The incident

During the incident, our service appeared to be partially functional. While we were still receiving and responding to some RPC requests, others were failing due to errors like this:

** (RuntimeError) could not lookup Ecto repo Platform.Repo because it was not started or it does not exist
    (ecto 3.10.3) lib/ecto/repo/registry.ex:22: Ecto.Repo.Registry.lookup/1
    (ecto 3.10.3) lib/ecto/repo/supervisor.ex:160: Ecto.Repo.Supervisor.tuplet/2
    (platform 0.0.1) lib/repo.ex:6: Platform.Repo.all/2

That means we could only respond to requests that didn’t interact with Platform.Repo, however that error still didn’t tell us much, except that the Platform.Repo we were looking for in the registry couldn’t be found for some reason while handling an RPC call. But why was this happening?

Root cause

Upon further investigation we’ve found logs below from that service’s pods.

Application platform exited: shutdown

That meant our application platform, one of our RPC service’s dependencies, which manages Platform.Repo had shut down. But if platform was down, how was our service still running? The answer lay in how the application was started. Since this was one of our legacy apps, it was still using Mix (instead of releases) to start the server on production. Specifically we were using a task to start the application and all its dependencies like this:

Application.ensure_all_started(:our_legacy_rpc)

Here is the key difference: In a Mix release, applications are started with a restart_type of permanent. However when using Application.ensure_all_started/1, the restart_type defaults to temporary. According to the documentation:

:temporary - if app terminates, it is reported but no other applications are terminated (the default).

This explains why our service kept running even though one of its critical dependencies (platform) had shut down. Erlang does nothing about shutdown applications, which makes sense, since the top-level supervisor of the application will attempt to restart its children up to max_retries within max_seconds (source) before giving up and eventually shutting down the whole application.

Resolution

To mitigate the issue we basically started all applications with a restart_type of permanent like so:

Application.ensure_all_start(:our_legacy_rpc, type: :permanent)

This guarantees that, if any dependency of our application (or the application itself) goes down for any reason, everything will be shutdown which will trigger Kubernetes to restart the container because of failing liveness checks.

:permanent - if app terminates, all other applications and the entire node are also terminated.

Just to note it here, to see restart_types of all started applications, you can run:

:application_controller.info()

Sources

Some links that helped me understand how Erlang manages applications and the role of restart_type.

• http://erlang.org/documentation/doc-5.2/doc/design_principles/applications.html
• https://blog.plataformatec.com.br/2015/04/build-embedded-and-start-permanent-in-elixir-1-0-4/
• https://stackoverflow.com/a/44665998/4796762
• https://stackoverflow.com/a/39662718/4796762

(Postgrex.Error ERROR 55000 (object_not_in_prerequisite_state) recovery is in progress

    hint: WAL control functions cannot be executed during recovery.)

In that consumer module, there was a function which was eventually trying to get the LSN from the DB by querying something like

SELECT pg_last_wal_replay_lsn()::varchar

It turns out we were actually querying a replica where WAL was enabled, figured that out quite quickly by asking in #postgresql IRC. (Well, if you’re querying the LSN it obviously means you’ve got WAL enabled, it’s even in the function name! But didn’t know it back then. ¯_(ツ)/¯_)

10:49 < Berge> Note that a replica (using WAL replication) is considered to be in recovery

Thank you Berge! TIL!

It is possible however for Firefox you need to add the preference below to your about:config and set it to true.

browser.urlbar.update2.engineAliasRefresh

Then you can go to Settings > Search and you’ll be able to see the “Add” button under the section “Search Shortcuts”. You can see an example of how to add a custom search engine for YouTube, which means that you’ll be able to search on YouTube simply by typing into your URL bar after you hit the shortcut “yt” while the focus is on the URL bar.

We have a CSV file that has some defaults for our business logic, the file barely changes and is not huge. The content of the file is transformed into a list of tuples using a script and the script is included in one of the modules as comments. Using this script (in an iex shell), you would get the output, which is a list of tuples, and paste it into appropriate function body, which would be used at runtime. Whenever the CSV file changes you would do the same things described above.

So the script looks something like this

# example CSV just in case you want to run this script.
# Just copy & paste the values below into some CSV file.
#
# service_name,base_price,duration
# Haircut,50,30
# Manicure,50,60
# Pedicure,70,90

"./lib/data.csv"
|> File.stream!()
|> CSV.decode!(headers: true)
|> Enum.map(fn row ->
  {
    row["service_name"],
    row["duration"],
    row["base_price"]
  }
end)

When run, this script outputs a list of tuples shown below.

[
  {"Haircut", "30", "50"},
  {"Manicure", "60", "50"},
  {"Pedicure", "90", "70"}
]

You would then take the list above and put it inside one of the functions, which would be used at runtime on production for whatever logic and the final module would look something like this.

defmodule Playground do
  # Script to use whenever necessary to
  # generate the body of some_important_function/0
  #
  # "/path/to/csv"
  # |> File.stream!()
  # |> CSV.decode!(headers: true)
  # |> Enum.map(fn row ->
  #   {
  #     row["service_name"],
  #     row["duration"],
  #     row["base_price"]
  #   }
  # end)

  def some_important_function do
    [
      {"Haircut", "30", "50"},
      {"Manicure", "60", "50"},
      {"Pedicure", "90", "70"}
    ]
  end
end

Everything is fine so far, except that you have to run the script manually every time the CSV file changes. It works anyway. The problem started when we wanted to translate some of the strings from the CSV file, which means we would need to wrap some of the strings with gettext/1 macro.

So the outcome we want to achieve is to have some_important_function as

def some_important_function do
  [
    {gettext("Haircut"), "30", "50"},
    {gettext("Manicure"), "60", "50"},
    {gettext("Pedicure"), "90", "70"}
  ]
end

We can’t directly wrap our strings with gettext/1 in our “script” above because,

1. The strings are not known at compile time, hence can’t be used with macros.
2. Moreover, it’s just a script and it’s not actually on production.

One solution would be to manually wrap the necessary strings with gettext/1 inside some_important_function/0 above. But that’s too much manual work and error prone.

Solving the problem at compile time with Elixir AST

Doing some research and asking around in Elixir Slack (shout out to @lostkobrakai), I figured we could solve the problem using Elixir AST.

What is the AST?

An AST, acronym for Abstact Syntax Tree, is a representation of code as a data structure. This data structure is used by Elixir to run Elixir code: either by interpreting it directly: executing the commands in the AST by recursively walking it; or by compiling it: translating the AST into another format, namely BEAM bytecode instructions, which then are saved to disk as .beam files.^{source: botsquad}

We will first read the CSV at compile time and create the tuple list, which will then be used to create an AST with gettext/1 macro used wherever necessary. Lastly we will unquote/1 this AST within our actual function body which will be used at runtime on production. Moreover, we will use @external_resource module attribute to make sure that whenever our CSV file changes our module will be recompiled, meaning that our code will always be up to date! No more running scripts manually. 🎉

@external_resource

Tools may use this information to ensure the module is recompiled in case any of the external resources change, see for example: mix compile.elixir. ^{source: Hexdocs}

defmodule Playground do
  # This way our module will be recompiled whenever
  # you make changes to the CSV file.
  # Make some changes to the CSV file and recompile to see it in action.
  @external_resource "./lib/foo.csv"

	# Nothing fancy here, we're just creating a list of tuples
  data =
    @external_resource
    |> File.stream!()
    |> CSV.decode!(headers: true)
    |> Enum.map(fn row ->
      {
        row["service_name"],
        row["duration"],
        row["base_price"]
      }
    end)

  IO.inspect(data)
  # =>
  # [
  #   {"Haircut", "30", "50"},
  #   {"Manicure", "60", "50"},
  #   {"Pedicure", "90", "70"}
  # ]

  # This is where we build the AST.
  data_ast =
    Enum.map(data, fn tuple ->
      list =
        tuple
        |> put_elem(0, {:gettext, [], [elem(tuple, 0)]})
        |> Tuple.to_list()

      {:{}, [], list}
    end)

  IO.inspect(data_ast)
  # =>
  # [
  #   {:{}, [], [{:gettext, [], ["Haircut"]}, "30", "50"]},
  #   {:{}, [], [{:gettext, [], ["Manicure"]}, "60", "50"]},
  #   {:{}, [], [{:gettext, [], ["Pedicure"]}, "90", "70"]}
  # ]

  data_ast
  |> Macro.to_string()
  |> IO.puts()
  # =>
  # [
  #   {gettext("Haircut"), "30", "50"},
  #   {gettext("Manicure"), "60", "50"},
  #   {gettext("Pedicure"), "90", "70"}
  # ]

  # Now, this function would return the same thing
  # that's returned by Macro.to_string/1 above
  def some_important_function do
    unquote(data_ast)
  end
end

When you start an Agent or GenServer they will be linked to the current process and the current process will crash if the linked Agent or GenServer crashes. However Supervisors do not crash even if the child processes that are linked to the supervisor crash. Well, the answer is obvious.

Because Supervisors trap the exit signals.^{source code}

Assuming that you’ve at least one child spec in your children list below, you can also confirm that doing:

{:ok, supervisor} = Supervisor.start_link(children, strategy: :one_for_one)

Process.info(supervisor, :trap_exit)
# => {:trap_exit, true}

I was wondering how Ecto.Changeset.cast/4 function casts attributes from a map into their approriate types in the DB since we do not provide any information about the types explicitly.

Turns out that if you are passing a schema struct to cast/4 Ecto uses the __changeset__ function on it to figure out the types of the fields. Otherwise if you are passing an Ecto.Changeset to cast/4 then Ecto uses the types field of the changeset struct.

To show you what I mean, we are going to create a Phoenix app called God with only a Human schema so we can create, read, update, delete a Human (wish it was that easy lol).

If you already know what I mean then you can basically skip to the breakdown.

Create a Phoenix app:

$ mix phx.new god

After our project is created, and we ran mix ecto.create inside the project directory we need to generate our controller, view and context for Human to be able to CRUD Humans.

╰─$ mix phx.gen.html Creature Human humans name surname to_be_born_on:utc_datetime age_to_die_at:integer will_get_married:boolean

* creating lib/god_web/controllers/human_controller.ex
* creating lib/god_web/templates/human/edit.html.eex
* creating lib/god_web/templates/human/form.html.eex
* creating lib/god_web/templates/human/index.html.eex
* creating lib/god_web/templates/human/new.html.eex
* creating lib/god_web/templates/human/show.html.eex
* creating lib/god_web/views/human_view.ex
* creating test/god_web/controllers/human_controller_test.exs
* creating lib/god/creature/human.ex
* creating priv/repo/migrations/20210716083409_create_humans.exs
* creating lib/god/creature.ex
* injecting lib/god/creature.ex
* creating test/god/creature_test.exs
* injecting test/god/creature_test.exs

Add the resource to your browser scope in lib/god_web/router.ex:

    resources "/humans", HumanController


Remember to update your repository by running migrations:

    $ mix ecto.migrate

Basically our Human will have a name, surname and we will also have a utc_datetime for them to_be_born_on, an age_to_die_at and finally whether they will_get_married.

After we follow the instructions given after running mix phx.gen.html above we will have a working God app. :D

We’ve already created (scheduled maybe?) a human to be born on 05/05/2022 at 07:07 who will sadly die at the age of 88. 🎉

Let’s get serious

So when we navigate to http://localhost:4000/humans/new we get the form below to create a human.

After we fill in the details in the form and click the save button, our request follows a path of

endpoint.ex -> router.ex -> HumanController.create/2

Below you can see the code for HumanController.create/2 which gets run eventually,

def create(conn, %{"human" => human_params}) do
  case Creature.create_human(human_params) do
    {:ok, human} ->
      conn
      |> put_flash(:info, "Human created successfully.")
      |> redirect(to: Routes.human_path(conn, :show, human))

    {:error, %Ecto.Changeset{} = changeset} ->
      render(conn, "new.html", changeset: changeset)
  end
end

which in fact calls a function Creature.create_human/1 which you can see below.

def create_human(attrs \\ %{}) do
  %Human{}
  |> Human.changeset(attrs)
  |> Repo.insert()
end

This function also calls Human.changeset/2 which you can see below.

def changeset(human, attrs) do
  human
  |> cast(attrs, [:name, :surname, :to_be_born_on, :age_to_die_at, :will_get_married])
  |> validate_required([:name, :surname, :to_be_born_on, :age_to_die_at, :will_get_married])
end

If we insert a require IEx; IEx.pry() after the function head and before the function end like below,

def changeset(human, attrs) do
  require IEx; IEx.pry()
  human = human
  |> cast(attrs, [:name, :surname, :to_be_born_on, :age_to_die_at, :will_get_married])
  |> validate_required([:name, :surname, :to_be_born_on, :age_to_die_at, :will_get_married])
  require IEx; IEx.pry()
  human
end

and inspect the attrs map. We will get:

pry(1)> attrs
%{
  "age_to_die_at" => "99",
  "name" => "John",
  "surname" => "Doe",
  "to_be_born_on" => %{
    "day" => "1",
    "hour" => "0",
    "minute" => "0",
    "month" => "1",
    "year" => "2016"
  },
  "will_get_married" => "true"
}

As you can see all the values are of string which is what we can only get after a form submission anyway.

If we also inspect the human after the piping we get the changeset below.

pry(1)> human
#Ecto.Changeset<
  action: nil,
  changes: %{
    age_to_die_at: 99,
    name: "John",
    surname: "Doe",
    to_be_born_on: ~U[2016-01-01 00:00:00Z],
    will_get_married: true
  },
  errors: [],
  data: #God.Creature.Human<>,
  valid?: true
>

The question is…

So here, how does Ecto.Changeset.cast/4 know how to cast attributes into their DB appropriate types? We have only passed Ecto.Changeset.cast/4 a Human schema struct and a map of attributes consisting of only string values.

The Breakdown

The breakdown will be two-fold. One for when you are passing a schema struct to cast/4 and the other one for when you are passing a Ecto.Changeset to cast/4.

In the case, when you pass a schema struct to Ecto.Changeset.cast/4, it turns out that there’s a __changeset__ function you can call on a schema struct which will give you

iex(12)> God.Creature.Human.__changeset__
%{
  age_to_die_at: :integer,
  id: :id,
  inserted_at: :naive_datetime,
  name: :string,
  surname: :string,
  to_be_born_on: :utc_datetime,
  updated_at: :naive_datetime,
  will_get_married: :boolean
}

and this __changeset__ function gets injected by a private schema/4 function which is called by schema and embedded_schema macros. You can take a look at them on GitHub.

If you further inspect the Ecto.Changeset.cast/4 function which accepts a struct, you will see that the __changeset__ function is being used like so:

def cast(%{__struct__: module} = data, params, permitted, opts) do
  cast(data, module.__changeset__(), %{}, params, permitted, opts)
end

If you create your own struct and try to pass it to Ecto.Changeset.cast/4 you will obviously get an error.

iex(18)> defmodule Foo do
...(18)> defstruct [:name, :age]
...(18)> end
{:module, Foo,
 <<70, 79, 82, 49, 0, 0, 6, 152, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 180,
   0, 0, 0, 18, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 8, 95, 95,
   105, 110, 102, 111, 95, 95, 10, 97, 116, ...>>, %Foo{age: nil, name: nil}}

iex(19)> Ecto.Changeset.cast(%Foo{}, %{name: "Burak"}, [:name])
** (UndefinedFunctionError) function Foo.__changeset__/0 is undefined or private
    Foo.__changeset__()
    (ecto 3.6.2) lib/ecto/changeset.ex:489: Ecto.Changeset.cast/4

In the case, when you pass an Ecto.Changeset to Ecto.Changeset.cast/4 below function gets invoked.

def cast(%Changeset{changes: changes, data: data, types: types, empty_values: empty_values} = changeset,
		    params, permitted, opts) do
  opts = Keyword.put_new(opts, :empty_values, empty_values)
  new_changeset = cast(data, types, changes, params, permitted, opts)
  cast_merge(changeset, new_changeset)
end

As you can also see below, there’s a field called types in an Ecto.Changeset hence when you pass an Ecto.Changeset to cast/4, Ecto will use the types field to figure out how to cast attributes properly.

iex(63)> ch = God.Creature.Human.changeset(%God.Creature.Human{}, %{name: "John", surname: "Doe", to_be_born_on: "2022-05-05 00:00:00", age_to_die_at: "99"})
#Ecto.Changeset<
  action: nil,
  changes: %{
    age_to_die_at: 99,
    name: "John",
    surname: "Doe",
    to_be_born_on: ~U[2022-05-05 00:00:00Z]
  },
  errors: [],
  data: #God.Creature.Human<>,
  valid?: true
>
iex(64)> ch.types
%{
  age_to_die_at: :integer,
  id: :id,
  inserted_at: :naive_datetime,
  name: :string,
  surname: :string,
  to_be_born_on: :utc_datetime,
  updated_at: :naive_datetime,
  will_get_married: :boolean
}

Lastly

You don’t always need a schema struct or an Ecto.Changeset to be able to use cast/4. As you can see in lib/ecto/changeset.ex

def cast({data, types}, params, permitted, opts) when is_map(data) do
  cast(data, types, %{}, params, permitted, opts)
end

You can just pass a tuple consisting of a data map and a types map which has the types of the values like so:

iex(15)> data = %{}
%{}
iex(16)> types = %{name: :string, age: :integer, likes_elixir: :boolean}
%{age: :integer, likes_elixir: :boolean, name: :string}
iex(17)> Ecto.Changeset.cast({ %{}, types}, %{name: "Burak", age: "99", likes_elixir: "true"}, [:name, :age, :likes_elixir])
#Ecto.Changeset<
  action: nil,
  changes: %{age: 99, likes_elixir: true, name: "Burak"},
  errors: [],
  data: %{},
  valid?: true
>

map ,P Orequire IEx; IEx.pry()<ESC>
map ,p orequire IEx; IEx.pry()<ESC>

After inserting above code into your .vimrc you can basically insert pry line into your code with the keyboard shortcuts ,p and ,P. Uppercase version of the shortcut inserts the line above the current line and lowercase version inserts the line below the current line.

If you take a look at assets/js/app.js, we can see that we call

Video.init(socket, document.getElementById("video"))

If you inspect where Video.init is defined, which is assets/js/video.js, you can see the snippet below,

let Video = {
  init(socket, element) { if(!element) { return }
    let playerId = element.getAttribute("data-player-id")
    let videoId  = element.getAttribute("data-id")
    socket.connect()
    Player.init(element.id, playerId, () => {
      this.onReady(videoId, socket)
    })
  },

What the Video.init function does is, if the element doesn’t exist on the page we directly return without trying to connecting to the socket. However if you basically insert a div with an id video on any page, you can see that the browser will try to connect to a websocket just because we have the video element on the page.

In templates/watch/show.html.eex you can see the snippet below which basically creates an element with an id of video, hence, the browser tries to connect to a socket on the video page.

<%= content_tag :div, id: "video", data: [id: @video.id, player_id: player_id(@video)] do %>
<% end %>

These operators expect true or false as their first argument:

a or b	# true if a is true; otherwise b
a and b # false if a is false; otherwise b
not a	# false if a is true; otherwise b

These operators take arguments of any type. Any value apart from nil or false is interpreted as true:

a || b	# a if a is truthy; otherwise b
a && b	# b if a is truthy; otherwise a
!a	# false if a is truthy; otherwise true

Examples:

iex(1)> 1 or 2
** (BadBooleanError) expected a boolean on left-side of "or", got: 1

iex(1)> 1 || 2
1

iex(1)> true or false
true

iex(2)> "foo" and "bar"
** (BadBooleanError) expected a boolean on left-side of "and", got: "foo"

iex(2)> "foo" && "bar"
"bar"

iex(3)> not "foo"
** (ArgumentError) argument error
    :erlang.not("foo")

iex(3)> !"foo"
false

In file included from binder.cpp:20:
./project.h:119:10: fatal error: 'openssl/ssl.h' file not found
#include <openssl/ssl.h>
         ^~~~~~~~~~~~~~~
1 error generated.
make: *** [binder.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/burak/.rvm/gems/ruby-3.0.1/gems/eventmachine-1.2.7 for inspection.

That was the case for me when I was trying to install jekyll gem using gem install jekyll.

Solution is to do:

gem install jekyll -- --with-cppflags=-I/opt/homebrew/opt/openssl/include