arcology-elixir/inotify-tools.org

Rebuilding Arcology DB Automatically

• Running the build commands
• State Transitions
• Footnotes

Here we weld arcology-db's shell calls to emacs to the file_system module, to signal Arcology to update its database whenever I change the files and Syncthing gets it to my server. Until I work through documenting my goal-state and doing some amount of risk/threat modeling in Arcology Publishing Workflow, this is going to generate the DB on file-updates automatically. It'll be its own OTP application implemented with GenServer that is almost entirely set apart from the rest of the system.

I have to think about how to design this thing, optimally I would prefer it not to spawn 150 Emacs works whenever my laptop connects to Wi-Fi, after all, so I need to build in some sort of rate-limiter or cooldown period in to this. I love building "distributed systems." I'm going to want to build another rate-limiter, a Plug and so-called "anti-viral" rate-limiter designed to keep my pages from "going viral" when I'm not paying attention. Luckily both of these rate limiters will opt to "drop" the traffic that is over-budget and ask them to try again later.

This is using the code from Arcology.DbCommands.

defmodule Arcology.FileNotifier do
  use GenServer
  
  def start_link(args) do
    GenServer.start_link(__MODULE__, args)
  end

  @impl true
  def init(_) do
    dir = Application.get_env(:arcology, :env)[:arcology_directory]
    {:ok, watcher_pid} = FileSystem.start_link(dirs: [dir], recursive: true)
    FileSystem.subscribe(watcher_pid)
    Process.send_after(self(), :timer_fire, 1000)
    {:ok, %{watcher_pid: watcher_pid, last_touch: nil, last_build: DateTime.utc_now()}}
  end

  <<filesystem_watcher>>
  <<check-and-build>>
  <<build-commands>>
  <<build-message>>
  <<build-helper>>
end

Running the build commands

These just call the functions in arcology-db to generate the command and then run 'em. These should probably be called from an Async task!

def update_db() do
  cmd = Arcology.DbCommands.build_command()
  System.cmd("bash", ["-c", cmd])
end

def build_db() do
  cmd = Arcology.DbCommands.build_command(full_build: true)
  System.cmd("bash", ["-c", cmd])
end

State Transitions

This is kind of finicky, to be honest, maybe even "janky"… The first message is when no files have been touched yet. The second message contains all the logic of checking when we should build, and doing it. The last message is returned by the supervised task in the inside of the second message handler, and it updates the last build time.

There is a timer which tries to run every second to check if any files have been touched. If no files have been touched since the last rebuild (or application startup), the system will yield for 1 second, and fire again.

@impl true
def handle_info(:timer_fire, %{last_touch: touch} = state)
    when is_nil(touch) do
  Process.send_after(self(), :timer_fire, 1000)
  {:noreply, state}
end

The FileSystem watcher set up and subscribed to by the init function will set last_touch on the state object to the current time, whenever an org file is touched.

@impl true
def handle_info({:file_event, _watched_pid, {path, _events}}, state) do
  if path |> String.ends_with?(".org") do
    {:noreply, %{state|last_touch: DateTime.utc_now()}}
  else
    {:noreply, state}
  end
end

This message handler checks if the "debounce" has waited at least 5 seconds since the last file was touched, and also checks that it has been at least 60 seconds since the last time a build was completed to keep the thing from running away and using a bunch of resources as I actively edit. The update itself runs in its own task which is not supervised by this GenServer but by the application itself in Arcology.TaskSupervisor defined in Arcology.Application.start/2.

@impl true
def handle_info(:timer_fire, %{last_touch: touch, last_build: build} = state) do
  now = DateTime.utc_now()
  touch_diff = DateTime.diff(now, touch)
  build_diff = DateTime.diff(now, build||~U[2020-01-01 12:00:00Z])

  Process.send_after(self(), :timer_fire, 1000)

  if touch_diff > 5 do
    if build_diff > 60 do
      Task.Supervisor.async_nolink(Arcology.TaskSupervisor, fn ->
        {:build_done, update_db()}
      end)

      {:noreply, %{state|last_touch: DateTime.utc_now()}, 2000}
    else
      {:noreply, state, 2000}
    end
  else
    {:noreply, state, 2000}
  end
end

When the update task finishes, a :build_done message is sent to the GenServer, which is used to reset the last_touch and last_build keys.

require Logger

def handle_info({_ref, {:build_done, {build_output, exit_code}}}, state) do
  Logger.debug("emacs exits #{exit_code}")
  Logger.debug(build_output)
  {:noreply, %{state | last_touch: nil, last_build: DateTime.utc_now()}}
end

Additionally we have a :timeout handler which attempts to re-start the loop if its lost. I am not sure how to design this better right now.

def handle_info(:timeout, state) do
  Process.send_after(self(), :timer_fire, 1000)
  {:noreply, state, 2000}
end

These handlers receive messages from the Supervisor, I am not 100% sure how to handle them yet:

def handle_info({:DOWN, _ref, _pid, _, _}, state) do
  {:noreply, state, 2000}
end

It is possible to trigger a full database build rather than an incremental update by using Arcology.FileNotifier.build/1; getting the pid of this GenServer from Supervisor.which_children isn't so hard.¹

def build(pid) do
  GenServer.cast(pid, :build)
end
  
def handle_cast(:build, state) do
  Task.Supervisor.async_nolink(Arcology.TaskSupervisor, fn ->
    {:build_done, build_db()}
  end)
  {:noreply, state}
end

Footnotes

---

¹

Shout to mixing elem/2 and Enum.at/2:

Supervisor.which_children(Arcology.Supervisor)
|> Enum.filter(fn sup -> elem(sup, 0) == Arcology.FileNotifier end)
|> Enum.at(0)
|> elem(1)
|> Arcology.FileNotifier.build()