arcology-elixir/inotify-tools.org

#+TITLE: Rebuilding Arcology DB Automatically
#+ROAM_TAGS: Arcology
#+ROAM_ALIAS: Arcology.FileNotifier "Arcology inotify"

#+PROPERTY: header-args            :mkdirp yes :results none

#+ARCOLOGY_KEY: arcology/inotify

Here we weld [[file:arcology_db.org][arcology-db]]'s shell calls to =emacs= to the [[https://github.com/falood/file_system/][file_system]] module, to signal Arcology to update its database whenever I change the files and [[file:../cce/syncthing.org][Syncthing]] gets it to my server. Until I work through documenting my goal-state and doing some amount of risk/threat modeling in [[file:publishing.org][Arcology Publishing Workflow]], this is going to generate the DB on file-updates automatically. It'll be its own OTP application implemented with [[file:../genserver_elixir_v1_11_2.org][GenServer]] that is almost entirely set apart from the rest of the system.

I have to think about how to design this thing, optimally I would prefer it not to spawn 150 Emacs works whenever my laptop connects to Wi-Fi, after all, so I need to build in some sort of rate-limiter or cooldown period in to this. I love building "distributed systems." I'm going to want to build another rate-limiter, a =Plug= and so-called "anti-viral" rate-limiter designed to keep my pages from "going viral" when I'm not paying attention. Luckily both of these rate limiters will opt to "drop" the traffic that is over-budget and ask them to try again later.

This is using the code from [[file:arcology_db.org][Arcology.DbCommands]].

#+begin_src elixir :tangle lib/arcology/file_notifier.ex :mkdirp yes :noweb yes
defmodule Arcology.FileNotifier do
  use GenServer
  
  def start_link(args) do
    GenServer.start_link(__MODULE__, args)
  end

  @impl true
  def init(_) do
    dir = Application.get_env(:arcology, :env)[:arcology_directory]
    {:ok, watcher_pid} = FileSystem.start_link(dirs: [dir], recursive: true)
    FileSystem.subscribe(watcher_pid)
    Process.send_after(self(), :timer_fire, 1000)
    {:ok, %{watcher_pid: watcher_pid, last_touch: nil, last_build: DateTime.utc_now()}}
  end

  <<filesystem_watcher>>
  <<check-and-build>>
  <<build-commands>>
  <<build-message>>
  <<build-helper>>
end
#+end_src

* Running the build commands

These just call the functions in [[file:arcology_db.org][arcology-db]] to generate the command and then run 'em. These should probably be called from an Async task!

#+begin_src elixir :noweb-ref build-commands
def update_db() do
  cmd = Arcology.DbCommands.build_command()
  System.cmd("bash", ["-c", cmd])
end

def build_db() do
  cmd = Arcology.DbCommands.build_command(full_build: true)
  System.cmd("bash", ["-c", cmd])
end
#+end_src

* State Transitions

This is kind of finicky, to be honest, maybe even "janky"... The first message is when no files have been touched yet. The second message contains all the logic of checking when we should build, and doing it. The last message is returned by the supervised task in the inside of the second message handler, and it updates the last build time.

There is a timer which tries to run every second to check if any files have been touched. If no files have been touched since the last rebuild (or application startup), the system will yield for 1 second, and fire again.

#+begin_src elixir :noweb-ref check-and-build
@impl true
def handle_info(:timer_fire, %{last_touch: touch} = state)
    when is_nil(touch) do
  Process.send_after(self(), :timer_fire, 1000)
  {:noreply, state}
end
#+end_src

The FileSystem watcher set up and subscribed to by the =init= function will set =last_touch= on the state object to the current time, whenever an =org= file is touched.

#+begin_src elixir :noweb-ref filesystem_watcher
@impl true
def handle_info({:file_event, _watched_pid, {path, _events}}, state) do
  if path |> String.ends_with?(".org") do
    {:noreply, %{state|last_touch: DateTime.utc_now()}}
  else
    {:noreply, state}
  end
end
#+end_src

This message handler checks if the "debounce" has waited at least 5 seconds since the last file was touched, and also checks that it has been at least 60 seconds since the last time a build was completed to keep the thing from running away and using a bunch of resources as I actively edit. The update itself runs in its own task which is not supervised by this GenServer but by the application itself in =Arcology.TaskSupervisor= defined in [[file:arcology.org][Arcology.Application.start/2]]. 

#+begin_src elixir :noweb-ref check-and-build
@impl true
def handle_info(:timer_fire, %{last_touch: touch, last_build: build} = state) do
  now = DateTime.utc_now()
  touch_diff = DateTime.diff(now, touch)
  build_diff = DateTime.diff(now, build||~U[2020-01-01 12:00:00Z])

  Process.send_after(self(), :timer_fire, 1000)

  if touch_diff > 5 do
    if build_diff > 60 do
      Task.Supervisor.async_nolink(Arcology.TaskSupervisor, fn ->
        {:build_done, update_db()}
      end)

      {:noreply, %{state|last_touch: DateTime.utc_now()}, 2000}
    else
      {:noreply, state, 2000}
    end
  else
    {:noreply, state, 2000}
  end
end 
#+end_src

When the update task finishes, a =:build_done= message is sent to the GenServer, which is used to reset the =last_touch= and =last_build= keys.

#+begin_src elixir :noweb-ref check-and-build
require Logger

def handle_info({_ref, {:build_done, {build_output, exit_code}}}, state) do
  Logger.debug("emacs exits #{exit_code}")
  Logger.debug(build_output)
  {:noreply, %{state | last_touch: nil, last_build: DateTime.utc_now()}}
end
#+end_src

Additionally we have a =:timeout= handler which attempts to re-start the loop if its lost. I am not sure how to design this better right now.

#+begin_src elixir :noweb-ref check-and-build
def handle_info(:timeout, state) do
  Process.send_after(self(), :timer_fire, 1000)
  {:noreply, state, 2000}
end
#+end_src

These handlers receive messages from the Supervisor, I am not 100% sure how to handle them yet:

#+begin_src elixir :noweb-ref check-and-build
def handle_info({:DOWN, _ref, _pid, _, _}, state) do
  {:noreply, state, 2000}
end
#+end_src

It is possible to trigger a full database build rather than an incremental update by using =Arcology.FileNotifier.build/1=; getting the pid of this GenServer from =Supervisor.which_children= isn't so hard.[fn:1]

#+begin_src elixir :noweb-ref build-message
def build(pid) do
  GenServer.cast(pid, :build)
end
  
def handle_cast(:build, state) do
  Task.Supervisor.async_nolink(Arcology.TaskSupervisor, fn ->
    {:build_done, build_db()}
  end)
  {:noreply, state}
end
#+end_src

* Footnotes

[fn:1] 

Shout to mixing =elem/2= and =Enum.at/2=:

#+begin_src elixir
Supervisor.which_children(Arcology.Supervisor)
|> Enum.filter(fn sup -> elem(sup, 0) == Arcology.FileNotifier end)
|> Enum.at(0)
|> elem(1)
|> Arcology.FileNotifier.build()
#+end_src