arcology-elixir/arcology_db.org

Arcology DB Creates a Queryable Cache of Knowledge

• Rebuilding the Arcology DB within Elixir
• Mix Tasks
• arcology-batch sets up a batch builder

I organize my personal Knowledge Base in org-roam and org-roam creates a SQLite cache to achieve its performance and dynamicity. I extend this further, adding caching of page keywords, tracking the upstream code but for now a fork of it. I would love to reintegrate this code some time. This elisp code is based on Jethro Kuan's org-roam code and shares copyright with him.

Copyright © 2020 Jethro Kuan <jethrokuan95@gmail.com> Copyright © 2020 Ryan Rix <ryan@whatthfuck.computer>

Right now, it's assumed that this code will run on the server which hosts the Elixir service, but that is not a hard requirement – it could be generated locally and then the sqlite3 file pushed to the server with Syncthing or so.

Rebuilding the Arcology DB within Elixir

The org-roam parser runs in Emacs and the Arcology does not. This is squared away by starting the Emacs parser using a shell-command.

This code is cool because it has three different types of string escaping in it. Good luck making sense of that!

defmodule Arcology.DbCommands do
  def build_command(), do: _build_command(false)
  def build_command(full_build: fb), do: _build_command(fb)

  defp _build_command(full_build) do
    arc_dir = Application.get_env(:arcology, :env)[:arcology_directory]

    org_roam_dir = Application.get_env(:arcology, :env)[:org_roam_source]

    db_location = Application.get_env(:arcology, :env)[:database]

    lisp_dir = to_string(:code.priv_dir(:arcology)) <> "/lisp"

    build_arg =
      case full_build do
        true -> "t"
        false -> "nil"
      end

    [
      ~s<pushd >, lisp_dir, ~s<; emacs -Q --batch >,
      ~s<--eval '(setq org-roam-source-directory ">, org_roam_dir, ~s<")' >,
      ~s<--eval '(setq org-roam-directory ">, arc_dir, ~s<")' >,
      ~s<--eval '(setq org-roam-db-location (expand-file-name ">, db_location, ~s<" ">, File.cwd!, ~s<"))' >,
      ~s<-l arcology-batch.el >,
      ~s<--eval '(org-roam-db-build-cache >, build_arg ,~s<)' >
    ]
    |> Enum.join() 
  end
end

Mix Tasks

These tasks use that build command and send it to Mix.shell(), easy.

The first rebuilds the DB from scratch:

defmodule Mix.Tasks.Arcology.Build do
  use Mix.Task

  @shortdoc "Create the arcology database"
  @impl Mix.Task
  def run(_) do
    Mix.Tasks.Arcology.build_command(full_build: true)
    |> Mix.shell().cmd()
  end
end

Update an existing DB:

defmodule Mix.Tasks.Arcology.Update do
  use Mix.Task

  @shortdoc "Update the arcology database"
  @impl Mix.Task
  def run(_) do
    Mix.Tasks.Arcology.build_command(full_build: false)
    |> Mix.shell().cmd()
  end
end

These mix tasks start Emacs with no configuration loaded before jamming its own configuration in, arcology-batch

arcology-batch sets up a batch builder

This relies on code from my Complete Computing Environment, i'm not sure what to do about that for now, but I don't think it matters so much. This elisp is a "batch" file to be run by the mix tasks, loading the required libraries and then configuring org-roam minimally. org-roam-source-directory, and optionally org-roam-directory is specified in Arcology configuration and shoved in using the Mix Task above.

(unless (boundp 'org-roam-directory)
  (setq org-roam-directory (file-truename "~/org/")))

(load-file (expand-file-name "cce/packaging.el" org-roam-directory))

(add-to-list 'load-path default-directory)
(add-to-list 'load-path org-roam-source-directory)

(use-package dash)
(use-package f)
(use-package s)
(use-package emacsql)
(use-package emacsql-sqlite3)
(require 'subr-x)
(require 'cl)

(require 'org-roam)
(require 'org-roam-keywords)

(setq org-roam-cached-keywords
      '("ARCOLOGY_RSS"
        "ARCOLOGY_KEY"
        "ARCOLOGY_TWITTER"
        "ARCOLOGY_FACEBOOK"
        "ARCOLOGY_OLD_PERMALINK"))