rrix
/
arcology
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
arcology/localapi.org

22 KiB
Raw Permalink Blame History

A Localhost API for the Arcology

By moving the database out of EmacSQL and in to Django, I have hobbled some of the Arcology's Knowledge Management Living Systems and meta-cognitive skills. arroyo-db-query no longer existing means that things like the Direnv arroyo-db integration stopped working, and the helpers in My NixOS configuration and elsewhere rely on the stringly-typed Arcology generate Command and a nix run invocation to function.

I think it's worth building a small HTTP JSON API which can serve some of the common queries, to at least see if it's worth the trouble..

Here's how to use this from Emacs Lisp:

  • (arcology-fetch-localapi-bearer-token) fetches the bearer token file shared with the localapi deployment.
  • (arcology-localapi-call method path) is an API helper which given an API path will fetch the data with authorization and return a deserialized JSON structure

  • these interactive commands will fetch a URL, putting it on your kill ring or clipboard if you call it with M-x or equivalent:

    • (arcology-key-to-url page-key &optional heading-id) will take an ARCOLOGY_KEY and return a URL
    • (arcology-file-to-url file-path &optional heading-id) will do the same with a file path from (buffer-file-name) or so.
    • (arcology-url-at-point) gets the org-id of the heading your cursor is in if it has one, and makes a URL that links directly to that.
    • (arcology-read-url) pops up a list of all the org-roam headings, and returns a URL to it.
    • (arcology-api-generator) calls in to The Arroyo Generators and returns the string of files they generate

Server view and Arroyo Emacs lisp scaffolding

I probably should use Django REST Framework for this, but. I'm just gonna beat two stones together until a JSON API falls out. This will probably change the first time I add a POST call or want to do more complex auth stuff.

The API will be simple, with a bearer token provided by a local state file:

(use-package plz)

(defvar arcology-localapi-bearer-token nil)
(defun arcology-fetch-localapi-bearer-token ()
  (interactive)
  (let ((tok (or arcology-localapi-bearer-token
                 (getenv "ARCOLOGY_LOCALAPI_BEARER_TOKEN")
                 (thread-last
                   "~/sync/private-files/.arcology-env"
                   (format ". %s && echo $ARCOLOGY_LOCALAPI_BEARER_TOKEN")
                   (shell-command-to-string)
                   (s-chop-suffix "\n" )))))
    (when (or (called-interactively-p)
              (not arcology-localapi-bearer-token))
      (customize-save-variable 'arcology-localapi-bearer-token tok))
    tok))

(defcustom arcology-api-base "http://127.0.0.1:29543/api/v1"
  "localapi base url")

(defun arcology-localapi-call (method path &rest plz-args)
  (let ((plz-args (or plz-args '(:as json-read))))
    (apply 'plz method (format "%s%s" arcology-api-base path)
           :headers `(("Authorization" . ,(format "Bearer %s" (arcology-fetch-localapi-bearer-token))))
             plz-args)))

These are the API URLs and basic imports… nothing to concern yourself with, probably!

from django.contrib import admin
from django.urls import path, re_path, include

from localapi import views

urlpatterns = [
    path("", views.index),
    path("generate/<slug:module>/<slug:role>", views.generate, name="keyword_by_key"),
    path("generate/<slug:module>", views.generate, name="keyword_by_key"),
    path("file/by-tag/<slug:key>", views.files_by_tag, name="files_by_tag"),
    path("keywords/<slug:key>", views.keyword_by_key, name="keyword_by_key"),
    re_path("keywords/(?P<key>[0-9a-zA-Z_-]+)/(?P<value>[0-9a-zA-Z/_\-]+)", views.keyword_by_key_value, name="keyword_by_key"),
    re_path("page/(?P<route_key>[0-9a-zA-Z/_\-]+)", views.page_metadata, name="page_metadata"),
    re_path("file/(?P<file_path>[0-9a-zA-Z/_\-\.]+)", views.file_metadata, name="file_metadata"),
]
from django.http import HttpRequest, HttpResponse, HttpResponseNotFound, Http404, JsonResponse
from django.shortcuts import render, get_object_or_404

from arcology.models import Page, Feed, Site
from roam.models import Link
from localapi.auth import authenticated

from prometheus_client import Counter, Histogram

import logging
logger = logging.getLogger(__name__)

The API is fairly simple and built-to-purpose, we start with an endpoint that can be queried to validate that the API Bearer token is valid:

@authenticated
def index(request):
    return JsonResponse(dict(state="ok :)"))

DONE Local API Auth Middleware

  • State "DONE" from [2024-03-13 Wed 17:01]

This API should be secured from CSRF and local scanners, we'll just put a bearer token on it. Since the API is meant to only be running on a CCE endpoint, and not on The Wobserver the API authentication can be smoothed out a bit and hopefully made transparent to the user. The Bearer token is set in the Arcology Project Configuration, and can be overridden in an environment variable ARCOLOGY_LOCALAPI_BEARER_TOKEN. The NixOS endpoint configuration module for this API will include token generation and placement so that this should be transparent to a local command user. Decorating a view with @authenticated will add this bearer token authentication to the view.

import re
from django.conf import settings
from django.http import HttpRequest, JsonResponse

def authenticate_request(request: HttpRequest):
    r = re.compile(r'Bearer (\S+)')
    bearer = request.headers.get("Authorization", "")
    match = r.match(bearer)
    if not match:
        return False

    tok = match.group(1)
    if tok != settings.LOCALAPI_BEARER_TOKEN:
        return False

    return True

def authenticated(func):
    def wrapper(*args, **kwargs):
        request=args[0]
        if not authenticate_request(request):
            return JsonResponse(dict(state="no :("), status=401)
        return func(*args, **kwargs)
    return wrapper

Keyword metadata

There is a simple set of HTTP GET APIs to query the file/key/value store:

import roam.models

def _json_keywords(keywords):
    return [
            dict(path=kw.path.path, keyword=kw.keyword, value=kw.value)
            for kw in keywords
        ]

/keywords/{key}

GET http://127.0.0.1:8000/api/v1/keywords/ARCOLOGY_KEY

@authenticated
def keyword_by_key(request, key):
    keywords = roam.models.Keyword.objects.filter(keyword=key).all()
    return JsonResponse(dict(
        state="ok :)",
        key=key,
        keywords=_json_keywords(keywords),
    ))

/keywords/{key}/{value}

GET http://127.0.0.1:8000/api/v1/keywords/ARCOLOGY_KEY/arcology/localapi

@authenticated
def keyword_by_key_value(request, key, value):
    keywords = roam.models.Keyword.objects.filter(keyword=key, value=value).all()
    return JsonResponse(dict(
        state="ok :)",
        key=key,
        value=value,
        keywords=_json_keywords(keywords),
    ))

NEXT add elisp APIs

Page and File metadata

And some really simple APIs to get information about Page and File objects out of the database:

import arcology.models

def _json_page(page):
    return dict(
        title=page.title,
        url=page.to_url(),
        site=page.site.title,
        keywords={
            kw.keyword: kw.value
            for kw in page.collect_keywords().all()
        },
        tags=list(set([tag.tag for tag in page.collect_tags()])),
        references=[reference.ref for reference in page.collect_references()],
    )

GET http://127.0.0.1:8000/api/v1/page/arcology/localapi or this Emacs Lisp command: A command which will take an ARCOLOGY_KEY and the ID of a heading on that page, and return a URL for it:

(defun arcology-key-to-url (page-key &optional heading-id)
  (interactive "sKey: \nsHeading ID: ")
  (let ((url (concat
              (thread-last
                (arcology-localapi-call 'get (format  "/page/%s" page-key))
                (alist-get 'page)
                (alist-get 'url))
              (when heading-id (format "#%s" heading-id)))))
    (when (called-interactively-p)
      (kill-new url)
      (message "%s" url))
    url))
@authenticated
def page_metadata(request, route_key):
    page = arcology.models.Page.objects.get(route_key=route_key)
    keywords = page.file.keyword_set.all()
    return JsonResponse(dict(
        state="ok :)",
        route_key=route_key,
        file=page.file.path,
        page=_json_page(page),
        keywords=_json_keywords(keywords)
    ))

GET http://127.0.0.1:8000/api/v1/file/arcology-django/localapi.org and the Emacs Lisp command which will do the same:

(defun arcology-file-metadata (file-path)
  (thread-last
    (file-relative-name file-path org-roam-directory)
    (format "/file/%s")
    (arcology-localapi-call 'get)
    (alist-get 'file)))

(defun arcology-file-to-url (file-path &optional heading-id)
  (interactive "fKey: \nsHeading ID: ")
  (let ((url (concat
              (thread-last
                file-path
                (arcology-file-metadata)
                (alist-get 'page)
                (alist-get 'url))
              (when heading-id (format "#%s" heading-id)))))
    (when (called-interactively-p)
      (kill-new url)
      (message "%s" url))
    url))
from django.conf import settings

@authenticated
def file_metadata(request, file_path):
    final_path = settings.ARCOLOGY_BASE_DIR.joinpath(file_path)
    the_file = roam.models.File.objects.get(path=final_path)
    page = the_file.page_set.first()
    return JsonResponse(dict(
        state="ok :)",
        file_path=file_path,
        file=dict(
            path=the_file.path,
            page=_json_page(page),
        ),
    ))

Tag search

Get all files by tag 

(defun arcology-files-by-tag (tag)
  (thread-last
    tag
    (format "/file/by-tag/%s")
    (arcology-localapi-call 'get)
    (alist-get 'files)
    (mapcar #'identity)))
@authenticated
def files_by_tag(request, key):
    # final_path = settings.ARCOLOGY_BASE_DIR.joinpath(file_path)
    the_tags = roam.models.Tag.objects.filter(tag=key).all()
    return JsonResponse(dict(
        state="ok :)",
        tag=key,
        files=list(set([
            tag.heading.path.path
            for tag in the_tags
        ]))
    ))

Some more ELisp helpers 

(defun arcology-url-at-point (pom)
  (interactive "m")
  (let ((url (arcology-key-to-url
              (thread-last
                (org-collect-keywords '("ARCOLOGY_KEY"))
                (first)
                (second))
              (and (> (org-outline-level) 0)
                   (org-id-get)))))
    (when (called-interactively-p)
      (kill-new url)
      (message "%s" url))
    url))

A command which will get the URL for a heading you can select using your completing-read:

(defun arcology-read-url ()
  (interactive)
  (let* ((node (org-roam-node-read))
         (node-path (file-relative-name (org-roam-node-file node)
                                        org-roam-directory))
         (url (arcology-file-to-url node-path
                                    (and (> (org-outline-level) 0)
                                         (org-id-get)))))
    (when (called-interactively-p)
      (kill-new url)
      (message "%s" url))))

DONE Arcology generate Command

  • State "DONE" from "NEXT" [2024-03-13 Wed 23:56]

emacs-lisp code along with the HTTP endpoints to populate things from the Generators

(defun arcology-api-generator (module &optional role destination do-sort)
  (interactive)
  (let ((url (format "/generate/%s%s" module
                     (if role (format "/%s" role) ""))))
    (let ((gen-text (with-current-buffer
                        (arcology-localapi-call 'get url :as 'buffer)
                      (buffer-substring-no-properties (point-min) (point-max))))
          (buf (if destination
                   (find-file destination)
                 (get-buffer-create (format "*%s out*" module)))))
      (with-current-buffer buf
        (erase-buffer)
        (insert gen-text)
        (when do-sort
          (sort-lines nil (point-min) (point-max)))
        (when destination
          (save-buffer))
        (when (called-interactively-p)
          (display-buffer))
        (buffer-string)))))
(provide 'cce/arcology-localapi-commands)
import tempfile
from django.core.management import call_command

@authenticated
def generate(request, module, role=None):
    f = tempfile.NamedTemporaryFile()
    fn = f.name
    # i hate this!!!
    if role:
        call_command('generate', destination=fn, module=module, role=role)
    else:
        call_command('generate', destination=fn, module=module)
    f.seek(0)

    return HttpResponse(f)

this is all full of jank. having to pass to a temporary file because the generate commands are locked away in a django management command is silly, i should refactor all of this. the branch is silly, too. But it's very easy now to get my literate programming helpers to quickly generate the code imports and whatnot as soon as the local server is running.

DONE Deploying the Arcology as a Local API via home-manager

  • State "DONE" from "INPROGRESS" [2024-03-20 Wed 12:37]
  • State "INPROGRESS" from "NEXT" [2024-03-18 Mon 16:08]

This thing needs to run as a systemd User Unit for the local tangles and whatnot to work. Bootstrapping this will be Fun, we'll just end up needing a bootstrap script to put the systemd user units in place for future deployments…

This is just the configurable module; the configuration itself is in the Arcology Project Configuration page like the Wobserver module. Here are the options:

LocalAPI Deployment Options

name type default mdDoc
packages.arcology package Arcology package to use
address str "localhost" Web interface listening address
port port 29543 Web interface listening port
workerCount number 4 gunicorn worker count; they recommend 2-4 workers per core.
environmentFile str "${cfg.dataDir}/env" A file containing ARCOLOGY_SYNCTHING_KEY and ARCOLOGY_LOCALAPI_BEARER_TOKEN.
logLevel enum ["ERROR" "WARN" "INFO" "DEBUG"] "INFO" Django root logger level
orgDir str "~/org" Directory containing the org-mode documents. Provide r/o access.
folderId str Syncthing folder ID containing the org files
cacheDir str "~/.cache/arcology2" Location to cache HTML files and the like.
dataDir str "~/.config/arcology2" Location to store the arcology metadata store.

That table does some really gross code generation:

(thread-last
  tbl
  (-map (pcase-lambda (`(,name ,type ,default ,doc))
          (let ((default-str
                 (cond ((numberp default)
                        (prin1-to-string default))
                       ((and (stringp default)
                             (string-empty-p default))
                        nil)
                       (t (prin1-to-string default)))))
            (concat
             name " = mkOption {\n"
             "  type = with types; " type ";\n"
             (if default-str
                 (concat "  default = " default-str ";\n")
               "")
             (if doc
                 (concat "  description = mdDoc ''" doc "'';\n")
               "\n")
             "};\n"))))
  (s-join "\n"))

Home Manager Service Definition

This does some code injection using the table above. It's configured from the Project Configuration, but basically it sets up a few wrapper scripts that configure the process environment and put the webserver and the watchsync commands as SystemD User units as well as providing an arcologyctl command in PATH. Eventually it would be nice to pull the Syncthing key from the local config.xml but maybe that's another management command…

{ pkgs, config, lib, ... }:

let
  cfg = config.services.arcology2;
  arcology = cfg.packages.arcology;

  env = [
    "ARCOLOGY_ENVIRONMENT=development"
    "ARCOLOGY_BASE_DIR=${cfg.orgDir}"
    "ARCOLOGY_DB_PATH=${cfg.dataDir}/db.sqlite3"
    "ARCOLOGY_LOG_LEVEL=${cfg.logLevel}"
    "ARCOLOGY_CACHE_PATH=${cfg.cacheDir}"
  ];

  preStartScript = pkgs.writeScriptBin "arcology-watchsync-prestart" ''
    mkdir -p ${cfg.dataDir}
    ${arcology}/bin/arcology migrate
    ${arcology}/bin/arcology seed || true
  '';
  watchsyncScript = pkgs.writeScriptBin "arcology-watchsync" ''
    exec ${arcology}/bin/arcology watchsync -f ${cfg.folderId}
  '';
  localapiScript = pkgs.writeScriptBin "arcology-localapi" ''
    exec ${arcology}/bin/arcology runserver --noreload ${toString cfg.address}:${toString cfg.port}
  '';
  wrapperScript = pkgs.writeScriptBin "arcology" ''
    set -eEuo pipefail

    ${lib.strings.concatMapStrings (e: "export ${e}\n") env}

    source ${cfg.environmentFile}
    export ARCOLOGY_SYNCTHING_KEY
    export ARCOLOGY_LOCALAPI_BEARER_TOKEN

    exec ${arcology}/bin/arcology "$@"
  '';
in {
  options.services.arcology2 = with lib; {
    enable = mkEnableOption (mdDoc "Arcology Local API");

    <<generate-hm-options()>>
  };

  config = {
    home.packages = [
      wrapperScript
    ];

    systemd.user.services = {
      arcology-watchsync = {
        Unit.Description = "Dynamically update the Arcology database.";
        Service = {
          ExecStartPre = "${pkgs.stdenv.shell} ${preStartScript}/bin/arcology-watchsync-prestart";
          ExecStart = "${pkgs.stdenv.shell} ${watchsyncScript}/bin/arcology-watchsync";

          Restart = "on-failure";
          RestartSec=5;
          RestartSteps=10;
          RestartMaxDelaySec="1min";
          Environment = env;
          EnvironmentFile = cfg.environmentFile;
        };
        Install.WantedBy = ["default.target"];
      };

      arcology-web = {
        Unit.Description = "Local hosted version of the Arcology";
        Service = {
          ExecStart = "${pkgs.stdenv.shell} ${localapiScript}/bin/arcology-localapi";

          Restart = "on-failure";
          RestartSec=5;
          RestartSteps=10;
          RestartMaxDelaySec="1min";
          Environment = env;
          EnvironmentFile = cfg.environmentFile;
        };
        Install.WantedBy = ["default.target"];
      };
    };
  };
}

NEXT DRY this between here and the Deploying the Arcology work.

NEXT is the arcology wobserver deployment better as a systemd user unit?

still need nixos module for the vhosts etc, but… hm.

running as my main user would make a lot of the environment stuff easier to deal with, i guess