arcology/configuration.org

600 lines
24 KiB
Org Mode

:PROPERTIES:
:ID: arcology/django/config
:END:
#+TITLE: Arcology Project Configuration
#+filetags: :Project:
#+OPTIONS: ^:nil
#+ARCOLOGY_KEY: arcology/configuration
,#+AUTO_TANGLE: vars:org-babel-default-header-args
#+ARROYO_NIXOS_MODULE: nixos/arcology2.nix
#+ARROYO_SYSTEM_ROLE: server
This file contains user-configuration. Some sections are generated from org-mode tables if you're meant to be editing or extending them. This is a cool feature of org-babel, where you can use tables as data for code which can output more code or even org headings or links. We use this to generate configuration.
* The Arcology's Site List
:PROPERTIES:
:ID: 20231229T164611.256424
:END:
These can be customized by the user:
#+NAME: sites-config
| title | key | css file | link color |
|--------------------------+-----------+---------------+------------|
| The Lion's Rear | lionsrear | lionsrear.css | #cfdcc2 |
| The Arcology Garden | garden | garden.css | #fcf6ed |
| The Complete Computer | cce | cce.css | #dcdcec |
| The Arcology Site Engine | arcology | arcology.css | #ccdfff |
| Local Dev Environment | localhost | arcology.css | #075192 |
#+NAME: domains-config
| key | domain |
|-----------+---------------------------|
| lionsrear | thelionsrear.com |
| lionsrear | rix.si |
| garden | arcology.garden |
| garden | whatthefuck.computer |
| cce | cce.whatthefuck.computer |
| cce | cce.rix.si |
| arcology | engine.arcology.garden |
| localhost | 127.0.0.1 |
| localhost | localhost |
| lionsrear | v2.thelionsrear.com |
| garden | v2.arcology.garden |
| cce | cce2.whatthefuck.computer |
| arcology | engine2.arcology.garden |
** The Code
Oh we doing some code gen in here, this is just gonna generate some JSON for now joining these two tables lul.
#+name: elisp-join-tables
#+begin_src emacs-lisp :var sites=sites-config domains=domains-config :results none
(json-encode
(-map (lambda (siterow)
(pcase-let* ((`(,title ,key ,css ,link) siterow))
`((title . ,title)
(key . ,key)
(css_file . ,(format "arcology/css/%s" css))
(link_color . ,link)
(domains . ,(-map #'cadr
(-filter (pcase-lambda (`(,dkey ,domain))
(equal key dkey))
domains))))))
sites))
#+end_src
That generates JSON that goes in to =arcology/settings/sites.json= for the [[id:20231217T154835.232283][Arcology Seed Command]] to use:
#+begin_src json :tangle arcology/settings/sites.json :noweb yes :comments none
<<elisp-join-tables()>>
#+end_src
#+name: get_allowed_hosts
#+begin_src emacs-lisp :var domains=domains-config :results none
(thread-last domains
(-map (pcase-lambda (`(,site ,domain))
domain))
(s-join ","))
#+end_src
** NEXT How to re-seed these if you change the configuration?
for now i just blow away the database but gosh!
* Per-site CSS
These are generated from a table, too, because I have brainworms.
Note that they're basically the same, but you can customize it further to your satisfaction.
#+NAME: site-colors
| Site | alert | primary | secondary | success | warning | white | gray2 | gray3 | gray4 | black |
|----------------+---------+---------+-----------+---------+---------+---------+---------+---------+---------+---------|
| default-colors | #cc6960 | #707231 | #ebbe7b | #67b4f8 | #7e5c41 | #fcf6ed | #f6e5cb | #baad9b | #82796c | #211f1c |
| garden | #cc6960 | #707231 | #ebbe7b | #67b4f8 | #7e5c41 | #fcf6ed | #f6e5cb | #baad9b | #82796c | #211f1c |
| lionsrear | #cc6960 | #707231 | #ebbe7b | #67b4f8 | #7e5c41 | #cfdcc2 | #87af87 | #a0aa96 | #82796c | #211f1c |
| cce | #cc6960 | #707231 | #ebbe7b | #67b4f8 | #7e5c41 | #fcfcfc | #dcdcec | #cacada | #808090 | #211f1c |
| arcology | #cc6960 | #707231 | #ebbe7b | #67b4f8 | #7e5c41 | #fcf6ed | #f6e5cb | #baad9b | #82796c | #211f1c |
** The Code
#+NAME: mk-site-colors
#+begin_src emacs-lisp :var colors=site-colors :var site="default-colors" :results none
(pcase-let ((`((,name ,alert ,primary ,secondary ,success ,warning ,white ,gray2 ,gray3 , gray4 ,black))
(-filter (lambda (row)
(equal (first row) site))
colors)))
(s-join "\n"
(list ":root {"
(concat " --alert: " alert ";")
(concat " --primary: " primary ";")
(concat " --secondary: " secondary ";")
(concat " --success: " success ";")
(concat " --warning: " warning ";")
(concat " --white: " white ";")
(concat " --light-gray: " gray2 ";")
(concat " --medium-gray: " gray3 ";")
(concat " --dark-gray: " gray4 ";")
(concat " --black: " black ";")
"}")))
#+end_src
#+begin_src css :tangle arcology/static/arcology/css/default-colors.css :noweb yes
<<mk-site-colors(site-colors, "default-colors")>>
#+end_src
#+begin_src css :tangle arcology/static/arcology/css/cce.css :noweb yes
<<mk-site-colors(site-colors, "cce")>>
#+end_src
#+begin_src css :tangle arcology/static/arcology/css/lionsrear.css :noweb yes
<<mk-site-colors(site-colors, "lionsrear")>>
#+end_src
#+begin_src css :tangle arcology/static/arcology/css/garden.css :noweb yes
<<mk-site-colors(site-colors, "garden")>>
#+end_src
#+begin_src css :tangle arcology/static/arcology/css/arcology.css :noweb yes
<<mk-site-colors(site-colors, "arcology")>>
#+end_src
* Deploy manifests for [[id:arroyo/nixos][Arroyo NixOS Generator]] or your own manifests
The [[id:arroyo/nixos][Arroyo NixOS Generator]] will integrate this or you can do this yourself. just import the nixos module above, and then configure it as below:
#+begin_src nix :tangle ~/arroyo-nix/nixos/arcology2.nix :noweb yes
{ pkgs, ... }:
let
arroyo_rs = pkgs.callPackage /home/rrix/org/arroyo/default.nix {};
arcology = pkgs.callPackage /home/rrix/org/arcology-django/default.nix { inherit arroyo_rs; };
in {
imports = [ ./arcology2-module.nix ];
fileSystems."/media/org" = {
device = "/home/rrix/org";
options = ["bind"];
};
services.arcology-ng = {
enable = true;
packages.arcology = arcology;
domains = pkgs.lib.splitString "," "<<get_allowed_hosts()>>";
orgDir = "/media/org";
folderId = "p1kld-oxnwd";
dataDir = "/srv/arcology";
logLevel = "INFO";
};
}
#+end_src
The NixOS module which defines =services.arcology-ng= is in [[id:20240213T124300.774781][Deploying the Arcology]].
** NEXT the package import needs to be much better than this.
* Deploy manifests for [[id:cce/home-manager][a Dynamic Home Manager Configuration]] of [[id:20240320T123253.558216][LocalAPI Deployment]]
This deploys [[id:20240313T153901.656967][A Localhost API for the Arcology]] to any Home Manager user using =systemd= User Units. See [[id:20240320T123253.558216][LocalAPI Deployment Options]] for the list of options that can be changed and how to use it:
#+begin_src nix :tangle ~/arroyo-nix/hm/arcology-localapi.nix
{ pkgs, ... }:
let
arroyo_rs = pkgs.callPackage /home/rrix/org/arroyo/default.nix {};
arcology = pkgs.callPackage /home/rrix/org/arcology-django/default.nix { inherit arroyo_rs; };
in {
imports = [ ./arcology-localapi-mod.nix ];
services.arcology2 = {
enable = true;
packages.arcology = arcology;
folderId = "p1kld-oxnwd";
logLevel = "DEBUG";
environmentFile = "/home/rrix/sync/private-files/.arcology-env";
};
}
#+end_src
** NEXT address the =callPackages= here...
* Service Configuration
:PROPERTIES:
:ID: 20231217T155611.177995
:END:
these are mostly defaults, a bit of this should be made user-configurable but a lot of this is baked in to how the project works.
#+begin_quote
Django settings for arcology project.
Generated by 'django-admin startproject' using Django 3.2.22.
For more information on this file, see
https://docs.djangoproject.com/en/3.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.2/ref/settings/
Quick-start development settings - unsuitable for production
See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/
#+end_quote
#+BEGIN_SRC python :tangle arcology/settings/__init__.py
from pathlib import Path
import os
from .generators import *
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent.parent
DATABASE_PATH = pathlib.Path(os.getenv("ARCOLOGY_DB_PATH", pathlib.Path(os.getcwd()) / "db.sqlite3")).expanduser()
#+END_SRC
** Environment Variables
=ARCOLOGY_DJANGO_SECRET= environment variable will be set in the NixOS deployment manifests. this is used for, uhh, security things
#+BEGIN_SRC python :tangle arcology/settings/__init__.py
SECRET_KEY = os.getenv("ARCOLOGY_DJANGO_SECRET", "devsecret")
#+END_SRC
Setting =ARCOLOGY_ENVIRONMENT=production= in the NixOS deployment manifests will modify a bunch of behavior, probably
#+BEGIN_SRC python :tangle arcology/settings/__init__.py
ARCOLOGY_ENVIRONMENT = os.getenv("ARCOLOGY_ENVIRONMENT", "dev")
DEBUG = (ARCOLOGY_ENVIRONMENT != "production")
#+END_SRC
The =ARCOLOGY_SYNCTHING_KEY= environment variable will be set to the [[id:cce/syncthing][Syncthing]] API key, each deployment will need to specify its own, i'll re-think this when it comes time to deploy this to many places.
#+BEGIN_SRC python :tangle arcology/settings/__init__.py
SYNCTHING_KEY = os.getenv("ARCOLOGY_SYNCTHING_KEY")
#+END_SRC
The =ARCOLOGY_CACHE_PATH= is set to a path that multi-process django can use to cache processed HTML and Atom between processes.
#+BEGIN_SRC python :tangle arcology/settings/__init__.py
BASE_CACHE_PATH = pathlib.Path(os.environ.get("ARCOLOGY_CACHE_PATH", '/var/tmp/django_cache')).expanduser()
CACHES = {
'default': {
'BACKEND': 'django_prometheus.cache.backends.filebased.FileBasedCache',
'LOCATION': BASE_CACHE_PATH,
},
}
#+END_SRC
#+begin_src python :tangle arcology/settings/__init__.py
LOCALAPI_BEARER_TOKEN = os.environ.get("ARCOLOGY_LOCALAPI_BEARER_TOKEN", "changeme!")
#+end_src
** NEXT Hostname configuration from =arcology.model.Site=, eventually
When I have the sites organized in an org-mode table, i'll reapproach the hostname list, and probably before then when i want to test domain-based routing.
#+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
ALLOWED_HOSTS = "<<get_allowed_hosts()>>".split(',')
#+END_SRC
** The Arcology is Modular
basically, each org file in this repository, and maybe one or two of your own, are all interoperable and build on top of the base data models of the org docs.
#+name: arcology_apps
| "roam" | [[id:arcology/django/roam][Arcology Roam Models]] |
| "arcology" | [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]] |
| "generators" | [[id:arroyo/django/generators][The Arroyo Generators]] |
| "syncthonk" | [[id:20231218T183551.765340][Arcology watchsync Command]] |
| "sitemap" | [[id:20240226T132507.817450][The Arcology's Site Maps and Discovery Mechanisms]] |
#+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
# Application definition
INSTALLED_APPS = [
<<gen-config-list(arcology_apps)>>
"django_htmx",
"django_prometheus",
"django.contrib.admin",
"django.contrib.auth",
"django.contrib.contenttypes",
"django.contrib.sessions",
"django.contrib.messages",
"django.contrib.staticfiles",
]
if ARCOLOGY_ENVIRONMENT != "production":
INSTALLED_APPS = ["localapi"] + INSTALLED_APPS
#+END_SRC
** Internationalization
I'll have to do this at some point. what does an internationalized arcology look like, how does one ship translations of this work?
#+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
# Internationalization
# https://docs.djangoproject.com/en/3.2/topics/i18n/
LANGUAGE_CODE = "en-us"
TIME_ZONE = "UTC"
USE_I18N = True
USE_L10N = True
USE_TZ = True
#+END_SRC
** Logging Configuration
#+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"console": {
"class": "logging.StreamHandler",
},
},
"loggers": {
"django": {
"handlers": ["console"],
"level": os.getenv("DJANGO_LOG_LEVEL", "INFO"),
"propagate": False,
},
"arcology.cache_decorator": { # left as an example to change later.
"handlers": ["console"],
"level": "DEBUG",
}
},
"root": {
"handlers": ["console"],
"level": os.getenv("ARCOLOGY_LOG_LEVEL", "INFO"),
},
}
#+END_SRC
* Generator Configuration
:PROPERTIES:
:ID: 20231217T155608.355979
:END:
** NEXT directory configuration
#+BEGIN_SRC python :tangle arcology/settings/generators.py :noweb yes
import pathlib
import os
ARCOLOGY_BASE_DIR = pathlib.Path(os.environ.get("ARCOLOGY_BASE_DIR", "~/org")).expanduser()
ARCOLOGY_EMACS_SNIPPETS_DIR = str(pathlib.Path("~/org/cce/").expanduser())
ARROYO_BASE_DIR = str(pathlib.Path("~/arroyo-nix").expanduser())
#+END_SRC
** Defining extension modules
:PROPERTIES:
:ID: 20240204T232637.286228
:ROAM_ALIASES: "Arcology Extractor and Arroyo Generator Configuration"
:END:
It should be feasible to add new data types and models and generators to the Arcology system to fit a user's needs. The defaults are specified here.
These models are only persisted if the page is going to be published and are used by the core [[id:arcology/django/arcology-models][The Arcology Web Server]] to set up the page and render backlinks and tag pages and whatnot.
#+name: arcology_extractors
| roam.models.Heading | [[id:arcology/django/roam][Arcology Roam Models]]'s heading entity |
| roam.models.Reference | [[id:arcology/django/roam][Arcology Roam Models]]'s [[id:cce/org-roam][org-roam]] reference entity |
| roam.models.Tag | [[id:arcology/django/roam][Arcology Roam Models]]'s heading tag entity |
| roam.models.HeadingProperty | [[id:arcology/django/roam][Arcology Roam Models]]'s heading properties map |
| roam.models.Link | [[id:arcology/django/roam][Arcology Roam Models]] page-to-page links |
| arcology.models.Page | [[id:arcology/django/arcology-models][The Arcology's Data Models]]'s Page entity |
| arcology.models.Feed | [[id:arcology/django/arcology-models][The Arcology's Data Models]]'s Feed entity, probably move to its own module some day... |
| arcology.models.FeedEntry | |
The [[id:arroyo/django/generators][Arroyo Generators]] are run regardless of whether a file is published. These are a dictionary so that the user can specify a module when invoking the [[id:20231217T154938.132553][Arcology =generate= Command]].
#+name: arroyo_extractors
| emacs | generators.models.EmacsSnippet |
| epkgs | generators.models.EmacsEpkg |
| nixos | generators.models.NixosModule |
| home-manager | generators.models.HomeManagerModule |
*** The Code
#+BEGIN_SRC python :tangle arcology/settings/generators.py :noweb yes
ARCOLOGY_EXTRACTORS = [
<<gen-config-list(arcology_extractors)>>
]
ARROYO_EXTRACTORS = {
<<gen-config-dict(arroyo_extractors)>>
}
#+END_SRC
** Enumerating System Roles
I would like to not need these enumerated here, it seems like it should be easier to add a new role than that. But for now here they are so that the DB seed works better.
#+NAME: arroyo-roles
| server | [[id:20220101T195846.044283][the Wobserver Configuration with Arroyo Nixos]] $ |
| endpoint | [[id:cce/my_nixos_configuration][My NixOS configuration]] |
| settop | [[id:20220131T152041.472624][NixOS Set Top Box]] |
| droid | [[id:20221106T113721.266425][CCE in Nix On Droid]] |
| waterboy | [[id:waterboy][waterboy]] icebox'd data collection platform |
#+BEGIN_SRC python :tangle arcology/settings/generators.py :noweb yes
ARROYO_ROLES = [
<<gen-config-list(arroyo-roles)>>
]
#+END_SRC
** Roam Allowed Keywords
These tables will be used to create an allowlist of =#+KEYWORD='s to store in the database.
These are used to provide system features or power the various generators:
#+NAME: roam_allowed_keywords
| ARCOLOGY_FEED | Add a route that will generate an Atom feed for this page |
| ARCOLOGY_KEY | Add a route that will generate an HTML page for this page |
| ARCOLOGY_ALLOW_CRAWL | Whether the Arcology page will be marked as allowed in =robots.txt= |
| ARCOLOGY_TOOT_VISIBILITY | This is set to an [[id:62538db5-d94a-47c3-9998-086ded91fd88][Mastadan]] API visibility mode like "public" or "unlisted" |
| ARROYO_EMACS_MODULE | Instruct [[id:arroyo/django/generators][The Arroyo Generators]] to include an Emacs snippet in the init. |
| ARROYO_HOME_MODULE | Instruct [[id:arroyo/django/generators][The Arroyo Generators]] to import the referenced home-manager module |
| ARROYO_NIXOS_MODULE | Instruct [[id:arroyo/django/generators][The Arroyo Generators]] to import the referenced nixos module |
| ARROYO_HOME_EPKGS | Instruct [[id:arroyo/django/generators][The Arroyo Generators]] to make a custom Emacs-Lisp package available to Arroyo Emacs |
| ARROYO_MODULE_WANTS | Define a dependency relationship with the referenced org-roam document (this doc depends on that) |
| ARROYO_MODULE_WANTED | Define a dependent relationship from the referenced org-roam document (that doc depends on this) |
| ARROYO_SYSTEM_ROLE | Constrain the Nix expressions on the page to only load in this role (can be specified repeatedly) |
| ARROYO_SYSTEM_EXCLUDE | Prevent the Nix expressions on the page from loading in the listed role (can be specified repeatedly) |
| ARROYO_DIRENV_DIR | Load [[id:45fc2a02-fcd0-40c6-a29e-897c0ee7b1c7][direnv]] from a different directory than the loaded buffer's directory. Useful for [[id:cce/literate_programming][Org Babel]]. |
| AUTHOR | Used to populate the author in RSS feeds and page metadata |
These are used in some of my custom reports:
#+NAME: my_roam_allowed_keywords
| BIRTHDAY | Used in [[id:8803fda8-20cd-4d62-878d-dcb1b7853183][People]] pages. |
| LOCATION | Used in [[id:8803fda8-20cd-4d62-878d-dcb1b7853183][People]] pages. |
| PRONOUNS | Used in [[id:8803fda8-20cd-4d62-878d-dcb1b7853183][People]] pages. |
| AGE_OF_MONEY | Used in financial review of [[id:426cef6a-f77d-4b2d-8acb-0793b3cef368][Monthly Review]] |
| JPMORGAN | Used in financial review of [[id:426cef6a-f77d-4b2d-8acb-0793b3cef368][Monthly Review]] |
| SAVINGS | Used in financial review of [[id:426cef6a-f77d-4b2d-8acb-0793b3cef368][Monthly Review]] |
| SPENT_THIS_MONTH | Used in financial review of [[id:426cef6a-f77d-4b2d-8acb-0793b3cef368][Monthly Review]] |
*** The Code
#+BEGIN_SRC python :tangle arcology/settings/generators.py :noweb yes
ROAM_ALLOWED_KEYWORDS = [
<<gen-config-list(roam_allowed_keywords)>>
<<gen-config-list(my_roam_allowed_keywords)>>
# used for deprecated universal-aggregator feed generation
"ARROYO_FEEDS",
"ARROYO_FEED_CATEGORY",
"ARROYO_FEED_URL",
# unused
"CCE_HOME_MODULE",
"ARROYO_EMACS_TANGLED",
"ARROYO_MODULE_DEP",
]
#+END_SRC
** Ignored Roam Tags
Headings with these tags will not be exported.
#+NAME: ignored_roam_tags
| NOEXPORT |
| noexport |
| Private |
*** The Code
#+BEGIN_SRC python :tangle arcology/settings/generators.py :noweb yes
IGNORED_ROAM_TAGS = [
<<gen-config-list(ignored_roam_tags)>>
]
#+END_SRC
* Look don't worry about the rest of these
#+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
MIDDLEWARE = [
"django_prometheus.middleware.PrometheusBeforeMiddleware",
"django_htmx.middleware.HtmxMiddleware",
"django.middleware.security.SecurityMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
"django.middleware.csrf.CsrfViewMiddleware",
"django.contrib.auth.middleware.AuthenticationMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"django.middleware.clickjacking.XFrameOptionsMiddleware",
"django_prometheus.middleware.PrometheusAfterMiddleware",
]
ROOT_URLCONF = "arcology.urls"
TEMPLATES = [
{
"BACKEND": "django.template.backends.django.DjangoTemplates",
"DIRS": [],
"APP_DIRS": True,
"OPTIONS": {
"context_processors": [
"django.template.context_processors.debug",
"django.template.context_processors.request",
"django.contrib.auth.context_processors.auth",
"django.contrib.messages.context_processors.messages",
],
},
},
]
WSGI_APPLICATION = "arcology.wsgi.application"
# Database
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases
DATABASES = {
"default": {
"ENGINE": "django_prometheus.db.backends.sqlite3",
"NAME": DATABASE_PATH,
}
}
# Password validation
# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
"NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",
},
{
"NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",
},
{
"NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",
},
{
"NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",
},
]
CSRF_TRUSTED_ORIGINS = list(map(lambda it: f"https://{it}", "<<get_allowed_hosts()>>".split(',')))
#+END_SRC
#+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.2/howto/static-files/
STATIC_URL = "/static/"
STORAGES = {
"default": {
"BACKEND": "django.core.files.storage.FileSystemStorage",
},
"staticfiles": {
"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
},
}
STATICFILES_DIRS = [
BASE_DIR / "arcology/static",
BASE_DIR / "sitemap/static",
]
STATIC_ROOT = os.getenv("ARCOLOGY_STATIC_ROOT", "/var/www/arcology")
# Default primary key field type
# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
#+END_SRC
* Helpers
#+NAME: gen-config-list
#+begin_src emacs-lisp :var tbl=roam_allowed_keywords
(mapconcat (lambda (row)
(format "\"%s\", # %s\n" (first row) (second row)))
tbl)
#+end_src
#+NAME: gen-config-dict
#+begin_src emacs-lisp :var tbl=arroyo_extractors
(mapconcat (lambda (row)
(format "\"%s\": \"%s\",\n" (first row) (second row)))
tbl)
#+end_src