The Arcology Garden

The Arcology Project

LifeTechEmacsArcology

Arcology is my site generator Project currently written in roam:FastAPI and backed by my org-mode wiki set up on top of org-roam. The Goal for the project is to be able to publish subsections of my notes and code and knowledge across multiple websites, and to be able to publish a feed of new and updated pages within those subsections over open protocols like roam:RSS and ActivityPub.

What's in a Name?

Arcology is an architectural and cultural school of design, a portmanteau of Ecology and roam:Architecture, a philosophy of design principles for densely populated, ecologically low-impact human habitats. In Sci-Fi they're closed-system environments where human needs are met without harming the wider environment, through technical, social, and automated means. A solar plant and a vertical farm feeding thousands of residents, with countless acres of land left to wild outside the Arcology. In the real world there's a prototype desert commune in northern Arizona.

My Arcology is a technical and cultural school of design, of building systems which mesh cleanly with those outside it and the user(s) within it. It's a self-sustaining system choosing to remain unreliant on proprietary and liberty-hostile systems. This is a Privacy-focused self-hosted document publishing platform, a Hypermedia application exposing the inner working of my knowledge archive, link archive, and inner thinking. It's part of A Modern User Agent built for myself for now, and for the family and community I will build.

Vision and Goals

The Arcology Project exists so that I can tag any page in my org-roam sysetm to the web for sharing among friends and family and my technical peers.

I am trying to fully develop this as an Literate Programming application, and so far the results have been mixed but still promising.

Why not ox-hugo or another Static Site Gen?

Project History

Arcology Static Site Generator

Arcology started as an Emacs Lisp library which produced a static site where individual org-mode headings would be exported using a custom templating system to provide a site that was built around Microformat metadata specifications and the informal protocols around the Indieweb community.

This is the software which generate[s/d/ing] https://notes.whatthefuck.computer and it worked well enough, but it's failed to scale with my current system, and especially with EXWM having a long syncronous process in my Emacs process was a nonstarter -- I stopped publishing my site because of how costly it was, and that must change.

I was also butting up against the organisational bad habits that I had picked up with regard to how my org-mode documents were organized and how I handled my personal Knowledge Base. I explored and began to realize that what I wanted was less of outline-style heirarchical organization of thought, and I was more interested in non-heirarchical structures like graphs and links.

Arcology Rails

Arcology Rails was the first attempt at a "dynamic thinking environment", a set of CRUD applications that could help me think, and remember. It was an experiment in building a simple graph structure without namespacing, and building intelligence on top of that.

I had visions of building dynamic applications on top of three components:

  • a content addressed store of files

  • a graph-database

  • a lens-based programming environment on top of them

I am very far from being interested in building this sort of plumbing in the style and systems I am investing in. If those exist some day in a decentralized, equitable, tactile option, I would love to see them.

A brief exploration of TiddlyWiki

TiddlyWiki is a very cool programmable notebook, and I spent some time working with it in January and February 2020. Very promising thing, and it lead me to the value of embedded interlinks over metadata. But I wanted a lot of things that it couldn't give me, and so I went back to the drawing board on a static site generator built on top of org-mode.

Arcology Gen

Arcology Gen was another attempt at a smarter, more streamlined static site generator, but this time outside the Emacs process. This is where I started to realize that I need a cache database, something queryable, and something persistent. I constructed some very simple org-mode parsers, and that was interesting for publishing a directory, but I needed metadata extraction to power indices and feeds.

Arcology Elixir

In early 2020, org-roam gave me that database, and with some extensions, I could build the website. Around this time, Tor and I started working on a Home Mesh project in Elixir, and co-hosting that with my web-site seems like a reasonable cross-over. I wanted to use Ecto to build the site, and could some day have provided static HTML, but it ended as a dynamic Elixir application, a mashup of Phoenix and Ecto and some custom code to glue it to an Emacs environment

Now in early 2021 I have a working minimum viable prototype of the Arcology at https://dev.arcology.garden and I must evaluate the next steps. Jethro of org-roam is currently evaluating a 2.0 refactor and simplification and the vision of it doesn't quite line up with what I want even if it's for the good of the software and the ecosystem. Alas, I'll have to modify the Arcology Link Routers to handle these opaque ID links, and I think i can, but this major version bump lays bare the cost of my dependencies.

The Literate Programming Elixir Phoenix project is worth evaluating in its own right, still: Is developing entire services as an org-mode meta application viable or for that matter fun? How does the interwoven format aid long-term maintenance, refactoring and development? I do intend to explore this at some point (open threads)

As of [2021-09-25] I haven't written those things, and have struggled to find the energy and will to work on the Elixir version of arcology. Work has pushed me towards picking up roam:FastAPI and now I am using that, hoping that I can get a small custom web app put together as I found that I wasn't using the dynamic high-scale, interesting things which Phoenix and Elixir provide.

Arcology FastAPI

The FastAPI version was a lightweight implementation of the ideas I fleshed out in the Elixir implementation. It's still literate-ly programmed. Flask would probably be good enough, but having the database stuff integated with SQLModel is going to be nice.

At this point I have boiled down the "Arcology as dynamic site" down to a set of essential functions and ideas which could lbe implemented in any language, and I intend to explore at least a Rust implementation of it next. However, this system is lithe and fun to work with.

The Arcology Project: Django Edition

I wrote The arroyo_rs Native Org Parser to peel the Emacs org-roam parser (Arroyo System Cache + Generators) out of the publishing process. Orgize 0.9 is nice enough, it's currently being refactored heavily in a branch, and I'm not looking forward to porting forward, but for now it works and is bound to a Python library using pyo3 and maturin. I wrote about this in [2023-06-14] at I am starting to experiment with a rust rewrite of the Arroyo Arcology Generator.

Rather than handle all the SQLite and web access in Rust, where I found few sqlite tools which I enjoyed using or were fit-to-purpose for the ingester and relation mapper I needed, and after some experiments with different libraries, I landed on shipping the Arcology on Django. Django makes it easy to prototype data-driven applications quickly and the ORM's design is a big part of that. The combination of the Django ORM, arroyo_rs and Django's web framework is a good powerful combination. It tracks file changes using the Syncthing API for instant updates.

The Arcology is now powered by this stack, as is the Arroyo generators which power the Concept Operating System's NixOS configuration, The Wobserver and init.el generators.

ActivityPub and Microblog

There has been a parallel project tumbling around in my head, the Mastadan-powered second brain, which could be implemented in the rust toolkit, too, since it can be used to generate org-mode docs too; I tried an elixir ActivityPub library but it was not ready for action yet; there is another one developed by the Lemmy folks which may be more reusable. I found another weird option too called Dialtone which could be a neat little Rust thing to integrate... Idk. Anyways, a little microblog that supported multi-actor posting and thread coalescing/search/export could be neat as heck. If the Bonfire ActivityPub library was ready for business I would probably be implementing all the site stuff in Elixir, but this seems like a more likely win; maybe if I take take enough the Elixir lib will be ready... :)

akkoma2org to bring my threads and replies in-system

Data Architecture

org-mode documents are the canonical datastore

I value plain-text formats and org-mode provides enough structurally significant syntax that I can extract any information that I need from the text, and re-encode that without a huge loss of fidelity.

arroyo-rs parses the documents and returns metadata

Django ORM persists the Arroyo objects

Django's rendering framework presents them.

Arcology FastAPI Task Work

HEAD support with opengraph and the social embed shit, rss feeds for site and page...

Work on Arcology once a week

SCHEDULED: <2024-02-10 Sat ++1w>

Base feature set

test coverage

webmention server

iNaturalist observations PESOS importer

A page for each plant, a heading for each observation.

journal entry

similar PESOS My Instagram posts in to blog

dogsheep's iNaturalist to Sqlite is half the work

[2020-10-27 Tue 22:31] Arcology should make sure that pages in the Archive are saved in the Internet Archive

file attachments + attached image rendering

it'd be nice to resize the images and strip exif data and whatnot, but maybe i just need to do that on "my" side...

ActivityPub server

using tsileo/little-boxes

calendar generator

Trailblazer Mode

Footnotes

Let's be honest; it's doing quite a bit more. It's caching pages and metadata, it's providing a Sitemap, it's providing high-quality Routing, and it allows me to build features a theoretical roam:Arcology Anti-Viral Technology, I would love it if external links and references could be automatically committed to the Internet Archive