dorodango

README.org

@@ -7,7 +7,7 @@
 
 * The Arcology Project
 
-[[id:1d917282-ecf4-4d4c-ba49-628cbb4bb8cc][The Arcology Project]] is the logistical layer of my [[id:60f710b2-6a1f-44be-bc13-dfe01e46d4e3][Concept Operating System]]'s [[id:knowledge_base][Knowledge Management]] system. It is a web publishing platform built to present my literate programming platform and my public knowledge. The goal, in short, is to be able to publish and link to any node in a set of [[id:1fb8fb45-fac5-4449-a347-d55118bb377e][org-mode]] documents.
+[[id:1d917282-ecf4-4d4c-ba49-628cbb4bb8cc][The Arcology Project]] is the social and  layer of my [[id:60f710b2-6a1f-44be-bc13-dfe01e46d4e3][Concept Operating System]]'s [[id:knowledge_base][Knowledge Management]] system. It is a web publishing platform built to present my literate programming platform and my public knowledge. The goal, in short, is to be able to publish and link to any node in a set of [[id:1fb8fb45-fac5-4449-a347-d55118bb377e][org-mode]] documents. When someone asks me "hey how do I do XYZ on Linux" I can just link them to a page answering that question, and showing them how to do it themselves.
 
 * The Arcology Project: Django Edition
 
@@ -17,9 +17,9 @@ The new [[id:20231023T115950.248543][arroyo_rs Native Org Parser]] removes the A
 
 This allows a user to host a webpage, and use [[roam:The Arcology Management Commands]] to query the structure of their notes.
 
-What I mean by the latter is that pages can be annotated with =key= / =value= pairs and these can be queried and built upon to implement the features of [[id:cce/cce][The Complete Computing Environment]] where any page in an [[id:cce/org-roam][org-roam]] directory can add code to your NixOS environment or Emacs configuration to give your computer and your knowledge base new powers.
+What I mean by the latter is that pages can be annotated with =key= / =value= pairs and these can be queried and built upon to implement the features of [[id:cce/cce][The Complete Computing Environment]] where any page in an [[id:cce/org-roam][org-roam]] directory can add code to your NixOS environment or Emacs configuration to give your computer and your knowledge base new powers. The Arcology contains much of the [[id:arroyo/arroyo][Arroyo Systems Management]] software, but I may change that in the future.
 
-The Complete Computing Environment provides a decent UX and starter configuration and document design for this system. The Arcology's Django edition promises to add new meta-powers to this programming toolkit while making it more feasible for others to replicate the system.
+The Complete Computing Environment provides a decent UX and starter configuration and document design for this system. The Arcology's Django edition promises to add new meta-powers to this programming toolkit while making it more feasible for others to replicate the system and build their own Arcology.
 
 * Usage
 
@@ -28,7 +28,7 @@ ofc i need to flesh all this out still. don't use this, for now.
 - [[id:20231113T195508.942155][Rebuild of The Complete Computer]]
 
 - [[id:arcology/django/config][Project Configuration]]
-- [[shell:nix run git+https://code.rix.si/rrix/arcology-django# ingest_files ~/org]] to create a DB
+- [[shell:nix run git+https://code.rix.si/rrix/arcology-django# ingestfiles ~/org]] to create a DB in CWD
 - [[shell:nix run git+https://code.rix.si/rrix/arcology-django# watchsync]] to create a DB and keep it updated by monitoring [[id:cce/syncthing][Syncthing]]
 - example =nix run= to start the webserver
 - example =nix run= to start an emacs running arroyo
@@ -40,10 +40,20 @@ ofc i need to flesh all this out still. don't use this, for now.
 This project is a living document. It's [[id:cce/literate_programming][Literate Programming]] powered by [[id:cce/literate_programming][Org Babel]]. You shouldn't need to read all the code to run it, but if you need to it oughta be well documented.
 
 - [[id:arcology/django/config][Arcology Project configuration]] describes and implements the knobs the user can twiddle to configure the Arcology's operation
-- [[id:arcology/django/interfaces][Interfacing with the Arcology]] is a set of management commands and a Syncthing client which will automatically ingest new files in to the database.
-- The [[id:arcology/django/scaffolding][Arcology Project Scaffolding]] contains the files necessary to run the project, the nix environment, and the python environment, and the Django configuration
+- [[id:arcology/django/interfaces][Interfacing with the Arcology]] is a set of management commands and a Syncthing client which will automatically ingest new files in to the database. This document describes all the tools a prospective user of the CCE will need to use for surface-level features.
+- The [[id:arcology/django/scaffolding][Arcology Project Scaffolding]] contains the files necessary to run the project, defines the nix environment, and the python environment, and the base Django apparatus.
 - The [[id:arcology/django/roam][Arcology Roam Models]] are the base metadata of the [[id:cce/org-roam][org-roam]] documents
 - [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]] are the web publishing platform and the data models it requires to be operated
-- [[id:arroyo/django/generators][The Arroyo Generators]] is the "API" which can be used to build a [[id:60f710b2-6a1f-44be-bc13-dfe01e46d4e3][Concept Operating System]] using literate programming in org-mode.
+- [[id:arroyo/django/generators][The Arroyo Generators]] are the "API" which can be used to build a [[id:60f710b2-6a1f-44be-bc13-dfe01e46d4e3][Concept Operating System]] using literate programming in org-mode.
+
+* Rough Timeline and Task List
+
+- Prometheus and basic page hit count/analytics
+- Deployment and NixOS module to testing-subdomains
+- Sitemap
+- Move [[id:arroyo/django/generators][The Arroyo Generators]] and perhaps [[id:arcology/django/roam][The Arcology Roam Models]] in to [[id:arroyo/arroyo][Arroyo Systems Management]]
+- [[id:20231113T195508.942155][Rebuild of The Complete Computer]]
+- Consider having a second sqlite3 with server-written state like hit counts and fedi urls and whatnot that i am nervous to store in a DB right now.
+- tests, unit tests, functional tests with org-documents, eugh.
 
 * [[id:20220116T143655.499306][Hey Smell This]]

arcology.org

@@ -2,10 +2,13 @@
 :ID:       arcology/django/arcology-models
 :END:
 #+TITLE: The Arcology's Data Models and Web Server
-#+filetags: :Project:
+#+filetags: :Project:Arcology:
 #+ARCOLOGY_KEY: arcology/django/arcology
 
-* Data Models for routing to org-roam pages
+* Data Models for Sites, Web Features, and Feeds
+:PROPERTIES:
+:ID:       20240204T234334.762591
+:END:
 
 #+begin_src python :tangle arcology/models.py
 from __future__ import annotations
@@ -13,6 +16,7 @@ from typing import Optional, List
 from django.db import models
 from django.conf import settings
 import arrow
+import functools
 
 import arroyo.arroyo_rs as native
 
@@ -22,6 +26,11 @@ import logging
 
 logger = logging.getLogger(__name__)
 logger.setLevel(logging.WARN)
+
+# used for some memoization
+class hashabledict(dict):
+    def __hash__(self):
+        return hash(tuple(sorted(self.items())))
 #+end_src
 
 ** Site
@@ -138,7 +147,7 @@ class Page(models.Model):
     title = models.CharField(max_length=512)
     allow_crawl = models.BooleanField(default=False)
 
-    def urlize_self(self):
+    def to_url(self):
         site = self.site
         return site.urlize_page(self)
 
@@ -175,8 +184,15 @@ class Page(models.Model):
         my_headings = self.file.heading_set.all()
         return set(roam.models.Link.objects.filter(dest_heading__in=my_headings))
 
+    def to_html(self, links, heading=None, include_subheadings=False):
+        return self._to_html_memoized(hashabledict(links), heading, include_subheadings, self.file.digest)
 
-    def to_html(self, links, headings=[], include_subheadings=False):
+    @functools.lru_cache(maxsize=500)
+    def _to_html_memoized(self, links, heading, include_subheadings, _file_digest):
+        if heading is not None:
+            headings = [heading]
+        else:
+            headings = []
         opts = native.ExportOptions(
             link_retargets=links,
             limit_headings=headings,
@@ -268,13 +284,6 @@ class Feed(models.Model):
     title = models.CharField(max_length=512)
     visibility = models.CharField(max_length=512, choices=POST_VISIBILITY)
 
-    def to_atom(self, links):
-        raise "Re-implement this!"
-        opts = native.ExportOptions(
-            link_retargets=links
-        )
-        return native.atomize_file(self.file.path, opts)
-
     @classmethod
     def create_from_arroyo(cls, doc: native.Document) -> Feed | None:
         route_key = next(iter(doc.collect_keywords("ARCOLOGY_FEED")), None)
@@ -342,26 +351,9 @@ migrations.CreateModel(
     ],
 ),
 #+end_src
-** NEXT FeedEntry
+** FeedEntry
 
-A FeedEntry is a Heading with a PUBDATE property that exists on a page w/ ARCOLOGY_FEED Keyword
-
-#+begin_src python
-feed_kws = roam.models.Keyword.objects.filter(value="garden/updates.xml", keyword="ARCOLOGY_FEED")
-
-headings = [
-    item
-    for kw in feed_kws
-    for item in kw.path.heading_set \
-      .filter(headingproperty__keyword="PUBDATE") \
-      .exclude(tag__tag__in=["noexport", "NOEXPORT"]) \
-      .all()
-]
-
-[
-    h.title for h in headings
-]
-#+end_src
+A FeedEntry is a Heading with a PUBDATE property that exists on a page w/ ARCOLOGY_FEED Keyword. These are used to construct =Feeds=
 
 #+begin_src python :tangle arcology/models.py
 class FeedEntry(models.Model):
@@ -390,6 +382,10 @@ class FeedEntry(models.Model):
     pubdate = models.DateTimeField(auto_now=False)
 
     def to_html(self, links):
+        return self._to_html_memoized(hashabledict(links), self.heading.path.digest)
+
+    @functools.lru_cache(maxsize=500)
+    def _to_html_memoized(self, links, _file_digest):
         opts = native.ExportOptions(
             link_retargets=links,
             limit_headings=[self.heading.node_id],
@@ -545,9 +541,9 @@ from roam.models import Link
 logger = logging.getLogger(__name__)
 #+end_src
 
-** index
+** NEXT =GET /= site index
 
-this will just call the org-page rendering handler tbh
+this will just call the org-page rendering handler for the site's index pages eventually
 
 #+begin_src python :tangle arcology/views.py
 def index(request):
@@ -558,7 +554,7 @@ def index(request):
     return HttpResponse(f"index page for {the_page.file.path}")
 #+end_src
 
-** org-page
+** Arcology Org Page handler 
 :PROPERTIES:
 :ID:       20240202T144002.656093
 :END:
@@ -566,7 +562,7 @@ def index(request):
 - State "INPROGRESS" from              [2023-12-20 Wed 17:48]
 :END:
 
-enough! it is time to get basic page HTML rendering working!
+This constructs a page key, tries to load that page and its HTML, and renders that along with a bunch of other metadata.
 
 #+begin_src python :tangle arcology/views.py
 def org_page(request, key):
@@ -605,7 +601,7 @@ The =page= template extends the app template defined below, which provides four
 {% extends "arcology/app.html" %}
 #+end_src
 
-the tab title is assembled from the page and site title:
+The tab title is assembled from the page and site title:
 
 #+begin_src jinja2 :tangle arcology/templates/arcology/page.html
 {% block title %}{{ head_title }}{% endblock %}
@@ -626,10 +622,11 @@ If the site has any feeds, they're injected in to the =<head>= along with any pa
 {% endblock %}
 #+end_src
 
-The main content block contains the =<main>= generated by the native parser, and a sidebar containing backlinks, and page metadata.
+The main =content= block contains the =<main>= generated by the native parser, and a sidebar containing backlinks, and page metadata, and other crap.
 
 #+begin_src jinja2 :tangle arcology/templates/arcology/page.html
 {% block content %}
+  {# HTML is sent through without HTML Escaping via | safe #}
   {{ html_content | safe }}
 
   <section class="sidebar">
@@ -669,6 +666,8 @@ The main content block contains the =<main>= generated by the native parser, and
 {% endblock %}
 #+end_src
 
+*** Org Page Stylings
+
 Most of the page CSS is defined below, but the content CSS is here, nearer the actual implementation of the flexbox:
 
 #+begin_src css :tangle static/arcology/css/app.css :mkdirp yes
@@ -726,26 +725,22 @@ Here are some [[https://medium.com/@massimo.cassandro/flexbox-separators-b284d6d
 }
 #+end_src
 
-** NEXT sitemap
+** INPROGRESS Atom Feed Handler
+:PROPERTIES:
+:ID:       20240204T234814.612917
+:END:
+:LOGBOOK:
+- State "INPROGRESS" from "NEXT"       [2024-02-04 Sun 23:48]
+:END:
 
-this is maybe its own django app...
-
-#+begin_src python :tangle arcology/views.py
-def sitemap(request):
-    # query links etc to create a JSON doc for SigmaJS
-    return HttpResponse(b"sitemap")
-#+end_src
-
-** NEXT feed
-
-all the pandoc based feed generator stuff will need to be recreated or
-bodged in, and all that probably should go in its own django app.
+This uses the sub-feature of the HTML exporter to export only certain sub-headings in [[id:20231023T115950.248543][The arroyo_rs Native Org Parser]]. The =FeedEntry='s defined above are used to construct the feed. I do some gnarly stuff including just stuffing a custom Django template filter in to there so that I can keep a bunch of =node ID= -> =$thing= maps so that when I make the feed entries I can just reach in to a few dicts instead of shaping that all on the handler. But [[roam:仕方がない][仕方がない]]...
 
 #+begin_src python :tangle arcology/views.py
 import arrow 
 import roam.models
 
 def feed(request, key):
+    # Get the site and construct the route key
     site = Site.from_request(request)
     if site.key == "localhost":
         full_key = key
@@ -753,42 +748,40 @@ def feed(request, key):
         site = Site.objects.filter(key=new_site_key).first()
     else:
         full_key = f"{site.key}/{key}"
-    logger.warn(site)
 
-    the_feed = get_object_or_404(Feed, route_key=full_key)
+    # Fetch page metadata
+    the_feed    = get_object_or_404(Feed, route_key=full_key)
+    entries     = the_feed.feedentry_set.order_by("-pubdate").all()[:10]
+    page_author = roam.models.Keyword.objects.get(keyword="AUTHOR", path=the_feed.file).value
+    page_url    = the_feed.file.page_set.first().to_url()
+    updated_at  = arrow.get(entries[0].pubdate).format(arrow.FORMAT_RFC3339) # entries is already sorted
+
+    # node-id -> URL
     links = the_feed.file.page_set.first().collect_links()
-    entries = the_feed.feedentry_set.order_by("-pubdate").all()[:10]
+    # node-id -> HTML
     html_map = {
         entry.heading.node_id: entry.to_html(links=links) for entry in entries
     }
+    # node-id -> PUBDATE heading property
     pubdate_map = {
         entry.heading.node_id: arrow.get(entry.pubdate).format(arrow.FORMAT_RFC3339) for entry in entries
     }
 
-    page_author = roam.models.Keyword.objects.get(keyword="AUTHOR", path=the_feed.file).value
-
     # return HttpResponse("",content_type="application/atom+xml")
     return render(request, "arcology/feed.xml", dict(
-        title="Test",
-        page_url=the_feed.file.page_set.first().urlize_self(),
+        title=the_feed.title,
+        page_url=page_url,
         author=page_author,
-        updated_at=arrow.get(entries[0].pubdate).format(arrow.FORMAT_RFC3339),
+        updated_at=updated_at,
 
         feed_entries=entries,
         htmls=html_map,
         pubdates=pubdate_map,
-
         links=links,
     ), content_type="application/atom+xml")
-
-from django.template.defaulttags import register
-
-@register.filter
-def get_item(dictionary, key):
-    return dictionary.get(key)
 #+end_src
 
-Feed template:
+An Atom feed is pretty simple, it's an XML document with multiple =<entry>='s and the metadata we collected above. For once i'm glad that Python templating treats strings as HTML-Unsafe and escapes the generated HTML used in the Summary for me. This bit me in the past, with the FastAPI version -- the stuff that goes inside of =type = "html"= elements isn't necessarily valid XML so it needs to get escaped. 
 
 #+begin_src jinja2 :tangle arcology/templates/arcology/feed.xml :mkdirp yes
 <?xml version="1.0" encoding="utf-8"?>
@@ -815,42 +808,28 @@ Feed template:
 </feed>
 #+end_src
 
-** Per-Site link color CSS endpoint
-:PROPERTIES:
-:ID:       20231229T215425.830707
-:END:
+*** NEXT move this function to somewhere else more reasonable
 
-This endpoint generates a dynamic CSS file that colorizes internal URLs based on the [[id:20231229T164611.256424][The Arcology's Site List]] which is stored in the database. It does something [[https://twitter.com/gotMLK7/status/1675994399086641152][extremely wicked]] to make the page links less jarring until you hover over them by faking an alpha-channel in to the color.
+This template relies on this custom Django template i [[https://stackoverflow.com/questions/8000022/django-template-how-to-look-up-a-dictionary-value-with-a-variable][nicked from StackOverflow]] to access a dict with a variable key.
 
 #+begin_src python :tangle arcology/views.py
-def site_css(request):
-    sites = Site.objects.all()
-    stanzas = []
-    for site in sites:
-      for domain in site.sitedomain_set.all():
-        stanzas.append(f'''
-          a[href*="//{domain.domain}"] {{
-            border-radius: 0.25em;
-            padding: 0.1em;
-            background-color: {site.link_color}66;
-          }}
-          a[href*="//{domain.domain}"]:hover {{
-            background-color: {site.link_color}FF !important;
-          }}
-        ''')
-    stanzas.append(f'''
-      a[href*="/404"] {{
-        color: var(--alert);
-        text-decoration: line-through;
-      }}
-      a[href*="/404"]::after {{
-        content: " ⚠";
-      }}
-      a[href*="/404"]::before {{
-        content: "⚠ ";
-      }}
-    ''')
-    return HttpResponse(stanzas, content_type="text/css")
+from django.template.defaulttags import register
+
+@register.filter
+def get_item(dictionary, key):
+    return dictionary.get(key)
+#+end_src
+
+*** NEXT [#A] see if the IDs are consistent with the old generator
+
+** NEXT sitemap
+
+this is maybe its own django app...
+
+#+begin_src python :tangle arcology/views.py
+def sitemap(request):
+    # query links etc to create a JSON doc for SigmaJS
+    return HttpResponse(b"sitemap")
 #+end_src
 
 ** Arcology Site Templates
@@ -922,7 +901,7 @@ A footer contains the oh-so-important copyright notice and a limited privacy pol
 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
     <footer>
       <hr/>
-      &copy; 02023 <a href="https://arcology.garden/people/rrix">Ryan Rix</a> &lt;<a href="mailto:site@whatthefuck.computer">site@whatthefuck.computer</a>&gt;
+      &copy; 02024 <a href="https://arcology.garden/people/rrix">Ryan Rix</a> &lt;<a href="mailto:site@whatthefuck.computer">site@whatthefuck.computer</a>&gt;
 
       <br/>
 
@@ -1045,7 +1024,7 @@ a::visited {
 
 There are per-site CSS in [[id:20231229T164611.256424][The Arcology's Site List]].
 
-*** Generataing =@font-face= rules for a bunch of fonts
+*** Generating =@font-face= rules for a bunch of fonts
 
 [[id:cce/vulfpeck_fonts_are_fun][Vulfpeck Fonts]] are pulled in with this code-gen because writing =@font-face= rules does not bring joy and I don't have the right to redistribute these files, so I won't check it in at all.
 
@@ -1084,3 +1063,41 @@ There are per-site CSS in [[id:20231229T164611.256424][The Arcology's Site List]
         tbl)
   (write-file "~/org/arcology-django/static/arcology/css/vulf.css"))
 #+end_src
+
+** Per-Site link color dynamic CSS endpoint
+:PROPERTIES:
+:ID:       20231229T215425.830707
+:END:
+
+This endpoint generates a dynamic CSS file that colorizes internal URLs based on the [[id:20231229T164611.256424][The Arcology's Site List]] which is stored in the database. It does something [[https://twitter.com/gotMLK7/status/1675994399086641152][extremely wicked]] to make the page links less jarring until you hover over them by faking an alpha-channel in to the color.
+
+#+begin_src python :tangle arcology/views.py
+def site_css(request):
+    sites = Site.objects.all()
+    stanzas = []
+    for site in sites:
+      for domain in site.sitedomain_set.all():
+        stanzas.append(f'''
+          a[href*="//{domain.domain}"] {{
+            border-radius: 0.25em;
+            padding: 0.1em;
+            background-color: {site.link_color}66;
+          }}
+          a[href*="//{domain.domain}"]:hover {{
+            background-color: {site.link_color}FF !important;
+          }}
+        ''')
+    stanzas.append(f'''
+      a[href*="/404"] {{
+        color: var(--alert);
+        text-decoration: line-through;
+      }}
+      a[href*="/404"]::after {{
+        content: " ⚠";
+      }}
+      a[href*="/404"]::before {{
+        content: "⚠ ";
+      }}
+    ''')
+    return HttpResponse(stanzas, content_type="text/css")
+#+end_src

arcology/models.py

@@ -1,9 +1,10 @@
-# [[file:../arcology.org::*Data Models for routing to org-roam pages][Data Models for routing to org-roam pages:1]]
+# [[file:../arcology.org::*Data Models for Sites, Web Features, and Feeds][Data Models for Sites, Web Features, and Feeds:1]]
 from __future__ import annotations
 from typing import Optional, List
 from django.db import models
 from django.conf import settings
 import arrow
+import functools
 
 import arroyo.arroyo_rs as native
 
@@ -13,7 +14,12 @@ import logging
 
 logger = logging.getLogger(__name__)
 logger.setLevel(logging.WARN)
-# Data Models for routing to org-roam pages:1 ends here
+
+# used for some memoization
+class hashabledict(dict):
+    def __hash__(self):
+        return hash(tuple(sorted(self.items())))
+# Data Models for Sites, Web Features, and Feeds:1 ends here
 
 # [[file:../arcology.org::*Site][Site:1]]
 # Sites and SiteDomains are created in django-admin or a seed rather than from arroyo parser, no create_from_arroyo..!
@@ -72,7 +78,7 @@ class Page(models.Model):
     title = models.CharField(max_length=512)
     allow_crawl = models.BooleanField(default=False)
 
-    def urlize_self(self):
+    def to_url(self):
         site = self.site
         return site.urlize_page(self)
 
@@ -109,8 +115,15 @@ class Page(models.Model):
         my_headings = self.file.heading_set.all()
         return set(roam.models.Link.objects.filter(dest_heading__in=my_headings))
 
+    def to_html(self, links, heading=None, include_subheadings=False):
+        return self._to_html_memoized(hashabledict(links), heading, include_subheadings, self.file.digest)
 
-    def to_html(self, links, headings=[], include_subheadings=False):
+    @functools.lru_cache(maxsize=500)
+    def _to_html_memoized(self, links, heading, include_subheadings, _file_digest):
+        if heading is not None:
+            headings = [heading]
+        else:
+            headings = []
         opts = native.ExportOptions(
             link_retargets=links,
             limit_headings=headings,
@@ -161,13 +174,6 @@ class Feed(models.Model):
     title = models.CharField(max_length=512)
     visibility = models.CharField(max_length=512, choices=POST_VISIBILITY)
 
-    def to_atom(self, links):
-        raise "Re-implement this!"
-        opts = native.ExportOptions(
-            link_retargets=links
-        )
-        return native.atomize_file(self.file.path, opts)
-
     @classmethod
     def create_from_arroyo(cls, doc: native.Document) -> Feed | None:
         route_key = next(iter(doc.collect_keywords("ARCOLOGY_FEED")), None)
@@ -196,7 +202,7 @@ class Feed(models.Model):
         )
 # Feed:1 ends here
 
-# [[file:../arcology.org::*FeedEntry][FeedEntry:2]]
+# [[file:../arcology.org::*FeedEntry][FeedEntry:1]]
 class FeedEntry(models.Model):
     POST_VISIBILITY = [
         ("unlisted", "Unlisted"),
@@ -223,6 +229,10 @@ class FeedEntry(models.Model):
     pubdate = models.DateTimeField(auto_now=False)
 
     def to_html(self, links):
+        return self._to_html_memoized(hashabledict(links), self.heading.path.digest)
+
+    @functools.lru_cache(maxsize=500)
+    def _to_html_memoized(self, links, _file_digest):
         opts = native.ExportOptions(
             link_retargets=links,
             limit_headings=[self.heading.node_id],
@@ -266,4 +276,4 @@ class FeedEntry(models.Model):
         # title = root_heading.title
 
         return rets
-# FeedEntry:2 ends here
+# FeedEntry:1 ends here

arcology/templates/arcology/app.html

@@ -49,7 +49,7 @@
 {# [[file:../../../arcology.org::*Arcology Site Templates][Arcology Site Templates:5]] #}
 <footer>
   <hr/>
-  &copy; 02023 <a href="https://arcology.garden/people/rrix">Ryan Rix</a> &lt;<a href="mailto:site@whatthefuck.computer">site@whatthefuck.computer</a>&gt;
+  &copy; 02024 <a href="https://arcology.garden/people/rrix">Ryan Rix</a> &lt;<a href="mailto:site@whatthefuck.computer">site@whatthefuck.computer</a>&gt;
 
   <br/>
 

arcology/templates/arcology/feed.xml

@@ -1,4 +1,4 @@
-{# [[file:../../../arcology.org::*feed][feed:2]] #}
+{# [[file:../../../arcology.org::*Atom Feed Handler][Atom Feed Handler:2]] #}
 <?xml version="1.0" encoding="utf-8"?>
 <feed xmlns="http://www.w3.org/2005/Atom">
 
@@ -21,4 +21,4 @@
 {% endfor %}
 
 </feed>
-{# feed:2 ends here #}
+{# Atom Feed Handler:2 ends here #}

arcology/templates/arcology/page.html

@@ -21,6 +21,7 @@
 
 {# [[file:../../../arcology.org::*Rendering the page][Rendering the page:4]] #}
 {% block content %}
+  {# HTML is sent through without HTML Escaping via | safe #}
   {{ html_content | safe }}
 
   <section class="sidebar">

arcology/views.py

@@ -9,16 +9,16 @@ from roam.models import Link
 logger = logging.getLogger(__name__)
 # the web server:2 ends here
 
-# [[file:../arcology.org::*index][index:1]]
+# [[file:../arcology.org::*=GET /= site index][=GET /= site index:1]]
 def index(request):
     site = Site.from_request(request)
     full_key = f"{site.key}/index"
 
     the_page = Page.objects.get(route_key=full_key)
     return HttpResponse(f"index page for {the_page.file.path}")
-# index:1 ends here
+# =GET /= site index:1 ends here
 
-# [[file:../arcology.org::*org-page][org-page:1]]
+# [[file:../arcology.org::*Arcology Org Page handler][Arcology Org Page handler:1]]
 def org_page(request, key):
     site = Site.from_request(request)
     if site.key == "localhost":
@@ -45,19 +45,14 @@ def org_page(request, key):
         keywords=the_page.collect_keywords().all(),
         references=the_page.collect_references(),
     ))
-# org-page:1 ends here
+# Arcology Org Page handler:1 ends here
 
-# [[file:../arcology.org::*sitemap][sitemap:1]]
-def sitemap(request):
-    # query links etc to create a JSON doc for SigmaJS
-    return HttpResponse(b"sitemap")
-# sitemap:1 ends here
-
-# [[file:../arcology.org::*feed][feed:1]]
+# [[file:../arcology.org::*Atom Feed Handler][Atom Feed Handler:1]]
 import arrow 
 import roam.models
 
 def feed(request, key):
+    # Get the site and construct the route key
     site = Site.from_request(request)
     if site.key == "localhost":
         full_key = key
@@ -65,42 +60,54 @@ def feed(request, key):
         site = Site.objects.filter(key=new_site_key).first()
     else:
         full_key = f"{site.key}/{key}"
-    logger.warn(site)
 
-    the_feed = get_object_or_404(Feed, route_key=full_key)
+    # Fetch page metadata
+    the_feed    = get_object_or_404(Feed, route_key=full_key)
+    entries     = the_feed.feedentry_set.order_by("-pubdate").all()[:10]
+    page_author = roam.models.Keyword.objects.get(keyword="AUTHOR", path=the_feed.file).value
+    page_url    = the_feed.file.page_set.first().to_url()
+    updated_at  = arrow.get(entries[0].pubdate).format(arrow.FORMAT_RFC3339) # entries is already sorted
+
+    # node-id -> URL
     links = the_feed.file.page_set.first().collect_links()
-    entries = the_feed.feedentry_set.order_by("-pubdate").all()[:10]
+    # node-id -> HTML
     html_map = {
         entry.heading.node_id: entry.to_html(links=links) for entry in entries
     }
+    # node-id -> PUBDATE heading property
     pubdate_map = {
         entry.heading.node_id: arrow.get(entry.pubdate).format(arrow.FORMAT_RFC3339) for entry in entries
     }
 
-    page_author = roam.models.Keyword.objects.get(keyword="AUTHOR", path=the_feed.file).value
-
     # return HttpResponse("",content_type="application/atom+xml")
     return render(request, "arcology/feed.xml", dict(
-        title="Test",
-        page_url=the_feed.file.page_set.first().urlize_self(),
+        title=the_feed.title,
+        page_url=page_url,
         author=page_author,
-        updated_at=arrow.get(entries[0].pubdate).format(arrow.FORMAT_RFC3339),
+        updated_at=updated_at,
 
         feed_entries=entries,
         htmls=html_map,
         pubdates=pubdate_map,
-
         links=links,
     ), content_type="application/atom+xml")
+# Atom Feed Handler:1 ends here
 
+# [[file:../arcology.org::*move this function to somewhere else more reasonable][move this function to somewhere else more reasonable:1]]
 from django.template.defaulttags import register
 
 @register.filter
 def get_item(dictionary, key):
     return dictionary.get(key)
-# feed:1 ends here
+# move this function to somewhere else more reasonable:1 ends here
 
-# [[file:../arcology.org::*Per-Site link color CSS endpoint][Per-Site link color CSS endpoint:1]]
+# [[file:../arcology.org::*sitemap][sitemap:1]]
+def sitemap(request):
+    # query links etc to create a JSON doc for SigmaJS
+    return HttpResponse(b"sitemap")
+# sitemap:1 ends here
+
+# [[file:../arcology.org::*Per-Site link color dynamic CSS endpoint][Per-Site link color dynamic CSS endpoint:1]]
 def site_css(request):
     sites = Site.objects.all()
     stanzas = []
@@ -129,4 +136,4 @@ def site_css(request):
       }}
     ''')
     return HttpResponse(stanzas, content_type="text/css")
-# Per-Site link color CSS endpoint:1 ends here
+# Per-Site link color dynamic CSS endpoint:1 ends here

configuration.org

@@ -7,7 +7,7 @@
 #+ARCOLOGY_KEY: arcology/django/configuration
 ,#+AUTO_TANGLE: vars:org-babel-default-header-args
 
-user-configuration ; these could be generated from org tables if i'm feeling like a proper sicko. They should at least be much more legible but that's true of all the code i'm pulling in here..
+user-configuration ; These are mostly generated from org-mode tables if you're meant to be editing or extending them. This is a cool feature of org-babel, where you can use tables as data for code which can output more code or even org headings or links. We use this to generate configuration.
 
 * The Arcology's Site List
 :PROPERTIES:
@@ -38,6 +38,8 @@ These can be customized by the user:
 | localhost | 127.0.0.1:8000           |
 | localhost | localhost:8000           |
 
+** The Code
+
 Oh we doing some code gen in here, this is just gonna generate some JSON for now joining these two tables lul.
 
 #+name: elisp-join-tables
@@ -63,7 +65,10 @@ That generates JSON that goes in to =arcology/settings/sites.json= for the [[id:
 #+end_src
 
 ** NEXT How to re-seed these if you change the configuration?
-** Per-site CSS
+
+for now i just blow away the database but gosh!
+
+* Per-site CSS
 
 These are generated from a table, too, because I have brainworms.
 Note that they're basically the same, but you can customize it further to your satisfaction.
@@ -77,6 +82,8 @@ Note that they're basically the same, but you can customize it further to your s
 | cce            | #cc6960 | #707231 | #ebbe7b   | #67b4f8 | #7e5c41 | #fcfcfc | #dcdcec | #cacada | #808090 | #211f1c |
 | arcology       | #cc6960 | #707231 | #ebbe7b   | #67b4f8 | #7e5c41 | #fcf6ed | #f6e5cb | #baad9b | #82796c | #211f1c |
 
+** The Code 
+
 #+NAME: mk-site-colors
 #+begin_src emacs-lisp :var colors=site-colors :var site="default-colors" :results none
 (pcase-let ((`((,name ,alert ,primary ,secondary ,success ,warning ,white ,gray2 ,gray3 , gray4 ,black))
@@ -265,6 +272,10 @@ ARROYO_BASE_DIR = str(pathlib.Path("~/arroyo-nix").expanduser())
 #+END_SRC
 
 ** Defining extension modules
+:PROPERTIES:
+:ID:       20240204T232637.286228
+:ROAM_ALIASES: "Arcology Extractor and Arroyo Generator Configuration"
+:END:
 
 It should be feasible to add new data types and models and generators to the Arcology system to fit a user's needs. The defaults are specified here.
 
@@ -302,7 +313,7 @@ ARROYO_EXTRACTORS = {
 
 ** Enumerating System Roles
 
-I would like to make this optional but define these for now...
+I would like to not need these enumerated here, it seems like it should be easier to add a new role than that. But for now here they are so that the DB seed works better.
 
 #+NAME: arroyo-roles
 | server   | [[id:20220101T195846.044283][the Wobserver Configuration with Arroyo Nixos]] $ |
@@ -319,7 +330,9 @@ ARROYO_ROLES = [
 
 ** Roam Allowed Keywords
 
-These are used to provide system features:
+These tables will be used to create an allowlist of =#+KEYWORD='s to store in the database.
+
+These are used to provide system features or power the various generators:
 #+NAME: roam_allowed_keywords
 | ARCOLOGY_FEED            | Add a route that will generate an Atom feed for this page                                             |
 | ARCOLOGY_KEY             | Add a route that will generate an HTML page for this page                                             |

roam.org

@@ -10,7 +10,11 @@
 
 at the top here, describe the class layout and the overall usage of this module.
 
-* Models
+* Org-Roam Caching Models
+:PROPERTIES:
+:ID:       20240204T234111.701754
+:ROAM_ALIASES: "Arcology Org-Roam Caching Models"
+:END:
 
 #+BEGIN_SRC python :tangle roam/models.py
 from __future__ import annotations
@@ -120,7 +124,7 @@ class Heading(models.Model):
 
     def to_url(self) -> str:
         page = self.path.page_set.first()
-        page_url = page.urlize_self()
+        page_url = page.to_url()
         if self.level == 0:
             return page_url
         else:
@@ -687,3 +691,5 @@ class RoamConfig(AppConfig):
     default_auto_field = "django.db.models.BigAutoField"
     name = "roam"
 #+end_src
+
+* NEXT move this in to [[id:arroyo/arroyo][Arroyo Systems Management]], along with the [[id:arroyo/django/generators][The Arroyo Generators]].

roam/models.py

@@ -100,7 +100,7 @@ class Heading(models.Model):
 
     def to_url(self) -> str:
         page = self.path.page_set.first()
-        page_url = page.urlize_self()
+        page_url = page.to_url()
         if self.level == 0:
             return page_url
         else:

static/arcology/css/app.css

@@ -1,4 +1,4 @@
-/* [[file:../../../arcology.org::*Rendering the page][Rendering the page:5]] */
+/* [[file:../../../arcology.org::*Org Page Stylings][Org Page Stylings:1]] */
 .content {
   margin-left: auto;
   margin-right: auto;
@@ -23,9 +23,9 @@
   flex-shrink: 1;
   flex-basis: 30ch;
 }
-/* Rendering the page:5 ends here */
+/* Org Page Stylings:1 ends here */
 
-/* [[file:../../../arcology.org::*Rendering the page][Rendering the page:6]] */
+/* [[file:../../../arcology.org::*Org Page Stylings][Org Page Stylings:2]] */
 section.sidebar {
   display: flex;
   flex-flow: column wrap;
@@ -34,9 +34,9 @@ section.sidebar {
 section.sidebar > div.backlinks {
   flex-grow: 1;
 }
-/* Rendering the page:6 ends here */
+/* Org Page Stylings:2 ends here */
 
-/* [[file:../../../arcology.org::*Rendering the page][Rendering the page:7]] */
+/* [[file:../../../arcology.org::*Org Page Stylings][Org Page Stylings:3]] */
 .content::before {
     align-self: stretch;
     content: '';
@@ -47,7 +47,7 @@ section.sidebar > div.backlinks {
 .content > *:first-child {
     order: -1;
 }
-/* Rendering the page:7 ends here */
+/* Org Page Stylings:3 ends here */
 
 /* [[file:../../../arcology.org::*CSS][CSS:1]] */
 body {