fix bug i introduced to watchsync

README.org

@@ -46,6 +46,7 @@ This project is a living document. It's [[id:cce/literate_programming][Literate
 - [[id:arcology/django/config][Arcology Project configuration]] describes and implements the knobs the user can twiddle to configure the Arcology's operation
 - [[id:arcology/django/interfaces][Interfacing with the Arcology]] provides a set of management commands and a Syncthing client which will automatically ingest new files in to the database. This document describes all the tools a prospective user of the CCE will need to use for surface-level features.
 - [[id:20240213T124300.774781][Deploying the Arcology]] lays out the basics of operating a Wobserver, but also provides a methodology for bootstrapping the Arcology on systems not already running NixOS.
+- [[id:20240226T132507.817450][The Arcology's Site Maps and Discovery Mechanisms]] show you how you can discover content published in an Arcology.
 - The [[id:arcology/django/scaffolding][Arcology Project Scaffolding]] contains the files necessary to run the project, defines the nix environment, and the python environment, and the base Django apparatus.
 - The [[id:arcology/django/roam][Arcology Roam Models]] are the base metadata of the [[id:cce/org-roam][org-roam]] documents
 - [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]] are the web publishing platform and the data models it requires to be operated
@@ -76,7 +77,11 @@ This project is a living document. It's [[id:cce/literate_programming][Literate
 - State "DONE"       from "NEXT"       [2024-02-17 Sat 21:27]
 :END:
 
-** NEXT Sitemap
+** NEXT [[id:20240226T132507.817450][Sitemap]]
+:LOGBOOK:
+CLOCK: [2024-02-26 Mon 15:22]--[2024-02-26 Mon 16:16] =>  0:54
+CLOCK: [2024-02-26 Mon 13:24]--[2024-02-26 Mon 14:59] =>  1:35
+:END:
 ** NEXT Move [[id:arroyo/django/generators][The Arroyo Generators]] and perhaps [[id:arcology/django/roam][The Arcology Roam Models]] in to [[id:arroyo/arroyo][Arroyo Systems Management]]
 ** NEXT [[id:20231113T195508.942155][Rebuild of The Complete Computer]]
 ** NEXT Consider having a second sqlite3 with server-written state like hit counts and fedi urls and whatnot that i am nervous to store in a DB right now.

arcology.org

@@ -543,10 +543,10 @@ urlpatterns = [
     path("", views.index),
     path("robots.txt", views.robots, name="robots_txt"),
     path("404", views.unpublished, name="page_not_found"),
-    path("sitemap", views.sitemap, name="sitemap"),
     path("sites.css", views.site_css, name="site-css"),
     path("feeds.json", views.feed_list, name="feed-list"),
     path("", include("django_prometheus.urls")),
+    path("", include("sitemap.urls")),
     # ensure these ones are last because they're greedy!
     re_path("(?P<key>[0-9a-zA-Z/_\-]+\.xml)", views.feed, name="feed"),
     re_path("(?P<key>[0-9a-zA-Z/_\-]+)", views.org_page, name="org-page"),
@@ -811,7 +811,11 @@ Here are some [[https://medium.com/@massimo.cassandro/flexbox-separators-b284d6d
 .content > *:first-child {
     order: -1;
 }
+#+end_src
 
+And some simple image wrangling:
+
+#+begin_src css :tangle arcology/static/arcology/css/app.css :mkdirp yes
 .content img {
   display: block;
   width: 80%;
@@ -819,7 +823,21 @@ Here are some [[https://medium.com/@massimo.cassandro/flexbox-separators-b284d6d
 }
 #+end_src
 
-And some simple image wrangling:
+These rules annotate task headings by inserting an icon before them.
+
+#+begin_src css :tangle arcology/static/arcology/css/app.css :mkdirp yes
+.task.task-DONE::before {
+  content: '☑️ ';
+}
+
+.task.task-NEXT::before {
+  content: '🆕 ';
+}
+
+.task.task-INPROGRESS::before {
+  content: '⏳ ';
+}
+#+end_src
 
 ** INPROGRESS Atom Feed Handler
 :PROPERTIES:
@@ -923,16 +941,6 @@ def get_item(dictionary, key):
 
 *** NEXT [#A] see if the IDs are consistent with the old generator
 
-** NEXT sitemap
-
-this is maybe its own django app...
-
-#+begin_src python :tangle arcology/views.py
-def sitemap(request):
-    # query links etc to create a JSON doc for SigmaJS
-    return HttpResponse(b"sitemap")
-#+end_src
-
 ** NEXT unpublished/not found endpoint
 
 There are plenty of links inside the Arcology which aren't meant to be clicked. =roam:= stub links will of course 
@@ -1033,6 +1041,7 @@ The base template provides some basic information and loads the CSS sheets neces
 
 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
   {% load static %}
+  {% load django_htmx %}
   <link rel="stylesheet" href="{% static 'arcology/css/app.css' %}"/>
   <link rel="stylesheet" href="{% static 'arcology/css/vulf.css' %}"/>
   <link rel="stylesheet" href="{% static 'arcology/css/default-colors.css' %}"/>
@@ -1044,14 +1053,15 @@ The base template provides some basic information and loads the CSS sheets neces
   <meta name="generator" content="Arcology Site Engine https://engine.arcology.garden/"/>
   <meta name="viewport" content="width=device-width, initial-scale=1">
   <title>{% block title %}{{head_title | default:"The Arcology Project" }}{% endblock %}</title>
+  <script src="{% static 'sitemap/js/htmx.js' %}" defer></script>
   {% block extra_head %}{% endblock %}
-  </head>
+</head>
 #+end_src
 
 The body consists of a header which has the site and page title (which can be overridden for example in the =index= handler to only show the site title) and links to the other sites. These should be loaded from the DB eventually.
 
 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
-  <body>
+<body>
   <header>
     <div class="header-content">
       {% block h1 %}
@@ -1072,15 +1082,15 @@ The body consists of a header which has the site and page title (which can be ov
 The =content= block is used in child templates to hide a =<main>=; the =content= div *should* be a main element instead but [[id:20231023T115950.248543][The arroyo_rs Native Org Parser]] wants to output a =<main>= and i'm not going to stop it, so the div is there to make the body's flexbox layout work.
 
 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
-    <div class="content">
+<div class="content">
   {% block content %}{% endblock %}
-    </div>
+</div>
 #+end_src
 
 A footer contains the oh-so-important copyright notice and a limited privacy policy which I should update before I ship this, along with links to the sitemap and to [[https://fediring.net][my fediring neighbors]].
 
 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
-    <footer>
+<footer>
   <hr/>
   &copy; 02024 <a href="https://arcology.garden/people/rrix">Ryan Rix</a> &lt;<a href="mailto:site@whatthefuck.computer">site@whatthefuck.computer</a>&gt;
 
@@ -1271,7 +1281,7 @@ def site_css(request):
     stanzas.append(f'''
       a[href*="/404"] {{
         color: var(--alert);
-        text-decoration: line-through;
+ /* text-decoration: line-through; */
       }}
       a[href*="/404"]::after {{
         content: " ⚠";

arcology/settings/__init__.py

@@ -33,7 +33,9 @@ INSTALLED_APPS = [
     "arcology", # [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]]
     "generators", # [[id:arroyo/django/generators][The Arroyo Generators]]
     "syncthonk", # [[id:20231218T183551.765340][Arcology watchsync Command]]
+    "sitemap", # [[id:20240226T132507.817450][The Arcology's Site Maps and Discovery Mechanisms]]
     
+    "django_htmx",
     "django_prometheus",
     "django.contrib.admin",
     "django.contrib.auth",
@@ -81,6 +83,7 @@ LOGGING = {
 # [[file:../../configuration.org::*Look don't worry about the rest of these][Look don't worry about the rest of these:1]]
 MIDDLEWARE = [
     "django_prometheus.middleware.PrometheusBeforeMiddleware",
+    "django_htmx.middleware.HtmxMiddleware",
     "django.middleware.security.SecurityMiddleware",
     "django.contrib.sessions.middleware.SessionMiddleware",
     "django.middleware.common.CommonMiddleware",
@@ -167,6 +170,7 @@ STORAGES = {
 }
 STATICFILES_DIRS = [
     BASE_DIR / "arcology/static",
+    BASE_DIR / "sitemap/static",
 ]
 
 STATIC_ROOT = os.getenv("ARCOLOGY_STATIC_ROOT", "/var/www/arcology")

arcology/static/arcology/css/app.css

@@ -47,13 +47,29 @@ section.sidebar > div.backlinks {
 .content > *:first-child {
     order: -1;
 }
+/* Org Page-specific CSS Stylings:3 ends here */
 
+/* [[file:../../../../arcology.org::*Org Page-specific CSS Stylings][Org Page-specific CSS Stylings:4]] */
 .content img {
   display: block;
   width: 80%;
   margin: 0 auto;
 }
-/* Org Page-specific CSS Stylings:3 ends here */
+/* Org Page-specific CSS Stylings:4 ends here */
+
+/* [[file:../../../../arcology.org::*Org Page-specific CSS Stylings][Org Page-specific CSS Stylings:5]] */
+.task.task-DONE::before {
+  content: '☑️ ';
+}
+
+.task.task-NEXT::before {
+  content: '🆕 ';
+}
+
+.task.task-INPROGRESS::before {
+  content: '⏳ ';
+}
+/* Org Page-specific CSS Stylings:5 ends here */
 
 /* [[file:../../../../arcology.org::*CSS][CSS:1]] */
 body {

arcology/templates/arcology/app.html

@@ -6,6 +6,7 @@
 
 {# [[file:../../../arcology.org::*Arcology Site Templates][Arcology Site Templates:2]] #}
 {% load static %}
+  {% load django_htmx %}
   <link rel="stylesheet" href="{% static 'arcology/css/app.css' %}"/>
   <link rel="stylesheet" href="{% static 'arcology/css/vulf.css' %}"/>
   <link rel="stylesheet" href="{% static 'arcology/css/default-colors.css' %}"/>
@@ -17,6 +18,7 @@
   <meta name="generator" content="Arcology Site Engine https://engine.arcology.garden/"/>
   <meta name="viewport" content="width=device-width, initial-scale=1">
   <title>{% block title %}{{head_title | default:"The Arcology Project" }}{% endblock %}</title>
+  <script src="{% static 'sitemap/js/htmx.js' %}" defer></script>
   {% block extra_head %}{% endblock %}
 </head>
 {# Arcology Site Templates:2 ends here #}

arcology/urls.py

@@ -9,10 +9,10 @@ urlpatterns = [
     path("", views.index),
     path("robots.txt", views.robots, name="robots_txt"),
     path("404", views.unpublished, name="page_not_found"),
-    path("sitemap", views.sitemap, name="sitemap"),
     path("sites.css", views.site_css, name="site-css"),
     path("feeds.json", views.feed_list, name="feed-list"),
     path("", include("django_prometheus.urls")),
+    path("", include("sitemap.urls")),
     # ensure these ones are last because they're greedy!
     re_path("(?P<key>[0-9a-zA-Z/_\-]+\.xml)", views.feed, name="feed"),
     re_path("(?P<key>[0-9a-zA-Z/_\-]+)", views.org_page, name="org-page"),

arcology/views.py

@@ -134,12 +134,6 @@ def get_item(dictionary, key):
     return dictionary.get(key)
 # move this function to somewhere else more reasonable:1 ends here
 
-# [[file:../arcology.org::*sitemap][sitemap:1]]
-def sitemap(request):
-    # query links etc to create a JSON doc for SigmaJS
-    return HttpResponse(b"sitemap")
-# sitemap:1 ends here
-
 # [[file:../arcology.org::*unpublished/not found endpoint][unpublished/not found endpoint:1]]
 def unpublished(request):
     key = request.GET.get("key")
@@ -210,7 +204,7 @@ def site_css(request):
     stanzas.append(f'''
       a[href*="/404"] {{
         color: var(--alert);
-        text-decoration: line-through;
+        /* text-decoration: line-through; */
       }}
       a[href*="/404"]::after {{
         content: " ⚠";

configuration.org

@@ -245,11 +245,13 @@ basically, each org file in this repository, and maybe one or two of your own, a
 | "arcology"   | [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]]         |
 | "generators" | [[id:arroyo/django/generators][The Arroyo Generators]]                             |
 | "syncthonk"  | [[id:20231218T183551.765340][Arcology watchsync Command]]                        |
+| "sitemap"    | [[id:20240226T132507.817450][The Arcology's Site Maps and Discovery Mechanisms]] |
 
 #+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
 # Application definition
 INSTALLED_APPS = [
     <<gen-config-list(arcology_apps)>>
+    "django_htmx",
     "django_prometheus",
     "django.contrib.admin",
     "django.contrib.auth",
@@ -442,6 +444,7 @@ IGNORED_ROAM_TAGS = [
 #+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
 MIDDLEWARE = [
     "django_prometheus.middleware.PrometheusBeforeMiddleware",
+    "django_htmx.middleware.HtmxMiddleware",
     "django.middleware.security.SecurityMiddleware",
     "django.contrib.sessions.middleware.SessionMiddleware",
     "django.middleware.common.CommonMiddleware",
@@ -528,6 +531,7 @@ STORAGES = {
 }
 STATICFILES_DIRS = [
     BASE_DIR / "arcology/static",
+    BASE_DIR / "sitemap/static",
 ]
 
 STATIC_ROOT = os.getenv("ARCOLOGY_STATIC_ROOT", "/var/www/arcology")

default.nix

@@ -23,6 +23,7 @@ python3.pkgs.buildPythonPackage rec {
     click
     django_4
     django-prometheus
+    django-htmx
     (django-stubs-ext.override { django = django_4; })
     (django-stubs.override { django = django_4; })
     gunicorn

interfaces.org

@@ -306,7 +306,7 @@ class Command(BaseCommand):
     def handle(self, *args, **kwargs):
         self.key = settings.SYNCTHING_KEY
         assert self.key != None
-        if "folder_id" in kwargs.keys():
+        if kwargs.get("folder_id") is not None and len(kwargs.get("folder_id")) > 0:
             self.folder_id = kwargs['folder_id']
             self.expanded_path = self.get_path_for_folder_id(self.folder_id)
             logger.info(f"fetched folder ID {self.folder_id} AKA {self.expanded_path} from syncthing API")

pyproject.toml

@@ -5,7 +5,7 @@ version = "0.0.1"
 description = "org-mode metadata query engine, publishing platform, and computer metaprogrammer"
 # license = "Hey Smell This"
 readme = "README.md"
-dependencies = ["click ~=8.1", "django ~= 4.2", "django-stub", "polling", "django-prometheus", "arroyo", "arrow ~= 1.3.0", "gunicorn ~= 21.0"]
+dependencies = ["click ~=8.1", "django ~= 4.2", "django-stub", "polling", "django-prometheus", "arroyo", "arrow ~= 1.3.0", "gunicorn ~= 21.0", "htmx ~= 1.17" ]
 requires-python = ">=3.10"
 authors = [
     { name = "Ryan Rix", email = "code@whatthefuck.computer" }
@@ -25,6 +25,10 @@ arcology = [
   'static/arcology/css/*',
   'static/arcology/fonts/*',
 ]
+sitemap = [
+  'static/sitemap/js/*',
+  'templates/sitemap/*',
+]
 
 [tool.setuptools.packages.find]
 where = ["."]

roam.org

@@ -419,6 +419,18 @@ class Tag(EMOM('tag'), models.Model):
     )
     tag = models.CharField(max_length=256)
 
+    def pages(self) -> List['Page']:
+        return self.__class__.pages_by_name(self.tag)
+
+    @classmethod
+    def pages_by_name(cls, tag_name: str) -> List['Page']:
+        import arcology.models
+        return set([
+            arcology.models.Page.objects.get(file=tag_obj.heading.path_id)
+            for tag_obj in cls.objects.filter(tag=tag_name).distinct()
+        ])
+
+
     @classmethod
     def create_from_arroyo(cls, doc: native.Document) -> List[Tag]:
         return [

roam/models.py

@@ -157,6 +157,18 @@ class Tag(EMOM('tag'), models.Model):
     )
     tag = models.CharField(max_length=256)
 
+    def pages(self) -> List['Page']:
+        return self.__class__.pages_by_name(self.tag)
+
+    @classmethod
+    def pages_by_name(cls, tag_name: str) -> List['Page']:
+        import arcology.models
+        return set([
+            arcology.models.Page.objects.get(file=tag_obj.heading.path_id)
+            for tag_obj in cls.objects.filter(tag=tag_name).distinct()
+        ])
+
+
     @classmethod
     def create_from_arroyo(cls, doc: native.Document) -> List[Tag]:
         return [

scaffolding.org

@@ -17,7 +17,7 @@ version = "0.0.1"
 description = "org-mode metadata query engine, publishing platform, and computer metaprogrammer"
 # license = "Hey Smell This"
 readme = "README.md"
-dependencies = ["click ~=8.1", "django ~= 4.2", "django-stub", "polling", "django-prometheus", "arroyo", "arrow ~= 1.3.0", "gunicorn ~= 21.0"]
+dependencies = ["click ~=8.1", "django ~= 4.2", "django-stub", "polling", "django-prometheus", "arroyo", "arrow ~= 1.3.0", "gunicorn ~= 21.0", "htmx ~= 1.17" ]
 requires-python = ">=3.10"
 authors = [
     { name = "Ryan Rix", email = "code@whatthefuck.computer" }
@@ -37,6 +37,10 @@ arcology = [
   'static/arcology/css/*',
   'static/arcology/fonts/*',
 ]
+sitemap = [
+  'static/sitemap/js/*',
+  'templates/sitemap/*',
+]
 
 [tool.setuptools.packages.find]
 where = ["."]
@@ -75,6 +79,7 @@ python3.pkgs.buildPythonPackage rec {
     click
     django_4
     django-prometheus
+    django-htmx
     (django-stubs-ext.override { django = django_4; })
     (django-stubs.override { django = django_4; })
     gunicorn
@@ -113,6 +118,7 @@ let
     arroyo_rs
     django_4
     django-prometheus
+    django-htmx
     (django-stubs-ext.override { django = django_4; })
     (django-stubs.override { django = django_4; })
     gunicorn

shell.nix

@@ -14,6 +14,7 @@ let
     arroyo_rs
     django_4
     django-prometheus
+    django-htmx
     (django-stubs-ext.override { django = django_4; })
     (django-stubs.override { django = django_4; })
     gunicorn

sitemap.org

@@ -0,0 +1,183 @@
+:PROPERTIES:
+:ID:       20240226T132507.817450
+:END:
+#+TITLE: The Arcology's Site Maps and Discovery Mechanisms
+
+#+ARCOLOGY_KEY: arcology/sitemaps
+#+ARCOLOGY_ALLOW_CRAWL: t
+
+These pages are perhaps the most "dynamic" or inteactive of the Arcology's features right now.
+
+First there is a set of views which renders an index of all the Pages' tags. Here we start to use [[roam:HTMX]] to render partials dynamically. Rather than load every tag's page listing, the =/tags/= endpoint allows you to drop-down tags with a button press to see posts with that tag.
+
+Then there is the Sitemap page which uses [[id:20220711T151820.326251][SigmaJS]] to render a connected topology of all the pages in the Arcology Project.
+
+* Django View Setup
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+import logging
+from django.http import HttpResponse, HttpResponseNotFound
+from django.shortcuts import render, get_object_or_404
+from django.db.models import Count
+
+from arcology.models import Page, Feed, Site
+from roam.models import Link, Tag
+
+# from prometheus_client import Counter, Histogram
+
+logger = logging.getLogger(__name__)
+#+end_src
+
+#+begin_src python :tangle sitemap/urls.py :mkdirp yes
+from django.contrib import admin
+from django.urls import path, re_path, include
+
+from sitemap import views
+
+urlpatterns = [
+    path("tags", views.tags_index, name="sitemap"),
+    path("tags/", views.tags_index, name="sitemap"),
+    path("tags/<slug:tag>", views.tag_page, name="sitemap"),
+    path("sitemap", views.sitemap, name="sitemap"),
+    path("sitemap.json", views.sitemap_data, name="sitemap"),
+]
+#+end_src
+
+* Tag aggregation pages
+:LOGBOOK:
+CLOCK: [2024-02-26 Mon 15:22]--[2024-02-26 Mon 15:22] =>  0:00
+:END:
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+from django.shortcuts import render, get_object_or_404
+
+def tags_index(request):
+    site = Site.from_request(request)
+    tags = Tag.objects.all().values('tag').annotate(total=Count('tag')).order_by('-total')
+    # tags = [
+    #     {**tag, 'pages': Tag.pages_by_name(tag['tag'])}
+    #     for tag in tags
+    #    ] 
+    return render(request, "sitemap/tags.html", dict(
+        tags=tags,
+        site=site,
+        feeds=site.feed_set.all()
+    ))
+#+end_src
+
+The =tags= template lists all the tags and lets you click on them to see the pages in one. .o(it probably should just show all the pages underneath it.)
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tags.html :mkdirp yes
+{% extends "arcology/app.html" %}
+#+end_src
+
+The tab title is assembled from the page and site title:
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tags.html
+{% block title %}The Arcology Project - Tag List{% endblock %}
+#+end_src
+
+If the site has any feeds, they're injected in to the =<head>= along with any particular web-crawler rules.
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tags.html
+{% block extra_head %}
+  {% for feed in feeds %}
+    <link rel="alternate" type="application/atom+xml" href="{{ feed.url }}" title="{{ feed.title }}" />
+  {% endfor %}
+{% endblock %}
+#+end_src
+
+The main =content= block contains the list of tags. It uses =HTMX= to make a dynamically loading list of pages with headings containing the Tag:
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tags.html
+{% block content %}
+<section>
+  <ul>
+    {% for tag in tags %}
+      <li>{{ tag.tag }}&nbsp;
+        (<a class="page_count"
+            href="/tags/{{ tag.tag }}"
+            hx-get="/tags/{{ tag.tag }}"
+            hx-swap="outerHTML"
+            hx-target="#{{tag.tag}}-pages">
+            <b>{{tag.total}}</b> Hits
+        </a>)
+        {# {% include tag-pages.html %} #}
+        <ul id="{{tag.tag}}-pages">
+        </ul>
+      </li>
+    {% endfor %}
+  </ul>
+</section>
+{% endblock %}
+#+end_src 
+
+* Individual Tag Pages (and list partial)
+
+This renders a partial depending on whether or not it's called by the HTMX declarations above.
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+def tag_page(request, tag: str):
+    site = Site.from_request(request)
+    pages = Tag.pages_by_name(tag)
+
+    if request.htmx:
+        base_template = "sitemap/tag_partial.html"
+    else:
+        base_template = "arcology/app.html"
+
+    return render(request, "sitemap/tag.html", dict(
+        base_template=base_template,
+
+        tag=tag,
+        pages=pages,
+        site=site,
+    ))
+#+end_src
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tag.html
+{% extends base_template %}
+
+{% block h1 %}<h1>Pages tagged with {{tag}} in the Arcology<h1>{% endblock %}
+{% block title %}Pages tagged with {{tag}} in the Arcology{% endblock %}
+
+{% block content %}
+<h1>Pages tagged with {{tag}}</h1>
+
+{{ list() }}
+{% endblock %}
+
+{% block list %}
+hi
+<ul id="{{tag}}-pages">
+{% for page in pages %}
+<li><a href="{{page.to_url}}">{{page.title}}</a></li>
+{% endfor %}
+</ul>
+{% endblock %}
+#+end_src
+
+That =request.htmx= branch will make sure we only render the list if the HTMX partial is called by swapping the base HTML template to render only the list.
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tag_partial.html
+{% block list %}{% endblock %}
+#+end_src
+
+
+* Sitemap JSON
+
+That this relies on [[id:20240204T234334.762591][Data Models for Sites, Web Features, and Feeds]] *and* [[id:arcology/django/roam][Arcology Roam Models]] tells me it may be needs to be in a different module, idk... the structure of these projects really does need to be worked on.
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+def sitemap_data(request):
+    pass
+#+end_src
+
+* Sitemap HTML Page
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+def sitemap(request):
+    pass
+#+end_src
+
+* Sitemap Frontend JS using Sigma JS

sitemap/templates/sitemap/tag.html

@@ -0,0 +1,21 @@
+{# [[file:../../../sitemap.org::*Individual Tag Pages (and list partial)][Individual Tag Pages (and list partial):2]] #}
+{% extends base_template %}
+
+{% block h1 %}<h1>Pages tagged with {{tag}} in the Arcology<h1>{% endblock %}
+{% block title %}Pages tagged with {{tag}} in the Arcology{% endblock %}
+
+{% block content %}
+<h1>Pages tagged with {{tag}}</h1>
+
+{{ list() }}
+{% endblock %}
+
+{% block list %}
+hi
+<ul id="{{tag}}-pages">
+{% for page in pages %}
+<li><a href="{{page.to_url}}">{{page.title}}</a></li>
+{% endfor %}
+</ul>
+{% endblock %}
+{# Individual Tag Pages (and list partial):2 ends here #}

sitemap/templates/sitemap/tag_partial.html

@@ -0,0 +1,3 @@
+{# [[file:../../../sitemap.org::*Individual Tag Pages (and list partial)][Individual Tag Pages (and list partial):3]] #}
+{% block list %}{% endblock %}
+{# Individual Tag Pages (and list partial):3 ends here #}

sitemap/templates/sitemap/tags.html

@@ -0,0 +1,38 @@
+{# [[file:../../../sitemap.org::*Tag aggregation pages][Tag aggregation pages:2]] #}
+{% extends "arcology/app.html" %}
+{# Tag aggregation pages:2 ends here #}
+
+{# [[file:../../../sitemap.org::*Tag aggregation pages][Tag aggregation pages:3]] #}
+{% block title %}The Arcology Project - Tag List{% endblock %}
+{# Tag aggregation pages:3 ends here #}
+
+{# [[file:../../../sitemap.org::*Tag aggregation pages][Tag aggregation pages:4]] #}
+{% block extra_head %}
+  {% for feed in feeds %}
+    <link rel="alternate" type="application/atom+xml" href="{{ feed.url }}" title="{{ feed.title }}" />
+  {% endfor %}
+{% endblock %}
+{# Tag aggregation pages:4 ends here #}
+
+{# [[file:../../../sitemap.org::*Tag aggregation pages][Tag aggregation pages:5]] #}
+{% block content %}
+<section>
+  <ul>
+    {% for tag in tags %}
+      <li>{{ tag.tag }}&nbsp;
+        (<a class="page_count"
+            href="/tags/{{ tag.tag }}"
+            hx-get="/tags/{{ tag.tag }}"
+            hx-swap="outerHTML"
+            hx-target="#{{tag.tag}}-pages">
+            <b>{{tag.total}}</b> Hits
+        </a>)
+        {# {% include tag-pages.html %} #}
+        <ul id="{{tag.tag}}-pages">
+        </ul>
+      </li>
+    {% endfor %}
+  </ul>
+</section>
+{% endblock %}
+{# Tag aggregation pages:5 ends here #}

sitemap/urls.py

@@ -0,0 +1,14 @@
+# [[file:../sitemap.org::*Django View Setup][Django View Setup:2]]
+from django.contrib import admin
+from django.urls import path, re_path, include
+
+from sitemap import views
+
+urlpatterns = [
+    path("tags", views.tags_index, name="sitemap"),
+    path("tags/", views.tags_index, name="sitemap"),
+    path("tags/<slug:tag>", views.tag_page, name="sitemap"),
+    path("sitemap", views.sitemap, name="sitemap"),
+    path("sitemap.json", views.sitemap_data, name="sitemap"),
+]
+# Django View Setup:2 ends here

sitemap/views.py

@@ -0,0 +1,59 @@
+# [[file:../sitemap.org::*Django View Setup][Django View Setup:1]]
+import logging
+from django.http import HttpResponse, HttpResponseNotFound
+from django.shortcuts import render, get_object_or_404
+from django.db.models import Count
+
+from arcology.models import Page, Feed, Site
+from roam.models import Link, Tag
+
+# from prometheus_client import Counter, Histogram
+
+logger = logging.getLogger(__name__)
+# Django View Setup:1 ends here
+
+# [[file:../sitemap.org::*Tag aggregation pages][Tag aggregation pages:1]]
+from django.shortcuts import render, get_object_or_404
+
+def tags_index(request):
+    site = Site.from_request(request)
+    tags = Tag.objects.all().values('tag').annotate(total=Count('tag')).order_by('-total')
+    # tags = [
+    #     {**tag, 'pages': Tag.pages_by_name(tag['tag'])}
+    #     for tag in tags
+    #    ] 
+    return render(request, "sitemap/tags.html", dict(
+        tags=tags,
+        site=site,
+        feeds=site.feed_set.all()
+    ))
+# Tag aggregation pages:1 ends here
+
+# [[file:../sitemap.org::*Individual Tag Pages (and list partial)][Individual Tag Pages (and list partial):1]]
+def tag_page(request, tag: str):
+    site = Site.from_request(request)
+    pages = Tag.pages_by_name(tag)
+
+    if request.htmx:
+        base_template = "sitemap/tag_partial.html"
+    else:
+        base_template = "arcology/app.html"
+
+    return render(request, "sitemap/tag.html", dict(
+        base_template=base_template,
+
+        tag=tag,
+        pages=pages,
+        site=site,
+    ))
+# Individual Tag Pages (and list partial):1 ends here
+
+# [[file:../sitemap.org::*Sitemap JSON][Sitemap JSON:1]]
+def sitemap_data(request):
+    pass
+# Sitemap JSON:1 ends here
+
+# [[file:../sitemap.org::*Sitemap HTML Page][Sitemap HTML Page:1]]
+def sitemap(request):
+    pass
+# Sitemap HTML Page:1 ends here

syncthonk/management/commands/watchsync.py

@@ -28,7 +28,7 @@ class Command(BaseCommand):
     def handle(self, *args, **kwargs):
         self.key = settings.SYNCTHING_KEY
         assert self.key != None
-        if "folder_id" in kwargs.keys():
+        if kwargs.get("folder_id") is not None and len(kwargs.get("folder_id")) > 0:
             self.folder_id = kwargs['folder_id']
             self.expanded_path = self.get_path_for_folder_id(self.folder_id)
             logger.info(f"fetched folder ID {self.folder_id} AKA {self.expanded_path} from syncthing API")