fix bug i introduced to watchsync

CSS nips and tucks
add sitemap module and expose cross-module interfaces for them
2024-02-26 16:20:46 -08:00 · 2024-02-26 16:20:40 -08:00 · 2024-02-26 16:20:40 -08:00
24 changed files with 4392 additions and 97 deletions
--- a/README.org
+++ b/README.org
@ -46,6 +46,7 @@ This project is a living document. It's [[id:cce/literate_programming][Literate
 - [[id:arcology/django/config][Arcology Project configuration]] describes and implements the knobs the user can twiddle to configure the Arcology's operation
 - [[id:arcology/django/interfaces][Interfacing with the Arcology]] provides a set of management commands and a Syncthing client which will automatically ingest new files in to the database. This document describes all the tools a prospective user of the CCE will need to use for surface-level features.
 - [[id:20240213T124300.774781][Deploying the Arcology]] lays out the basics of operating a Wobserver, but also provides a methodology for bootstrapping the Arcology on systems not already running NixOS.
+- [[id:20240226T132507.817450][The Arcology's Site Maps and Discovery Mechanisms]] show you how you can discover content published in an Arcology.
 - The [[id:arcology/django/scaffolding][Arcology Project Scaffolding]] contains the files necessary to run the project, defines the nix environment, and the python environment, and the base Django apparatus.
 - The [[id:arcology/django/roam][Arcology Roam Models]] are the base metadata of the [[id:cce/org-roam][org-roam]] documents
 - [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]] are the web publishing platform and the data models it requires to be operated
@ -76,7 +77,11 @@ This project is a living document. It's [[id:cce/literate_programming][Literate
 - State "DONE"       from "NEXT"       [2024-02-17 Sat 21:27]
 :END:

-** NEXT Sitemap
+** NEXT [[id:20240226T132507.817450][Sitemap]]
+:LOGBOOK:
+CLOCK: [2024-02-26 Mon 15:22]--[2024-02-26 Mon 16:16] =>  0:54
+CLOCK: [2024-02-26 Mon 13:24]--[2024-02-26 Mon 14:59] =>  1:35
+:END:
 ** NEXT Move [[id:arroyo/django/generators][The Arroyo Generators]] and perhaps [[id:arcology/django/roam][The Arcology Roam Models]] in to [[id:arroyo/arroyo][Arroyo Systems Management]]
 ** NEXT [[id:20231113T195508.942155][Rebuild of The Complete Computer]]
 ** NEXT Consider having a second sqlite3 with server-written state like hit counts and fedi urls and whatnot that i am nervous to store in a DB right now.
--- a/arcology.org
+++ b/arcology.org
@ -543,10 +543,10 @@ urlpatterns = [
    path("", views.index),
    path("robots.txt", views.robots, name="robots_txt"),
    path("404", views.unpublished, name="page_not_found"),
-    path("sitemap", views.sitemap, name="sitemap"),
    path("sites.css", views.site_css, name="site-css"),
    path("feeds.json", views.feed_list, name="feed-list"),
    path("", include("django_prometheus.urls")),
+    path("", include("sitemap.urls")),
    # ensure these ones are last because they're greedy!
    re_path("(?P<key>[0-9a-zA-Z/_\-]+\.xml)", views.feed, name="feed"),
    re_path("(?P<key>[0-9a-zA-Z/_\-]+)", views.org_page, name="org-page"),
@ -811,7 +811,11 @@ Here are some [[https://medium.com/@massimo.cassandro/flexbox-separators-b284d6d
 .content > *:first-child {
    order: -1;
 }
+#+end_src

+And some simple image wrangling:
+
+#+begin_src css :tangle arcology/static/arcology/css/app.css :mkdirp yes
 .content img {
  display: block;
  width: 80%;
@ -819,7 +823,21 @@ Here are some [[https://medium.com/@massimo.cassandro/flexbox-separators-b284d6d
 }
 #+end_src

-And some simple image wrangling:
+These rules annotate task headings by inserting an icon before them.
+
+#+begin_src css :tangle arcology/static/arcology/css/app.css :mkdirp yes
+.task.task-DONE::before {
+  content: '☑️ ';
+}
+
+.task.task-NEXT::before {
+  content: '🆕 ';
+}
+
+.task.task-INPROGRESS::before {
+  content: '⏳ ';
+}
+#+end_src

 ** INPROGRESS Atom Feed Handler
 :PROPERTIES:
@ -923,16 +941,6 @@ def get_item(dictionary, key):

 *** NEXT [#A] see if the IDs are consistent with the old generator

-** NEXT sitemap
-
-this is maybe its own django app...
-
-#+begin_src python :tangle arcology/views.py
-def sitemap(request):
-    # query links etc to create a JSON doc for SigmaJS
-    return HttpResponse(b"sitemap")
-#+end_src
-
 ** NEXT unpublished/not found endpoint

 There are plenty of links inside the Arcology which aren't meant to be clicked. =roam:= stub links will of course 
@ -1032,88 +1040,90 @@ for now it's largely lifted from [[id:arcology/fastapi/base.html.j2][Base HTML T
 The base template provides some basic information and loads the CSS sheets necessary to make things look nice, along with some page and author metadata. It provides a template block =extra_head= so that child templates can shove more =<head>= elements in here.

 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
-    {% load static %}
-    <link rel="stylesheet" href="{% static 'arcology/css/app.css' %}"/>
-    <link rel="stylesheet" href="{% static 'arcology/css/vulf.css' %}"/>
-    <link rel="stylesheet" href="{% static 'arcology/css/default-colors.css' %}"/>
-    <link rel="stylesheet" href="{% url 'site-css' %}"/>
-    {% if site and site.css_file %}
-      <link rel="stylesheet" href="{% static site.css_file %}"/>
-    {% endif %}
-    <meta name="author" content="Ryan Rix"/>
-    <meta name="generator" content="Arcology Site Engine https://engine.arcology.garden/"/>
-    <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>{% block title %}{{head_title | default:"The Arcology Project" }}{% endblock %}</title>
-    {% block extra_head %}{% endblock %}
-  </head>
+  {% load static %}
+  {% load django_htmx %}
+  <link rel="stylesheet" href="{% static 'arcology/css/app.css' %}"/>
+  <link rel="stylesheet" href="{% static 'arcology/css/vulf.css' %}"/>
+  <link rel="stylesheet" href="{% static 'arcology/css/default-colors.css' %}"/>
+  <link rel="stylesheet" href="{% url 'site-css' %}"/>
+  {% if site and site.css_file %}
+  <link rel="stylesheet" href="{% static site.css_file %}"/>
+  {% endif %}
+  <meta name="author" content="Ryan Rix"/>
+  <meta name="generator" content="Arcology Site Engine https://engine.arcology.garden/"/>
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>{% block title %}{{head_title | default:"The Arcology Project" }}{% endblock %}</title>
+  <script src="{% static 'sitemap/js/htmx.js' %}" defer></script>
+  {% block extra_head %}{% endblock %}
+</head>
 #+end_src

 The body consists of a header which has the site and page title (which can be overridden for example in the =index= handler to only show the site title) and links to the other sites. These should be loaded from the DB eventually.

 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
-  <body>
-    <header>
-      <div class="header-content">
-        {% block h1 %}
-          <h1><a href='/'>{{ site.title }}</a></h1>
-          <h2>{{ page.title }}</h2>
-        {% endblock %}
-        <div>
-          &bull; <a class="internal" href="https://thelionsrear.com">Life</a>
-          &bull; <a class="internal" href="https://arcology.garden">Tech</a>
-          &bull; <a class="internal" href="https://cce.whatthefuck.computer">Emacs</a>
-          &bull; <a class="internal" href="https://engine.arcology.garden">Arcology</a>
-          &bull;
-        </div>
+<body>
+  <header>
+    <div class="header-content">
+      {% block h1 %}
+      <h1><a href='/'>{{ site.title }}</a></h1>
+      <h2>{{ page.title }}</h2>
+      {% endblock %}
+      <div>
+        &bull; <a class="internal" href="https://thelionsrear.com">Life</a>
+        &bull; <a class="internal" href="https://arcology.garden">Tech</a>
+        &bull; <a class="internal" href="https://cce.whatthefuck.computer">Emacs</a>
+        &bull; <a class="internal" href="https://engine.arcology.garden">Arcology</a>
+        &bull;
      </div>
-    </header>
+    </div>
+  </header>
 #+end_src

 The =content= block is used in child templates to hide a =<main>=; the =content= div *should* be a main element instead but [[id:20231023T115950.248543][The arroyo_rs Native Org Parser]] wants to output a =<main>= and i'm not going to stop it, so the div is there to make the body's flexbox layout work.

 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
-    <div class="content">
-      {% block content %}{% endblock %}
-    </div>
+<div class="content">
+  {% block content %}{% endblock %}
+</div>
 #+end_src

 A footer contains the oh-so-important copyright notice and a limited privacy policy which I should update before I ship this, along with links to the sitemap and to [[https://fediring.net][my fediring neighbors]].

 #+begin_src jinja2 :tangle arcology/templates/arcology/app.html :mkdirp yes
-    <footer>
-      <hr/>
-      &copy; 02024 <a href="https://arcology.garden/people/rrix">Ryan Rix</a> &lt;<a href="mailto:site@whatthefuck.computer">site@whatthefuck.computer</a>&gt;
+<footer>
+  <hr/>
+  &copy; 02024 <a href="https://arcology.garden/people/rrix">Ryan Rix</a> &lt;<a href="mailto:site@whatthefuck.computer">site@whatthefuck.computer</a>&gt;

-      <br/>
+  <br/>

-      <p>
-        Care has been taken to publish accurate information to
-        long-lived URLs, but context and content as well as URLs may
-        change without notice.
-      </p>
+  <p>
+    Care has been taken to publish accurate information to
+    long-lived URLs, but context and content as well as URLs may
+    change without notice.
+  </p>

-      <p>
-        This site collects no personal information from visitors, nor
-        stores any identifying tokens. If you or your personal
-        information ended up in public notes please email me for
-        correction or removal. A single bit cookie may be stored on
-        your device if you choose to change appearance settings below.
-      </p>
+  <p>
+    This site collects no personal information from visitors, nor
+    stores any identifying tokens. If you or your personal
+    information ended up in public notes please email me for
+    correction or removal. A single bit cookie may be stored on
+    your device if you choose to change appearance settings below.
+  </p>

-      <p>
-        Email me with questions, comments, insights, kind criticism.
-        blow horn, good luck.
-      </p>
+  <p>
+    Email me with questions, comments, insights, kind criticism.
+    blow horn, good luck.
+  </p>

-      <p>
-        <a href="/sitemap/">View the Site Map</a>
-      </p>
+  <p>
+    <a href="/sitemap/">View the Site Map</a>
+  </p>

-      <p>
-        <a href="https://fediring.net/previous?host=arcology.garden">&larr;</a>
-        <a href="https://fediring.net/">Fediring</a>
-        <a href="https://fediring.net/next?host=arcology.garden">&rarr;</a>
-      </p>
+  <p>
+    <a href="https://fediring.net/previous?host=arcology.garden">&larr;</a>
+    <a href="https://fediring.net/">Fediring</a>
+    <a href="https://fediring.net/next?host=arcology.garden">&rarr;</a>
+  </p>
 #+end_src

 The FastaAPI site had a "boredom mode" which would disable fonts and colors because some nerds were mean to me. This one will not have that until some nerds are mean to me.
@ -1271,7 +1281,7 @@ def site_css(request):
    stanzas.append(f'''
      a[href*="/404"] {{
        color: var(--alert);
-        text-decoration: line-through;
+ /* text-decoration: line-through; */
      }}
      a[href*="/404"]::after {{
        content: " ⚠";
--- a/arcology/settings/init.py
+++ b/arcology/settings/init.py
@ -33,7 +33,9 @@ INSTALLED_APPS = [
    "arcology", # [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]]
    "generators", # [[id:arroyo/django/generators][The Arroyo Generators]]
    "syncthonk", # [[id:20231218T183551.765340][Arcology watchsync Command]]
+    "sitemap", # [[id:20240226T132507.817450][The Arcology's Site Maps and Discovery Mechanisms]]
    
+    "django_htmx",
    "django_prometheus",
    "django.contrib.admin",
    "django.contrib.auth",
@ -81,6 +83,7 @@ LOGGING = {
 # [[file:../../configuration.org::*Look don't worry about the rest of these][Look don't worry about the rest of these:1]]
 MIDDLEWARE = [
    "django_prometheus.middleware.PrometheusBeforeMiddleware",
+    "django_htmx.middleware.HtmxMiddleware",
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.common.CommonMiddleware",
@ -167,6 +170,7 @@ STORAGES = {
 }
 STATICFILES_DIRS = [
    BASE_DIR / "arcology/static",
+    BASE_DIR / "sitemap/static",
 ]

 STATIC_ROOT = os.getenv("ARCOLOGY_STATIC_ROOT", "/var/www/arcology")
--- a/arcology/static/arcology/css/app.css
+++ b/arcology/static/arcology/css/app.css
@ -47,13 +47,29 @@ section.sidebar > div.backlinks {
 .content > *:first-child {
    order: -1;
 }
+/* Org Page-specific CSS Stylings:3 ends here */

+/* [[file:../../../../arcology.org::*Org Page-specific CSS Stylings][Org Page-specific CSS Stylings:4]] */
 .content img {
  display: block;
  width: 80%;
  margin: 0 auto;
 }
-/* Org Page-specific CSS Stylings:3 ends here */
+/* Org Page-specific CSS Stylings:4 ends here */
+
+/* [[file:../../../../arcology.org::*Org Page-specific CSS Stylings][Org Page-specific CSS Stylings:5]] */
+.task.task-DONE::before {
+  content: '☑️ ';
+}
+
+.task.task-NEXT::before {
+  content: '🆕 ';
+}
+
+.task.task-INPROGRESS::before {
+  content: '⏳ ';
+}
+/* Org Page-specific CSS Stylings:5 ends here */

 /* [[file:../../../../arcology.org::*CSS][CSS:1]] */
 body {
--- a/arcology/templates/arcology/app.html
+++ b/arcology/templates/arcology/app.html
@ -6,17 +6,19 @@

 {# [[file:../../../arcology.org::*Arcology Site Templates][Arcology Site Templates:2]] #}
 {% load static %}
+  {% load django_htmx %}
  <link rel="stylesheet" href="{% static 'arcology/css/app.css' %}"/>
  <link rel="stylesheet" href="{% static 'arcology/css/vulf.css' %}"/>
  <link rel="stylesheet" href="{% static 'arcology/css/default-colors.css' %}"/>
  <link rel="stylesheet" href="{% url 'site-css' %}"/>
  {% if site and site.css_file %}
-    <link rel="stylesheet" href="{% static site.css_file %}"/>
+  <link rel="stylesheet" href="{% static site.css_file %}"/>
  {% endif %}
  <meta name="author" content="Ryan Rix"/>
  <meta name="generator" content="Arcology Site Engine https://engine.arcology.garden/"/>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>{% block title %}{{head_title | default:"The Arcology Project" }}{% endblock %}</title>
+  <script src="{% static 'sitemap/js/htmx.js' %}" defer></script>
  {% block extra_head %}{% endblock %}
 </head>
 {# Arcology Site Templates:2 ends here #}
@ -26,8 +28,8 @@
  <header>
    <div class="header-content">
      {% block h1 %}
-        <h1><a href='/'>{{ site.title }}</a></h1>
-        <h2>{{ page.title }}</h2>
+      <h1><a href='/'>{{ site.title }}</a></h1>
+      <h2>{{ page.title }}</h2>
      {% endblock %}
      <div>
        &bull; <a class="internal" href="https://thelionsrear.com">Life</a>
--- a/arcology/urls.py
+++ b/arcology/urls.py
@ -9,10 +9,10 @@ urlpatterns = [
    path("", views.index),
    path("robots.txt", views.robots, name="robots_txt"),
    path("404", views.unpublished, name="page_not_found"),
-    path("sitemap", views.sitemap, name="sitemap"),
    path("sites.css", views.site_css, name="site-css"),
    path("feeds.json", views.feed_list, name="feed-list"),
    path("", include("django_prometheus.urls")),
+    path("", include("sitemap.urls")),
    # ensure these ones are last because they're greedy!
    re_path("(?P<key>[0-9a-zA-Z/_\-]+\.xml)", views.feed, name="feed"),
    re_path("(?P<key>[0-9a-zA-Z/_\-]+)", views.org_page, name="org-page"),
--- a/arcology/views.py
+++ b/arcology/views.py
@ -134,12 +134,6 @@ def get_item(dictionary, key):
    return dictionary.get(key)
 # move this function to somewhere else more reasonable:1 ends here

-# [[file:../arcology.org::*sitemap][sitemap:1]]
-def sitemap(request):
-    # query links etc to create a JSON doc for SigmaJS
-    return HttpResponse(b"sitemap")
-# sitemap:1 ends here
-
 # [[file:../arcology.org::*unpublished/not found endpoint][unpublished/not found endpoint:1]]
 def unpublished(request):
    key = request.GET.get("key")
@ -210,7 +204,7 @@ def site_css(request):
    stanzas.append(f'''
      a[href*="/404"] {{
        color: var(--alert);
-        text-decoration: line-through;
+        /* text-decoration: line-through; */
      }}
      a[href*="/404"]::after {{
        content: " ⚠";
--- a/configuration.org
+++ b/configuration.org
@ -241,15 +241,17 @@ ALLOWED_HOSTS = "<<get_allowed_hosts()>>".split(',')
 basically, each org file in this repository, and maybe one or two of your own, are all interoperable and build on top of the base data models of the org docs.

 #+name: arcology_apps
-| "roam"       | [[id:arcology/django/roam][Arcology Roam Models]]                      |
-| "arcology"   | [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]] |
-| "generators" | [[id:arroyo/django/generators][The Arroyo Generators]]                     |
-| "syncthonk"  | [[id:20231218T183551.765340][Arcology watchsync Command]]                |
+| "roam"       | [[id:arcology/django/roam][Arcology Roam Models]]                              |
+| "arcology"   | [[id:arcology/django/arcology-models][The Arcology's Data Models and Web Server]]         |
+| "generators" | [[id:arroyo/django/generators][The Arroyo Generators]]                             |
+| "syncthonk"  | [[id:20231218T183551.765340][Arcology watchsync Command]]                        |
+| "sitemap"    | [[id:20240226T132507.817450][The Arcology's Site Maps and Discovery Mechanisms]] |

 #+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
 # Application definition
 INSTALLED_APPS = [
    <<gen-config-list(arcology_apps)>>
+    "django_htmx",
    "django_prometheus",
    "django.contrib.admin",
    "django.contrib.auth",
@ -442,6 +444,7 @@ IGNORED_ROAM_TAGS = [
 #+BEGIN_SRC python :tangle arcology/settings/__init__.py :noweb yes
 MIDDLEWARE = [
    "django_prometheus.middleware.PrometheusBeforeMiddleware",
+    "django_htmx.middleware.HtmxMiddleware",
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.common.CommonMiddleware",
@ -528,6 +531,7 @@ STORAGES = {
 }
 STATICFILES_DIRS = [
    BASE_DIR / "arcology/static",
+    BASE_DIR / "sitemap/static",
 ]

 STATIC_ROOT = os.getenv("ARCOLOGY_STATIC_ROOT", "/var/www/arcology")
--- a/default.nix
+++ b/default.nix
@ -23,6 +23,7 @@ python3.pkgs.buildPythonPackage rec {
    click
    django_4
    django-prometheus
+    django-htmx
    (django-stubs-ext.override { django = django_4; })
    (django-stubs.override { django = django_4; })
    gunicorn
--- a/interfaces.org
+++ b/interfaces.org
@ -306,7 +306,7 @@ class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        self.key = settings.SYNCTHING_KEY
        assert self.key != None
-        if "folder_id" in kwargs.keys():
+        if kwargs.get("folder_id") is not None and len(kwargs.get("folder_id")) > 0:
            self.folder_id = kwargs['folder_id']
            self.expanded_path = self.get_path_for_folder_id(self.folder_id)
            logger.info(f"fetched folder ID {self.folder_id} AKA {self.expanded_path} from syncthing API")
--- a/pyproject.toml
+++ b/pyproject.toml
@ -5,7 +5,7 @@ version = "0.0.1"
 description = "org-mode metadata query engine, publishing platform, and computer metaprogrammer"
 # license = "Hey Smell This"
 readme = "README.md"
-dependencies = ["click ~=8.1", "django ~= 4.2", "django-stub", "polling", "django-prometheus", "arroyo", "arrow ~= 1.3.0", "gunicorn ~= 21.0"]
+dependencies = ["click ~=8.1", "django ~= 4.2", "django-stub", "polling", "django-prometheus", "arroyo", "arrow ~= 1.3.0", "gunicorn ~= 21.0", "htmx ~= 1.17" ]
 requires-python = ">=3.10"
 authors = [
    { name = "Ryan Rix", email = "code@whatthefuck.computer" }
@ -25,6 +25,10 @@ arcology = [
  'static/arcology/css/*',
  'static/arcology/fonts/*',
 ]
+sitemap = [
+  'static/sitemap/js/*',
+  'templates/sitemap/*',
+]

 [tool.setuptools.packages.find]
 where = ["."]
--- a/roam.org
+++ b/roam.org
@ -96,7 +96,7 @@ class RoamFileTest(TestCase):
            .rstrip()
        )
 #+end_src
- 
+
 test =create_from_arroyo=, parse this document and see if we can get a =File= out of it lul

 #+begin_src python :tangle roam/tests.py
@ -366,7 +366,7 @@ class RoamHeadingPropertyTest(TestCase):
        self.native = parse_file(str(settings.BASE_DIR / "./README.org"))
        self.expected_path = str(settings.BASE_DIR / "./README.org")
 #+end_src
- 
+
 - test =create_from_arroyo=, parse this doc, create file and headings, and properties, validate properties are populated properly
 - including top-level file-properties (this will fail right now, i think)

@ -379,7 +379,7 @@ class RoamHeadingPropertyTest(TestCase):

 this will raise because level 0 file properties are not persisted, I need to fix it in [[id:20231023T115950.248543][arroyo_rs]]. It's not included in the test, but I'd like to be able to once I fix arroyo_rs.

-#+begin_src python 
+#+begin_src python
        # self.assertEquals(len(props), 2)
 #+end_src

@ -419,6 +419,18 @@ class Tag(EMOM('tag'), models.Model):
    )
    tag = models.CharField(max_length=256)

+    def pages(self) -> List['Page']:
+        return self.__class__.pages_by_name(self.tag)
+
+    @classmethod
+    def pages_by_name(cls, tag_name: str) -> List['Page']:
+        import arcology.models
+        return set([
+            arcology.models.Page.objects.get(file=tag_obj.heading.path_id)
+            for tag_obj in cls.objects.filter(tag=tag_name).distinct()
+        ])
+
+
    @classmethod
    def create_from_arroyo(cls, doc: native.Document) -> List[Tag]:
        return [
--- a/roam/models.py
+++ b/roam/models.py
@ -157,6 +157,18 @@ class Tag(EMOM('tag'), models.Model):
    )
    tag = models.CharField(max_length=256)

+    def pages(self) -> List['Page']:
+        return self.__class__.pages_by_name(self.tag)
+
+    @classmethod
+    def pages_by_name(cls, tag_name: str) -> List['Page']:
+        import arcology.models
+        return set([
+            arcology.models.Page.objects.get(file=tag_obj.heading.path_id)
+            for tag_obj in cls.objects.filter(tag=tag_name).distinct()
+        ])
+
+
    @classmethod
    def create_from_arroyo(cls, doc: native.Document) -> List[Tag]:
        return [
--- a/scaffolding.org
+++ b/scaffolding.org
@ -17,7 +17,7 @@ version = "0.0.1"
 description = "org-mode metadata query engine, publishing platform, and computer metaprogrammer"
 # license = "Hey Smell This"
 readme = "README.md"
-dependencies = ["click ~=8.1", "django ~= 4.2", "django-stub", "polling", "django-prometheus", "arroyo", "arrow ~= 1.3.0", "gunicorn ~= 21.0"]
+dependencies = ["click ~=8.1", "django ~= 4.2", "django-stub", "polling", "django-prometheus", "arroyo", "arrow ~= 1.3.0", "gunicorn ~= 21.0", "htmx ~= 1.17" ]
 requires-python = ">=3.10"
 authors = [
    { name = "Ryan Rix", email = "code@whatthefuck.computer" }
@ -37,6 +37,10 @@ arcology = [
  'static/arcology/css/*',
  'static/arcology/fonts/*',
 ]
+sitemap = [
+  'static/sitemap/js/*',
+  'templates/sitemap/*',
+]

 [tool.setuptools.packages.find]
 where = ["."]
@ -75,6 +79,7 @@ python3.pkgs.buildPythonPackage rec {
    click
    django_4
    django-prometheus
+    django-htmx
    (django-stubs-ext.override { django = django_4; })
    (django-stubs.override { django = django_4; })
    gunicorn
@ -113,6 +118,7 @@ let
    arroyo_rs
    django_4
    django-prometheus
+    django-htmx
    (django-stubs-ext.override { django = django_4; })
    (django-stubs.override { django = django_4; })
    gunicorn
--- a/shell.nix
+++ b/shell.nix
@ -14,6 +14,7 @@ let
    arroyo_rs
    django_4
    django-prometheus
+    django-htmx
    (django-stubs-ext.override { django = django_4; })
    (django-stubs.override { django = django_4; })
    gunicorn
--- a/sitemap.org
+++ b/sitemap.org
@ -0,0 +1,183 @@
+:PROPERTIES:
+:ID:       20240226T132507.817450
+:END:
+#+TITLE: The Arcology's Site Maps and Discovery Mechanisms
+
+#+ARCOLOGY_KEY: arcology/sitemaps
+#+ARCOLOGY_ALLOW_CRAWL: t
+
+These pages are perhaps the most "dynamic" or inteactive of the Arcology's features right now.
+
+First there is a set of views which renders an index of all the Pages' tags. Here we start to use [[roam:HTMX]] to render partials dynamically. Rather than load every tag's page listing, the =/tags/= endpoint allows you to drop-down tags with a button press to see posts with that tag.
+
+Then there is the Sitemap page which uses [[id:20220711T151820.326251][SigmaJS]] to render a connected topology of all the pages in the Arcology Project.
+
+* Django View Setup
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+import logging
+from django.http import HttpResponse, HttpResponseNotFound
+from django.shortcuts import render, get_object_or_404
+from django.db.models import Count
+
+from arcology.models import Page, Feed, Site
+from roam.models import Link, Tag
+
+# from prometheus_client import Counter, Histogram
+
+logger = logging.getLogger(__name__)
+#+end_src
+
+#+begin_src python :tangle sitemap/urls.py :mkdirp yes
+from django.contrib import admin
+from django.urls import path, re_path, include
+
+from sitemap import views
+
+urlpatterns = [
+    path("tags", views.tags_index, name="sitemap"),
+    path("tags/", views.tags_index, name="sitemap"),
+    path("tags/<slug:tag>", views.tag_page, name="sitemap"),
+    path("sitemap", views.sitemap, name="sitemap"),
+    path("sitemap.json", views.sitemap_data, name="sitemap"),
+]
+#+end_src
+
+* Tag aggregation pages
+:LOGBOOK:
+CLOCK: [2024-02-26 Mon 15:22]--[2024-02-26 Mon 15:22] =>  0:00
+:END:
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+from django.shortcuts import render, get_object_or_404
+
+def tags_index(request):
+    site = Site.from_request(request)
+    tags = Tag.objects.all().values('tag').annotate(total=Count('tag')).order_by('-total')
+    # tags = [
+    #     {**tag, 'pages': Tag.pages_by_name(tag['tag'])}
+    #     for tag in tags
+    #    ] 
+    return render(request, "sitemap/tags.html", dict(
+        tags=tags,
+        site=site,
+        feeds=site.feed_set.all()
+    ))
+#+end_src
+
+The =tags= template lists all the tags and lets you click on them to see the pages in one. .o(it probably should just show all the pages underneath it.)
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tags.html :mkdirp yes
+{% extends "arcology/app.html" %}
+#+end_src
+
+The tab title is assembled from the page and site title:
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tags.html
+{% block title %}The Arcology Project - Tag List{% endblock %}
+#+end_src
+
+If the site has any feeds, they're injected in to the =<head>= along with any particular web-crawler rules.
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tags.html
+{% block extra_head %}
+  {% for feed in feeds %}
+    <link rel="alternate" type="application/atom+xml" href="{{ feed.url }}" title="{{ feed.title }}" />
+  {% endfor %}
+{% endblock %}
+#+end_src
+
+The main =content= block contains the list of tags. It uses =HTMX= to make a dynamically loading list of pages with headings containing the Tag:
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tags.html
+{% block content %}
+<section>
+  <ul>
+    {% for tag in tags %}
+      <li>{{ tag.tag }}&nbsp;
+        (<a class="page_count"
+            href="/tags/{{ tag.tag }}"
+            hx-get="/tags/{{ tag.tag }}"
+            hx-swap="outerHTML"
+            hx-target="#{{tag.tag}}-pages">
+            <b>{{tag.total}}</b> Hits
+        </a>)
+        {# {% include tag-pages.html %} #}
+        <ul id="{{tag.tag}}-pages">
+        </ul>
+      </li>
+    {% endfor %}
+  </ul>
+</section>
+{% endblock %}
+#+end_src 
+
+* Individual Tag Pages (and list partial)
+
+This renders a partial depending on whether or not it's called by the HTMX declarations above.
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+def tag_page(request, tag: str):
+    site = Site.from_request(request)
+    pages = Tag.pages_by_name(tag)
+
+    if request.htmx:
+        base_template = "sitemap/tag_partial.html"
+    else:
+        base_template = "arcology/app.html"
+
+    return render(request, "sitemap/tag.html", dict(
+        base_template=base_template,
+
+        tag=tag,
+        pages=pages,
+        site=site,
+    ))
+#+end_src
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tag.html
+{% extends base_template %}
+
+{% block h1 %}<h1>Pages tagged with {{tag}} in the Arcology<h1>{% endblock %}
+{% block title %}Pages tagged with {{tag}} in the Arcology{% endblock %}
+
+{% block content %}
+<h1>Pages tagged with {{tag}}</h1>
+
+{{ list() }}
+{% endblock %}
+
+{% block list %}
+hi
+<ul id="{{tag}}-pages">
+{% for page in pages %}
+<li><a href="{{page.to_url}}">{{page.title}}</a></li>
+{% endfor %}
+</ul>
+{% endblock %}
+#+end_src
+
+That =request.htmx= branch will make sure we only render the list if the HTMX partial is called by swapping the base HTML template to render only the list.
+
+#+begin_src jinja2 :tangle sitemap/templates/sitemap/tag_partial.html
+{% block list %}{% endblock %}
+#+end_src
+
+
+* Sitemap JSON
+
+That this relies on [[id:20240204T234334.762591][Data Models for Sites, Web Features, and Feeds]] *and* [[id:arcology/django/roam][Arcology Roam Models]] tells me it may be needs to be in a different module, idk... the structure of these projects really does need to be worked on.
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+def sitemap_data(request):
+    pass
+#+end_src
+
+* Sitemap HTML Page
+
+#+begin_src python :tangle sitemap/views.py :mkdirp yes
+def sitemap(request):
+    pass
+#+end_src
+
+* Sitemap Frontend JS using Sigma JS
--- a/sitemap/static/sitemap/js/htmx.js
+++ b/sitemap/static/sitemap/js/htmx.js
--- a/sitemap/static/sitemap/js/htmx.min.js
+++ b/sitemap/static/sitemap/js/htmx.min.js
--- a/sitemap/templates/sitemap/tag.html
+++ b/sitemap/templates/sitemap/tag.html
@ -0,0 +1,21 @@
+{# [[file:../../../sitemap.org::*Individual Tag Pages (and list partial)][Individual Tag Pages (and list partial):2]] #}
+{% extends base_template %}
+
+{% block h1 %}<h1>Pages tagged with {{tag}} in the Arcology<h1>{% endblock %}
+{% block title %}Pages tagged with {{tag}} in the Arcology{% endblock %}
+
+{% block content %}
+<h1>Pages tagged with {{tag}}</h1>
+
+{{ list() }}
+{% endblock %}
+
+{% block list %}
+hi
+<ul id="{{tag}}-pages">
+{% for page in pages %}
+<li><a href="{{page.to_url}}">{{page.title}}</a></li>
+{% endfor %}
+</ul>
+{% endblock %}
+{# Individual Tag Pages (and list partial):2 ends here #}
--- a/sitemap/templates/sitemap/tag_partial.html
+++ b/sitemap/templates/sitemap/tag_partial.html
@ -0,0 +1,3 @@
+{# [[file:../../../sitemap.org::*Individual Tag Pages (and list partial)][Individual Tag Pages (and list partial):3]] #}
+{% block list %}{% endblock %}
+{# Individual Tag Pages (and list partial):3 ends here #}
--- a/sitemap/templates/sitemap/tags.html
+++ b/sitemap/templates/sitemap/tags.html
@ -0,0 +1,38 @@
+{# [[file:../../../sitemap.org::*Tag aggregation pages][Tag aggregation pages:2]] #}
+{% extends "arcology/app.html" %}
+{# Tag aggregation pages:2 ends here #}
+
+{# [[file:../../../sitemap.org::*Tag aggregation pages][Tag aggregation pages:3]] #}
+{% block title %}The Arcology Project - Tag List{% endblock %}
+{# Tag aggregation pages:3 ends here #}
+
+{# [[file:../../../sitemap.org::*Tag aggregation pages][Tag aggregation pages:4]] #}
+{% block extra_head %}
+  {% for feed in feeds %}
+    <link rel="alternate" type="application/atom+xml" href="{{ feed.url }}" title="{{ feed.title }}" />
+  {% endfor %}
+{% endblock %}
+{# Tag aggregation pages:4 ends here #}
+
+{# [[file:../../../sitemap.org::*Tag aggregation pages][Tag aggregation pages:5]] #}
+{% block content %}
+<section>
+  <ul>
+    {% for tag in tags %}
+      <li>{{ tag.tag }}&nbsp;
+        (<a class="page_count"
+            href="/tags/{{ tag.tag }}"
+            hx-get="/tags/{{ tag.tag }}"
+            hx-swap="outerHTML"
+            hx-target="#{{tag.tag}}-pages">
+            <b>{{tag.total}}</b> Hits
+        </a>)
+        {# {% include tag-pages.html %} #}
+        <ul id="{{tag.tag}}-pages">
+        </ul>
+      </li>
+    {% endfor %}
+  </ul>
+</section>
+{% endblock %}
+{# Tag aggregation pages:5 ends here #}
--- a/sitemap/urls.py
+++ b/sitemap/urls.py
@ -0,0 +1,14 @@
+# [[file:../sitemap.org::*Django View Setup][Django View Setup:2]]
+from django.contrib import admin
+from django.urls import path, re_path, include
+
+from sitemap import views
+
+urlpatterns = [
+    path("tags", views.tags_index, name="sitemap"),
+    path("tags/", views.tags_index, name="sitemap"),
+    path("tags/<slug:tag>", views.tag_page, name="sitemap"),
+    path("sitemap", views.sitemap, name="sitemap"),
+    path("sitemap.json", views.sitemap_data, name="sitemap"),
+]
+# Django View Setup:2 ends here
--- a/sitemap/views.py
+++ b/sitemap/views.py
@ -0,0 +1,59 @@
+# [[file:../sitemap.org::*Django View Setup][Django View Setup:1]]
+import logging
+from django.http import HttpResponse, HttpResponseNotFound
+from django.shortcuts import render, get_object_or_404
+from django.db.models import Count
+
+from arcology.models import Page, Feed, Site
+from roam.models import Link, Tag
+
+# from prometheus_client import Counter, Histogram
+
+logger = logging.getLogger(__name__)
+# Django View Setup:1 ends here
+
+# [[file:../sitemap.org::*Tag aggregation pages][Tag aggregation pages:1]]
+from django.shortcuts import render, get_object_or_404
+
+def tags_index(request):
+    site = Site.from_request(request)
+    tags = Tag.objects.all().values('tag').annotate(total=Count('tag')).order_by('-total')
+    # tags = [
+    #     {**tag, 'pages': Tag.pages_by_name(tag['tag'])}
+    #     for tag in tags
+    #    ] 
+    return render(request, "sitemap/tags.html", dict(
+        tags=tags,
+        site=site,
+        feeds=site.feed_set.all()
+    ))
+# Tag aggregation pages:1 ends here
+
+# [[file:../sitemap.org::*Individual Tag Pages (and list partial)][Individual Tag Pages (and list partial):1]]
+def tag_page(request, tag: str):
+    site = Site.from_request(request)
+    pages = Tag.pages_by_name(tag)
+
+    if request.htmx:
+        base_template = "sitemap/tag_partial.html"
+    else:
+        base_template = "arcology/app.html"
+
+    return render(request, "sitemap/tag.html", dict(
+        base_template=base_template,
+
+        tag=tag,
+        pages=pages,
+        site=site,
+    ))
+# Individual Tag Pages (and list partial):1 ends here
+
+# [[file:../sitemap.org::*Sitemap JSON][Sitemap JSON:1]]
+def sitemap_data(request):
+    pass
+# Sitemap JSON:1 ends here
+
+# [[file:../sitemap.org::*Sitemap HTML Page][Sitemap HTML Page:1]]
+def sitemap(request):
+    pass
+# Sitemap HTML Page:1 ends here
--- a/syncthonk/management/commands/watchsync.py
+++ b/syncthonk/management/commands/watchsync.py
@ -28,7 +28,7 @@ class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        self.key = settings.SYNCTHING_KEY
        assert self.key != None
-        if "folder_id" in kwargs.keys():
+        if kwargs.get("folder_id") is not None and len(kwargs.get("folder_id")) > 0:
            self.folder_id = kwargs['folder_id']
            self.expanded_path = self.get_path_for_folder_id(self.folder_id)
            logger.info(f"fetched folder ID {self.folder_id} AKA {self.expanded_path} from syncthing API")
Author	SHA1	Message	Date
Ryan Rix	8fbbd106f6	fix bug i introduced to watchsync	2024-02-26 16:20:46 -08:00
Ryan Rix	baa4ebe02a	CSS nips and tucks	2024-02-26 16:20:40 -08:00
Ryan Rix	9cdbca6c40	add sitemap module and expose cross-module interfaces for them	2024-02-26 16:20:40 -08:00