arcology-fastapi/arcology-sitemaps.org

:PROPERTIES:
:ID:       arcology/sitemaps
:ROAM_ALIASES: "Arcology Sitemap"
:END:
#+TITLE: Navigating the Arcology Site Graph with SigmaJS
#+filetags: :Project:Arcology:Development:
#+ARCOLOGY_KEY: arcology/sitemaps
#+ARCOLOGY_ALLOW_CRAWL: t

#+AUTO_TANGLE: t

* What is SigmaJS
:PROPERTIES:
:ID:       20220711T151820.326251
:ROAM_REFS: https://www.sigmajs.org/
:ROAM_ALIASES: SigmaJS
:END:

[[https://www.sigmajs.org/][SigmaJS]] is a JavaScript library for visualizing graphs. My [[id:cce/org-roam][org-roam]] knowledge base is a graph and [[id:arroyo/system-cache][arroyo-db]] contains the public pages and links. As nodes and edges, this makes a fine graph and a lot more "logical" than an XML sitemap or linear walk through the sites.

* Getting SigmaJS

Download and vendor the minified versions:

#+begin_src shell
export GRAPHOLOGY_VER=0.24.1
export SIGMAJS_VER=v2.3.1

curl --location -v -o arcology/static/sigmajs.min.js https://github.com/jacomyal/sigma.js/releases/download/$SIGMAJS_VER/sigma.min.js

curl --location -v -o arcology/static/graphology.min.js https://github.com/graphology/graphology/releases/download/$GRAPHOLOGY_VER/graphology.min.js

curl --location -v -o arcology/static/graphology-library.min.js https://github.com/graphology/graphology/releases/download/$GRAPHOLOGY_VER/graphology-library.min.js
#+end_src

#+results:

At this point the [[id:arcology/fastapi/page.html.j2][Arcology Page Template]] should be able to import these thanks to the [[id:arcology/fastapi][Arcology FastAPI]] router exposing that static directory. I could set up a typescript dev environment and webpack and all this sort of shit, but I don't really need it.... It's fine! [[roam:This Is Fine]]!

* Generating the Nodes and Edges bundle from [[id:arcology/arroyo-page][Arroyo Arcology]]

From [[id:arroyo/system-cache][arroyo-db]] load all [[id:arcology/arroyo/page][Arcology Page]] as Nodes shaped like:

#+begin_example json
{
  "id": "n0",
  "label": "A node",
  "x": 0,
  "y": 0,
  "size": 3
}
#+end_example

From [[id:arroyo/system-cache][arroyo-db]] load all [[id:arcology/arroyo/link][Arcology Links]] as Edges shaped like:

#+begin_example json
{
  "id": "e0",
  "source": "n0",
  "target": "n1"
}
#+end_example

compose together and return in a single dict for JSON rendering shaped like:

#+begin_example json
{
  "nodes": [
  <<all nodes>>
  ]
  "edges": [
  <<all edges>>
  ]
}
#+end_example

** Making SigmaJS Nodes
:PROPERTIES:
:ID:       20220712T101916.751167
:END:

Nodes are simple:

#+begin_src python :tangle arcology/sigma.py
import hashlib

from sqlmodel import Session, select

import arcology.arroyo as arroyo
from arcology.arroyo import Page

def make_loc_hash(page: Page, salt, max_q=700):
  key = page.get_file() + str(salt)
  hash = hashlib.sha224(key.encode("utf-8")).digest()
  return int.from_bytes(hash, byteorder="big") % max_q

def nodes(engine):
  # collect all pages
  with Session(engine) as s:
    pages = s.exec(select(Page)).all()

    nodes = [
      dict(
        key=page.get_key(),
        attributes=dict(
          label=page.get_title(),
          x=make_loc_hash(page, 1),
          y=make_loc_hash(page, 2),
          size=min((len(page.backlinks) + len(page.outlinks)) / 2, 10),
          color=page.get_site().node_color,
          href=page.get_arcology_key().to_url()
        )
      )
      for page in pages
    ]

    return nodes
#+end_src

** Making SigmaJS Edges
:PROPERTIES:
:ID:       20220712T101918.675312
:END:

Edges are too, we just have to make sure the IDs line up:

#+begin_src python :tangle arcology/sigma.py
from sqlmodel import Session, select

import arcology.arroyo as arroyo
from arcology.arroyo import Link, engine

from arcology.parse import parse_sexp

# HACK: https://stackoverflow.com/questions/1151658/python-hashable-dicts
class hashabledict(dict):
    def __hash__(self):
        return hash(tuple(sorted(self.items())))

def edges(engine):
  # collect all pages
  with Session(engine) as s:
    q_links = s.exec(select(Link)).all()

    links = set()
    for link in q_links:
      source = link.source_page.get_key()
      dest = link.dest_page.get_key()
      links.add(
        hashabledict(
          key=f"{source}-{dest}",
          source=source,
          target=dest,
        )
      )

    return list(links)
#+end_src

Compose these together, let the [[id:20220225T175638.482695][Router]] handle turning it in to JSON. I clobber this in to an LRU cache like I do in a bunch of other places with a key brute-forced to map to the SHA hashes of every file in the public wiki... Woof! Glad [[id:personal_software_can_be_shitty][Personal Software Can Be Shitty]]!

#+begin_src python :tangle arcology/sigma.py
import functools

@functools.lru_cache(maxsize=20)
def sigma_lru(cache_key):
  return dict(
    # this is a different import of the same engine...
    nodes=nodes(engine),
    edges=edges(engine)
  )

def sigma(engine):
  with Session(engine) as session:
    pages = session.exec(select(Page).order_by(Page.key))
    hashes =[page.hash for page in pages]

    cache_key = ''.join(
      hashes
    )
    return sigma_lru(cache_key)
#+end_src

* Sitemap HTML Template
:PROPERTIES:
:ID:       arcology/fastapi/sitemap.html.j2
:END:

This derives from the [[id:arcology/fastapi/base.html.j2][Base HTML Template]] for the sites. It's decomposed a little bit with [[id:09779ac0-4d5f-40db-a340-49595c717e03][noweb syntax]] so that the JavaScript elements can be worked on as JavaScript instead of as Jinja2 templates.

#+begin_src jinja2 :tangle arcology/templates/sitemap.html.j2 :mkdirp yes :noweb yes
{% extends "base.html.j2" %}

{% block h1 %}
  <h1><a href='/'>{{ site.title }}</a></h1>
  <h2>The Site Map</h2>
{% endblock %}

{% block head %}
  <title>Arcology System Site Map</title>
  <script src="/static/graphology.min.js"></script>
  <script src="/static/graphology-library.min.js"></script>
  <script src="/static/sigmajs.min.js"></script>
{% endblock %}

{% block body %}
  <main>
    <section class="body">
      <p>
        This is a network graph of all the pages published in my
        Arcology network. The color of the node corresponds to its
        site, and you can click on any of them to jump to that page or
        zoom in to see more of their labels. I will be adding more
        features to this sitemap like search and highlighting
        "relative" neighbors later on. Feel free to view
        <a href="https://engine.arcology.garden/sitemaps">the implementation</a>
        within the Arcology pages, as usual.
      </p>

      <div id="sitemap-container">
      </div>

    </section>
    <script type="application/javascript" src="/static/sitemap.js"></script>
  </main>
{% endblock %}
#+end_src

** JavaScript to set up and render the SigmaJS graph

The =SigmaJS= code is loaded in the =<head>=, and the graph is injected in to the =#sitemap-container= =<div>=.

#+begin_src javascript :noweb-ref sigmaInit :noweb yes :tangle arcology/static/sitemap.js
const container = document.getElementById("sitemap-container");
fetch('/sitemap/json')
    .then(response => response.json())
    .then((data) => {
        const graph = new graphology.Graph.from(data);

        <<forceAtlas2-setup>>

        const renderer = new Sigma(graph, container);

        <<hoverReducer>>

        <<forceAtlas2-invoke-worker>>

        <<clickNode>>

        return {renderer, graph}
    });
#+end_src

A [[https://graphology.github.io/standard-library/layout-forceatlas2.html][ForceAtlas2]] layout algorithm is applied to the page for 5 seconds to allow the nodes to coalesce in to a stable layout.

#+begin_src javascript :noweb-ref forceAtlas2-setup
var FA2Layout = graphologyLibrary.FA2Layout;
var forceAtlas2 = graphologyLibrary.layoutForceAtlas2;
const sensibleSettings = forceAtlas2.inferSettings(graph);
const layout = new FA2Layout(graph, {
    settings: sensibleSettings
});
#+end_src

The layout work is done in a [[https://graphology.github.io/standard-library/layout-forceatlas2.html#webworker][background task]]:

#+begin_src javascript :noweb-ref forceAtlas2-invoke-worker
layout.start();
setTimeout(() => {
    layout.stop()
}, 5000);
#+end_src

An event handler is defined for the =clickNode= event to open the URL embedded in the node's attributes added in [[id:20220712T101916.751167][Making SigmaJS Nodes]].

#+begin_src javascript :noweb-ref clickNode
renderer.on("clickNode", ({ node }) => {
    var realnode = graph._nodes.get(node);
    window.location = realnode.attributes.href;
});
#+end_src

** Focusing on Nodes in the Graph

The =<<hoverReducer>>= code to "highlight" a node by de-emphasizing the nodes it does not neighbor is taken from the [[https://codesandbox.io/s/github/jacomyal/sigma.js/tree/main/examples/use-reducers][SigmaJS examples]] and converted from TypeScript to JavaScript by hand. No I will not set up a NodeJS build chain.

#+begin_src javascript :noweb-ref hoverReducer
state = {};
function setHoveredNode(node) {
    if (node) {
        state.hoveredNode = node;
        state.hoveredNeighbors = new Set(graph.neighbors(node));
    } else {
            state.hoveredNode = undefined;
            state.hoveredNeighbors = undefined;
        }

        // Refresh rendering:
        renderer.refresh();
    }

    renderer.on("enterNode", ({ node }) => {
        setHoveredNode(node);
    });
    renderer.on("leaveNode", () => {
        setHoveredNode(undefined);
    });

    // Render nodes accordingly to the internal state:
    // 1. If a node is selected, it is highlighted
    // 2. If there is query, all non-matching nodes are greyed
    // 3. If there is a hovered node, all non-neighbor nodes are greyed
    renderer.setSetting("nodeReducer", (node, data) => {
        const res = { ...data };

        if (state.hoveredNeighbors && !state.hoveredNeighbors.has(node) && state.hoveredNode !== node) {
            res.label = "";
            res.color = "#f6f6f6";
        }

        if (state.selectedNode === node) {
            res.highlighted = true;
        } else if (state.suggestions && !state.suggestions.has(node)) {
            res.label = "";
            res.color = "#f6f6f6";
        }

        return res;
    });

    // Render edges accordingly to the internal state:
    // 1. If a node is hovered, the edge is hidden if it is not connected to the
    //    node
    // 2. If there is a query, the edge is only visible if it connects two
    //    suggestions
    renderer.setSetting("edgeReducer", (edge, data) => {
        const res = { ...data };

        if (state.hoveredNode && !graph.hasExtremity(edge, state.hoveredNode)) {
            res.hidden = true;
        }

        if (state.suggestions && (!state.suggestions.has(graph.source(edge)) || !state.suggestions.has(graph.target(edge)))) {
            res.hidden = true;
        }

        return res;
    });

#+end_src

* NEXT [[id:arcology/arroyo/ref][Arcology References]] as edges...