11 KiB
Navigating the Arcology Site Graph with SigmaJS
- What is SigmaJS
- Getting SigmaJS
- Generating the Nodes and Edges bundle from Arroyo Arcology
- Sitemap HTML Template
- NEXT Arcology References as edges…
What is SigmaJS
Getting SigmaJS
Download and vendor the minified versions:
export GRAPHOLOGY_VER=0.24.1
export SIGMAJS_VER=v2.3.1
curl --location -v -o arcology/static/sigmajs.min.js https://github.com/jacomyal/sigma.js/releases/download/$SIGMAJS_VER/sigma.min.js
curl --location -v -o arcology/static/graphology.min.js https://github.com/graphology/graphology/releases/download/$GRAPHOLOGY_VER/graphology.min.js
curl --location -v -o arcology/static/graphology-library.min.js https://github.com/graphology/graphology/releases/download/$GRAPHOLOGY_VER/graphology-library.min.jsAt this point the Arcology Page Template should be able to import these thanks to the Arcology FastAPI router exposing that static directory. I could set up a typescript dev environment and webpack and all this sort of shit, but I don't really need it…. It's fine! roam:This Is Fine!
Generating the Nodes and Edges bundle from Arroyo Arcology
From arroyo-db load all Arcology Page as Nodes shaped like:
{
"id": "n0",
"label": "A node",
"x": 0,
"y": 0,
"size": 3
}
From arroyo-db load all Arcology Links as Edges shaped like:
{
"id": "e0",
"source": "n0",
"target": "n1"
}
compose together and return in a single dict for JSON rendering shaped like:
{
"nodes": [
<<all nodes>>
]
"edges": [
<<all edges>>
]
}
Making SigmaJS Nodes
Nodes are simple:
import hashlib
from sqlmodel import Session, select
import arcology.arroyo as arroyo
from arcology.arroyo import Page
def make_loc_hash(page: Page, salt, max_q=700):
key = page.get_file() + str(salt)
hash = hashlib.sha224(key.encode("utf-8")).digest()
return int.from_bytes(hash, byteorder="big") % max_q
def nodes(engine):
# collect all pages
with Session(engine) as s:
pages = s.exec(select(Page)).all()
nodes = [
dict(
key=page.get_key(),
attributes=dict(
label=page.get_title(),
x=make_loc_hash(page, 1),
y=make_loc_hash(page, 2),
size=min((len(page.backlinks) + len(page.outlinks)) / 2, 10),
color=page.get_site().node_color,
href=page.get_arcology_key().to_url()
)
)
for page in pages
]
return nodesMaking SigmaJS Edges
Edges are too, we just have to make sure the IDs line up:
from sqlmodel import Session, select
import arcology.arroyo as arroyo
from arcology.arroyo import Link, engine
from arcology.parse import parse_sexp
# HACK: https://stackoverflow.com/questions/1151658/python-hashable-dicts
class hashabledict(dict):
def __hash__(self):
return hash(tuple(sorted(self.items())))
def edges(engine):
# collect all pages
with Session(engine) as s:
q_links = s.exec(select(Link)).all()
links = set()
for link in q_links:
source = link.source_page.get_key()
dest = link.dest_page.get_key()
links.add(
hashabledict(
key=f"{source}-{dest}",
source=source,
target=dest,
)
)
return list(links)Compose these together, let the Router handle turning it in to JSON. I clobber this in to an LRU cache like I do in a bunch of other places with a key brute-forced to map to the SHA hashes of every file in the public wiki… Woof! Glad Personal Software Can Be Shitty!
import functools
@functools.lru_cache(maxsize=20)
def sigma_lru(cache_key):
return dict(
# this is a different import of the same engine...
nodes=nodes(engine),
edges=edges(engine)
)
def sigma(engine):
with Session(engine) as session:
pages = session.exec(select(Page).order_by(Page.key))
hashes =[page.hash for page in pages]
cache_key = ''.join(
hashes
)
return sigma_lru(cache_key)Sitemap HTML Template
This derives from the Base HTML Template for the sites. It's decomposed a little bit with noweb syntax so that the JavaScript elements can be worked on as JavaScript instead of as Jinja2 templates.
{% extends "base.html.j2" %}
{% block h1 %}
<h1><a href='/'>{{ site.title }}</a></h1>
<h2>The Site Map</h2>
{% endblock %}
{% block head %}
<title>Arcology System Site Map</title>
<script src="/static/graphology.min.js"></script>
<script src="/static/graphology-library.min.js"></script>
<script src="/static/sigmajs.min.js"></script>
{% endblock %}
{% block body %}
<main>
<section class="body">
<p>
This is a network graph of all the pages published in my
Arcology network. The color of the node corresponds to its
site, and you can click on any of them to jump to that page or
zoom in to see more of their labels. I will be adding more
features to this sitemap like search and highlighting
"relative" neighbors later on. Feel free to view
<a href="https://engine.arcology.garden/sitemaps">the implementation</a>
within the Arcology pages, as usual.
</p>
<div id="sitemap-container">
</div>
</section>
<script type="application/javascript" src="/static/sitemap.js"></script>
</main>
{% endblock %}JavaScript to set up and render the SigmaJS graph
The SigmaJS code is loaded in the <head>, and the graph is injected in to the #sitemap-container <div>.
const container = document.getElementById("sitemap-container");
fetch('/sitemap/json')
.then(response => response.json())
.then((data) => {
const graph = new graphology.Graph.from(data);
<<forceAtlas2-setup>>
const renderer = new Sigma(graph, container);
<<hoverReducer>>
<<forceAtlas2-invoke-worker>>
<<clickNode>>
return {renderer, graph}
});A ForceAtlas2 layout algorithm is applied to the page for 5 seconds to allow the nodes to coalesce in to a stable layout.
var FA2Layout = graphologyLibrary.FA2Layout;
var forceAtlas2 = graphologyLibrary.layoutForceAtlas2;
const sensibleSettings = forceAtlas2.inferSettings(graph);
const layout = new FA2Layout(graph, {
settings: sensibleSettings
});The layout work is done in a background task:
layout.start();
setTimeout(() => {
layout.stop()
}, 5000);
An event handler is defined for the clickNode event to open the URL embedded in the node's attributes added in Making SigmaJS Nodes.
renderer.on("clickNode", ({ node }) => {
var realnode = graph._nodes.get(node);
window.location = realnode.attributes.href;
});Focusing on Nodes in the Graph
The <<hoverReducer>> code to "highlight" a node by de-emphasizing the nodes it does not neighbor is taken from the SigmaJS examples and converted from TypeScript to JavaScript by hand. No I will not set up a NodeJS build chain.
state = {};
function setHoveredNode(node) {
if (node) {
state.hoveredNode = node;
state.hoveredNeighbors = new Set(graph.neighbors(node));
} else {
state.hoveredNode = undefined;
state.hoveredNeighbors = undefined;
}
// Refresh rendering:
renderer.refresh();
}
renderer.on("enterNode", ({ node }) => {
setHoveredNode(node);
});
renderer.on("leaveNode", () => {
setHoveredNode(undefined);
});
// Render nodes accordingly to the internal state:
// 1. If a node is selected, it is highlighted
// 2. If there is query, all non-matching nodes are greyed
// 3. If there is a hovered node, all non-neighbor nodes are greyed
renderer.setSetting("nodeReducer", (node, data) => {
const res = { ...data };
if (state.hoveredNeighbors && !state.hoveredNeighbors.has(node) && state.hoveredNode !== node) {
res.label = "";
res.color = "#f6f6f6";
}
if (state.selectedNode === node) {
res.highlighted = true;
} else if (state.suggestions && !state.suggestions.has(node)) {
res.label = "";
res.color = "#f6f6f6";
}
return res;
});
// Render edges accordingly to the internal state:
// 1. If a node is hovered, the edge is hidden if it is not connected to the
// node
// 2. If there is a query, the edge is only visible if it connects two
// suggestions
renderer.setSetting("edgeReducer", (edge, data) => {
const res = { ...data };
if (state.hoveredNode && !graph.hasExtremity(edge, state.hoveredNode)) {
res.hidden = true;
}
if (state.suggestions && (!state.suggestions.has(graph.source(edge)) || !state.suggestions.has(graph.target(edge)))) {
res.hidden = true;
}
return res;
});