somehow terra-firma's uuids aren't univerally unique.

gnus_adaptive_scoring.org

@@ -3,19 +3,20 @@
 :END:
 #+TITLE: Gnus Adaptive Scoring
 #+filetags: :Emacs:CCE:Communications:
-#+PROPERTY: header-args            :mkdirp yes :results none
-#+PROPERTY: header-args:emacs-lisp :tangle gnus-adaptive.el
-#+ARROYO_EMACS_MODULE: gnus-adaptive
+#+PROPERTY: header-args            :mkdirp yes 
+#+PROPERTY: header-args:emacs-lisp :tangle gnus-adaptive.el :results none
+
 #+ARCOLOGY_KEY: cce/gnus-adaptive
-#+CCE_PRIORITY: 71
-#+ARROYO_MODULE_WANTS: 	cce/gnus.org
 #+ARCOLOGY_ALLOW_CRAWL: t
 
-I believe that [[id:cce/gnus][Gnus]] has a single feature which keeps me using it, stuck around me like the world's lightest anchor. Gnus has a very powerful scoring system, where each message and each mailing list and each IMAP folder will develop a score which can be used as a sorting key.
+#+ARROYO_EMACS_MODULE: gnus-adaptive
+#+ARROYO_MODULE_WANTS: 	cce/gnus.org
 
-This scoring system can be programmed with explicit commands: "When you see a message from my mom, make sure that sorts to the top". "Never show me emails from this person who sends me angry emails any time I publish a document." "When my CPA or healthcare provider email me, I need to see those mails."
+I believe that [[id:cce/gnus][Gnus]] has a single feature which keeps me using it, stuck around me like the world's lightest anchor. Gnus has a very powerful "scoring" system, a message sorting algorithm where each message and each mailing list and each IMAP folder will develop over time a score which can be used as a sorting key to bubble the most relevant things to the top of my attention.
 
-This scoring system, critically, can also be scored with an automatically evolving "Adaptive Score" file. Gnus adds a single bit of information that must be provided while I am reading my mail: whether this email is relevant or irrelevant. When Gnus closes a folder, it will look at each mail, and will add or subtract points to the people, words, topics, sentences and groups I open most often.
+This scoring system can be programmed with explicit commands: "When you see a message from my mom, make sure that sorts to the top (+10000 points)". "Never show me emails from this person who sends me angry emails any time I publish a document. (-5000 points)" "When my CPA or healthcare provider email me, I need to see those mails. (+1000 points)"
+
+This scoring system, critically, can also be scored with an automatically evolving "Adaptive Score" file. Gnus adds a single bit of information that must be provided while I am reading my mail: whether this email is relevant or irrelevant. When Gnus closes a folder, it will look at each mail, and will add or subtract points to the people, words, topics, sentences and groups I interact with most often.
 
 These score files can be combined in to a system which learns from use over time but can also be overridden by the user when the system fails to intuit properly, and this is not a kludge like when GMail fails to mark a message as important, going in and cleaning up behind it and hoping it doesn't happen next time. If I have a need to see a class of messages, I have a firm guarantee that they will be near the top of my inbox.
 
@@ -23,7 +24,15 @@ Anything that scores below a certain threshold is semi-visible: The messages app
 
 With this simple workflow, with this one bit of information and a terribly inefficient "Artificial Stupidity", I am confident in my ability to triage both mail and news sources. This provides a highly personalized news reading expierence with minimal overhead and the more I use it the smarter it becomes.
 
-As an example: When I would take time off at [[id:38b6cc2e-55a8-496e-b8bd-813e1b2c83af][Uber]], I would often do so for two weeks or more at a time, and when I would do so I would *fully* disconnect. You could page me if you needed me, you could call me and I'd respond, and my team knew this, but I would come back to work with six or seven hundred unread mails, some critical, some that could be answered by maybe me and three other folks, a lot of update emails and newsletters, and a fair bit of crap. And I would often be free of the pile by olunch time, having brought my inbox down to the 30 or so emails that I would actually need to address, and I would still have a vague idea of what had happened in the two weeks by reading the tl;dr section often included in the top of update and newsletter emails. I heavily attribute my success as a technical lead in that highly-social distributed development environment to my ability to process and filter information effectively, and retain the important bits to disseminate to my colleagues and include in development practice.
+As an example: When I would take time off at [[id:38b6cc2e-55a8-496e-b8bd-813e1b2c83af][Uber]], I would often do so for two weeks or more at a time, and when I would do so I would *fully* disconnect. You could page me if you needed me, you could call me and I'd respond, and my team knew this, but I would come back to work with six or seven hundred unread mails, some critical, some that could be answered by maybe me and three other folks, a lot of update emails and newsletters, and a fair bit of crap. And I would often be free of the pile by lunch time, having brought my inbox down to the 30 or so emails that I would actually need to address, and I would still have a vague idea of what had happened in the two weeks by reading the tl;dr section often included in the top of update and newsletter emails. I heavily attribute my success as a technical lead in that highly-social distributed development environment to my ability to process and filter information effectively, and retain the important bits to disseminate to my colleagues and include in development practice.
+
+But at what cost? Each directory has to be processed locally for scores -- there's not really a great efficient way to do this on the backend. I had some idea for [[id:20210921T150711.001619][IMAP-based Smart Scoring]] but never bothered figuring out how to implement it. But running this sort of algorithm on the client is fine for some folks, especially for me because [[id:personal_software_can_be_shitty][Personal Software Can Be Shitty]], it doesn't *need* to scale to web-scale or even community-scale if it just runs on my device. Shipping the score files around is a bit annoying but definitely viable with  [[id:cce/syncthing][Syncthing]]. I would love to have this sort of behavior for more clients like in the [[id:62538db5-d94a-47c3-9998-086ded91fd88][Fediverse]]. I had, for some time, a [[id:cce/universal_aggregator][Universal Aggregator]] script which would ingest my Twitter posts in to Maildir so that I could read them in [[id:cce/gnus][Gnus]] and run the adaptive scoring on them but that is not a good way to read twitter. It works *great* on RSS feeds though where people are less likely to talk "around" a topic and more directly write about a topic.
+
+It's difficult for me to express how powerful this system becomes as it ages, though. I have yet to find a news reader or email client which can emulate these features, which has lead to this strange situation where I don't want to read my email on my phone, where I must be on a desktop or a laptop to be able to keep up with the world around me. There are many systems which will claim to have a "smart" recommendation engine. Because Gnus can rely on the horsepower of my own machine, rather than some CPU time on a datacenter cluster that is being tracked for cost efficiency, my system can get away with brute-forcing this behavior in a way that may make many uncomfortable because it is not an optimal solution, and it is not a good model for how you want to solve this at scale.
+
+[[roam:#MePersonally]], I believe that the [[id:645d757d-97ab-48fb-9f8d-a650c3bfff8e][Technology]] industry's reliance on opaque machine learning models is creating inequitable technology, and has been part of a decades long trend of stripping users of control and telling us that we're better off for it. It's only now that we are starting to grapple with the negative effects and costs of automated decision making, and we are finding much to be mad about. Simple, transparent systems which are not immediately effective but which grow over time in a healthy auditable fashion, are an improvement over the network weight models which are easy to scale but effectively magic to the public and to even technical pundits and thinkers.
+
+* Configuring Adaptive Scoring
 
 #+begin_src emacs-lisp
 (provide 'cce/gnus-adaptive)
@@ -68,7 +77,3 @@ Finally, I mentioned in the introductory text above that this also worked on *gr
 #+begin_src emacs-lisp
 (add-hook 'gnus-summary-exit-hook 'gnus-summary-bubble-group)
 #+end_src
-
-It's difficult for me to express how powerful this system becomes as it ages. I have yet to find a news reader or email client which can emulate these features, which has lead to this strange situation where I don't want to read my email on my phone, where I must be on a desktop or a laptop to be able to keep up with the world around me. There are many systems which will claim to have a "smart" recommendation engine. Because Gnus can rely on the horsepower of my own machine, rather than some CPU time on a datacenter cluster that is being tracked for cost efficiency, my system can get away with brute-forcing this behavior in a way that may make many uncomfortable because it is not an optimal solution, and it is not a good model for how you want to solve this.
-
-On the contrary, I believe that the [[id:645d757d-97ab-48fb-9f8d-a650c3bfff8e][Technology]] industry's reliance on opaque machine learning models is creating inequitable technology, and has been part of a decades long trend of stripping users of control and telling us that we're better off for it. It's only now that we are starting to grapple with the negative effects and costs of automated decision making, and we are finding much to be mad about. Simple, transparent systems which are not immediately effective but which grow over time in a healthy auditable fashion, are an improvement over the network weight models which are easy to scale but effectively magic to the public and to even technical pundits and thinkers.

nextcloud.org

@@ -18,6 +18,8 @@ Here is how it's deployed to [[id:20211120T220054.226284][The Wobserver]]:
 { ... }:
 
 {
+
+  services.postgresql.ensureDatabases = ["nextcloud"];
   services.postgresql.ensureUsers = [
     {
       name = "nextcloud";

nginx.org

@@ -125,10 +125,11 @@ I use [[https://letsencrypt.org/][Lets Encrypt]] for my DNS, I really like 'em.
 
 {
   services.nginx.virtualHosts = {
-    "fontkeming.fail".root = "/var/www/html";
+    "fontkeming.fail".root = "/srv/static-sites/default";
+    "fontkeming.fail".default = true;
 
     # additional home.rix.si stuff in wobserver-observability!
-    "home.rix.si".root = "/var/www/html";
+    "home.rix.si".root = "/srv/static-sites/default";
     "home.rix.si".locations."/fdroid".root = "/srv/fdroid/repo";
 
     "afd.fontkeming.fail".root = "/srv/afdsew/SEW";

nixos-wobserver.org

@@ -51,9 +51,12 @@ rec {
 
   services.openssh.enable = true;
 
+  home-manager.users.rrix.services.syncthing.tray.enable = lib.mkForce false;
+
   boot = {
     kernelParams = [ "console=ttyS0" "boot.shell_on_fail" ];
     loader.timeout = 5;
+    zfs.devNodes = lib.mkForce "/dev/disk/by-id";
   };
 }
 #+end_src
@@ -70,6 +73,7 @@ rec {
 
 #+results: arroyo_nixos_imports
 #+begin_example
+../../nixos/arcology-config.nix
 ../../nixos/cachix.nix
 ../../nixos/emacs.nix
 ../../nixos/gnupg-pam.nix
@@ -83,7 +87,9 @@ rec {
 ../../nixos/rixpkgs.nix
 ../../nixos/rrix.nix
 ../../nixos/ssh_client.nix
+../../nixos/syncthing.nix
 ../../nixos/tailscale.nix
+../../nixos/wobservability.nix
 ../../nixos/zfs.nix
 #+end_example
 
@@ -120,6 +126,7 @@ rec {
 ../../hm/shell-helpers.nix
 ../../hm/spell-check.nix
 ../../hm/ssh_client.nix
+../../hm/syncthing.nix
 #+end_example
 
 * Packages in the Wobserver
@@ -132,28 +139,31 @@ NixOS modules:
  (--map (caar (arroyo-db-query [:select file :from keywords :where (= value $s1)] it)))
  (--map (car (org-roam-db-query [:select [id title] :from nodes :where (= file $s1) :and (= level 0)] it)))
  (--map (format "- [[id:%s][%s]]" (first it) (second it)))
+ (-sort #'string<)
  (s-join "\n")
 )
 #+end_src
 
 #+results:
 :results:
-- [[id:cce/wobserver/postgres][PostgreSQL on the Wobserver]]
-- [[id:20220210T155158.671084][From Wireguard to Tailscale]]
-- [[id:e4998eda-d14a-48ee-9661-3d7d1bead53c][Wobserver Nginx Frontends]]
 - [[id:20220101T180015.306163][Nextcloud on Wobserver]]
+- [[id:20220101T190353.843667][Wobserver Observability]]
+- [[id:20220210T155158.671084][From Wireguard to Tailscale]]
 - [[id:20220526T143555.660133]["The manual appears to depend on the location of Nixpkgs"]]
-- [[id:cce/gnupg_configuration][GnuPG Configuration]]
-- [[id:c75d20e6-8888-4c5a-ac97-5997e2f1c711][CCE Nixos Core]]
-- [[id:cce/public_keys][My Public Keys and NixOS user]]
-- [[id:cce/nix_community_cachix][Nix Community Cachix]]
-- [[id:c4c2d068-c5b7-4cbc-b34a-297a68fb7c26][Where I Am At]]
-- [[id:47ff77f9-3eae-43eb-886c-7513d05f047f][Secure Backup Infrastructure]]
 - [[id:20221021T115008.329657][Arroyo Nix Support]]
-- [[id:cce/home-manager][Generate a Dynamic Home Manager Configuration]]
-- [[id:cce/syncthing][Nearly Stateless Computing Using Syncthing]]
+- [[id:47ff77f9-3eae-43eb-886c-7513d05f047f][Secure Backup Infrastructure]]
+- [[id:arcology/poetry][Arcology Poetry Pyproject]]
 - [[id:arroyo/emacs][Arroyo Emacs Generator]]
+- [[id:c4c2d068-c5b7-4cbc-b34a-297a68fb7c26][Where I Am At]]
+- [[id:c75d20e6-8888-4c5a-ac97-5997e2f1c711][CCE Nixos Core]]
+- [[id:cce/gnupg_configuration][GnuPG Configuration]]
+- [[id:cce/home-manager][Generate a Dynamic Home Manager Configuration]]
+- [[id:cce/nix_community_cachix][Nix Community Cachix]]
+- [[id:cce/public_keys][My Public Keys and NixOS user]]
 - [[id:cce/ssh_configuration][SSH Configuration]]
+- [[id:cce/syncthing][Nearly Stateless Computing Using Syncthing]]
+- [[id:cce/wobserver/postgres][PostgreSQL on the Wobserver]]
+- [[id:e4998eda-d14a-48ee-9661-3d7d1bead53c][Wobserver Nginx Frontends]]
 :end:
 
 Things I need to package:

postgresql.org

@@ -16,18 +16,17 @@ My go-to multi-write database server is postgres. I like how it has JSON columns
 {
   services.postgresql = {
     enable = true;
-    ensureDatabases = [ "gitea" "nextcloud" "synapse" ];
     settings = {
       wal_level = "replica";
       archive_mode = "on";
-      archive_command = "test ! -f /svc/pgsql/archive/wal/%f && cp %p /svc/pgsql/archive/wal/%f";
+      archive_command = "test ! -f /srv/pgsql/archive/wal/%f && cp %p /srv/pgsql/archive/wal/%f";
     };
   };
   # backup weekly
   services.postgresqlBackup = {
     enable = true;
     backupAll = true;
-    location = "/backup/pgsql/";
+    location = "/srv/pgsql/backups/";
     startAt = "Mon *-*-* 02:00:00";
   };
 }

wobserver-observability.org

@@ -7,6 +7,11 @@
 #+ARCOLOGY_ALLOW_CRAWL: t
 #+ARCOLOGY_KEY: cce/wobserver/observability
 
+#+AUTO_TANGLE: t
+
+#+ARROYO_NIXOS_MODULE: nixos/wobservability.nix
+#+ARROYO_NIXOS_ROLE: server
+
 It is critical that the [[id:20211120T220054.226284][Wobserver]] can be "self-healing" -- not that all faults can be automatically fixed, but that with a human in the loop all common failure modes can be managed and normal operation can be maintained.
 
 Most of the problems are disk usage and IO utilization, failed disks and the like.
@@ -20,13 +25,14 @@ I keep wanting to set up telegraf, but for now I'll use =prometheus= since it ma
 
 I need to set up alerts and dashboards for the most common operations, and I'd love some declarative way to share these. Oh some day.
 
-#+begin_src nix
-{ ... }:
+#+begin_src nix :tangle ~/arroyo-nix/nixos/wobservability.nix
+{ pkgs, config, ... }:
 
 rec {
   services.prometheus = {
     retentionTime = "60d";
     # scrapConfigs = [];
+    listenAddress = "127.0.0.1";
     exporters = {
       node = {
         enable = true;
@@ -35,21 +41,59 @@ rec {
     };
   };
 
-  services.grafana = {
+  # services.prometheus.exporters.pihole = {};
+  # services.prometheus.exporters.nginx = {};
+  # services.prometheus.exporters.nginxlog = {};
+  # services.prometheus.exporters.unifi = {};
+
+  services.prometheus.exporters.process = {
     enable = true;
-    addr = "127.0.0.1";
-    analytics.reporting.enable = false;
-    dataDir = "/svc/grafana";
-    rootUrl = "https://home.rix.si/grafana";
-
-    database = {
-      name = "grafana";
-      type = "postgres";
-      user = "grafana";
-    };
-
+    settings.process_names = [
+      # Remove nix store path from process name
+      {
+        name = "{{.Matches.Wrapped}} {{ .Matches.Args }}";
+        cmdline = [ "^/nix/store[^ ]*/(?P<Wrapped>[^ /]*) (?P<Args>.*)" ];
+      }
+    ];
   };
 
+  services.prometheus.exporters.smartctl.enable = true;
+  services.smartd.enable = true;
+  environment.systemPackages = [ pkgs.smartmontools ];
+
+  services.prometheus.exporters.postgres = {
+    enable = true;
+    runAsLocalSuperUser = true;
+  };
+  services.prometheus.exporters.zfs.enable = true;
+
+  services.grafana = {
+    enable = true;
+    dataDir = "/srv/grafana";
+
+    settings = {
+      analytics.reporting_enable = false;
+      database = {
+        name = "grafana";
+        type = "postgres";
+        user = "grafana";
+        host = "/run/postgresql/";
+      };
+      server = {
+        enable_gzip = true;
+        http_addr = "127.0.0.1";
+        http_port = 3000;
+        root_url = "https://home.rix.si/grafana";
+      };
+    };
+  };
+
+  services.grafana-image-renderer = {
+    enable = true;
+    provisionGrafana = true;
+  };
+
+  services.postgresql.ensureDatabases = ["grafana"];
   services.postgresql.ensureUsers = [
     {
       name = "grafana";
@@ -61,11 +105,11 @@ rec {
 
   services.nginx.virtualHosts."home.rix.si" = {
     locations."/prom" = {
-      proxyPass = "http://127.0.0.1:${services.prometheus.port}";
+      proxyPass = "http://127.0.0.1:${toString config.services.prometheus.port}";
     };
     
     locations."/grafana" = {
-      proxyPass = "http://${services.grafana.addr}:${services.grafana.port}";
+      proxyPass = "http://${config.services.grafana.settings.server.http_addr}:${toString config.services.grafana.settings.server.http_port}";
       extraConfig = ''
         proxy_set_header Host $host;
       '';