209 lines
9.4 KiB
Org Mode
209 lines
9.4 KiB
Org Mode
:PROPERTIES:
|
|
:ID: arroyo/feed-cache
|
|
:ROAM_ALIASES: "Arroyo Feed Cache" arroyo-feeds
|
|
:END:
|
|
#+TITLE: Arroyo Feed Cache Generator
|
|
#+PROPERTY: header-args:emacs-lisp :tangle arroyo-feeds.el :comments link
|
|
#+filetags: :Project:
|
|
#+ARCOLOGY_KEY: cce/arroyo/feeds
|
|
#+ARCOLOGY_ALLOW_CRAWL: t
|
|
*this isn't being used right now, sorry.*
|
|
|
|
Arroyo Feed Cache is used to generate a list of RSS and Atom URLs from across a [[id:cce/org-roam][org-roam]] [[id:knowledge_base][Knowledge Base]] in a format which can be used by the [[https://github.com/sloonz/ua][Universal Aggregator]] to present a =Maildir= cache of the feeds' contents so that they can be viewed in [[id:cce/gnus][Gnus]]. [[id:cce/universal_aggregator][Universal Aggregator]] is a [[https://xkcd.com/2347/][small but important]] part of my [[id:26c9e4fd-4501-4b8b-95ce-a2a5230d7c1e][Email and News and Information Pipelines]], but it's fairly simple and quite robust. I'll have documentation for operating UA itself defined elsewhere, Arroyo Feed Generator puts the =ggsrc= file in place.
|
|
|
|
It is managed and updated and curated in fits and bursts and when I want to expand my news horizon in some fashion. see [[id:79db5d1e-b52e-41e2-b75c-8a7c5c5a05c9][YouTube Feeds]] as an example in use; automatically generate from a table, copy to server, restart =UA= with [[id:arroyo/feed-cache-ggsrc][=arroyo-feeds-flood=]].
|
|
|
|
Arroyo Feed Cache consumes tabular feed information from pages throughout my [[id:knowledge_base][Knowledge Base]], as well as =ARROYO_FEED_URL= keywords stored in the [[id:arroyo/system-cache][Arroyo System Cache]].
|
|
|
|
* [[id:cce/literate_programming][Literate Programming]] helpers for engaging with the Feed Cache
|
|
:PROPERTIES:
|
|
:ID: arroyo/feed-cache-helpers
|
|
:END:
|
|
|
|
Using this is "indirect" compared to the rest of the arroyo systems -- rather than running in the context of an org-tangle process and an external coordinator process like [[id:arroyo/home-manager][Arroyo Home Manager]], =CALL= blocks must be used to create these files and read the table data in the context of [[id:cce/literate_programming][Org Babel]]. Given a table with three columns, and an invocation shaped like this the source example below, =CALL='ing that tangle-file function will collect the file and export it; this needs to somehow be automated! for now, eval it when the feed list changes to make sure the tangles are up to date. the tangled version of this can be fed directly in to [[id:cce/yasnippets][Yasnippet]], btw:
|
|
|
|
#+begin_src org :tangle arroyo-feed.yasnippet.org_archive :comments none
|
|
,#+name: tangle-file
|
|
,#+CALL: arroyo/arroyo-feed-cache.org:tangle-file[:results drawer](str=$1-ggs(), "ggs/70-$1.ggs")
|
|
|
|
,#+NAME: $1-ggs
|
|
,#+CALL: arroyo/arroyo-feed-cache.org:make-string(tbl=$1)
|
|
|
|
,#+name: $1
|
|
| <name> | <feed> | <category> |
|
|
#+end_src
|
|
|
|
This cross-file =CALL='ing is quite cool, those references are here and can be made absolute if Arroyo is not cloned in to =org-roam-directory=.
|
|
|
|
#+name: make-string
|
|
#+begin_src emacs-lisp :var tbl="" :exports code :tangle no
|
|
(arroyo-feeds-make-string tbl)
|
|
#+end_src
|
|
|
|
#+results: make-string
|
|
|
|
#+name: tangle-file
|
|
#+begin_src emacs-lisp :var str="" path="" :exports code :tangle no
|
|
(arroyo-feeds-write-file str path)
|
|
#+end_src
|
|
|
|
* Code for tabular file =ggsrc= fragment generation
|
|
:PROPERTIES:
|
|
:ID: arroyo/feed-cache-tab-fragments
|
|
:END:
|
|
|
|
#+begin_src emacs-lisp :results none
|
|
(defun arroyo-feeds-make-string (tbl)
|
|
"Converts an babelized org-mode tabel TBL in to an ggsrc command.
|
|
|
|
The table is three columns, NAME, FEED_URL, CATEGORY."
|
|
(->> tbl
|
|
(seq-map (lambda (line)
|
|
(format "rss \"%s\" %s # \"%s\"\n" (second line) (third line) (first line))))
|
|
(apply #'s-concat)))
|
|
#+end_src
|
|
|
|
#+begin_src emacs-lisp
|
|
(defun arroyo-feeds-write-file (ggsrc-block export-path)
|
|
(let* ((cce-directory (expand-file-name "cce" org-roam-directory))
|
|
(fname (expand-file-name export-path cce-directory)))
|
|
(with-current-buffer (find-file-noselect (expand-file-name export-path cce-directory))
|
|
(erase-buffer)
|
|
(insert ggsrc-block)
|
|
(save-buffer)
|
|
(format "[[%s][%s]]\n#+ARROYO_FEEDS: %s" fname export-path export-path))))
|
|
#+end_src
|
|
|
|
* Code for Keyword Metadata =ggsrc= fragment generation
|
|
|
|
Of course Arroyo can query the [[id:arroyo/system-cache][System Cache]] for the key/value pairs on individual pages in the form:
|
|
|
|
#+begin_example
|
|
,#+ARROYO_FEED_URL: https://afd.fontkeming.fail/AFDSEW.xml
|
|
,#+ARROYO_FEED_CATEGORY: News
|
|
#+end_example
|
|
|
|
[[(keyword-fragment)][arroyo-feeds--keyword-fragment]] returns a string which does this:
|
|
|
|
#+begin_src emacs-lisp
|
|
(defun arroyo-feeds--keyword-fragment ()
|
|
(->> (arroyo-db-get "ARROYO_FEED_URL")
|
|
;; add titles
|
|
(-map (pcase-lambda (`(,file ,url))
|
|
(let ((res (org-roam-db-query [:select title :from nodes
|
|
:where (= level 0) :and (= file $s1)]
|
|
file)))
|
|
(list file url (car (first res))))))
|
|
;; add feed category
|
|
(-map (pcase-lambda (`(,file ,url ,title))
|
|
(let ((res (arroyo-db-get "ARROYO_FEED_CATEGORY" file)))
|
|
(list file url title (first res)))))
|
|
;; format and output
|
|
(-map (pcase-lambda (`(,file ,url ,title ,category))
|
|
(concat "rss \"" url "\" " category
|
|
" # " (format "[[%s][%s]]" file title))))
|
|
(s-join "\n")))
|
|
#+end_src
|
|
|
|
* Generating a =ggsrc= file for [[id:cce/universal_aggregator][Universal Aggregator]]
|
|
:PROPERTIES:
|
|
:ID: arroyo/feed-cache-ggsrc
|
|
:END:
|
|
|
|
[[elisp:(arroyo-feeds-flood)][(arroyo-feeds-flood)]] will call the export functions in each file and then insert all of the fragments in to the final =ggsrc= configuration file.
|
|
|
|
The Feed Cache uses [[id:arroyo/system-cache][Arroyo System Cache]] to find files which can have feeds extracted from them:
|
|
|
|
#+begin_src emacs-lisp :results none
|
|
(add-to-list 'arroyo-db-keywords "ARROYO_FEEDS")
|
|
(add-to-list 'arroyo-db-keywords "ARROYO_FEED_URL")
|
|
(add-to-list 'arroyo-db-keywords "ARROYO_FEED_CATEGORY")
|
|
#+end_src
|
|
|
|
#+begin_src emacs-lisp -r
|
|
(defcustom arroyo-feeds-ggsrc-path "/ssh:fontkeming.fail:~/Maildir/ggsrc"
|
|
"The location to write final collated GGSRC file. This can be local or TRAMP path."
|
|
:group 'arroyo
|
|
:group 'arroyo-feeds
|
|
:type 'string)
|
|
|
|
(defun arroyo-feeds-flood (&optional no-restart)
|
|
"Create a ggsrc file suitable for the grey goo spawner. With prefix argument skip restarting service"
|
|
(interactive "P")
|
|
(with-current-buffer (find-file arroyo-feeds-ggsrc-path)
|
|
(erase-buffer)
|
|
(->> (arroyo-db-get "ARROYO_FEEDS")
|
|
(-sort #'arroyo-feeds--sort-pairs) ;; (ref:sort-pairs)
|
|
(-map (pcase-lambda (`(,file ,dest))
|
|
(arroyo-feeds--call-file file) ;; (ref:call-file)
|
|
dest)) ;; done with file, return only dest
|
|
(append (list "ggs/0-rss-command.ggsrc"))
|
|
(arroyo-feeds--insert-files)) ;; (ref:insert-files)
|
|
(goto-char (point-max))
|
|
(insert (arroyo-feeds--keyword-fragment)) ;; (ref:keyword-fragment)
|
|
(save-buffer))
|
|
|
|
(unless no-restart
|
|
(let ((default-directory "/ssh:fontkeming.fail:/home/rrix"))
|
|
(async-shell-command "sudo systemctl restart ua && echo \"done\""))))
|
|
#+end_src
|
|
|
|
[[(sort-pairs)][arroyo-feeds--sort-pairs]] is simple, except that I've gone mad with the power of =pcase=; it sorts the database results lexically by the path of the output file, which is generally of the form =ggs/$SORT_INTEGER-name.ggsrc= so that boils down to a lexical sort of the two digit sort integer. Use it or weep. [[id:cce/universal_aggregator][Universal Aggregator]] supplies a stage-zero with commands defined in it and I will probably move that in to Arroyo itself soon.
|
|
|
|
#+begin_src emacs-lisp :results none
|
|
(defun arroyo-feeds--sort-pairs (one two)
|
|
(pcase-let ((`(,_ ,one-path) one)
|
|
(`(,_ ,two-path) two))
|
|
(s-less? one-path two-path)))
|
|
#+end_src
|
|
|
|
[[(call-file)][arroyo-feeds--call-file]] is also simple, the discovery of =org-sbe= is fortuitous.
|
|
|
|
#+begin_src emacs-lisp :results none
|
|
(defun arroyo-feeds--ugly-execute-src-block (block-name)
|
|
(save-excursion
|
|
(goto-char (point-min))
|
|
(while (and (search-forward-regexp
|
|
(org-babel-named-src-block-regexp-for-name block-name)
|
|
nil t)
|
|
(org-babel-execute-src-block)))))
|
|
|
|
(defun arroyo-feeds--call-file (path)
|
|
(with-current-buffer (find-file-noselect path)
|
|
(arroyo-feeds--ugly-execute-src-block "tangle-file")))
|
|
#+end_src
|
|
|
|
[[(ref:insert-files)][arroyo-feeds--insert-files]] iterates over the list of fully expanded file names in to the target file:
|
|
|
|
#+begin_src emacs-lisp :results none
|
|
(defun arroyo-feeds--insert-files (files)
|
|
(let ((cce-directory (expand-file-name "cce" org-roam-directory)))
|
|
(dolist (fragment files)
|
|
(goto-char (point-max))
|
|
(insert-file-contents (expand-file-name fragment cce-directory)))))
|
|
#+end_src
|
|
|
|
** DONE support =ARROYO_FEED_URL= and =ARROYO_FEED_CATEGORY=
|
|
:LOGBOOK:
|
|
- State "DONE" from "NEXT" [2021-08-29 Sun 21:18]
|
|
:END:
|
|
** NEXT deploy the file to the server (for now with shell script, eventually deploy nixos)
|
|
|
|
* [[id:cce/universal_aggregator][Universal Aggregator]] "top-matter"
|
|
|
|
#+begin_src shell :tangle ../cce/ggs/0-rss-command.ggs
|
|
ROOTDIR=/data/feeds
|
|
CACHE=/data/ua-cache
|
|
default_timeout=30
|
|
rss() {
|
|
command 2000 "rss2json \"$1\" | ua-inline | maildir-put -cache ${CACHE} -redis -redis-db 1 -root ${ROOTDIR} -folder \"$2\""
|
|
}
|
|
#+end_src
|
|
|
|
* Footmatter
|
|
|
|
#+begin_src emacs-lisp
|
|
(provide 'arroyo-feeds)
|
|
#+end_src
|