rrix
/
arcology-fastapi
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
arcology-fastapi/arcology-poetry.org

531 lines
18 KiB
Org Mode
Raw Permalink Blame History

:PROPERTIES:
:ID: arcology/poetry
:END:
#+TITLE: Arcology Poetry Pyproject
#+filetags: :Project:Arcology:
#+ARCOLOGY_KEY: arcology/poetry
#+ARCOLOGY_ALLOW_CRAWL: t
#+AUTO_TANGLE: t
Okay so the [[id:arcology/fastapi][Arcology FastAPI]] package is built with =poetry=. I run the commands, look at the output, and copy it back in here... This is not very ergonomic right now, but I don't have a better idea on how to manage these literately.
#+begin_src conf-toml :tangle pyproject.toml
[tool.poetry]
name = "arcology"
version = "0.1.0"
description = "The Arcology is a Multi-domain Web Site Engine for Org Roam Files"
authors = ["Ryan Rix <code@whatthefuck.computer>"]
include = ["static", "templates", "pandoc"]
#+end_src
#+begin_src conf-toml :tangle pyproject.toml
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
#+end_src
* Arcology Python Dependencies
:PROPERTIES:
:ID: arcology/poetry/dependencies
:END:
Here's what I use:
I don't have a good workflow for bringing this literate doc in sync to the tools output regularly, to de-tangle changes made by =poetry add --lock='ing new dependencies or updating their versions. The best bet is to run =poetry add --lock= and then nix-shell in -- this will update the package environment and lock file, and then the updated =pyproject.toml= will need to be de-tangled back to here by-hand...
#+begin_src conf-toml :tangle pyproject.toml
[tool.poetry.dependencies]
python = "^3.9"
fastapi = "^0.70"
uvicorn = "^0.16"
sqlmodel = "^0.0.11"
# https://github.com/tiangolo/sqlmodel/issues/315
# sqlalchemy = "1.4.35"
sexpdata = "^0.0.3"
pypandoc = "^1.7"
Jinja2 = "^3.0"
prometheus-fastapi-instrumentator = "^5.7"
asyncinotify = "^2.0.2"
transitions = "^0.8.11"
graphviz = "^0.19.1"
pygraphviz = "^1.9"
async-lru = "^1.0.3"
[tool.poetry.dev-dependencies]
ipdb = "^0.13"
#+end_src
* Poetry creates Startup Scripts in =PATH=
Poetry can create wrapper scripts for modules in the package which will be automatically stuck in to =PATH= or the appropriate spot.
#+begin_src conf-toml :tangle pyproject.toml
[tool.poetry.scripts]
arcology-inotify = 'arcology.inotify:start'
arcology-fastapi = 'arcology.server:start'
#+end_src
The [[id:20211218T222408.578567][arcology inotify-watcher]] is invoked simply, right? it's just a little thing, doesn't even need command line arguments since it's configured in the environment. Same with the [[id:arcology/fastapi][Arcology FastAPI]] server!
* Nix Derivations
** =poetry2nix= will package the Arcology application up based on Poetry =TOML= in =default.nix=
:PROPERTIES:
:ID: arcology/nix/poetry2nix
:ROAM_ALIASES: arcology/default.nix
:END:
The Poetry application is factored out so that it can be used in the [[id:arcology/nix/shell][=nix-shell= below]]. Using =poetry2nix= to extract the package information from =pyproject.toml= is a pretty simple affair with [[https://github.com/nix-community/poetry2nix][=poetry2nix=]] bundled with [[id:c75d20e6-8888-4c5a-ac97-5997e2f1c711][nixpkgs]].
#+begin_src nix :tangle default.nix :noweb yes
{ pkgs ? import <nixpkgs> {},
poetry2nix ? import ((import <arroyo/versions.nix> {}).poetry2nix {}) {},
stdenv ? pkgs.stdenv,
python ? pkgs.python3 }:
let
mkPoetryApplication' =
<<mkPoetryApplicationPrime>>
;
in
mkPoetryApplication' {
inherit python;
projectDir = ./.;
propagatedBuildInputs = [
pkgs.coreutils
pkgs.pandoc
];
overrides = [
(poetry2nix.defaultPoetryOverrides.overrideOverlay (
self: super: {
sqlmodel = super.sqlmodel.overridePythonAttrs (
old: {
buildInputs = (old.buildInputs or [ ]) ++ [self.poetry-core];
patchPhase = (old.patchPhase or "") + ''
# fix pyproject.toml version?
substituteInPlace pyproject.toml --replace 'version = "0"' 'version = "${old.version}"'
'';
}
);
traitlets = super.traitlets.overridePythonAttrs (
old: {
buildInputs = (old.buildInputs or [ ]) ++ [self.hatchling];
}
);
pypandoc = super.pypandoc.overridePythonAttrs (
old: {
buildInputs = (old.buildInputs or [ ]) ++ [self.poetry-core];
}
);
asyncinotify = super.asyncinotify.overridePythonAttrs (
old: {
buildInputs = (old.buildInputs or [ ]) ++ [self.setuptools];
}
);
sqlalchemy2-stubs = super.sqlalchemy2-stubs.overridePythonAttrs (
old: {
buildInputs = (old.buildInputs or [ ]) ++ [self.setuptools];
}
);
}
))
];
editablePackageSources = {
arcology = ./.;
};
}
#+end_src
*** =mkPoetryApplication'= exists to allow for unified editable source shell and application
:PROPERTIES:
:ID: arcology/poetry/mkPoetryApplicationPrime
:END:
From [[https://github.com/nix-community/poetry2nix/issues/423][nix-community/poetry2nix#423]]:
#+begin_quote
I am developing a few applications using this awesome projects. One thing that bothered me for a while that it takes a lot of code duplication to maintain a derivation for packaging using =mkPoetryApplication= and a second one for nix-shell using e.g. =mkPoetryEnv=. Unfortunately there is no easy way to override the =mkPoetryPackages= call in =mkPoetryApplication= to pass in editablePackageSources and mkPoetryEnv doesn't support the same hooks that buildPythonPackage supports. In my packages I e.g. need to call makeWrapper on the generated executables which cannot be easily done currently.
I use the following hack to get around this limitation:
#+end_quote
and so do i. This can be included as a noweb (or, i guess, properly =imported= but we hackin' here.)
#+begin_src nix :noweb-ref mkPoetryApplicationPrime
{ projectDir,
editablePackageSources,
overrides ? poetry2nix.defaultPoetryOverrides,
... }@args:
let
# pass all args which are not specific to mkPoetryEnv
app = poetry2nix.mkPoetryApplication (builtins.removeAttrs args [ "editablePackageSources" ]);
# pass args specific to mkPoetryEnv and all remaining arguments to mkDerivation
editableEnv = stdenv.mkDerivation (
{
name = "editable-env";
src = poetry2nix.mkPoetryEnv {
inherit projectDir editablePackageSources overrides;
};
# copy all the output of mkPoetryEnv so that patching and wrapping of outputs works
installPhase = ''
mkdir -p $out
cp -a * $out
'';
} // builtins.removeAttrs args [ "projectDir" "editablePackageSources" "overrides" ]
);
in
app.overrideAttrs (super: {
passthru = super.passthru // { inherit editableEnv; };
})
#+end_src
** =poetry2nix= composes with =nix-shell= to get a development environment in =shell.nix=
:PROPERTIES:
:ID: arcology/nix/shell
:ROAM_ALIASES: arcology/shell.nix
:END:
This imports =myApp= from [[id:arcology/nix/poetry2nix][above]] so that it has a common =gcroot= -- nothing more annoying than opening a file in this project and having [[id:45fc2a02-fcd0-40c6-a29e-897c0ee7b1c7][direnv]] block while =pytest= runs. This is a [[id:ec508a23-6214-4b6b-998d-1c61d714efa8][fnord]], but it gets me to the point where I can use =direnv-mode= to have [[help:lsp-org][lsp-org]] to work with [[id:cce/python][pyright LSP Mode]].
#+begin_src nix -r :tangle shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
myApp = import ./default.nix { inherit pkgs; };
myAppEnv = myApp.editableEnv;
in pkgs.mkShell {
packages = [
myAppEnv
pkgs.pandoc
pkgs.poetry
# inotify-tools
];
}
#+end_src
This gets me to having a thing i can =nix-shell= in to and have dependencies available, where I can run =uvicorn arcology.server:app=
** =arcology-with-assets= can be bundled in to a simple Docker container with =dockerTools.buildImage=
:PROPERTIES:
:ID: arcology/nix/docker-image
:ROAM_ALIASES: arcology/docker.nix
:END:
#+begin_src nix :tangle docker.nix
{
arroyo ? import <arroyo> {},
emacsOverlay ? arroyo.lib.pkgVersions.emacsOverlay {},
pkgs ? import <nixpkgs> { overlays = [
(import emacsOverlay)
(import <arroyo/overlay.nix>)
];
},
python ? pkgs.python3
}:
let
app = import ./default.nix { inherit pkgs; inherit python; };
env = app.dependencyEnv;
myEmacs = (import /home/rrix/arroyo-nix/pkgs/emacs.nix { inherit pkgs; });
in pkgs.dockerTools.buildLayeredImage {
name = "arcology";
tag = "latest";
contents = [ app myEmacs pkgs.pandoc pkgs.coreutils ];
config = {
Env = [
"ARCOLOGY_DIRECTORY=/data"
"ARCOLOGY_SRC=/data/arcology-fastapi"
"ARROYO_SRC=/data/arroyo"
"ARCOLOGY_DB=/databases/arcology.db"
"ORG_ROAM_DB=/databases/org-roam.db"
"ARCOLOGY_ENV=prod"
];
Volumes = {
"/data"={};
"/databases"={};
};
WorkingDir = "${app}/lib/python${python.pythonVersion}/site-packages/";
ExposedPorts = {
"8000/tcp" = {};
};
# Cmd = ["${app}/bin/arcology-fastapi" ];
};
}
#+end_src
These dockerfiles modify the starting command -- i couldn't figure out how to do this with dockerTools' fromImage argument, the invocation was giving me some wonky error within poetry2nix?? daft.
#+begin_src dockerfile :tangle Dockerfile-inotify
FROM arcology:latest
CMD ["/bin/arcology-inotify" ]
#+end_src
#+begin_src dockerfile :tangle Dockerfile-fastapi
FROM arcology:latest
CMD ["/bin/arcology-fastapi" ]
#+end_src
Build these like so:
- base layered image [[shell:nix-build docker.nix && docker load -i result &]]
- fastapi command [[shell:docker build -f Dockerfile-fastapi -t docker.fontkeming.fail/arcology-fastapi . &]]
- inotify command [[shell:docker build -f Dockerfile-inotify -t docker.fontkeming.fail/arcology-inotify . &]]
Here's an all-in-one
#+begin_src shell :results drawer :tangle build.sh :shebang #!/run/current-system/sw/bin/env bash
set -e
nix-build docker.nix
docker load -i result
docker build -f Dockerfile-fastapi -t docker.fontkeming.fail/arcology-fastapi .
docker build -f Dockerfile-inotify -t docker.fontkeming.fail/arcology-inotify .
docker push docker.fontkeming.fail/arcology-fastapi
docker push docker.fontkeming.fail/arcology-inotify
ssh fontkeming "docker pull docker.fontkeming.fail/arcology-fastapi && docker pull docker.fontkeming.fail/arcology-inotify && sudo systemctl restart arcology-fastapi arcology-inotify"
#+end_src
(lol, sorry... i'll automate all this with [[id:20211120T220054.226284][The Wobserver]]'s [[id:c75d20e6-8888-4c5a-ac97-5997e2f1c711][NixOS]] port some day) *Ralphy voice i'm a systems engineer*
*** INPROGRESS This needs to load in [[id:arroyo/emacs][Arroyo Emacs]] built out of [[id:nix_community_emacs_overlay][nix-community/emacs-overlay]] somehow...
:LOGBOOK:
- State "INPROGRESS" from "NEXT" [2022-04-02 Sat 20:51]
:END:
Maybe easier to just rebuild the [[id:20211120T220054.226284][Wobserver]] run this on [[id:c75d20e6-8888-4c5a-ac97-5997e2f1c711][NixOS]] LMAO
*** DONE Environment configure BaseSettings
:LOGBOOK:
- State "DONE" from "NEXT" [2022-02-26 Sat 12:35]
:END:
*** DONE Volumes mount volumes
:LOGBOOK:
- State "DONE" from "NEXT" [2022-02-26 Sat 12:35]
:END:
*** DONE cmd -- need a wrapper which can either inotify or fastapi -- maybe split those in to different packages
:LOGBOOK:
- State "DONE" from "NEXT" [2022-02-26 Sat 12:35]
:END:
*** NEXT refactor all this bullshit overlay stuff lmao
** All of this can be encapsulated by a Nix Flake
[[https://nixos.wiki/wiki/Flakes][Flakes]] are the new hotness everyone in Nix land says you should use except it's unstable and may change out from underneath you but it can make it easy to distribute your package from JitHub so you should do it. At the very least =nix develop= is nicer than nix-shell in theory.
In the spirit of [[id:20220116T143655.499306][Hey Smell This]] I'll provide a flake that in theory you can invoke to run the Arcology on any system with Nix installed. Probably? it probably won't work since i relative-import [[id:cce/cce][CCE]] modules but in theory this should next pull a flake-ified version of [[id:arroyo/emacs][Arroyo Emacs]] in. some day. for now you get to keep the pieces and i get to run [[shell:nix run .#arcology-fastapi]] to start the project.
#+begin_src nix :tangle flake.nix :noweb yes
{
description = "arcology org-mode publishing";
inputs = {
nixpkgs.url = "github:nixos/nixpkgs";
flake-utils.url = "github:numtide/flake-utils";
};
outputs = {self, nixpkgs, flake-utils}:
flake-utils.lib.eachDefaultSystem (system:
let pkgs = nixpkgs.legacyPackages.${system}; in
rec {
devShell = import ./shell.nix { inherit pkgs; };
packages = flake-utils.lib.flattenTree {
arcology = import ./default.nix { inherit pkgs; };
docker = import ./docker.nix { inherit pkgs; };
};
defaultPackage = packages.arcology;
apps.arcology-fastapi = flake-utils.lib.mkApp {
drv = packages.arcology-with-assets;
exePath = "/bin/arcology-fastapi";
};
apps.arcology-inotify = flake-utils.lib.mkApp {
drv = packages.arcology-with-assets;
exePath = "/bin/arcology-inotify";
};
}
);
}
#+end_src
Everyone who wants to write maintainable flakes have to pull in a random 3rd party dependency first called [[https://github.com/numtide/flake-utils][flake-utils]].
I should noweb this but for now i'm just copying it in here directly, I donno if I even want to keep using this since my systems compose on the filesystem using [[id:cce/syncthing][Syncthing]].
* Deploying Arcology to [[id:c75d20e6-8888-4c5a-ac97-5997e2f1c711][NixOS]]
:PROPERTIES:
:ID: 20221128T194356.350889
:END:
Arcology is deployed to [[id:20211120T220054.226284][The Wobserver]] using [[id:arroyo/nixos][Arroyo NixOS Generator]], it uses the same Metadata extraction engine which powers the Arcology itself to generate a NixOS configuration file which will build my systems and deploy them through [[id:cce/morph][Morph]]. Deploying Arcology in this manner should not be so difficult, but I'm not so sure how to make this replicable by others -- of course the source can be pulled from [[roam:My Gitea Instance]] but I don't want to always be pushing code to remotes to iterate on the server...
#+begin_src nix :tangle ~/arroyo-nix/nixos/arcology.nix
{ config, lib, options, pkgs, ... }:
with lib;
let
cfg = config.services.arcology;
env = {
ARCOLOGY_ENV = cfg.environment;
ARCOLOGY_DIRECTORY = cfg.orgDir;
ARCOLOGY_SRC = "${cfg.orgDir}/arcology-fastapi";
ARROYO_SRC = "${cfg.orgDir}/arroyo";
ARROYO_EMACS = "${cfg.packages.emacs}/bin/emacs";
ARCOLOGY_DB = "${cfg.dataDir}/databases/arcology.db";
ORG_ROAM_DB = "${cfg.dataDir}/databases/org-roam.db";
};
domainVHosts = {
services.nginx.virtualHosts."${head cfg.domains}" = {
serverAliases = tail cfg.domains;
locations."/".proxyPass = "http://localhost:8000";
extraConfig = ''
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header Host $host;
'';
};
};
in {
options = {
services.arcology = {
packages.arcology = mkOption {
type = types.package;
default = pkgs.arcology;
description = mdDoc ''
'';
};
packages.emacs = mkOption {
type = types.package;
default = pkgs.arroyo-emacs;
};
domains = mkOption {
type = types.listOf types.string;
};
environment = mkOption {
type = types.enum ["prod" "dev"];
default = "prod";
};
dataDir = mkOption {
type = types.path;
default = "/var/lib/arcology";
description = mdDoc ''
Directory to store Arcology cache files, database, etc. Service User's home directory.
'';
};
orgDir = mkOption {
type = types.path;
description = mdDoc ''
Directory containing the org-mode documents.
Arcology needs read-only access to this directory.
'';
};
};
};
config = {
# fix this probably...
ids.uids.arcology = 900;
ids.gids.arcology = 900;
users.users.arcology = {
group = "arcology";
home = cfg.dataDir;
createHome = true;
shell = "${pkgs.bash}/bin/bash";
isSystemUser = true;
uid = config.ids.uids.arcology;
};
users.groups.arcology = {
gid = config.ids.gids.arcology;
};
systemd.services.arcology-web = {
description = "Arcology Web Site Engine's FastAPI web worker";
after = ["network.target" "arcology-inotify.service"];
wantedBy = ["multi-user.target"];
environment = env;
serviceConfig = {
Type = "simple";
User = "arcology";
Group = "arcology";
# include in pkg
# WorkingDirectory = "${cfg.packages.arcology}/lib/python${pkgs.python3.pythonVersion}/site-packages/";
WorkingDirectory = "${cfg.orgDir}/arcology-fastapi";
ExecStart = "${cfg.packages.arcology}/bin/arcology-fastapi";
Restart = "on-failure";
UMask = "0077";
# todo hardening
};
};
systemd.services.arcology-inotify = {
description = "Arcology Web Site Engine's indexing worker";
after = ["network.target"];
wantedBy = ["multi-user.target"];
environment = env;
serviceConfig = {
Type = "simple";
User = "arcology";
Group = "arcology";
# include in pkg
# WorkingDirectory = "${cfg.packages.arcology}/lib/python${pkgs.python3.pythonVersion}/site-packages/";
WorkingDirectory = "${cfg.orgDir}/arcology-fastapi";
ExecStart = "${cfg.packages.arcology}/bin/arcology-inotify";
Restart = "on-failure";
UMask = "0077";
# todo hardening
};
};
} // domainVHosts;
}
#+end_src
Using this is pretty simple:
#+ARROYO_NIXOS_MODULE: nixos/arcology-config.nix
#+ARROYO_NIXOS_ROLE: server
#+begin_src nix :tangle ~/arroyo-nix/nixos/arcology-config.nix
{ ... }:
{
imports = [ <arroyo/nixos/arcology.nix> ];
fileSystems."/media/org" = {
device = "/home/rrix/org";
options = ["bind"];
};
services.arcology = {
orgDir = "/media/org";
dataDir = "/srv/arcology";
domains = [
"engine.arcology.garden"
"arcology.garden"
"thelionsrear.com"
"cce.whatthefuck.computer" "cce.rix.si"
"doc.rix.si"
];
};
}
#+end_src
Reference in New Issue View Git Blame Copy Permalink