complete-computing-environment/morph-wrapper.org

Wrapping Morph commands for more ergonomic deployment

• Shell Environment and Pyproject manifest

After I tried setting up deploy-rs and it (and flakes) is kind of not very good for what I am doing with my computers yesterday, I found that I still want to simplify my deploy tooling to make it easier to ship updates of my Nix systems to their hosts.

a dogshit vince mcmahon meme i made showing the progression from morph, to flake experiments, and back to morph

I landed on a solution inspired by roam:Xe Iaso's hosts.toml setup. See hosts.toml ">Deploying from my hosts.toml for how this file is created, structured, and used in the morph commands themselves.

This page outlines a very simple script which ingests that hosts.toml file and provides a handful of options to make it easy for me to just specify hostnames and have the system figure out which manifest they should apply to and what to do with them.

We use toml and click and some builtins.

import toml
import click

import os
import socket

import subprocess

Short help options are good imo, customize that.

CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help'])

deploy-targets will print out the hostnames of all the hosts in the hosts.toml file suitable for using in arroyo-flood or so to interactively pick hostnames to deploy to.

Usage: deploy-targets [OPTIONS]

  print a list of all the hosts in the hosts.toml

Options:
  -f FILENAME  hosts.toml file path
  -h, --help   Show this message and exit.

@click.command(context_settings=CONTEXT_SETTINGS)
@click.option('-f', 'hosts_file',
              envvar="HOSTS_TOML", type=click.File('r'),
              help="hosts.toml file path",
              default="/home/rrix/arroyo-nix/networks/hosts.toml")
def list_hosts(hosts_file):
    """
    print a list of all the hosts in the hosts.toml
    """
    network = toml.load(hosts_file)
    for netname, host in get_all_hosts(network):
        print(host)

deploy does the thing. If you try to deploy to a host which cannot be pinged, it will be skipped. This doesn't read targetHost from the Morph network file, but those are lined up for me. -f will override this behavior, this is useful to me when I am bootstrapping a node.

Usage: deploy [OPTIONS] [HOSTS]...

  build or deploy one or more hosts

Options:
  -f FILENAME                     hosts.toml file path
  --all                           deploy to all hosts in the manifest
  --deploy / -b, --build          choose whether to deploy or just build.
  -a, --action TEXT               choose the deploy action
  -c, --confirm / -C, --no-confirm
                                  ask before running morph commands
  -h, --help                      Show this message and exit.

@click.command(context_settings=CONTEXT_SETTINGS)
@click.option('-f', 'hosts_file',
              envvar="HOSTS_TOML",
              help="hosts.toml file path",
              type=click.File('r'),
              default="/home/rrix/arroyo-nix/networks/hosts.toml")
@click.option('--all', 'deploy_all', is_flag=True, default=False,
              help="deploy to all hosts in the manifest")
@click.option(' /-b', '--deploy/--build', 'do_deploy',
              help="choose whether to deploy or just build.", is_flag=True, default=True)
@click.option('-a', '--action', 'deploy_action',
              help="choose the deploy action", default="switch")
@click.option('-c/-C', '--confirm/--no-confirm', 'confirm',
              help="ask before running morph commands", is_flag=True, default=False)
@click.option('-f', '--force/--no-force', 'force',
              help="Don't skip unavailable hosts", is_flag=True, default=False)
@click.argument('hosts', nargs=-1)
def wrap(hosts_file, deploy_all, hosts, do_deploy, deploy_action, confirm, force):
    """
    build or deploy one or more hosts
    """
    network = toml.load(hosts_file)

    if deploy_all:
        hosts = [h for net,h in get_all_hosts(network)]
    elif len(hosts) == 0:
        hosts = (socket.gethostname(),)

    subnets_to_deploy = get_pairs(network, hosts)

    for network, host in subnets_to_deploy:

        if do_deploy:
            if force or host_available(host):
                cmd = f"morph deploy --on={host} --passwd ~/arroyo-nix/networks/{network}.nix {deploy_action} --keep-result"
            else:
                click.echo(f"{host} is unavailable, skipping...")
                continue 
        else:
            cmd = f"morph build --on={host} ~/arroyo-nix/networks/{network}.nix --keep-result"

        click.echo(f"Prepared to run '{cmd}'")
        if confirm:
            input("or hit ctrl-c... ")
        os.system(cmd)

def host_available(hostname: str) -> bool:
    proc = subprocess.run(f"ping -w 1 -c 1 {hostname}", shell=True, capture_output=True)
    return proc.returncode == 0

And these ugly list-comprehensions help to munge the TOML file in to a form the python commands here would like to use.

def get_all_hosts(network):
    return [
        (name, host)
        for name, net in network.items()
        for host in net['hosts'].keys()
    ]

def get_pairs(network, hosts):
    return [
        (netname, hostname)
        for netname, subnet in network.items()
        for hostname in hosts
        if hostname in subnet['hosts'].keys()
    ]

This is made available to my system builds in my rixpkgs overlay, and added like this:

{ pkgs, ... }:

{
  environment.systemPackages = [ pkgs.morph-wrapper ];
}

So now when I make changes I can type deploy -b in to my nearest terminal to see my local system's build come together, then deploy to deploy it to this machine, then deploy $hostnames to deploy it to any number of my hostnames or deploy --all to deploy it everywhere.

Shell Environment and Pyproject manifest

This uses poetry and poetry2nix to generate a python application and nix derivation for use in my systems, a shell.nix is provided as well:

[tool.poetry]
name = "morph-wrapper"
version = "0.1.0"
description = ""
authors = ["Ryan Rix <code@whatthefuck.computer>"]

include = []

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = "^3.10"
toml = "^0.10.2"
click = "^8.1.3"

[tool.poetry.scripts]
deploy = 'morph_wrapper.wrapper:wrap'
deploy-targets = 'morph_wrapper.wrapper:list_hosts'

{ pkgs ? import <nixpkgs> {} }:
let
  python-with-my-packages = pkgs.python3.withPackages (p: with p; [
    toml
    click
  ]);
in
pkgs.mkShell {
  packages = [
    python-with-my-packages
    pkgs.poetry
  ];
}

{ pkgs ? import <nixpkgs> {},
  poetry2nix ? pkgs.poetry2nix,
  stdenv ? pkgs.stdenv,
  python ? pkgs.python3 }:

poetry2nix.mkPoetryApplication {
  inherit python;
  projectDir = ./.;
  propagatedBuildInputs = [];
}