rrix
/
arcology-elixir
1
0
Fork 0
Code Releases Wiki Activity
arcology-elixir/deploying.org

10 KiB
Raw Permalink Blame History

Deploying the Arcology

,#+CCE_MODULE: arcology-deploy

I have a fair bit of existing automation infrastructure for managing my systems, and I plan to rely heavily on it for Arcology. Things will be a little bit … odd. First of all, rather than shipping a Dockerfile Arcology will generate Dynamic Ansible Bender Playbooks which will build containers using Ansible, Buildah.io, and Podman. Then, there will be some Ansible to deploy the container to my server, set up the SystemD services1, and make sure it's running. It'll all be sort of templated and weird and hard to reason about, and I intend to make that better over time – for now, the design of my new server system is in flux and getting a deployment out is higher priority.

I would like, eventually, to enable the hot-reloading features and other reliability systems the Erlang runtimes provide, but for now it's gonna be a simple container thing; I don't worry about downtime2, so systemd will just SIGTERM the container.

Quick list for deploying Arcology, these links can be clicked or executed with C-c C-o.

Distillery Config

This is mostly boilerplate generated by mix distillery.init, there is some added to the release to support Runtime Configurable Elements.

use Distillery.Releases.Config,
    default_release: :default,
    default_environment: Mix.env()

environment :prod do
  set include_erts: true
  set include_src: false
  set cookie: :"As1{bTJ;}(>HQZi=@6ln>r<{wG2RZIte(>`:;C:xK(rApd&u^xJ}PnITd{Q3M|W!"
  set vm_args: "rel/vm.args"
end

release :arcology do
  set version: current_version(:arcology)
  set applications: [
    :runtime_tools
  ]
  set config_providers: [
    {Distillery.Releases.Config.Providers.Elixir, ["${RELEASE_ROOT_DIR}/etc/config.exs"]}
  ]
  set overlays: [
    {:copy, "rel/config/config.exs", "etc/config.exs"}
  ]
end

the vm.args file is used to configure the low-level Erlang VM:

-name <%= release_name %>@127.0.0.1
-setcookie <%= release.profile.cookie %>
-smp auto

Distillery Releases

There is this shell script which creates a tarball for the release using Distillery:

set -e

APP_NAME="$(grep 'app:' mix.exs | sed -e 's/\[//g' -e 's/ //g' -e 's/app://' -e 's/[:,]//g')"
APP_VSN="$(grep 'version:' mix.exs | cut -d '"' -f2)"

mkdir -p ./rel/artifacts

# Install updated versions of hex/rebar
mix local.rebar --force
mix local.hex --if-missing --force

export MIX_ENV=prod

# Fetch deps and compile
mix deps.get
# Run an explicit clean to remove any build artifacts from the host
mix do clean, compile --force
# Digest assets
mix phx.digest
# Build the release
mix distillery.release
# Copy tarball to output
cp "_build/prod/rel/arcology/releases/$APP_VSN/arcology.tar.gz" rel/artifacts/"arcology-$APP_VSN.tar.gz"

Container build using Ansible Bender and Podman

This generates a dynamic playbook and then tells you how to run it, execute the body, then the link that it spits out. If any of the variables in this script are changed, make sure to update the shell link in the quick list at the top of the document. See Dynamic Ansible Bender Playbooks for the explanation of this dumb stupid contraption. :)

cat <<EOF | python ~/org/cce/make-container-playbook.py
template: ~/org/cce/containers/simple.yml
friendly: Arcology Site Engine
service_name: arcology
build_roles:
- arcology-build
build_reqs:
- elixir
cmd: /opt/arcology/bin/arcology foreground
task_tags:
- arcology
EOF
- name: deps
  dnf:
    state: installed
    name:
    - pandoc
    - emacs
    - inotify-tools

- name: target directory exists
  file:
    state: directory
    path: /opt/arcology

- name: explode in tarball
  unarchive:
    dest: /opt/arcology/
    src: ~/org/arcology/rel/artifacts/arcology-0.1.0.tar.gz

Container build with Nix

trying to get buildMix' to use a rebar3 that uses port compiler

self: super:

let
  packagesWith = erlang: let
    packages = super.beam.packagesWith erlang;
    rebar3 = packages.rebar3.overrideAttrs(oldAttrs: { compilePorts=true; });
  in packages // rec {
    inherit rebar3;
    buildMix'' = packages.buildMix'.override { inherit rebar3; } ;
  };
  packages = super.lib.attrsets.mapAttrs (name: _: packagesWith super.beam.interpreters.${name}) super.beam.packages;
in {
  beam = super.beam // {
    inherit packagesWith packages;
  };
}
{
  mixOverlay ? builtins.fetchGit {
     url = "https://github.com/hauleth/nix-elixir.git";
     rev = "8bdf6e73e3c5fa0706b2d1936bceb1baf692ed96";
   },
  rebarOverlay ? ./rebar-overlay.nix,
  pkgs ? import <nixpkgs> {
    overlays = [ (import mixOverlay) (import rebarOverlay) ];
  },

  version,
  installDebugTools ? false,
  src ? ./.,
  mixSha256 ? "0c19n0gn0xvhjg60dhsvdnvxk3kvdyf3pr12p8xzhb60jp17sw5z", 
  # mixSha256 ? pkgs.stdenv.lib.fakeSha256,
}:

let
  erlang = pkgs.beam.interpreters.erlangR23;
  packages = pkgs.beam.packagesWith erlang;

  elixir = packages.elixir_1_11;

  nodejs = pkgs.nodejs-10_x;
in packages.buildMix'' {
  name = "arcology";
  pname = "arcology";
  inherit version;
  inherit src;
  inherit mixSha256;

  buildInputs = [
    elixir
    nodejs

    # needed for distillery release invocation
    pkgs.bashInteractive
    pkgs.coreutils
    pkgs.gnused
    pkgs.gnugrep
    pkgs.gawk
    pkgs.zlib

    pkgs.gcc

    # libraries specified in autoPatchelfHook for the distillery package
    pkgs.openssl
    pkgs.systemd
    pkgs.ncurses

    pkgs.inotify-tools
    pkgs.pandoc
    pkgs.emacs-nox

    pkgs.emacs26Packages.use-package
    pkgs.emacs26Packages.elixir-mode
    pkgs.emacs26Packages.nix-mode
    pkgs.emacs26Packages.erlang
    pkgs.emacs26Packages.yaml-mode
  ];
  nativeBuildInputs = [
    pkgs.autoPatchelfHook 
    pkgs.gnumake
    pkgs.git
  ];
  preConfigure = ''
                 make tangle
                 echo "tangled"
                 '';
  buildPhase = ''
               mix compile --no-deps-check
               mix phx.digest
               mkdir -p $out/opt/arcology/
               mix release --path $out/opt/arcology
               '';
}
with import <nixpkgs> {};

pkgs.mkShell {
  buildInputs = [
    (import ./default.nix {
      version = "0.1.0";
    })
  ];
}
with import <nixpkgs> {};

let 
  httpPort = "4000";
  arcology = import ./default.nix {
    src = builtins.fetchGit {
      url = "https://code.rix.si/rrix/arcology.git";
      ref = "nix-docker-checkpoint";
      rev = "0a0d4b8cbbb4e53d175c8cae2e9301fdf2a90628";
    };
    version = "0.1.0";
  };
  installDebugTools = true;
in
dockerTools.buildImage {
  name = "arcology";
  contents = [ arcology ] ++ 
               pkgs.lib.lists.optional installDebugTools [
               pkgs.vim
               pkgs.file
               pkgs.strace
               pkgs.dcfldd
             ];
  created = "now";
  tag = "latest";
  config = {
    Cmd = [ "/opt/arcology/bin/arcology" "start" ];
    Env = [
      ("PATH=${pkgs.coreutils}/bin/"
       + ":${pkgs.bashInteractive}/bin/"
       + ":${pkgs.gnugrep}/bin/"
       + ":${pkgs.gnused}/bin/"
       + ":${pkgs.gawk}/bin/"
       + ":${pkgs.emacs}/bin/"
       + ":${pkgs.pandoc}/bin/"
       + ":${pkgs.strace}/bin/"
       + ":${pkgs.file}/bin/")
      "SHELL=${pkgs.bash}/bin/bash"
      "ARCOLOGY_DIRECTORY=/org"
      "ORG_ROAM_SOURCE=/org-roam"
      "ARCOLOGY_DATABASE=/arcology.db"
      "ARCOLOGY_PORT=${httpPort}"
    ];
    ExposedPorts = {
      "${httpPort}/tcp" = {};
    };
    Volumes = {
      "/org" = {};
      "/arcology.db" = {};
      "/org-roam" = {};
    };
  };
}

Footnotes

1

The systemd services use docker run to start the containers since I still use docker-ce on the server, to be aleviated eventually through work being done in CCE Server Design Thinking.

2

Personal Software Can Be Shitty, and my time is finite. I of course am not out there to make my site go down for weeks, but I don't need Site Reliability Engineering brainworms infecting all my work.