rrix
/
complete-computing-environment
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
complete-computing-environment/nixos_automatic_partitionin...

22 KiB
Raw Permalink Blame History

NixOS Automatic Partitioning Installer

There are a few "Supported" methods for installing NixOS: you can use the installer image to manually partition and prepare a system, you can LUSTRATE an existing Linux installation with Nix installed on it, or you can use a kexec script generally referred to as justdoit which is distributed as a script which boots the system in to a NixOS system with a partitioning script which blindly partitions the rootDevice and installs NixOS on to that partition. I spent some time this month trying to get the kexec method working and it was pretty frustrating. The auto-partitioner and the easy setup through kexec.justdoit is nice, but getting the kexec working out of the box was frustrating1[Fedora Linux]] plus home-manager environment struggled to get virtualization running, and the testing cycle was pretty painful. The repository I based this around was broken OOTB and even the testing scripts included in it didn't work reliably. The justdoit itself took a fair bit of tweaking to get to booting, too. I'll reintegrate kexec some day.], so for now I'll stick with ISO and ISO-as-SD with the justdoit work included.

To build an ISO image: shell:pushd ~/arroyo-nix/kexec && nix-build '<nixpkgs/nixos>&%2339;%20-A%20config.system.build.isoImage%20-I%20nixos-config=configuration.nix%20& The file will get symlinked to /rrix/complete-computing-environment/src/branch/main/~/arroyo-nix/kexec/result/iso.

My NixOS configuration

Configuring the Installer

This record describes a basic NixOS installation which runs an installer script and can optionally be configured to automatically reboot (see below).

{ lib, pkgs, config, ... }:

let
  overlayFn = import <arroyo/overlay.nix>;
  pkgs' = overlayFn pkgs {};
in with lib;
  {
    imports = [
      <nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix>
      ./kexec.nix ./justdoit.nix
    ];

    nixpkgs.overlays = [
      (overlayFn)
    ];

    boot.supportedFilesystems = [ "zfs" ];
    boot.loader.grub.enable = false;
    boot.kernelParams = [
      "console=ttyS0,115200"          # allows certain forms of remote access, if the hardware is setup right
      "panic=30" "boot.panic_on_fail" # reboot the machine upon fatal boot issues
    ];
    systemd.services.sshd.wantedBy = mkForce [ "multi-user.target" ];
    networking.hostName = "kexec";
    # for the children
    networking.wireless.enable = false;
    networking.networkmanager.enable = true; 
    # hahaha! yes!
    users.users.root.openssh.authorizedKeys.keys = pkgs'.lib.publicKeys.rrix;

    hardware.video.hidpi.enable = true;

    isoImage = {
      includeSystemBuildDependencies = false;
      makeUsbBootable = true;
    };

    kexec.justdoit = {
      hostName = "terra-firma";
      rootDevice = "/dev/sde"; 
      swapSize = 16384;
      poolName = "terra-firma";
      bootType = "vfat";
      bootSize = 1024;

      luksEncrypt = false;
      uefi = false;
      nvme = false;

      zfsPools = {
        terra-firma = {
          devices = "$ROOT_DEVICE";
          volumes = {
            root = {
              snapshot = false;
              compression = false;
              mountPoint = "/";
            };
            nix = {
              snapshot = false;
              compression = false;
              mountPoint = "/nix";
            };
          };
        };
        tank = {
          devices = "mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd";
          volumes = {
            home = {
              snapshot = true;
              compression = "lz4";
              mountPoint = "/home";
            };
            media = {
              snapshot = true;
              compression = "lz4";
              mountPoint = "/media";
            };
            srv = {
              snapshot = true;
              compression = false;
              mountPoint = "/srv";
            };
          };
        };
      };
    };
  }

That noweb call /rrix/complete-computing-environment/src/branch/main/(gen_call) gets my public ssh key this way. i should and could and will define this in my SSH Configuration:

[ -f ~/.ssh/id_rsa.pub ] && \
    cat ~/.ssh/id_rsa.pub \
        | awk 'BEGIN {print "["} {print "\"" $1 " " $2 "\"" } END {print "];"}' \        (ref:tr)
        | tr \\n " "

LMAO there must be a way to get /rrix/complete-computing-environment/src/branch/main/(tr)'s awk output on to one line but ORS doesn't do what i expect…

Minimal target-config.nix

A subset of My NixOS configuration, enough to get the rest Morph deployed to it. Head over there for in-depth discussion.

{ config, pkgs, ... }:

{
  imports = [ ./hardware-configuration.nix ./generated.nix ];

  # boot.loader.systemd-boot.enable = true;
  boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

  services.openssh.enable = true;

  boot.zfs.forceImportRoot = true;

  boot.kernelParams = [
    "boot.shell_on_fail"
  ];

  # less nix crap
  nix.gc.automatic = true;
  nix.gc.dates = "23:30";

  # system clock
  time.timeZone = "America/Los_Angeles";

  users.groups.humans = {
    name = "humans";
    gid = 1000;
  };
  users.users.rrix = {
    isNormalUser = true;
    home = "/home/rrix";
    description = "Ryan Rix";
    extraGroups = [ "wheel" "networkmanager" "adbusers" ];
    uid = 1000;
    group = "humans";
    initialPassword = "changeme!";
    openssh.authorizedKeys.keys = <<get_ssh_pubkey()>>
  };
  users.users.root.openssh.authorizedKeys.keys = <<get_ssh_pubkey()>>   # (ref:gen_call)

  hardware.video.hidpi.enable = true;

  # networking
  networking.wireless.enable = false;
  networking.networkmanager.enable = true;

  # hahaha! yes
  nixpkgs.config = { allowUnfree = true; };

  # power management
  powerManagement.enable = true;

  environment.systemPackages = (with pkgs.libsForQt5; [
    pkgs.vim
  ]);
}

NEXT Would be pretty neat to add a bootstrap script here…

open up a konsole and run bootstrap or bootstrap-local to get a backup restored on to the machine? bootstrap-push might be better but a bootstrap script is so alluring!

config.system.build.justdoit formats and installs a system 

{ config, pkgs, lib, ... }:

with lib;
let
  cfg = config.kexec.justdoit;
  x = if cfg.nvme then "p" else "";
in {
  options = {
    kexec.justdoit = {
      hostName = mkOption {
        type = types.str;
        description = "set the networking.hostName of the installed system";
      };
      rootDevice = mkOption {
        type = types.str;
        default = "/dev/sda";
        description = "the root block device that justdoit will nuke from orbit and force nixos onto";
      };
      bootSize = mkOption {
        type = types.int;
        default = 256;
        description = "size of /boot in mb";
      };
      bootType = mkOption {
        type = types.enum [ "ext4" "vfat" "zfs" ];
        default = "ext4";
      };
      swapSize = mkOption {
        type = types.int;
        default = 1024;
        description = "size of swap in mb";
      };
      poolName = mkOption {
        type = types.str;
        default = "tank";
        description = "zfs pool name";
      };
      luksEncrypt = mkOption {
        type = types.bool;
        default = false;
        description = "encrypt all of zfs and swap";
      };
      uefi = mkOption {
        type = types.bool;
        default = false;
        description = "create a uefi install";
      };
      nvme = mkOption {
        type = types.bool;
        default = false;
        description = "rootDevice is nvme";
      };
      zfsPools = mkOption {
        type = types.attrs;
        default = {
          "${cfg.poolName}" = {
            devices = "$ROOT_DEVICE";
            volumes = {
              nix = { mountPoint = "/nix"; };
              root = { mountPoint = "/"; };
            };
          };
        };
        description = "Extra ZFS pools to create and mount";
        example = {
          kiddypool = {
            devices = "$ROOT_DEVICE";
            volumes = {
              nix = {
                snapshot = false;
                compression = false;
                mountPoint = "/nix";
              };
              root ={
                snapshot = false;
                compression = "lz4";
                mountPoint = "/";
              };
            };
          };
          deepend = {
            devices = "mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd";
            volumes = {
              home = {
                snapshot = true;
                compression = "lz4";
                mountPoint = "/home";
              };
              media = {
                snapshot = true;
                compression = false;
                mountPoint = "/media";
              };
            };
          };
        };
      };
    };
  };
  config = let
    mkBootTable = {
      ext4 = "mkfs.ext4 $NIXOS_BOOT -L NIXOS_BOOT";
      vfat = "mkfs.vfat $NIXOS_BOOT -n NIXOS_BOOT";
      zfs = "";
    };

    mkZfsCreateCmd =
      (poolName: volName: vol:
        "zfs create ${lib.optionalString (lib.isString vol.compression) "-o compression=${vol.compression}"} -o mountpoint=legacy ${poolName}/${volName}");
    zfsPoolSetup = concatStringsSep "\n"
      (lib.mapAttrsToList 
        (poolName: pool:
          lib.concatStringsSep "\n"
            (["zpool create -f -o altroot=/mnt/${poolName} ${poolName} ${pool.devices}"] ++
             (lib.mapAttrsToList (mkZfsCreateCmd poolName) pool.volumes)))
        cfg.zfsPools);

    mkSnapshotCmd = (poolName: volName: vol:
      "zfs set com.sun:auto-snapshot=${boolToString vol.snapshot} ${poolName}/${volName}");
    zfsPoolSnapshotRules = concatStringsSep "\n" 
      (lib.mapAttrsToList 
        (poolName: pool:
          lib.concatStringsSep "\n"
            (["zfs set com.sun:auto-snapshot=false ${poolName}"] ++
             (lib.mapAttrsToList (mkSnapshotCmd poolName) pool.volumes)))
        cfg.zfsPools);

    # mounts need to be sorted so that /mnt doesn't occlude /mnt/home etc later on...
    mkMountCmd = (poolName: volName: vol:
      {sortKey = "/mnt${vol.mountPoint}";
       theCommand = "mkdir -p /mnt${vol.mountPoint} && mount -t zfs ${poolName}/${volName} /mnt${vol.mountPoint}";});
    mkZfsPoolMountCommands = cmd:
      (lists.toposort (p1: p2: hasPrefix p1.sortKey p2.sortKey)
        (flatten
          (mapAttrsToList 
            (poolName: pool:
              (mapAttrsToList (cmd poolName) pool.volumes))
            cfg.zfsPools)));
    zfsPoolMountCommands =
      concatStringsSep "\n" 
        (map (pair: pair.theCommand)
          (mkZfsPoolMountCommands mkMountCmd).result);

    mkUmountCmd = (poolName: volName: vol:
      {sortKey = "/mnt${vol.mountPoint}";
       theCommand = "umount /mnt${vol.mountPoint}";});
    zfsPoolUmountCommands =
      concatStringsSep "\n" 
        (reverseList
          (map (pair: pair.theCommand)
            (mkZfsPoolMountCommands mkUmountCmd).result));

  in lib.mkIf true {
    system.build.justdoit = pkgs.writeScriptBin "justdoit" ''
      #!${pkgs.stdenv.shell}
      set -e
      vgchange -a n
      wipefs -a ${cfg.rootDevice}
      dd if=/dev/zero of=${cfg.rootDevice} bs=512 count=10000
      sfdisk ${cfg.rootDevice} <<EOF
      label: gpt
      device: ${cfg.rootDevice}
      unit: sectors
      ${lib.optionalString (cfg.bootType != "zfs") "1 : size=${toString (2048 * cfg.bootSize)}, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B"}
      ${lib.optionalString (! cfg.uefi) "4 : size=4096, type=21686148-6449-6E6F-744E-656564454649"}
      2 : size=${toString (2048 * cfg.swapSize)}, type=0657FD6D-A4AB-43C4-84E5-0933C84B4F4F
      3 : type=0FC63DAF-8483-4772-8E79-3D69D8477DE4
      EOF
      ${if cfg.luksEncrypt then ''
        cryptsetup luksFormat ${cfg.rootDevice}${x}2
        cryptsetup open --type luks ${cfg.rootDevice}${x}2 swap
        cryptsetup luksFormat ${cfg.rootDevice}${x}3
        cryptsetup open --type luks ${cfg.rootDevice}${x}3 root
        export ROOT_DEVICE=/dev/mapper/root
        export SWAP_DEVICE=/dev/mapper/swap
      '' else ''
        export ROOT_DEVICE=${cfg.rootDevice}${x}3
        export SWAP_DEVICE=${cfg.rootDevice}${x}2
      ''}
      ${lib.optionalString (cfg.bootType != "zfs") "export NIXOS_BOOT=${cfg.rootDevice}${x}1"}
      ${mkBootTable.${cfg.bootType}}
      mkswap $SWAP_DEVICE -L NIXOS_SWAP
      ${lib.optionalString (cfg.zfsPools != {}) zfsPoolSetup}
      ${lib.optionalString (cfg.zfsPools != {}) zfsPoolSnapshotRules}
      swapon $SWAP_DEVICE
      ${lib.optionalString (cfg.zfsPools != {}) zfsPoolMountCommands}
      mkdir -p /mnt/boot
      ${lib.optionalString (cfg.bootType != "zfs") "mount $NIXOS_BOOT /mnt/boot/"}
      nixos-generate-config --root /mnt/
      hostId=$(echo $(head -c4 /dev/urandom | od -A none -t x4))
      cp ${./target-config.nix} /mnt/etc/nixos/configuration.nix
      cat > /mnt/etc/nixos/generated.nix <<EOF
      { ... }:
      {
        ${lib.optionalString (cfg.hostName != "") "networking.hostName = \"${cfg.hostName}\";"}
        ${if cfg.uefi then ''
          # boot.loader.grub.efiInstallAsRemovable = true;
          # boot.loader.grub.efiSupport = true;
          # boot.loader.grub.device = "nodev";
          boot.loader.efi.canTouchEfiVariables = true;
          boot.loader.systemd-boot.enable = true;
        '' else ''
          boot.loader.grub.enable = true;
          # boot.loader.grub.device = "${cfg.rootDevice}";
          boot.loader.grub.device = "/dev/sdf";
        ''}
        networking.hostId = "$hostId"; # required for zfs use
      ${if cfg.luksEncrypt ''
        boot.zfs.devNodes = "/dev/mapper";                          # (ref:devNodes)
        boot.initrd.luks.devices = {
          "swap" = { name = "swap"; device = "${cfg.rootDevice}${x}2"; preLVM = true; };
          "root" = { name = "root"; device = "${cfg.rootDevice}${x}3"; preLVM = true; };
      };
      '' else ''
        boot.zfs.devNodes = "/dev/disk/by-uuid";                          # (ref:devNodes)
      ''}
      }
      EOF
      nix-channel --add https://github.com/nix-community/home-manager/archive/master.tar.gz home-manager
      export NIX_PATH="home-manager=/nix/var/nix/profiles/per-user/root/channels/home-manager/:$NIX_PATH"
      nix-channel --update
      nixos-install
      ${lib.optionalString (cfg.bootType != "zfs") "umount /mnt/boot"}
      ${lib.optionalString (cfg.zfsPools != {}) zfsPoolUmountCommands}
      zpool set cachefile=none ${cfg.poolName}
      zpool export ${cfg.poolName}
      swapoff $SWAP_DEVICE
    '';
    environment.systemPackages = [ config.system.build.justdoit ];
    boot.supportedFilesystems = [ "zfs" ];
  };
}

system.build.kexec_tarball produces a kexecutable file 

{ pkgs, config, ... }:

{
  system.build = rec {
    image = pkgs.runCommand "image" { buildInputs = [ pkgs.nukeReferences ]; } ''
      mkdir $out
      cp ${config.system.build.kernel}/bzImage $out/kernel
      cp ${config.system.build.netbootRamdisk}/initrd $out/initrd
      echo "init=${builtins.unsafeDiscardStringContext config.system.build.toplevel}/init ${toString config.boot.kernelParams}" > $out/cmdline
      nuke-refs $out/kernel
    '';
    kexec_script = pkgs.writeTextFile {
      executable = true;
      name = "kexec-nixos";
      text = ''
        #!${pkgs.stdenv.shell}
        export PATH=${pkgs.kexectools}/bin:${pkgs.cpio}/bin:$PATH
        set -x
        set -e
        cd $(mktemp -d)
        pwd
        mkdir initrd
        pushd initrd
        if [ -e /ssh_pubkey ]; then
          cat /ssh_pubkey >> authorized_keys
        fi
        find -type f | cpio -o -H newc | gzip -9 > ../extra.gz
        popd
        cat ${image}/initrd extra.gz > final.gz
        kexec -l ${image}/kernel --initrd=final.gz --append="init=${builtins.unsafeDiscardStringContext config.system.build.toplevel}/init ${toString config.boot.kernelParams}"
        sync
        echo "executing kernel, filesystems will be improperly umounted"
        kexec -e
        '';
    };
  };
  boot.initrd.postMountCommands = ''
    mkdir -p /mnt-root/root/.ssh/
    cp /authorized_keys /mnt-root/root/.ssh/
  '';
  system.build.kexec_tarball = pkgs.callPackage <nixpkgs/nixos/lib/make-system-tarball.nix> {
    storeContents = [
      { object = config.system.build.kexec_script; symlink = "/kexec_nixos"; }
    ];
    contents = [];
  };
}

Work Stream

DONE get justdoit working

  • State "DONE" from "INPROGRESS" [2021-04-03 Sat 02:20]

NEXT include the full installation closure in to the image

NEXT home-manager-alike for nixos configuration.nix generation (??)

DONE basic nixops setup to get the boot configuration stuff parameterized

  • State "DONE" from "WAITING" [2021-04-05 Mon 02:02]
  • Note taken on [2021-04-02 Fri 21:53]
    I'm not sure how dynamic configuration.nix generation will compose with nixops
  • State "WAITING" from "NEXT" [2021-04-02 Fri 21:53]

DONE rewrite from kexec to iso installation procedures

  • State "DONE" from "NEXT" [2021-04-05 Mon 02:02]

NEXT pull public key out of SSH Configuration

NEXT document my deviations from the upstream justdoit.nix

NEXT bootstrap home-manager on My NixOS configuration

NEXT bootstrap wireguard

NEXT bootstrap backup and restore

Footnotes

1

nothing worth complaining about: fedora couldn't kexec the images, fedora couldn't expand the images without doing TMPDIR=/var/tmp because tmp is a ramdisk… my mongrel Fedora Linux plus home-manager environment struggled to get virtualization running, and the testing cycle was pretty painful. The repository I based this around was broken OOTB and even the testing scripts included in it didn't work reliably. The justdoit itself took a fair bit of tweaking to get to booting, too. I'll reintegrate kexec some day.