rrix
/
complete-computing-environment
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
complete-computing-environment/nixos_automatic_partitionin...

592 lines
22 KiB
Org Mode
Raw Permalink Blame History

:PROPERTIES:
:ID: cce/nixos_automatic_partitioning_installer
:ROAM_ALIASES: justdoit.nix
:END:
#+TITLE: NixOS Automatic Partitioning Installer
#+filetags: :Project:CCE:
#+PROPERTY: header-args :mkdirp yes
#+ARCOLOGY_KEY: cce/nixos-installer
#+ARCOLOGY_ALLOW_CRAWL: t
There are a few "Supported" methods for installing NixOS: you can use the [[https://nixos.org/manual/nixos/stable/index.html#sec-installation][installer image]] to manually partition and prepare a system, you can [[https://nixos.org/manual/nixos/stable/index.html#sec-installing-from-other-distro][LUSTRATE]] an existing Linux installation with Nix installed on it, or you can use a [[https://github.com/cleverca22/nix-tests/tree/master/kexec][kexec script]] generally referred to as =justdoit= which is distributed as a script which boots the system in to a NixOS system with a partitioning script which blindly partitions the =rootDevice= and installs NixOS on to that partition. I spent some time this month trying to get the [[https://github.com/cleverca22/nix-tests/tree/master/kexec][kexec method]] working and it was pretty frustrating. The auto-partitioner and the easy setup through =kexec.justdoit= is nice, but getting the =kexec= working out of the box was frustrating[fn:1:nothing worth complaining about: fedora couldn't kexec the images, fedora couldn't expand the images without doing =TMPDIR=/var/tmp= because tmp is a ramdisk... my mongrel [[id:fedora_linux][Fedora Linux]] plus [[id:cce/home-manager][home-manager]] environment struggled to get virtualization running, and the testing cycle was pretty painful. The [[https://github.com/cleverca22/nix-tests/tree/master/kexec][repository]] I based this around was broken OOTB and even the testing scripts included in it didn't work reliably. The =justdoit= itself took a fair bit of tweaking to get to booting, too. I'll reintegrate kexec some day.], so for now I'll stick with ISO and ISO-as-SD with the =justdoit= work included.
To build an ISO image: [[shell:pushd ~/arroyo-nix/kexec && nix-build '<nixpkgs/nixos>' -A config.system.build.isoImage -I nixos-config=configuration.nix &]] The file will get symlinked to [[file:~/arroyo-nix/kexec/result/iso]].
* [[id:cce/my_nixos_configuration][My NixOS configuration]]
* Configuring the Installer
This record describes a basic NixOS installation which runs an installer script and can optionally be configured to automatically reboot (see below).
#+begin_src nix :tangle ~/arroyo-nix/kexec/configuration.nix :noweb yes
{ lib, pkgs, config, ... }:
let
overlayFn = import <arroyo/overlay.nix>;
pkgs' = overlayFn pkgs {};
in with lib;
{
imports = [
<nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix>
./kexec.nix ./justdoit.nix
];
nixpkgs.overlays = [
(overlayFn)
];
boot.supportedFilesystems = [ "zfs" ];
boot.loader.grub.enable = false;
boot.kernelParams = [
"console=ttyS0,115200" # allows certain forms of remote access, if the hardware is setup right
"panic=30" "boot.panic_on_fail" # reboot the machine upon fatal boot issues
];
systemd.services.sshd.wantedBy = mkForce [ "multi-user.target" ];
networking.hostName = "kexec";
# for the children
networking.wireless.enable = false;
networking.networkmanager.enable = true;
# hahaha! yes!
users.users.root.openssh.authorizedKeys.keys = pkgs'.lib.publicKeys.rrix;
hardware.video.hidpi.enable = true;
isoImage = {
includeSystemBuildDependencies = false;
makeUsbBootable = true;
};
kexec.justdoit = {
hostName = "terra-firma";
rootDevice = "/dev/sde";
swapSize = 16384;
poolName = "terra-firma";
bootType = "vfat";
bootSize = 1024;
luksEncrypt = false;
uefi = false;
nvme = false;
zfsPools = {
terra-firma = {
devices = "$ROOT_DEVICE";
volumes = {
root = {
snapshot = false;
compression = false;
mountPoint = "/";
};
nix = {
snapshot = false;
compression = false;
mountPoint = "/nix";
};
};
};
tank = {
devices = "mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd";
volumes = {
home = {
snapshot = true;
compression = "lz4";
mountPoint = "/home";
};
media = {
snapshot = true;
compression = "lz4";
mountPoint = "/media";
};
srv = {
snapshot = true;
compression = false;
mountPoint = "/srv";
};
};
};
};
};
}
#+end_src
That [[id:09779ac0-4d5f-40db-a340-49595c717e03][noweb]] call [[(gen_call)]] gets my public ssh key this way. i should and could and will define this in my [[id:cce/ssh_configuration][SSH Configuration]]:
#+NAME: get_ssh_pubkey
#+begin_src shell :results drawer -n
[ -f ~/.ssh/id_rsa.pub ] && \
cat ~/.ssh/id_rsa.pub \
| awk 'BEGIN {print "["} {print "\"" $1 " " $2 "\"" } END {print "];"}' \ (ref:tr)
| tr \\n " "
#+end_src
LMAO there must be a way to get [[(tr)]]'s awk output on to one line but ORS doesn't do what i expect...
** Minimal =target-config.nix=
A subset of [[id:cce/my_nixos_configuration][My NixOS configuration]], enough to get the rest [[id:cce/morph][Morph]] deployed to it. Head over there for in-depth discussion.
#+begin_src nix :tangle ~/arroyo-nix/kexec/target-config.nix :noweb yes
{ config, pkgs, ... }:
{
imports = [ ./hardware-configuration.nix ./generated.nix ];
# boot.loader.systemd-boot.enable = true;
boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
services.openssh.enable = true;
boot.zfs.forceImportRoot = true;
boot.kernelParams = [
"boot.shell_on_fail"
];
# less nix crap
nix.gc.automatic = true;
nix.gc.dates = "23:30";
# system clock
time.timeZone = "America/Los_Angeles";
users.groups.humans = {
name = "humans";
gid = 1000;
};
users.users.rrix = {
isNormalUser = true;
home = "/home/rrix";
description = "Ryan Rix";
extraGroups = [ "wheel" "networkmanager" "adbusers" ];
uid = 1000;
group = "humans";
initialPassword = "changeme!";
openssh.authorizedKeys.keys = <<get_ssh_pubkey()>>
};
users.users.root.openssh.authorizedKeys.keys = <<get_ssh_pubkey()>> # (ref:gen_call)
hardware.video.hidpi.enable = true;
# networking
networking.wireless.enable = false;
networking.networkmanager.enable = true;
# hahaha! yes
nixpkgs.config = { allowUnfree = true; };
# power management
powerManagement.enable = true;
environment.systemPackages = (with pkgs.libsForQt5; [
pkgs.vim
]);
}
#+end_src
*** NEXT Would be pretty neat to add a bootstrap script here...
open up a konsole and run =bootstrap= or =bootstrap-local= to get a backup restored on to the machine? =bootstrap-push= might be better but a bootstrap script is so alluring!
** =config.system.build.justdoit= formats and installs a system
:PROPERTIES:
:ID: nixos_justdoit
:END:
#+begin_src nix :tangle ~/arroyo-nix/kexec/justdoit.nix
{ config, pkgs, lib, ... }:
with lib;
let
cfg = config.kexec.justdoit;
x = if cfg.nvme then "p" else "";
in {
options = {
kexec.justdoit = {
hostName = mkOption {
type = types.str;
description = "set the networking.hostName of the installed system";
};
rootDevice = mkOption {
type = types.str;
default = "/dev/sda";
description = "the root block device that justdoit will nuke from orbit and force nixos onto";
};
bootSize = mkOption {
type = types.int;
default = 256;
description = "size of /boot in mb";
};
bootType = mkOption {
type = types.enum [ "ext4" "vfat" "zfs" ];
default = "ext4";
};
swapSize = mkOption {
type = types.int;
default = 1024;
description = "size of swap in mb";
};
poolName = mkOption {
type = types.str;
default = "tank";
description = "zfs pool name";
};
luksEncrypt = mkOption {
type = types.bool;
default = false;
description = "encrypt all of zfs and swap";
};
uefi = mkOption {
type = types.bool;
default = false;
description = "create a uefi install";
};
nvme = mkOption {
type = types.bool;
default = false;
description = "rootDevice is nvme";
};
zfsPools = mkOption {
type = types.attrs;
default = {
"${cfg.poolName}" = {
devices = "$ROOT_DEVICE";
volumes = {
nix = { mountPoint = "/nix"; };
root = { mountPoint = "/"; };
};
};
};
description = "Extra ZFS pools to create and mount";
example = {
kiddypool = {
devices = "$ROOT_DEVICE";
volumes = {
nix = {
snapshot = false;
compression = false;
mountPoint = "/nix";
};
root ={
snapshot = false;
compression = "lz4";
mountPoint = "/";
};
};
};
deepend = {
devices = "mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd";
volumes = {
home = {
snapshot = true;
compression = "lz4";
mountPoint = "/home";
};
media = {
snapshot = true;
compression = false;
mountPoint = "/media";
};
};
};
};
};
};
};
config = let
mkBootTable = {
ext4 = "mkfs.ext4 $NIXOS_BOOT -L NIXOS_BOOT";
vfat = "mkfs.vfat $NIXOS_BOOT -n NIXOS_BOOT";
zfs = "";
};
mkZfsCreateCmd =
(poolName: volName: vol:
"zfs create ${lib.optionalString (lib.isString vol.compression) "-o compression=${vol.compression}"} -o mountpoint=legacy ${poolName}/${volName}");
zfsPoolSetup = concatStringsSep "\n"
(lib.mapAttrsToList
(poolName: pool:
lib.concatStringsSep "\n"
(["zpool create -f -o altroot=/mnt/${poolName} ${poolName} ${pool.devices}"] ++
(lib.mapAttrsToList (mkZfsCreateCmd poolName) pool.volumes)))
cfg.zfsPools);
mkSnapshotCmd = (poolName: volName: vol:
"zfs set com.sun:auto-snapshot=${boolToString vol.snapshot} ${poolName}/${volName}");
zfsPoolSnapshotRules = concatStringsSep "\n"
(lib.mapAttrsToList
(poolName: pool:
lib.concatStringsSep "\n"
(["zfs set com.sun:auto-snapshot=false ${poolName}"] ++
(lib.mapAttrsToList (mkSnapshotCmd poolName) pool.volumes)))
cfg.zfsPools);
# mounts need to be sorted so that /mnt doesn't occlude /mnt/home etc later on...
mkMountCmd = (poolName: volName: vol:
{sortKey = "/mnt${vol.mountPoint}";
theCommand = "mkdir -p /mnt${vol.mountPoint} && mount -t zfs ${poolName}/${volName} /mnt${vol.mountPoint}";});
mkZfsPoolMountCommands = cmd:
(lists.toposort (p1: p2: hasPrefix p1.sortKey p2.sortKey)
(flatten
(mapAttrsToList
(poolName: pool:
(mapAttrsToList (cmd poolName) pool.volumes))
cfg.zfsPools)));
zfsPoolMountCommands =
concatStringsSep "\n"
(map (pair: pair.theCommand)
(mkZfsPoolMountCommands mkMountCmd).result);
mkUmountCmd = (poolName: volName: vol:
{sortKey = "/mnt${vol.mountPoint}";
theCommand = "umount /mnt${vol.mountPoint}";});
zfsPoolUmountCommands =
concatStringsSep "\n"
(reverseList
(map (pair: pair.theCommand)
(mkZfsPoolMountCommands mkUmountCmd).result));
in lib.mkIf true {
system.build.justdoit = pkgs.writeScriptBin "justdoit" ''
#!${pkgs.stdenv.shell}
set -e
vgchange -a n
wipefs -a ${cfg.rootDevice}
dd if=/dev/zero of=${cfg.rootDevice} bs=512 count=10000
sfdisk ${cfg.rootDevice} <<EOF
label: gpt
device: ${cfg.rootDevice}
unit: sectors
${lib.optionalString (cfg.bootType != "zfs") "1 : size=${toString (2048 * cfg.bootSize)}, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B"}
${lib.optionalString (! cfg.uefi) "4 : size=4096, type=21686148-6449-6E6F-744E-656564454649"}
2 : size=${toString (2048 * cfg.swapSize)}, type=0657FD6D-A4AB-43C4-84E5-0933C84B4F4F
3 : type=0FC63DAF-8483-4772-8E79-3D69D8477DE4
EOF
${if cfg.luksEncrypt then ''
cryptsetup luksFormat ${cfg.rootDevice}${x}2
cryptsetup open --type luks ${cfg.rootDevice}${x}2 swap
cryptsetup luksFormat ${cfg.rootDevice}${x}3
cryptsetup open --type luks ${cfg.rootDevice}${x}3 root
export ROOT_DEVICE=/dev/mapper/root
export SWAP_DEVICE=/dev/mapper/swap
'' else ''
export ROOT_DEVICE=${cfg.rootDevice}${x}3
export SWAP_DEVICE=${cfg.rootDevice}${x}2
''}
${lib.optionalString (cfg.bootType != "zfs") "export NIXOS_BOOT=${cfg.rootDevice}${x}1"}
${mkBootTable.${cfg.bootType}}
mkswap $SWAP_DEVICE -L NIXOS_SWAP
${lib.optionalString (cfg.zfsPools != {}) zfsPoolSetup}
${lib.optionalString (cfg.zfsPools != {}) zfsPoolSnapshotRules}
swapon $SWAP_DEVICE
${lib.optionalString (cfg.zfsPools != {}) zfsPoolMountCommands}
mkdir -p /mnt/boot
${lib.optionalString (cfg.bootType != "zfs") "mount $NIXOS_BOOT /mnt/boot/"}
nixos-generate-config --root /mnt/
hostId=$(echo $(head -c4 /dev/urandom | od -A none -t x4))
cp ${./target-config.nix} /mnt/etc/nixos/configuration.nix
cat > /mnt/etc/nixos/generated.nix <<EOF
{ ... }:
{
${lib.optionalString (cfg.hostName != "") "networking.hostName = \"${cfg.hostName}\";"}
${if cfg.uefi then ''
# boot.loader.grub.efiInstallAsRemovable = true;
# boot.loader.grub.efiSupport = true;
# boot.loader.grub.device = "nodev";
boot.loader.efi.canTouchEfiVariables = true;
boot.loader.systemd-boot.enable = true;
'' else ''
boot.loader.grub.enable = true;
# boot.loader.grub.device = "${cfg.rootDevice}";
boot.loader.grub.device = "/dev/sdf";
''}
networking.hostId = "$hostId"; # required for zfs use
${if cfg.luksEncrypt ''
boot.zfs.devNodes = "/dev/mapper"; # (ref:devNodes)
boot.initrd.luks.devices = {
"swap" = { name = "swap"; device = "${cfg.rootDevice}${x}2"; preLVM = true; };
"root" = { name = "root"; device = "${cfg.rootDevice}${x}3"; preLVM = true; };
};
'' else ''
boot.zfs.devNodes = "/dev/disk/by-uuid"; # (ref:devNodes)
''}
}
EOF
nix-channel --add https://github.com/nix-community/home-manager/archive/master.tar.gz home-manager
export NIX_PATH="home-manager=/nix/var/nix/profiles/per-user/root/channels/home-manager/:$NIX_PATH"
nix-channel --update
nixos-install
${lib.optionalString (cfg.bootType != "zfs") "umount /mnt/boot"}
${lib.optionalString (cfg.zfsPools != {}) zfsPoolUmountCommands}
zpool set cachefile=none ${cfg.poolName}
zpool export ${cfg.poolName}
swapoff $SWAP_DEVICE
'';
environment.systemPackages = [ config.system.build.justdoit ];
boot.supportedFilesystems = [ "zfs" ];
};
}
#+end_src
** =system.build.kexec_tarball= produces a =kexecutable= file
:PROPERTIES:
:ID: nixos_kexec_tarball
:END:
#+begin_src nix :tangle ~/arroyo-nix/kexec/kexec.nix
{ pkgs, config, ... }:
{
system.build = rec {
image = pkgs.runCommand "image" { buildInputs = [ pkgs.nukeReferences ]; } ''
mkdir $out
cp ${config.system.build.kernel}/bzImage $out/kernel
cp ${config.system.build.netbootRamdisk}/initrd $out/initrd
echo "init=${builtins.unsafeDiscardStringContext config.system.build.toplevel}/init ${toString config.boot.kernelParams}" > $out/cmdline
nuke-refs $out/kernel
'';
kexec_script = pkgs.writeTextFile {
executable = true;
name = "kexec-nixos";
text = ''
#!${pkgs.stdenv.shell}
export PATH=${pkgs.kexectools}/bin:${pkgs.cpio}/bin:$PATH
set -x
set -e
cd $(mktemp -d)
pwd
mkdir initrd
pushd initrd
if [ -e /ssh_pubkey ]; then
cat /ssh_pubkey >> authorized_keys
fi
find -type f | cpio -o -H newc | gzip -9 > ../extra.gz
popd
cat ${image}/initrd extra.gz > final.gz
kexec -l ${image}/kernel --initrd=final.gz --append="init=${builtins.unsafeDiscardStringContext config.system.build.toplevel}/init ${toString config.boot.kernelParams}"
sync
echo "executing kernel, filesystems will be improperly umounted"
kexec -e
'';
};
};
boot.initrd.postMountCommands = ''
mkdir -p /mnt-root/root/.ssh/
cp /authorized_keys /mnt-root/root/.ssh/
'';
system.build.kexec_tarball = pkgs.callPackage <nixpkgs/nixos/lib/make-system-tarball.nix> {
storeContents = [
{ object = config.system.build.kexec_script; symlink = "/kexec_nixos"; }
];
contents = [];
};
}
#+end_src
* Testing it :noexport:
#+begin_example
nix-build simple-test.nix -A luks_legacy
nix-build simple-test.nix -A luks_nvme
#... etc
#+end_example
#+begin_src nix :tangle ~/arroyo-nix/kexec/simple-test.nix
let
pkgs = import <nixpkgs> { config = {}; };
packages = with pkgs.lib; self: {
nvme = false;
uefi = false;
virtio = true;
configuration = {};
configuration1 = {
imports = [ ./configuration.nix self.configuration ];
};
config = (import <nixpkgs/nixos> { configuration = self.configuration1; }).config;
justdoit = self.config.system.build.justdoit;
image = self.config.system.build.image;
interface = if self.nvme then "none" else (if self.virtio then "virtio" else "scsi");
commonFlags = [
"-fw_cfg" "opt/com.angeldsis/simple-string,string=foobarbaz"
"-serial" "mon:stdio"
"-net" "user,hostfwd=tcp:127.0.0.2:2222-:22" "-net" "nic"
"-m" "2048"
"-drive" "index=0,id=drive1,file=dummy_root.qcow2,cache=writeback,werror=report,if=${self.interface}"
] ++ optional self.nvme "-device nvme,drive=drive1,serial=1234"
++ optional self.uefi "-drive if=pflash,format=raw,readonly,file=${pkgs.OVMF.fd}/FV/OVMF.fd -drive if=pflash,format=raw,file=my_uefi_vars.bin";
qemu_test1 = pkgs.writeScriptBin "qemu_test1" ''
#!${pkgs.stdenv.shell}
export PATH=${pkgs.qemu_kvm}/bin/:$PATH
if ! test -e dummy_root.qcow2; then
qemu-img create -f qcow2 dummy_root.qcow2 20G
fi
if ! test -e my_uefi_vars.bin; then
cp ${pkgs.OVMF.fd}/FV/OVMF_VARS.fd my_uefi_vars.bin
chmod +w my_uefi_vars.bin
fi
qemu-kvm -kernel ${self.image}/kernel -initrd ${self.image}/initrd \
-append "init=${builtins.unsafeDiscardStringContext self.config.system.build.toplevel}/init ${toString self.config.boot.kernelParams}" \
${toString self.commonFlags}
'';
qemu_test2 = pkgs.writeScriptBin "qemu_test2" ''
#!${pkgs.stdenv.shell}
export PATH=${pkgs.qemu_kvm}/bin/:$PATH
qemu-kvm ${toString self.commonFlags}
# -chardev stdio,id=qemu-debug-out -device isa-debugcon,chardev=qemu-debug-out
'';
# -debugcon file:debug.log -global isa-debugcon.iobase=0x402 \
qemu_test = pkgs.buildEnv {
name = "qemu_test";
paths = with self; [ qemu_test1 qemu_test2 ];
};
};
self = pkgs.lib.makeScope pkgs.newScope packages;
makeTest = { uefi ? false, nvme ? false, virtio ? false, luks ? false, bootType ? (if uefi then "vfat" else "ext4")}: let
pkgs2 = with pkgs.lib; self.overrideScope' (self: super: {
inherit uefi nvme virtio;
configuration = {
kexec.justdoit = {
rootDevice = mkForce (if nvme then "/dev/nvme0n1" else (if virtio then "/dev/vda" else "/dev/sda"));
nvme = mkForce nvme;
luksEncrypt = mkForce luks;
uefi = mkForce uefi;
inherit bootType;
};
};
});
in pkgs2.qemu_test // { justdoit = pkgs2.justdoit; };
in {
legacy_sata = makeTest {};
uefi_sata = makeTest { uefi = true; };
legacy_virtio = makeTest { virtio = true; };
nvme = makeTest { uefi = true; nvme = true; };
luks_legacy = makeTest { luks = true; virtio = true; };
virtio_no_boot = makeTest { virtio = true; bootType = "zfs"; };
luks_nvme = makeTest { luks = true; uefi = true; nvme = true; };
}
#+end_src
* Work Stream
** DONE get justdoit working
- State "DONE" from "INPROGRESS" [2021-04-03 Sat 02:20]
** NEXT include the full installation closure in to the image
** NEXT [[id:cce/home-manager][home-manager]]-alike for nixos configuration.nix generation (??)
** DONE basic nixops setup to get the boot configuration stuff parameterized
- State "DONE" from "WAITING" [2021-04-05 Mon 02:02]
- Note taken on [2021-04-02 Fri 21:53] \\
I'm not sure how dynamic configuration.nix generation will compose with nixops
- State "WAITING" from "NEXT" [2021-04-02 Fri 21:53]
** DONE rewrite from kexec to iso installation procedures
- State "DONE" from "NEXT" [2021-04-05 Mon 02:02]
** NEXT pull public key out of [[id:cce/ssh_configuration][SSH Configuration]]
** NEXT document my deviations from the upstream justdoit.nix
** NEXT bootstrap home-manager on [[id:cce/my_nixos_configuration][My NixOS configuration]]
** NEXT bootstrap wireguard
** NEXT bootstrap backup and restore
* Footnotes
[fn:1] nothing worth complaining about: fedora couldn't kexec the images, fedora couldn't expand the images without doing =TMPDIR=/var/tmp= because tmp is a ramdisk... my mongrel [[id:fedora_linux][Fedora Linux]] plus [[id:cce/home-manager][home-manager]] environment struggled to get virtualization running, and the testing cycle was pretty painful. The [[https://github.com/cleverca22/nix-tests/tree/master/kexec][repository]] I based this around was broken OOTB and even the testing scripts included in it didn't work reliably. The =justdoit= itself took a fair bit of tweaking to get to booting, too. I'll reintegrate kexec some day.
Reference in New Issue View Git Blame Copy Permalink