arcology-phx/setup.org

:PROPERTIES:
:ID:       20230302T132603.040866
:END:
#+TITLE: Arcology Phoenix Setup
#+filetags: :Project:

#+ARCOLOGY_KEY: arcology/phx/setup
#+ARCOLOGY_ALLOW_CRAWL: t

#+AUTO_TANGLE: t
=nixpkgs= has pretty decent support for Elixir projects with some good helpers. In 2021 I tried an earlier incarnation of [[id:1d917282-ecf4-4d4c-ba49-628cbb4bb8cc][The Arcology Project]] in Elixir+Phoenix, got it working on my laptop but when time came to use Nix to put together a Docker container I couldn't get =ecto_sqlite3= to build. Since then, I have shipped a fully function [[id:arcology/fastapi][Arcology FastAPI]] prototype, but I am still quite interested in shipping this on the BEAM. Elixir is cool, and the [[id:20220225T175638.482695][Arcology Public Router]] could "just" be a =Phoenix.Plug=. Now that I have a proper database layer in the [[id:arcology/arroyo-page][Arroyo Arcology Generator]] many of the design shortcomings of the original Arcology Phoenix prototype can be worked around. I'll be writing a Content-Addressed Store, too, I think, for caching the pages.

* =nix-shell= for base dependencies

Install these packages from =nixpkgs=:

#+begin_src nix :noweb-ref packages
elixir_1_14
elixir_ls
inotify-tools
mix2nix

# probably will need this.... 🥴
nodePackages.node2nix
#+end_src

Set these basic environment variables in the shell:

#+begin_src nix :noweb-ref environment
LANG="C.UTF-8";
ERL_AFLAGS="-kernel shell_history enabled";
#+end_src

Put dependency files in =$PWD/.nix-shell=:
(/This is taken from [[https://elixirforum.com/t/flake-nix-phoenix-postgresql/52622][elixir forum]]/)

#+begin_src nix :noweb-ref environment
NIX_SHELL_DIR="$PWD/.nix-shell";
MIX_HOME="$NIX_SHELL_DIR/.mix";
MIX_ARCHIVES="$MIX_HOME/archives";
HEX_HOME="$NIX_SHELL_DIR/.hex";
PATH="$HEX_HOME/bin:$MIX_HOME/escripts:$MIX_HOME/bin:$PATH";
LIVEBOOK_HOME="$PWD";
#+end_src

Install a small wrapper script which will install =hex= package manager and the Phoenix mix tasks; run [[shell:setup-mix-phx][setup-mix-phx]] the first time this project is set up.

#+begin_src shell :noweb-ref packages
(writeScriptBin "setup-mix-phx" ''
 ${elixir}/bin/mix local.hex
 ${elixir}/bin/mix local.rebar
 ${elixir}/bin/mix archive.install hex phx_new
 '')
#+end_src

Assemble all that in to a =pkgs.mkShell= for =nix-shell=

#+begin_src nix :noweb yes :tangle shell.nix
{ ... }:

let
  pkgs = import <nixpkgs> {};
  # import <arroyo> {};
in with pkgs; mkShell {
  packages = [
    <<packages>>
  ];

  shellHook = ''
    export <<environment>>

    <<shellHook>>

    ${elixir}/bin/mix --version
    ${elixir}/bin/iex --version
  '';
}
#+end_src

And set up [[id:45fc2a02-fcd0-40c6-a29e-897c0ee7b1c7][=direnv=]] (don't forget to [[shell:direnv allow][=direnv allow=]]):

#+begin_src shell :tangle .envrc
use nix
#+end_src

This doesn't set up [[id:cce/wobserver/postgres][PostgreSQL]] or anything like that -- we're in [[roam:Sqlite]] country.

* =nix build= for deploying the project

I had a fully functional Arcology Phoenix a number of years back but [[https://discourse.nixos.org/t/nix-elixir-and-dependencies-with-port-compilation/11208][wasn't able]] to get =ecto_sqlite3= to build in NixOS. Better start on with that now!

Start with a stub =default.nix= that sets up a =callPackage=:

#+begin_src nix :tangle default.nix
{ ... }:

let pkgs = import <nixpkgs> {};
in
  pkgs.callPackage ./arcology.nix {}
#+end_src

Then this thing can have its nix dependencies declared:

#+begin_src nix :tangle arcology.nix :noweb yes
{ lib, beamPackages, callPackage, ... }:

beamPackages.mixRelease rec {
  pname = "arcology";
  version = "0.0.1";

  src = ./.;

  dontStrip = true;

  <<override-mix-deps>>

  meta = with lib; {
    description = "Arcology Org-mode Web Engine";
    homepage = "https://engine.arcology.garden";
    license = licenses.unfree;
    maintainers = with maintainers; [ rrix ];
  };
}
#+end_src

Nix's mix deps are provided by =mix2nix=, run [[shell:mix2nix > mix.nix]] when the =mix.exs= deps are updated;

this basically works, except that =exqlite= which =ecto_sqlite3= uses as a database driver. It will try to write "something" to =XDG_CACHE_HOME= ... I'm not sure why I need to define this twice -- defining it in only one of the root Arcology =mixRelease= or =exqlite='s =mixRelease= will cause the build to fail...

#+begin_src nix :noweb-ref override-mix-deps
XDG_CACHE_HOME = "/tmp/elixir-cache";
mixNixDeps = import ./mix.nix {
  inherit beamPackages lib;
  overrides = (final: prev: {
    exqlite = prev.exqlite.overrideAttrs (pprev: pprev // {
      XDG_CACHE_HOME = "/tmp/elixir-cache";
    });
  });
};
#+end_src

* =mix.exs= project configuration

#+begin_src elixir :noweb-ref project
def project do
  [
    app: :arcology,
    version: "0.1.0",
    elixir: "~> 1.14",
    elixirc_paths: elixirc_paths(Mix.env()),
    start_permanent: Mix.env() == :prod,
    aliases: aliases(),
    deps: deps()
  ]
end
#+end_src

This instructs Mix to load [[roam:Arcology.Application]] as the entrypoint:

#+begin_src elixir :noweb-ref application
def application do
  [
    mod: {Arcology.Application, []},
    extra_applications: [:logger, :runtime_tools]
  ]
end
#+end_src

Specifies which paths to compile per environment.

#+begin_src elixir :noweb-ref paths

defp elixirc_paths(:test), do: ["lib", "test/support"]
defp elixirc_paths(_), do: ["lib"]
#+end_src

Here's what we depend on:

#+begin_src elixir :noweb-ref deps
defp deps do
  [
    {:phoenix, "~> 1.7.1"},
    {:phoenix_ecto, "~> 4.4"},
    {:ecto_sql, "~> 3.6"},
    {:ecto_sqlite3, ">= 0.0.0"},
    {:phoenix_html, "~> 3.3"},
    {:phoenix_live_reload, "~> 1.2", only: :dev},
    {:phoenix_live_view, "~> 0.18.16"},
    {:floki, ">= 0.30.0", only: :test},
    {:phoenix_live_dashboard, "~> 0.7.2"},
    {:esbuild, "~> 0.5", runtime: Mix.env() == :dev},
    {:tailwind, "~> 0.1.8", runtime: Mix.env() == :dev},
    {:swoosh, "~> 1.3"},
    {:finch, "~> 0.13"},
    {:telemetry_metrics, "~> 0.6"},
    {:telemetry_poller, "~> 1.0"},
    {:gettext, "~> 0.20"},
    {:jason, "~> 1.2"},
    {:plug_cowboy, "~> 2.5"}
  ]
end
#+end_src

Aliases are shortcuts or tasks specific to the current project.
For example, to install project dependencies and perform other setup tasks, run [[shell:mix setup]]. See [[https://hexdocs.pm/mix/1.12/Mix.html#module-aliases][the documentation]] for `Mix` for more info on aliases.

#+begin_src elixir :noweb-ref aliases
defp aliases do
  [
    setup: ["deps.get", "ecto.setup", "assets.setup", "assets.build"],
    "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
    "ecto.reset": ["ecto.drop", "ecto.setup"],
    test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"],
    "assets.setup": ["tailwind.install --if-missing", "esbuild.install --if-missing"],
    "assets.build": ["tailwind default", "esbuild default"],
    "assets.deploy": ["tailwind default --minify", "esbuild default --minify", "phx.digest"]
  ]
end
#+end_src

#+begin_src elixir :tangle mix.exs :noweb yes
defmodule Arcology.MixProject do
  use Mix.Project

  <<project>>

  <<application>>

  <<paths>>

  <<deps>>

  <<aliases>>
end
#+end_src

* Runtime Project Configuration

#+begin_src elixir :tangle config/config.exs
# This file is responsible for configuring your application
# and its dependencies with the aid of the Config module.
#
# This configuration file is loaded before any dependency and
# is restricted to this project.

# General application configuration
import Config
#+end_src

Tell Phoenix to load [[roam:Arcology.Repo]]

#+begin_src elixir :tangle config/config.exs
config :arcology,
  ecto_repos: [Arcology.Repo]
#+end_src

Configure [[id:20230303T104626.041201][=ArcologyWeb.Endpoint=]], instruct it to use =Arcology.PubSub= as its message-passing layer.

#+begin_src elixir :tangle config/config.exs
config :arcology, ArcologyWeb.Endpoint,
  url: [host: "localhost"],
  render_errors: [
    formats: [html: ArcologyWeb.ErrorHTML, json: ArcologyWeb.ErrorJSON],
    layout: false
  ],
  pubsub_server: Arcology.PubSub,
  live_view: [signing_salt: "OgcteMC0"]
#+end_src

Configures the mailer

By default it uses the "Local" adapter which stores the emails
locally. You can see the emails in your browser, at "/dev/mailbox".

For production it's recommended to configure a different adapter
at the `config/runtime.exs`.

#+begin_src elixir :tangle config/config.exs
config :arcology, Arcology.Mailer, adapter: Swoosh.Adapters.Local
#+end_src

Configure =esbuild= for Javascript packaging. Sure beats a webpack setup!! I hope!!!!

#+begin_src elixir :tangle config/config.exs
config :esbuild,
  version: "0.14.41",
  default: [
    args:
      ~w(js/app.js --bundle --target=es2017 --outdir=../priv/static/assets --external:/fonts/* --external:/images/*),
    cd: Path.expand("../assets", __DIR__),
    env: %{"NODE_PATH" => Path.expand("../deps", __DIR__)}
  ]
#+end_src

Configure Tailwind for components CSS; I might just ship the same 40 CSS rules I've been using for a while, though, tbh. This stuff is auto-generated:

#+begin_src elixir :tangle config/config.exs
# Configure tailwind (the version is required)
config :tailwind,
  version: "3.2.4",
  default: [
    args: ~w(
      --config=tailwind.config.js
      --input=css/app.css
      --output=../priv/static/assets/app.css
    ),
    cd: Path.expand("../assets", __DIR__)
  ]
#+end_src

Configures Elixir's Logger:

#+begin_src elixir :tangle config/config.exs
config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id]
#+end_src

Use [[https://github.com/michalmuskala/jason][Jason]] for JSON parsing in Phoenix; in the past I used Poison which require a NIF, but this seems like a good middle-ground between fast and cheap.

#+begin_src elixir :tangle config/config.exs
config :phoenix, :json_library, Jason
#+end_src

Import environment specific config. This must remain at the bottom of this file so it overrides the configuration defined above.

#+begin_src elixir :tangle config/config.exs
import_config "#{config_env()}.exs"
#+end_src

** Dev

This will override the code in the parent =config.exs=.

#+begin_src elixir :tangle config/dev.exs
import Config
#+end_src

In production this will read an environment variable, but in dev we can just use the local directory to store the [[id:arcology/arroyo-page][Arroyo Arcology Generator]]'s DB.

#+begin_src elixir :tangle config/dev.exs
config :arcology, Arcology.Repo,
  database: Path.expand("../arcology.db", Path.dirname(__ENV__.file)),
  pool_size: 5,
  stacktrace: true,
  show_sensitive_data_on_connection_error: true
#+end_src

Dev build will bind to [[http://localhost:4000]] and have debug helpers installed code reloading enabled for Elixir, CSS, and Javascript.

#+begin_src elixir :tangle config/dev.exs
config :arcology, ArcologyWeb.Endpoint,
  # Binding to loopback ipv4 address prevents access from other machines.
  # Change to `ip: {0, 0, 0, 0}` to allow access from other machines.
  http: [ip: {127, 0, 0, 1}, port: 4000],
  check_origin: false,
  code_reloader: true,
  debug_errors: true,
  secret_key_base: "PkJnuIpBpXQpUqGrc7ynUHYyXvAB40b03rjay/9WuKmzEN9H9GaELBdGOok2GQih",
  watchers: [
    esbuild: {Esbuild, :install_and_run, [:default, ~w(--sourcemap=inline --watch)]},
    tailwind: {Tailwind, :install_and_run, [:default, ~w(--watch)]}
  ]
#+end_src

Phoenix's LiveReload is pretty slick -- it will automatically reload your browser page if CSS, JS, or Elixir Web modules, etc are changed:

#+begin_src elixir :tangle config/dev.exs
config :arcology, ArcologyWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"priv/gettext/.*(po)$",
      ~r"lib/arcology_web/(controllers|live|components)/.*(ex|heex)$"
    ]
  ]
#+end_src

Enable dev routes for dashboard and mailbox

#+begin_src elixir :tangle config/dev.exs
config :arcology, dev_routes: true
#+end_src

Do not include metadata nor timestamps in development logs

#+begin_src elixir :tangle config/dev.exs
config :logger, :console, format: "[$level] $message\n"
#+end_src

Set a higher stacktrace during development. The auto-generated comment says "Avoid configuring such in production as building large stacktraces may be expensive."

#+begin_src elixir :tangle config/dev.exs
config :phoenix, :stacktrace_depth, 20
#+end_src

Initialize plugs at runtime for faster development compilation:

#+begin_src elixir :tangle config/dev.exs
config :phoenix, :plug_init_mode, :runtime
#+end_src

Disable [[https://github.com/swoosh/swoosh][swoosh]] api client as it is only required for production adapters, this is for the Mailer, and I probably don't care to set this up in prod, but yanno...

#+begin_src elixir :tangle config/dev.exs
config :swoosh, :api_client, false
#+end_src

** Test

I promise myself I'll figure out how to write tests for Arcology this time around.............

#+begin_src elixir :tangle config/test.exs
import Config

# Configure your database
#
# The MIX_TEST_PARTITION environment variable can be used
# to provide built-in test partitioning in CI environment.
# Run `mix help test` for more information.
config :arcology, Arcology.Repo,
  database: Path.expand("../arcology_test.db", Path.dirname(__ENV__.file)),
  pool_size: 5,
  pool: Ecto.Adapters.SQL.Sandbox

# We don't run a server during test. If one is required,
# you can enable the server option below.
config :arcology, ArcologyWeb.Endpoint,
  http: [ip: {127, 0, 0, 1}, port: 4002],
  secret_key_base: "4EDyvZHi6tCmgax6tx/v9e6L9kQmXt8qnhsqYE78k7P82uy71Ge7Zl6jPMolA+RR",
  server: false

# In test we don't send emails.
config :arcology, Arcology.Mailer, adapter: Swoosh.Adapters.Test

# Disable swoosh api client as it is only required for production adapters.
config :swoosh, :api_client, false

# Print only warnings and errors during test
config :logger, level: :warning

# Initialize plugs at runtime for faster test compilation
config :phoenix, :plug_init_mode, :runtime
#+end_src

** NEXT Prod

This is mostly overridden by the runtime configuration below.

#+begin_src elixir :tangle config/prod.exs
import Config

# For production, don't forget to configure the url host
# to something meaningful, Phoenix uses this information
# when generating URLs.

# Note we also include the path to a cache manifest
# containing the digested version of static files. This
# manifest is generated by the `mix phx.digest` task,
# which you should run after static files are built and
# before starting your production server.
config :arcology, ArcologyWeb.Endpoint, cache_static_manifest: "priv/static/cache_manifest.json"

# Configures Swoosh API Client
config :swoosh, api_client: Swoosh.ApiClient.Finch, finch_name: Arcology.Finch

# Do not print debug messages in production
config :logger, level: :info

# Runtime production configuration, including reading
# of environment variables, is done on config/runtime.exs.
#+end_src

** Runtime

These helpers are used to set up the process with environment variables at runtime. These will be important when it comes to deploying Arcology on [[id:20211120T220054.226284][the Wobserver]].

Only start the web server if =PHX_SERVER= is set; this is done automatically by the =bin/server= wrapper script installed by =mix phx.gen.release=.

#+begin_src elixir :tangle config/runtime.exs
if System.get_env("PHX_SERVER") do
  config :arcology, ArcologyWeb.Endpoint, server: true
end
#+end_src

Only =prod= is runtime configured:

#+begin_src elixir :tangle config/runtime.exs
if config_env() == :prod do
#+end_src

The database is pointed to with =DATABASE_PATH=:

#+begin_src elixir :tangle config/runtime.exs
  database_path =
    System.get_env("DATABASE_PATH") ||
      raise """
      environment variable DATABASE_PATH is missing.
      For example: /etc/arcology/arcology.db
      """
  
  config :arcology, Arcology.Repo,
    database: database_path,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "5")
#+end_src

The secret key base is used to sign/encrypt cookies and other secrets. A default value is used in config/dev.exs and config/test.exs but you want to use a different value for prod and you most likely don't want to check this value into version control, so we use an environment variable instead.

#+begin_src elixir :tangle config/runtime.exs
  secret_key_base =
    System.get_env("SECRET_KEY_BASE") ||
      raise """
      environment variable SECRET_KEY_BASE is missing.
      You can generate one by calling: mix phx.gen.secret
      """
#+end_src
=PHX_HOST= and =PORT= are used to configure the route/URL generation. The app is configured to listen only to the loopback IPv4 interface since [[id:e4998eda-d14a-48ee-9661-3d7d1bead53c][Nginx]] will be handling the actual edge and SSL and whatnot. Otherwise it would be configured according to [[https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html][Plug.Cowboy]] documentation.

#+begin_src elixir :tangle config/runtime.exs
  host = System.get_env("PHX_HOST") || "example.com"
  port = String.to_integer(System.get_env("PORT") || "4000")

  config :arcology, ArcologyWeb.Endpoint,
    url: [host: host, port: 443, scheme: "https"],
    http: [
      ip: {127, 0, 0, 1},
      port: port
    ],
    secret_key_base: secret_key_base

#+end_src

These aren't used right now:

#+begin_src elixir 
  # ## SSL Support
  #
  # To get SSL working, you will need to add the `https` key
  # to your endpoint configuration:
  #
  #     config :arcology, ArcologyWeb.Endpoint,
  #       https: [
  #         ...,
  #         port: 443,
  #         cipher_suite: :strong,
  #         keyfile: System.get_env("SOME_APP_SSL_KEY_PATH"),
  #         certfile: System.get_env("SOME_APP_SSL_CERT_PATH")
  #       ]
  #
  # The `cipher_suite` is set to `:strong` to support only the
  # latest and more secure SSL ciphers. This means old browsers
  # and clients may not be supported. You can set it to
  # `:compatible` for wider support.
  #
  # `:keyfile` and `:certfile` expect an absolute path to the key
  # and cert in disk or a relative path inside priv, for example
  # "priv/ssl/server.key". For all supported SSL configuration
  # options, see https://hexdocs.pm/plug/Plug.SSL.html#configure/1
  #
  # We also recommend setting `force_ssl` in your endpoint, ensuring
  # no data is ever sent via http, always redirecting to https:
  #
  #     config :arcology, ArcologyWeb.Endpoint,
  #       force_ssl: [hsts: true]
  #
  # Check `Plug.SSL` for all available options in `force_ssl`.

  # ## Configuring the mailer
  #
  # In production you need to configure the mailer to use a different adapter.
  # Also, you may need to configure the Swoosh API client of your choice if you
  # are not using SMTP. Here is an example of the configuration:
  #
  #     config :arcology, Arcology.Mailer,
  #       adapter: Swoosh.Adapters.Mailgun,
  #       api_key: System.get_env("MAILGUN_API_KEY"),
  #       domain: System.get_env("MAILGUN_DOMAIN")
  #
  # For this example you need include a HTTP client required by Swoosh API client.
  # Swoosh supports Hackney and Finch out of the box:
  #
  #     config :swoosh, :api_client, Swoosh.ApiClient.Hackney
  #
  # See https://hexdocs.pm/swoosh/Swoosh.html#module-installation for details.
#+end_src

#+begin_src elixir :tangle config/runtime.exs
end
#+end_src