arcology-elixir/phoenix.org

#+TITLE: Project Configuration for arcology
#+ROAM_ALIAS: "Arcology Phoenix" "Arcology .gitignore" "Arcology configuration"
#+ARCOLOGY_KEY: arcology/project
#+CREATED: [2020-09-22]
#+MODIFIED: [2020-11-12]
#+AUTO_TANGLE: t

This file describes the general Elixir and Phoenix plumbing required to get any Elixir application stood up. In general, when I work in a code module it should get moved in to its own file, but I don't know how to address that for these "infrastructure" elements. For now they'll live here. I also need to design some build systems and build scripts, and set up my =.gitignore=

* Project =.gitignore= file
:PROPERTIES:
:ID:       arcology/gitignore
:END:

This project is intended to be developed as a [[file:~/org/cce/literate_programming.org][Literate Programming]] [[file:~/org/org-mode.org][org-mode]] application, developed within my [[file:~/org/cce/org-roam.org][org-roam]] environment, published as part of my [[file:README.org][Arcology]] system. The project is its own subdirectory under my =org-roam-directory= and it can be on yours as well, or provided standalone.

The repo will contain docs and assets and a build script that will call up an [[file:../cce/emacs.org][Emacs]] that will "tangle" the files out and build the application. It will *not* contain the compiled code checked in, the docs must be considered the source of truth. We'll have facilities to "de-tangle" the source code eventually to aide contribution from folks who aren't running Emacs. For now, though, I'm gonna be a bastard 😄

This starts with a .gitignore. When updating this, the order of operation is to delete the file from the index, update this, tangle it, and then commit. it's a bit of a mess, maybe detangling is better. I gotta improve upon this quite a bit still, I would love to automatically generate the gitignore some time.

#+begin_src conf :tangle .gitignore
# elixir and mix outputs
/_build/
/cover/
/deps/
/doc/
/.fetch

# debug and dumps
erl_crash.dump
npm-debug.log

# Archives
,*.ez
arcology-*.tar

# The directory NPM downloads your dependencies sources to.
/assets/node_modules/
/priv/static/

# ansible roles
/roles/
systemd.service.j2
deploy.yml
#+end_src

Files which are provided from tangles will need to be managed here:

#+begin_src conf :tangle .gitignore
/mix.exs
# /mix.lock is checked in.
.formatter.exs
/config/*.exs
!/config/prod.secret.exs
/tmp/

# Arcology module
/lib/arcology.ex
/lib/arcology/*.ex
/lib/arcology/roam.ex
/lib/arcology/roam/*.ex
/lib/arcology/link_router/*.ex
/lib/arcology/link_router.ex

# ArcologyWeb module
/lib/arcology_web.ex
/lib/arcology_web/*.ex

# Channels, Controllers, Views, Templates
/lib/arcology_web/channels/
/lib/arcology_web/controllers/
/lib/arcology_web/views/
/lib/arcology_web/templates/
/lib/arcology_web/sigma/
/test/arcology_web/controllers/
/test/arcology_web/views/
/test/arcology/
/test/arcology_web/

/test/test_helper.exs
/test/support/

# mix tasks
/lib/mix/

/priv/lisp/arcology.db
/priv/lisp/arcology-batch.el

# Assets
/assets/js/app.js
/assets/js/foundation.js
/assets/js/sw.js
/assets/js/s1gma.js

# Build crap
/rel/artifacts/
/rel/config.exs
/rel/config/config.exs
/rel/vm.args.eex
/bin/build
/Dockerfile

# nixos crap
shell.nix
#+end_src

This file should be checked in when it's changed, along with =mix.lock=

* Mix Configuration ([[./mix.exs]])

Mix is the main tool interface for Elixir, an Elixir app almost always has a mix.exs in its root directory, and this is ours: it uses [[file:../noweb_syntax.org][noweb syntax]] to tangle the file from multiple source code blocks so that the documentation can be placed around each important segment.

#+begin_src elixir :tangle mix.exs :noweb tangle
defmodule Arcology.MixProject do
  use Mix.Project

  <<project-definition>>
  <<application-definition>>
  <<dependencies>>
  <<mix-aliases>>
  <<elixirc-paths>>
end
#+end_src

Basic Mix project configuration, appliation version, required elixir version, similar things are returned by =project/0=, this calls in to some other functions in the module. My [[file:../fedora_linux.org][Fedora Linux]] host has Elixir 1.5 installed on it now, I assume I'm going to have to update this sooner or later, though!

#+project_definition:
#+begin_src elixir :noweb-ref project-definition :noweb yes
def project do
  [
    app: :arcology,
    version: "0.1.1",
    elixir: "~> 1.7",
    elixirc_paths: elixirc_paths(Mix.env()),
    compilers: [:phoenix, :gettext] ++ Mix.compilers(),
    start_permanent: Mix.env() == :prod,
    aliases: aliases(),
    deps: deps(),
    releases: [
      <<releases>>
    ]
  ]
end
#+end_src

These releases are used by =mix release= and more details can by found in [[file:deploying.org][Deploying the Arcology]]:

#+begin_src elixir :noweb-ref releases
arcology: [
  include_executables_for: [:unix],
  steps: [
    :assemble,
    (fn rel ->
      File.cp_r("lisp", rel.path <> "/lisp")
      rel
    end),
    :tar
  ]
]
#+end_src

Configuration for the OTP application. Type =mix help compile.app= for more information. This is extended with support for =sqlite_ecto2= for the Arcology DB.

#+begin_src elixir :noweb-ref application-definition
  def application do
    [
      mod: {Arcology.Application, []},
      extra_applications: [:logger, :runtime_tools]
    ]
  end
#+end_src

The project dependencies are specified here, this is a simple Phoenix web app for now, we'll probably only need a few other things to ship this.

#+begin_src elixir :noweb-ref dependencies
defp deps do
  [
    {:sqlite_ecto2, "~> 2.2"},
    {:symbolic_expression, git: "https://github.com/rob-brown/SymbolicExpression", tag: "1.0.3"},
    {:panpipe, "~> 0.1"},
    {:phoenix, "~> 1.5"},
    {:phoenix_pubsub, "~> 2.0"},
    {:phoenix_pubsub_redis, "~> 3.0"},
    {:phoenix_html, "~> 2.14"},
    {:phoenix_live_reload, "~> 1.3", only: :dev},
    {:gettext, "~> 0.11"},
    {:jason, "~> 1.0"},
    {:plug_cowboy, "~> 2.0"},
    {:memoize, "~>1.3"},
    {:mix_test_watch, "~> 1.0", only: :dev, runtime: false},
    {:file_system, "~> 0.2"}
  ]
end
#+end_src


Aliases are shortcuts or tasks specific to the current project. For example, to create, migrate and run the seeds file at once: =mix ecto.setup=

#+begin_src elixir :noweb-ref mix-aliases
  defp aliases do
    [
      "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
      "ecto.reset": ["ecto.drop", "ecto.setup"],
      test: ["arcology.build", "test"]
    ]
  end
#+end_src

In =:test= environment

#+begin_src elixir :noweb-ref elixirc-paths
  defp elixirc_paths(:test), do: ["lib", "test/support"]
  defp elixirc_paths(_), do: ["lib"]
#+end_src

* Elixir Formatter Configurations

There are two =.formatter.exs= files, one in the project root, and one stored with the migrations. I'm not sure if this will be used by the org-babel programming, but I'll provide them for now so that I can =git clean=

#+begin_src elixir :tangle .formatter.exs
[
  import_deps: [:ecto, :phoenix],
  inputs: ["*.{ex,exs}", "priv/*/seeds.exs", "{config,lib,test}/**/*.{ex,exs}"],
  subdirectories: ["priv/*/migrations"]
]
#+end_src

This one is in =priv/repo/migrations/=

#+begin_src elixir :tangle priv/repo/migrations/.formatter.exs
[
  inputs: ["*.exs"]
]
#+end_src

* Runtime Configurations

This is *not* noweb-tangled since it's not wrapped in a =defmodule=, just tangled line-by-line and concatenated.

This file is responsible for configuring your application and its dependencies with the aid of the Mix.Config module:

#+begin_src elixir :tangle config/config.exs
use Mix.Config
#+end_src
=sqlite_ecto2= is the only repository I use, configure the Arcology DB to load from the DB.

#+begin_src elixir :tangle config/config.exs :noweb-ref arcology-repo-config
config :arcology, Arcology.Repo,
  adapter: Sqlite.Ecto2,
  database: System.get_env("ARCOLOGY_DATABASE") || "./tmp/arcology.db",
  pool: Ecto.Adapters.SQL.Sandbox

config :arcology, ecto_repos: [Arcology.Repo]
#+end_src

The Phoenix endpoint configuration is pretty straightforward, we use Phoenix's PostgreSQL PubSub adapter these days to have a durable message passing channel between frontend and backend. And this damned secret key will have to be provided for production builds somehow, that'll be fun. The only thing that is weird here is that Arcology.PubSub is disabled right now because I'm not using it directly, and don't want to take on a Redis or PG2 dependency just to have one lying about, even if my production systems de-facto have them there's no reason my endpoints and development environments need them running.

#+begin_src elixir :tangle config/config.exs
config :arcology, ArcologyWeb.Endpoint,
  url: [host: "localhost"],
  secret_key_base: "qjGyFdguDBSm8O7R4TI5M084mtgZ2vPE5p7a/2kTMuDCaPRUz9JiFdLK1RNUBggy",
  render_errors: [view: ArcologyWeb.ErrorView, accepts: ~w(html json)],
  pubsub_server: Arcology.PubSub
#+end_src

Logger configuration is simple, I haven't changed this, though I probably would like to have JSON logs some day.

#+begin_src elixir :tangle config/config.exs
config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id]
#+end_src

Use Jason for the Phoenix JSON library, this is also a default. I've used Poison in the past for personal projects, but I don't have anything resembling strong feelings.

#+begin_src elixir :tangle config/config.exs
config :phoenix, :json_library, Jason
#+end_src

My org-mode source files location for the renderer. Even though I have =~/org= symlinked on one of my hosts, the elisp code will/would store the actual path, and so I need to be able to switch across that. This will need to be replaced with a system environment variable closer to production deployment. These are used in [[file:arcology_db.org][Arcology DB]]

#+begin_src elixir :tangle config/config.exs :noweb-ref arcology-env
config :arcology, :env,
  database: System.get_env("ARCOLOGY_DATABASE") || "./tmp/arcology.db",
  org_roam_source: System.get_env("ORG_ROAM_SOURCE") || "~/Code/org-roam/",
  arcology_directory: System.get_env("ARCOLOGY_DIRECTORY") ||"/home/rrix/org"
#+end_src

Lastly, config.exs loads environment-specific configuration.

#+begin_src elixir :tangle config/config.exs
import_config "#{Mix.env()}.exs"
#+end_src

** Development Configuration

#+begin_src elixir :tangle config/dev.exs
use Mix.Config
#+end_src

The database configuration for local development is pretty simple, a local postgres instance is required, which makes it a bit heavy to do on older hardware, but this is an Elixir decision not mine. There is a sqlite for ecto2, but I only use that in the [[file:../arcology.org][Arcology]] project right now.

Disable the cache and enable debugging and hot code reloading, as well as setting up the webpack development server for assets.

#+begin_src elixir :tangle config/dev.exs
config :arcology, ArcologyWeb.Endpoint,
  http: [port: 4000],
  debug_errors: true,
  code_reloader: true,
  check_origin: false,
  watchers: [
    node: [
      "node_modules/webpack/bin/webpack.js",
      "--mode",
      "development",
      "--watch-stdin",
      cd: Path.expand("../assets", __DIR__)
    ]
  ]
#+end_src

Configure Phoenix's LiveReload functionality, which watches some file paths and will recompile and reload the browser page if any of the files change:

#+begin_src elixir :tangle config/dev.exs
config :arcology, ArcologyWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"priv/gettext/.*(po)$",
      ~r"lib/arcology_web/{live,views}/.*(ex)$",
      ~r"lib/arcology_web/templates/.*(eex)$"
    ]
  ]
#+end_src

In development we don't need timestamps or other metadata in the logs, and provide more stacktrace context when there is an error.

#+begin_src elixir :tangle config/dev.exs
config :logger, :console, format: "[$level] $message\n"
config :phoenix, :stacktrace_depth, 20
#+end_src

"Initialize plugs at runtime for faster development compilation", sure!

#+begin_src elixir :tangle config/dev.exs
config :phoenix, :plug_init_mode, :runtime
#+end_src

** Testing Configuration

Testing is pretty simple and similar to development, just less stuff going on. The server doesn't run, the logger is less loud, warnings and errors only. The test suite uses the project itself as the Arcology Directory

#+begin_src elixir :tangle config/test.exs
use Mix.Config

config :arcology, ArcologyWeb.Endpoint,
  http: [port: 4002],
  server: false

config :logger, level: :warn

config :arcology, :env,
  arcology_directory: File.cwd!,
  database: System.get_env("ARCOLOGY_DATABASE") || "./tmp/arcology-test.db"

config :arcology, Arcology.Repo,
  database: System.get_env("ARCOLOGY_DATABASE") || "./tmp/arcology-test.db"
#+end_src

** Production Configuration

#+begin_src elixir :tangle config/prod.exs
use Mix.Config
#+end_src

Note we also include the path to a cache manifest containing the digested version of static files. This manifest is generated by the `mix phx.digest` task, which you should run after static files are built and before starting your production server.

#+begin_src elixir :tangle config/prod.exs :noweb-ref arcology-endpoint
config :arcology, ArcologyWeb.Endpoint,
  http: [:inet6, port: System.get_env("ARCOLOGY_PORT") || 5000],
  url: [host: "dev.arcology.garden", port: System.get_env("ARCOLOGY_PORT")],
  cache_static_manifest: "priv/static/cache_manifest.json",
  server: true,
  root: "."
#+end_src

Don't print debug messages in production.

#+begin_src elixir :tangle config/prod.exs
config :logger, level: :info
#+end_src

Keeping this secrets file in sync is an [[file:../open_threads.org][open thread]].

#+begin_src elixir :tangle config/prod.exs
# import_config "prod.secret.exs"
#+end_src

When deployed, we'll probably do SSL termination in =nginx=, it's not so hard.

*** Runtime Configurable Elements repeated for [[file:deploying.org][Deploying the Arcology]] with =mix release=
:PROPERTIES:
:ID:       63c0724e-3065-42e4-8ced-80eccb526821
:END:

According to [[https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-runtime-configuration][mix release's documentation]], the =config.exs= above is evaluated at compile time, not runtime. I want to be able to specify runtime configuration as process environment injected in to the container at invocation, which I have set up to work for "local" configuration in the =System.get_env= calls above. To get these to load at runtime, this will configure the Distillery release to inject a smaller runtime-only =config.exs= which will be evaluated at runtime which uses [[file:../noweb_syntax.org][noweb syntax]] to transclude code blocks defined above.

#+begin_src elixir :noweb yes :tangle config/releases.exs :mkdirp yes :comments none
import Config

<<arcology-repo-config>>
<<arcology-env>>
<<arcology-endpoint>>
#+end_src

* Error Views and other base views

These are default, I'll change them some day.

#+begin_src elixir :tangle lib/arcology_web/views/error_view.ex
defmodule ArcologyWeb.ErrorView do
  use ArcologyWeb, :view

  # If you want to customize a particular status code
  # for a certain format, you may uncomment below.
  # def render("500.html", _assigns) do
  #   "Internal Server Error"
  # end

  # By default, Phoenix returns the status message from
  # the template name. For example, "404.html" becomes
  # "Not Found".
  def template_not_found(template, _assigns) do
    Phoenix.Controller.status_message_from_template(template)
  end
end
#+end_src

#+begin_src elixir :tangle lib/arcology_web/views/error_helpers.ex
defmodule ArcologyWeb.ErrorHelpers do
  @moduledoc """
  Conveniences for translating and building error messages.
  """

  use Phoenix.HTML

  @doc """
  Generates tag for inlined form input errors.
  """
  def error_tag(form, field) do
    Enum.map(Keyword.get_values(form.errors, field), fn error ->
      content_tag(:span, translate_error(error), class: "help-block")
    end)
  end

  @doc """
  Translates an error message using gettext.
  """
  def translate_error({msg, opts}) do
    # When using gettext, we typically pass the strings we want
    # to translate as a static argument:
    #
    #     # Translate "is invalid" in the "errors" domain
    #     dgettext("errors", "is invalid")
    #
    #     # Translate the number of files with plural rules
    #     dngettext("errors", "1 file", "%{count} files", count)
    #
    # Because the error messages we show in our forms and APIs
    # are defined inside Ecto, we need to translate them dynamically.
    # This requires us to call the Gettext module passing our gettext
    # backend as first argument.
    #
    # Note we use the "errors" domain, which means translations
    # should be written to the errors.po file. The :count option is
    # set by Ecto and indicates we should also apply plural rules.
    if count = opts[:count] do
      Gettext.dngettext(ArcologyWeb.Gettext, "errors", msg, msg, count, opts)
    else
      Gettext.dgettext(ArcologyWeb.Gettext, "errors", msg, opts)
    end
  end
end
#+end_src

#+begin_src elixir :tangle test/arcology_web/views/error_view_test.exs :mkdirp yes
defmodule ArcologyWeb.ErrorViewTest do
  use ArcologyWeb.ConnCase, async: true

  # Bring render/3 and render_to_string/3 for testing custom views
  import Phoenix.View

  test "renders 404.html" do
    assert render_to_string(ArcologyWeb.ErrorView, "404.html", []) == "Not Found"
  end

  test "renders 500.html" do
    assert render_to_string(ArcologyWeb.ErrorView, "500.html", []) == "Internal Server Error"
  end
end
#+end_src

IDK why I need to hold on to this one.

#+begin_src elixir :tangle lib/arcology_web/views/layout_view.ex :mkdirp yes
defmodule ArcologyWeb.LayoutView do
  use ArcologyWeb, :view
end
#+end_src