Small improvements to make setup process easier for OS contributors (#1171)

* Remove _build folders for umbrella apps

For umbrella apps everything goes into /_build directory so there no need to ignore directories that should never be created

* Change mix aliases to be more aligned with what OS community would expect

1. We want ecto.create and ecto.migrate to be run on each tests, this will simplify setup steps (no need to run migrations manually)

2. ecto.remigrate is not needed because now you can just run ecto.drop and on tests migrations would be executed anyways.

* Rename docker-compose step name in CONTRIBUTING.md

The step was renamed here: dd67baf629 (diff-67a4805fdcc6145d7b3ada2a6099a9b2e91c9d0fd108c22f95d2f01d219793d1R10)

* Remove .devcontainer

This an is opinionated change. Right now devcontainer doesn't work but should be easy to fix (with renaming step name), but at the same time it forces developers that use VS code to have unified development environment (including plugins for the editor itself).

I feel like it's not a good path to go for OS and for small team - everyone should be allowed to use setup they like. Especially for people like me that tend to recompile ls-elixir for Elixir plugin from master branch.

Plus it's yet another thing to maintain while nobody on the team is using it, which means it will be always causing issues.

* Make fz_http mix.exs aliases aligned with umbrella app ones

* Redirect stderr to stdout in a command called from dev.exs

Otherwise I'm getting this on my MacOS (that has a `route` implementation that doesn't show interfaces) when `mix phx.server` is executed:
```
usage: route [-dnqtv] command [[modifiers] args]
```

* Fix race condition due to static device field values

Both public_key and name are unique and we should not use static values for field covered by unique index, otherwise deadlocks and slow tests are expected.

* Remove unwanted transaction block

The changeset code doesn't have any code that accesses the database and individual Ecto.SQL commands are already wrapped in transactions by default, so there is no need to start it manually and hold for longer than expected (while irrelevant Elixir code is running).

* Use netstat to identify egress interface on MacOS

* Rename uninstall.sh to omnibus-uninstall.sh

* Fix uninstall path in omnibus_build.yml

.devcontainer/.gitignore

@@ -1 +0,0 @@
-.tool-versions

.devcontainer/Dockerfile

@@ -1,121 +0,0 @@
-FROM ubuntu:bionic AS dev-base
-
-ARG USER_UID=1000
-ARG USER_GID=$USER_UID
-ARG USERNAME=vscode
-
-ENV DEBIAN_FRONTEND=noninteractive
-
-RUN echo "APT::Install-Recommends 0;" >> /etc/apt/apt.conf.d/01norecommends \
-  && echo "APT::Install-Suggests 0;" >> /etc/apt/apt.conf.d/01norecommends \
-  && apt update \
-  && apt upgrade -y \
-  && apt install -y \
-    apt-utils \
-    dialog \
-    wget \
-    net-tools \
-    wireguard \
-    nftables \
-    inotify-tools \
-    ca-certificates \
-    build-essential \
-    less \
-    automake \
-    autoconf \
-    libreadline-dev \
-    libncurses-dev \
-    libssl-dev \
-    libyaml-dev \
-    libxslt-dev \
-    libffi-dev \
-    libtool \
-    zlib1g-dev \
-    unixodbc-dev \
-    unzip \
-    curl \
-    git \
-    vim \
-    sudo \
-    bsdmainutils \
-    gpg \
-    dirmngr \
-    jq \
-    locales \
-  # Clean up
-  && apt-get autoremove -y \
-  && apt-get clean -y \
-  && rm -rf /var/lib/apt/lists/* \
-  && groupadd --gid $USER_GID $USERNAME \
-  && useradd --uid $USER_GID --gid $USERNAME --shell /bin/bash --create-home --groups sudo $USERNAME \
-  && echo "%${USERNAME}    ALL=(ALL)   NOPASSWD:    ALL" >> /etc/sudoers.d/${USERNAME} \
-  && mkdir -p /home/$USERNAME/.vscode-server/extensions \
-      /home/$USERNAME/.vscode-server-insiders/extensions \
-  && chown -R $USERNAME \
-      /home/$USERNAME/.vscode-server \
-      /home/$USERNAME/.vscode-server-insiders \
-  && mkdir -p /workspace \
-  && chown -R $USERNAME /workspace
-
-# Set the locale to en_US.UTF-8 and TZ to UTC
-RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
-    locale-gen
-ENV LANG en_US.UTF-8
-ENV LANGUAGE en_US:en
-ENV LC_ALL en_US.UTF-8
-ENV TZ 'UTC'
-
-ENV DEBIAN_FRONTEND=dialog
-
-FROM dev-base AS setup-asdf
-
-ARG USERNAME=vscode
-
-RUN git clone https://github.com/asdf-vm/asdf.git /opt/asdf \
-  && cd /opt/asdf \
-  && git checkout "$(git describe --abbrev=0 --tags)" \
-  && mkdir /opt/asdf-data \
-  && chown -R $USERNAME /opt/asdf-data
-
-FROM setup-asdf AS dev-env
-
-ARG USERNAME=vscode
-
-USER $USERNAME
-WORKDIR /home/$USERNAME
-ENV HOME=/home/$USERNAME \
-  ASDF_DIR="/opt/asdf" \
-  ASDF_DATA_DIR="/opt/asdf-data"
-
-RUN echo '\n. /opt/asdf/asdf.sh' >> ~/.bashrc \
-    && echo '\n. /opt/asdf/completions/asdf.bash' >> ~/.bashrc
-
-SHELL ["/bin/bash", "-ic"]
-
-RUN asdf plugin-add erlang https://github.com/asdf-vm/asdf-erlang.git && \
-    asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git && \
-    asdf plugin-add nodejs https://github.com/asdf-vm/asdf-nodejs.git && \
-    asdf plugin-add python https://github.com/danhper/asdf-python.git && \
-    asdf plugin-add ruby https://github.com/asdf-vm/asdf-ruby.git
-
-# Need global erlang version to install Elixir -     https://github.com/asdf-vm/asdf-elixir/issues/113
-COPY .tool-versions /home/vscode/.tool-versions
-
-# No order to asdf, so elixir error without Erlang
-RUN asdf install erlang && asdf install
-
-RUN mix local.hex --force
-RUN mix local.rebar --force
-
-VOLUME ["${ASDF_DATA_DIR}"]
-
-# Pre-commit install with deps
-RUN pip install setuptools wheel
-RUN pip install pre-commit
-RUN gem install rubocop
-
-ENV PATH=${HOME}/.local/bin:/opt/asdf-data/shims:/opt/asdf/bin:${PATH}
-
-RUN git config --global --add safe.directory /workspace
-
-CMD ["/bin/bash"]

.devcontainer/README.md

@@ -1,6 +0,0 @@
-# DevContainer
-
-Files in this directory are used exclusively for VS Code / Github Codespaces.
-
-For a general overview of how to run Firezone in Docker locally, see [our
-contributing guide](../CONTRIBUTING.md).

.devcontainer/devcontainer.json

@@ -1,30 +0,0 @@
-// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
-// https://github.com/microsoft/vscode-dev-containers/tree/v0.231.6/containers/elixir-phoenix-postgres
-{
-	"name": "Firezone",
-	"dockerComposeFile": "docker-compose.yml",
-	"service": "elixir",
-	"workspaceFolder": "/workspace",
-
-	// Set *default* container specific settings.json values on container create.
-	"settings": {},
-
-	"initializeCommand": [".devcontainer/init"],
-
-	// Add the IDs of extensions you want installed when the container is created.
-	"extensions": [
-		"jakebecker.elixir-ls",
-		"phoenixframework.phoenix",
-		"rebornix.ruby"
-	],
-
-	// Use 'forwardPorts' to make a list of ports inside the container available locally.
-	// This can be used to network with other containers or with the host.
-	"forwardPorts": [80, 443, 4000, 5432],
-
-	// Use 'postCreateCommand' to run commands after the container is created.
-	// "postCreateCommand": "mix deps.get"
-
-	// Uncomment to connect as a non-root user. See https://aka.ms/vscode-remote/containers/non-root.
-	"remoteUser": "vscode"
-}

.devcontainer/docker-compose.yml

@@ -1,45 +0,0 @@
-version: "3.8"
-
-services:
-  caddy:
-    image: caddy:2
-    volumes:
-      - ./Caddyfile:/etc/caddy/Caddyfile
-    network_mode: service:firezone
-
-  firezone:
-    build:
-      context: .
-      dockerfile: Dockerfile
-      args:
-        # Elixir Version: 1.9, 1.10, 1.10.4, ...
-        VARIANT: "1.14.0"
-        # Phoenix Version: 1.4.17, 1.5.4, ...
-        PHOENIX_VERSION: "1.6.12"
-        # Node Version: 12, 14, ...
-        NODE_VERSION: "16"
-        RUBY_VERSION: "2.7.6"
-
-    volumes:
-      - ..:/workspace:cached
-    # Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
-    network_mode: service:postgres
-
-    environment:
-      LOCAL_AUTH_ENABLED: 'true'
-
-    # Overrides default command so things don't shut down after the process ends.
-    command: sleep infinity
-
-  postgres:
-    image: postgres:13
-    restart: unless-stopped
-    volumes:
-      - postgres-data:/var/lib/postgresql/data
-    environment:
-      POSTGRES_USER: postgres
-      POSTGRES_PASSWORD: postgres
-      POSTGRES_DB: firezone_dev
-
-volumes:
-  postgres-data:

.devcontainer/init

@@ -1,3 +0,0 @@
-#!/bin/sh
-
-cp .tool-versions .devcontainer/.tool-versions

.devcontainer/init.cmd

@@ -1 +0,0 @@
-copy .\.tool-versions .\.devcontainer\.tool-versions

.dockerignore

@@ -2,10 +2,6 @@ apps/fz_http/assets/node_modules
 apps/fz_http/priv/static/dist
 apps/fz_http/priv/cert
 _build
-apps/fz_http/_build
-apps/fz_wall/_build
-apps/fz_vpn/_build
-apps/fz_common/_build
 **/cover
 docs
 .DS_Store

.github/workflows/omnibus_build.yml

@@ -100,7 +100,7 @@ jobs:
       - name: Cleanup
         if: always()
         run: |
-          sudo scripts/uninstall.sh
+          sudo scripts/omnibus-uninstall.sh
           sudo rm -rf /tmp/firezone*
           rm -rf omnibus/pkg/*
 

.gitignore

@@ -1,7 +1,7 @@
 # macOS cruft
 .DS_Store
 
-.devcontainer/pki/authorities/local/
+priv/pki/authorities/local/
 
 # The directory Mix will write compiled artifacts to.
 /_build/

CONTRIBUTING.md

@@ -15,8 +15,6 @@ started.
   * [The .env File](#the-env-file)
   * [Bootstrapping](#bootstrapping)
   * [Ensure Everything Works](#ensure-everything-works)
-  * [Running this inside a Devcontainer](#running-this-inside-a-devcontainer)
-    * [Note: Devcontainer on Windows](#note--devcontainer-on-windows)
 * [Reporting Bugs](#reporting-bugs)
 * [Opening a Pull Request](#opening-a-pull-request)
   * [Run Tests](#run-tests)
@@ -88,7 +86,7 @@ machine, and out to the external Internet should work as well.
 ### Local HTTPS
 
 We use Caddy as a development proxy. The `docker-compose.yml` is set up to link
-Caddy's local root cert into your `.devcontainer/pki/authorities/local/` directory.
+Caddy's local root cert into your `priv/pki/authorities/local/` directory.
 
 Simply add the `root.crt` file to your browser and/or OS certificate store in
 order to have working local HTTPS. This file is generated when Caddy launches for
@@ -139,7 +137,7 @@ To start the local development cluster, follow these steps:
 ```
 docker compose build
 docker compose up -d postgres
-docker compose run --rm elixir mix ecto.setup
+docker compose run --rm firezone mix ecto.setup
 docker compose up
 ```
 
@@ -151,7 +149,7 @@ and sign in with email `firezone@localhost` and password `firezone1234`.
 There is a `client` container in the docker-compose configuration that
 can be used to simulate a WireGuard client connecting to Firezone. It's already
 provisioned in the Firezone development cluster and has a corresponding
-WireGuard configuration located at .devcontainer/wg0.client.conf.
+WireGuard configuration located at `priv/wg0.client.conf`.
 It's attached to the `isolation` Docker network which is isolated from the other
 Firezone Docker services. By connecting to Firezone from the `client`
 container, you can test the WireGuard tunnel is set up correctly by pinging the
@@ -163,29 +161,6 @@ container, you can test the WireGuard tunnel is set up correctly by pinging the
 
 If the above commands indicate success, you should be good to go!
 
-### Running this inside a Devcontainer
-
-You can run this using Github Codespaces or your own devcontainer using Docker.
-
-On GitHub Codespaces, follow the instructions above but pass in your Codespace
-external url:
-
-`EXTERNAL_URL=[your_devcontainer_url] MIX_ENV=dev mix start`
-
-or using the `.env` file
-
-`env $(cat .env | grep -v \# | xargs) mix start`
-
-On Github Codespaces you can find your EXTERNAL_URL by issuing the following
-command in the terminal:
-
-`echo "https://${CODESPACE_NAME}-4000.githubpreview.dev"`
-
-#### Note: Devcontainer on Windows
-
-If you are on Windows, make sure your git config `core.autocrlf` is off. Otherwise,
-the `\r` characters confuse asdf, which in turn fails the devcontainer build.
-
 ## Reporting Bugs
 
 We appreciate any and all bug reports.

apps/fz_http/assets/package-lock.json

@@ -37,14 +37,14 @@
       }
     },
     "../../../deps/phoenix": {
-      "version": "1.6.13",
+      "version": "1.7.0-rc.0",
       "license": "MIT"
     },
     "../../../deps/phoenix_html": {
       "version": "3.2.0"
     },
     "../../../deps/phoenix_live_view": {
-      "version": "0.18.2",
+      "version": "0.18.3",
       "license": "MIT"
     },
     "local_modules/admin-one-bulma-dashboard": {

apps/fz_http/lib/fz_http/devices.ex

@@ -63,24 +63,17 @@ defmodule FzHttp.Devices do
   def get_device!(id), do: Repo.get!(Device, id)
 
   def create_device(attrs \\ %{}) do
-    # XXX: insert sometimes fails with deadlock errors, probably because
-    # of the giant SELECT in queries/inet.ex. Find a way to do this more gracefully.
-    {:ok, result} =
-      Repo.transaction(fn ->
-        %Device{}
-        |> Device.create_changeset(attrs)
-        |> Repo.insert()
-      end)
-
-    case result do
-      {:ok, _device} ->
+    attrs
+    |> Device.create_changeset()
+    |> Repo.insert()
+    |> case do
+      {:ok, device} ->
         Telemetry.add_device()
+        {:ok, device}
 
-      _ ->
-        nil
+      {:error, changeset} ->
+        {:error, changeset}
     end
-
-    result
   end
 
   def update_device(%Device{} = device, attrs) do

apps/fz_http/lib/fz_http/devices/device.ex

@@ -61,7 +61,7 @@ defmodule FzHttp.Devices.Device do
     timestamps(type: :utc_datetime_usec)
   end
 
-  def create_changeset(device, attrs) do
+  def create_changeset(device \\ %__MODULE__{}, attrs) do
     device
     |> shared_cast(attrs)
     |> put_next_ip(:ipv4)

apps/fz_http/mix.exs

@@ -107,14 +107,10 @@ defmodule FzHttp.MixProject do
   # See the documentation for `Mix` for more info on aliases.
   defp aliases do
     [
-      "ecto.seed": "run priv/repo/seeds.exs",
-      "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
+      "ecto.seed": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
+      "ecto.setup": ["ecto.create", "ecto.migrate"],
       "ecto.reset": ["ecto.drop", "ecto.setup"],
-      test: [
-        "ecto.create --quiet",
-        "ecto.migrate",
-        "test"
-      ]
+      test: ["ecto.create --quiet", "ecto.migrate", "test"]
     ]
   end
 end

apps/fz_http/test/fz_http/rules_test.exs

@@ -172,7 +172,7 @@ defmodule FzHttp.RulesTest do
   describe "as_settings/0" do
     setup [:create_rules]
 
-    test "Maps rules to projections", %{rules: rules} do
+    test "maps rules to projections", %{rules: rules} do
       expected_rules = Enum.map(rules, &Rules.setting_projection/1) |> MapSet.new()
 
       assert Rules.as_settings() == expected_rules

apps/fz_http/test/support/fixtures/devices_fixtures.ex

@@ -15,12 +15,16 @@ defmodule FzHttp.DevicesFixtures do
 
     default_attrs = %{
       user_id: user_id,
-      public_key: "test-pubkey",
-      name: "factory",
+      public_key: "test-pubkey-#{counter()}",
+      name: "factory #{counter()}",
       description: "factory description"
     }
 
     {:ok, device} = Devices.create_device(Map.merge(default_attrs, attrs))
     device
   end
+
+  defp counter do
+    System.unique_integer([:positive])
+  end
 end

config/dev.exs

@@ -31,8 +31,15 @@ config :fz_http, FzHttpWeb.Endpoint,
   ]
 
 get_egress_interface = fn ->
-  egress_interface_cmd = "route | grep '^default' | grep -o '[^ ]*$'"
-  System.cmd("/bin/sh", ["-c", egress_interface_cmd]) |> elem(0) |> String.trim()
+  egress_interface_cmd =
+    case :os.type() do
+      {:unix, :darwin} -> "netstat -rn -finet | grep '^default' | awk '{print $NF;}'"
+      {_os_family, _os_name} -> "route | grep '^default' | grep -o '[^ ]*$'"
+    end
+
+  System.cmd("/bin/sh", ["-c", egress_interface_cmd], stderr_to_stdout: true)
+  |> elem(0)
+  |> String.trim()
 end
 
 egress_interface = System.get_env("EGRESS_INTERFACE") || get_egress_interface.()

docker-compose.yml

@@ -18,8 +18,8 @@ services:
   caddy:
     image: caddy:2
     volumes:
-      - ./.devcontainer/Caddyfile:/etc/caddy/Caddyfile
-      - ./.devcontainer/pki:/data/caddy/pki
+      - ./priv/Caddyfile:/etc/caddy/Caddyfile
+      - ./priv/pki:/data/caddy/pki
     ports:
       - 80:80
       - 443:443
@@ -209,7 +209,7 @@ services:
       - TZ=UTC
       - ALLOWEDIPS="0.0.0.0/0,::/0"
     volumes:
-      - ./.devcontainer/wg0.client.conf:/config/wg0.conf
+      - ./priv/wg0.client.conf:/config/wg0.conf
     cap_add:
       - NET_ADMIN
       - SYS_MODULE

docs/docs/administer/uninstall.mdx

@@ -27,10 +27,10 @@ rm -rf $installDir
 <TabItem label="Omnibus" value="omnibus">
 
 To completely remove Omnibus-based deployments of Firezone run the [uninstall.sh
-script](https://github.com/firezone/firezone/blob/master/scripts/uninstall.sh):
+script](https://github.com/firezone/firezone/blob/master/scripts/omnibus-uninstall.sh):
 
 ```bash
-sudo /bin/bash -c "$(curl -fsSL https://github.com/firezone/firezone/raw/master/scripts/uninstall.sh)"
+sudo /bin/bash -c "$(curl -fsSL https://github.com/firezone/firezone/raw/master/scripts/omnibus-uninstall.sh)"
 ```
 
 </TabItem>

mix.exs

@@ -70,10 +70,10 @@ defmodule FirezoneUmbrella.MixProject do
 
   defp aliases do
     [
-      "ecto.seed": "run apps/fz_http/priv/repo/seeds.exs",
-      "ecto.setup": ["ecto.create", "ecto.migrate", "ecto.seed"],
-      "ecto.remigrate": ["ecto.drop", "ecto.create", "ecto.migrate"],
+      "ecto.seed": ["ecto.create", "ecto.migrate", "run apps/fz_http/priv/repo/seeds.exs"],
+      "ecto.setup": ["ecto.create", "ecto.migrate"],
       "ecto.reset": ["ecto.drop", "ecto.setup"],
+      test: ["ecto.create --quiet", "ecto.migrate", "test"],
       start: ["compile --no-validate-compile-env", "phx.server", "run --no-halt"]
     ]
   end

omnibus/config/projects/firezone.rb

@@ -50,7 +50,6 @@ dependency 'firezone-cookbooks'
 
 # XXX: Ensure all development resources aren't included
 exclude '.env'
-exclude '.devcontainer'
 exclude '.github'
 exclude '.vagrant'
 exclude '.ci'