feat(connlib): create flow on ICMP error "prohibited" (#10462)

In Firezone, a Client requests an "access authorization" for a Resource
on the fly when it sees the first packet for said Resource going through
the tunnel. If we don't have a connection to the Gateway yet, this is
also where we will establish a connection and create the WireGuard
tunnel.

In order for this to work, the access authorization state between the
Client and the Gateway MUST NOT get out of sync. If the Client thinks it
has access to a Resource, it will just route the traffic to the Gateway.
If the access authorization on the Gateway has expired or vanished
otherwise, the packets will be black-holed.

Starting with #9816, the Gateway sends ICMP errors back to the
application whenever it filters a packet. This can happen either because
the access authorization is gone or because the traffic wasn't allowed
by the specific filter rules on the Resource.

With this patch, the Client will attempt to create a new flow (i.e.
re-authorize) traffic for this resource whenever it sees such an ICMP
error, therefore acting as a way of synchronizing the view of the world
between Client and Gateway should they ever run out of sync.

Testing turned out to be a bit tricky. If we let the authorization on
the Gateway lapse naturally, we portal will also toggle the Resource off
and on on the Client, resulting in "flushing" the current
authorizations. Additionally, it the Client had only access to one
Resource, then the Gateway will gracefully close the connection, also
resulting in the Client creating a new flow for the next packet.

To actually trigger this new behaviour we need to:

- Access at least two resources via the same Gateway
- Directly send `reject_access` to the Gateway for this particular
resource

To achieve this, we dynamically eval some code on the API node and
instruct the Gateway channel to send `reject_access`. The connection
stays intact because there is still another active access authorization
but packets for the other resource are answered with ICMP errors.

To achieve a safe roll-out, the new behaviour is feature-flagged. In
order to still test it, we now also allow feature flags to be set via
env variables.

Resolves: #10074

---------

Co-authored-by: Mariusz Klochowicz <mariusz@klochowicz.com>

scripts/tests/create-flow-from-icmp-error.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+
+source "./scripts/tests/lib.sh"
+
+# Authorize resource 1
+client_curl_resource "172.20.0.100/get"
+client_curl_resource "[172:20:0::100]/get"
+
+# Authorize resource 2 (important, otherwise the Gateway will close the connection on the last resource being removed)
+client_ping_resource download.httpbin
+
+# Revoke access to resource 1
+api_send_reject_access "mycro-aws-gws" "MyCorp Network"        # This is the 172.20.0.1/16 network
+api_send_reject_access "mycro-aws-gws" "MyCorp Network (IPv6)" # This is the 172:20:0::1/64 network
+
+# Try to access resource 1 again
+# First one for each IP will fail because we get an ICMP error.
+expect_error client_curl_resource "172.20.0.100/get"
+expect_error client_curl_resource "[172:20:0::100]/get"
+
+client_curl_resource "172.20.0.100/get"
+client_curl_resource "[172:20:0::100]/get"

scripts/tests/lib.sh

@@ -33,6 +33,23 @@ function client_nslookup() {
     client timeout 30 sh -c "nslookup $1 | tee >(cat 1>&2) | tail -n +4"
 }
 
+function api_send_reject_access() {
+    local site_name="$1"
+    local resource_name="$2"
+
+    docker compose exec -T api bin/api rpc "
+Application.ensure_all_started(:domain)
+account_id = \"c89bcc8c-9392-4dae-a40d-888aef6d28e0\"
+
+[gateway_group] = Domain.Gateways.Group.Query.not_deleted() |> Domain.Gateways.Group.Query.by_account_id(account_id) |> Domain.Gateways.Group.Query.by_name(\"$site_name\") |> Domain.Repo.all()
+[gateway_id | _] = Domain.Gateways.Presence.Group.list(gateway_group.id) |> Map.keys()
+[client_id | _] = Domain.Clients.Presence.Account.list(account_id) |> Map.keys()
+[resource] = Domain.Resources.Resource.Query.not_deleted() |> Domain.Resources.Resource.Query.by_account_id(account_id) |> Domain.Repo.all() |> Enum.filter(&(&1.name == \"$resource_name\"))
+
+Domain.PubSub.Account.broadcast(account_id, {{:reject_access, gateway_id}, client_id, resource.id})
+"
+}
+
 function assert_equals() {
     local actual="$1"
     local expected="$2"
@@ -69,3 +86,13 @@ function create_token_file {
     # cut into a release.
     sudo cp "$TOKEN_PATH" "$TOKEN_PATH.txt"
 }
+
+# Expects a command to fail (non-zero exit code)
+# Usage: expect_error your_command arg1 arg2
+function expect_error() {
+    if "$@"; then
+        return 1
+    else
+        return 0
+    fi
+}