932f6791fb
Our `phoenix-channel` component is responsible for maintaining a WebSocket connection to the portal. In case that connection fails, we want to reconnect to it using an exponential backoff, eventually giving up after a certain amount of time. Unfortunately, the code we have today doesn't quite do that. An `ExponentialBackoff` has a setting for the `max_elapsed_time`. Regardless of how many and how often we retry something, we won't ever wait longer than this amount of time. For the Relay, this is set to 15min. For other components its indefinite (Gateway, headless-client), or very long (30 days for Android, 1 day for Apple). The point in time from which this duration is counted is when the `ExponentialBackoff` is **constructed** which translates to when we **first** connected to the portal. As a result, our backoff would immediately fail on the first error if it has been longer than `max_elapsed_time` since we first connected. For most components, this codepath is not relevant because the `max_elapsed_time` is so long. For the Relay however, that is only 15 minutes so chances are, the Relay would immediately fail (and get rebooted) on the first connection error with the portal. To fix this, we now lazily create the `ExponentialBackoff` on the first error. This bug has some interesting consequences: When a relay reboots, it looses all its state, i.e. allocations, channel bindings, available nonces etc, stamp-secret. Thus, all credentials and state that got distributed to Clients and Gateways get invalidated, causing disconnects from the Relay. We have observed these alerts in Sentry for a while and couldn't explain them. Most likely, this is the root cause for those because whilst a Relay disconnects, the portal also cannot detect its presence and pro-actively inform Clients and Gateways to no longer use this Relay.
relay
This crate houses a minimalistic STUN & TURN server.
Features
We aim to support the following feature set:
- STUN binding requests
- TURN allocate requests
- TURN refresh requests
- TURN channel bind requests
- TURN channel data requests
Relaying of data through other means such as DATA frames is not supported.
Building
You can build the relay using: cargo build --release --bin firezone-relay
You should then find a binary in target/release/firezone-relay.
Running
The Firezone Relay supports Linux only. To run the Relay binary on your Linux host:
- Generate a new Relay token from the "Relays" section of the admin portal and save it in your secrets manager.
- Ensure the
FIREZONE_TOKEN=<relay_token>environment variable is set securely in your Relay's shell environment. The Relay expects this variable at startup. - Now, you can start the Firezone Relay with:
firezone-relay
To view more advanced configuration options pass the --help flag:
firezone-relay --help
Ports
By default, the relay listens on port udp/3478. This is the standard port for
STUN/TURN. Additionally, the relay needs to have access to the port range
49152 - 65535 for the allocations.
Portal Connection
When given a token, the relay will connect to the Firezone portal and wait for
an init message before commencing relay operations.
Metrics
The relay parses the OTLP_GRPC_ENDPOINT env variable.
Traces and metrics will be sent to an OTLP collector listening on that endpoint.
It is recommended to set additional environment variables to scope your metrics:
OTEL_SERVICE_NAME: Translates to theservice.name.OTEL_RESOURCE_ATTRIBUTES: Additional, comma-separated key=value attributes.
By default, we set the following OTEL attributes:
service.name=relayservice.namespace=firezone
The docker-init-relay.sh script integrates with GCE.
When OTEL_METADATA_DISCOVERY_METHOD=gce_metadata, the service.instance.id
variables is set to the instance ID of the VM.
Design
The relay is designed in a sans-IO fashion, meaning the core components do not cause side effects but operate as pure, synchronous state machines. They take in data and emit commands: wake me at this point in time, send these bytes to this peer, etc.
This allows us to very easily unit-test all kinds of scenarios because all inputs are simple values.
The main server runs in a single task and spawns one additional task for each allocation. Incoming data that needs to be relayed is forwarded to the main task where it gets authenticated and relayed on success.