feat(gateway): add flow-logs MVP (#10576)

Network flow logs are a common feature of VPNs. Due to the nature of a
shared exit node, it is of great interest to a network analyst, which
TCP connections are getting routed through the tunnel, who is initiating
them, for long do they last and how much traffic is sent across them.

With this PR, the Firezone Gateway gains the ability of detecting the
TCP and UDP flows that are being routed through it. The information we
want to attach to these flows is spread out over several layers of the
packet handling code. To simplify the implementation and not complicate
the APIs unnecessarily, we chose to rely on TLS (thread-local storage)
for gathering all the necessary data as a packet gets passed through the
various layers. When using a const initializer, the overhead of a TLS
variable over an actual local variable is basically zero. The entire
routing state of the Gateway is also never sent across any threads,
making TLS variables a particularly good choice for this problem.

In its MVP form, the detected flows are only emitted on stdout and also
that only if `flow_logs=trace` is set using `RUST_LOG`. Early adopters
of this feature are encouraged to enable these logs as described and
then ingest the Gateway's logs into the SIEM of their choice for further
analysis.

Related: #8353
This commit is contained in:
Thomas Eizinger
2025-10-22 14:10:21 +11:00
committed by GitHub
parent 80331b4e93
commit 6a538368cb
12 changed files with 1124 additions and 29 deletions

View File

@@ -103,6 +103,8 @@ jobs:
test:
- script: create-flow-from-icmp-error
min_client_version: 1.5.4
- script: download-rst
min_gateway_version: 1.4.18
- script: curl-api-down
- script: curl-api-restart
- script: curl-ecn
@@ -113,31 +115,43 @@ jobs:
- name: dns-systemd-resolved
script: systemd/dns-systemd-resolved
- script: tcp-dns
# Setting both client and gateway to random masquerade will force relay-relay candidate pair
- script: download-concurrent
min_gateway_version: 1.4.18
- name: download-double-symmetric-nat
script: download
# Setting both client and gateway to random masquerade will force relay-relay candidate pair
client_masquerade: random
gateway_masquerade: random
rust_log: debug
rust_log: debug,flow_logs=trace
single_relay: true # Force single relay
min_gateway_version: 1.4.18
- script: download-packet-loss
rust_log: debug
- script: download-roaming-network
# Too noisy can cause flaky tests due to the amount of data
rust_log: debug
min_gateway_version: 1.4.18
rust_log: debug,flow_logs=trace # Too noisy can cause flaky tests due to the amount of data
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: ./.github/actions/ghcr-docker-login
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
- name: Check minimum client version
id: version_check
id: client_version_check
if: ${{ matrix.test.min_client_version }}
continue-on-error: true
run: |
ACTUAL_VERSION=$(docker run ${{ inputs.client_image }}:${{ inputs.client_tag }} firezone-headless-client --version | awk '{print $2}')
MIN_VERSION="${{ matrix.test.min_client_version }}"
[ "$(printf '%s\n' "$MIN_VERSION" "$ACTUAL_VERSION" | sort --version-sort | head -n1)" == "$MIN_VERSION" ]
- name: Check minimum gateway version
id: gateway_version_check
if: ${{ matrix.test.min_gateway_version }}
continue-on-error: true
run: |
ACTUAL_VERSION=$(docker run ${{ inputs.gateway_image }}:${{ inputs.gateway_tag }} firezone-gateway --version | awk '{print $2}')
MIN_VERSION="${{ matrix.test.min_gateway_version }}"
[ "$(printf '%s\n' "$MIN_VERSION" "$ACTUAL_VERSION" | sort --version-sort | head -n1)" == "$MIN_VERSION" ]
# We need at least Docker v28.1 which is not yet available on GitHub actions runners
- uses: docker/setup-docker-action@b60f85385d03ac8acfca6d9996982511d8620a19 # v4.3.0
@@ -189,7 +203,7 @@ jobs:
sudo ethtool -K docker0 tx off
- run: ./scripts/tests/${{ matrix.test.script }}.sh
if: ${{ steps.version_check.outcome != 'failure' }} # Run the script if version check succeeds or is skipped
if: ${{ steps.client_version_check.outcome != 'failure' && steps.gateway_version_check.outcome != 'failure' }} # Run the script if version checks succeed or are skipped
- name: Ensure Client emitted no warnings
if: "!cancelled()"