0079f76ebd
Any communication between user-space and the eBPF kernel happens via
maps. The keys and values in these maps are serialised to bytes, meaning
the endianness of how these values are encoded matters!
When debugging why the eBPF kernels were not performing as much as we
thought they would, I noticed that only very small packets were getting
relayed. In particular, only packets encoded as channel-data packets
were getting unwrapped correctly. The reverse didn't happen at all.
Turning the log-level up to TRACE did reveal that we do in fact see
these packets but they don't get handled.
Here is the relevant section that handles these packets:
74ccf8e0b2/rust/relay/ebpf-turn-router/src/main.rs (L127-L151)
We can see the `trace!` log in the logs and we know that it should be
handled by the first `if`. But for some reason it doesn't.
x86 systems like the machines running in GCP are typically
little-endian. Network-byte ordering is big-endian. My current theory is
that we are comparing the port range with the wrong endianness and
therefore, this branch never gets hit, causing the relaying to be
offloaded to user space.
By storing the fields within `Config` in byte-arrays, we can take
explicit control over which endianness is used to store these fields.
Rust development guide
Firezone uses Rust for all data plane components. This directory contains the Linux and Windows clients, and low-level networking implementations related to STUN/TURN.
We target the last stable release of Rust using rust-toolchain.toml.
If you are using rustup, that is automatically handled for you.
Otherwise, ensure you have the latest stable version of Rust installed.
Reading Client logs
The Client logs are written as JSONL for machine-readability.
To make them more human-friendly, pipe them through jq like this:
cd path/to/logs # e.g. `$HOME/.cache/dev.firezone.client/data/logs` on Linux
cat *.log | jq -r '"\(.time) \(.severity) \(.message)"'
Resulting in, e.g.
2024-04-01T18:25:47.237661392Z INFO started log
2024-04-01T18:25:47.238193266Z INFO GIT_VERSION = 1.0.0-pre.11-35-gcc0d43531
2024-04-01T18:25:48.295243016Z INFO No token / actor_name on disk, starting in signed-out state
2024-04-01T18:25:48.295360641Z INFO null
Benchmarking on Linux
The recommended way for benchmarking any of the Rust components is Linux' perf utility.
For example, to attach to a running application, do:
- Ensure the binary you are profiling is compiled with the
releaseprofile. sudo perf record -g --freq 10000 --pid $(pgrep <your-binary>).- Run the speed test or whatever load-inducing task you want to measure.
sudo perf script > profile.perf- Open profiler.firefox.com and load
profile.perf
Instead of attaching to a process with --pid, you can also specify the path to executable directly.
That is useful if you want to capture perf data for a test or a micro-benchmark.