mirror of
https://github.com/outbackdingo/firezone.git
synced 2026-01-27 10:18:54 +00:00
ac5e44d5d0
Sufficiently large receive buffers are important to sustain high-throughput as latency increases. If the receive buffer in the kernel is too small, packets need to be dropped on arrival. Firefox uses 1MB in its QUIC stack [0]. `quic-go` recommends to set send and receive buffers to 7.5 MB [1]. Power users of Firezone are likely receiving a lot more traffic than the average Firefox user (especially with Internet Resource activated) so setting it to 10 MB seems reasonable. Sending packets is likely not as critical because we have back-pressure through our system such that we will stop reading IP packets when we cannot write to our UDP socket. The UDP socket is sitting in a separate thread and those threads are connected with dedicated queues which act as another buffer. However, as the data below shows, some systems have really small send buffers which are currently likely a speed bottleneck because we need to suspend writing so frequently. Assuming a 50ms latency, the bandwidth-delay product tells us that we can (in theory) saturate a 1.6 Gbps link with a 10MB receive buffer (assuming the OS also has large enough buffer sizes in its TCP or QUIC stack): ``` 80 Mb / 0.05s = 1600Mbps ``` Experiments and research [2] show the following: |OS|Receive buffer (default)|Receive buffer (this PR)|Send buffer (default)|Send buffer (this PR)| |---|---|---|---|---| |Windows|65KB|10MB|65KB|1MB| |MacOS|786KB|8MB|9KB|1MB| |Linux|212KB|212KB|212KB|212KB| With the exception of Linux, the OSes appear to be quite generous with how big they allow receive buffers to be. On Linux, these limit can be changed by setting the `core.net.rmem_max` and `core.net.wmem_max` parameters using `sysctl`. Most of our users are on Windows and MacOS, meaning they immediately benefit from this without having to change any system settings. Larger client-side UDP receive buffers are critical for any "download" scenario which is likely the majority of usecases that Firezone is used for. On Windows, increasing this receive buffer almost doubles the throughput in an iperf3 download test. [0]: https://github.com/mozilla/neqo/pull/2470 [1]: https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes [2]: https://unix.stackexchange.com/a/424381 --------- Signed-off-by: Thomas Eizinger <thomas@eizinger.io> Co-authored-by: Jamil <jamilbk@users.noreply.github.com>
Rust development guide
Firezone uses Rust for all data plane components. This directory contains the Linux and Windows clients, and low-level networking implementations related to STUN/TURN.
We target the last stable release of Rust using rust-toolchain.toml.
If you are using rustup, that is automatically handled for you.
Otherwise, ensure you have the latest stable version of Rust installed.
Reading Client logs
The Client logs are written as JSONL for machine-readability.
To make them more human-friendly, pipe them through jq like this:
cd path/to/logs # e.g. `$HOME/.cache/dev.firezone.client/data/logs` on Linux
cat *.log | jq -r '"\(.time) \(.severity) \(.message)"'
Resulting in, e.g.
2024-04-01T18:25:47.237661392Z INFO started log
2024-04-01T18:25:47.238193266Z INFO GIT_VERSION = 1.0.0-pre.11-35-gcc0d43531
2024-04-01T18:25:48.295243016Z INFO No token / actor_name on disk, starting in signed-out state
2024-04-01T18:25:48.295360641Z INFO null
Benchmarking on Linux
The recommended way for benchmarking any of the Rust components is Linux' perf utility.
For example, to attach to a running application, do:
- Ensure the binary you are profiling is compiled with the
releaseprofile. sudo perf record -g --freq 10000 --pid $(pgrep <your-binary>).- Run the speed test or whatever load-inducing task you want to measure.
sudo perf script > profile.perf- Open profiler.firefox.com and load
profile.perf
Instead of attaching to a process with --pid, you can also specify the path to executable directly.
That is useful if you want to capture perf data for a test or a micro-benchmark.