By default, Docker creates network interfaces with a txqueuelen of 1000.
This is pretty small and causes unnecessary packet drops when running
perf tests in that setup.
Jitter causes packets to get re-ordered which makes it really hard to
get predictable performance results. With jitter disabled, we get more
consistent performance numbers.
Now that we have a more realistic network setup in our compose file, we
can extend our router containers to apply the latency on the network
path. This means any use of the compose file has a latency by default,
simplifying our CI setup. It also allows us to restart containers
without having to re-apply the latency which is useful during
performance testing.
By default, RPS (Receive Packet Steering) is disabled on Linux which
means the CPU handling the interrupt for an incoming packet also handles
the packet. Under high-load, this can causes packet reordering in your
test setup where at least two routers are in the path between Client and
Gateway.
To ensure our test suite is deterministic, we enable RPS and set it to
1, meaning always CPU 1 will handle all packets.
Local testing has shown that this fixes the warnings of "packet counter
too old" on the Gateway and instead, all packets arrive entirely in
order.
Source:
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/performance_tuning_guide/network-rps
Currently, the setup we have in docker-compose does not reflect
real-world scenarios very well because most components share the same
subnet. In reality, Clients, Gateways, relays and the backend are all in
separate subnets, connected via multiple routers on the Internet.
The current setup makes it hard to properly test relayed connections. To
fix this, we move all components into their own subnet with a dedicated
router container that performs source and destination NAT as well as
acts as a firewall for the client and gateway containers to not allow
inbound traffic.
This setup will allow us to more easily test #10286 which requires port
randomization for outgoing traffic on the Client and Gateway side.
