ci: prevent packet reordering by router containers (#10328)

By default, RPS (Receive Packet Steering) is disabled on Linux which
means the CPU handling the interrupt for an incoming packet also handles
the packet. Under high-load, this can causes packet reordering in your
test setup where at least two routers are in the path between Client and
Gateway.

To ensure our test suite is deterministic, we enable RPS and set it to
1, meaning always CPU 1 will handle all packets.

Local testing has shown that this fixes the warnings of "packet counter
too old" on the Gateway and instead, all packets arrive entirely in
order.

Source:
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/performance_tuning_guide/network-rps

docker-compose.yml

@@ -224,6 +224,7 @@ services:
     cap_add:
       - NET_ADMIN
     sysctls: *ip-forwarding
+    privileged: true
     environment:
       PORT_FORWARDS: |
         8081 172.28.0.100 tcp
@@ -420,6 +421,7 @@ services:
       context: scripts/router
     cap_add:
       - NET_ADMIN
+    privileged: true
     sysctls: *ip-forwarding
     environment:
       MASQUERADE_TYPE: ${CLIENT_MASQUERADE:-}
@@ -490,6 +492,7 @@ services:
       context: scripts/router
     cap_add:
       - NET_ADMIN
+    privileged: true
     sysctls: *ip-forwarding
     environment:
       MASQUERADE_TYPE: ${GATEWAY_MASQUERADE:-}
@@ -617,6 +620,7 @@ services:
       context: scripts/router
     cap_add:
       - NET_ADMIN
+    privileged: true
     sysctls: *ip-forwarding
     environment:
       PORT_FORWARDS: |
@@ -694,6 +698,7 @@ services:
       context: scripts/router
     cap_add:
       - NET_ADMIN
+    privileged: true
     sysctls: *ip-forwarding
     environment:
       PORT_FORWARDS: |

scripts/router/router.sh

@@ -72,6 +72,12 @@ nft -f "$CONFIG_FILE"
 
 rm "$CONFIG_FILE"
 
+for iface in internal internet; do
+    # Enable RPS (Receive Packet Steering) to always use CPU 1 to handle packets.
+    # This prevents packet reordering where otherwise the CPU which handles the interrupt would handle the packet.
+    echo 1 >"/sys/class/net/$iface/queues/rx-0/rps_cpus" 2>/dev/null
+done
+
 echo "1" >/tmp/setup_done # Health check marker
 
 # Keep container running