github-personal/firezone
1
0
Fork 0
mirror of https://github.com/outbackdingo/firezone.git synced 2026-01-27 10:18:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(ci): Architect e2e test suite (#3043)

Browse Source 
Adds an overview document with a proposed plan and test bed architecture
for e2e testing the clients.

Drivers are available for [each platform using
Appium](https://appium.io/docs/en/2.3/ecosystem/drivers/).
This commit is contained in:
Jamil
2024-01-02 08:07:13 -08:00
committed by GitHub
parent 110e128b93
commit 57133122e0
9 changed files with 226 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
.github/README_CI.md vendored
View File

@@ -33,25 +33,6 @@ difference using a larger runner.
## Self-hosted runners
We've also configured a self-hosted M1 runner to run macOS workloads:
- `macos-14`
You can target it with either the `macos-14` label, `self-hosted` label, or a
combination of the two. It's running at the Firezone HQ and basically costs
nothing to execute jobs on.
It should have the following software on it:
- Docker Desktop
- Xcode 15
- Homebrew
<!-- TODO: Add instructions when Dogfood is working
You may log into the self-hosted runner remotely via Apple Remote Desktop
if you need to make any changes to its configuration. To do so, make
sure your Firezone client is connected to the Dogfood account, then:
1. Open Screen Sharing.app
2. Connect to macos-14.firezone.dev
3. Log in with the "macOS M1 Firezone Builder GitHub Actions self-hosted Runner" credentials in Firezone Engineering 1Password
-->
We maintain a baremetal testbed for running our end-to-end test suite. See
[the `e2e`](../e2e) directory. Please don't target those runners unless you're
specifically trying to run workflows that require a baremetal runner.

6
.github/workflows/ci.yml vendored
View File

@@ -174,12 +174,6 @@ jobs:
fail-fast: false
matrix:
include:
# TODO
# - Run control plane components as services
# - Test clients
# - Test with different NAT types
# - Test IPv6
# - Test end-to-end critical paths
- test_name: Relayed flow
artifact_name: relayed
setup: |

85
.github/workflows/e2e.yml vendored Normal file
View File

@@ -0,0 +1,85 @@
name: End to end tests
on:
# TODO
# push:
# branches:
# - main
workflow_dispatch:
defaults:
working-directory: ./e2e
concurrency:
group: "e2e-${{ github.workflow }}-${{ github.ref }}"
cancel-in-progress: true
jobs:
setup-e2e:
runs-on: ubuntu-22.04-firezone
steps:
- uses: actions/checkout@v4
- run: echo "Spin up docker test environment"
macos:
needs: setup-e2e
runs-on: macos-14-firezone
steps:
- uses: actions/checkout@v4
- run: echo 'Noop'
windows:
needs: setup-e2e
runs-on: windows-11-firezone
steps:
- uses: actions/checkout@v4
- run: echo 'Noop'
linux:
needs: setup-e2e
runs-on: ubuntu-22.04-firezone
steps:
- uses: actions/checkout@v4
- run: echo 'Noop'
android:
needs: setup-e2e
runs-on: ubuntu-22.04-firezone
steps:
- uses: actions/checkout@v4
- run: echo 'Noop'
chromeos:
needs: setup-e2e
runs-on: ubuntu-22.04-firezone
steps:
- uses: actions/checkout@v4
- run: echo 'Noop'
ios:
needs: setup-e2e
runs-on: macos-14-firezone
steps:
- uses: actions/checkout@v4
- run: echo 'Noop'
ipados:
needs: setup-e2e
runs-on: macos-14-firezone
steps:
- uses: actions/checkout@v4
- run: echo 'Noop'
teardown-e2e:
needs:
- macos
- windows
- linux
- android
- chromeos
- ios
- ipados
if: 'always()'
runs-on: ubuntu-22.04-firezone
steps:
- uses: actions/checkout@v4
- run: echo 'Teardown e2e environment'

8
CONTRIBUTING.md
View File

@@ -36,7 +36,7 @@ To maximize your chances of getting your pull request approved, please abide by
the following general guidelines:
1. Please adhere to our [code of conduct](CODE_OF_CONDUCT.md).
1. Please test with your code and include unit tests when possible.
1. Please test your code and include unit tests when possible.
1. It is up to you, the contributor, to make a case for why your change is a
good idea.
1. For any security issues, please **do not** open a Github Issue. Please follow
@@ -79,10 +79,10 @@ After this you will have running:
- A portal
- A gateway connected to the portal
- A client connected to the portal
- A headless Linux client connected to the portal
- A relay connected to the portal
- A resource on a network only with the gateway
- The ip is `172.20.0.100` (And any other dependency for these to run)
- A resource with IP `172.20.0.100` on a separate network shared with the
gateway
```sh
# To test that a client can ping the resource

67
MAINTAINERS.md Normal file
View File

@@ -0,0 +1,67 @@
# Maintainer's Guide
This document contains instructions for maintaining the code in this repo
including the product, website, and documentation.
Table of Contents:
- [Releasing](#releasing)
- [Publishing Clients](#publishing-clients)
- [Apple Client](#apple-client)
## Releasing
**Note**: The version gets set from the Makefile
- Go to Actions tab in GH
- For apple the workflow is `Swift`
- For Android the workflow is `Kotlin`
- Click the edit button on the Draft 1.0.0 release
- Give the release a name (manually - right now we're using
`1.0.0-pre.<num-here>`)
- Create a new tag as well with the same name as the release (click
`create new tag`)
- Double check that the body text of the release is what is expected
- **IMPORTANT**: Scroll to the bottom and check the `Set as latest release` and
uncheck `Set as pre-release`
- Click `Publish Release`
- The `Publish` workflow is now run Note: This will deploy to production and the
following will happen
- All logged in users on the portal will be logged out, but the clients will
not be logged out
- All the websockets will be disconnected and should automatically reconnect
## Publish Clients
### Apple Client
- Log in to the following URL: https://appstoreconnect.apple.com/
- Go to Apps
- Go to Firezone
- Click on TestFlight
- Note: You can't delete a `Version` in TestFlight
- There is "internal testing" and "external testing"
- "internal testing" is only the Firezone team
- "external testing" is the beta customers
- Click on the testing group you want to release to and on the testing group
page:
- Click the `+` on the `Builds` sections
- Select the build you want to push out
- Check the `Automatically notify testers`
- Type a description of what you want users to see in the notification sent to
users (e.g. a small change log of what's in this release)
- Click `Submit for Review`
- Then you have to wait for it to be reviewed (has been a matter of minutes
as of late)
(Alternative way to push out a release)
- After login go to `Builds` (select either ios/macos)
- Find the `Version` section you want to release and drop down to show list of
builds
- Find the build you want to push out, hover over the `Groups` column and select
the `+` icon
- From here it's the same as the instructions above to type a description,
etc...

65
e2e/README.md Normal file
View File

@@ -0,0 +1,65 @@
# End-to-end Tests
This directory contains our suite of end-to-end tests for the following clients,
running on baremetal hardware:
- Windows
- Android
- ChromeOS
- macOS
- iOS
- iPadOS
We use [Appium](https://appium.io) as our test orchestrator because it supports
all of the above platforms with a consistent architecture and test language.
## Running
You shouldn't run these tests yourself. Instead, these tests are intended to be
run from our bare metal testbed running in the Firezone HQ. See the
[e2e.yml](../.github/workflows/e2e.yml) workflow for how these tests are
triggered and run.
## Architecture
Our baremetal testbed consists of the following hardware:
| Hostname | Purpose | OS / Model | CPU | RAM | Disk | Network |
| ----------------------- | ----------------------------------------------------- | ------------------------------------ | ----------------------- | -------------------- | --------------- | -------------------------------------- |
| `linux.firezone.test` | Orchestration server + Android / ChromeOS test runner | Ubuntu 22.04 | Ryzen 5950x | 128 GB DDR4 3200 ECC | 500 GB NVMe SSD | 10 GbE internet uplink + 1 GbE testnet |
| `macos.firezone.test` | macOS, iPadOS, iOS test driver | macOS 14 | Apple M1 | 8 GB | 256 GB | 1 GbE testnet + WiFi 6 |
| `windows.firezone.test` | Windows test driver | Windows 11 Home | Intel i3-1115G4 | 8 GB | 256 GB | 1 GbE testnet + WiFi 6 |
| `ios.firezone.test` | iOS test client | iOS 17 | A14 Bionic (iPhone 12) | 4 GB | 128 GB | WiFi 6 |
| `ipados.firezone.test` | iPadOS test client | iPadOS 17 | A12 (iPad 8th gen) | 3 GB | 128 GB | 802.11ax WiFi |
| `android.firezone.test` | Android test client | Android 14 (Pixel 6) | Tensor SoC | 8 GB | 128 GB | WiFi 6 |
| `gateway.firezone.test` | Remote access (using a Firezone gateway!) | Raspbian 12 (Raspberry Pi 2 Model B) | Cortex A7 4-core armv7l | 4 GB | 32 GB | 1 GbE testnet |
## Topology
Since we can't run the test driver directly on mobile platforms, we run it on
the host instead and connect over USB to the client. iOS/iPadOS clients are
tested from the `macos.firezone.test` host, and the Android/ChromeOS clients are
tested from the `linux.firezone.test` host.
There's also a Raspberry Pi in the testnet running Firezone, used for remote
access.
### Network diagram
![network diagram](./diagram.png)
### Addressing
| Host | Interface | IPv4 Address | IPv6 Address |
| ----------------------- | ---------------- | --------------- | ------------ |
| `linux.firezone.test` | egress | `192.168.1.222` | TODO |
| `linux.firezone.test` | testnet ethernet | `10.0.0.1` | TODO |
| `gateway.firezone.test` | testnet ethernet | `10.0.0.254` | TODO |
| `linux.firezone.test` | testnet WiFi | `10.0.1.1` | TODO |
| `macos.firezone.test` | testnet ethernet | `10.0.0.2` | TODO |
| `windows.firezone.test` | testnet ethernet | `10.0.0.3` | TODO |
| `macos.firezone.test` | testnet WiFi | `10.0.1.2` | TODO |
| `windows.firezone.test` | testnet WiFi | `10.0.1.3` | TODO |
| `ios.firezone.test` | testnet WiFi | `10.0.1.4` | TODO |
| `ipados.firezone.test` | testnet WiFi | `10.0.1.5` | TODO |
| `android.firezone.test` | testnet WiFi | `10.0.1.6` | TODO |

BIN
e2e/diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 79 KiB

2
website/package.json
View File

@@ -1,5 +1,5 @@
{
"name": "www2",
"name": "website",
"version": "0.1.0",
"private": true,
"scripts": {

2
website/src/app/about/team.tsx
View File

@@ -135,7 +135,7 @@ export default function Team() {
see for yourself
</Link>
. The team has experience building enterprise networking solutions
at companies like Cisco, Marqeta, Tailscale, and more.
at companies like Cisco, Marqeta, Instacart, and more.
</p>
</div>
<div className="mt-16 grid sm:grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-4 md:gap-8 lg:gap-16">