## Changelog
- Updates connlib parameter API_URL (formerly known under different
names as `CONTROL_PLANE_URL`, `PORTAL_URL`, `PORTAL_WS_URL`, and
friends) to be configured as an "advanced" or "hidden" feature at
runtime so that we can test production builds on both staging and
production.
- Makes `AUTH_BASE_URL` configurable at runtime too
- Moves `CONNLIB_LOG_FILTER_STRING` to be configured like this as well
and simplifies its naming
- Fixes a timing attack bug on Android when comparing the `csrf` token
- Adds proper account ID validation to Android to prevent invalid URL
parameter strings from being saved and used
- Cleans up a number of UI / view issues on Android regarding typos,
consistency, etc
- Hides vars from from the `relay` CLI we may not want to expose just
yet
- `get_device_id()` is flawed for connlib components -- SMBios is rarely
available. Data plane components now require a `FIREZONE_ID` now instead
to use for upserting.
Fixes#2482Fixes#2471
---------
Signed-off-by: Jamil <jamilbk@users.noreply.github.com>
Co-authored-by: Gabi <gabrielalejandro7@gmail.com>
- rebuild and publish gateway and relay binaries to currently drafted
release
- re-tag current relay/gateway images and push to ghcr.io
Stacked on #2341 to prevent conflicts
Fixes#2223Fixes#2205Fixes#2202Fixes#2239
~~Still TODO: `arm64` images and binaries...~~ Edit: added via
`cross-rs`
Fixes#2363
* Rename `relay` package to `firezone-relay` so that binaries outputted
match the `firezone-*` cli naming scheme
* Rename `firezone-headless-client` package to `firezone-linux-client`
for consistency
* Add READMEs for user-facing CLI components (there will also be docs
later)
Upsides:
1. We don't need to maintain a separate repo and Dockerfile just for
Elixir image (permissions, runner labels, etc)
2. No need to push intermediate images to the container registry
3. No need to copy-paste alpine/erlang/elixir version and hashes from
`firezone/containers` to `elixir/dockerfile` every time they change
4. No need to cross-compile for local dev environments, better
experience building with slow internet connection
5. One command to test if our code works on our containers but a
different alpine/erlang/elixir version
Downsides:
1. Locally devs will need to compile Erlang at least once per version,
but the whole build takes ~6 minutes on my M1 Max. It also takes only 8
minutes on the free GitHub Actions runner without any cache.
2. Worse experience on slow machines
FYI: there is no performance penalty once we have cache layers, still
takes 30 seconds on CI.
This patch-set aims to make several improvements to our CI caching:
1. Use of registry as build cache: Pushes a separate image to our docker
registry at GCP that contains the cache layers. This happens for every
PR & main. As a result, we can restore from **both** which should make
repeated runs of CI on an individual PR faster and give us a good
baseline cache for new PRs from `main`. See
https://docs.docker.com/build/ci/github-actions/cache/#registry-cache
for details. As a nice side-effect, this allows us to use the 10 GB we
have on GitHub actions for other jobs.
2. We make better use of `restore-keys` by also attempting to restore
the cache if the fingerprint of our lockfiles doesn't match. This is
useful for CI runs that upgrade dependencies. Those will restore a cache
that is still useful although doesn't quite match. That is better[^1]
than not hitting the cache at all.
3. There were two tiny bugs in our Swift and Android builds:
a. We used `rustup show` in the wrong directory and thus did not
actually install the toolchain properly.
b. We used `shared-key` instead of `key` for the
https://github.com/Swatinem/rust-cache action and thus did not
differentiate between jobs properly.
5. Our Dockerfile for Rust had a bug where it did not copy in the
`rust-toolchain.toml` file in the `chef` layer and thus also did not use
the correctly toolchain.
6. We remove the dedicated gradle cache because the build action already
comes with a cache configuration:
https://github.com/firezone/firezone/actions/runs/6416847209/job/17421412150#step:10:25
[^1]: Over time, this may mean that our caches grow a bit. In an ideal
world, we automatically remove files from the caches that haven't been
used in a while. The cache action we use for Rust does that
automatically:
https://github.com/Swatinem/rust-cache?tab=readme-ov-file#cache-details.
As a workaround, we can just purge all caches every now and then.
---------
Signed-off-by: Jamil <jamilbk@users.noreply.github.com>
Co-authored-by: Jamil <jamilbk@users.noreply.github.com>
Getting some weird behavior with AppLinks. They don't seem to work upon
first use and require a few tries to function correctly.
Edit: Found the issue: Android Studio doesn't like when the Manifest
contains variables for AppLinks. I added a note in the Manifest.
@conectado To test Applinks are working correctly, you can use the App
Link Assistant:
<img width="930" alt="Screenshot 2023-09-28 at 11 15 11 PM"
src="https://github.com/firezone/firezone/assets/167144/e4bd4674-d562-44ec-bdb8-3a5f97250b84">
Then from there you can click "Test App Links":
<img width="683" alt="Screenshot 2023-09-28 at 11 15 30 PM"
src="https://github.com/firezone/firezone/assets/167144/f3dc8e0d-f58a-4a4b-9855-62472096dc9e">
Basically we were having a panic inside a panic before, when I tried to
drop the runtime in `on_disconnect` since you can't drop a runtime
within a runtime. This PR spawns a new thread that listen for
disconnection and stops the runtime right there.
This also fixes the timer for reconnections.
Note: That I first stop it and the drop it which is redundant but I
rather be safe :)
This PR allows the TURN allocation binding to be optionally configured
by `TURN_LOWEST_PORT` and `TURN_HIGHEST_PORT` environment variables.
This will allow client app developers to test their apps against a
fully-working local development cluster in Docker Desktop for
Linux/macOS/Windows, allowing us to remove the PortalMock, Connlib Mock,
and SwiftMock codepaths entirely.
cc @roop @pratikvelani
Previously, we required the user to specify a `LISTEN_IP4_ADDR` and/or a
`LISTEN_IP6_ADDR` parameter. This is cumbersome because dynamically
fetching the address of the local interface is not trivial in all
environments.
We remove this parameter in exchange for listening on all interfaces.
This is a trade-off. The relay will now listen on all interfaces, even
the ones not exposed to the public internet. This is true for the main
socket on port 3478 and for all created allocations. Actually relaying
data relies on the 4-tuple of a "connection", i.e. the source and
destination address and port. Technically, I think it is possible with
this change to send traffic to a relay via an interface that was not
intended to be used for that. I think this will still require spoofing
the source address which is a known and accepted problem.
It is still recommended that operators put appropriate firewall rules in
place to not allow ingress traffic on any interface other than the one
intended for relaying.
I've tested locally that we are correctly using the `IPV6_ONLY` flag. In
other words, a relay listening on the `0.0.0.0` wildcard interface will
not accept IPv6 traffic and vice versa.
Resolves#1886.
This PR should fix the way we handle the `length` field in the
`DataChannel` messages, previous to this fix relaying data (using the
`webrtc-rs` crate) was impossible)
The new way to handle this is if the actual message is bigger than what
this data field says we ignore the extra bytes (which I think is the
correct way to do it according to spec)
Also, I added an integration test to verify relay messages using
`iptables`, not the cleanest way to do it but the easiest, in this vein
I tried to fix the caching for rust containers since 2 integration test
in our current state would take ~20 minutes each.
Previously, I thought it might be helpful to refuse a insecure
connections to the portal unless the user explicitly opts-in to this. In
our CI and testing environment, this however proved to cause more
headaches than it helps.
This PR removes this flag and assumes that users are smart enough that
they should protect self-hosted portals with transport-level encryption.
This PR fixes a bunch of small things to allow a new flow to test
clients pinging a resource within docker compose.
Masquerade/Forwarding is enabled directly in the container for now, this
might change in the future.
Also added a README to be able to run this locally.
---------
Signed-off-by: Gabi <gabrielalejandro7@gmail.com>
Co-authored-by: Jamil <jamilbk@users.noreply.github.com>
With this PR the full control-plane message flow is working.
Meaning that if you do:
```
docker compose up -d
docker compose exec -it client "ping 172.20.0.2" # will fix this IP later
```
Messages start flowing to gateway. The gateway still not correctly
forwards the messages to the resource since masquerading is still not
working, although I suspect there might be an additional problem. Will
fix this in my next PR along with a README on how to test this whole
flow.
This PR also fixes how we sent the stamp secret to the gateway from the
relay, but I still see some warnings in the webrtc that I'm sure that
are due to a mismatch between how webrtc-rs and the relay handle
messages (The most important being `bind() failed: unexpected response
type`), I will take a look at that and a way to test that the flow works
when:
1. hole-punching is available
2. through relay when it's not
Since the flow right now works without hole-punching or relay since the
gateway is in the same network in the docker compose.
This PR fixes `docker compose up` but it doesn't have the test client ->
resource flow working but it prevent anything from erroring at startup.
This fixes:
* tokens (use the correct token for the client user agent we are using)
* randomize `name_suffix` at start up for connlib (we will eventually
allow options to set it manually)
* remove port ranges for relay (see firezone/product#613)
**Update CONTRIBUTING.md**
Why:
* The CONTRIBUTING.md doc seems to have fallen slightly out of date with
how Firezone now works. This commit updates the doc to provide a
quick start guide for getting all of the various Firezone components
up and running as quick as possible. The doc then links to the more
specific `Elixir` and `Rust` README.md files in the respective
directories to help developers who would like to contribute.
**Update docker-compose vault health check**
Why:
* The current Vault health check listed in the docker-compose file does
not seem to be working when using `localhost` in the `wget` command.
Updating the URL to use `127.0.0.1` seems to have fixed it.
---------
Signed-off-by: bmanifold <bmanifold@users.noreply.github.com>
Co-authored-by: Jamil <jamilbk@users.noreply.github.com>
There are problems building the docker images in macos using musl due to
ring's problems therefore we started using slim-debian with glibc for
development.
This brindgs connlib from its own separated repo to firezone's monorepo.
On top of bringing connlib we also add and unify the Dockerfile for all
rust binaries and add a docker-compose that can run a headless client, a
relay and a gateway which eventually will test the whole flow between a
client and a resource. For this to work we also incorporated some elixir
scripts to generate portal tokens for those components.
TODO:
- [x] Cluster formation for all API and web nodes
- [x] Injest Docker logs to Stackdriver
- [x] Fix assets building for prod
To finish later:
- [ ] Structured logging:
https://issuetracker.google.com/issues/285950891
- [ ] Better networking policy (eg. use public postmark ranges and deny
all unwanted egress)
- [ ] OpenTelemetry collector for Google Stackdriver
- [ ] LoggerJSON.Plug integration
---------
Signed-off-by: Andrew Dryga <andrew@dryga.com>
Co-authored-by: Jamil <jamilbk@users.noreply.github.com>
- [x] All configs should support ENV variable overrides over DB values
- [ ] ~Adding a new field to DB value should automatically write ENV
config to DB on app boot (so that we don't need migrations)~
- [x] Validate configs and report human-readable errors when something
is wrong, telling where it's invalid (eg. env key X) and what's wrong
with it
- [x] Reuse Changeset validations (we still have a DB schema and UI
form, and want to make sure it's valid)
- [x] Auto-generate docs
- [x] Merge `Config` and `Configurations` into one `Config` context
- [x] Lock out UI fields for configurations when there is an ENV
override
- [x] Lock out corresponding REST API configuration field if overridden
via ENV var
- [x] Log a warning when deprecated legacy var is used
- [x] Document precedence: ENV -> Legacy ENV -> File -> DB
- [x] Change type to `inet[]` for `configurations.{default_client_dns,
default_client_allowed_ips}`, `devices.{dns, allowed_ips}`,
- [x] Drop `EctoNetwork` dep
- [x] `s/phoenix_port/phoenix_http_port` because it doesn't configure
HTTPS server
- [x] Do not load DB configs when config can be resolved from other
sources
Maybe:
- [ ] ~Auto-generate Ecto types to automatically cast/dump values
to/from DB~
- [ ] Allow JSON file config source
- [x] DB-related configs will not be validated?
Closes#1162Closes#1313Closes#1374Closes#1432
Regression was introduced in #1350: the path order did not allow auth
callback to be ever called.
Additionally, acceptance tests for SAML are added and we added a
blacklist of SAML config ids to prevent further route collisions.
Fixes#1362
This codepath was being skipped in the `test` env, which is no longer
necessary. This caused a runtime error that failed to show up in tests
because the codepath was being skipped.
Refs #1341
