Closes#7175
Also fixes a bug with the initialization order of Tokio and Sentry.
Previously:
1. Start Tokio, executor threads inherit main thread context
2. Load device ID and set it on the main telemetry hub
Now:
1. Load device ID and set it on the main telemetry hub
2. Start Tokio, executor threads inherit main thread context
The context and possibly tags didn't seem to propagate from the main hub
if we set them after the worker threads spawned.
Based on this understanding, the IPC service process is still wrong, but
a fix will have to wait, because telemetry in the IPC service is more
complicated than in the GUI process.
<img width="818" alt="image"
src="https://github.com/user-attachments/assets/9c9efec8-fc55-4863-99eb-5fe9ba5b36fa">
When building inside a docker container, like we do for the
headless-client and gateway, the `.git` directory is not available.
Thus, determining what our current version is fails and gets reported as
"unknown". We are now also using this for Sentry which is not very
helpful if all errors are categorised under the same version.
In case somebody builds a gateway / client from source, we will have the
full version available. Most users will use our docker containers
though, meaning the version will only always be for a full release.
Resolves: #7184.
With the recent lobbying effort in `quinn-udp`, we were able to get
`try_send` APIs for the UDP socket that doesn't silence any errors while
sending datagrams. Originally, the reasoning in `quinn-udp` was that
because UDP is an unreliable protocol anyway, errors don't need to be
surfaced because there must be upper-level mechanisms for retrying
messages. Whilst that is true, getting immediate feedback that something
isn't working can also be very beneficial. For example, if you don't
have proper IPv6 connectivity on a socket, the syscall will immediately
fail with `DestinationUnreachable`.
Within Firezone, we use these UDP sockets to send all kinds of messages,
including DNS queries to upstream servers. In case that doesn't work,
failing instantly allows us to send a SERVFAIL error back to the OS
right away instead of having to wait for a timeout.
Additionally, `quinn-udp` logs these send errors on WARN which cause
unnecessary noise in Sentry.
Resolves: #6353.
Closes#7171
If the assets aren't bundled, Tauri will warn about it in `tracing`,
that will get sent to Sentry, and then it will be interpreted as an
error.
Timeline to prove that this fixes the false positive error in Sentry,
all times UTC on October 29th:
- 21:01:26 - Most recent events in Sentry as of 21:20:19
- 21:11:09 - Restarted CI while CD is quiet
- 21:14:01 - First smoke test begins
- 21:19:39 - Last smoke test ends
If edns0 doesn't work correctly DNS servers might respond with messages
bigger than our maximum udp size.
In that case we need to truncate those messages when forwarding the
respond back to the interface and expect the OS to retry with TCP.
Otherwise we aren't able to allocate a packet big enough for this.
Fixes#7121
---------
Co-authored-by: Thomas Eizinger <thomas@eizinger.io>
This should give us much more context for a particular error without
having to bother a customer with sending us the logs / digging for them
ourselves in our staging or production environment.
Resolves: #7176.
In order to release #6941, we need to bump the gateway's version to
1.4.0. The portal has a version gate that only allows connection clients
which have version >= 1.4.0. Thus, in order to test #6941 on staging,
the version must not yet be bumped and is thus split out into this PR.
We introduced a boolean bug in #7163 that causes us to attempt to add
host candidates much more often than necessary. This spams the logs on
DEBUG level but was otherwise not harmful.
During normal operation, we should never lose connectivity to the set of
assigned relays in a client or gateway. In the presence of odd network
conditions and partitions however, it is possible that we disconnect
from a relay that is in fact only temporarily unavailable. Without an
explicit mechanism to retrieve new relays, this means that both clients
and gateways can end up with no relays at all. For clients, this can be
fixed by either roaming or signing out and in again. For gateways, this
can only be fixed by a restart!
Without connected relays, no connections can be established. With #7163,
we will at least be able to still establish direct connections. Yet,
that isn't good enough and we need a mechanism for restoring full
connectivity in such a case.
We creating a new connection, we already sample one of our relays and
assign it to this particular connection. This ensures that we don't
create an excessive amount of candidates for each individual connection.
Currently, this selection is allowed to be silently fallible. With this
PR, we make this a hard-error and bubble up the error that all the way
to the client's and gateway's event-loop. There, we initiate a reconnect
to the portal as a compensating action. Reconnecting to the portal means
we will receive another `init` message that allows us to reconnect the
relays.
Due to the nature of this implementation, this fix may only apply with a
certain delay from when we actually lost connectivity to the last relay.
However, this design has the advantage that we don't have to introduce
an additional state within `snownet`: Connections now simply fail to
establish and the next one soon after _should_ succeed again because we
will have received a new `init` message.
Resolves: #7162.
As part of maintaining an allocation, we also perform STUN with our
relays to discover our server-reflexive address. At the moment, these
candidates are scoped to an `Allocation`. This is unnecessarily
restrictive. Similar to host candidates, server-reflexive candidate
entirely depend on the socket you send data from and are thus
independent of the allocation's state.
During normal operation, this doesn't really matter because all relay
traffic is sent through the same sockets so all `Allocation`s end up
with the same server-reflexive candidates. Where this does matter is
when we disconnect from relay's for one reason or another (for example:
#7162). The fact that all but host-candidates are scoped to
`Allocation`s means that without `Allocation`s, we cannot make any new
connections, not even direct ones. This is unnecessarily restrictive and
causes bugs within `Allocation` to have a bigger blast radius than
necessary.
With this PR, we keep server-reflexive candidates in the same set as
host candidates. This allows us to at least establish direct connections
in case something is wrong with the relays or our state tracking of
relays on the client side.
It appears that I have misunderstood the documentation of
`sentry-tracing`. When a message gets logged as an event (rather than an
"exception") `std::error::Error`s attached as tracing `Value`s do not
get recorded. It doesn't really matter whether we record our events as
exceptions or messages. We should ideally look at all of them and
particularly noisy ones can be muted forever in Sentry so we don't end
up in a "boy who cried wolf" situation. Therefore, this PR changes our
event filter to also submit WARNs as exceptions to make sure they get
logged accordingly.
Resolves: #7161.
Related: https://github.com/getsentry/sentry-rust/issues/702.
In order to make Firezone more mobile-friendly, waking up the CPU less
often is key. In #6845, we introduced a low-power mode into `snownet`
that sends STUN messages on a longer interval if the connection is idle.
Whilst poking around `boringtun` as part integrating our fork into the
main codebase, I noticed that we are updating `boringtun`'s timers every
second - even on idle connections.
This PR applies the same idea of #6845 to the timers within `Node`: Idle
connections get "woken" less and if all connections are idle, we avoid
waking the `Node` altogether (unless we need to refresh allocations /
channels).
Calling `handle_timeout` less often revealed an issue in the tests where
we didn't fully process the state changes after invalidating a candidate
from the remote. To fix this, we now call `handle_timeout` directly
after `{add,remove}_remote_candidate`. This isn't super clean because at
first glance, it looks like `handle_timeout` should just be part of the
add/remove candidate function. It is quite common for sans-IO designs to
require calling `handle_timeout` after state has been changed. In
`{Client,Server}Node`, we do it implicitely so that we don't have to do
it in the tests and the event-loop.
It would be great to test this in some automated fashion but I haven't
figured out how yet. I did temporarily add an `info!` log to the
event-loop of the client and with this patch applied, the entire
event-loop goes to sleep on idle connections. It still does get woken
every now and then but no longer every second!
With the introduction of the `tracing-sentry` integration in #7105, we
started sending tracing spans to Sentry. By default, all spans with
level INFO and above get sampled at the configured rate and sent to
Sentry.
This results in a lot of useless transaction in Sentry because we use
INFO level spans in multiple places in connlib to attach contextual
information like the current connection ID.
This PR introduces the concept of `telemetry` spans which - similar to
the `telemetry` log target in #7147 - qualifies a span for being sent to
Sentry. By convention, these are also defined as requiring the TRACE
level. This ensures we won't ever see them as part of regular log
output.
As a first step for integration Sentry into the Android app, we launch
the Sentry Rust agent as soon as a `connlib` session starts up. At a
later point, we can also integrate Sentry into the Android app itself
using the Java / Kotlin SDK.
---------
Signed-off-by: Thomas Eizinger <thomas@eizinger.io>
This starts up telemetry together with each `connlib` session. At a
later point, we can also integrate the native Swift SDK into the MacOS /
iOS app to catch non-connlib specific problems.
---------
Signed-off-by: Thomas Eizinger <thomas@eizinger.io>
Closes#4883
Refs #7005
Adds support for Ubuntu 24.04, drops support for Ubuntu 20.04
Known issues:
- On Ubuntu 22.04, sometimes GNOME shows the wrong tray icon
- On Ubuntu 24.04, the first time you open the tray menu, GNOME takes a
long time to open the menu.
---------
Signed-off-by: Reactor Scram <ReactorScram@users.noreply.github.com>
With #7105, all ERROR events from `tracing` get logged as exceptions in
Sentry and all WARN events get logged as "messages". We don't want to
fill up the user's harddrive with logs which means we have to be
somewhat conservative, what gets logged on INFO and above (with INFO
being the default log level). There are certain events though where it
would be useful to know, how often they happen because too many of them
can indicate a problem.
To solve this problem, we introduce a dedicated `telemetry` log target
that the tracing-sentry integration layer watches for. Events for the
`telemetry` log target that gets logged on TRACE will be sampled at a
rate of 1% and submitted as messages to Sentry.
Now that we have Sentry integrated with `tracing`, using `warn!` logs a
bit more liberally allows us to detect edge-cases that customers might
run into.
All the logs touched in this PR represent some kind of problem that it
would be good to know about.
Using a trait means the call-site of the log message will always be the
`log_unwrap` module, despite the `#[track_caller]` annotation. That one
only works for `std::panic::Location` unfortunately which `tracing`
isn't using.
Macros will be evaluated earlier and thus the messages will show up with
the correct module name.
Applications may query domains for HTTPS RR using the HTTPS record type.
`connlib` operates on OSI layer 3 and thus can only hand out IPs for the
particular domains. The correct way to signal this to applications is to
answer the HTTPS query with NOERROR and return an empty set of records.
[RFC9460](https://www.rfc-editor.org/rfc/rfc9460.html#name-client-behavior)
says the following:
> 4. If one or more "compatible" ([Section
8](https://www.rfc-editor.org/rfc/rfc9460.html#mandatory)) ServiceMode
records are returned, these represent the alternative endpoints. Sort
the records by ascending SvcPriority.
> 5. Otherwise, SVCB resolution has failed, and the list of available
endpoints is empty.
This implies that returning no records is valid behaviour and forces the
client to consider the HTTPS DNS query as failed and query for A / AAAA
records instead (if it didn't do so via happy-eyeballs already).
My theory for this issue is that we receive a UDP DNS response from an
upstream server that is bigger than our MTU and thus forwarding it
fails.
This PR doesn't fix that issue by itself but only mitigates the actual
panic. To properly fix the underlying issue, we need to parse the DNS
message. Truncate it and set the TC bit.
Related: #7121.
The `fmt::Display` implementation of `tokio::task::JoinError` already
does exactly what we do here: Extracting the panic message if there is
one. Thus, we can simplify this code why just moving the `JoinError`
into the `DisconnectError` as its source.
For simplicity sake I assumed that any packet using port 53 would be a
dns packet, but I forgot to exclude it from the range of possible ports
used.
This can cause spurious CI failures
Using the `sentry-tracing` integration, we can automatically capture
events based on what we log via `tracing`. The mapping is defined as
follows:
- ERROR: Gets captured as a fatal error
- WARN: Gets captured as a message
- INFO: Gets captured as a breadcrumb
- `_`: Does not get captured at all
If telemetry isn't active / configured, this integration does nothing.
It is therefore safe to just always enable it.
Similar to the GUI and headless clients, adding error reporting via
Sentry should give us much better insight into how well gateways are
performing.
Resolves: #7099.
---------
Signed-off-by: Thomas Eizinger <thomas@eizinger.io>
Co-authored-by: Jamil <jamilbk@users.noreply.github.com>
- We don't need to control our deb's deps since we're sticking with
Tauri
- Specifying `pnpm tauri` fixes an odd issue on one dev system
- `tauri-cli` is a dev dep, not a runtime dep
Flushing events to Sentry requires us to be able to resolve domain
names. This is only possible while connlib is active or completely
disabled.
Without this, stopping telemetry pretty much always times out for me on
my local machine when using the headless-client.
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.129 to
1.0.132.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/serde-rs/json/releases">serde_json's
releases</a>.</em></p>
<blockquote>
<h2>1.0.132</h2>
<ul>
<li>Improve binary size and compile time for JSON array and JSON object
deserialization by about 50% (<a
href="https://redirect.github.com/serde-rs/json/issues/1205">#1205</a>)</li>
<li>Improve performance of JSON array and JSON object deserialization by
about 8% (<a
href="https://redirect.github.com/serde-rs/json/issues/1206">#1206</a>)</li>
</ul>
<h2>1.0.131</h2>
<ul>
<li>Implement Deserializer and IntoDeserializer for <code>Map<String,
Value></code> and <code>&Map<String, Value></code> (<a
href="https://redirect.github.com/serde-rs/json/issues/1135">#1135</a>,
thanks <a
href="https://github.com/swlynch99"><code>@swlynch99</code></a>)</li>
</ul>
<h2>1.0.130</h2>
<ul>
<li>Support converting and deserializing <code>Number</code> from i128
and u128 (<a
href="https://redirect.github.com/serde-rs/json/issues/1141">#1141</a>,
thanks <a
href="https://github.com/druide"><code>@druide</code></a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="86d933cfd7"><code>86d933c</code></a>
Release 1.0.132</li>
<li><a
href="f45b422a3b"><code>f45b422</code></a>
Merge pull request <a
href="https://redirect.github.com/serde-rs/json/issues/1206">#1206</a>
from dtolnay/hasnext</li>
<li><a
href="f2082d2a04"><code>f2082d2</code></a>
Clearer order of comparisons</li>
<li><a
href="0f54a1a0df"><code>0f54a1a</code></a>
Handle early return sooner on eof in seq or map</li>
<li><a
href="2a4cb44f7c"><code>2a4cb44</code></a>
Rearrange 'match peek'</li>
<li><a
href="4cb90ce66d"><code>4cb90ce</code></a>
Merge pull request <a
href="https://redirect.github.com/serde-rs/json/issues/1205">#1205</a>
from dtolnay/hasnext</li>
<li><a
href="b71ccd2d8f"><code>b71ccd2</code></a>
Reduce duplicative instantiation of logic in SeqAccess and
MapAccess</li>
<li><a
href="a810ba9850"><code>a810ba9</code></a>
Release 1.0.131</li>
<li><a
href="0d084c5038"><code>0d084c5</code></a>
Touch up PR 1135</li>
<li><a
href="b4954a9561"><code>b4954a9</code></a>
Merge pull request <a
href="https://redirect.github.com/serde-rs/json/issues/1135">#1135</a>
from swlynch99/map-deserializer</li>
<li>Additional commits viewable in <a
href="https://github.com/serde-rs/json/compare/1.0.129...1.0.132">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Our logging library, `tracing` supports structured logging. This is
useful because it preserves the more than just the string representation
of a value and thus allows the active logging backend(s) to capture more
information for a particular value.
In the case of errors, this is especially useful because it allows us to
capture the sources of a particular error.
Unfortunately, recording an error as a tracing value is a bit cumbersome
because `tracing::Value` is only implemented for `&dyn
std::error::Error`. Casting an error to this is quite verbose. To make
it easier, we introduce two utility functions in `firezone-logging`:
- `std_dyn_err`
- `anyhow_dyn_err`
Tracking errors as correct `tracing::Value`s will be especially helpful
once we enable Sentry's `tracing` integration:
https://docs.rs/sentry-tracing/latest/sentry_tracing/#tracking-errors
Currently, tests only send ICMP packets back and forth, to expand our
coverage and later on permit us cover filters and resource picking this
PR implements sending UDP and TCP packets as part of that logic too.
To make this PR simpler in this stage TCP packets don't track an actual
TCP connection, just that they are forwarded back and forth, this will
be fixed in a future PR by emulating TCP sockets.
We also unify how we handle CIDR/DNS/Non Resources to reduce the number
of transitions.
Fixes#7003
---------
Signed-off-by: Gabi <gabrielalejandro7@gmail.com>
Co-authored-by: Thomas Eizinger <thomas@eizinger.io>
When we query upstream DNS servers through the tunnel via TCP DNS, we
will always be successful in establishing a tunnel, regardless of how
many concurrent queries we send because the TCP stack will keep
re-trying. Thus, tracking, which resources we are connected to after
sending a bunch of DNS queries needs to be split by UDP and TCP.
For UDP, only the "first" resource will be connected, however, with
concurrent TCP and UDP DNS queries, "first" isn't necessarily the order
in which we send the queries because with TCP DNS, one packet doesn't
equate to one query anymore.
This is quite hacky but it will get completely deleted once we buffer
packets during the connection setup.
This PR implements the new idempotent control protocol for the gateway.
We retain backwards-compatibility with old clients to allow admins to
perform a disruption-free update to the latest version.
With this new control protocol, we are moving the responsibility of
exchanging the proxy IPs we assigned to DNS resources to a p2p protocol
between client and gateway. As a result, wildcard DNS resources only get
authorized on the first access. Accessing a new domain within the same
resource will thus no longer require a roundtrip to the portal.
Overall, users will see a greatly decreased connection setup latency. On
top of that, the new protocol will allow us to more easily implement
packet buffering which will be another UX boost for Firezone.
At present, `connlib` only supports DNS over UDP on port 53. Responses
over UDP are size-constrained on the IP MTU and thus, not all DNS
responses fit into a UDP packet. RFC9210 therefore mandates that all DNS
resolvers must also support DNS over TCP to overcome this limitation
[0].
Handling UDP packets is easy, handling TCP streams is more difficult
because we need to effectively implement a valid TCP state machine.
Building on top of a lot of earlier work (linked in issue), this is
relatively easy because we can now simply import
`dns_over_tcp::{Client,Server}` which do the heavy lifting of sending
and receiving the correct packets for us.
The main aspects of the integration that are worth pointing out are:
- We can handle at most 10 concurrent DNS TCP connections _per defined
resolver_. The assumption here is that most applications will first
query for DNS records over UDP and only fall back to TCP if the response
is truncated. Additionally, we assume that clients will close the TCP
connections once they no longer need it.
- Errors on the TCP stream to an upstream resolver result in `SERVFAIL`
responses to the client.
- All TCP connections to upstream resolvers get reset when we roam, all
currently ongoing queries will be answered with `SERVFAIL`.
- Upon network reset (i.e. roaming), we also re-allocate new local ports
for all TCP sockets, similar to our UDP sockets.
Resolves: #6140.
[0]: https://www.ietf.org/rfc/rfc9210.html#section-3-5
