These are expected during deploys, so don't log them as errors. If the
Supervisor fails to start us after exhausting all attempts, it will log
an error.
The `background_jobs_enabled` config in an ENV var that needs to be set
for a specific configuration key. It's not set on the top-level
`:domain` config by default.
Instead, it's used to enable / disable specific modules to start by the
application's Supervisor.
The `Domain.Events.ReplicationConnection` module is updated in this PR
to follow this convention.
The `web` and `api` application use `domain` as a dependency in their
`mix.exs`. This means by default their Supervisor will start the
Domain's supervision tree as well.
The author did not realize this at the time of implementation, and so we
now leverage the convention in place for restricting tasks to `domain`
nodes, the `background_jobs_enabled` application configuration
parameter.
We also add an info log when the replication slot is being started so we
can verify the node it's starting on.
When deploying, the cluster state diverges temporarily, which allows
more than one `ReplicationConnection` process to start on the new nodes.
(One of) the old nodes still has an active slot, and we get an "object
in use" error `(Postgrex.Error) ERROR 55006 (object_in_use) replication
slot "events_slot" is active for PID 603037`.
Rather than use ReplicationConnection's restart behavior (which logs
tons of errors with Logger.error), we can use the Supervisor here
instead, and continue to try and start the ReplicationConnection until
successful.
Note that if the process name is registered (globally) and running,
ReplicationConnection.start_link/1 simply returns `{:ok, pid}` instead
of erroring out with `:already_running`, so eventually one of the nodes
will succeed and the remaining ones will return the globally-registered
pid.
The LoggerJSON Redactor only redacts top-level keys, so we need to
redact the entire `connection_opts` param to redact its contained
password.
We also don't need to pass around `connection_opts` across the entire
ReplicationConnection process state, only for the initial connection, so
we refactor that out of the `state`.
Firezone's control plane is a realtime, distributed system that relies
on a broadcast/subscribe system to function. In many cases, these events
are broadcasted whenever relevant data in the DB changes, such as an
actor losing access to a policy, a membership being deleted, and so
forth.
Today, this is handled in the application layer, typically happening at
the place where the relevant DB call is made (i.e. in an
`after_commit`). While this approach has worked thus far, it has several
issues:
1. We have no guarantee that the DB change will issue a broadcast. If
the application is deployed or the process crashes after the DB changes
are made but before the broadcast happens, we will have potentially
failed to update any connected clients or gateways with the changes.
2. We have no guarantee that the order of DB updates will be maintained
in order for broadcasts. In other words, app server A could win its DB
operation against app server B, but then proceed to lose being the first
to broadcast.
3. If the cluster is in a bad state where broadcasts may return an error
(i.e. https://github.com/firezone/firezone/issues/8660), we will never
retry the broadcast.
To fix the above issues, we introduce a WAL logical decoder that process
the event stream one message at a time and performs any needed work.
Serializability is guaranteed since we only process the WAL in a single,
cluster-global process, `ReplicationConnection`. Durability is also
guaranteed since we only ACK WAL segments after we've successfully
ingested the event.
This means we will only advance the position of our WAL stream after
successfully broadcasting the event.
This PR only introduces the WAL stream processing system but does not
introduce any changes to our current broadcasting behavior - that's
saved for another PR.
Prevents more than one sync-enabled adapter per account in order to
prepare for eventually adding a unique constraint on
`provider_identifier` for identities and groups per account.
Related: #6294
---------
Signed-off-by: Jamil <jamilbk@users.noreply.github.com>
Co-authored-by: Brian Manifold <bmanifold@users.noreply.github.com>
Our current bespoke job system, while it's worked out well so far, has
the following shortcomings:
- No retry logic
- No robust to guarantee job isolation / uniqueness without resorting to
row-level locking
- No support for cron-based scheduling
This PR adds the boilerplate required to get started with
[Oban](https://hexdocs.pm/oban/Oban.html), the job management system for
Elixir.
There was slight API change in the way LoggerJSON's configuration is
generation, so I took the time to do a little fixing and cleanup here.
Specifically, we should be using the `new/1` callback to create the
Logger config which fixes the below exception due to missing config
keys:
```
FORMATTER CRASH: {report,[{formatter_crashed,'Elixir.LoggerJSON.Formatters.GoogleCloud'},{config,[{metadata,{all_except,[socket,conn]}},{redactors,[{'Elixir.LoggerJSON.Redactors.RedactKeys',[<<"password">>,<<"secret">>,<<"nonce">>,<<"fragment">>,<<"state">>,<<"token">>,<<"public_key">>,<<"private_key">>,<<"preshared_key">>,<<"session">>,<<"sessions">>]}]}]},{log_event,#{meta => #{line => 15,pid => <0.308.0>,time => 1744145139650804,file => "lib/logger.ex",gl => <0.281.0>,domain => [elixir],application => libcluster,mfa => {'Elixir.Cluster.Logger',info,2}},msg => {string,<<"[libcluster:default] connected to :\"web@web.cluster.local\"">>},level => info}},{reason,{error,{badmatch,[{metadata,{all_except,[socket,conn]}},{redactors,[{'Elixir.LoggerJSON.Redactors.RedactKeys',[<<"password">>,<<"secret">>,<<"nonce">>,<<"fragment">>,<<"state">>,<<"token">>,<<"public_key">>,<<"private_key">>,<<"preshared_key">>,<<"session">>,<<"sessions">>]}]}]},[{'Elixir.LoggerJSON.Formatters.GoogleCloud',format,2,[{file,"lib/logger_json/formatters/google_cloud.ex"},{line,148}]}]}}]}
```
Supersedes #8714
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Why:
* The Okta IdP sync job needs to make sure it is always using the latest
access token available. If not, there is the possibility for the job to
take too long to complete and the access token that the job started with
might time out. This commit updates the Okta API client to always check
and make sure it is using the latest access token for each request to
the Okta API.
- Attaches the Sentry Logging hook in each of [api, web, domain]
- Removes errant Sentry logging configuration in config/config.exs
- Fixes the exception logger to default to logging exceptions, use
`skip_sentry: true` to skip
Tested successfully in dev. Hopefully the cluster behaves the same way.
Fixes#8639
The Google API will often return a missing `members` key alongside a
`200` response from their members API. The documentation here isn't
clear whether this key is expected or not, but since the sync has been
working fine up until #8608, we can only surmise that the missing key in
fact means the group has no members.
This PR updates the Google API client so that a `default_if_missing` can
be passed in which is returned if the API response is missing the JSON
key to fetch.
For the users, groups, and organization units fetches, we consider a
missing key to be an error and we return `{:error, :invalid_response}`
since this most likely indicates an API problem.
For the members endpoint, we consider the missing key to be the empty
set.
Additionally, a bug is fixed that was introduced in #8608 whereupon we
returned `{:error, :retry_later}` for newly-accounted-for API responses,
which would have caused a "sync failed" email to be sent to the admins
on the instance.
Instead, we want to return `{:error, :invalid_response}` which will stop
the sync from progressing, and log it internally.
After merging #8608, we discovered that we receive unexpected API
responses on the regular. This adds improved logging to uncover what
exactly these unexpected API responses are.
When syncing identities from an identity, we have logic in place that
resurrects any soft-deleted identities in order to maintain their
session history, group memberships and any other relevant data. Users
can be temporarily suspended from their identity provider and then
resumed.
Groups, however, based on cursory research, can never be temporarily
suspended at the identity provider. However, this doesn't mean that we
can't see the group disappear and reappear at a later point in time.
This can happen due to a temporary sync issue, or in the upcoming Group
Filters PR: #8381.
This PR adds more robust testing to ensure we can in fact resurrect
identities as expected.
It also updates the group sync logic to similarly resurrect soft-deleted
groups if they are seen again in a subsequent sync.
To achieve this, we need to update the `UNIQUE CONSTRAINT` used in the
upsert clause during the sync. Before, it was possible for two (or more)
groups to exist with the same provider_identifier and provider_id, if
`deleted_at IS NOT NULL`. Now, we need to ensure that only one group
with the same `account_id, provider_id, provider_identifier` can exist,
since we want to resurrect and not recreate these.
To do this, we use a migration that does the following:
1. Ensures any potentially problematic data is permanently deleted
2. Drops the existing unique constraint
3. Recreates it, omitting `WHERE DELETED_AT IS NULL` from the partial
index.
Based on exploring the production DB data, this should not cause any
issues, but it would be a good idea to double-check before rolling this
out to prod.
Lastly, the final missing piece to the resurrection story is Policies.
This is saved for a future PR since we need to first define the
difference between a policy that was soft-deleted via a sync job, and a
policy that was "perma" deleted by a user.
Related: #8187
When calling the various directory sync endpoints, we had error cases
that matched a few of the possible error scenarios in an appropriate way
by returning either `{:error, :retry_later}` or the `{:error, ...}`
tuples.
However, as we've recently learned in [this
thread](https://firezonehq.slack.com/archives/C069H865MHP/p1743521884037159),
it's possible for identity provider APIs to return all kinds of bogus
data here, and we need a more defensive approach.
The specific issue this PR addresses is the case where we receive a
`2xx` response, but without the expected JSON key in the response body.
That will result in the `list*` functions returning an empty list, which
the calling code paths then use to soft-delete all existing record types
in the DB.
This is wrong. If the JSON response is missing a key we're expecting, we
instead log a warning and return `{:error, :retry_later}`. It's
currently unknown when exactly this happens and why, but with better
monitoring here we'll have a much better picture as to why.
When reading through these modules, it's helpful to know that the actual
sync data update doesn't occur more often than 10 minutes due to a
database check.
When a customer signs up for Starter or Team, we don't enable tax
calculation by default. This means customers can upgrade to Team, start
paying invoices, and we won't collect taxes.
This creates a management issue and possible tax liability since I need
to manually reconcile these.
Instead, since we have Stripe Tax configured on our account, we can
enable automatic tax calculation when the subscription is created. Any
products (Starter/Team/Enterprise) therefore in the subscription will
automatically collect tax appropriately.
In most cases in the US, the tax rate is 0. In EU transactions, for B2B
sales, the tax rate for us is also 0 (reverse charge basis). If we sell
a Team subscription to an individual, however, we need to collect VAT.
There doesn't seem to be a way to block consumer EU transactions in
Stripe, so we'll likely need to register for VAT in the EU if we cross
the reporting threshold.
A regression was introduced in d0f0de0f8d
whereupon we started using the updated policy record for broadcasting
the `delete_policy` and `expire_flows` events. This caused a security
issue because if the actor group changed from `Everyone` to `thomas`,
for example, we'd only expire flows and broadcast policy removal (i.e.
resource removal) events for `thomas`, and `Everyone` would still have
access granted by the old policy.
To fix this, we broadcast the destructive events to the old policy, so
that its `actor_group_id` and `resource_id` are used, and not the new
policy's.
Fixes#8549
Why:
* This commit will allow account admins to send a request through the
Firezone portal to schedule a deletion of their account, rather than
having the account admins email their request manually. Doing this
through the portal allows us to verify that the request actually came
from an admin of the account.
I was debugging some of this just now and realized our naming / comments
are incorrect here, so thought I'd open a PR to tidy things up for the
next person reading this.
Resource CIDRs actually occupy the `100.96.0.0/11` range (and IPv6
equivalent), but the portal doesn't generate these.
Why:
* Previously, when running a directory sync with the Google Workspace
IdP adapter, if a service account had been configured but there was a
problem getting an access token for the service account, the sync job
would fall back to using a personal access token. We no longer want to
rely on any personal access token once a service account has been
configured. This commit will make sure that if a service account is
configured there is no way to fall back to any personal access token.
Fixes#8409
Finishes up the Internet Resource migration by enforcing:
- No internet resources in non-internet sites
- No regular resources in internet sites
- Removing the prompt to migrate
~~I've already migrated the existing internet resources in customer's
accounts. No one that was using the internet resource hadn't already
migrated.~~
Edit: I started to head down that path, then decided doing this here in
a data migration was going to be a better approach.
Fixes#8212
Introduces a simple `search_domain` field embed into our existing
`Accounts.Account.Config` embedded schema. This will be sent to clients
to append to single-label DNS queries.
UI and API changes will come in subsequent PRs: this one adds field and
(lots of) validations only.
Related: #8365
- Adds more actor groups to the existing `oidc_provider`
- Configures a rand seed so our seed data is reproducible across
machines
- Formats the seeds file to allow for some refactoring a later PR
- Adds a `Mock` identity provider adapter with sync enabled
Adds the following endpoints:
- `PUT /clients/:id` for updating the `name`
- `PUT /clients/:client_id/verify` for verifying a client
- `PUT /clients/:client_id/unverify` for unverifying a client
- `GET /clients` for listing clients in an account
- `GET /clients/:id` for getting a single client
- `DELETE /clients/:id` for deleting a client
Related: #8081
If the websocket connection between a relay and the portal experiences a
temporary network split, the portal will immediately send the
disconnected id of the relay to any connected clients and gateways, and
all relayed connections (and current allocations) will be immediately
revoked by connlib.
This tight coupling is needlessly disruptive. As we've seen in staging
and production logs, relay disconnects can happen randomly, and in the
vast majority of cases immediately reconnect. Currently we see about 1-2
dozen of these **per day**.
To better account for this, we introduce a debounce mechanism in the
portal for `relays_presence` disconnects that works as follows:
- When a relay disconnects, record its `stamp_secret` (this is somewhat
tricky as we don't get this at the time of disconnect - we need to cache
it by relay_id beforehand)
- If the same `relay_id` reconnects again with the same `stamp_secret`
within `relays_presence_debounce_timeout` -> no-op
- If the same `relay_id` reconnects again with a **different**
`stamp_secret` -> disconnect immediately
- If it doesn't reconnect, **then** send the `relays_presence` with the
disconnected_id after the `relays_presence_debounce_timeout`
There are several ways connlib detects a relay is down:
1. Binding requests time out. These happen every 25s, so on average we
don't know a Relay is down for 12.5s + backoff timer.
2. `relays_presence` - this is currently the fastest way to detect
relays are down. With this change, the caveat is we will now detect this
with a delay of `relays_presence_debounce_timer`.
Fixes#8301
We had a number of validation issues:
- DNS resources allow address `1.1.1.1` or `1.1.1.1/32`. These are not
valid and will cause issues during resolution.
- IP resources were allowing basically any string character on `edit`
caused by a logic bug in the changeset
- CIDR resources, same as above
- `*.*.*.*.google.com` and similar DNS wildcard resources were not
allowed
This PR beefs all of those up so that we have a higher degree of
certainty that our data is valid. If invalid data reaches connlib, it
will cause a panic.
This PR also introduces a migration to migrate any invalid resources
into the proper format in the DB.
Fixes#8287
Why:
* Rather than using a persistent_id field in Resources/Policies, it was
decided that we should allow "breaking changes" to these entities. This
means that Resources/Policies will now be able to update all fields on
the schema without changing the primary key ID of the entity.
* This change will greatly help the API and Terraform provider
development.
@jamilbk, would you like me to put a migration in this PR to actually
get rid of all of the existing soft deleted entities?
@thomaseizinger, I tagged you on this, because I wanted to make sure
that these changes weren't going to break any expectations in the client
and/or gateways.
---------
Signed-off-by: Brian Manifold <bmanifold@users.noreply.github.com>
Co-authored-by: Jamil <jamilbk@users.noreply.github.com>
Currently, it would theoretically be possible for an admin to connect
non-internet Resources to the Internet site. This PR fixes that by
enforcing only the `internet` Resource type can belong to the `Internet`
gateway group.
Related: #6834
This PR fixes two issues:
1. Since we weren't updating any actual fields in the telemetry reporter
log record, it was never being updated, thus optimistic locking was not
taking effect. To fix this, we use `Repo.update(force: true)`.
2. If a buffer is full, we write immediately, but we provider an empty
`%Log{}` which causes a repetitive `the current value of last_flushed_at
is nil and will not be used as a filter for optimistic locking.`
Some customers have already picked the `Internet` name, which is making
our migrations fail.
This scopes the unique name index by `managed_by` so that our attempts
to create them succeed.
By specifying the `before_send` hook, we can easily drop events based on
their data, such as `original_exception` which contains the original
exception instance raised.
Leveraging this, we can add a `report_to_sentry` parameter to
`Web.LiveErrors.NotFound` to optionally ignore certain not found errors
from going to Sentry.
When flushing metrics to GCP, we sometimes get the following error:
```
{400, "{\n \"error\": {\n \"code\": 400,\n \"message\": \"One or more TimeSeries could not be written: timeSeries[0-51]: write for resource=gce_instance{zone:us-east1-d,instance_id:6130184649770384727} failed with: One or more points were written more frequently than the maximum sampling period configured for the metric.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.monitoring.v3.CreateTimeSeriesSummary\",\n \"totalPointCount\": 52,\n \"successPointCount\": 48,\n \"errors\": [\n {\n \"status\": {\n \"code\": 9\n },\n \"pointCount\": 4\n }\n ]\n }\n ]\n }\n}\n"}
```
It would be helpful to know exactly which metrics are failing to flush
so we can further troubleshoot any issues.
