Commit Graph

935 Commits

Author SHA1 Message Date
dependabot[bot]
fc256d7e37 build(deps): bump phoenix_ecto from 4.6.4 to 4.6.5 in /elixir (#9755)
Bumps [phoenix_ecto](https://github.com/phoenixframework/phoenix_ecto)
from 4.6.4 to 4.6.5.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/phoenixframework/phoenix_ecto/blob/main/CHANGELOG.md">phoenix_ecto's
changelog</a>.</em></p>
<blockquote>
<h2>v4.6.5</h2>
<ul>
<li>Bug fixes
<ul>
<li>Unallow existing allowances when attempting to allow a Plug to
access a connection</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/phoenixframework/phoenix_ecto/commits/v4.6.5">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=phoenix_ecto&package-manager=hex&previous-version=4.6.4&new-version=4.6.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-07-03 23:05:48 +00:00
Brian Manifold
83e71f45b8 fix(portal): catch all errors when sending welcome email (#9776)
Why:

* We were previously only catching the `:rate_limited` error when
sending welcome emails. This update adds a catch-all case to gracefully
handle the error and alert us.

---------

Signed-off-by: Brian Manifold <bmanifold@users.noreply.github.com>
Co-authored-by: Jamil <jamilbk@users.noreply.github.com>
2025-07-03 21:41:12 +00:00
dependabot[bot]
ad126f72ba build(deps): bump nimble_csv from 1.2.0 to 1.3.0 in /elixir (#9757)
Bumps [nimble_csv](https://github.com/dashbitco/nimble_csv) from 1.2.0
to 1.3.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/dashbitco/nimble_csv/blob/master/CHANGELOG.md">nimble_csv's
changelog</a>.</em></p>
<blockquote>
<h2>v1.3.0 (2025-06-24)</h2>
<ul>
<li>Require Elixir 1.15+</li>
<li>Add <code>generated: true</code> to <code>newlines_separator</code>
macro</li>
<li>Fix warnings on Elixir 1.20+</li>
<li>Document OWASP official recommendations for CSV injections</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="2fc3cbf4b5"><code>2fc3cbf</code></a>
Release v1.3.0</li>
<li><a
href="1c3b383dd9"><code>1c3b383</code></a>
Update ex_doc</li>
<li><a
href="da38c6ac6b"><code>da38c6a</code></a>
Document OWASP official recommendations for CSV injections (<a
href="https://redirect.github.com/dashbitco/nimble_csv/issues/86">#86</a>)</li>
<li><a
href="2b4a5e0da6"><code>2b4a5e0</code></a>
Support Elixir 1.18 in CI (<a
href="https://redirect.github.com/dashbitco/nimble_csv/issues/84">#84</a>)</li>
<li><a
href="020ea8297e"><code>020ea82</code></a>
Update docs</li>
<li><a
href="106e298ec7"><code>106e298</code></a>
Fix warnings on Elixir 1.20+ (<a
href="https://redirect.github.com/dashbitco/nimble_csv/issues/83">#83</a>)</li>
<li><a
href="57e0858100"><code>57e0858</code></a>
CI housekeeping (<a
href="https://redirect.github.com/dashbitco/nimble_csv/issues/82">#82</a>)</li>
<li><a
href="8cf912e204"><code>8cf912e</code></a>
Add generated to newlines_separator macro (<a
href="https://redirect.github.com/dashbitco/nimble_csv/issues/81">#81</a>)</li>
<li><a
href="3b2e2f7f2a"><code>3b2e2f7</code></a>
Document parse error, closes <a
href="https://redirect.github.com/dashbitco/nimble_csv/issues/78">#78</a></li>
<li><a
href="0b2390440a"><code>0b23904</code></a>
Add benchmarks using benchee (<a
href="https://redirect.github.com/dashbitco/nimble_csv/issues/76">#76</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/dashbitco/nimble_csv/compare/v1.2.0...v1.3.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=nimble_csv&package-manager=hex&previous-version=1.2.0&new-version=1.3.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-07-01 17:13:03 +00:00
dependabot[bot]
afbadf1acb build(deps-dev): bump mix_audit from 2.1.4 to 2.1.5 in /elixir (#9753)
Bumps [mix_audit](https://github.com/mirego/mix_audit) from 2.1.4 to
2.1.5.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/mirego/mix_audit/blob/main/CHANGELOG.md">mix_audit's
changelog</a>.</em></p>
<blockquote>
<h2>2.1.5 (2025-06-09)</h2>
<ul>
<li>Update dependencies</li>
<li>Use <code>System.stop/1</code> instead of
<code>System.halt/1</code></li>
<li>Use <code>System.user_home()</code> instead of
<code>System.get_env(&quot;HOME&quot;)</code></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="a3a6c1cbf4"><code>a3a6c1c</code></a>
v2.1.5</li>
<li><a
href="aa630f7e91"><code>aa630f7</code></a>
Update CHANGELOG.md</li>
<li><a
href="14f3511a92"><code>14f3511</code></a>
Bump ex_doc from 0.38.1 to 0.38.2</li>
<li><a
href="14698645b0"><code>1469864</code></a>
refactor: use <code>System.stop/1</code> to enable caller to rescue
tasks</li>
<li><a
href="5b43bcb989"><code>5b43bcb</code></a>
Bump ex_doc from 0.37.3 to 0.38.1</li>
<li><a
href="572a0bfdb3"><code>572a0bf</code></a>
Bump ex_doc from 0.34.2 to 0.37.3</li>
<li><a
href="7247db4543"><code>7247db4</code></a>
Bump jason from 1.4.3 to 1.4.4</li>
<li><a
href="11580d4125"><code>11580d4</code></a>
Change home to platform independent function</li>
<li><a
href="41c96c476a"><code>41c96c4</code></a>
Revert a commit that was temporary to test something locally 🤦‍♂️</li>
<li><a
href="876645af75"><code>876645a</code></a>
Use latest Ubuntu and support Elixir 1.18 and 1.17 in CI</li>
<li>Additional commits viewable in <a
href="https://github.com/mirego/mix_audit/compare/v2.1.4...v2.1.5">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=mix_audit&package-manager=hex&previous-version=2.1.4&new-version=2.1.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-07-01 17:12:43 +00:00
dependabot[bot]
c700d30371 build(deps): bump ecto_sql from 3.12.1 to 3.13.2 in /elixir (#9748)
Bumps [ecto_sql](https://github.com/elixir-ecto/ecto_sql) from 3.12.1 to
3.13.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/elixir-ecto/ecto_sql/blob/master/CHANGELOG.md">ecto_sql's
changelog</a>.</em></p>
<blockquote>
<h2>v3.13.2 (2025-06-24)</h2>
<h3>Enhancements</h3>
<ul>
<li>[sandbox] Allow passing through opts in
<code>Ecto.Adapters.SQL.Sandbox.allow/4</code> calls</li>
<li>[sql] Add support for <code>ON DELETE SET DEFAULT</code></li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>[postgres] Fix nested array generated time columns</li>
</ul>
<h2>v3.13.1 (2025-06-20)</h2>
<h3>Bug fixes</h3>
<ul>
<li>[postgres] Fix nested array generated columns</li>
</ul>
<h2>v3.13.0 (2025-06-18)</h2>
<h3>Enhancements</h3>
<ul>
<li>[Ecto.Migration] Add support for index directions</li>
<li>[sql] Support <code>:log_stacktrace_mfa</code> for filtering or
modifying stacktrace-derived info in query logs</li>
<li>[mysql] Support arrays using JSON for MariaDB</li>
<li>[mysql] Allow to specify <code>:prepare</code> per operation</li>
<li>[postgres] Add support for collations in Postgres</li>
<li>[postgres] Allow source fields in
<code>json_extract_path</code></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="cf5080c1a4"><code>cf5080c</code></a>
Release v3.13.2</li>
<li><a
href="b87638180f"><code>b876381</code></a>
Refactor generated handling in column_type</li>
<li><a
href="62603f88b6"><code>62603f8</code></a>
Fix generated nested time array (<a
href="https://redirect.github.com/elixir-ecto/ecto_sql/issues/680">#680</a>)</li>
<li><a
href="701c99e97f"><code>701c99e</code></a>
Add support for <code>ON DELETE SET DEFAULT</code> (<a
href="https://redirect.github.com/elixir-ecto/ecto_sql/issues/677">#677</a>)</li>
<li><a
href="79590224dc"><code>7959022</code></a>
Allow passing through opts in Ecto.Adapters.SQL.Sandbox.allow/4 calls
(<a
href="https://redirect.github.com/elixir-ecto/ecto_sql/issues/678">#678</a>)</li>
<li><a
href="22c71121b7"><code>22c7112</code></a>
Release v3.13.1</li>
<li><a
href="35e27985ec"><code>35e2798</code></a>
Fix nested array generated columns (<a
href="https://redirect.github.com/elixir-ecto/ecto_sql/issues/676">#676</a>)</li>
<li><a
href="955f0fbf8f"><code>955f0fb</code></a>
Release v3.13.0</li>
<li><a
href="aa9a3291f7"><code>aa9a329</code></a>
Remove unused argument from private helper (<a
href="https://redirect.github.com/elixir-ecto/ecto_sql/issues/672">#672</a>)</li>
<li><a
href="3084d7150d"><code>3084d71</code></a>
Better docs for Repos that use <code>Ecto.Adapters.SQL.Adapter</code>
(<a
href="https://redirect.github.com/elixir-ecto/ecto_sql/issues/671">#671</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/elixir-ecto/ecto_sql/compare/v3.12.1...v3.13.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=ecto_sql&package-manager=hex&previous-version=3.12.1&new-version=3.13.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-07-01 17:10:59 +00:00
Jamil
29d8881c54 fix(seeds): remove unused vars (#9731)
This fixes some warnings introduced by #9692.
2025-06-30 19:33:11 +00:00
Jamil
865442eabe chore(portal): increase queue_target on bg nodes to 5000 (#9732)
The domain nodes that run background jobs aren't user-facing and as
such, don't need short queue_interval times. They can afford to wait a
few seconds for a connection from the pool to become available.

This should be a relatively low-risk change as it does not increase the
resources used on the app server and database, it only relaxes the
length of time we want for a connection to be available for us to use.
2025-06-30 19:33:01 +00:00
Jamil
23c43c12dd chore(portal): log wal status every 60s (#9729)
It would be helpful to see these more often in the logs to better
understand our current processing position.
2025-06-30 18:25:19 +00:00
Jamil
972ece507d chore(portal): downgrade expected wal log to info (#9726)
This is expected during deploys so we downgrade it to info to avoid
sending to Sentry.
2025-06-30 15:35:35 +00:00
Jamil
47fe7b388e chore(portal): ack WAL records more often (#9703)
- log connection module for replication manager logs
- simplify bypass conditionals
- ACK write messages to avoid PG resending data
2025-06-28 20:34:26 +00:00
Jamil
a24f582ff5 fix(portal): increase change_log lag warning threshold (#9702)
This is needlessly short and has already tripped a false alarm once.
2025-06-27 20:58:58 +00:00
Jamil
3760536afd chore(portal): add unique index to lsn (#9699) 2025-06-27 20:58:20 +00:00
Jamil
dddd1b57fc refactor(portal): remove flow_activities (#9693)
This has been dead code for a long time. The feature this was meant to
support, #8353, will require a different domain model, views, and user
flows.

Related: #8353
2025-06-27 20:40:25 +00:00
Jamil
9655dacc04 fix(portal): restart wal connection from manager proc (#9701)
When the ReplicationConnection dies, its Manager will die too on all
other nodes, and all domain Application supervisors on all nodes will
attempt to restart them. This allows the connection to migrate to a
healthy node automagically.

However, the default Supervisor behavior is to allow 3 restarts in 5
seconds before the whole tree is taken down. To prevent this, we trap
the exit in the ReplicationManager and attempt to reconnect right away,
beginning the backoff process.
2025-06-27 20:40:04 +00:00
Jamil
6c0a62aa73 fix(tests): wait for visible els before click (#9697)
We had an old bug in one of our acceptance tests that is just now being
hit again due to the faster runners.

- We need to wait for the dropdown to become visible before clicking
- We fix a minor timer issue that was calculating elapsed time
incorrectly when determining when time out finding an el.
2025-06-27 19:06:59 +00:00
Jamil
3247b7c5d2 fix(portal): don't log soft-deleted deletes (#9698) 2025-06-27 19:06:45 +00:00
Jamil
0b09d9f2f5 refactor(portal): don't rely on flows.expires_at (#9692)
The `expires_at` column on the `flows` table was never used outside of
the context in which the flow was created in the Client Channel. This
ephemeral state, which is created in the `Domain.Flows.authorize_flow/4`
function, is never read from the DB in any meaningful capacity, so it
can be safely removed.

The `expire_flows_for` family of functions now simply reads the needed
fields from the flows table in order to broadcast `{:expire_flow,
flow_id, client_id, resource_id}` directly to the subscribed entities.

This PR is step 1 in removing the reliance on `Flows` to manage
ephemeral access state. In a subsequent PR we will actually change the
structure of what state is kept in the channel PIDs such that reliance
on this Flows table will no longer be necessary.

Additionally, in a few places, we were referencing a Flows.Show view
that was never available in production, so this dead code has been
removed.

Lastly, the `flows` table subscription and associated hook processing
has been completely removed as it is no longer needed. We've implemented
in #9667 logic to remove publications from removed table subscriptions,
so we can expect to get a couple ingest warnings when we deploy this as
the `Hooks.Flows` processor no longer exists, and the WAL data may have
lingering flows records in the queue. These can be safely ignored.
2025-06-27 18:29:12 +00:00
Jamil
fbf48a207a chore(portal): handle lag up to 30m (#9681)
Now that we know the bypass system works, it might be a good idea to
allow it to lag data up to 30m so that events accrued during deploys are
not lost.

Also, this PR fixes a small bug where we triggered the threshold _after_
a transaction already committed (`COMMIT`), instead of before the data
came through (`BEGIN`). Since the timestamps are identical (see below),
it would be more accurate to read the timestamp of the transaction
before acting on the data contained within.

```
[(domain 0.1.0+dev) lib/domain/change_logs/replication_connection.ex:4: Domain.ChangeLogs.ReplicationConnection.handle_message/3]
"BEGIN #{commit_timestamp}" #=> "BEGIN 2025-06-26 04:22:45.283151Z"

[(domain 0.1.0+dev) lib/domain/change_logs/replication_connection.ex:4: Domain.ChangeLogs.ReplicationConnection.handle_message/3]
"END #{commit_timestamp}" #=> "END 2025-06-26 04:22:45.283151Z"
```

---------

Signed-off-by: Jamil <jamilbk@users.noreply.github.com>
Co-authored-by: Brian Manifold <bmanifold@users.noreply.github.com>
2025-06-26 13:38:40 +00:00
Jamil
59fa7fa4f1 fix(portal): diff = now - past (#9680)
We were performing the diff backwards, so the bypass never kicked in.
2025-06-25 18:10:28 -07:00
Jamil
e7756a9be5 fix(portal): bypass delayed events past threshold (#9679)
When attempting to process a WAL that's _very_ far behind, it's helpful
to have a time past which we simply `noop` the handlers.
2025-06-25 22:32:15 +00:00
Jamil
eed8343e8f fix(portal): wait 30s for agm query (#9678)
These queries are timing out, so we wait longer for them.
2025-06-25 22:22:21 +00:00
Jamil
42e3027c34 fix(portal): use replication config in dev (#9676) 2025-06-25 21:02:01 +00:00
Jamil
855c427688 chore(portal): don't log found nodes (#9674)
These are better logged elsewhere and this is just noise.
2025-06-25 18:40:48 +00:00
Jamil
9badf1fe7c chore(portal): bump elixir 1.18.4, otp 27.3.4.1 (#9673) 2025-06-25 18:39:20 +00:00
Jamil
bebc69e2bc fix(portal): use distinct slot names (#9672)
These were being configured using the same default `events_` value.
2025-06-25 17:28:17 +00:00
Jamil
343717b502 refactor(portal): broadcast client struct when updated (#9664)
When a client is updated, we may need to re-initialize it if "breaking"
fields are updated. If non-breaking fields are changed, such as name, we
don't need to re-initialize the client.

This PR also adds a helper `struct_from_params/2` which will create a
schema struct from WAL data in order to type cast any needed data for
convenience. This avoid having to do a DB hit - we _already have the
data from the DB_ - we just need to format and send it.

Related: #9501
2025-06-25 17:04:41 +00:00
Jamil
02dd21018d fix(portal): log error when connected_nodes crossed (#9668)
To avoid log spam, we only log an error when the threshold boundary is
crossed.
2025-06-24 21:47:17 -07:00
Jamil
95624211cd fix(portal): update publications when config changes (#9667)
Creating a table publication(s) (and associated replication slot) is
sticky. These will outlive the lifetime of the process that created
them.

We don't want to remove them on shutdown, because this will pause WAL
writing to disk.

However, when starting the _new_ application, it's possible
`table_subscriptions` has changed (such as if we decide we no longer
want events for a certain table). We weren't updating the created
publication(s) with these added/removed tables, so this PR updates the
replication connection setup state machine to pass through a few
conditionals to get these properly updated with the diff of old vs new.
2025-06-24 21:31:40 -07:00
Jamil
a9f49629ae feat(portal): add change_logs table and insert data (#9553)
Building on the WAL consumer that's been in development over the past
several weeks, we introduce a new `change_logs` table that stores very
lightly up-fitted data decoded from the WAL:

- `account_id` (indexed): a foreign key reference to an account.
- `inserted_at` (indexed): the timestamp of insert, for truncating rows
later.
- `table`: the table where the op took place.
- `op`: the operation performed (insert/update/delete)
- `old_data`: a nullable map of the old row data (update/delete)
- `data`: a nullable map of the new row data(insert/update)
- `vsn`: an integer version field we can bump to signify schema changes
in the data in case we need to apply operations to only new or only old
data.

Judging from our prod metrics, we're currently average about 1,000 write
operations a minute, which will generate about 1-2 dozen changelogs / s.
Doing the math on this, 30 days at our current volume will yield about
50M / month, which should be ok for some time, since this is an
append-only, rarely (if ever) read from table.

The one aspect of this we may need to handle sooner than later is
batch-inserting these. That raises an issue though - currently, in this
PR, we process each WAL event serially, ending with the final
acknowledgement `:ok` which will signal to Postgres our status in
processing the WAL.

If we do anything async here, this processing "cursor" then becomes
inaccurate, so we may need to think about what to track and what data we
care about.

Related: #7124
2025-06-25 02:06:20 +00:00
Jamil
ff5a632d2a fix(portal): only show never synced correctly (#9652)
It's confusing that we clear this field upon sync failure. Instead, we
let it track the time of the last sync.

Will be cleaned up in #6294 so just applying a minimal fix now.

Fixes #7715
2025-06-24 22:54:30 +00:00
Jamil
caa21accf9 feat(portal): add mock sync adapter staging (#9660)
This needs to be enabled here too.
2025-06-24 19:08:58 +00:00
Jamil
933d51e3d0 feat(portal): send account_slug in gateway init (#9653)
Adds the `account_slug` to the gateway's `init` message. When the
account slug is changed, the gateway's socket is disconnected using the
same mechanism as gateway deletion, which causes the gateway to
reconnect immediately and receive a new `init`.

Related: #9545
2025-06-24 18:35:06 +00:00
Brian Manifold
27f482e061 fix(portal): trim whitespace in all remaining forms (#9654)
Why:

* After updating the Auth Provider changesets to trim all whitespace
from user editable string fields we realized we needed to do the same
for all forms/entities within Firezone. This commit updates all entities
to trim whitespace on string fields.

Fixes: #9579
2025-06-24 14:28:51 +00:00
Jamil
0cd919a5e2 fix(portal): use account_id index in flow expiration (#9623)
There were a couple more instances where we weren't using the
`account_id` which prevented use of the index, causing a DB Connection
queue drop.
2025-06-23 21:51:21 +00:00
Jamil
f55596be4e fix(portal): index auth_providers on adapter (#9625)
The `refresh_tokens` job for each auth provider uses a cross-account
query that unfortunately hits no indexes. This can cause slow queries
each time the job runs for the adapter.

We add a simple sparse index to speed this query up.

Related:
https://firezone-inc.sentry.io/issues/6346235615/?project=4508756715569152&query=is%3Aunresolved&referrer=issue-stream&stream_index=1
2025-06-23 18:50:22 +00:00
Jamil
0af7582ab6 fix(portal): flush metrics as we accumulate (#9622)
Unfortunately #9608 did not handle the case where we receive more than
200 compressed metrics in a single call. To fix this, we ensure we
`flush` the metrics buffer inside the `reduce` so that we never grow the
accumulated metrics buffer larger than 200 points.

The log string was updated to roll the issue over in Sentry as well as
the old issue was set to delete and destroy to prevent issue spam.

---------

Signed-off-by: Jamil <jamilbk@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-06-23 14:58:18 +00:00
Jamil
c783b23bae refactor(portal): rename conditional->manual (#9612)
These only have one condition - to run manually. `manual migrations`
better implies that these migrations _must_ typically be run manually.
2025-06-21 21:17:33 +00:00
Jamil
2523bedd19 fix(portal): add if not exists to concurrent index (#9611)
With `@disable_ddl_transaction` this needs to be added.

See
https://firezonehq.slack.com/archives/C04HRQTFY0Z/p1750516438992329?thread_ts=1750510766.640919&cid=C04HRQTFY0Z
2025-06-21 15:42:51 +00:00
Jamil
e113def903 fix(portal): flush metrics buffer before exceeding limit (#9608)
Instead of checking for buffer surpass _after_ adding new timeseries to
it, we should check before.

Variables were renamed to be a little more clear on what they represent.
2025-06-20 21:44:52 +00:00
Jamil
a1677494b5 chore(portal): drop index concurrently (#9609)
Looks like postgres does support this, so adding for good measure.
2025-06-20 14:55:23 -07:00
Jamil
975057f9b4 fix(portal): add account_id,type index on actors (#9607)
`Repo.aggregate(:count)` which performs a `COUNT(*)` query should be
relatively fast if it's able to do an index-only scan. For that to
happen we need to ensure all of the fields in the WHERE clause are
indexed. Currently, we're missing an index on `actors.type` so a full
row scan is executed per account each time we calculate Billing limits,
every 5 minutes, for all accounts.

If we need to check these limits more often and/or our data grows in
size, it could be worth moving these to a limits counter field on
`accounts` which is maintained via INSERT/DELETE triggers.

Related:
https://firezone-inc.sentry.io/issues/6346235615/events/588a61860e0b4875a5dbe8531dbb806a/?project=4508756715569152&referrer=next-event
2025-06-20 20:53:09 +00:00
Jamil
6f87f5ea2c fix(portal): use account_id in index for agm hook (#9606)
When reacting to `ActorGroupMembership` updates, we were issuing a query
to expire Flows given an `actor_id, actor_group_id` combination.

Unfortunately, this query never included an `account_id` to scope it,
causing a table scan of flows and associated join tables to resolve it.

To fix this, we introduce the `account_id` and ensure the expire flows
uses this field to ensure only data for an account is considered in the
query.

Related:
https://firezone-inc.sentry.io/issues/6346235615/events/e225e1c488cb4ea3896649aabd529c50
2025-06-20 20:40:31 +00:00
Jamil
ddb3dc8ce0 refactor(portal): compile_config macro to env_var_to_config (#9605)
The `compile_config` macro only works on environment and DB variables.
This caused recent confusion when determining where `database_pool_size`
was coming from.

To fix this issue, we rename `compile_config` to be more clear.

We also remove the technical debt around supporting "legacy keys" and
DB-based configuration.

The configuration compiler now works exclusively on environment
variables only, where it is still useful for:

- Casting environment variables to their expected type
- Alerting us when one is missing that should be set
2025-06-20 20:39:06 +00:00
Brian Manifold
5bd5a7f6ad fix(portal): trim whitespace in auth provider forms (#9587)
Why:

* We recently had an issue where a space was entered into a provider
form field and caused our system to not be able to authenticate the
admin when setting up the auth provider and directory sync. To mitigate
this moving forward we are making sure all white space is trimmed in the
form fields. This commit focuses on the form fields for the auth
providers.

related: #9579
2025-06-20 18:44:33 +00:00
Jamil
fc3a9d17b9 fix(portal): broadcast before possible query errors out (#9601)
When handling some side effects, if the query fails for whatever reason,
we don't want these preventing handling side effects.

Related:
https://firezone-inc.sentry.io/issues/6346235615/events/d30d222f8a3e436d8058a54c0b2a508c/?project=4508756715569152&query=is%3Aunresolved&referrer=previous-event&stream_index=3
2025-06-20 17:03:42 +00:00
Jamil
e5a0bdc3b1 fix(portal): ensure sentry reports conditional migrations (#9582)
Sentry isn't started when this runs, so start it and manually capture a
message to ensure we're reminded about pending conditional migrations.

Verified that this works with the Release script.
2025-06-19 17:28:38 +00:00
Jamil
2d6e478a44 fix(portal): check conditional migrations with repo started (#9577)
In #9562, we introduced a bug where the pending conditional migrations
check was run without the repo being started. Wrapping it with
`with_repo` fixes that.
2025-06-18 22:40:24 +00:00
Jamil
6cd077cd18 feat(portal): populate otel resource attributes (#9574)
If the otel collector runs on a different node from the elixir apps,
these won't be set anymore.

Luckily, all of the info we need is already in the application's env, so
we can simply copy them over to the needed attributes.

Related: https://github.com/firezone/infra/pull/47
Related:
https://github.com/firezone/infra/pull/47#discussion_r2149291472
Related:
https://opentelemetry.io/docs/languages/erlang/resources/#adding-resources-with-os-and-otp-application-environment-variables
Related: https://opentelemetry.io/docs/specs/semconv/resource/#service
2025-06-18 19:26:10 +00:00
Jamil
236c21111a refactor(portal): don't rely on db to gate metric reporting (#9565)
This table was added to try and gate the request rate to Google's
Metrics API.

However, this was a flawed endeavor as we later discovered that the time
series points need to be spaced apart at least 5s, not the API requests
themselves.

This PR gets rid of the table and therefore the problematic DB query
which is timing out quite often due to the contention involved in 12
elixir nodes trying to grab a lock every 5s.

Related: #9539 
Related:
https://firezone-inc.sentry.io/issues/6346235615/?project=4508756715569152&query=is%3Aunresolved&referrer=issue-stream&stream_index=1
2025-06-18 18:40:33 +00:00
Jamil
a20989a819 feat(portal): conditional migrations on prod (#9562)
Some migrations take a long time to run because they require locks or
modify large amounts of data. To prevent this from causing issues during
deploy, we leverage Ecto's native support for loading migrations from
multiple directories to introduce a `conditional_migrations/` directory
that houses any conditional migrations we want to run.

To run these migrations, you'll need to do one of the following:

- `dev, test`: The `mix ecto.migrate` will run them by default because
we have aliased this to load conditional_migrations for dev
- `prod`: Set the `RUN_CONDITIONAL_MIGRATIONS` env var to `true` before
starting a prod server using the `bin/migrate` script.
- `dev, test, prod`: Run `Domain.Release.migrate(conditional: true)`
from an IEx shell.

If conditional migrations were found that weren't executed during
`Domain.Release.migrate`, a warning is logged to remind us to run them.

---------

Signed-off-by: Jamil <jamilbk@users.noreply.github.com>
2025-06-18 18:08:25 +00:00