a9f49629ae
Building on the WAL consumer that's been in development over the past several weeks, we introduce a new `change_logs` table that stores very lightly up-fitted data decoded from the WAL: - `account_id` (indexed): a foreign key reference to an account. - `inserted_at` (indexed): the timestamp of insert, for truncating rows later. - `table`: the table where the op took place. - `op`: the operation performed (insert/update/delete) - `old_data`: a nullable map of the old row data (update/delete) - `data`: a nullable map of the new row data(insert/update) - `vsn`: an integer version field we can bump to signify schema changes in the data in case we need to apply operations to only new or only old data. Judging from our prod metrics, we're currently average about 1,000 write operations a minute, which will generate about 1-2 dozen changelogs / s. Doing the math on this, 30 days at our current volume will yield about 50M / month, which should be ok for some time, since this is an append-only, rarely (if ever) read from table. The one aspect of this we may need to handle sooner than later is batch-inserting these. That raises an issue though - currently, in this PR, we process each WAL event serially, ending with the final acknowledgement `:ok` which will signal to Postgres our status in processing the WAL. If we do anything async here, this processing "cursor" then becomes inaccurate, so we may need to think about what to track and what data we care about. Related: #7124
Firezone Elixir Development
Before reading this doc, make sure you've read through our CONTRIBUTING guide.
Getting Started
This is not an in depth guide for setting up all dependencies, but it should give you a starting point.
Prerequisites:
- All prerequisites in the CONTRIBUTING guide
- Install ASDF and all plugins/tools from
.tool-versionin the top level of the Firezone repo - Install pnpm
From the top level director of the Firezone repo start the Postgres container:
docker compose up -d postgres
Inside the /elixir directory run the following commands:
# Install dependencies
# --------------------
> mix deps.get
# Install npm packages and build assets
# -------------------------------------
> cd apps/web/
> mix setup
# Setup and seed the DB
# ---------------------
> cd ../..
> mix ecto.seed
# Start all of the portal Elixir apps:
# ------------------------------------
> iex -S mix
The web and api applications should now be running:
- Web -> http://localhost:13000/
- API -> http://localhost:13001/
Stripe integration for local development
Prerequisites:
- Stripe account
- Stripe CLI
Steps:
-
Use static seeds to provision account ID that corresponds to staging setup on Stripe:
STATIC_SEEDS=true mix do ecto.reset, ecto.seed -
Start Stripe CLI webhook proxy:
stripe listen --forward-to localhost:13001/integrations/stripe/webhooks -
Start the Phoenix server with enabled billing from the elixir/ folder using a test mode token:
cd elixir/ BILLING_ENABLED=true STRIPE_SECRET_KEY="...copy from stripe dashboard..." STRIPE_WEBHOOK_SIGNING_SECRET="...copy from stripe cli tool.." mix phx.server
When updating the billing plan in stripe, use the Stripe Testing Docs for how to add test payment info
WorkOS integration for local development
Prerequisites:
- WorkOS account
WorkOS is currently being used for JumpCloud directory sync integration. This allows JumpCloud users to use SCIM on the JumpCloud side, rather than having to give Firezone an admin JumpCloud API token.
Connecting WorkOS in dev mode for manual testing
If you are not planning to use the JumpCloud provider in your local development setup, then no additional setup is needed. However, if you need to use the JumpCloud provider locally, you will need to obtain an API Key and Client ID from the WorkOS Dashboard.
After obtaining WorkOS API credentials, you will need to make sure they are set in the environment ENVs when starting your local dev instance of Firezone. As an example:
WORKOS_API_KEY="..." WORKOS_CLIENT_ID="..." mix phx.server
Acceptance tests
You can disable headless mode for the browser by adding
@tag debug: true
feature ....