* add gosimport to make fmt and run it
* move installation to tools.sh
* correct weird spacing issue
* Update Makefile
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* fix a weird issue
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
This PR introduces a new testonly endpoint for introspecting the
RequestLimiter state. It makes use of the endpoint to verify that changes to
the request_limiter config are honored across reload.
In the future, we may choose to make the sys/internal/request-limiter/status
endpoint available in normal binaries, but this is an expedient way to expose
the status for testing without having to rush the design.
In order to re-use as much of the existing command package utility funcionality
as possible without introducing sprawling code changes, I introduced a new
server_util.go and exported some fields via accessors.
The tests shook out a couple of bugs (including a deadlock and lack of
locking around the core limiterRegistry state).
This commit introduces two new adaptive concurrency limiters in Vault,
which should handle overloading of the server during periods of
untenable request rate. The limiter adjusts the number of allowable
in-flight requests based on latency measurements performed across the
request duration. This approach allows us to reject entire requests
prior to doing any work and prevents clients from exceeding server
capacity.
The limiters intentionally target two separate vectors that have been
proven to lead to server over-utilization.
- Back pressure from the storage backend, resulting in bufferbloat in
the WAL system. (enterprise)
- Back pressure from CPU over-utilization via PKI issue requests
(specifically for RSA keys), resulting in failed heartbeats.
Storage constraints can be accounted for by limiting logical requests
according to their http.Method. We only limit requests with write-based
methods, since these will result in storage Puts and exhibit the
aforementioned bufferbloat.
CPU constraints are accounted for using the same underlying library and
technique; however, they require special treatment. The maximum number
of concurrent pki/issue requests found in testing (again, specifically
for RSA keys) is far lower than the minimum tolerable write request
rate. Without separate limiting, we would artificially impose limits on
tolerable request rates for non-PKI requests. To specifically target PKI
issue requests, we add a new PathsSpecial field, called limited,
allowing backends to specify a list of paths which should get
special-case request limiting.
For the sake of code cleanliness and future extensibility, we introduce
the concept of a LimiterRegistry. The registry proposed in this PR has
two entries, corresponding with the two vectors above. Each Limiter
entry has its own corresponding maximum and minimum concurrency,
allowing them to react to latency deviation independently and handle
high volumes of requests to targeted bottlenecks (CPU and storage).
In both cases, utilization will be effectively throttled before Vault
reaches any degraded state. The resulting 503 - Service Unavailable is a
retryable HTTP response code, which can be handled to gracefully retry
and eventually succeed. Clients should handle this by retrying with
jitter and exponential backoff. This is done within Vault's API, using
the go-retryablehttp library.
Limiter testing was performed via benchmarks of mixed workloads and
across a deployment of agent pods with great success.
* Re-process .well-known redirects with a recursive handler call rather than a 302 redirect
* Track when the RequestURI mismatches path (in a redirect) and add it to the audit log
* call cancelFunc
* Work towards removing the feature flag that disabled eventlogger for audit events
* Removed audited headers from LogRequest and LogResponse and clean up
* make clear we don't use a method param, and comment tweak
* Moved BenchmarkAuditFile_request to audit_broker_test and renamed. Clean up
* fixed calls from tests to Factory's
* waffling godoc for a ported and tweaked test
* Remove duplicate code from previous merges, remove uneeded code
* Refactor file audit backend tests
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* Support reloading database plugins across multiple mounts
* Add clarifying comment to MountEntry.Path field
* Tests: Replace non-parallelisable t.Setenv with plugin env settings
* VAULT-22481: Audit filter node (#24465)
* Initial commit on adding filter nodes for audit
* tests for audit filter
* test: longer filter - more conditions
* copywrite headers
* Check interface for the right type
* Add audit filtering feature (#24554)
* Support filter nodes in backend factories and add some tests
* More tests and cleanup
* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)
* invert control of the pipelines/nodes to the audit broker vs. within each backend
* update noop audit test code to implement the pipeliner interface
* noop mount path has trailing slash
* attempting to make NoopAudit more friendly
* NoopAudit uses known salt
* Refactor audit.ProcessManual to support filter nodes
* HasFiltering
* rename the pipeliner
* use exported AuditEvent in Filter
* Add tests for registering and deregistering backends on the audit broker
* Add missing licence header to one file, fix a typo in two tests
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Add changelog file
* update bexpr datum to use a strong type
* go docs updates
* test path
* PR review comments
* handle scenarios/outcomes from broker.send
* don't need to re-check the complete sinks
* add extra check to deregister to ensure that re-registering non-filtered device sets sink threshold
* Ensure that the multierror is appended before attempting to return it
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Export audit event
* Move older tests away from audit behavior that didn't use eventlogger
* spelling--;
* no more struct initialization of NoopAudit outside of NewNoopAudit
* locking since we're accessing the shared backend
* Refactor plugin catalog into its own package
* Fix some unnecessarily slow tests due to accidentally running multiple plugin processes
* Clean up MakeTestPluginDir helper
* Move getBackendVersion tests to plugin catalog package
* Use corehelpers.MakeTestPlugin consistently
* Fix semgrep failure: check for nil value from logical.Storage
* Re-implementation of API redirects with more deterministic matching
* add missing file
* Handle query params properly
* licensing
* Add single src deregister
* Implement specifically RFC 5785 (.well-known) redirects.
Also implement a unit test for HA setups, making sure the standby node redirects to the active (as usual), and that then the active redirects the .well-known request to a backend, and that that is subsequently satisfied.
* Remove test code
* Rename well known redirect logic
* comments/cleanup
* PR feedback
* Remove wip typo
* Update http/handler.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Fix registrations with trailing slashes
---------
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* wip
* more pruning
* Integrate OCSP into binary paths PoC
- Simplify some of the changes to the router
- Remove the binary test PKI endpoint
- Switch OCSP to use the new binary paths backend variable
* Fix proto generation and test compilation
* Add unit test for binary request handling
---------
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
* Update the default kv mount to kv.Factory
* Imports
* Set some tests that care about leaseapssthroughbackend to use it
* extra newline
* More test updates
* Test updates
* Refactor KV mounting in tests
* Re-add comment
* create custom type for disable-replication-status-endpoints context key
make use of custom context key type in middleware function
* clean up code to remove various compiler warnings
unnecessary return statement
if condition that is always true
fix use of deprecated ioutil.NopCloser
empty if block
* remove unused unexported function
* clean up code
remove unnecessary nil check around a range expression
* clean up code
removed redundant return statement
* use http.StatusTemporaryRedirect constant instead of literal integer
* create custom type for context key for max_request_size parameter
* create custom type for context key for original request path
Teach stubmaker how to work with methods, not just funcs. Fix some stubs defined in #23557 which either had the wrong signature or needed to be public.
* CI: Pre-emptively delete logs dir after cache restore in test-collect-reports (#23600)
* Fix OktaNumberChallenge (#23565)
* remove arg
* changelog
* exclude changelog in verifying doc/ui PRs (#23601)
* Audit: eventlogger sink node reopen on SIGHUP (#23598)
* ensure nodes are asked to reload audit files on SIGHUP
* added changelog
* Capture errors emitted from all nodes during proccessing of audit pipelines (#23582)
* Update security-scan.yml
* Listeners: Redaction only for TCP (#23592)
* redaction should only work for TCP listeners, also fix bug that allowed custom response headers for unix listeners
* fix failing test
* updates from PR feedback
* fix panic when unlocking unlocked user (#23611)
* VAULT-18307: update rotation period for aws static roles on update (#23528)
* add disable_replication_status_endpoints tcp listener config parameter
* add wrapping handler for disabled replication status endpoints setting
* adapt disable_replication_status_endpoints configuration parsing code to refactored parsing code
* refactor configuration parsing code to facilitate testing
* fix a panic when parsing configuration
* update refactored configuration parsing code
* fix merge corruption
* add changelog file
* document new TCP listener configuration parameter
* make sure disable_replication_status_endpoints only has effect on TCP listeners
* use active voice for explanation of disable_replication_status_endpoints
* fix minor merge issue
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
Co-authored-by: Angel Garbarino <Monkeychip@users.noreply.github.com>
Co-authored-by: Hamid Ghaf <83242695+hghaf099@users.noreply.github.com>
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
Co-authored-by: Mark Collao <106274486+mcollao-hc@users.noreply.github.com>
Co-authored-by: davidadeleon <56207066+davidadeleon@users.noreply.github.com>
Co-authored-by: kpcraig <3031348+kpcraig@users.noreply.github.com>
* implement user lockout logger
* formatting
* make user lockout log interval configurable
* create func to get locked user count, and fix potential deadlock
* fix test
* fix test
* add changelog
* add redaction config settings to listener
* sys seal redaction + test modification for default handler properties
* build date should be redacted by 'redact_version' too
* sys-health redaction + test fiddling
* sys-leader redaction
* added changelog
* Lots of places need ListenerConfig
* Renamed options to something more specific for now
* tests for listener config options
* changelog updated
* updates based on PR comments
* updates based on PR comments - removed unrequired test case field
* fixes for docker tests and potentially server dev mode related flags
The WebSocket tests have been very flaky because we weren't able to tell when a WebSocket was fully connected and subscribed to events.
We reworked the websocket subscription code to accept the websocket only after subscribing.
This should eliminate all flakiness in these tests. 🤞 (We can follow-up in an enterprise PR to simplify some of the tests after this fix is merged.)
I ran this locally a bunch of times and with data race detection enabled, and did not see any failures.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
We grab the state lock and check that the core is not shutting down.
This panic mostly seems to happen if Vault is shutting down, usually
in a test.
Also, we try clean up the go-bexpr test by sending duplicates, and
deduplicating in the receive loop.
before starting access loop checker.
We were seeing rare data races because the `core.IsRoot()` check was
being called after the subscriber access loop checker was starter.
The subscriber access loop calls `core.CheckToken()`, which can modify
the values that `IsRoot()` is reading.
When #22835 was merged, it was auto-squashed, so the `experiments`
import was removed, but the test still referenced it.
This removes the (now unnecessary) experiment from the test.
Subscribing to events through a WebSocket now support boolean
expressions to filter only the events wanted based on the fields
* `event_type`
* `operation`
* `source_plugin_mount`
* `data_path`
* `namespace`
Example expressions:
These can be passed to `vault events subscribe`, e.g.,:
* `event_type == abc`
* `source_plugin_mount == secret/`
* `event_type != def and operation != write`
```sh
vault events subscribe -filter='source_plugin_mount == secret/' 'kv*'
```
The docs for the `vault events subscribe` command and API endpoint
will be coming shortly in a different PR, and will include a better
specification for these expressions, similar to (or linking to)
https://developer.hashicorp.com/boundary/docs/concepts/filtering
The flag `events.alpha1` will no longer do anything, but we keep it
to prevent breaking users who have it in their configurations or
startup flags, or if it is referenced in other code.
The more I looked at this test, the more I realized it wasn't testing
anything except that the namespace parameter was being parsed by the
websocket.
So, I moved that parameter parsing check to `TestEventsSubscribe()`,
which is not flaky, and removed the flaky test altogether.
There is a similar set of tests in the enterprise repo that I will
try to simplify and make less flaky, though.
Previously, when a user initiated a websocket subscription,
the access to the `sys/events/subscribe` endpoint was checked then,
and only once.
Now, perform continuous policy checks:
* We check access to the `sys/events/subscribe` endpoint every five
minutes. If this check fails, then the websocket is terminated.
* Upon receiving any message, we verify that the `subscribe`
capability is present for that namespace, data path, and event type.
If it is not, then the message is not delivered. If the message is
allowed, we cache that result for five minutes.
Tests for this are in a separate enterprise PR.
Documentation will be updated in another PR.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
CI is sometimes slow, so 100ms was not enough time for all events
to be sent and processed in `http/events_test.go`.
We bumped that timeout to a full 1 second, but also added a trick at
the end to shorten the timeout once the expected number of events
have been receieved. This way, once the test has passed, we only
wait 100ms for any "extra" events to make the test fail, instead
of waiting for the full 1 second before we let the test pass. This
should keep the test relatively fast, while still allowing for it to
be slow sometimes.
* reduce calls to DetermineRoleFromLoginRequest from 3 to 1 for aws auth method
* change ordering of LoginCreateToken args
* replace another determineRoleFromLoginRequest function with role from context
* add changelog
* Check for role in context if not there make call to DeteremineRoleFromLoginRequest
* move context role check below nanmespace check
* Update changelog/22583.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* revert signature to same order
* make sure resp is last argument
* retrieve role from context closer to where role variable is needed
* remove failsafe for role in mfa login
* Update changelog/22583.txt
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
