* VAULT-22481: Audit filter node (#24465)
* Initial commit on adding filter nodes for audit
* tests for audit filter
* test: longer filter - more conditions
* copywrite headers
* Check interface for the right type
* Add audit filtering feature (#24554)
* Support filter nodes in backend factories and add some tests
* More tests and cleanup
* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)
* invert control of the pipelines/nodes to the audit broker vs. within each backend
* update noop audit test code to implement the pipeliner interface
* noop mount path has trailing slash
* attempting to make NoopAudit more friendly
* NoopAudit uses known salt
* Refactor audit.ProcessManual to support filter nodes
* HasFiltering
* rename the pipeliner
* use exported AuditEvent in Filter
* Add tests for registering and deregistering backends on the audit broker
* Add missing licence header to one file, fix a typo in two tests
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Add changelog file
* update bexpr datum to use a strong type
* go docs updates
* test path
* PR review comments
* handle scenarios/outcomes from broker.send
* don't need to re-check the complete sinks
* add extra check to deregister to ensure that re-registering non-filtered device sets sink threshold
* Ensure that the multierror is appended before attempting to return it
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Export audit event
* Move older tests away from audit behavior that didn't use eventlogger
* spelling--;
* no more struct initialization of NoopAudit outside of NewNoopAudit
* locking since we're accessing the shared backend
* Refactor plugin catalog into its own package
* Fix some unnecessarily slow tests due to accidentally running multiple plugin processes
* Clean up MakeTestPluginDir helper
* Move getBackendVersion tests to plugin catalog package
* Use corehelpers.MakeTestPlugin consistently
* Fix semgrep failure: check for nil value from logical.Storage
* Re-implementation of API redirects with more deterministic matching
* add missing file
* Handle query params properly
* licensing
* Add single src deregister
* Implement specifically RFC 5785 (.well-known) redirects.
Also implement a unit test for HA setups, making sure the standby node redirects to the active (as usual), and that then the active redirects the .well-known request to a backend, and that that is subsequently satisfied.
* Remove test code
* Rename well known redirect logic
* comments/cleanup
* PR feedback
* Remove wip typo
* Update http/handler.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Fix registrations with trailing slashes
---------
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* wip
* more pruning
* Integrate OCSP into binary paths PoC
- Simplify some of the changes to the router
- Remove the binary test PKI endpoint
- Switch OCSP to use the new binary paths backend variable
* Fix proto generation and test compilation
* Add unit test for binary request handling
---------
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
* Update the default kv mount to kv.Factory
* Imports
* Set some tests that care about leaseapssthroughbackend to use it
* extra newline
* More test updates
* Test updates
* Refactor KV mounting in tests
* Re-add comment
* create custom type for disable-replication-status-endpoints context key
make use of custom context key type in middleware function
* clean up code to remove various compiler warnings
unnecessary return statement
if condition that is always true
fix use of deprecated ioutil.NopCloser
empty if block
* remove unused unexported function
* clean up code
remove unnecessary nil check around a range expression
* clean up code
removed redundant return statement
* use http.StatusTemporaryRedirect constant instead of literal integer
* create custom type for context key for max_request_size parameter
* create custom type for context key for original request path
Teach stubmaker how to work with methods, not just funcs. Fix some stubs defined in #23557 which either had the wrong signature or needed to be public.
* CI: Pre-emptively delete logs dir after cache restore in test-collect-reports (#23600)
* Fix OktaNumberChallenge (#23565)
* remove arg
* changelog
* exclude changelog in verifying doc/ui PRs (#23601)
* Audit: eventlogger sink node reopen on SIGHUP (#23598)
* ensure nodes are asked to reload audit files on SIGHUP
* added changelog
* Capture errors emitted from all nodes during proccessing of audit pipelines (#23582)
* Update security-scan.yml
* Listeners: Redaction only for TCP (#23592)
* redaction should only work for TCP listeners, also fix bug that allowed custom response headers for unix listeners
* fix failing test
* updates from PR feedback
* fix panic when unlocking unlocked user (#23611)
* VAULT-18307: update rotation period for aws static roles on update (#23528)
* add disable_replication_status_endpoints tcp listener config parameter
* add wrapping handler for disabled replication status endpoints setting
* adapt disable_replication_status_endpoints configuration parsing code to refactored parsing code
* refactor configuration parsing code to facilitate testing
* fix a panic when parsing configuration
* update refactored configuration parsing code
* fix merge corruption
* add changelog file
* document new TCP listener configuration parameter
* make sure disable_replication_status_endpoints only has effect on TCP listeners
* use active voice for explanation of disable_replication_status_endpoints
* fix minor merge issue
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
Co-authored-by: Angel Garbarino <Monkeychip@users.noreply.github.com>
Co-authored-by: Hamid Ghaf <83242695+hghaf099@users.noreply.github.com>
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
Co-authored-by: Mark Collao <106274486+mcollao-hc@users.noreply.github.com>
Co-authored-by: davidadeleon <56207066+davidadeleon@users.noreply.github.com>
Co-authored-by: kpcraig <3031348+kpcraig@users.noreply.github.com>
* implement user lockout logger
* formatting
* make user lockout log interval configurable
* create func to get locked user count, and fix potential deadlock
* fix test
* fix test
* add changelog
* add redaction config settings to listener
* sys seal redaction + test modification for default handler properties
* build date should be redacted by 'redact_version' too
* sys-health redaction + test fiddling
* sys-leader redaction
* added changelog
* Lots of places need ListenerConfig
* Renamed options to something more specific for now
* tests for listener config options
* changelog updated
* updates based on PR comments
* updates based on PR comments - removed unrequired test case field
* fixes for docker tests and potentially server dev mode related flags
The WebSocket tests have been very flaky because we weren't able to tell when a WebSocket was fully connected and subscribed to events.
We reworked the websocket subscription code to accept the websocket only after subscribing.
This should eliminate all flakiness in these tests. 🤞 (We can follow-up in an enterprise PR to simplify some of the tests after this fix is merged.)
I ran this locally a bunch of times and with data race detection enabled, and did not see any failures.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
We grab the state lock and check that the core is not shutting down.
This panic mostly seems to happen if Vault is shutting down, usually
in a test.
Also, we try clean up the go-bexpr test by sending duplicates, and
deduplicating in the receive loop.
before starting access loop checker.
We were seeing rare data races because the `core.IsRoot()` check was
being called after the subscriber access loop checker was starter.
The subscriber access loop calls `core.CheckToken()`, which can modify
the values that `IsRoot()` is reading.
When #22835 was merged, it was auto-squashed, so the `experiments`
import was removed, but the test still referenced it.
This removes the (now unnecessary) experiment from the test.
Subscribing to events through a WebSocket now support boolean
expressions to filter only the events wanted based on the fields
* `event_type`
* `operation`
* `source_plugin_mount`
* `data_path`
* `namespace`
Example expressions:
These can be passed to `vault events subscribe`, e.g.,:
* `event_type == abc`
* `source_plugin_mount == secret/`
* `event_type != def and operation != write`
```sh
vault events subscribe -filter='source_plugin_mount == secret/' 'kv*'
```
The docs for the `vault events subscribe` command and API endpoint
will be coming shortly in a different PR, and will include a better
specification for these expressions, similar to (or linking to)
https://developer.hashicorp.com/boundary/docs/concepts/filtering
The flag `events.alpha1` will no longer do anything, but we keep it
to prevent breaking users who have it in their configurations or
startup flags, or if it is referenced in other code.
The more I looked at this test, the more I realized it wasn't testing
anything except that the namespace parameter was being parsed by the
websocket.
So, I moved that parameter parsing check to `TestEventsSubscribe()`,
which is not flaky, and removed the flaky test altogether.
There is a similar set of tests in the enterprise repo that I will
try to simplify and make less flaky, though.
Previously, when a user initiated a websocket subscription,
the access to the `sys/events/subscribe` endpoint was checked then,
and only once.
Now, perform continuous policy checks:
* We check access to the `sys/events/subscribe` endpoint every five
minutes. If this check fails, then the websocket is terminated.
* Upon receiving any message, we verify that the `subscribe`
capability is present for that namespace, data path, and event type.
If it is not, then the message is not delivered. If the message is
allowed, we cache that result for five minutes.
Tests for this are in a separate enterprise PR.
Documentation will be updated in another PR.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
CI is sometimes slow, so 100ms was not enough time for all events
to be sent and processed in `http/events_test.go`.
We bumped that timeout to a full 1 second, but also added a trick at
the end to shorten the timeout once the expected number of events
have been receieved. This way, once the test has passed, we only
wait 100ms for any "extra" events to make the test fail, instead
of waiting for the full 1 second before we let the test pass. This
should keep the test relatively fast, while still allowing for it to
be slow sometimes.
* reduce calls to DetermineRoleFromLoginRequest from 3 to 1 for aws auth method
* change ordering of LoginCreateToken args
* replace another determineRoleFromLoginRequest function with role from context
* add changelog
* Check for role in context if not there make call to DeteremineRoleFromLoginRequest
* move context role check below nanmespace check
* Update changelog/22583.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* revert signature to same order
* make sure resp is last argument
* retrieve role from context closer to where role variable is needed
* remove failsafe for role in mfa login
* Update changelog/22583.txt
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Events from multiple namespaces can be subscribed to via
glob patterns passed to the subscription.
This does not do policy enforcement yet -- that will come in PR soon.
I tested this manually as well by pulling it into Vault Enterprise
so I could create namespaces and check that subscriptions work as
expected.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Seal HA: Use new SealWrappedValue type to abstract seal wrapped values
Introduce SealWrappedValue to abstract seal wrapped values.
Make SealWrappedValue capable of marshalling into a BlobInfo, when there is
plaintext or a single encryption, or to a custom serialization consisting of a
header, length and a marshalled MultiWrapValue protobuf.
* Vault-13769: Support configuring and using multiple seals for unsealing
* Make sealWrapBackend start using multiple seals
* Make seal.Access no longer implement wrapping.Wrapper.
Instead, add the Encrypt and Decrypt methods to the Access interface.
* Make raft snapshot system use funcs SealWrapValue + UnsealWrapValue.
Move the snapshot.Sealer implementation to the vault package to
avoid circular imports.
* Update sealWrapBackend to use multiple seals for encryption.
Use all the encryption wrappers when storing seal wrapped values.
Try do decrypt using the highest priority wrapper, but try all
combinations of encrypted values and wrappers if necessary.
* Allow the use of multiple seals for entropy augmentation
Add seal_name variable in entropy stanza
Add new MultiSourcer to accommodate the new entropy augmentation behavior.
* Individually health check each wrapper, and add a sys/seal-backend-status endpoint.
* Address a race, and also a failed test mock that I didn't catch
* Track partial wrapping failures...
... where one or more but not all access.Encrypts fail for a given write.
Note these failures by adding a time ordered UUID storage entry containing
the path in a special subdirectory of root storage. Adds a callback
pattern to accomplish this, with certain high value writes like initial
barrier key storage not allowing a partial failure. The followup work
would be to detect return to health and iterate through these storage
entries, rewrapping.
* Add new data structure to track seal config generation (#4492)
* Add new data structure to track seal config generation
* Remove import cycle
* Fix undefined variable errors
* update comment
* Update setSeal response
* Fix setSealResponse in operator_diagnose
* Scope the wrapper health check locks individually (#4491)
* Refactor setSeal function in server.go. (#4505)
Refactor setSeal function in server.go.
* Decouple CreateSecureRandomReaderFunc from seal package.
Instead of using a list of seal.SealInfo structs, make
CreateSecureRandomReaderFunc use a list of new EntropySourcerInfo structs. This
brakes the denpency of package configutil on the seal package.
* Move SealGenerationInfo tracking to the seal Access.
* Move SealGenerationInfo tracking to the seal Access.
The SealGenerationInfo is now kept track by a Seal's Access instead of by the
Config object. The access implementation now records the correct generation
number on seal wrapped values.
* Only store and read SealGenerationInfo if VAULT_ENABLE_SEAL_HA_BETA is true.
* Add MultiWrapValue protobuf message
MultiWrapValue can be used to keep track of different encryptions of a value.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* Use generation to determine if a seal wrapped value is up-to-date. (#4542)
* Add logging to seal Access implementation.
* Seal HA buf format run (#4561)
* Run buf format.
* Add buf.lock to ensure go-kms-wrapping module is imported.
* Vault-18958: Add unit tests for config checks
* Add safety logic for seal configuration changes
* Revert "Add safety logic for seal configuration changes"
This reverts commit 7fec48035a5cf274e5a4d98901716d08d766ce90.
* changes and tests for checking seal config
* add ent tests
* remove check for empty name and add type into test cases
* add error message for empty name
* fix no seals test
---------
Co-authored-by: divyapola5 <divya@hashicorp.com>
* Handle migrations between single-wrapper and multi-wrapper autoSeals
* Extract method SetPhysicalSealConfig.
* Extract function physicalSealConfig.
The extracted function is the only code now reading SealConfig entries from
storage.
* Extract function setPhysicalSealConfig.
The extracted function is the only code now writing SealConfig entries from
storage (except for migration from the old recovery config path).
* Move SealConfig to new file vault/seal_config.go.
* Add SealConfigType quasy-enumeration.
SealConfigType is to serve as the typed values for field SealConfig.Type.
* Rename Seal.RecoveryType to RecoverySealConfigType.
Make RecoverySealConfigType return a SealConfigType instead of a string.
* Rename Seal.BarrierType to BarrierSealConfigType.
Make BarrierSealConfigType return a SealConfigType.
Remove seal.SealType (really a two-step rename to SealConfigType).
* Add Seal methods ClearBarrierConfig and ClearRecoveryConfig.
* Handle autoseal <-> multiseal migrations.
While going between single-wrapper and multiple-wrapper autoseals are not
migrations that require an unwrap seal (such as going from shamir to autoseal),
the stored "barrier" SealConfig needs to be updated in these cases.
Specifically, the value of SealConfg.Type is "multiseal" for autoSeals that have
more than one wrapper; on the other hand, for autoseals with a single wrapper,
SealConfig.Type is the type of the wrapper.
* Remove error return value from NewAutoSeal constructor.
* Automatically rewrap partially seal wrapped values on an interval
* Add in rewrapping of partially wrapped values on an interval, regardless of seal health/status.
* Don't set SealGenerationInfo Rewrapped flag in the partial rewrap call.
* Unexport the SealGenerationInfo's Rewrapped field, add a mutex to it for thread safe access, and add accessor methods for it.
* Add a success callback to the manual seal rewrap process that updates the SealGenerationInfo's rewrapped field. This is done via a callback to avoid an import cycle in the SealRewrap code.
* Fix a failing seal wrap backend test which was broken by the unexporting of SealGenerationInfo's Rewrapped field.
* Nil check the seal rewrap success callback before calling it.
* Change SealGenerationInfo rewrapped parameter to an atomic.Bool rather than a sync.RWMutex for simplicity and performance.
* Add nil check for SealAccess before updating SealGenerationInfo rewrapped status during seal rewrap call.
* Update partial rewrap check interval from 10 seconds to 1 minute.
* Update a reference to SealGenerationInfo Rewrapped field to use new getter method.
* Fix up some data raciness in partial rewrapping.
* Account for possibly nil storage entry when retrieving partially wrapped value.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
* Restore propagation of wrapper configuration errors by setSeal.
Function setSeal is meant to propagate non KeyNotFound errors returned by calls
to configutil.ConfigureWrapper.
* Remove unused Access methods SetConfig and Type.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
Make it possible for an autoSeal that uses multiple wrappers to include disabled
wrappers that can be used to decrypt entries, but are skipped for encryption.
e an unwrapSeal when there are disabled seals.
* Fix bug with not providing name (#4580)
* add suffix to name defaults
* add comment
* only change name for disabled seal
* Only attempt to rewrap partial values when all seals are healthy.
* Only attempt to rewrap partial values when all seals are healthy.
* Change logging level from info to debug for notice about rewrap skipping based on seal health.
* Remove stale TODOs and commented out code.
---------
Co-authored-by: rculpepper <rculpepper@hashicorp.com>
Co-authored-by: Larroyo <95649169+DeLuci@users.noreply.github.com>
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
Co-authored-by: Divya Pola <87338962+divyapola5@users.noreply.github.com>
Co-authored-by: Matt Schultz <matt.schultz@hashicorp.com>
Co-authored-by: divyapola5 <divya@hashicorp.com>
Co-authored-by: Rachel Culpepper <84159930+rculpepper@users.noreply.github.com>
Biggest change: we rename `Send` to `SendEvent` in `logical.EventSender`..
Initially we picked `Send` to match the underlying go-eventlogger
broker's `Send` method, and to avoid the stuttering of `events.SendEvent`.
However, I think it is more useful for the `logical.EventSender`
interface to use the method `SendEvent` so that, for example,
`framework.Backend` can implement it.
This is a relatively change now that should not affect anything
except the KV plugin, which is being fixed in another PR.
Another change: if the `secret_path` metadata is present, then
the plugin-aware `EventBus` will prepend it with the plugin mount.
This allows the `secret_path` to be the full path to any referenced
secret.
This change is also backwards compatible, since this field was not
present in the KV plugin. (It did use the slightly different `path`
field, which we can keep for now.)
For now, only the leader of a cluster can handle subscription requests,
so we forward the connection request otherwise.
We forward using a 307 temporary redirect (the fallback way).
Forwarding a request over gRPC currently only supports a single request
and response, but a websocket connection is long-lived with potentially
many messages back and forth.
We modified the `vault events subscribe` command to honor those
redirects. `wscat` supports them with the `-L` flag.
In the future, we may add a gRPC method to handle forwarding WebSocket
requests, but doing so adds quite a bit of complexity (even over
normal request forwarding) due to the intricate nature of the `http` /
`vault.Core` interactions required. (I initially went down this path.)
I added tests for the forwarding header, and also tested manually.
(Testing with `-dev-three-node` is a little clumsy since it does not
properly support experiments, for some reason.)
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* add escape hatch to use feature flag for reversion of audit behavior
* Setup pipeline which ends with a NoopSink
* explicitly call out old way of running test
* old behavior for audit trail tests
* More manual forcing of tests to legacy audit system
* Add NOTE: to suggest that the feature flag is temporary
* Initial oss-patch apply
* Added changelog
* Renamed changelog txt
* Added the imports to the handler file
* Added a check that no two ports are the same, and modified changelog
* Edited go sum entry
* Tidy up using go mod
* Use strutil instead
* Revert go sum and go mod
* Revert sdk go sum
* Edited go.sum to before
* Edited go.sum again to initial
* Revert changes
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* add hashfunc field to EntryFormatter struct and adjust NewEntryFormatter function and tests
* add HeaderAdjuster interface and require it in EntryFormatter
dquote> adjust all references to NewEntryFormatter to include a HeaderAdjuster parameter
* replace use of hash function in AuditedHeadersConfig's ApplyConfig method with Salter interface instance
* fixup! replace use of hash function in AuditedHeadersConfig's ApplyConfig method with Salter interface instance
* review feedback
* Go doc typo
* add another test function
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
