* Base Binary Cert and CSR Parse functions.
* Add otherSANS parsing.
* Notate what doesn't exist on a CSR.
* Fix otherSans call err-checking and add basic-constriants to CSR
* Move BasicConstraint parsing to be optionally set.
* Refactored to use existing ParseBasicConstraintsExtension.
* Add handling for the ChangeSubjectName ext on CSR that is needed for EST
* Remove ChangeSubjectName - it's an attribute, not an extension, and there is no clean way to parse it, so pair down for now.
* Make these public methods, so they can be used in vault.
* Add unit tests for certutil.ParseCertificateToCreationParameters.
Also add unit tests for certutil.ParseCertificateToFields.
* Cleanup TestParseCertificate.
* Add unit tests for certutil.ParseCsrToCreationParameters and ParseCsrToFields.
* Fix return values for "add_basic_constraints" in certutil.ParseCsrToFields.
Add a test for parsing CSRs where "add_basic_constraints" is false.
* Clear up some todos.
* Add a test for certutil.ParseCertificateToCreationParameters for non-CA cert.
* Tweak TestParseCertificate/full_non_CA_cert.
* Basics of three remaining fields - keyUsage; extKeyUsage; PolicyIdentifiers
* Fix tests and err handling
* Add unit tests for policy_identifiers; ext_key_usage_oids; key_usage
* Add test on ext_key_usage_oids
* Remove duplicate usages elsewhere.
* Add error handling to csr-checks.
* Remove extranames on returned types.
* Remove useless function.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* add new plugin wif fields to AWS Secrets Engine
* add changelog
* go get awsutil v0.3.0
* fix up changelog
* fix test and field parsing helper
* godoc on new test
* require role arn when audience set
* make fmt
---------
Co-authored-by: Austin Gebauer <agebauer@hashicorp.com>
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
This commit introduces two new adaptive concurrency limiters in Vault,
which should handle overloading of the server during periods of
untenable request rate. The limiter adjusts the number of allowable
in-flight requests based on latency measurements performed across the
request duration. This approach allows us to reject entire requests
prior to doing any work and prevents clients from exceeding server
capacity.
The limiters intentionally target two separate vectors that have been
proven to lead to server over-utilization.
- Back pressure from the storage backend, resulting in bufferbloat in
the WAL system. (enterprise)
- Back pressure from CPU over-utilization via PKI issue requests
(specifically for RSA keys), resulting in failed heartbeats.
Storage constraints can be accounted for by limiting logical requests
according to their http.Method. We only limit requests with write-based
methods, since these will result in storage Puts and exhibit the
aforementioned bufferbloat.
CPU constraints are accounted for using the same underlying library and
technique; however, they require special treatment. The maximum number
of concurrent pki/issue requests found in testing (again, specifically
for RSA keys) is far lower than the minimum tolerable write request
rate. Without separate limiting, we would artificially impose limits on
tolerable request rates for non-PKI requests. To specifically target PKI
issue requests, we add a new PathsSpecial field, called limited,
allowing backends to specify a list of paths which should get
special-case request limiting.
For the sake of code cleanliness and future extensibility, we introduce
the concept of a LimiterRegistry. The registry proposed in this PR has
two entries, corresponding with the two vectors above. Each Limiter
entry has its own corresponding maximum and minimum concurrency,
allowing them to react to latency deviation independently and handle
high volumes of requests to targeted bottlenecks (CPU and storage).
In both cases, utilization will be effectively throttled before Vault
reaches any degraded state. The resulting 503 - Service Unavailable is a
retryable HTTP response code, which can be handled to gracefully retry
and eventually succeed. Clients should handle this by retrying with
jitter and exponential backoff. This is done within Vault's API, using
the go-retryablehttp library.
Limiter testing was performed via benchmarks of mixed workloads and
across a deployment of agent pods with great success.
Adds the ability to pin a version for a specific plugin type + name to enable an easier plugin upgrade UX. After pinning and reloading, that version should be the only version in use.
No HTTP API implementation yet for managing pins, so no user-facing effects yet.
* Validate audit filter option against filter selectors referencing unsupported fields
* Test updates due to filter validation
* Test all properties of the log input bexpr datum struct in filters
* Remove redundant cloning of the client in external tests for audit filtering
* TestAuditFilteringFilterForUnsupportedField now also tests the same behaviour with skip_test option set to true
* Add filter validation test cases to unit tests for audit backends
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Migration of OtherSANs Parsing Call to SDK helper from pki-issuer
* Based on PR feedback from Steve, remove internal variable, reference certutil directly.
* Work towards removing the feature flag that disabled eventlogger for audit events
* Removed audited headers from LogRequest and LogResponse and clean up
* make clear we don't use a method param, and comment tweak
* Moved BenchmarkAuditFile_request to audit_broker_test and renamed. Clean up
* fixed calls from tests to Factory's
* waffling godoc for a ported and tweaked test
* Remove duplicate code from previous merges, remove uneeded code
* Refactor file audit backend tests
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* VAULT-22481: Audit filter node (#24465)
* Initial commit on adding filter nodes for audit
* tests for audit filter
* test: longer filter - more conditions
* copywrite headers
* Check interface for the right type
* Add audit filtering feature (#24554)
* Support filter nodes in backend factories and add some tests
* More tests and cleanup
* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)
* invert control of the pipelines/nodes to the audit broker vs. within each backend
* update noop audit test code to implement the pipeliner interface
* noop mount path has trailing slash
* attempting to make NoopAudit more friendly
* NoopAudit uses known salt
* Refactor audit.ProcessManual to support filter nodes
* HasFiltering
* rename the pipeliner
* use exported AuditEvent in Filter
* Add tests for registering and deregistering backends on the audit broker
* Add missing licence header to one file, fix a typo in two tests
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Add changelog file
* initial work on global metrics for sink success/failure
* initial work to add a fallback device for audit
* Return when we have outright errors
* Improve comment
* Remove unneeded options on NewBroker and remove the policy opts elsewhere
* Remove duplicate node registration code
* Add more tests for audit backends
* ensure we return the multierror as soon as possible, and append it correctly
* error tweaks for audit: log req/resp
* extract the registration for fallback/normal devices, and ensure we always add to backends when successful
* slightly nicer error message rather than returning the raw err
* refactor the deregister methods for audit broker
* Prevent issues if fallback device is the first device added
* Bail early when the user tries adding more than one fallback audit device
* Check if there is an existing fallback audit device when setting the required sinks threshold for an audit broker
* Use the right ParseBool in audit backends
* Tweak the way we check for the threshold to make it clear why we ignore fallback
* Ensure all 'fallback' settings look the same
* nicer formatting of error
* broker tests for Register
* Deregister tests
* Deregister checks if registered before attempting
* Comment improvement
* Multiple Deregister calls are OK
* Fallback not required in this test
* Sanitise input for Deregister
* Locking mixup
* fix test
* Add changelog
* Check fallback broker's sink success threshold for register/deregister
* Remove changelog
* updated
* better name for the audit metrics labelers
* extra test
* remove name from metric counter type
* update func calls for NewMetricsCounter
* labelers should be pointers to the instance
* revert audit_test complaints about the header
* use constant value for the metric label on a fallback miss
* remove vault prefix from metric labels
* US spelling for labeler and adjust the way the labels are returned
* Fixed name and type we're testing for
* Defensive addition to HasFiltering (no nodemap no filter node)
* Remove dupe code block
* Revert to using armon/go-metrics
* Fallback miss fix
* PR feedback updates
* consistent format for configure methods
* Updated telemetry set up based on PR feedback
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* Support reloading database plugins across multiple mounts
* Add clarifying comment to MountEntry.Path field
* Tests: Replace non-parallelisable t.Setenv with plugin env settings
* VAULT-22481: Audit filter node (#24465)
* Initial commit on adding filter nodes for audit
* tests for audit filter
* test: longer filter - more conditions
* copywrite headers
* Check interface for the right type
* Add audit filtering feature (#24554)
* Support filter nodes in backend factories and add some tests
* More tests and cleanup
* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)
* invert control of the pipelines/nodes to the audit broker vs. within each backend
* update noop audit test code to implement the pipeliner interface
* noop mount path has trailing slash
* attempting to make NoopAudit more friendly
* NoopAudit uses known salt
* Refactor audit.ProcessManual to support filter nodes
* HasFiltering
* rename the pipeliner
* use exported AuditEvent in Filter
* Add tests for registering and deregistering backends on the audit broker
* Add missing licence header to one file, fix a typo in two tests
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Add changelog file
* update bexpr datum to use a strong type
* go docs updates
* test path
* PR review comments
* handle scenarios/outcomes from broker.send
* don't need to re-check the complete sinks
* add extra check to deregister to ensure that re-registering non-filtered device sets sink threshold
* Ensure that the multierror is appended before attempting to return it
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* update node and pipeline registration to prevent overwriting, strip some unused bits of NewTestCluster, tweak to prevent auditing on a test that is flaking
* tidy imports
* PKI refactoring to start breaking apart monolith into sub-packages
- This was broken down by commit within enterprise for ease of review
but would be too difficult to bring back individual commits back
to the CE repository. (they would be squashed anyways)
- This change was created by exporting a patch of the enterprise PR
and applying it to CE repository
* Fix TestBackend_OID_SANs to not be rely on map ordering
* Refactor plugin catalog into its own package
* Fix some unnecessarily slow tests due to accidentally running multiple plugin processes
* Clean up MakeTestPluginDir helper
* Move getBackendVersion tests to plugin catalog package
* Use corehelpers.MakeTestPlugin consistently
* Fix semgrep failure: check for nil value from logical.Storage
- Noticed that our documentation was out of date, we allow 8192
bit RSA keys to be used as an argument to the various PKI
issuer/key creation APIs.
- Augument some unit tests to verify this continues to work
* allow to skip TLS check in acme http-01 challenge
* remove configurable logic, just ignore TLS
* add changelog
* Add test case
---------
Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
* Address a panic export RSA public keys in transit
- When attempting to export the public key for an RSA key that
we only have a private key for, the export panics with a nil
deference.
- Add additional tests around Transit key exporting
* Add cl
- Try to avoid these build failures as our proxy does seem to have
issues around pulling images with the 'latest' tag at times.
```
acme_test.go:206:
Error Trace: /home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/builtin/logical/pkiext/pkiext_binary/acme_test.go:206
/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/builtin/logical/pkiext/pkiext_binary/acme_test.go:75
Error: Received unexpected error:
container create failed: Error response from daemon: No such image: docker.mirror.hashicorp.services/curlimages/curl:latest
Test: Test_ACME/group/caddy_http_eab
Messages: could not start cURL container
```
* wip
* more pruning
* Integrate OCSP into binary paths PoC
- Simplify some of the changes to the router
- Remove the binary test PKI endpoint
- Switch OCSP to use the new binary paths backend variable
* Fix proto generation and test compilation
* Add unit test for binary request handling
---------
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
* fix lease revocation when config token exists in one namespace but can create tokens in another
* add test
* Add similar check for admin partition
* Add admin partition test
---------
Co-authored-by: robmonte <17119716+robmonte@users.noreply.github.com>
* Forbid setting auto_rotate_period on transit managed keys
- Prevent and guard against auto-rotating managed keys as we
generate an invalid key version without the uuid field set.
- Hook in the datakey generation api into managed key encryption.
* Add cl
* auth/aws: fix panic in IAM-based login when client config doesn't exist
* add changelog
* adds known issue for 1.15.0
* fixes up known issue with workaround
* fix link
* maintain behavior of client config not needing to exist for IAM login
* update changelog
