* Move cert auth backend setup into initialize
In further review with new understanding after #18244, loading
configuration and CRLs within the backend's initialize function is the
ideal approach: Factory construction is strictly serial, resulting in
backend initialization blocking until config and CRLs are loaded.
By using an InitializeFunc(...), we delay loading until after all
backends are constructed (either right on startup in 1.12+, else during
the initial PeriodicFunc(...) invocation on 1.11 and earlier).
We also invoke initialize automatically on test Factory construction.
Resolves: #17847
Co-authored-by: valli_0x <personallune@mail.ru>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: valli_0x <personallune@mail.ru>
This PR modifies every test in `builtin/credentials/approle/path_role_test.go` with new validation checks to ensure that approle/path_role successful responses align with the declared response schema.
It also introduces a test helper in `sdk/helper/testhelpers`:
```go
func FindResponseSchema(t *testing.T, ...)
```
This test helper will be useful for all plugins that require similar response schema validation in tests.
### Background
This PR is part of the ongoing work to add structured responses in Vault OpenAPI (VLT-234)
This was not copied over when the this code was
copied in https://github.com/hashicorp/vault/pull/12340.
Also adds a stub for the `.copywrite.hcl` file (for when
Vault is onboarded to Copywrite) and adds the `pkcs7` and
`ui/node_modules` to the ignore pattern.
Bad news: the hot patch we were using breaks in Go 1.19.4: 6109c07ec4
Good news: we can now patch with an environment variable at runtime.
Co-authored-by: Christopher Swenson <christopher.swenson@hashicorp.com>
This PR modifies the path schema of `approle/path_role.go`, switching the old `Callbacks` to the equivalent `Operations` objects with a list of response fields for the 200 responses. This will allow us to generate a response structures in openapi.json. This PR is split out from #18055 along with #18192.
### Example
For `GET "/auth/approle/role/{role_name}/bind-secret-id"` path, it will update the response as follows:
```diff
"responses": {
"200": {
"description": "OK",
++ "content": {
++ "application/json": {
++ "schema": {
++ "$ref": "#/components/schemas/ApproleRoleBindSecretIdResponse"
++ }
++ }
}
}
}
```
And will add the actual response structure:
```diff
++ "ApproleRoleBindSecretIdResponse": {
++ "type": "object",
++ "properties": {
++ "bind_secret_id": {
++ "type": "boolean",
++ "description": "Impose secret_id to be presented when logging in using this role. Defaults to 'true'."
++ }
++ }
++ },
```
* wip
* Add cached OCSP client support to Cert Auth
* ->pointer
* Code cleanup
* Fix unit tests
* Use an LRU cache, and only persist up to 1000 of the most recently used values to stay under the storage entry limit
* Fix caching, add fail open mode parameter to cert auth roles
* reduce logging
* Add the retry client and GET then POST logic
* Drop persisted cache, make cache size configurable, allow for parallel testing of multiple servers
* dead code
* Update builtin/credential/cert/path_certs.go
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Hook invalidate to reinit the ocsp cache size
* locking
* Conditionally init the ocsp client
* Remove cache size config from cert configs, it's a backend global
* Add field
* Remove strangely complex validity logic
* Address more feedback
* Rework error returning logic
* More edge cases
* MORE edge cases
* Add a test matrix with a builtin responder
* changelog
* Use an atomic for configUpdated
* Actually use ocsp_enabled, and bind to a random port for testing
* Update builtin/credential/cert/path_login.go
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor unit tests
* Add status to cache
* Make some functions private
* Rename for testing, and attribute
* Up to date gofumpt
* remove hash from key, and disable the vault dependent unit test
* Comment out TestMultiOCSP
* imports
* more imports
* Address semgrep results
* Attempt to pass some sort of logging to test_responder
* fix overzealous search&replace
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
When running the test suite in CI (where requests are centralized from
relatively few IPs), we'd occasionally hit Dockerhub's rate limits.
Luckily Hashicorp runs a (limited) public mirror of the containers we
need, so we can switch to them here in the tests.
For consistency between developer and CI, we've opted to have the tests
always pull from the Hashicorp mirror, rather than updating the CI
runner to prefer the mirror.
We exclude nomad and influxdb as we don't presently mirror these repos.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor Docker command execution
This refactor will allow others to interact with containers more easily,
providing two interfaces (RunCmdWithOutput and RunCmdInBackground) for
executing commands in running containers if they don't wish to do so
manually.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow building containerfiles in tests
By adding image building capabilities to testhelpers (and coupled with
the better command execution support), we can begin to build better,
more reliable integration tests on top of public base images without
needing to maintain separate forks of these images out-of-tree for any
shortcomings they might have.
In particular, rather than doing the rather messy echo hack for writing
clients.conf, it is far better to provision this via a slim
Containerfile overlay on top of the stock jumanjiman/radiusd:latest
image.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly parse stdout/stderr in RunCmdWithOutput
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* ctx -> bCtx for BuildContext
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update errors to use %w instead of %v
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
The periodic function only runs every 50ms, so waiting 60ms means we
might not be done fetching the CRL on slower CI systems or with high
test parallelism.
Tested with:
> untilfail -parallel=-9 ../../../cert.test -test.run=TestCRLFetch -test.count=1 -test.v
And shown to reliably fail before, fixed after.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Check if plugin version matches running version
When registering a plugin, we check if the request version matches the
self-reported version from the plugin. If these do not match, we log a
warning.
This uncovered a few missing pieces for getting the database version
code fully working.
We added an environment variable that helps us unit test the running
version behavior as well, but only for approle, postgresql, and consul
plugins.
Return 400 on plugin not found or version mismatch
Populate the running SHA256 of plugins in the mount and auth tables (#17217)
There were two races here:
1. Tests racing against periodic func on updating the backend.
2. Tests racing internally to itself, to access the http-served
CRL data.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fetch CRLs from a user defined CDP (PoC)
* Handle no param sent
* Move CRL fetch to a periodFunc. Use configured CA certs + system root as trusted certs for CRL fetch
* comments
* changelog
* Just use root trust
* cdp->url in api
* Store CRL and populate it initially in cdlWrite
* Update docs
* Update builtin/credential/cert/path_crls.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Handle pre-verification of a CRL url better
* just in case
* Fix crl write locking
* Add a CRL fetch unit test
* Remove unnecessary validity clear
* Better func name
* Don't exit early updating CRLs
* lock in updateCRLs
* gofumpt
* err-
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Allow exposing access to the underlying container
This exposes the Container response from the Docker API, allowing
consumers of the testhelper to interact with the newly started running
container instance. This will be useful for two reasons:
1. Allowing radiusd container to start its own daemon after modifying
its configuration.
2. For loading certificates into a future similar integration test
using the PKI secrets engine.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow any client to connect to test radiusd daemon
This fixes test failures of the following form:
> 2022-09-07T10:46:19.332-0400 [TRACE] core: adding local paths: paths=[]
> 2022-09-07T10:46:19.333-0400 [INFO] core: enabled credential backend: path=mnt/ type=test
> 2022-09-07T10:46:19.334-0400 [WARN] Executing test step: step_number=1
> 2022-09-07T10:46:19.334-0400 [WARN] Executing test step: step_number=2
> 2022-09-07T10:46:29.334-0400 [WARN] Executing test step: step_number=3
> 2022-09-07T10:46:29.335-0400 [WARN] Executing test step: step_number=4
> 2022-09-07T10:46:39.336-0400 [WARN] Requesting RollbackOperation
> --- FAIL: TestBackend_acceptance (28.56s)
> testing.go:364: Failed step 4: erroneous response:
>
> &logical.Response{Secret:<nil>, Auth:<nil>, Data:map[string]interface {}{"error":"context deadline exceeded"}, Redirect:"", Warnings:[]string(nil), WrapInfo:(*wrapping.ResponseWrapInfo)(nil), Headers:map[string][]string(nil)}
> FAIL
> FAIL github.com/hashicorp/vault/builtin/credential/radius 29.238s
In particular, radiusd container ships with a default clients.conf which
restricts connections to ranges associated with the Docker daemon. When
creating new networks (such as in CircleCI) or when running via Podman
(which has its own set of network ranges), this initial config will no
longer be applicable. We thus need to write a new config into the image;
while we could do this by rebuilding a new image on top of the existing
layers (provisioning our config), we then need to manage these changes
and give hooks for the service setup to build it.
Thus, post-startup modification is probably easier to execute in our
case.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add fields 'ttl' and 'num_uses' to SecretID generation.
Add fields 'ttl' and 'num_uses' when generating/obtaining a SecretID.
Rather than just being able to use the Role's SecretID ttl and num uses. #14390
* Add secret_id_num_uses response field to generating SecretID
Add the response field secret_id_num_uses to the endpoints for generating
SecretIDs. Used in testing but also to supply the vendor with this variable.
* Add tests for new ttl and num_uses SecretID generation fields
Add tests to assert the new TTL and NumUses option in the SecretID entry.
Separate test for testing with just parameters vs a -force example.
* Patch up test for ttl and num_uses fields
* Add changelog entry for auth/approle 'ttl' and 'num_uses' fields
* Add fields to API Docs and AppRole Auth Docs example
* Correct error message for failing test on missing field.
Change the error message produced when a test fails due to a missing field.
Previous values did not map to correct fields.
* Remove unnecessary int cast to int "secret_id_num_uses" field.
Unnecessary cast to int where type already is int.
* Move numUses field check to after assignment.
* Remove metadata entry in sample payload to limit change to changes made.
Remove metadata entry in sample payload for custom-secret-id. The metadata was not
changed in the features pull request.
* Bind fields 'ttl' and 'num_uses' to role's configuration.
Rather than implicitly overriding, error when the ttl is lower than and the num
uses higher than the role's configuration. #14390
* Update changelog 14474 with a more detailed description.
More elaborate description for the changelog. Specifying the per-request based fields.
* Elaborate more on the bounds of the 'ttl' and 'num_uses' field.
Specify in both the api-docs and the CLI the limits of the fields.
Specify that the role's configuration is still the leading factor.
* Upper bound ttl with role secret id ttl
Upper bound ttl with role secret id ttl when creating a secret id
Adding test cases for infinite ttl and num uses
Adding test cases for negative ttl and num uses
Validation on infinite ttl and num uses
* Formatting issues. Removed unnecessary newline
* Update documentation for AppRole Secret ID and Role
Changed that TTL is not allowed to be shorter to longer
* Cleanup approle secret ID test and impl
* Define ttl and num_uses in every test
Define ttl and num_uses in every test despite them not being tested.
This is to ensure that no unexpected behaviour comes to mind.
* Rename test RoleSecretID -> RoleSecretIDWithoutFields
* Test secret id generation defaults to Role's config
Test secret id generation defaults to Role's configuration entries.
* Change finit -> finite
Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>
* Rephrase comments to the correct validation check
* Rephrase role-secret-id option description
* Remove "default" incorrect statement about ttl
* Remove "default" incorrect statement about ttl for custom secret id
* Touch up approle.mdx to align more with path_role documentation
Co-authored-by: Remco Buddelmeijer <r.buddelmeijer@fullstaq.com>
Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>
* enable registering backend muxed plugins in plugin catalog
* set the sysview on the pluginconfig to allow enabling secrets/auth plugins
* store backend instances in map
* store single implementations in the instances map
cleanup instance map and ensure we don't deadlock
* fix system backend unit tests
move GetMultiplexIDFromContext to pluginutil package
fix pluginutil test
fix dbplugin ut
* return error(s) if we can't get the plugin client
update comments
* refactor/move GetMultiplexIDFromContext test
* add changelog
* remove unnecessary field on pluginClient
* add unit tests to PluginCatalog for secrets/auth plugins
* fix comment
* return pluginClient from TestRunTestPlugin
* add multiplexed backend test
* honor metadatamode value in newbackend pluginconfig
* check that connection exists on cleanup
* add automtls to secrets/auth plugins
* don't remove apiclientmeta parsing
* use formatting directive for fmt.Errorf
* fix ut: remove tls provider func
* remove tlsproviderfunc from backend plugin tests
* use env var to prevent test plugin from running as a unit test
* WIP: remove lazy loading
* move non lazy loaded backend to new package
* use version wrapper for backend plugin factory
* remove backendVersionWrapper type
* implement getBackendPluginType for plugin catalog
* handle backend plugin v4 registration
* add plugin automtls env guard
* modify plugin factory to determine the backend to use
* remove old pluginsets from v5 and log pid in plugin catalog
* add reload mechanism via context
* readd v3 and v4 to pluginset
* call cleanup from reload if non-muxed
* move v5 backend code to new package
* use context reload for for ErrPluginShutdown case
* add wrapper on v5 backend
* fix run config UTs
* fix unit tests
- use v4/v5 mapping for plugin versions
- fix test build err
- add reload method on fakePluginClient
- add multiplexed cases for integration tests
* remove comment and update AutoMTLS field in test
* remove comment
* remove errwrap and unused context
* only support metadatamode false for v5 backend plugins
* update plugin catalog errors
* use const for env variables
* rename locks and remove unused
* remove unneeded nil check
* improvements based on staticcheck recommendations
* use const for single implementation string
* use const for context key
* use info default log level
* move pid to pluginClient struct
* remove v3 and v4 from multiplexed plugin set
* return from reload when non-multiplexed
* update automtls env string
* combine getBackend and getBrokeredClient
* update comments for plugin reload, Backend return val and log
* revert Backend return type
* allow non-muxed plugins to serve v5
* move v5 code to existing sdk plugin package
* do next export sdk fields now that we have removed extra plugin pkg
* set TLSProvider in ServeMultiplex for backwards compat
* use bool to flag multiplexing support on grpc backend server
* revert userpass main.go
* refactor plugin sdk
- update comments
- make use of multiplexing boolean and single implementation ID const
* update comment and use multierr
* attempt v4 if dispense fails on getPluginTypeForUnknown
* update comments on sdk plugin backend
* auth/cert: Add metadata to identity-alias
Add the possibility to include certificate metadata in the created
logical.Alias (the identity alias), in addition to the metadata added
to logical.Auth. This is analogous to the behaviour of the ldap and
approle auth providers.
This possibility can be configured by the config endpoint of the
auth method mount and is disabled by default. We added the read
operation on this config endpoint as well.
Fixes: #14418
Signed-off-by: Peter Verraedt <peter.verraedt@kuleuven.be>
* Add changelog for #14751
Signed-off-by: Peter Verraedt <peter.verraedt@kuleuven.be>
* Test the usage of cert metadata in ACL policies
Signed-off-by: Peter Verraedt <peter@verraedt.be>
Signed-off-by: Peter Verraedt <peter.verraedt@kuleuven.be>
Signed-off-by: Peter Verraedt <peter@verraedt.be>
Remove SHA1 for certs in prep for Go 1.18
* Remove certs with SHA1 from tests
* Use default SHA-256 with PKCS7 in AWS
* Update SHA1 deprecation note
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
Update AWS auth method certificates
Add tests that the `rsa2048` document can also be verified using the
`pkcs7` field for AWS auth.
Due to the use of SHA-1-based signatures for the `identity` and `pkcs7`
methods, we want to encourage moving toward using the RSA 2048 workflow,
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/verify-rsa2048.html
This doesn't require code changes for Vault necessarily, but adding in
the (many) certificates will help end users.
Also adds `rsa2048` option to API to fetch the RSA 2048 signature.
I will make a PR to update to the AWS auth docs to document the RSA 2048
flow soon after this.
* Fix handling of username_as_alias during LDAP authentication
There is a bug that was introduced in the LDAP authentication method by https://github.com/hashicorp/vault/pull/11000.
It was thought to be backward compatible but has broken a number of users. Later
a new parameter `username_as_alias` was introduced in https://github.com/hashicorp/vault/pull/14324
to make it possible for operators to restore the previous behavior.
The way it is currently working is not completely backward compatible thought
because when username_as_alias is set, a call to GetUserAliasAttributeValue() will
first be made, then this value is completely discarded in pathLogin() and replaced
by the username as expected.
This is an issue because it makes useless calls to the LDAP server and will break
backward compatibility if one of the constraints in GetUserAliasAttributeValue()
is not respected, even though the resulting value will be discarded anyway.
In order to maintain backward compatibility here we have to only call
GetUserAliasAttributeValue() if necessary.
Since this change of behavior was introduced in 1.9, this fix will need to be
backported to the 1.9, 1.10 and 1.11 branches.
* Add changelog
* Add tests
* Format code
* Update builtin/credential/ldap/backend.go
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
* Format and fix declaration
* Reword changelog
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
* POC of Okta Auth Number Challenge verification
* switch from callbacks to operations, forward validate to primary
* cleanup and nonce description update
* add changelog
* error on empty nonce, no forwarding, return correct_answer instead
* properly clean up verify goroutine
* add docs on new endpoint and parameters
* change polling frequency when WAITING to 1s
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
