* PKI refactoring to start breaking apart monolith into sub-packages
- This was broken down by commit within enterprise for ease of review
but would be too difficult to bring back individual commits back
to the CE repository. (they would be squashed anyways)
- This change was created by exporting a patch of the enterprise PR
and applying it to CE repository
* Fix TestBackend_OID_SANs to not be rely on map ordering
* Pulls in github.com/go-secure-stdlib/plugincontainer@v0.3.0 which exposes a new `Config.Rootless` option to opt in to extra container configuration options that allow establishing communication with a non-root plugin within a rootless container runtime.
* Adds a new "rootless" option for plugin runtimes, so Vault needs to be explicitly told whether the container runtime on the machine is rootless or not. It defaults to false as rootless installs are not the default.
* Updates `run_config.go` to use the new option when the plugin runtime is rootless.
* Adds new `-rootless` flag to `vault plugin runtime register`, and `rootless` API option to the register API.
* Adds rootless Docker installation to CI to support tests for the new functionality.
* Minor test refactor to minimise the number of test Vault cores that need to be made for the external plugin container tests.
* Documentation for the new rootless configuration and the new (reduced) set of restrictions for plugin containers.
* As well as adding rootless support, we've decided to drop explicit support for podman for now, but there's no barrier other than support burden to adding it back again in future so it will depend on demand.
* wip
* Work on the tuneable allowance and some bugs
* Call handleCancellableRequest instead, which gets the audit order more correct and includes the preauth response
* Get rid of no longer needed operation
* Phew, this wasn't necessary
* Add auth error handling by the backend, and fix a bug with handleInvalidCredentials
* Cleanup req/resp naming
* Use the new form, and data
* Discovered that tokens werent really being checked because isLoginRequest returns true for the re-request into the backend, when it shouldnt
* Add a few more checks in the delegated request handler for bad inputs
- Protect the delegated handler from bad inputs from the backend such
as an empty accessor, a path that isn't registered as a login request
- Add similar protections for bad auth results as we do in the normal
login request paths. Technically not 100% needed but if somehow the
handleCancelableRequest doesn't use the handleLoginRequest code path
we could get into trouble in the future
- Add delegated-auth-accessors flag to the secrets tune command and
api-docs
* Unit tests and some small fixes
* Remove transit preauth test, rely on unit tests
* Cleanup and add a little more commentary in tests
* Fix typos, add another failure use-case which we reference a disabled auth mount
* PR Feedback
- Use router to lookup mount instead of defining a new lookup method
- Enforce auth table types and namespace when mount is found
- Define a type alias for the handleInvalidCreds
- Fix typos/grammar
- Clean up globals in test
* Additional PR feedback
- Add test for delegated auth handler
- Force batch token usage
- Add a test to validate failures if a non-batch token is used
- Check for Data member being nil in test cases
* Update failure error message around requiring batch tokens
* Trap MFA requests
* Reword some error messages
* Add test and fixes for delegated response wrapping
* Move MFA test to dedicated mount
- If the delegated auth tests were running in parallel, the MFA test
case might influence the other tests, so move the MFA to a dedicated
mount
* PR feedback: use textproto.CanonicalMIMEHeaderKey
- Change the X-Vault-Wrap-Ttl constant to X-Vault-Wrap-TTL
and use textproto.CanonicalMIMEHeaderKey to format it
within the delete call.
- This protects the code around changes of the constant typing
* PR feedback
- Append Error to RequestDelegatedAuth
- Force error interface impl through explicit nil var assignment on
RequestDelegatedAuthError
- Clean up test factory and leverage NewTestSoloCluster
- Leverage newer maps.Clone as this is 1.16 only
---------
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
* Handle expired OCSP responses from server
- If a server replied with what we considered an expired OCSP response (nextUpdate is now or in the past), and it was our only response we would panic due to missing error handling logic.
* Add cl
* Add test to demonstrate a split-brain active node when using Consul
* Add Consul session check to prevent split-brain updates
* It's not right
Co-authored-by: Josh Black <raskchanky@gmail.com>
---------
Co-authored-by: Josh Black <raskchanky@gmail.com>
* Ignore nonces when encrypting without convergence or with convergence versions > 1
* Honor nonce use warning in non-FIPS modes
* Revert "Honor nonce use warning in non-FIPS modes"
This reverts commit 2aee3dbdc11c4e333ecb20503539c7993b24ee57.
* Add a test func that removes a nonce when not needed
* err out rather than ignore the nonce
* Alter unit test to cover, also cover convergent version 3
* More unit test work
* Fix test 14
* changelog
* tests not already in a nonce present path
* Update unit test to not assume warning when nonce provided incorrectly
* remove unused test field
* Fix auto-squash events experiments
When #22835 was merged, it was auto-squashed, so the `experiments`
import was removed, but the test still referenced it.
This removes the (now unnecessary) experiment from the test.
* Allow nonces for managed keys, because we have no way of knowing if the backing cipher/mode needs one
---------
Co-authored-by: Christopher Swenson <christopher.swenson@hashicorp.com>
* Also makes plugin directory optional when registering container plugins
* And threads plugin runtime settings through to plugin execution config
* Add runsc to github runner for plugin container tests
* Fix transit panic with invalid PEM
When an invalid (non-PEM) public key is given to Transit's import, this
fails with a panic in server logs:
2023-09-05T08:11:11.526-0400 [INFO] http: panic serving 127.0.0.1:42414: runtime error: invalid memory address or nil pointer dereference
goroutine 950 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0x8371620?, 0x1050b390?})
/usr/local/go/src/runtime/panic.go:920 +0x270
github.com/hashicorp/vault/sdk/helper/keysutil.(*Policy).ImportPublicOrPrivate(0xc003fff440, {0xaf02918, 0xc004509920}, {0xaf03670, 0xc0032e4180}, {0xc004532ea0, 0x188, 0x1a0}, 0x0, {0xae7f5e0, ...})
/home/cipherboy/GitHub/cipherboy/vault/sdk/helper/keysutil/policy.go:1538 +0x687
github.com/hashicorp/vault/sdk/helper/keysutil.(*LockManager).ImportPolicy(0xc001a29410, {0xaf02918, 0xc004509920}, {{0xaf03670, 0xc0032e4180}, {0xc003eb5ab5, 0xb}, 0x3, 0x0, 0x0, ...}, ...)
/home/cipherboy/GitHub/cipherboy/vault/sdk/helper/keysutil/lock_manager.go:517 +0x38a
This is unfortunate and doesn't reveal the cause of the failure: input
was not provided in PEM format, per docs.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix additional PEM decode without error check
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Implements running plugins in containers to give them some degree
of isolation from the main Vault process and other plugins. It only
supports running on Linux initially, where it is easiest to manage unix
socket communication across the container boundary.
Additionally
* Adds -env arg to vault plugin register.
* Don't return env from 'vault plugin info'
Historically it's been omitted, and it could conceivably have secret information in
it, so if we want to return it in the response, it should probably only be via explicit
opt-in. Skipping for now though as it's not the main purpose of the commit.
* setup initial boilerplate code for sign csr endpoint
* add function to sign csr
* working version of sign csr endpoint
* improving errors for csr create and sign endpoint
* initial implementation for import leaf certificate endpoint
* check if more than one certificate was provided in the ceritificate chain
* improve validate cert public key matches transit key
* convert provided cert chain from PEM to DER so it can be parsed by
x509.ParseCertificates and fixing other bugs
* fix creation of csr from csrTemplate
* add missing persist of certificate chain after validations in set-certificate endpoint
* allow exporting a certificate-chain
* move function declaration to end of page
* improving variable and function names, removing comments
* fix certificate chain parsing - work in progress
* test for signCsr endpoint
* use Operations instead of Callbacks in framework.Path
* setup test for set-certificate endpoint
fix problems with sign-csr endpoint returning base64
* finish set-certificate endpoint test
* use public key KeyEntry fields instead of retrieving public key from private
* improve error message and make better distinction between client and server error
also moved check of key types before checking if key match to endpoint handler
* check if private key has been imported for key version selected when signing a csr
* improve errors
* add endpoint description and synopsis
* fix functions calls in backend as function names changed
* improve import cert chain test
* trim whitespaces on export certificate chain
* changelog
* pass context from handler function to policy Persist
* make fmt run
* fix: assign returned error from PersistCertificateChain to err so it can be evaluated
* additional validations and improvements to parseCertificateChain function
* add validation to check if there is only one certificate in the certificate chain and it is in the first position
* import cert chain test: move creation of cluster to exported test function
* move check of end-cert pub key algorithm and key transit algorithm match into a separate function
* test export certificate chain
* Update sdk/helper/keysutil/policy.go
Co-authored-by: Alexander Scheel <alexander.m.scheel@gmail.com>
* fix validateLeafCertPosition
* reject certificate actions on policies that allow key derivation and remove derived checks
* return UserError from CreateCSR SDK function as 400 in transit API handler
* add derived check for ED5519 keys on CreateCSR SDK func
* remove unecessary calls of x509.CreateCertificateRequest
* move validate key type match back into SDK ValidateLeafCertMatch function
* add additional validations (ValidateLeafCertKeyMatch, etc) in SDK PersistCertificateChain function
* remove uncessary call of ValidateLeafCertKeyMatch in parseImportCertChainWrite
* store certificate chain as a [][]byte instead of []*x509.Certificate
* include persisted ca chain in import cert-chain response
* remove NOTE comment
* allow exporting cert-chain even if exportable is set as false
* remove NOTE comment
* add certifcate chain to formatKeyPublic if present
also added an additional check to validate if field is added when
certchain is present
---------
Co-authored-by: Alexander Scheel <alexander.m.scheel@gmail.com>
* Initial oss-patch apply
* Added changelog
* Renamed changelog txt
* Added the imports to the handler file
* Added a check that no two ports are the same, and modified changelog
* Edited go sum entry
* Tidy up using go mod
* Use strutil instead
* Revert go sum and go mod
* Revert sdk go sum
* Edited go.sum to before
* Edited go.sum again to initial
* Revert changes
* Refactor sign-intermediate API response
- Allow the sign-intermediate response handling code to be shared
across different API calls.
* Add missing cieps.go
* Migrate protobuf generation to Buf
Buf simplifies the generation story and allows us to lean
into other features in the Buf ecosystem, such as dependency
management, linting, breaking change detection, formatting
and remote plugins.
* Format all protobuf files with buf
Also add a CI job to ensure formatting remains consistent
* Add CI job to warn on proto generate diffs
Some files were not regenerated with the latest version
of the protobuf binary. This CI job will ensure we are always
detect if the protobuf files need regenerating.
* Add CI job for linting protobuf files
* OpenAPI: Define default response structure for ListOperations
Almost all Vault ListOperation responses have an identical response
schema. Update the OpenAPI generator to know this, and remove a few
instances where that standard response schema had been manually
copy/pasted into place in individual endpoints.
* changelog
* Only render StandardListResponse schema, if an operation uses it
* Teach the response schema validation test helper about the default list schema too
---------
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Replace all time.ParseDurations with testutil.ParseDurationSeconds
* Changelog
* Import formatting
* Import formatting
* Import formatting
* Import formatting
* Semgrep rule that runs as part of CI
* Allow CSRs with basic constraint extension with IsCA=false
- We previously forbid any CSR with a basic constraint extension within the CSR.
- It was discovered that some ACME clients (Proxmox ACME client) do send us this extension with a value of IsCA to false.
- So allow the extension to be set within the ACME CSR with
a value of IsCA set to false
- Add a test for both the IsCA=true and IsCA=false use-cases and make sure we don't actually set the extension back within the generated certificate.
* PR feedback
- Move basic constraint function to sdk, increase test coverage
- Error out on extra characters being returned from the asn1 unmarshalling.
* make fmt
* Fix export of HMAC typed keys
When initially implemented, exporting HMAC keys resulted in returning
the unused, internal HMACKey value rather than the main Key value that
is used for HMAC operations.
This is a breaking change.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Consistently handle HMAC keys in keysutil
When generating HMAC-typed keys, set HMACKey = Key consistently, to
allow users of HMAC-typed keys to use them backwards compatibly.
Notably, this could discard the (unused) HMACKey field set today.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test proving export of HMAC keys work
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add WrapKey capabilities to keysutil
This allows one keysutil to wrap another key, assuming that key has an
type matching one of keysutil's allowed KeyTypes. This allows completing
the BYOK import loop with Transit, allowing imported wrapping keys to
export (wrap) other keys in transit, without having them leave in
plaintext.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add /byok-export/:dst/:src[/:version] to Transit
Still respecting exportable, we allow encrypted-only export of transit
keys to another cluster using the BYOK semantics.
In particular, this allows an operator to securely establish key
material between two separate Transit installations. This potentially
allows one cluster to be used as a source cluster (encrypting a large
amount of data) and a second cluster to decrypt this data later. This
might be useful in hybrid or site-specific deployments of Vault for
instance.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing dependency to sdk/, vault/
Also updates to a newer version while we're here.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation on BYOK export
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for BYOK export/import
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update website/content/api-docs/secret/transit.mdx
* Update builtin/logical/transit/path_byok.go
Co-authored-by: Matt Schultz <975680+schultz-is@users.noreply.github.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Matt Schultz <975680+schultz-is@users.noreply.github.com>
