Fix missing log files: we need to use an absolute path, since go test chdirs into the test package dir before running tests. Move the cleanup-on-success behaviour from NewTestCluster into NewTestLogger so it applies more broadly.
This causes the registry to now contain ent plugins on ent; previously it did not, though that appears to have been the intention. I believe this is because of the order in which inits were run.
Having changed this, various tests broke that were relying on the incorrect behaviour. Several tests were changed to rely less on opaque counts of expected plugins, instead they're now using explicit comparison by name.
* create custom type for disable-replication-status-endpoints context key
make use of custom context key type in middleware function
* clean up code to remove various compiler warnings
unnecessary return statement
if condition that is always true
fix use of deprecated ioutil.NopCloser
empty if block
* remove unused unexported function
* clean up code
remove unnecessary nil check around a range expression
* clean up code
removed redundant return statement
* use http.StatusTemporaryRedirect constant instead of literal integer
* create custom type for context key for max_request_size parameter
* create custom type for context key for original request path
* Add test to demonstrate a split-brain active node when using Consul
* Add Consul session check to prevent split-brain updates
* It's not right
Co-authored-by: Josh Black <raskchanky@gmail.com>
---------
Co-authored-by: Josh Black <raskchanky@gmail.com>
The flag `events.alpha1` will no longer do anything, but we keep it
to prevent breaking users who have it in their configurations or
startup flags, or if it is referenced in other code.
Implements running plugins in containers to give them some degree
of isolation from the main Vault process and other plugins. It only
supports running on Linux initially, where it is easiest to manage unix
socket communication across the container boundary.
Additionally
* Adds -env arg to vault plugin register.
* Don't return env from 'vault plugin info'
Historically it's been omitted, and it could conceivably have secret information in
it, so if we want to return it in the response, it should probably only be via explicit
opt-in. Skipping for now though as it's not the main purpose of the commit.
* Seal HA: Use new SealWrappedValue type to abstract seal wrapped values
Introduce SealWrappedValue to abstract seal wrapped values.
Make SealWrappedValue capable of marshalling into a BlobInfo, when there is
plaintext or a single encryption, or to a custom serialization consisting of a
header, length and a marshalled MultiWrapValue protobuf.
* Vault-13769: Support configuring and using multiple seals for unsealing
* Make sealWrapBackend start using multiple seals
* Make seal.Access no longer implement wrapping.Wrapper.
Instead, add the Encrypt and Decrypt methods to the Access interface.
* Make raft snapshot system use funcs SealWrapValue + UnsealWrapValue.
Move the snapshot.Sealer implementation to the vault package to
avoid circular imports.
* Update sealWrapBackend to use multiple seals for encryption.
Use all the encryption wrappers when storing seal wrapped values.
Try do decrypt using the highest priority wrapper, but try all
combinations of encrypted values and wrappers if necessary.
* Allow the use of multiple seals for entropy augmentation
Add seal_name variable in entropy stanza
Add new MultiSourcer to accommodate the new entropy augmentation behavior.
* Individually health check each wrapper, and add a sys/seal-backend-status endpoint.
* Address a race, and also a failed test mock that I didn't catch
* Track partial wrapping failures...
... where one or more but not all access.Encrypts fail for a given write.
Note these failures by adding a time ordered UUID storage entry containing
the path in a special subdirectory of root storage. Adds a callback
pattern to accomplish this, with certain high value writes like initial
barrier key storage not allowing a partial failure. The followup work
would be to detect return to health and iterate through these storage
entries, rewrapping.
* Add new data structure to track seal config generation (#4492)
* Add new data structure to track seal config generation
* Remove import cycle
* Fix undefined variable errors
* update comment
* Update setSeal response
* Fix setSealResponse in operator_diagnose
* Scope the wrapper health check locks individually (#4491)
* Refactor setSeal function in server.go. (#4505)
Refactor setSeal function in server.go.
* Decouple CreateSecureRandomReaderFunc from seal package.
Instead of using a list of seal.SealInfo structs, make
CreateSecureRandomReaderFunc use a list of new EntropySourcerInfo structs. This
brakes the denpency of package configutil on the seal package.
* Move SealGenerationInfo tracking to the seal Access.
* Move SealGenerationInfo tracking to the seal Access.
The SealGenerationInfo is now kept track by a Seal's Access instead of by the
Config object. The access implementation now records the correct generation
number on seal wrapped values.
* Only store and read SealGenerationInfo if VAULT_ENABLE_SEAL_HA_BETA is true.
* Add MultiWrapValue protobuf message
MultiWrapValue can be used to keep track of different encryptions of a value.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* Use generation to determine if a seal wrapped value is up-to-date. (#4542)
* Add logging to seal Access implementation.
* Seal HA buf format run (#4561)
* Run buf format.
* Add buf.lock to ensure go-kms-wrapping module is imported.
* Vault-18958: Add unit tests for config checks
* Add safety logic for seal configuration changes
* Revert "Add safety logic for seal configuration changes"
This reverts commit 7fec48035a5cf274e5a4d98901716d08d766ce90.
* changes and tests for checking seal config
* add ent tests
* remove check for empty name and add type into test cases
* add error message for empty name
* fix no seals test
---------
Co-authored-by: divyapola5 <divya@hashicorp.com>
* Handle migrations between single-wrapper and multi-wrapper autoSeals
* Extract method SetPhysicalSealConfig.
* Extract function physicalSealConfig.
The extracted function is the only code now reading SealConfig entries from
storage.
* Extract function setPhysicalSealConfig.
The extracted function is the only code now writing SealConfig entries from
storage (except for migration from the old recovery config path).
* Move SealConfig to new file vault/seal_config.go.
* Add SealConfigType quasy-enumeration.
SealConfigType is to serve as the typed values for field SealConfig.Type.
* Rename Seal.RecoveryType to RecoverySealConfigType.
Make RecoverySealConfigType return a SealConfigType instead of a string.
* Rename Seal.BarrierType to BarrierSealConfigType.
Make BarrierSealConfigType return a SealConfigType.
Remove seal.SealType (really a two-step rename to SealConfigType).
* Add Seal methods ClearBarrierConfig and ClearRecoveryConfig.
* Handle autoseal <-> multiseal migrations.
While going between single-wrapper and multiple-wrapper autoseals are not
migrations that require an unwrap seal (such as going from shamir to autoseal),
the stored "barrier" SealConfig needs to be updated in these cases.
Specifically, the value of SealConfg.Type is "multiseal" for autoSeals that have
more than one wrapper; on the other hand, for autoseals with a single wrapper,
SealConfig.Type is the type of the wrapper.
* Remove error return value from NewAutoSeal constructor.
* Automatically rewrap partially seal wrapped values on an interval
* Add in rewrapping of partially wrapped values on an interval, regardless of seal health/status.
* Don't set SealGenerationInfo Rewrapped flag in the partial rewrap call.
* Unexport the SealGenerationInfo's Rewrapped field, add a mutex to it for thread safe access, and add accessor methods for it.
* Add a success callback to the manual seal rewrap process that updates the SealGenerationInfo's rewrapped field. This is done via a callback to avoid an import cycle in the SealRewrap code.
* Fix a failing seal wrap backend test which was broken by the unexporting of SealGenerationInfo's Rewrapped field.
* Nil check the seal rewrap success callback before calling it.
* Change SealGenerationInfo rewrapped parameter to an atomic.Bool rather than a sync.RWMutex for simplicity and performance.
* Add nil check for SealAccess before updating SealGenerationInfo rewrapped status during seal rewrap call.
* Update partial rewrap check interval from 10 seconds to 1 minute.
* Update a reference to SealGenerationInfo Rewrapped field to use new getter method.
* Fix up some data raciness in partial rewrapping.
* Account for possibly nil storage entry when retrieving partially wrapped value.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
* Restore propagation of wrapper configuration errors by setSeal.
Function setSeal is meant to propagate non KeyNotFound errors returned by calls
to configutil.ConfigureWrapper.
* Remove unused Access methods SetConfig and Type.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
Make it possible for an autoSeal that uses multiple wrappers to include disabled
wrappers that can be used to decrypt entries, but are skipped for encryption.
e an unwrapSeal when there are disabled seals.
* Fix bug with not providing name (#4580)
* add suffix to name defaults
* add comment
* only change name for disabled seal
* Only attempt to rewrap partial values when all seals are healthy.
* Only attempt to rewrap partial values when all seals are healthy.
* Change logging level from info to debug for notice about rewrap skipping based on seal health.
* Remove stale TODOs and commented out code.
---------
Co-authored-by: rculpepper <rculpepper@hashicorp.com>
Co-authored-by: Larroyo <95649169+DeLuci@users.noreply.github.com>
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
Co-authored-by: Divya Pola <87338962+divyapola5@users.noreply.github.com>
Co-authored-by: Matt Schultz <matt.schultz@hashicorp.com>
Co-authored-by: divyapola5 <divya@hashicorp.com>
Co-authored-by: Rachel Culpepper <84159930+rculpepper@users.noreply.github.com>
* add escape hatch to use feature flag for reversion of audit behavior
* Setup pipeline which ends with a NoopSink
* explicitly call out old way of running test
* old behavior for audit trail tests
* More manual forcing of tests to legacy audit system
* Add NOTE: to suggest that the feature flag is temporary
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* Automatically track subloggers in allLoggers
This PR introduces a new interface called `SubloggerAdder` for tracking
allLoggers across startup phases.
The interface enables us to register a single `SubloggerHook` during
initial logger creation and hand off management of `allLoggers` during
the three phases we need to handle:
- Before `CoreConfig` is created, the `SubloggerHook`
(`AppendToAllLoggers`) appends new subloggers to
`ServerCommand.allLoggers`.
- After `CoreConfig` is created and before `NewCore` returns, new subloggers
are added to `CoreConfig.AllLoggers`. Intermediate state must also be
kept in sync within NewCore to track new subloggers before we return
to the server command and register the `Core.SubloggerAdder`
implementation.
- After `NewCore` returns to the server command, we register Core as the
implementer of `ServerCommand.SubloggerAdder` ensuring that all new
subloggers are appended to `Core.allLoggers`.
* Wire up the sublogger hook in NewTestLogger
* Use Hashicorp docker proxy for CI docker images
* Use library/consul as the mirror path instead of hashicorp/consul
- Looks like the older 1.4.4 image was not published within the
hashicorp/consul space, only newer images are.
- Switch to library/consul which seems to have both versions
* Migrate protobuf generation to Buf
Buf simplifies the generation story and allows us to lean
into other features in the Buf ecosystem, such as dependency
management, linting, breaking change detection, formatting
and remote plugins.
* Format all protobuf files with buf
Also add a CI job to ensure formatting remains consistent
* Add CI job to warn on proto generate diffs
Some files were not regenerated with the latest version
of the protobuf binary. This CI job will ensure we are always
detect if the protobuf files need regenerating.
* Add CI job for linting protobuf files
* add hashfunc field to EntryFormatter struct and adjust NewEntryFormatter function and tests
* add HeaderAdjuster interface and require it in EntryFormatter
dquote> adjust all references to NewEntryFormatter to include a HeaderAdjuster parameter
* replace use of hash function in AuditedHeadersConfig's ApplyConfig method with Salter interface instance
* fixup! replace use of hash function in AuditedHeadersConfig's ApplyConfig method with Salter interface instance
* review feedback
* Go doc typo
* add another test function
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* add useEventLogger argument to audit Factory functions
* adjusting Factory functions defined in tests
* fixup! adjusting Factory functions defined in tests
* CreateOperation should only be implemented alongside ExistenceCheck
Closes#12329
Vault treats all POST or PUT HTTP requests equally - they default to
being treated as UpdateOperations, but, if a backend implements an
ExistenceCheck function, CreateOperations can be separated out when the
existence check returns false.
It follows, then, that if a CreateOperation handler is implemented
without an ExistenceCheck function, this is unreachable code - a coding
error. It's a fairly minor error in the grand scheme of things, but it
causes the generated OpenAPI spec to include x-vault-createSupported for
operations on which create can never actually be invoked - and promotes
muddled understanding of the create/update feature.
In this PR:
1) Implement a new test, which checks all builtin auth methods and
secrets engines can be successfully initialized. (This is important
to validate the next part.)
2) Expand upon the existing coding error checks built in to
framework.Backend, adding a check for this misuse of CreateOperation.
3) Fix up instances of improper CreateOperation within the Vault
repository - just two, transit and mock.
Note: At this point, the newly added test will **fail**.
There are improper uses of CreateOperation in all of the following:
vault-plugin-auth-cf
vault-plugin-auth-kerberos
vault-plugin-auth-kubernetes
vault-plugin-secrets-ad
vault-plugin-secrets-gcpkms
vault-plugin-secrets-kubernetes
vault-plugin-secrets-kv
vault-plugin-secrets-openldap
vault-plugin-secrets-terraform
each of which needs to be fixed and updated in go.mod here, before this
new check can be added.
* Add subtests
* Add in testing of KV v2, which otherwise doesn't get tested
This is a surprisingly complicated special case
* The database plugin needs special handling as well, and add in help invocations of the builtin backends too
* Fix extra package prefix
* Add changelog
* Update 6 out of 9 plugins to needed new versions
Note, this IS an upgrade despite the apparent version numbers going
down. (That's a consequence of slightly odd release management occurring
in the plugin repositories.)
* Update to deal with code changes since branch originally created
* Perform necessary update of vault-plugin-secrets-kubernetes so that CI checks on PR can run
* Fix another instance of incorrect CreateOperation, for a test-only endpoint
By being hidden behind a Go build constraint, it had evaded notice until
now.
* Add an opportunistic test of sys/internal/specs/openapi too
* Ent only ADP Metrics
* Added change log
* Changed changelog name
* Restored previous impl
* Moved to mount_util
* Change impl
* Add same file
* Moved to registry_util
* Edited corehelpers mock registry
* Edited chagnelog
* Edited changelog
* Edited build tag
* Added back function
* Delete core.go.rej
* Edited mount
* Changed spacing
The upcoming event main plugin will use a very similar pattern
as the database plugin map, so it makes sense to refactor this and move
this map out. It also cleans up the database plugin backend so that
it does not have to keep track of the lock.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
The main point of this work is to move runFilteredPathsEvaluation inside mountInternal/enableCredentialInternal, while we hold the lock. This requires that we grab both the mountsLock and authLock, for reasons that only make sense when looking at the ent code.
While fixing this I noticed that ListAuths grabs the wrong lock, so I also fixed that.
And I modified ClusterSetup: as a convenience, it now populates the factories for all the builtin engines in the config it returns.
* Adds an ACME validation failure test for certbot that doesn't run in CI unless a particular regression test env var is provided. Also includes a helper function to determine whether or not CI is running and if the regression test env var is provided.
* Rename and move the local or regression test env check. Sinkhole our invalid domain for ACME certbot test to avoid spamming someone's domain if it's registered in the future.
