* begin refactoring of event package into audit package
* audit options additions
* rename option structs
* Trying to remove 'audit' from the start of names.
* typo
* typo
* typo
* newEvent required params
* typo
* comments on noop sink
* more refactoring - merge json/jsonx formatters
* fix file backend and tests
* Moved unexported funcs to formatter, fixed file tests
* typos, comments, moved func
* fix corehelpers
* fix backends (syslog, socket)
* Moved some sinks back to generic event package.
* return of the file sink
* remove unneeded sink params/return vars
* Implement Register and Deregister Audit Devices for EventLogger Framework (#21940)
* add function to create StdoutSinkNode
* add boolean argument to audit Factory function
* create eventlogger nodes in backend factory functions
* simplify NewNoopSink function and remove DiscardSinkNode
* make the sanity test in the file backend mutually exclusive based on useEventLogger value
* remove test cases that no longer made sense and were failing
* NewFileSink attempts to open file for sanity check
* fix FileSink tests and update FileSink to remove discard, stdout but add /dev/null
* Moved WithPrefix from FileSink to EventFormatter
* move prefix in backend
* NewFormatterConfig and Options (tests fixed)
* Little tidy up
* add test where audit file is created with useEventLogger set to true
* only create eventlogger.Node instances when useEventLogger is true
fix failing test due to invalid string conversion of FileMode value
* moved variable definition to more appropriate scope
---------
Co-authored-by: Marc Boudreau <marc.boudreau@hashicorp.com>
* Add missing `Query: true` metadata to API definitions
Also improve the documentation comment for `Query` to guide people better how they should be setting `Query` in the future.
Endpoints affected:
- auth/approle/role/{role_name}/secret-id/destroy
- auth/approle/role/{role_name}/secret-id-accessor/destroy
- auth/token/lookup
- auth/token/lookup-self
- sys/internal/specs/openapi
- sys/wrapping/lookup
- identity/oidc/provider/{name}/authorize
There are also endpoints in the `aws` and `gcp` secrets engines which need the same treatment in their own PRs.
When working on the `auth/token/lookup-self` path, I discovered that it
had a parameter which was completely pointless - it was even documented
as unused. It only existed because the `auth/token/lookup-self` code
path was implemented by bodging the current token into the request data
and passing control to the `auth/token/lookup` handler directly -
instead of just factoring out the common code to a reusable function -
so I fixed that whilst I was there.
Note that two of the affected endpoints currently have one form of their
OpenAPI operation ID set to something mentioning "with-parameters":
- identity/oidc/provider/{name}/authorize
- sys/internal/specs/openapi
These operation IDs should be changed, as they perpetuate
a misunderstanding - both read (GET) and update (POST/PUT) forms of
these APIs are **equally** capable of being used with parameters.
* I failed to spot that the aws plugin is in-repo! Update that too.
* Remove code cleanup changes from this PR
* Wording and wrapping adjustment as requested.
* Fix regexes for `sys/raw/` and `sys/leases/lookup/` to match prevailing conventions
There are several endpoints in Vault which take an arbitrary path as the
last parameter. Many of these are defined in terms of the
`framework.MatchAllRegex` helper. Some were not, and were defined using
custom regexes which gave rise to multiple OpenAPI endpoints - one with
the path parameter, and one without.
We need to fix these definitions, because they give rise to a very
unnatural result when used to generate a client API - for example, you
end up with `LeasesLookUp()` which is only capable of being used to list
the very top level of the hierarchical collection of leases, and
`LeasesLookUpWithPrefix(prefix)` which must be used for all deeper
levels.
This PR changes the regexes used for `sys/raw/` and `sys/leases/lookup/`
to be consistent with the approach used for other well-known similar
endpoints, such as `cubbyhole/`, `kv-v1/` and `kv-v2/metadata/`.
This PR does have a very small compatibility issue, which I think is
tolerable: prior to this change, `sys/raw` with no trailing slash was
considered a valid endpoint, and now it will no longer be.
One way to observe this is to try `vault path-help sys/raw` - before
this change, it would work, after, it will not. You would have to
instead use `vault path-help sys/raw/foobar` to see the help.
I also considered whether losing the ability to read/write/delete
`sys/raw` would be an issue. In each case, the precise HTTP result code
will change, but each of these were meaningless operations that make no
sense - you cannot read/write/delete a "file" at the "root directory" of
the underlying Vault storage.
In fact, during testing, I discovered that currently, `vault write
sys/raw x=y` when using Raft storage, will permanently break the Vault
instance - it causes a panic within the Raft FSM, which re-occurs
immediately on restarting the server! This PR also closes off that
footgun / DoS vector.
None of these issues apply to `sys/leases/lookup/`, as the existing
regex in that case was already not matching the path without the
trailing slash.
* changelog
* Realign hardcoded sudo paths with updated OpenAPI spec
* Docs: fix inaccurate claim that audit log contains all requests
* I realised there is another category of paths to add
* Unauthenticated requests such as pki/cert/FINGERPRINT are logged
So, remove "authenticated" qualifier.
We further optimize the CI workflow for better costs and speed.
We tested the Go CI workflows across several instance classes
and update our compute choices. We achieve an average execution
speed improvement of 2-2.5 minutes per test workflow while
reducing the infrastructure cost by about 20%. We also also save
another ~2 minutes by installing `gotestsum` from the Github release
instead of downloading the Go modules and compiling it every time.
In addition to the speed improvements, we also further reduced our cache
usage by updating the `security-scan` workflow to not cache Go modules.
We also use the `cache/save` and `cache/restore` actions for timing
caches. This results is saving half as many cache results for timing
data.
*UI test results*
results for 2x runs:
* c6a.2xlarge (12m54s, 11m55s)
* c6a.4xlarge (10m47s, 11m6s)
* c6a.8xlarge (11m32s, 10m51s)
* m5.2xlarge (15m23s, 14m16s)
* m5.4xlarge (14m48s, 12m54s)
* m5.8xlarge (12m27s, 12m24s)
* m6a.2xlarge (11m55s, 12m20s)
* m6a.4xlarge (10m54s, 10m43s)
* m6a.8xlarge (10m33s, 10m51s)
Current runner:
m5.2xlarge (15m23s, 14m16s, avg 14m50s) @ 0.448/hr = $0.11
Faster candidates
* c6a.2xlarge (12m54s, 11m55s, avg 12m24s) @ 0.3816/hr = $0.078
* m6a.2xlarge (11m55s, 12m20s, avg 12m8s) @ 0.4032/hr = $0.081
* c6a.4xlarge (10m47s, 11m6s, avg 10m56s) @ 0.7632/hr = $0.139
* m6a.4xlarge (10m54s, 10m43s, avg 10m48s) @ 0.8064/hr = $0.140
Best bang for the buck for test-ui:
m6a.2xlarge, > 25% cost savings from current and we save ~2.5 minutes.
*Go test results*
During testing the external replication tests, when not broken up, will
always take the longest. Our original analysis focuses on this job.
Most other tests groups will finish ~3m faster so we'll use subtract
that time when estimating the cost for the whole job.
external replication job results:
* c6a.2xlarge (20m49s, 19m20s, avg 20m5s)
* c6a.4xlarge (19m1s, 19m38s, avg 19m20s)
* c6a.8xlarge (19m51s, 18m54s, avg 19m23s)
* m5.2xlarge (22m12s, 20m29s, avg 21m20s)
* m5.4xlarge (20m7s, 19m3s, avg 20m35s)
* m5.8xlarge (20m24s, 19m42s, avg 20m3s)
* m6a.2xlarge (21m10s, 19m37s, avg 20m23s)
* m6a.4xlarge (18m58s, 19m51s, avg 19m24s)
* m6a.8xlarge (19m27s, 18m47s, avg 19m7s)
There is little separation in time when we increase class size. In the
best case a class size increase yields about a ~5% performance increase
and doubles the cost. For test-go our best bang for the buck is
certainly going to be in the 2xlarge class.
Current runner:
m5.2xlarge (22m12s, 20m29s, avg 21m20s) @ 0.448/hr (16@avg-3m + 1@avg) = $2.35
Candidates in the same class
* c6a.2xlarge (20m49s, 19m20s, avg 20m5s) @ 0.3816/hr (16@avg-3m + 1@avg) = $1.86
* m6a.2xlarge (21m10s, 19m37s, avg 20m23s) @ 0.4032/hr (16@avg-3m + 1@avg) = $2.00
Best bang for the buck for test-go:
c6a.2xlarge: 20% cost savings and save about ~2.25 minutes.
We ran the tests with similar instances and saw similar execution times as
with test-go. Therefore we can use the same recommended instance sizes.
After breaking up test-go's external replication tests, the longest group
was shorter on average. I choose to look at group 3 as it was usually the
longest grouping:
* c6a.2xlarge: (14m51s, 14m48s)
* c6a.4xlarge: (14m14s, 14m15)
* c6a.8xlarge: (14m0s, 13m54s)
* m5.2xlarge: (15m36s, 15m35s)
* m5.4xlarge: (14m46s, 14m49s)
* m5.8xlarge: (14m25s, 14m25s)
* m6a.2xlarge: 14m51s, 14m53s)
* m6a.4xlarge: 14m16s, 14m16s)
* m6a.8xlarge: (14m2s, 13m57s)
Again, we see ~5% performance gains between the 2x and 8x instance classes
at quadruple the cost. The c6a and m6a families are almost identical, with
the c6a class being cheaper.
*Notes*
* UI and Go Test timing results: https://github.com/hashicorp/vault-enterprise/actions/runs/5556957460/jobs/10150759959
* Go Test with data race detection timing results: https://github.com/hashicorp/vault-enterprise/actions/runs/5558013192
* Go Test with replication broken up: https://github.com/hashicorp/vault-enterprise/actions/runs/5558490899
Signed-off-by: Ryan Cragun <me@ryan.ec>
* Sync missing scenarios and modules
* Clean up variables and examples vars
* Add a `lint` make target for enos
* Update enos `fmt` workflow to run the `lint` target.
* Always use ipv4 addresses in target security groups.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* created new JsonTemplate component
* used JsonTemplate in modal PolicyTemplate to replace code there
* renamed component and fixed when the editor content shows up
* changed PolicyForm to render example modal only conditionally. added desription to policy-example.js
* fixed bug in policy-example.js & edited description of that file, removed functionality from policy-template.js (it is already in policy-example.js)
* changed margin on text to better match Figma design, added example modal for when editing a policy
* added tests for PolicyExample in policy-example-tests
* added PolicyForm tests for (1) cancelling the creation/edit of policy and (2) properly rendering the policy example modal
* add changelog
* clean up code by removing unnecessary comments
* changed a conditional in policy-form.hbs for better readability (Kianna's comment)
* fixed description in policy-example.js, changed wording for RGP example, changed wording in policy-form-test.js
* added 2 more asserts in policy-form-test.js. Changed some naming for selectors in the test file
* added EGP policy to PolicyExample component, moved some functionality from .hbs to .js file and vise versa
* added tests to policy-exammple-test.js and policy-form-test.js to account for new EGP policy
* simplified all PolicyExample tests in policy-exmaple-test.js
* removed beforeEach hook in policy-exmaple-test.js
* add useEventLogger argument to audit Factory functions
* adjusting Factory functions defined in tests
* fixup! adjusting Factory functions defined in tests
* limit test comments
* remove unecessary tee
* fix go test condition
* fix
* fail test
* remove ailways entirely
* fix columns
* make a bunch of tests fail
* separate line
* include Failures:
* remove test fails
* fix whitespace
* Update libraries.mdx
Hello! We now maintain a dotnet client for Vault, updating the docs to include this info.
* Update libraries.mdx
Removed version flag from Install-Package.
Changed formatting slightly of ansible-vault to match as I think the hyphen + capital first letter looks nicer.
* Remove extra '('
---------
Co-authored-by: Yoko Hyakuna <yoko@hashicorp.com>
* Fix sudo paths missing from OpenAPI and docs
Various sudo (a.k.a. root-protected) paths are implemented in
non-standard ways, and as a result:
* are not declared as x-vault-sudo in the OpenAPI spec
* and as a result of that, are not included in the hardcoded patterns
powering the Vault CLI `-output-policy` flag
* and in some cases are missing from the table of all sudo paths in the
docs too
Fix these problems by:
* Adding `seal` and `step-down` to the list of root paths for the system
backend. They don't need to be there for enforcement, as those two
special endpoints bypass the standard request handling code, but they
do need to be there for the OpenAPI generator to be able to know they
require sudo.
The way in which those two endpoints do things differently can be
observed in the code search results for `RootPrivsRequired`:
https://github.com/search?q=repo%3Ahashicorp%2Fvault%20RootPrivsRequired&type=code
* Fix the implementation of `auth/token/revoke-orphan` to implement
endpoint sudo requirements in the standard way. Currently, it has an
**incorrect** path declared in the special paths metadata, and then
compensates with custom code throwing an error within the request
handler function itself.
* changelog
* As discussed in PR, delete test which is just testing equality of a constant
* Restore sudo check as requested, and add comment
* Update vault/token_store.go
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
---------
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* fix multiline
* shellcheck, and success message for builds
* add full path
* cat the summary
* fix and faster
* fix if condition
* base64 in a separate step
* echo
* check against empty string
* add echo
* only use matrix ids
* only id
* echo matrix
* remove wrapping array
* tojson
* try echo again
* use jq to get packages
* don't quote
* only run binary tests once
* only run binary tests once
* test what's wrong with the binary
* separate file
* use matrix file
* failed test
* update comment on success
* correct variable name
* bae64 fix
* output to file
* use multiline
* fix
* fix formatting
* fix newline
* fix whitespace
* correct body, remove comma
* small fixes
* shellcheck
* another shellcheck fix
* fix deprecation checker
* only run comments for prs
* Update .github/workflows/test-go.yml
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* Update .github/workflows/test-go.yml
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* fixes
---------
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* CreateOperation should only be implemented alongside ExistenceCheck
Closes#12329
Vault treats all POST or PUT HTTP requests equally - they default to
being treated as UpdateOperations, but, if a backend implements an
ExistenceCheck function, CreateOperations can be separated out when the
existence check returns false.
It follows, then, that if a CreateOperation handler is implemented
without an ExistenceCheck function, this is unreachable code - a coding
error. It's a fairly minor error in the grand scheme of things, but it
causes the generated OpenAPI spec to include x-vault-createSupported for
operations on which create can never actually be invoked - and promotes
muddled understanding of the create/update feature.
In this PR:
1) Implement a new test, which checks all builtin auth methods and
secrets engines can be successfully initialized. (This is important
to validate the next part.)
2) Expand upon the existing coding error checks built in to
framework.Backend, adding a check for this misuse of CreateOperation.
3) Fix up instances of improper CreateOperation within the Vault
repository - just two, transit and mock.
Note: At this point, the newly added test will **fail**.
There are improper uses of CreateOperation in all of the following:
vault-plugin-auth-cf
vault-plugin-auth-kerberos
vault-plugin-auth-kubernetes
vault-plugin-secrets-ad
vault-plugin-secrets-gcpkms
vault-plugin-secrets-kubernetes
vault-plugin-secrets-kv
vault-plugin-secrets-openldap
vault-plugin-secrets-terraform
each of which needs to be fixed and updated in go.mod here, before this
new check can be added.
* Add subtests
* Add in testing of KV v2, which otherwise doesn't get tested
This is a surprisingly complicated special case
* The database plugin needs special handling as well, and add in help invocations of the builtin backends too
* Fix extra package prefix
* Add changelog
* Update 6 out of 9 plugins to needed new versions
Note, this IS an upgrade despite the apparent version numbers going
down. (That's a consequence of slightly odd release management occurring
in the plugin repositories.)
* Update to deal with code changes since branch originally created
* Perform necessary update of vault-plugin-secrets-kubernetes so that CI checks on PR can run
* Fix another instance of incorrect CreateOperation, for a test-only endpoint
By being hidden behind a Go build constraint, it had evaded notice until
now.
* Add an opportunistic test of sys/internal/specs/openapi too
This is a code cleanup and addition of an explanatory comment. For some
reason, the code related to the CLI guessing whether a path requires
sudo, has been interleaved into plugin_helpers.go, which was previously
purely code used on the server side in the implementation of Vault
plugins.
This remedies that by dividing the sudo paths code to a separate file,
and adds a comment to plugin_helpers.go providing future readers with
information about the overall theme of the file.
No code has been changed - only moved and documented.
* Fix ACME tidy to not reference acmeCtx
acmeContext is useful for when we need to reference things with a ACME
base URL, but everything used in tidy doesn't need this URL as it is not
coming from an ACME request.
Refactor tidy to remove references to acmeContext, including dependent
functions in acme_state.go.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove spurious log message
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Draft Tidy Acme Test with Backdate Storage + Backdate Sysxsx
* Fixes to ACME tidy testing
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly set account kid to update account status
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add TestTidyAcmeWithSafetyBuffer
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for disabling tidy operation
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add acme_account_safety_buffer to auto-tidy config
Resolve: #21872
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests verifying tidy safety buffers
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add account status validations and order cleanup tests
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: kitography <khaines@mit.edu>
Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
* report build failures in a PR comment
* address action linter
* linter
* add an id
* change permission
* report failure from build yaml
* linter fix
* report workflow url
* reorder jobs
* complete boolean eval
* single quote
* experiment getting failed jobs
* linter
* pass failed jobs one by one
* failed jobs are reported cancelled
* use * instead of @
* some polishing
* find comment ID, create or update it
* some clean up
* missing }
* imprv: Add a parameter to allow ExtKeyUsage field usage from a role
* chore: Add the changelog entry
* imprv: Reword UI and changelog
* doc: Add allow_role_extkeyusage in parameter list
* imprv: Align variable names with existing fields/codebase
* Add unit test and tweak some labels
---------
Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
* removing caseSensitivityKey invalidation
* add changelog
* remove caseSensitivityKey
* removing loadCaseSensitivityKeyStore
* add go test to check caseSensitivityKey deletion
* edit return error messages during deletion of caseSensitivityKey
* adding log line to record caseSensitivityKey value
* modifying error check
* addressing comments
