* Don't allow setting dead server last contact threshold to less than 1 minute
* add changelog
* document the minimum dead server last contact threshold
* When support for service tags was added, the only way we had to parse
and dedup a list of strings also forced them to be lowercase. Now there's
another helper func that doesn't smash the case so use that instead.
* update Consul 'service_tag' documentation to include case sensitivity
* added upgrade guide for 1.15
* test for service tags
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
In my recent #21942, I overlooked the need to sort another part of the
OpenAPI document to ensure stable output.
I've also removed `strings.ToLower()` from the code I copied from, as
this code is sorting Vault API parameter names, which are all lowercase
anyway!
* OpenAPI: Define default response structure for ListOperations
Almost all Vault ListOperation responses have an identical response
schema. Update the OpenAPI generator to know this, and remove a few
instances where that standard response schema had been manually
copy/pasted into place in individual endpoints.
* changelog
* Only render StandardListResponse schema, if an operation uses it
* Teach the response schema validation test helper about the default list schema too
---------
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* OpenAPI: Fix generation of correct fields
Currently, the OpenAPI generator logic is wrong about how it maps from
Vault framework fields to OpenAPI. This manifests most obviously with
endpoints making use of `framework.OptionalParamRegex` or similar
regex-level optional path parameters, and results in various incorrect
fields showing up in the generated request structures.
The fix is a bit complicated, but in essence is just rewriting the
OpenAPI logic to properly parallel the real request processing logic.
With these changes:
* A path parameter in an optional part of the regex, no longer gets
erroneously treated as a body parameter when creating OpenAPI
endpoints that do not include the optional parameter.
* A field marked as `Query: true` no longer gets incorrectly skipped
when creating OpenAPI `POST` operations.
* changelog
This is a follow-up to #21772.
Historically, for some reason, `auth/token/revoke-orphan` was
sudo-protected by writing custom code in its handler function, instead
of via the usual declarative PathsSpecial.Root mechanism.
In fact, there was a declaration mentioning revoke-orphan in the token
backend's PathsSpecial.Root, but it was incorrect! That was corrected
in #21772, making the custom code in the handler function redundant.
However, removal of the now-redundant code was deferred to this
follow-up PR, out of an abundance of caution, and wanting extra eyes on
a change deleting a security check.
* created KubernetesCertificate component, moved functionality from configuration.hbs to kubernetes-certificate.hbs, added .scss file, fixed bug in the certificate
* change new component name from KubernetesCertificate to CertificateCard
* switched to using Hds::Card in certificate-card.hbs. Added additional padding class to spacing.scss
* added tests for the CertificateCard in certificate-card-test.js
* added changelog and copyright header
* change naming on CSS class
* changed param name used in CertificateCard to be more generic. removed one class from certificate-card.scss
* changed width of CertificateCard by adding helper class to layout.scss. removed component stylesheet for CertificateCard
* begin refactoring of event package into audit package
* audit options additions
* rename option structs
* Trying to remove 'audit' from the start of names.
* typo
* typo
* typo
* newEvent required params
* typo
* comments on noop sink
* more refactoring - merge json/jsonx formatters
* fix file backend and tests
* Moved unexported funcs to formatter, fixed file tests
* typos, comments, moved func
* fix corehelpers
* fix backends (syslog, socket)
* Moved some sinks back to generic event package.
* return of the file sink
* remove unneeded sink params/return vars
* Implement Register and Deregister Audit Devices for EventLogger Framework (#21940)
* add function to create StdoutSinkNode
* add boolean argument to audit Factory function
* create eventlogger nodes in backend factory functions
* simplify NewNoopSink function and remove DiscardSinkNode
* make the sanity test in the file backend mutually exclusive based on useEventLogger value
* remove test cases that no longer made sense and were failing
* NewFileSink attempts to open file for sanity check
* fix FileSink tests and update FileSink to remove discard, stdout but add /dev/null
* Moved WithPrefix from FileSink to EventFormatter
* move prefix in backend
* NewFormatterConfig and Options (tests fixed)
* Little tidy up
* add test where audit file is created with useEventLogger set to true
* only create eventlogger.Node instances when useEventLogger is true
fix failing test due to invalid string conversion of FileMode value
* moved variable definition to more appropriate scope
---------
Co-authored-by: Marc Boudreau <marc.boudreau@hashicorp.com>
* Add missing `Query: true` metadata to API definitions
Also improve the documentation comment for `Query` to guide people better how they should be setting `Query` in the future.
Endpoints affected:
- auth/approle/role/{role_name}/secret-id/destroy
- auth/approle/role/{role_name}/secret-id-accessor/destroy
- auth/token/lookup
- auth/token/lookup-self
- sys/internal/specs/openapi
- sys/wrapping/lookup
- identity/oidc/provider/{name}/authorize
There are also endpoints in the `aws` and `gcp` secrets engines which need the same treatment in their own PRs.
When working on the `auth/token/lookup-self` path, I discovered that it
had a parameter which was completely pointless - it was even documented
as unused. It only existed because the `auth/token/lookup-self` code
path was implemented by bodging the current token into the request data
and passing control to the `auth/token/lookup` handler directly -
instead of just factoring out the common code to a reusable function -
so I fixed that whilst I was there.
Note that two of the affected endpoints currently have one form of their
OpenAPI operation ID set to something mentioning "with-parameters":
- identity/oidc/provider/{name}/authorize
- sys/internal/specs/openapi
These operation IDs should be changed, as they perpetuate
a misunderstanding - both read (GET) and update (POST/PUT) forms of
these APIs are **equally** capable of being used with parameters.
* I failed to spot that the aws plugin is in-repo! Update that too.
* Remove code cleanup changes from this PR
* Wording and wrapping adjustment as requested.
* Fix regexes for `sys/raw/` and `sys/leases/lookup/` to match prevailing conventions
There are several endpoints in Vault which take an arbitrary path as the
last parameter. Many of these are defined in terms of the
`framework.MatchAllRegex` helper. Some were not, and were defined using
custom regexes which gave rise to multiple OpenAPI endpoints - one with
the path parameter, and one without.
We need to fix these definitions, because they give rise to a very
unnatural result when used to generate a client API - for example, you
end up with `LeasesLookUp()` which is only capable of being used to list
the very top level of the hierarchical collection of leases, and
`LeasesLookUpWithPrefix(prefix)` which must be used for all deeper
levels.
This PR changes the regexes used for `sys/raw/` and `sys/leases/lookup/`
to be consistent with the approach used for other well-known similar
endpoints, such as `cubbyhole/`, `kv-v1/` and `kv-v2/metadata/`.
This PR does have a very small compatibility issue, which I think is
tolerable: prior to this change, `sys/raw` with no trailing slash was
considered a valid endpoint, and now it will no longer be.
One way to observe this is to try `vault path-help sys/raw` - before
this change, it would work, after, it will not. You would have to
instead use `vault path-help sys/raw/foobar` to see the help.
I also considered whether losing the ability to read/write/delete
`sys/raw` would be an issue. In each case, the precise HTTP result code
will change, but each of these were meaningless operations that make no
sense - you cannot read/write/delete a "file" at the "root directory" of
the underlying Vault storage.
In fact, during testing, I discovered that currently, `vault write
sys/raw x=y` when using Raft storage, will permanently break the Vault
instance - it causes a panic within the Raft FSM, which re-occurs
immediately on restarting the server! This PR also closes off that
footgun / DoS vector.
None of these issues apply to `sys/leases/lookup/`, as the existing
regex in that case was already not matching the path without the
trailing slash.
* changelog
* Realign hardcoded sudo paths with updated OpenAPI spec
* Docs: fix inaccurate claim that audit log contains all requests
* I realised there is another category of paths to add
* Unauthenticated requests such as pki/cert/FINGERPRINT are logged
So, remove "authenticated" qualifier.
We further optimize the CI workflow for better costs and speed.
We tested the Go CI workflows across several instance classes
and update our compute choices. We achieve an average execution
speed improvement of 2-2.5 minutes per test workflow while
reducing the infrastructure cost by about 20%. We also also save
another ~2 minutes by installing `gotestsum` from the Github release
instead of downloading the Go modules and compiling it every time.
In addition to the speed improvements, we also further reduced our cache
usage by updating the `security-scan` workflow to not cache Go modules.
We also use the `cache/save` and `cache/restore` actions for timing
caches. This results is saving half as many cache results for timing
data.
*UI test results*
results for 2x runs:
* c6a.2xlarge (12m54s, 11m55s)
* c6a.4xlarge (10m47s, 11m6s)
* c6a.8xlarge (11m32s, 10m51s)
* m5.2xlarge (15m23s, 14m16s)
* m5.4xlarge (14m48s, 12m54s)
* m5.8xlarge (12m27s, 12m24s)
* m6a.2xlarge (11m55s, 12m20s)
* m6a.4xlarge (10m54s, 10m43s)
* m6a.8xlarge (10m33s, 10m51s)
Current runner:
m5.2xlarge (15m23s, 14m16s, avg 14m50s) @ 0.448/hr = $0.11
Faster candidates
* c6a.2xlarge (12m54s, 11m55s, avg 12m24s) @ 0.3816/hr = $0.078
* m6a.2xlarge (11m55s, 12m20s, avg 12m8s) @ 0.4032/hr = $0.081
* c6a.4xlarge (10m47s, 11m6s, avg 10m56s) @ 0.7632/hr = $0.139
* m6a.4xlarge (10m54s, 10m43s, avg 10m48s) @ 0.8064/hr = $0.140
Best bang for the buck for test-ui:
m6a.2xlarge, > 25% cost savings from current and we save ~2.5 minutes.
*Go test results*
During testing the external replication tests, when not broken up, will
always take the longest. Our original analysis focuses on this job.
Most other tests groups will finish ~3m faster so we'll use subtract
that time when estimating the cost for the whole job.
external replication job results:
* c6a.2xlarge (20m49s, 19m20s, avg 20m5s)
* c6a.4xlarge (19m1s, 19m38s, avg 19m20s)
* c6a.8xlarge (19m51s, 18m54s, avg 19m23s)
* m5.2xlarge (22m12s, 20m29s, avg 21m20s)
* m5.4xlarge (20m7s, 19m3s, avg 20m35s)
* m5.8xlarge (20m24s, 19m42s, avg 20m3s)
* m6a.2xlarge (21m10s, 19m37s, avg 20m23s)
* m6a.4xlarge (18m58s, 19m51s, avg 19m24s)
* m6a.8xlarge (19m27s, 18m47s, avg 19m7s)
There is little separation in time when we increase class size. In the
best case a class size increase yields about a ~5% performance increase
and doubles the cost. For test-go our best bang for the buck is
certainly going to be in the 2xlarge class.
Current runner:
m5.2xlarge (22m12s, 20m29s, avg 21m20s) @ 0.448/hr (16@avg-3m + 1@avg) = $2.35
Candidates in the same class
* c6a.2xlarge (20m49s, 19m20s, avg 20m5s) @ 0.3816/hr (16@avg-3m + 1@avg) = $1.86
* m6a.2xlarge (21m10s, 19m37s, avg 20m23s) @ 0.4032/hr (16@avg-3m + 1@avg) = $2.00
Best bang for the buck for test-go:
c6a.2xlarge: 20% cost savings and save about ~2.25 minutes.
We ran the tests with similar instances and saw similar execution times as
with test-go. Therefore we can use the same recommended instance sizes.
After breaking up test-go's external replication tests, the longest group
was shorter on average. I choose to look at group 3 as it was usually the
longest grouping:
* c6a.2xlarge: (14m51s, 14m48s)
* c6a.4xlarge: (14m14s, 14m15)
* c6a.8xlarge: (14m0s, 13m54s)
* m5.2xlarge: (15m36s, 15m35s)
* m5.4xlarge: (14m46s, 14m49s)
* m5.8xlarge: (14m25s, 14m25s)
* m6a.2xlarge: 14m51s, 14m53s)
* m6a.4xlarge: 14m16s, 14m16s)
* m6a.8xlarge: (14m2s, 13m57s)
Again, we see ~5% performance gains between the 2x and 8x instance classes
at quadruple the cost. The c6a and m6a families are almost identical, with
the c6a class being cheaper.
*Notes*
* UI and Go Test timing results: https://github.com/hashicorp/vault-enterprise/actions/runs/5556957460/jobs/10150759959
* Go Test with data race detection timing results: https://github.com/hashicorp/vault-enterprise/actions/runs/5558013192
* Go Test with replication broken up: https://github.com/hashicorp/vault-enterprise/actions/runs/5558490899
Signed-off-by: Ryan Cragun <me@ryan.ec>
* Sync missing scenarios and modules
* Clean up variables and examples vars
* Add a `lint` make target for enos
* Update enos `fmt` workflow to run the `lint` target.
* Always use ipv4 addresses in target security groups.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* created new JsonTemplate component
* used JsonTemplate in modal PolicyTemplate to replace code there
* renamed component and fixed when the editor content shows up
* changed PolicyForm to render example modal only conditionally. added desription to policy-example.js
* fixed bug in policy-example.js & edited description of that file, removed functionality from policy-template.js (it is already in policy-example.js)
* changed margin on text to better match Figma design, added example modal for when editing a policy
* added tests for PolicyExample in policy-example-tests
* added PolicyForm tests for (1) cancelling the creation/edit of policy and (2) properly rendering the policy example modal
* add changelog
* clean up code by removing unnecessary comments
* changed a conditional in policy-form.hbs for better readability (Kianna's comment)
* fixed description in policy-example.js, changed wording for RGP example, changed wording in policy-form-test.js
* added 2 more asserts in policy-form-test.js. Changed some naming for selectors in the test file
* added EGP policy to PolicyExample component, moved some functionality from .hbs to .js file and vise versa
* added tests to policy-exammple-test.js and policy-form-test.js to account for new EGP policy
* simplified all PolicyExample tests in policy-exmaple-test.js
* removed beforeEach hook in policy-exmaple-test.js
* add useEventLogger argument to audit Factory functions
* adjusting Factory functions defined in tests
* fixup! adjusting Factory functions defined in tests
* limit test comments
* remove unecessary tee
* fix go test condition
* fix
* fail test
* remove ailways entirely
* fix columns
* make a bunch of tests fail
* separate line
* include Failures:
* remove test fails
* fix whitespace
* Update libraries.mdx
Hello! We now maintain a dotnet client for Vault, updating the docs to include this info.
* Update libraries.mdx
Removed version flag from Install-Package.
Changed formatting slightly of ansible-vault to match as I think the hyphen + capital first letter looks nicer.
* Remove extra '('
---------
Co-authored-by: Yoko Hyakuna <yoko@hashicorp.com>
* Fix sudo paths missing from OpenAPI and docs
Various sudo (a.k.a. root-protected) paths are implemented in
non-standard ways, and as a result:
* are not declared as x-vault-sudo in the OpenAPI spec
* and as a result of that, are not included in the hardcoded patterns
powering the Vault CLI `-output-policy` flag
* and in some cases are missing from the table of all sudo paths in the
docs too
Fix these problems by:
* Adding `seal` and `step-down` to the list of root paths for the system
backend. They don't need to be there for enforcement, as those two
special endpoints bypass the standard request handling code, but they
do need to be there for the OpenAPI generator to be able to know they
require sudo.
The way in which those two endpoints do things differently can be
observed in the code search results for `RootPrivsRequired`:
https://github.com/search?q=repo%3Ahashicorp%2Fvault%20RootPrivsRequired&type=code
* Fix the implementation of `auth/token/revoke-orphan` to implement
endpoint sudo requirements in the standard way. Currently, it has an
**incorrect** path declared in the special paths metadata, and then
compensates with custom code throwing an error within the request
handler function itself.
* changelog
* As discussed in PR, delete test which is just testing equality of a constant
* Restore sudo check as requested, and add comment
* Update vault/token_store.go
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
---------
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
