* add gosimport to make fmt and run it
* move installation to tools.sh
* correct weird spacing issue
* Update Makefile
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* fix a weird issue
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Validate audit filter option against filter selectors referencing unsupported fields
* Test updates due to filter validation
* Test all properties of the log input bexpr datum struct in filters
* Remove redundant cloning of the client in external tests for audit filtering
* TestAuditFilteringFilterForUnsupportedField now also tests the same behaviour with skip_test option set to true
* Add filter validation test cases to unit tests for audit backends
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Re-process .well-known redirects with a recursive handler call rather than a 302 redirect
* Track when the RequestURI mismatches path (in a redirect) and add it to the audit log
* call cancelFunc
* audit: entry_formatter update to ensure no race detection issues
* in progress with looking at a clone method for LogInput
* Tidy up LogInput Clone method
* less memory allocation
* fix hmac key clone
* Work towards removing the feature flag that disabled eventlogger for audit events
* Removed audited headers from LogRequest and LogResponse and clean up
* make clear we don't use a method param, and comment tweak
* Moved BenchmarkAuditFile_request to audit_broker_test and renamed. Clean up
* fixed calls from tests to Factory's
* waffling godoc for a ported and tweaked test
* Remove duplicate code from previous merges, remove uneeded code
* Refactor file audit backend tests
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* VAULT-22481: Audit filter node (#24465)
* Initial commit on adding filter nodes for audit
* tests for audit filter
* test: longer filter - more conditions
* copywrite headers
* Check interface for the right type
* Add audit filtering feature (#24554)
* Support filter nodes in backend factories and add some tests
* More tests and cleanup
* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)
* invert control of the pipelines/nodes to the audit broker vs. within each backend
* update noop audit test code to implement the pipeliner interface
* noop mount path has trailing slash
* attempting to make NoopAudit more friendly
* NoopAudit uses known salt
* Refactor audit.ProcessManual to support filter nodes
* HasFiltering
* rename the pipeliner
* use exported AuditEvent in Filter
* Add tests for registering and deregistering backends on the audit broker
* Add missing licence header to one file, fix a typo in two tests
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Add changelog file
* initial work on global metrics for sink success/failure
* initial work to add a fallback device for audit
* Return when we have outright errors
* Improve comment
* Remove unneeded options on NewBroker and remove the policy opts elsewhere
* Remove duplicate node registration code
* Add more tests for audit backends
* ensure we return the multierror as soon as possible, and append it correctly
* error tweaks for audit: log req/resp
* extract the registration for fallback/normal devices, and ensure we always add to backends when successful
* slightly nicer error message rather than returning the raw err
* refactor the deregister methods for audit broker
* Prevent issues if fallback device is the first device added
* Bail early when the user tries adding more than one fallback audit device
* Check if there is an existing fallback audit device when setting the required sinks threshold for an audit broker
* Use the right ParseBool in audit backends
* Tweak the way we check for the threshold to make it clear why we ignore fallback
* Ensure all 'fallback' settings look the same
* nicer formatting of error
* broker tests for Register
* Deregister tests
* Deregister checks if registered before attempting
* Comment improvement
* Multiple Deregister calls are OK
* Fallback not required in this test
* Sanitise input for Deregister
* Locking mixup
* fix test
* Add changelog
* Check fallback broker's sink success threshold for register/deregister
* Remove changelog
* updated
* better name for the audit metrics labelers
* extra test
* remove name from metric counter type
* update func calls for NewMetricsCounter
* labelers should be pointers to the instance
* revert audit_test complaints about the header
* use constant value for the metric label on a fallback miss
* remove vault prefix from metric labels
* US spelling for labeler and adjust the way the labels are returned
* Fixed name and type we're testing for
* Defensive addition to HasFiltering (no nodemap no filter node)
* Remove dupe code block
* Revert to using armon/go-metrics
* Fallback miss fix
* PR feedback updates
* consistent format for configure methods
* Updated telemetry set up based on PR feedback
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* VAULT-22481: Audit filter node (#24465)
* Initial commit on adding filter nodes for audit
* tests for audit filter
* test: longer filter - more conditions
* copywrite headers
* Check interface for the right type
* Add audit filtering feature (#24554)
* Support filter nodes in backend factories and add some tests
* More tests and cleanup
* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)
* invert control of the pipelines/nodes to the audit broker vs. within each backend
* update noop audit test code to implement the pipeliner interface
* noop mount path has trailing slash
* attempting to make NoopAudit more friendly
* NoopAudit uses known salt
* Refactor audit.ProcessManual to support filter nodes
* HasFiltering
* rename the pipeliner
* use exported AuditEvent in Filter
* Add tests for registering and deregistering backends on the audit broker
* Add missing licence header to one file, fix a typo in two tests
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Add changelog file
* update bexpr datum to use a strong type
* go docs updates
* test path
* PR review comments
* handle scenarios/outcomes from broker.send
* don't need to re-check the complete sinks
* add extra check to deregister to ensure that re-registering non-filtered device sets sink threshold
* Ensure that the multierror is appended before attempting to return it
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Export audit event
* Move older tests away from audit behavior that didn't use eventlogger
* spelling--;
* no more struct initialization of NoopAudit outside of NewNoopAudit
* locking since we're accessing the shared backend
* add escape hatch to use feature flag for reversion of audit behavior
* Setup pipeline which ends with a NoopSink
* explicitly call out old way of running test
* old behavior for audit trail tests
* More manual forcing of tests to legacy audit system
* Add NOTE: to suggest that the feature flag is temporary
* use non-persistent Salter for logging test message
* adjust tests based on code changes to ProcessManual
* suggestion for log test message fix (#22320)
* clean up test code and fix misnamed elements
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* add hashfunc field to EntryFormatter struct and adjust NewEntryFormatter function and tests
* add HeaderAdjuster interface and require it in EntryFormatter
* adjust all references to NewEntryFormatter to include a HeaderAdjuster parameter
* replace use of hash function in AuditedHeadersConfig's ApplyConfig method with Salter interface instance
* export audit.NewEvent function and adjust function signature
* add eventlogger based handling in LogRequest
* adjust eventlogger.Broker according to number of backends
* record auditing metrics
* only send events if a pipeline is registered
* remove TODO comments
* remove unused struct and method
* move setup of audited headers earlier into Core's initialization
* adjust entry_formatter to properly handle request headers
* protect against potential segmentation fault
* moved common code out of both switch cases
* protect against case where a.Data.Request or a.Data.Request.Headers is nil
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* add hashfunc field to EntryFormatter struct and adjust NewEntryFormatter function and tests
* add HeaderAdjuster interface and require it in EntryFormatter
dquote> adjust all references to NewEntryFormatter to include a HeaderAdjuster parameter
* replace use of hash function in AuditedHeadersConfig's ApplyConfig method with Salter interface instance
* fixup! replace use of hash function in AuditedHeadersConfig's ApplyConfig method with Salter interface instance
* review feedback
* Go doc typo
* add another test function
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* begin refactoring of event package into audit package
* audit options additions
* rename option structs
* Trying to remove 'audit' from the start of names.
* typo
* typo
* typo
* newEvent required params
* typo
* comments on noop sink
* more refactoring - merge json/jsonx formatters
* fix file backend and tests
* Moved unexported funcs to formatter, fixed file tests
* typos, comments, moved func
* fix corehelpers
* fix backends (syslog, socket)
* Moved some sinks back to generic event package.
* return of the file sink
* remove unneeded sink params/return vars
* Implement Register and Deregister Audit Devices for EventLogger Framework (#21940)
* add function to create StdoutSinkNode
* add boolean argument to audit Factory function
* create eventlogger nodes in backend factory functions
* simplify NewNoopSink function and remove DiscardSinkNode
* make the sanity test in the file backend mutually exclusive based on useEventLogger value
* remove test cases that no longer made sense and were failing
* NewFileSink attempts to open file for sanity check
* fix FileSink tests and update FileSink to remove discard, stdout but add /dev/null
* Moved WithPrefix from FileSink to EventFormatter
* move prefix in backend
* NewFormatterConfig and Options (tests fixed)
* Little tidy up
* add test where audit file is created with useEventLogger set to true
* only create eventlogger.Node instances when useEventLogger is true
fix failing test due to invalid string conversion of FileMode value
* moved variable definition to more appropriate scope
---------
Co-authored-by: Marc Boudreau <marc.boudreau@hashicorp.com>
* add useEventLogger argument to audit Factory functions
* adjusting Factory functions defined in tests
* fixup! adjusting Factory functions defined in tests
This PR relates to a feature request logged through HashiCorp commercial
support.
Vault lacks pagination in its APIs. As a result, certain list operations
can return **very** large responses. The user's chosen audit sinks may
experience difficulty consuming audit records that swell to tens of
megabytes of JSON.
In our case, one of the systems consuming audit log data could not cope,
and failed.
The responses of list operations are typically not very interesting, as
they are mostly lists of keys, or, even when they include a "key_info"
field, are not returning confidential information. They become even less
interesting once HMAC-ed by the audit system.
Some example Vault "list" operations that are prone to becoming very
large in an active Vault installation are:
auth/token/accessors/
identity/entity/id/
identity/entity-alias/id/
pki/certs/
In response, I've coded a new option that can be applied to audit
backends, `elide_list_responses`. When enabled, response data is elided
from audit logs, only when the operation type is "list".
For added safety, the elision only applies to the "keys" and "key_info"
fields within the response data - these are conventionally the only
fields present in a list response - see logical.ListResponse, and
logical.ListResponseWithInfo. However, other fields are technically
possible if a plugin author writes unusual code, and these will be
preserved in the audit log even with this option enabled.
The elision replaces the values of the "keys" and "key_info" fields with
an integer count of the number of entries. This allows even the elided
audit logs to still be useful for answering questions like "Was any data
returned?" or "How many records were listed?".
* fix adding clientID to request in audit log
* fix boolean statement
* use standard encoding for client ID instead of urlEncoding
* change encoding in tests
* add in client counts to request handling
* remove redundant client ID generation in request handling
* directly add clientID to req after handling token usage
* Send a test message before committing a new audit device.
Also, lower timeout on connection attempts in socket device.
* added changelog
* go mod vendor (picked up some unrelated changes.)
* Skip audit device check in integration test.
Co-authored-by: swayne275 <swayne@hashicorp.com>
