Combined Database Backend: Static Accounts (#6834)

* Add priority queue to sdk

* fix issue of storing pointers and now copy

* update to use copy structure

* Remove file, put Item struct def. into other file

* add link

* clean up docs

* refactor internal data structure to hide heap method implementations. Other cleanup after feedback

* rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods

* updates after feedback

* refactoring/renaming

* guard against pushing a nil item

* minor updates after feedback

* Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC

* Initial Combined database backend implementation of static accounts and automatic rotation

* vendor updates

* initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation

* add lock and setup of rotation queue

* vendor the queue

* rebase on new method signature of queue

* remove mongo tests for now

* update default role sql

* gofmt after rebase

* cleanup after rebasing to remove checks for ErrNotFound error

* rebase cdcr-priority-queue

* vendor dependencies with 'go mod vendor'

* website database docs for Static Role support

* document the rotate-role API endpoint

* postgres specific static role docs

* use constants for paths

* updates from review

* remove dead code

* combine and clarify error message for older plugins

* Update builtin/logical/database/backend.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* cleanups from feedback

* code and comment cleanups

* move db.RLock higher to protect db.GenerateCredentials call

* Return output with WALID if we failed to delete the WAL

* Update builtin/logical/database/path_creds_create.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* updates after running 'make fmt'

* update after running 'make proto'

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* update comment and remove and rearrange some dead code

* Update website/source/api/secret/databases/index.html.md

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* cleanups after review

* Update sdk/database/dbplugin/grpc_transport.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* code cleanup after feedback

* remove PasswordLastSet; it's not used

* document GenerateCredentials and SetCredentials

* Update builtin/logical/database/path_rotate_credentials.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* wrap pop and popbykey in backend methods to protect against nil cred rotation queue

* use strings.HasPrefix instead of direct equality check for path

* Forgot to commit this

* updates after feedback

* re-purpose an outdated test to now check that static and dynamic roles cannot share a name

* check for unique name across dynamic and static roles

* refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials

* remove commented out code

* refactor to have loadstaticwals filter out wals for roles that no longer exist

* return error if nil input given

* add nil check for input into setStaticAccount

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* add constant for queue tick time in seconds, used for comparrison in updates

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* code cleanup after review

* remove misplaced code comment

* remove commented out code

* create a queue in the Factory method, even if it's never used

* update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type

* document new method

* move rotation things into a specific file

* rename test file and consolidate some static account tests

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* update code comments, method names, and move more methods into rotation.go

* update comments to be capitalized

* remove the item from the queue before we try to destroy it

* findStaticWAL returns an error

* use lowercase keys when encoding WAL entries

* small cleanups

* remove vestigial static account check

* remove redundant DeleteWAL call in populate queue

* if we error on loading role, push back to queue with 10 second backoff

* poll in initqueue to make sure the backend is setup and can write/delete data

* add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false

* add code comments on read-only loop

* code comment updates

* re-push if error returned from find static wal

* add locksutil and acquire locks when pop'ing from the queue

* grab exclusive locks for updating static roles

* Add SetCredentials and GenerateCredentials stubs to mockPlugin

* add a switch in initQueue to listen for cancelation

* remove guard on zero time, it should have no affect

* create a new context in Factory to pass on and use for closing the backend queue

* restore master copy of vendor dir

sdk/plugin/pb/backend.pb.go

@@ -10,6 +10,8 @@ import (
 	timestamp "github.com/golang/protobuf/ptypes/timestamp"
 	logical "github.com/hashicorp/vault/sdk/logical"
 	grpc "google.golang.org/grpc"
+	codes "google.golang.org/grpc/codes"
+	status "google.golang.org/grpc/status"
 	math "math"
 )
 
@@ -3036,6 +3038,32 @@ type BackendServer interface {
 	Type(context.Context, *Empty) (*TypeReply, error)
 }
 
+// UnimplementedBackendServer can be embedded to have forward compatible implementations.
+type UnimplementedBackendServer struct {
+}
+
+func (*UnimplementedBackendServer) HandleRequest(ctx context.Context, req *HandleRequestArgs) (*HandleRequestReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method HandleRequest not implemented")
+}
+func (*UnimplementedBackendServer) SpecialPaths(ctx context.Context, req *Empty) (*SpecialPathsReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method SpecialPaths not implemented")
+}
+func (*UnimplementedBackendServer) HandleExistenceCheck(ctx context.Context, req *HandleExistenceCheckArgs) (*HandleExistenceCheckReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method HandleExistenceCheck not implemented")
+}
+func (*UnimplementedBackendServer) Cleanup(ctx context.Context, req *Empty) (*Empty, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Cleanup not implemented")
+}
+func (*UnimplementedBackendServer) InvalidateKey(ctx context.Context, req *InvalidateKeyArgs) (*Empty, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method InvalidateKey not implemented")
+}
+func (*UnimplementedBackendServer) Setup(ctx context.Context, req *SetupArgs) (*SetupReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Setup not implemented")
+}
+func (*UnimplementedBackendServer) Type(ctx context.Context, req *Empty) (*TypeReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Type not implemented")
+}
+
 func RegisterBackendServer(s *grpc.Server, srv BackendServer) {
 	s.RegisterService(&_Backend_serviceDesc, srv)
 }
@@ -3265,6 +3293,23 @@ type StorageServer interface {
 	Delete(context.Context, *StorageDeleteArgs) (*StorageDeleteReply, error)
 }
 
+// UnimplementedStorageServer can be embedded to have forward compatible implementations.
+type UnimplementedStorageServer struct {
+}
+
+func (*UnimplementedStorageServer) List(ctx context.Context, req *StorageListArgs) (*StorageListReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method List not implemented")
+}
+func (*UnimplementedStorageServer) Get(ctx context.Context, req *StorageGetArgs) (*StorageGetReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Get not implemented")
+}
+func (*UnimplementedStorageServer) Put(ctx context.Context, req *StoragePutArgs) (*StoragePutReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Put not implemented")
+}
+func (*UnimplementedStorageServer) Delete(ctx context.Context, req *StorageDeleteArgs) (*StorageDeleteReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Delete not implemented")
+}
+
 func RegisterStorageServer(s *grpc.Server, srv StorageServer) {
 	s.RegisterService(&_Storage_serviceDesc, srv)
 }
@@ -3555,6 +3600,44 @@ type SystemViewServer interface {
 	PluginEnv(context.Context, *Empty) (*PluginEnvReply, error)
 }
 
+// UnimplementedSystemViewServer can be embedded to have forward compatible implementations.
+type UnimplementedSystemViewServer struct {
+}
+
+func (*UnimplementedSystemViewServer) DefaultLeaseTTL(ctx context.Context, req *Empty) (*TTLReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method DefaultLeaseTTL not implemented")
+}
+func (*UnimplementedSystemViewServer) MaxLeaseTTL(ctx context.Context, req *Empty) (*TTLReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method MaxLeaseTTL not implemented")
+}
+func (*UnimplementedSystemViewServer) SudoPrivilege(ctx context.Context, req *SudoPrivilegeArgs) (*SudoPrivilegeReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method SudoPrivilege not implemented")
+}
+func (*UnimplementedSystemViewServer) Tainted(ctx context.Context, req *Empty) (*TaintedReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Tainted not implemented")
+}
+func (*UnimplementedSystemViewServer) CachingDisabled(ctx context.Context, req *Empty) (*CachingDisabledReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method CachingDisabled not implemented")
+}
+func (*UnimplementedSystemViewServer) ReplicationState(ctx context.Context, req *Empty) (*ReplicationStateReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method ReplicationState not implemented")
+}
+func (*UnimplementedSystemViewServer) ResponseWrapData(ctx context.Context, req *ResponseWrapDataArgs) (*ResponseWrapDataReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method ResponseWrapData not implemented")
+}
+func (*UnimplementedSystemViewServer) MlockEnabled(ctx context.Context, req *Empty) (*MlockEnabledReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method MlockEnabled not implemented")
+}
+func (*UnimplementedSystemViewServer) LocalMount(ctx context.Context, req *Empty) (*LocalMountReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method LocalMount not implemented")
+}
+func (*UnimplementedSystemViewServer) EntityInfo(ctx context.Context, req *EntityInfoArgs) (*EntityInfoReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method EntityInfo not implemented")
+}
+func (*UnimplementedSystemViewServer) PluginEnv(ctx context.Context, req *Empty) (*PluginEnvReply, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method PluginEnv not implemented")
+}
+
 func RegisterSystemViewServer(s *grpc.Server, srv SystemViewServer) {
 	s.RegisterService(&_SystemView_serviceDesc, srv)
 }