Combined Database Backend: Static Accounts (#6834)

* Add priority queue to sdk

* fix issue of storing pointers and now copy

* update to use copy structure

* Remove file, put Item struct def. into other file

* add link

* clean up docs

* refactor internal data structure to hide heap method implementations. Other cleanup after feedback

* rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods

* updates after feedback

* refactoring/renaming

* guard against pushing a nil item

* minor updates after feedback

* Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC

* Initial Combined database backend implementation of static accounts and automatic rotation

* vendor updates

* initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation

* add lock and setup of rotation queue

* vendor the queue

* rebase on new method signature of queue

* remove mongo tests for now

* update default role sql

* gofmt after rebase

* cleanup after rebasing to remove checks for ErrNotFound error

* rebase cdcr-priority-queue

* vendor dependencies with 'go mod vendor'

* website database docs for Static Role support

* document the rotate-role API endpoint

* postgres specific static role docs

* use constants for paths

* updates from review

* remove dead code

* combine and clarify error message for older plugins

* Update builtin/logical/database/backend.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* cleanups from feedback

* code and comment cleanups

* move db.RLock higher to protect db.GenerateCredentials call

* Return output with WALID if we failed to delete the WAL

* Update builtin/logical/database/path_creds_create.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* updates after running 'make fmt'

* update after running 'make proto'

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* update comment and remove and rearrange some dead code

* Update website/source/api/secret/databases/index.html.md

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* cleanups after review

* Update sdk/database/dbplugin/grpc_transport.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* code cleanup after feedback

* remove PasswordLastSet; it's not used

* document GenerateCredentials and SetCredentials

* Update builtin/logical/database/path_rotate_credentials.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* wrap pop and popbykey in backend methods to protect against nil cred rotation queue

* use strings.HasPrefix instead of direct equality check for path

* Forgot to commit this

* updates after feedback

* re-purpose an outdated test to now check that static and dynamic roles cannot share a name

* check for unique name across dynamic and static roles

* refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials

* remove commented out code

* refactor to have loadstaticwals filter out wals for roles that no longer exist

* return error if nil input given

* add nil check for input into setStaticAccount

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* add constant for queue tick time in seconds, used for comparrison in updates

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* code cleanup after review

* remove misplaced code comment

* remove commented out code

* create a queue in the Factory method, even if it's never used

* update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type

* document new method

* move rotation things into a specific file

* rename test file and consolidate some static account tests

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* update code comments, method names, and move more methods into rotation.go

* update comments to be capitalized

* remove the item from the queue before we try to destroy it

* findStaticWAL returns an error

* use lowercase keys when encoding WAL entries

* small cleanups

* remove vestigial static account check

* remove redundant DeleteWAL call in populate queue

* if we error on loading role, push back to queue with 10 second backoff

* poll in initqueue to make sure the backend is setup and can write/delete data

* add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false

* add code comments on read-only loop

* code comment updates

* re-push if error returned from find static wal

* add locksutil and acquire locks when pop'ing from the queue

* grab exclusive locks for updating static roles

* Add SetCredentials and GenerateCredentials stubs to mockPlugin

* add a switch in initQueue to listen for cancelation

* remove guard on zero time, it should have no affect

* create a new context in Factory to pass on and use for closing the backend queue

* restore master copy of vendor dir

plugins/database/cassandra/cassandra.go

@@ -239,3 +239,12 @@ func (c *Cassandra) RotateRootCredentials(ctx context.Context, statements []stri
 	c.rawConfig["password"] = password
 	return c.rawConfig, nil
 }
+
+// GenerateCredentials returns a generated password
+func (c *Cassandra) GenerateCredentials(ctx context.Context) (string, error) {
+	password, err := c.GeneratePassword()
+	if err != nil {
+		return "", err
+	}
+	return password, nil
+}

plugins/database/cassandra/connection_producer.go

@@ -12,6 +12,7 @@ import (
 
 	"github.com/gocql/gocql"
 	"github.com/hashicorp/errwrap"
+	"github.com/hashicorp/vault/sdk/database/dbplugin"
 	"github.com/hashicorp/vault/sdk/database/helper/connutil"
 	"github.com/hashicorp/vault/sdk/database/helper/dbutil"
 	"github.com/hashicorp/vault/sdk/helper/certutil"
@@ -278,3 +279,13 @@ func (c *cassandraConnectionProducer) secretValues() map[string]interface{} {
 		c.PemJSON:   "[pem_json]",
 	}
 }
+
+// SetCredentials uses provided information to set/create a user in the
+// database. Unlike CreateUser, this method requires a username be provided and
+// uses the name given, instead of generating a name. This is used for creating
+// and setting the password of static accounts, as well as rolling back
+// passwords in the database in the event an updated database fails to save in
+// Vault's storage.
+func (c *cassandraConnectionProducer) SetCredentials(ctx context.Context, statements dbplugin.Statements, staticUser dbplugin.StaticUserConfig) (username, password string, err error) {
+	return "", "", dbutil.Unimplemented()
+}

plugins/database/hana/hana.go

@@ -293,3 +293,12 @@ func (h *HANA) revokeUserDefault(ctx context.Context, username string) error {
 func (h *HANA) RotateRootCredentials(ctx context.Context, statements []string) (map[string]interface{}, error) {
 	return nil, errors.New("root credentaion rotation is not currently implemented in this database secrets engine")
 }
+
+// GenerateCredentials returns a generated password
+func (h *HANA) GenerateCredentials(ctx context.Context) (string, error) {
+	password, err := h.GeneratePassword()
+	if err != nil {
+		return "", err
+	}
+	return password, nil
+}

plugins/database/influxdb/connection_producer.go

@@ -8,7 +8,9 @@ import (
 	"time"
 
 	"github.com/hashicorp/errwrap"
+	"github.com/hashicorp/vault/sdk/database/dbplugin"
 	"github.com/hashicorp/vault/sdk/database/helper/connutil"
+	"github.com/hashicorp/vault/sdk/database/helper/dbutil"
 	"github.com/hashicorp/vault/sdk/helper/certutil"
 	"github.com/hashicorp/vault/sdk/helper/parseutil"
 	"github.com/hashicorp/vault/sdk/helper/tlsutil"
@@ -261,3 +263,13 @@ func isUserAdmin(cli influx.Client, user string) (bool, error) {
 	}
 	return false, fmt.Errorf("the provided username is not a valid user in the influxdb")
 }
+
+// SetCredentials uses provided information to set/create a user in the
+// database. Unlike CreateUser, this method requires a username be provided and
+// uses the name given, instead of generating a name. This is used for creating
+// and setting the password of static accounts, as well as rolling back
+// passwords in the database in the event an updated database fails to save in
+// Vault's storage.
+func (i *influxdbConnectionProducer) SetCredentials(ctx context.Context, statements dbplugin.Statements, staticUser dbplugin.StaticUserConfig) (username, password string, err error) {
+	return "", "", dbutil.Unimplemented()
+}

plugins/database/influxdb/influxdb.go

@@ -242,3 +242,12 @@ func (i *Influxdb) RotateRootCredentials(ctx context.Context, statements []strin
 	i.rawConfig["password"] = password
 	return i.rawConfig, nil
 }
+
+// GenerateCredentials returns a generated password
+func (i *Influxdb) GenerateCredentials(ctx context.Context) (string, error) {
+	password, err := i.GeneratePassword()
+	if err != nil {
+		return "", err
+	}
+	return password, nil
+}

plugins/database/mongodb/connection_producer.go

@@ -15,6 +15,7 @@ import (
 	"time"
 
 	"github.com/hashicorp/errwrap"
+	"github.com/hashicorp/vault/sdk/database/dbplugin"
 	"github.com/hashicorp/vault/sdk/database/helper/connutil"
 	"github.com/hashicorp/vault/sdk/database/helper/dbutil"
 	"github.com/mitchellh/mapstructure"
@@ -153,6 +154,16 @@ func (c *mongoDBConnectionProducer) Close() error {
 	return nil
 }
 
+// SetCredentials uses provided information to set/create a user in the
+// database. Unlike CreateUser, this method requires a username be provided and
+// uses the name given, instead of generating a name. This is used for creating
+// and setting the password of static accounts, as well as rolling back
+// passwords in the database in the event an updated database fails to save in
+// Vault's storage.
+func (c *mongoDBConnectionProducer) SetCredentials(ctx context.Context, statements dbplugin.Statements, staticUser dbplugin.StaticUserConfig) (username, password string, err error) {
+	return "", "", dbutil.Unimplemented()
+}
+
 func parseMongoURL(rawURL string) (*mgo.DialInfo, error) {
 	url, err := url.Parse(rawURL)
 	if err != nil {

plugins/database/mongodb/mongodb.go

@@ -224,3 +224,12 @@ func (m *MongoDB) RevokeUser(ctx context.Context, statements dbplugin.Statements
 func (m *MongoDB) RotateRootCredentials(ctx context.Context, statements []string) (map[string]interface{}, error) {
 	return nil, errors.New("root credential rotation is not currently implemented in this database secrets engine")
 }
+
+// GenerateCredentials returns a generated password
+func (m *MongoDB) GenerateCredentials(ctx context.Context) (string, error) {
+	password, err := m.GeneratePassword()
+	if err != nil {
+		return "", err
+	}
+	return password, nil
+}

plugins/database/mssql/mssql.go

@@ -381,3 +381,12 @@ END
 const rotateRootCredentialsSQL = `
 ALTER LOGIN [{{username}}] WITH PASSWORD = '{{password}}' 
 `
+
+// GenerateCredentials returns a generated password
+func (m *MSSQL) GenerateCredentials(ctx context.Context) (string, error) {
+	password, err := m.GeneratePassword()
+	if err != nil {
+		return "", err
+	}
+	return password, nil
+}

plugins/database/mysql/mysql.go

@@ -315,3 +315,12 @@ func (m *MySQL) RotateRootCredentials(ctx context.Context, statements []string)
 	m.RawConfig["password"] = password
 	return m.RawConfig, nil
 }
+
+// GenerateCredentials returns a generated password
+func (m *MySQL) GenerateCredentials(ctx context.Context) (string, error) {
+	password, err := m.GeneratePassword()
+	if err != nil {
+		return "", err
+	}
+	return password, nil
+}

plugins/database/postgresql/postgresql.go

@@ -26,6 +26,10 @@ ALTER ROLE "{{name}}" VALID UNTIL '{{expiration}}';
 `
 	defaultPostgresRotateRootCredentialsSQL = `
 ALTER ROLE "{{username}}" WITH PASSWORD '{{password}}';
+`
+
+	defaultPostgresRotateCredentialsSQL = `
+ALTER ROLE "{{name}}" WITH PASSWORD '{{password}}';
 `
 )
 
@@ -88,6 +92,86 @@ func (p *PostgreSQL) getConnection(ctx context.Context) (*sql.DB, error) {
 	return db.(*sql.DB), nil
 }
 
+// SetCredentials uses provided information to set/create a user in the
+// database. Unlike CreateUser, this method requires a username be provided and
+// uses the name given, instead of generating a name. This is used for creating
+// and setting the password of static accounts, as well as rolling back
+// passwords in the database in the event an updated database fails to save in
+// Vault's storage.
+func (p *PostgreSQL) SetCredentials(ctx context.Context, statements dbplugin.Statements, staticUser dbplugin.StaticUserConfig) (username, password string, err error) {
+	if len(statements.Creation) == 0 {
+		return "", "", errors.New("empty creation statements")
+	}
+
+	username = staticUser.Username
+	password = staticUser.Password
+	if username == "" || password == "" {
+		return "", "", errors.New("must provide both username and password")
+	}
+
+	// Grab the lock
+	p.Lock()
+	defer p.Unlock()
+
+	// Get the connection
+	db, err := p.getConnection(ctx)
+	if err != nil {
+		return "", "", err
+	}
+
+	// Check if the role exists
+	var exists bool
+	err = db.QueryRowContext(ctx, "SELECT exists (SELECT rolname FROM pg_roles WHERE rolname=$1);", username).Scan(&exists)
+	if err != nil && err != sql.ErrNoRows {
+		return "", "", err
+	}
+
+	// Default to using Creation statements, which are required by the Vault
+	// backend. If the user exists, use the rotation statements, using the default
+	// ones if there are none provided
+	stmts := statements.Creation
+	if exists {
+		stmts = statements.Rotation
+		if len(stmts) == 0 {
+			stmts = []string{defaultPostgresRotateCredentialsSQL}
+		}
+	}
+
+	// Start a transaction
+	tx, err := db.BeginTx(ctx, nil)
+	if err != nil {
+		return "", "", err
+	}
+	defer func() {
+		_ = tx.Rollback()
+	}()
+
+	// Execute each query
+	for _, stmt := range stmts {
+		for _, query := range strutil.ParseArbitraryStringSlice(stmt, ";") {
+			query = strings.TrimSpace(query)
+			if len(query) == 0 {
+				continue
+			}
+
+			m := map[string]string{
+				"name":     staticUser.Username,
+				"password": password,
+			}
+			if err := dbtxn.ExecuteTxQuery(ctx, tx, m, query); err != nil {
+				return "", "", err
+			}
+		}
+	}
+
+	// Commit the transaction
+	if err := tx.Commit(); err != nil {
+		return "", "", err
+	}
+
+	return username, password, nil
+}
+
 func (p *PostgreSQL) CreateUser(ctx context.Context, statements dbplugin.Statements, usernameConfig dbplugin.UsernameConfig, expiration time.Time) (username string, password string, err error) {
 	statements = dbutil.StatementCompatibilityHelper(statements)
 
@@ -129,7 +213,6 @@ func (p *PostgreSQL) CreateUser(ctx context.Context, statements dbplugin.Stateme
 	defer func() {
 		tx.Rollback()
 	}()
-	// Return the secret
 
 	// Execute each query
 	for _, stmt := range statements.Creation {
@@ -267,7 +350,7 @@ func (p *PostgreSQL) defaultRevokeUser(ctx context.Context, username string) err
 		return err
 	}
 
-	if exists == false {
+	if !exists {
 		return nil
 	}
 
@@ -424,3 +507,12 @@ func (p *PostgreSQL) RotateRootCredentials(ctx context.Context, statements []str
 	p.RawConfig["password"] = password
 	return p.RawConfig, nil
 }
+
+// GenerateCredentials returns a generated password
+func (p *PostgreSQL) GenerateCredentials(ctx context.Context) (string, error) {
+	password, err := p.GeneratePassword()
+	if err != nil {
+		return "", err
+	}
+	return password, nil
+}

plugins/database/postgresql/postgresql_test.go

@@ -317,6 +317,84 @@ func TestPostgreSQL_RevokeUser(t *testing.T) {
 	}
 }
 
+func TestPostgresSQL_SetCredentials(t *testing.T) {
+	cleanup, connURL := preparePostgresTestContainer(t)
+	defer cleanup()
+
+	connectionDetails := map[string]interface{}{
+		"connection_url": connURL,
+	}
+
+	db := new()
+	_, err := db.Init(context.Background(), connectionDetails, true)
+	if err != nil {
+		t.Fatalf("err: %s", err)
+	}
+
+	password, err := db.GenerateCredentials(context.Background())
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	usernameConfig := dbplugin.StaticUserConfig{
+		Username: "test",
+		Password: password,
+	}
+
+	// Test with no configured Creation Statement
+	username, password, err := db.SetCredentials(context.Background(), dbplugin.Statements{}, usernameConfig)
+	if err == nil {
+		t.Fatalf("err: %s", err)
+	}
+
+	statements := dbplugin.Statements{
+		Creation: []string{testPostgresStaticRole},
+	}
+	// User should not exist, make sure we can create
+	username, password, err = db.SetCredentials(context.Background(), statements, usernameConfig)
+	if err != nil {
+		t.Fatalf("err: %s", err)
+	}
+
+	if err := testCredsExist(t, connURL, username, password); err != nil {
+		t.Fatalf("Could not connect with new credentials: %s", err)
+	}
+
+	// call SetCredentials again, the user will already exist, password will
+	// change. Without rotation statements, this should use the defaults
+	newPassword, _ := db.GenerateCredentials(context.Background())
+	usernameConfig.Password = newPassword
+	username, password, err = db.SetCredentials(context.Background(), statements, usernameConfig)
+	if err != nil {
+		t.Fatalf("err: %s", err)
+	}
+
+	if password != newPassword {
+		t.Fatal("passwords should have changed")
+	}
+
+	if err := testCredsExist(t, connURL, username, password); err != nil {
+		t.Fatalf("Could not connect with new credentials: %s", err)
+	}
+
+	// generate a new password and supply owr own rotation statements
+	newPassword2, _ := db.GenerateCredentials(context.Background())
+	usernameConfig.Password = newPassword2
+	statements.Rotation = []string{testPostgresStaticRoleRotate, testPostgresStaticRoleGrant}
+	username, password, err = db.SetCredentials(context.Background(), statements, usernameConfig)
+	if err != nil {
+		t.Fatalf("err: %s", err)
+	}
+
+	if password != newPassword2 {
+		t.Fatal("passwords should have changed")
+	}
+
+	if err := testCredsExist(t, connURL, username, password); err != nil {
+		t.Fatalf("Could not connect with new credentials: %s", err)
+	}
+}
+
 func testCredsExist(t testing.TB, connURL, username, password string) error {
 	t.Helper()
 	// Log in with the new creds
@@ -398,3 +476,18 @@ REVOKE USAGE ON SCHEMA public FROM "{{name}}";
 
 DROP ROLE IF EXISTS "{{name}}";
 `
+
+const testPostgresStaticRole = `
+CREATE ROLE "{{name}}" WITH
+  LOGIN
+  PASSWORD '{{password}}';
+GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO "{{name}}";
+`
+
+const testPostgresStaticRoleRotate = `
+ALTER ROLE "{{name}}" WITH PASSWORD '{{password}}';
+`
+
+const testPostgresStaticRoleGrant = `
+GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO "{{name}}";
+`