First draft of Seal HA documentation (#23252)

* wip

* Initial draft of Seal HA docs

* nav data

* Fix env var name

* title

* Note partially wrapped values and disabled seal participation

* Update website/data/docs-nav-data.json

Co-authored-by: Steven Clark <steven.clark@hashicorp.com>

* correct initial upgrade limitation

* Add note about shamir seals and migration

* fix nav json

* snapshot note

* availability note

* seal-backend-status

* Add a couple more clarifying statements

* header typo

* correct initial upgrade wording

* Update website/content/docs/configuration/seal/seal-ha.mdx

Co-authored-by: Steven Clark <steven.clark@hashicorp.com>

* Update website/content/docs/concepts/seal.mdx

Co-authored-by: Steven Clark <steven.clark@hashicorp.com>

---------

Co-authored-by: Steven Clark <steven.clark@hashicorp.com>

website/content/api-docs/system/seal-backend-status.mdx

@@ -0,0 +1,47 @@
+---
+layout: api
+page_title: /sys/seal-backend-status - HTTP API
+description: The `/sys/seal-backend-status` endpoint is used to check the status of each seal backend.
+---
+
+# `/sys/seal-backend-status`
+
+The `/sys/seal-backend-status` endpoint is used to display the status of
+each backing seal, namely whether each backend is healthy.
+
+## Seal Backend Status
+
+This endpoint returns the seal status of the Vault. This is an unauthenticated
+endpoint.
+
+| Method | Path                       |
+| :----- | :------------------------- |
+| `GET`  | `/sys/seal-backend-status` |
+
+### Sample request
+
+```shell-session
+$ curl \
+    http://127.0.0.1:8200/v1/sys/seal-backend-status
+```
+
+### Sample response
+
+
+```json
+{ 
+  "healthy": false,
+  "unhealthy_since": "2023-09-22T08:34:11Z",
+  "backends": [
+     {
+      "name": "hsm",
+      "healthy": true
+     },
+     {
+      "name": "transit",
+      "healthy": false,
+      "unhealthy_since": "2023-09-22T08:34:11Z"
+     }
+  ]
+}
+```

website/content/docs/concepts/seal.mdx

@@ -315,3 +315,76 @@ When the quorum of nodes are back up, Raft will elect a leader and the leader
 node that will perform the migration. The migrated information will be replicated to
 all other cluster peers and when the peers eventually become the leader,
 migration will not happen again on the peer nodes.
+
+## Seal High Availability (Enterprise, Beta)
+
+
+-> **Warning:** This feature is available as a Beta for evaluation and should not 
+be used in production deployments of Vault.
+
+Seal High Availability (Seal HA) allows the configuration of more than one auto 
+seal mechanism such that Vault can tolerate the temporary loss of a seal service 
+or device for a time.  With Seal HA Vault can also start up and unseal if one of the 
+configured seals is still available (though Vault will remain in a degraded mode in 
+this case). While seals are unavailable, seal wrapping and entropy augmentation can 
+still occur using the remaining seals, and values produced while a seal is down will
+be re-wrapped with all the seals when all seals become healthy again.
+
+When an operator configures an additional seal or removes a seal (one at a time)
+and restarts Vault, Vault will automatically detect that it needs to re-wrap
+CSPs and seal wrapped values, and will start the process.  Seal re-wrapping can
+be monitored via the logs or via the `sys/seal-status` endpoint.  While a 
+re-wrap is in progress (or could not complete successfully), changes to the
+seal configuration are not allowed.
+
+In additional to high availability, Seal HA can be used to migrate between two 
+auto seals in a simplified manner.  To migrate in this way:
+
+1. Edit the Vault configuration, and add the new, target seal configuration.
+1. Restart Vault, allowing re-wrapping to complete.
+1. Edit the Vault configuration, removing the old seal configuration.
+1. Restart Vault, again allowing re-wrapping to complete.
+
+By default, Seal HA is unavailable.  To try the beta, one must set an environment
+variable which enables it:
+
+`VAULT_ENABLE_SEAL_HA_BETA=true`
+
+### Use and Configuration
+
+Refer to the [configuration](/vault/docs/configuration/seal/seal-ha) section
+for details on configuring Seal HA.
+
+### Seal Re-Wrapping
+
+Whenever seal configuration changes, Vault must re-wrap all CSPs and seal
+wrapped values, to ensure each value has an entry encrypted by all configured
+seals.  Vault detects these configuration changes automatically, and triggers
+a re-wrap.  Re-wraps can take some time, depending on the number of
+seal wrapped values.  While re-wrapping is in progress, no configuration changes
+to the seals can be made.
+
+Progress of the re-wrap can be monitored using 
+the [`sys/sealwrap/rewrap`](/vault/api-docs/system/sealwrap-rewrap) endpoint.
+
+### Limitations and Known Issues
+
+As Seal HA is in beta, there are certain restrictions that may not be present in 
+the generally available version of the feature:
+
+* Changing seal configurations requires a restart of Vault.
+* Vault must be configured for a single seal at the time of initialization.
+Extra seals can then be added.
+* Upon upgrading to Vault 1.15 or higher, Vault must start with a
+conventional single seal and the beta flag enabled.  Afterward an additional seal 
+may be added.
+* Seals must be added or removed one at a time.
+* Only auto seals can be used in HA configurations.  Shamir and auto cannot
+be mixed.
+* A maximum of three seals can be configured.
+* As seal wrapped values must be wrapped by all configured seals, it is possible
+that large values may fail to persist as the size of the entry is multiplied by 
+the number of seals causing it to exceed the storage entry size limit.   An example
+would be storing a large document in KVv2 with seal wrapping enabled.
+
+

website/content/docs/configuration/entropy-augmentation.mdx

@@ -15,6 +15,9 @@ include: [PKCS11 seal](/vault/docs/configuration/seal/pkcs11), [AWS KMS](/vault/
 Vault Enterprises's external entropy support is activated by the presence of an `entropy "seal"`
 block in Vault's configuration file.
 
+~> **Note**: If using the Seal High Availability Beta, entropy will be retrieved
+from seals in priority order, using bytes from the first available and online seal.
+
 ## Requirements
 
 A valid Vault Enterprise license is required for Entropy Augmentation.

website/content/docs/configuration/seal/index.mdx

@@ -28,7 +28,7 @@ Seal configuration can be done through the Vault configuration file using the
 `seal` stanza:
 
 ```hcl
-seal [NAME] {
+seal [TYPE] {
   # ...
 }
 ```
@@ -44,4 +44,4 @@ seal "pkcs11" {
 For configuration options which also read an environment variable, the
 environment variable will take precedence over values in the configuration file.
 
-[sealwrap]: /vault/docs/enterprise/sealwrap
+[sealwrap]: /vault/docs/enterprise/sealwrap

website/content/docs/configuration/seal/seal-ha.mdx

@@ -0,0 +1,104 @@
+---
+layout: docs
+page_title: Seal High Availability (Beta) - Seals - Configuration
+description: |-
+  How to configure multiple Seals for high availability.
+---
+
+# Seal High Availability (Beta)
+
+<Note title="Seal HA functionality requires Vault Enterprise">
+
+  Seal High Availability requires Vault Enterprise.
+
+</Note>
+
+-> **Warning:** This feature is available as a Beta for evaluation and should not 
+be used in production deployments of Vault.
+
+[Seal High Availability](/vault/docs/concepts/seal#seal-high-availability-enterprise-beta) 
+is the ability to configure more than one seal in order to have resilience against
+outage of a seal service or mechanism.   
+
+Using Seal HA involves configuring extra seals in Vault's server configuration file
+and restarting Vault, after having enabled the Seal HA beta feature by setting
+the environment variable:
+
+`VAULT_ENABLE_SEAL_HA_BETA=true`
+
+Before using Seal HA, one must have upgraded to Vault 1.15 or higher and
+started with the environment variable and a single seal.
+
+## Adding and Removing Seals
+
+Seal HA adds two new fields to the seal configuration stanza, `name`, and
+`priority`:
+
+```hcl
+seal [TYPE] {
+  name = "seal-name"
+  priority = "1"
+  # ...
+}
+```
+
+Name is optional, and if not specified is set to the type of the seal.  Names
+must be unique.  If using two seals of the same type name must be specified.
+Internally, name is used to disambiguate seal wrapped values in some cases,
+so renaming seals should be avoided if possible.
+
+Priority is mandatory if more than one seal is specified.  Priority tells Vault
+the order in which to try seals during unseal (least priority first), 
+in the case more than one seal can unwrap a seal wrapped value, the order
+in which to attempt decryption, and which order to attempt to source entropy
+for entropy augmentation.  This can be useful if your seals have different
+performance or cost characteristics.
+
+All configured, healthy seals are used to seal wrap values.  This means that
+for every write of a seal wrapped value or CSP, an encryption is requested
+from every configured seal, and the results are stored in the storage entry.
+When seals are unhealthy, Vault keeps track of values that could not be fully 
+wrapped and will re-wrap them once seals become healthy again.  Disabled seals 
+can still be used for decryption of wrapped values, but will be avoided when
+encrypting values.
+
+When reading a CSP or seal wrapped value, Vault will try to decrypt with the
+highest priority available seal, and then try other seals on failure.
+
+To add an additional seal, simply add another seal stanza, specifying priority
+and optionally name, and restart Vault.
+
+To remove a seal, remove the corresponding seal stanza and restart.  There must
+be at least one seal remaining.
+
+It is highly recommended to take a snapshot of your Vault storage before applying 
+any seal configuration change.
+
+Once Vault unseals with the new seal configuration, it will be available to process
+traffic even as re-wrapping proceeds.
+
+### Safety checks
+
+Vault will reject seal configuration changes in the following circumstances,
+as a safety mechanism:
+
+* The old seal configuration and new seal configuration do not share one seal
+in common.  This is necessary as there would be no seal capable of decrypting
+CSPs or seal wrapped values previously written.
+
+* Seal re-wrapping is in progress.  Vault must be in a clean, fully wrapped state
+on the previous configuration before attempting a configuration change.
+
+* More than one seal is being added or removed at a time.  
+
+### Interaction with Shamir Seals
+
+Seal HA is only supported with auto seal mechanisms.  To use Seal HA when
+running a Shamir seal, first use the traditional 
+[seal migration](/vault/docs/concepts/seal#seal-migration) mechanism to migrate to
+an auto seal of your choice.  Afterwards you may follow the above
+instructions to add a second auto seal.
+
+Correspondingly, to migrate back to a shamir seal, first use the above
+instructions to move to a single auto seal, and use the traditional
+migration method to migrate back to a Shamir seal.

website/content/docs/configuration/seal/transit.mdx

@@ -67,6 +67,10 @@ These parameters apply to the `seal` stanza in the Vault configuration file:
   decryption. This may also be supplied using the `VAULT_TRANSIT_SEAL_KEY_NAME`
   environment variable.
 
+- `key_id_prefix` `(string: "")`: An optional string to add to the key id
+  of values wrapped by this transit seal.  This can help disambiguate between
+  two transit seals.
+
 - `mount_path` `(string: <required>)`: The mount path to the transit secret engine.
   This may also be supplied using the `VAULT_TRANSIT_SEAL_MOUNT_PATH` environment
   variable.

website/data/api-docs-nav-data.json

@@ -673,6 +673,10 @@
         "title": "<code>/sys/seal-status</code>",
         "path": "system/seal-status"
       },
+      {
+        "title": "<code>/sys/seal-backend-status</code>",
+        "path": "system/seal-backend-status"
+      },
       {
         "title": "<code>/sys/sealwrap/rewrap</code>",
         "path": "system/sealwrap-rewrap"

website/data/docs-nav-data.json

@@ -308,6 +308,14 @@
             "title": "Overview",
             "path": "configuration/seal"
           },
+          { "title": "High Availability (Beta)",
+           "badge": {
+              "text": "ENT",
+              "type": "outlined",
+              "color": "neutral"
+            },
+            "path": "configuration/seal/seal-ha"
+          },
           {
             "title": "AliCloud KMS",
             "path": "configuration/seal/alicloudkms"