Update considerations for multiple issuers (#15442)

Also adds auditing section about suggested un-HMAC'd request/response
parameters.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

website/content/docs/secrets/pki/considerations.mdx

@@ -32,25 +32,46 @@ root CA (which may be a second mount of the `pki` secrets engine).
 ### Managed Keys
 
 Since 1.10, Vault Enterprise can access private key material in a
-[_managed key_](../enterprise/managed-keys).  In this case Vault never sees the
+[_managed key_](../enterprise/managed-keys).  In this case, Vault never sees the
 private key, and the external KMS or HSM performs certificate signing operations.
 Managed keys are configured by selecting the `kms` type when generating a root
 or intermediate.
 
 ## One CA Certificate, One Secrets Engine
 
-In order to vastly simplify both the configuration and codebase of the PKI
-secrets engine, only one CA certificate is allowed per secrets engine. If you
-want to issue certificates from multiple CAs, mount the PKI secrets engine at
-multiple mount points with separate CA certificates in each.
-
-This also provides a convenient method of switching to a new CA certificate
-while keeping CRLs valid from the old CA certificate; simply mount a new secrets
-engine and issue from there.
+Since Vault 1.11.0, the PKI Secrets Engine supports multiple issuers in a single
+mount. However, in order to simplify the configuration, it is _strongly_
+recommended that operators limit a mount to a single issuer. If you want to issue
+certificates from multiple disparate CAs, mount the PKI secrets engine at multiple
+mount points with separate CA certificates in each.
 
 A common pattern is to have one mount act as your root CA and to use this CA
 only to sign intermediate CA CSRs from other PKI secrets engines.
 
+To keep old CAs active, there's two approaches to achieving rotation:
+
+ 1. Use multiple secrets engines. This allows a fresh start, preserving the
+    old issuer and CRL. Vault ACL policy can be updated to deny new issuance
+    under the old mount point and roles can be re-evaluated before being
+    imported into the new mount point.
+ 2. Use multiple issuers in the same mount point. The usage of the old issuer
+    can be restricted to CRL signing, and existing roles and ACL policy can be
+    kept as-is. This allows cross-signing within the same mount, and consumers
+    of the mount won't have to update their configuration. Once the transitional
+    period for this rotation has completed and all past issued certificate have
+    expired, it is encouraged to fully remove the old issuer and any unnecessary
+    cross-signed issuers from the mount point.
+
+Another suggested use case for multiple issuers in the same mount is splitting
+issuance by TTL lifetime. For short-lived certificates, an intermediate
+stored in Vault will often out-perform a HSM-backed intermediate. For
+longer-lived certificates, however, it is often important to have the
+intermediate key material secured throughout the lifetime of the end-entity
+certificate. This means that two intermediates in the same mount -- one backed
+by the HSM and one backed by Vault -- can satisfy both use cases. Operators
+can make roles setting maximum TTLs for each issuer and consumers of the
+mount can decide which to use.
+
 ## Keep certificate lifetimes short, for CRL's sake
 
 This secrets engine aligns with Vault's philosophy of short-lived secrets. As
@@ -64,7 +85,9 @@ be used; alternately a root token can be used to revoke the certificate using
 the certificate's serial number. Any revocation action will cause the CRL to be
 regenerated. When the CRL is regenerated, any expired certificates are removed
 from the CRL (and any revoked, expired certificate are removed from secrets
-engine storage).
+engine storage). This is an expensive operation! Due to the structure of the
+CRL standard, Vault must read **all** revoked certificates into memory in order
+to rebuild the CRL and clients must fetch the regenerated CRL.
 
 This secrets engine does not support multiple CRL endpoints with sliding date
 windows; often such mechanisms will have the transition point a few days apart,
@@ -75,9 +98,16 @@ your maximum comfortable CRL lifetime. Alternately, you can control CRL caching
 behavior on the client to ensure that checks happen more often.
 
 Often multiple endpoints are used in case a single CRL endpoint is down so that
-clients don't have to figure out what to do with a lack of response. Run Vault in HA mode, and the CRL endpoint should be available even if a particular node
+clients don't have to figure out what to do with a lack of response. Run Vault
+in HA mode, and the CRL endpoint should be available even if a particular node
 is down.
 
+~> Note: Since Vault 1.11.0, with multiple issuers in the same mount point,
+   different issuers may have different CRLs (depending on subject and key
+   material). This means that Vault may need to regenerate multiple CRLs.
+   This is again a rationale for keeping TTLs short and avoiding revocation
+   if possible.
+
 ## You must configure issuing/CRL/OCSP information _in advance_
 
 This secrets engine serves CRLs from a predictable location, but it is not
@@ -108,16 +138,84 @@ endpoint.
 
 Unlike many secrets engines which replicate their state to all clusters
 in multi-cluster architectures, the PKI secrets engine stores leaf certificates
-issued without `no_store` set to true local to the cluster that issued them.
+issued with `no_store` set to `false` local to the cluster that issued them.
 This allows for both primary and secondary clusters' nodes to issue
 certificates for greater scalability.  As a result, these certificates
-and any revocations are visible only on the issuing cluster.
+and any revocations are visible only on the issuing cluster. This additionally
+means each cluster has its own set of CRLs, distinct from other clusters. These
+CRLs should either be unified into a single CRL for distribution from a single
+URI, or server operators should know to fetch all CRLs from all clusters.
 
 ## Telemetry
 
 Beyond Vault's default telemetry around request processing, PKI exposes count and
- duration metrics for the issue, sign, sign-verbatim, and revoke calls. The
-metrics keys take the form `mount-path,operation,[failure]` with labels for namespace and role name.
+duration metrics for the issue, sign, sign-verbatim, and revoke calls. The
+metrics keys take the form `mount-path,operation,[failure]` with labels for
+namespace and role name.
+
+Note that these metrics are per-node and thus would need to be aggregated across
+nodes and clusters.
+
+## Auditing
+
+Because Vault HMACs audit string keys by default, it is necessary to tune
+PKI secrets mounts to get an accurate view of issuance that is occurring under
+this mount.
+
+Some suggested keys to un-HMAC for requests are as follows:
+
+ - `csr` - the requested CSR to sign,
+ - `certificate` - the requested self-signed certificate to re-sign or
+   when importing issuers,
+ - Various issuance-related overriding parameters, such as:
+   - `issuer_ref` - the issuer requested to sign this certificate,
+   - `common_name` - the requested common name,
+   - `alt_names` - alternative requested DNS-type SANs for this certificate,
+   - `other_sans` - other (non-DNS, non-Email, non-IP, non-URI) requested SANs for this certificate,
+   - `ip_sans` - requested IP-type SANs for this certificate,
+   - `uri_sans` - requested URI-type SANs for this certificate,
+   - `ttl` - requested expiration date of this certificate,
+   - `not_after` - requested expiration date of this certificate,
+   - `serial_number` - the subject's requested serial number,
+   - `key_type` - the requested key type,
+   - `private_key_format` - the requested key format which is also
+     used for the public certificate format as well,
+ - Various role- or issuer-related generation parameters, such as:
+   - `managed_key_name` - when creating an issuer, the requested managed
+     key name,
+   - `managed_key_id` -  when creating an issuer, the requested managed
+     key identifier,
+   - `ou` - the subject's organizational unit,
+   - `organization` - the subject's organization,
+   - `country` - the subject's country code,
+   - `locality` - the subject's locality,
+   - `province` - the subject's province,
+   - `street_address` - the subject's street address,
+   - `postal_code` - the subject's postal code,
+   - `permitted_dns_domains` - permitted DNS domains,
+   - `policy_identifiers` - the requested policy identifiers when creating a role, and
+   - `ext_key_usage_oids` - the extended key usage OIDs for the requested certificate.
+
+Some suggested keys to un-HMAC for responses are as follows:
+
+ - `certificate` - the certificate that was issued,
+ - `issuing_ca` - the certificate of the CA which issued the requested
+   certificate,
+ - `serial_number` - the serial number of the certificate that was issued,
+ - `error` - to show errors associated with the request, and
+ - `ca_chain` - optional due to noise; the full CA chain of the issuer of
+   the requested certificate.
+
+~> Note: These list of parameters to un-HMAC are provided as a suggestion and
+   may not be exhaustive.
+
+The following keys are suggested **NOT** to un-HMAC, due to their sensitive
+nature:
+
+ - `private_key` - this response parameter contains the private keys
+   generated by Vault during issuance, and
+ - `pem_bundle` this request parameter is only used on the issuer-import
+   paths and may contain sensitive private key material.
 
 ## Learn