Forward websocket event subscription requests (#22446)

For now, only the leader of a cluster can handle subscription requests,
so we forward the connection request otherwise.

We forward using a 307 temporary redirect (the fallback way).
Forwarding a request over gRPC currently only supports a single request
and response, but a websocket connection is long-lived with potentially
many messages back and forth.

We modified the `vault events subscribe` command to honor those
redirects. `wscat` supports them with the `-L` flag.

In the future, we may add a gRPC method to handle forwarding WebSocket
requests, but doing so adds quite a bit of complexity (even over
normal request forwarding) due to the intricate nature of the `http` /
`vault.Core` interactions required. (I initially went down this path.)

I added tests for the forwarding header, and also tested manually.
(Testing with `-dev-three-node` is a little clumsy since it does not
properly support experiments, for some reason.)

Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>

http/handler.go

@@ -84,6 +84,7 @@ var (
 	// the always forward list
 	perfStandbyAlwaysForwardPaths = pathmanager.New()
 	alwaysRedirectPaths           = pathmanager.New()
+	websocketPaths                = pathmanager.New()
 
 	injectDataIntoTopRoutes = []string{
 		"/v1/sys/audit",
@@ -109,7 +110,9 @@ var (
 		"/v1/sys/rotate",
 		"/v1/sys/wrapping/wrap",
 	}
-
+	websocketRawPaths = []string{
+		"/v1/sys/events/subscribe",
+	}
 	oidcProtectedPathRegex = regexp.MustCompile(`^identity/oidc/provider/\w(([\w-.]+)?\w)?/userinfo$`)
 )
 
@@ -119,6 +122,10 @@ func init() {
 		"sys/storage/raft/snapshot-force",
 		"!sys/storage/raft/snapshot-auto/config",
 	})
+	websocketPaths.AddPaths(websocketRawPaths)
+	for _, path := range websocketRawPaths {
+		alwaysRedirectPaths.AddPaths([]string{strings.TrimPrefix(path, "/v1/")})
+	}
 }
 
 type HandlerAnchor struct{}
@@ -1017,6 +1024,15 @@ func respondStandby(core *vault.Core, w http.ResponseWriter, reqURL *url.URL) {
 		RawQuery: reqURL.RawQuery,
 	}
 
+	// WebSockets schemas are ws or wss
+	if websocketPaths.HasPath(reqURL.Path) {
+		if finalURL.Scheme == "http" {
+			finalURL.Scheme = "ws"
+		} else {
+			finalURL.Scheme = "wss"
+		}
+	}
+
 	// Ensure there is a scheme, default to https
 	if finalURL.Scheme == "" {
 		finalURL.Scheme = "https"