Support rootless plugin containers (#24236)

* Pulls in github.com/go-secure-stdlib/plugincontainer@v0.3.0 which exposes a new `Config.Rootless` option to opt in to extra container configuration options that allow establishing communication with a non-root plugin within a rootless container runtime.
* Adds a new "rootless" option for plugin runtimes, so Vault needs to be explicitly told whether the container runtime on the machine is rootless or not. It defaults to false as rootless installs are not the default.
* Updates `run_config.go` to use the new option when the plugin runtime is rootless.
* Adds new `-rootless` flag to `vault plugin runtime register`, and `rootless` API option to the register API.
* Adds rootless Docker installation to CI to support tests for the new functionality.
* Minor test refactor to minimise the number of test Vault cores that need to be made for the external plugin container tests.
* Documentation for the new rootless configuration and the new (reduced) set of restrictions for plugin containers.
* As well as adding rootless support, we've decided to drop explicit support for podman for now, but there's no barrier other than support burden to adding it back again in future so it will depend on demand.
This commit is contained in:
Tom Proctor
2023-11-28 14:07:07 +00:00
committed by GitHub
parent 3726d8fb1d
commit 030bba4e68
22 changed files with 242 additions and 149 deletions

View File

@@ -286,14 +286,37 @@ jobs:
"runsc": {
"path": "/usr/local/bin/runsc",
"runtimeArgs": [
"--host-uds=all",
"--host-fifo=open"
"--host-uds=create"
]
}
}
}
EOF
sudo systemctl reload docker
- name: Install rootless Docker
# Enterprise repo runners do not allow sudo, so can't system packages there yet.
if: ${{ !inputs.enterprise }}
run: |
sudo apt-get install -y uidmap dbus-user-session
export FORCE_ROOTLESS_INSTALL=1
curl -fsSL https://get.docker.com/rootless | sh
mkdir -p ~/.config/docker/
tee ~/.config/docker/daemon.json <<EOF
{
"runtimes": {
"runsc": {
"path": "/usr/local/bin/runsc",
"runtimeArgs": [
"--host-uds=create",
"--ignore-cgroups"
]
}
}
}
EOF
systemctl --user restart docker
# Ensure the original rootful Docker install is still the default.
docker context use default
- id: run-go-tests
name: Run Go tests
timeout-minutes: ${{ fromJSON(env.TIMEOUT_IN_MINUTES) }}
@@ -356,8 +379,9 @@ jobs:
fi
fi
export VAULT_TEST_LOG_DIR=$(pwd)/test-results/go-test/logs-${{ matrix.id }}
mkdir -p $VAULT_TEST_LOG_DIR
VAULT_TEST_LOG_DIR="$(pwd)/test-results/go-test/logs-${{ matrix.id }}"
export VAULT_TEST_LOG_DIR
mkdir -p "$VAULT_TEST_LOG_DIR"
# shellcheck disable=SC2086 # can't quote RERUN_FAILS

View File

@@ -66,6 +66,7 @@ type RegisterPluginRuntimeInput struct {
CgroupParent string `json:"cgroup_parent,omitempty"`
CPU int64 `json:"cpu_nanos,omitempty"`
Memory int64 `json:"memory_bytes,omitempty"`
Rootless bool `json:"rootless,omitempty"`
}
// RegisterPluginRuntime registers the plugin with the given information.

3
changelog/24236.txt Normal file
View File

@@ -0,0 +1,3 @@
```release-note:improvement
plugins: Containerized plugins can be run fully rootless with the runsc runtime.
```

View File

@@ -26,6 +26,7 @@ type PluginRuntimeRegisterCommand struct {
flagCgroupParent string
flagCPUNanos int64
flagMemoryBytes int64
flagRootless bool
}
func (c *PluginRuntimeRegisterCommand) Synopsis() string {
@@ -88,6 +89,15 @@ func (c *PluginRuntimeRegisterCommand) Flags() *FlagSets {
Usage: "Memory limit to set per container in bytes. Defaults to no limit.",
})
f.BoolVar(&BoolVar{
Name: "rootless",
Target: &c.flagRootless,
Completion: complete.PredictAnything,
Usage: "Whether the container runtime is configured to run as a " +
"non-privileged (non-root) user. Required if the plugin container " +
"image is also configured to run as a non-root user.",
})
return set
}
@@ -151,6 +161,7 @@ func (c *PluginRuntimeRegisterCommand) Run(args []string) int {
CgroupParent: cgroupParent,
CPU: c.flagCPUNanos,
Memory: c.flagMemoryBytes,
Rootless: c.flagRootless,
}); err != nil {
c.UI.Error(fmt.Sprintf("Error registering plugin runtime %s: %s", runtimeName, err))
return 2

View File

@@ -130,7 +130,7 @@ func TestPluginRuntimeFlagParsing(t *testing.T) {
cgroupParent string
cpu int64
memory int64
args []string
rootless bool
expectedPayload string
}{
"minimal": {
@@ -145,7 +145,8 @@ func TestPluginRuntimeFlagParsing(t *testing.T) {
ociRuntime: "runtime",
cpu: 5678,
memory: 1234,
expectedPayload: `{"type":1,"cgroup_parent":"/cpulimit/","memory_bytes":1234,"cpu_nanos":5678,"oci_runtime":"runtime"}`,
rootless: true,
expectedPayload: `{"type":1,"cgroup_parent":"/cpulimit/","memory_bytes":1234,"cpu_nanos":5678,"oci_runtime":"runtime","rootless":true}`,
},
} {
tc := tc
@@ -167,6 +168,9 @@ func TestPluginRuntimeFlagParsing(t *testing.T) {
if tc.cpu != 0 {
args = append(args, fmt.Sprintf("-cpu_nanos=%d", tc.cpu))
}
if tc.rootless {
args = append(args, "-rootless=true")
}
if tc.runtimeType != api.PluginRuntimeTypeUnsupported {
args = append(args, "-type="+tc.runtimeType.String())

5
go.mod
View File

@@ -99,7 +99,7 @@ require (
github.com/hashicorp/go-memdb v1.3.4
github.com/hashicorp/go-msgpack v1.1.5
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/go-plugin v1.5.2
github.com/hashicorp/go-plugin v1.6.0
github.com/hashicorp/go-raftchunking v0.6.3-0.20191002164813-7e9e8525653a
github.com/hashicorp/go-retryablehttp v0.7.4
github.com/hashicorp/go-rootcerts v1.0.2
@@ -389,7 +389,7 @@ require (
github.com/hashicorp/go-metrics v0.5.1 // indirect
github.com/hashicorp/go-msgpack/v2 v2.0.0 // indirect
github.com/hashicorp/go-secure-stdlib/fileutil v0.1.0 // indirect
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.2.2 // indirect
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.3.0 // indirect
github.com/hashicorp/go-slug v0.12.1 // indirect
github.com/hashicorp/go-tfe v1.33.0 // indirect
github.com/hashicorp/jsonapi v0.0.0-20210826224640-ee7dae0fb22d // indirect
@@ -419,6 +419,7 @@ require (
github.com/jeffchao/backoff v0.0.0-20140404060208-9d7fd7aa17f2 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/joshlf/go-acl v0.0.0-20200411065538-eae00ae38531 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kelseyhightower/envconfig v1.4.0 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect

16
go.sum
View File

@@ -1548,7 +1548,6 @@ github.com/docker/docker v1.4.2-0.20190924003213-a8608b5b67c7/go.mod h1:eEKB0N0r
github.com/docker/docker v20.10.7+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker v20.10.17+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker v20.10.20+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker v24.0.5+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker v24.0.7+incompatible h1:Wo6l37AuwP3JaMnZa226lzVXGA3F9Ig1seQen0cKYlM=
github.com/docker/docker v24.0.7+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker-credential-helpers v0.6.3/go.mod h1:WRaJzqw3CTB9bk10avuGsjVBZsD05qeibJ1/TYlvc0Y=
@@ -2206,9 +2205,8 @@ github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHh
github.com/hashicorp/go-multierror v1.1.0/go.mod h1:spPvp8C1qA32ftKqdAHm4hHTbPw+vmowP0z+KUhOZdA=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/hashicorp/go-plugin v1.5.1/go.mod h1:w1sAEES3g3PuV/RzUrgow20W2uErMly84hhD3um1WL4=
github.com/hashicorp/go-plugin v1.5.2 h1:aWv8eimFqWlsEiMrYZdPYl+FdHaBJSN4AWwGWfT1G2Y=
github.com/hashicorp/go-plugin v1.5.2/go.mod h1:w1sAEES3g3PuV/RzUrgow20W2uErMly84hhD3um1WL4=
github.com/hashicorp/go-plugin v1.6.0 h1:wgd4KxHJTVGGqWBq4QPB1i5BZNEx9BR8+OFmHDmTk8A=
github.com/hashicorp/go-plugin v1.6.0/go.mod h1:lBS5MtSSBZk0SHc66KACcjjlU6WzEVP/8pwz68aMkCI=
github.com/hashicorp/go-raftchunking v0.6.3-0.20191002164813-7e9e8525653a h1:FmnBDwGwlTgugDGbVxwV8UavqSMACbGrUpfc98yFLR4=
github.com/hashicorp/go-raftchunking v0.6.3-0.20191002164813-7e9e8525653a/go.mod h1:xbXnmKqX9/+RhPkJ4zrEx4738HacP72aaUPlT2RZ4sU=
github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
@@ -2241,8 +2239,8 @@ github.com/hashicorp/go-secure-stdlib/parseutil v0.1.7 h1:UpiO20jno/eV1eVZcxqWnU
github.com/hashicorp/go-secure-stdlib/parseutil v0.1.7/go.mod h1:QmrqtbKuxxSWTN3ETMPuB+VtEiBJ/A9XhoYGv8E1uD8=
github.com/hashicorp/go-secure-stdlib/password v0.1.1 h1:6JzmBqXprakgFEHwBgdchsjaA9x3GyjdI568bXKxa60=
github.com/hashicorp/go-secure-stdlib/password v0.1.1/go.mod h1:9hH302QllNwu1o2TGYtSk8I8kTAN0ca1EHpwhm5Mmzo=
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.2.2 h1:lNWQ5KVsLmzjvN11LYqaTXtMrCP7CyxfmTeR3h0l3s8=
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.2.2/go.mod h1:7xQt0+IfRmzYBLpFx+4MYfLpBdd1PT1VatGKRswf7xE=
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.3.0 h1:KMWpBsC65ZBXDpoxJ0n2/zVfZaZIW73k2d8cy5Dv/Kk=
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.3.0/go.mod h1:qKYwSZ2EOpppko5ud+Sh9TrUgiTAZSaQCr8XWIYXsbM=
github.com/hashicorp/go-secure-stdlib/reloadutil v0.1.1 h1:SMGUnbpAcat8rIKHkBPjfv81yC46a8eCNZ2hsR2l1EI=
github.com/hashicorp/go-secure-stdlib/reloadutil v0.1.1/go.mod h1:Ch/bf00Qnx77MZd49JRgHYqHQjtEmTgGU2faufpVZb0=
github.com/hashicorp/go-secure-stdlib/strutil v0.1.1/go.mod h1:gKOamz3EwoIoJq7mlMIRBpVTAUn8qPCrEclOKKWhD3U=
@@ -2372,7 +2370,6 @@ github.com/hashicorp/vault/vault/hcp_link/proto v0.0.0-20230201201504-b741fa893d
github.com/hashicorp/vault/vault/hcp_link/proto v0.0.0-20230201201504-b741fa893d77/go.mod h1:a2crHoMWwY6aiL8GWT8hYj7vKD64uX0EdRPbnsHF5wU=
github.com/hashicorp/vic v1.5.1-0.20190403131502-bbfe86ec9443 h1:O/pT5C1Q3mVXMyuqg7yuAWUg/jMZR1/0QTzTRdNR6Uw=
github.com/hashicorp/vic v1.5.1-0.20190403131502-bbfe86ec9443/go.mod h1:bEpDU35nTu0ey1EXjwNwPjI9xErAsoOCmcMb9GKvyxo=
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM=
github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM=
github.com/hashicorp/yamux v0.1.1 h1:yrQxtgseBDrq9Y652vSRDvsKCJKOUD+GzTS4Y0Y8pvE=
github.com/hashicorp/yamux v0.1.1/go.mod h1:CtWFDAQgb7dxtzFs4tWbplKIe2jSi3+5vKbgIO0SLnQ=
@@ -2501,6 +2498,10 @@ github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22
github.com/jonboulle/clockwork v0.2.2/go.mod h1:Pkfl5aHPm1nk2H9h0bjmnJD/BcgbGXUBGnn1kMkgxc8=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/joshlf/go-acl v0.0.0-20200411065538-eae00ae38531 h1:hgVxRoDDPtQE68PT4LFvNlPz2nBKd3OMlGKIQ69OmR4=
github.com/joshlf/go-acl v0.0.0-20200411065538-eae00ae38531/go.mod h1:fqTUQpVYBvhCNIsMXGl2GE9q6z94DIP6NtFKXCSTVbg=
github.com/joshlf/testutil v0.0.0-20170608050642-b5d8aa79d93d h1:J8tJzRyiddAFF65YVgxli+TyWBi0f79Sld6rJP6CBcY=
github.com/joshlf/testutil v0.0.0-20170608050642-b5d8aa79d93d/go.mod h1:b+Q3v8Yrg5o15d71PSUraUzYb+jWl6wQMSBXSGS/hv0=
github.com/joyent/triton-go v0.0.0-20180628001255-830d2b111e62/go.mod h1:U+RSyWxWd04xTqnuOQxnai7XGS2PrPY2cfGoDKtMHjA=
github.com/joyent/triton-go v1.7.1-0.20200416154420-6801d15b779f h1:ENpDacvnr8faw5ugQmEF1QYk+f/Y9lXFvuYmRxykago=
github.com/joyent/triton-go v1.7.1-0.20200416154420-6801d15b779f/go.mod h1:KDSfL7qe5ZfQqvlDMkVjCztbmcpp/c8M77vhQP8ZPvk=
@@ -4366,6 +4367,7 @@ google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGO
google.golang.org/grpc v1.56.1/go.mod h1:I9bI3vqKfayGqPUAwGdOSu7kt6oIJLixfffKrpXqQ9s=
google.golang.org/grpc v1.56.2/go.mod h1:I9bI3vqKfayGqPUAwGdOSu7kt6oIJLixfffKrpXqQ9s=
google.golang.org/grpc v1.57.0/go.mod h1:Sd+9RMTACXwmub0zcNY2c4arhtrbBYD1AUHI/dt16Mo=
google.golang.org/grpc v1.57.1/go.mod h1:Sd+9RMTACXwmub0zcNY2c4arhtrbBYD1AUHI/dt16Mo=
google.golang.org/grpc v1.57.2/go.mod h1:Sd+9RMTACXwmub0zcNY2c4arhtrbBYD1AUHI/dt16Mo=
google.golang.org/grpc v1.58.2/go.mod h1:tgX3ZQDlNJGU96V6yHh1T/JeoBQ2TXdr43YbYSsCJk0=
google.golang.org/grpc v1.58.3 h1:BjnpXut1btbtgN/6sp+brB2Kbm2LjNXnidYujAVbSoQ=

View File

@@ -24,13 +24,13 @@ require (
github.com/hashicorp/go-kms-wrapping/entropy/v2 v2.0.0
github.com/hashicorp/go-kms-wrapping/v2 v2.0.8
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/go-plugin v1.5.2
github.com/hashicorp/go-plugin v1.6.0
github.com/hashicorp/go-retryablehttp v0.7.1
github.com/hashicorp/go-secure-stdlib/base62 v0.1.2
github.com/hashicorp/go-secure-stdlib/mlock v0.1.2
github.com/hashicorp/go-secure-stdlib/parseutil v0.1.7
github.com/hashicorp/go-secure-stdlib/password v0.1.1
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.2.2
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.3.0
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2
github.com/hashicorp/go-secure-stdlib/tlsutil v0.1.2
github.com/hashicorp/go-sockaddr v1.0.2
@@ -82,6 +82,7 @@ require (
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/pgtype v1.14.0 // indirect
github.com/jackc/pgx/v4 v4.18.1 // indirect
github.com/joshlf/go-acl v0.0.0-20200411065538-eae00ae38531 // indirect
github.com/klauspost/compress v1.16.5 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect

View File

@@ -247,8 +247,8 @@ github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHh
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/hashicorp/go-plugin v1.0.1/go.mod h1:++UyYGoz3o5w9ZzAdZxtQKrWWP+iqPBn3cQptSMzBuY=
github.com/hashicorp/go-plugin v1.5.2 h1:aWv8eimFqWlsEiMrYZdPYl+FdHaBJSN4AWwGWfT1G2Y=
github.com/hashicorp/go-plugin v1.5.2/go.mod h1:w1sAEES3g3PuV/RzUrgow20W2uErMly84hhD3um1WL4=
github.com/hashicorp/go-plugin v1.6.0 h1:wgd4KxHJTVGGqWBq4QPB1i5BZNEx9BR8+OFmHDmTk8A=
github.com/hashicorp/go-plugin v1.6.0/go.mod h1:lBS5MtSSBZk0SHc66KACcjjlU6WzEVP/8pwz68aMkCI=
github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
github.com/hashicorp/go-retryablehttp v0.5.4/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
github.com/hashicorp/go-retryablehttp v0.7.1 h1:sUiuQAnLlbvmExtFQs72iFW/HXeUn8Z1aJLQ4LJJbTQ=
@@ -265,8 +265,8 @@ github.com/hashicorp/go-secure-stdlib/parseutil v0.1.7 h1:UpiO20jno/eV1eVZcxqWnU
github.com/hashicorp/go-secure-stdlib/parseutil v0.1.7/go.mod h1:QmrqtbKuxxSWTN3ETMPuB+VtEiBJ/A9XhoYGv8E1uD8=
github.com/hashicorp/go-secure-stdlib/password v0.1.1 h1:6JzmBqXprakgFEHwBgdchsjaA9x3GyjdI568bXKxa60=
github.com/hashicorp/go-secure-stdlib/password v0.1.1/go.mod h1:9hH302QllNwu1o2TGYtSk8I8kTAN0ca1EHpwhm5Mmzo=
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.2.2 h1:lNWQ5KVsLmzjvN11LYqaTXtMrCP7CyxfmTeR3h0l3s8=
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.2.2/go.mod h1:7xQt0+IfRmzYBLpFx+4MYfLpBdd1PT1VatGKRswf7xE=
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.3.0 h1:KMWpBsC65ZBXDpoxJ0n2/zVfZaZIW73k2d8cy5Dv/Kk=
github.com/hashicorp/go-secure-stdlib/plugincontainer v0.3.0/go.mod h1:qKYwSZ2EOpppko5ud+Sh9TrUgiTAZSaQCr8XWIYXsbM=
github.com/hashicorp/go-secure-stdlib/strutil v0.1.1/go.mod h1:gKOamz3EwoIoJq7mlMIRBpVTAUn8qPCrEclOKKWhD3U=
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 h1:kes8mmyCpxJsI7FTwtzRqEy9CdjCtrXrXGuOpxEA7Ts=
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2/go.mod h1:Gou2R9+il93BqX25LAKCLuM+y9U2T4hlwvT1yprcna4=
@@ -351,6 +351,9 @@ github.com/jhump/protoreflect v1.15.1 h1:HUMERORf3I3ZdX05WaQ6MIpd/NJ434hTp5YiKgf
github.com/jimlambrt/gldap v0.1.4 h1:PoB5u4ND0E+6W99JtQJvcjGFw+iKi3Gx3M60oOJBOqE=
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/joshlf/go-acl v0.0.0-20200411065538-eae00ae38531 h1:hgVxRoDDPtQE68PT4LFvNlPz2nBKd3OMlGKIQ69OmR4=
github.com/joshlf/go-acl v0.0.0-20200411065538-eae00ae38531/go.mod h1:fqTUQpVYBvhCNIsMXGl2GE9q6z94DIP6NtFKXCSTVbg=
github.com/joshlf/testutil v0.0.0-20170608050642-b5d8aa79d93d h1:J8tJzRyiddAFF65YVgxli+TyWBi0f79Sld6rJP6CBcY=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=

View File

@@ -13,4 +13,5 @@ type PluginRuntimeConfig struct {
CgroupParent string `json:"cgroup_parent" structs:"cgroup_parent"`
CPU int64 `json:"cpu" structs:"cpu"`
Memory int64 `json:"memory" structs:"memory"`
Rootless bool `json:"rootless" structs:"rootlesss"`
}

View File

@@ -146,6 +146,7 @@ func (rc runConfig) makeConfig(ctx context.Context) (*plugin.ClientConfig, error
Group: strconv.Itoa(containerCfg.GroupAdd),
TempDir: os.Getenv("VAULT_PLUGIN_TMPDIR"),
}
clientConfig.GRPCBrokerMultiplex = true
}
return clientConfig, nil
}
@@ -161,7 +162,7 @@ func (rc runConfig) containerConfig(ctx context.Context, env []string) (*pluginc
SHA256: fmt.Sprintf("%x", rc.sha256),
Env: env,
GroupAdd: os.Getgid(),
GroupAdd: os.Getegid(),
Runtime: consts.DefaultContainerPluginOCIRuntime,
CapIPCLock: rc.mlockEnabled(),
Labels: map[string]string{
@@ -188,6 +189,9 @@ func (rc runConfig) containerConfig(ctx context.Context, env []string) (*pluginc
if rc.runtimeConfig.OCIRuntime != "" {
cfg.Runtime = rc.runtimeConfig.OCIRuntime
}
if rc.runtimeConfig.Rootless {
cfg.Rootless = true
}
}
return cfg, nil

View File

@@ -7,6 +7,7 @@ import (
"context"
"encoding/hex"
"fmt"
"os"
"os/exec"
"strings"
"testing"
@@ -19,67 +20,73 @@ import (
"github.com/hashicorp/vault/sdk/logical"
)
func testClusterWithContainerPlugin(t *testing.T, pluginType consts.PluginType, version string) (*Core, pluginhelpers.TestPlugin) {
coreConfig := &CoreConfig{
CredentialBackends: map[string]logical.Factory{},
}
cluster := NewTestCluster(t, coreConfig, &TestClusterOptions{
Plugins: &TestPluginConfig{
Typ: pluginType,
Versions: []string{version},
func testClusterWithContainerPlugins(t *testing.T, types []consts.PluginType) (*Core, []pluginhelpers.TestPlugin) {
var plugins []*TestPluginConfig
for _, typ := range types {
plugins = append(plugins, &TestPluginConfig{
Typ: typ,
Versions: []string{"v1.0.0"},
Container: true,
},
})
}
cluster := NewTestCluster(t, &CoreConfig{}, &TestClusterOptions{
Plugins: plugins,
})
cluster.Start()
t.Cleanup(cluster.Cleanup)
c := cluster.Cores[0].Core
TestWaitActive(t, c)
plugins := cluster.Plugins
core := cluster.Cores[0].Core
TestWaitActive(t, core)
return c, plugins[0]
return core, cluster.Plugins
}
func TestExternalPluginInContainer_MountAndUnmount(t *testing.T) {
for name, tc := range map[string]struct {
pluginType consts.PluginType
}{
"auth": {
pluginType: consts.PluginTypeCredential,
},
"secrets": {
pluginType: consts.PluginTypeSecrets,
},
} {
t.Run(name, func(t *testing.T) {
c, plugin := testClusterWithContainerPlugin(t, tc.pluginType, "v1.0.0")
t.Run("rootful docker runtimes", func(t *testing.T) {
t.Setenv("DOCKER_HOST", "unix:///var/run/docker.sock")
c, plugins := testClusterWithContainerPlugins(t, []consts.PluginType{
consts.PluginTypeCredential,
consts.PluginTypeSecrets,
})
t.Run("default", func(t *testing.T) {
for _, plugin := range plugins {
t.Run(plugin.Typ.String(), func(t *testing.T) {
t.Run("default runtime", func(t *testing.T) {
if _, err := exec.LookPath("runsc"); err != nil {
t.Skip("Skipping test as runsc not found on path")
}
mountAndUnmountContainerPlugin_WithRuntime(t, c, plugin, "")
mountAndUnmountContainerPlugin_WithRuntime(t, c, plugin, "", false)
})
t.Run("runc", func(t *testing.T) {
mountAndUnmountContainerPlugin_WithRuntime(t, c, plugin, "runc")
mountAndUnmountContainerPlugin_WithRuntime(t, c, plugin, "runc", false)
})
t.Run("runsc", func(t *testing.T) {
if _, err := exec.LookPath("runsc"); err != nil {
t.Skip("Skipping test as runsc not found on path")
}
mountAndUnmountContainerPlugin_WithRuntime(t, c, plugin, "runsc")
mountAndUnmountContainerPlugin_WithRuntime(t, c, plugin, "runsc", false)
})
})
}
})
t.Run("rootless runsc", func(t *testing.T) {
t.Setenv("DOCKER_HOST", fmt.Sprintf("unix:///run/user/%d/docker.sock", os.Getuid()))
c, plugins := testClusterWithContainerPlugins(t, []consts.PluginType{consts.PluginTypeCredential})
if _, err := exec.LookPath("runsc"); err != nil {
t.Skip("Skipping test as runsc not found on path")
}
mountAndUnmountContainerPlugin_WithRuntime(t, c, plugins[0], "runsc", true)
})
}
func mountAndUnmountContainerPlugin_WithRuntime(t *testing.T, c *Core, plugin pluginhelpers.TestPlugin, ociRuntime string) {
func mountAndUnmountContainerPlugin_WithRuntime(t *testing.T, c *Core, plugin pluginhelpers.TestPlugin, ociRuntime string, rootless bool) {
if ociRuntime != "" {
registerPluginRuntime(t, c.systemBackend, ociRuntime, ociRuntime)
registerPluginRuntime(t, c.systemBackend, ociRuntime, rootless)
}
registerContainerPlugin(t, c.systemBackend, plugin.Name, plugin.Typ.String(), "1.0.0", plugin.ImageSha256, plugin.Image, ociRuntime)
@@ -90,7 +97,7 @@ func mountAndUnmountContainerPlugin_WithRuntime(t *testing.T, c *Core, plugin pl
if plugin.Typ == consts.PluginTypeCredential {
pluginPath = "auth/foo/bar"
}
match := c.router.MatchingMount(namespace.RootContext(nil), pluginPath)
match := c.router.MatchingMount(namespace.RootContext(context.Background()), pluginPath)
if expectMatch && match != strings.TrimSuffix(pluginPath, "bar") {
t.Fatalf("missing mount, match: %q", match)
}
@@ -105,25 +112,13 @@ func mountAndUnmountContainerPlugin_WithRuntime(t *testing.T, c *Core, plugin pl
}
func TestExternalPluginInContainer_GetBackendTypeVersion(t *testing.T) {
for name, tc := range map[string]struct {
pluginType consts.PluginType
setRunningVersion string
}{
"external credential plugin": {
pluginType: consts.PluginTypeCredential,
setRunningVersion: "v1.2.3",
},
"external secrets plugin": {
pluginType: consts.PluginTypeSecrets,
setRunningVersion: "v1.2.3",
},
"external database plugin": {
pluginType: consts.PluginTypeDatabase,
setRunningVersion: "v1.2.3",
},
} {
t.Run(name, func(t *testing.T) {
c, plugin := testClusterWithContainerPlugin(t, tc.pluginType, tc.setRunningVersion)
c, plugins := testClusterWithContainerPlugins(t, []consts.PluginType{
consts.PluginTypeCredential,
consts.PluginTypeSecrets,
consts.PluginTypeDatabase,
})
for _, plugin := range plugins {
t.Run(plugin.Typ.String(), func(t *testing.T) {
for _, ociRuntime := range []string{"runc", "runsc"} {
t.Run(ociRuntime, func(t *testing.T) {
if _, err := exec.LookPath(ociRuntime); err != nil {
@@ -144,7 +139,7 @@ func TestExternalPluginInContainer_GetBackendTypeVersion(t *testing.T) {
var version logical.PluginVersion
var err error
if tc.pluginType == consts.PluginTypeDatabase {
if plugin.Typ == consts.PluginTypeDatabase {
version, err = c.pluginCatalog.getDatabaseRunningVersion(context.Background(), entry)
} else {
version, err = c.pluginCatalog.getBackendRunningVersion(context.Background(), entry)
@@ -152,8 +147,8 @@ func TestExternalPluginInContainer_GetBackendTypeVersion(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if version.Version != tc.setRunningVersion {
t.Errorf("Expected to get version %v but got %v", tc.setRunningVersion, version.Version)
if version.Version != plugin.Version {
t.Errorf("Expected to get version %v but got %v", plugin.Version, version.Version)
}
})
}
@@ -170,19 +165,20 @@ func registerContainerPlugin(t *testing.T, sys *SystemBackend, pluginName, plugi
"version": version,
"runtime": runtime,
}
resp, err := sys.HandleRequest(namespace.RootContext(nil), req)
resp, err := sys.HandleRequest(namespace.RootContext(context.Background()), req)
if err != nil || (resp != nil && resp.IsError()) {
t.Fatalf("err:%v resp:%#v", err, resp)
}
}
func registerPluginRuntime(t *testing.T, sys *SystemBackend, name, ociRuntime string) {
func registerPluginRuntime(t *testing.T, sys *SystemBackend, ociRuntime string, rootless bool) {
t.Helper()
req := logical.TestRequest(t, logical.UpdateOperation, fmt.Sprintf("plugins/runtimes/catalog/%s/%s", consts.PluginRuntimeTypeContainer, name))
req := logical.TestRequest(t, logical.UpdateOperation, fmt.Sprintf("plugins/runtimes/catalog/%s/%s", consts.PluginRuntimeTypeContainer, ociRuntime))
req.Data = map[string]interface{}{
"oci_runtime": ociRuntime,
"rootless": rootless,
}
resp, err := sys.HandleRequest(namespace.RootContext(nil), req)
resp, err := sys.HandleRequest(namespace.RootContext(context.Background()), req)
if err != nil || (resp != nil && resp.IsError()) {
t.Fatalf("err:%v resp:%#v", err, resp)
}

View File

@@ -70,10 +70,12 @@ func TestCore_EnableExternalPlugin(t *testing.T) {
}
cluster := NewTestCluster(t, coreConfig, &TestClusterOptions{
Plugins: &TestPluginConfig{
Plugins: []*TestPluginConfig{
{
Typ: tc.pluginType,
Versions: []string{""},
},
},
})
cluster.Start()

View File

@@ -47,10 +47,12 @@ func getClusterWithFileAuditBackend(t *testing.T, typ consts.PluginType, numCore
cluster := vault.NewTestCluster(t, coreConfig, &vault.TestClusterOptions{
TempDir: pluginDir,
NumCores: numCores,
Plugins: &vault.TestPluginConfig{
Plugins: []*vault.TestPluginConfig{
{
Typ: typ,
Versions: []string{""},
},
},
HandlerFunc: vaulthttp.Handler,
})
@@ -73,10 +75,12 @@ func getCluster(t *testing.T, typ consts.PluginType, numCores int) *vault.TestCl
cluster := vault.NewTestCluster(t, coreConfig, &vault.TestClusterOptions{
TempDir: pluginDir,
NumCores: numCores,
Plugins: &vault.TestPluginConfig{
Plugins: []*vault.TestPluginConfig{
{
Typ: typ,
Versions: []string{""},
},
},
HandlerFunc: vaulthttp.Handler,
})
@@ -101,10 +105,12 @@ func TestExternalPlugin_RollbackAndReload(t *testing.T) {
cluster := vault.NewTestCluster(t, coreConfig, &vault.TestClusterOptions{
TempDir: pluginDir,
NumCores: 1,
Plugins: &vault.TestPluginConfig{
Plugins: []*vault.TestPluginConfig{
{
Typ: consts.PluginTypeSecrets,
Versions: []string{""},
},
},
HandlerFunc: vaulthttp.Handler,
})

View File

@@ -809,6 +809,7 @@ func (b *SystemBackend) handlePluginRuntimeCatalogUpdate(ctx context.Context, _
if memory < 0 {
return logical.ErrorResponse("runtime memory in bytes cannot be negative"), nil
}
rootless := d.Get("rootless").(bool)
if err = b.Core.pluginRuntimeCatalog.Set(ctx,
&pluginruntimeutil.PluginRuntimeConfig{
Name: runtimeName,
@@ -817,6 +818,7 @@ func (b *SystemBackend) handlePluginRuntimeCatalogUpdate(ctx context.Context, _
CgroupParent: cgroupParent,
CPU: cpu,
Memory: memory,
Rootless: rootless,
}); err != nil {
return logical.ErrorResponse(err.Error()), nil
}
@@ -889,6 +891,7 @@ func (b *SystemBackend) handlePluginRuntimeCatalogRead(ctx context.Context, _ *l
"cgroup_parent": conf.CgroupParent,
"cpu_nanos": conf.CPU,
"memory_bytes": conf.Memory,
"rootless": conf.Rootless,
}}, nil
}
@@ -928,6 +931,7 @@ func (b *SystemBackend) handlePluginRuntimeCatalogList(ctx context.Context, _ *l
"cgroup_parent": conf.CgroupParent,
"cpu_nanos": conf.CPU,
"memory_bytes": conf.Memory,
"rootless": conf.Rootless,
})
}
}
@@ -6288,6 +6292,10 @@ This path responds to the following HTTP methods.
"Memory limit to set per container in bytes. Defaults to no limit.",
"",
},
"plugin-runtime-catalog_rootless": {
"Whether the container runtime is run as a non-privileged (non-root) user.",
"",
},
"leases": {
`View or list lease metadata.`,
`

View File

@@ -2143,6 +2143,10 @@ func (b *SystemBackend) pluginsRuntimesCatalogCRUDPath() *framework.Path {
Type: framework.TypeInt64,
Description: strings.TrimSpace(sysHelp["plugin-runtime-catalog_memory-bytes"][0]),
},
"rootless": {
Type: framework.TypeBool,
Description: strings.TrimSpace(sysHelp["plugin-runtime-catalog_rootless"][0]),
},
},
Operations: map[logical.Operation]framework.OperationHandler{
@@ -2212,6 +2216,11 @@ func (b *SystemBackend) pluginsRuntimesCatalogCRUDPath() *framework.Path {
Description: strings.TrimSpace(sysHelp["plugin-runtime-catalog_memory-bytes"][0]),
Required: true,
},
"rootless": {
Type: framework.TypeBool,
Description: strings.TrimSpace(sysHelp["plugin-runtime-catalog_rootless"][0]),
Required: true,
},
},
}},
},

View File

@@ -6104,6 +6104,7 @@ func TestSystemBackend_pluginRuntimeCRUD(t *testing.T) {
CgroupParent: "/cpulimit/",
CPU: 1,
Memory: 10000,
Rootless: true,
}
// Register the plugin runtime
@@ -6113,6 +6114,7 @@ func TestSystemBackend_pluginRuntimeCRUD(t *testing.T) {
"cgroup_parent": conf.CgroupParent,
"cpu_nanos": conf.CPU,
"memory_bytes": conf.Memory,
"rootless": conf.Rootless,
}
resp, err := b.HandleRequest(namespace.RootContext(nil), req)
@@ -6153,6 +6155,7 @@ func TestSystemBackend_pluginRuntimeCRUD(t *testing.T) {
"cgroup_parent": conf.CgroupParent,
"cpu_nanos": conf.CPU,
"memory_bytes": conf.Memory,
"rootless": conf.Rootless,
}
if !reflect.DeepEqual(resp.Data, readExp) {
t.Fatalf("got: %#v expect: %#v", resp.Data, readExp)

View File

@@ -5,6 +5,10 @@ FROM docker.mirror.hashicorp.services/ubuntu:22.04
ARG plugin
RUN groupadd nonroot && useradd -g nonroot nonroot
USER nonroot
COPY ${plugin} /bin/plugin
ENTRYPOINT [ "/bin/plugin" ]

View File

@@ -1256,7 +1256,7 @@ type TestClusterOptions struct {
NoDefaultQuotas bool
Plugins *TestPluginConfig
Plugins []*TestPluginConfig
// if populated, the callback is called for every request
RequestResponseCallback func(logical.Backend, *logical.Request, *logical.Response)
@@ -1699,8 +1699,10 @@ func NewTestCluster(t testing.T, base *CoreConfig, opts *TestClusterOptions) *Te
opts.ClusterLayers = inmemCluster
}
if opts != nil && opts.Plugins != nil {
if opts.Plugins.Container && runtime.GOOS != "linux" {
if opts != nil && len(opts.Plugins) != 0 {
var plugins []pluginhelpers.TestPlugin
for _, pluginType := range opts.Plugins {
if pluginType.Container && runtime.GOOS != "linux" {
t.Skip("Running plugins in containers is only supported on linux")
}
@@ -1713,14 +1715,14 @@ func NewTestCluster(t testing.T, base *CoreConfig, opts *TestClusterOptions) *Te
t.Cleanup(func() { cleanup(t) })
}
var plugins []pluginhelpers.TestPlugin
for _, version := range opts.Plugins.Versions {
plugin := pluginhelpers.CompilePlugin(t, opts.Plugins.Typ, version, coreConfig.PluginDirectory)
if opts.Plugins.Container {
for _, version := range pluginType.Versions {
plugin := pluginhelpers.CompilePlugin(t, pluginType.Typ, version, coreConfig.PluginDirectory)
if pluginType.Container {
plugin.Image, plugin.ImageSha256 = pluginhelpers.BuildPluginContainerImage(t, plugin, coreConfig.PluginDirectory)
}
plugins = append(plugins, plugin)
}
}
testCluster.Plugins = plugins
}

View File

@@ -78,6 +78,10 @@ supplied type and name.
- `name` `(string: <required>)`  Part of the request URL. Specifies the plugin runtime name.
Use the runtime name to look up plugin runtimes in the catalog.
- `rootless` `(bool: false)` - Whether the container runtime is running as a
non-privileged user. Must be set if plugin container images are also configured
to run as a non-root user.
- `oci_runtime` `(string: <optional>)` Specifies OCI-compliant container runtime to use.
Default is "runsc", gVisor's OCI runtime.

View File

@@ -45,6 +45,10 @@ flags](/vault/docs/commands) included on all commands.
- `-type` `(string: <required>)` - Plugin runtime type. Vault currently only
supports `container` as a runtime type.
- `-rootless` `(bool: false)` - Whether the container runtime is running as a
non-privileged user. Must be set if plugin container images are also configured
to run as a non-root user.
- `-cgroup_parent` `(string: "")` - Parent cgroup to set for each container.
Use `cgroup_parent` to control the total resource usage for a group of plugins.

View File

@@ -39,7 +39,7 @@ increases the isolation between plugins, and between plugins and Vault.
All plugins have the following basic requirements to be containerized:
- **Your plugin must be built with at least v1.5.0 of the HashiCorp
- **Your plugin must be built with at least v1.6.0 of the HashiCorp
[`go-plugin`](https://github.com/hashicorp/go-plugin) library**.
- **The image entrypoint should run the plugin binary**.
@@ -52,39 +52,39 @@ in [supported configurations](#supported-configurations).
Vault's containerized plugins are compatible with a variety of configurations.
In particular, it has been tested with the following:
- Docker and Podman.
- Default and rootless container engine.
- OCI runtimes runsc and runc.
- Plugin container images with root and non-root users.
- Default and [rootless](https://docs.docker.com/engine/security/rootless/) Docker.
- OCI-compatible runtimes `runsc` and `runc`.
- Plugin container images running as root and non-root users.
- [Mlock](/vault/docs/configuration#disable_mlock) disabled or enabled.
Not all combinations work and some have additional requirements, listed below.
If you use a configuration that matches multiple headings, you should combine
the requirements from each matching heading.
### Rootless installation with non-root container user
### `runsc` runtime
Not currently supported. We are hoping to provide support in future.
- You must pass an additional `--host-uds=create` flag to the `runsc` runtime.
### runsc runtime
### Rootless Docker with `runsc` runtime
- You must pass an additional `--host-uds=all` flag to the `runsc` runtime.
### Rootless installation with `runsc`
- Does not currently support cgroup limits.
- You must pass an additional `--ignore-cgroups` flag to the `runsc` runtime.
- Cgroup limits are not currently supported for this configuration.
### Rootless Docker with non-root container user
- You must use a container plugin runtime with
[`rootless`](/vault/docs/commands/plugin/runtime/register#rootless) enabled.
- Your filesystem must have Posix 1e ACL support, available by default in most
modern Linux file systems.
- Only supported for gVisor's `runsc` runtime.
### Rootless Docker with mlock enabled
- Only supported for gVisor's `runsc` runtime.
### Non-root container user with mlock enabled
- You must set the IPC_LOCK capability on the plugin binary.
### Rootless container engine with mlock enabled
- You must set the IPC_LOCK capability on the container engine's binary.
- You do not need to set the IPC_LOCK capability if running with Docker and runsc.
The `runsc` runtime supports mlock syscalls in rootless Docker without needing
IPC_LOCK itself.
- You must set the `IPC_LOCK` capability on the plugin binary.
## Container lifecycle and metadata