Commit Graph

4628 Commits

Author SHA1 Message Date
Patrick Ohly
d6851ec735 scheduler_perf: fail when input YAML is invalid
The YAML files get decoded into an unstructured object, without validation, and
then sent to the apiserver with a generic client. The default behavior is to
issue a warning to the client, which gets logged by client-go. What we want
instead is an error that causes the test to fail in a clean way right at the
beginning.
2024-02-29 09:53:16 +01:00
Tim Hockin
8b579b2347 Change the build_binaries path to use modules
This makes "new" and "old" setup_env functions. In subsequent commits,
all callers of the "old" form will be fixed, and the "new" will be
renamed back.

The old and new functions diff:

```diff
--- /tmp/a	2023-12-14 09:02:57.804092696 -0800
+++ /tmp/b	2023-12-14 09:03:09.679999585 -0800
@@ -1,4 +1,4 @@
-kube::golang::old::setup_env() {
+kube::golang::new::setup_env() {
   kube::golang::verify_go_version

   # Set up GOPATH.  We have tools which depend on being in a GOPATH (see
@@ -7,9 +7,9 @@
   # Even in module mode, we need to set GOPATH for `go build` and `go install`
   # to work.  We build various tools (usually via `go install`) from a lot of
   # scripts.
-  #   * We can't set GOBIN because that does not work on cross-compiles.
-  #   * We could use `go build -o <something>`, but it's subtle when it comes
-  #     to cross-compiles and whether the <something> is a file or a directory,
+  #   * We can't just set GOBIN because that does not work on cross-compiles.
+  #   * We could always use `go build -o <something>`, but it's subtle wrt
+  #     cross-compiles and whether the <something> is a file or a directory,
   #     and EVERY caller has to get it *just* right.
   #   * We could leave GOPATH alone and let `go install` write binaries
   #     wherever the user's GOPATH says (or doesn't say).
@@ -20,16 +20,6 @@
   #
   # Eventually, when we no longer rely on run-in-gopath.sh we may be able to
   # simplify this some.
-  local go_pkg_dir="${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}"
-  local go_pkg_basedir
-  go_pkg_basedir=$(dirname "${go_pkg_dir}")
-
-  mkdir -p "${go_pkg_basedir}"
-
-  # TODO: This symlink should be relative.
-  if [[ ! -e "${go_pkg_dir}" || "$(readlink "${go_pkg_dir}")" != "${KUBE_ROOT}" ]]; then
-    ln -snf "${KUBE_ROOT}" "${go_pkg_dir}"
-  fi
   export GOPATH="${KUBE_GOPATH}"

   # If these are not set, set them now.  This ensures that any subsequent
@@ -40,24 +30,10 @@
   # Make sure our own Go binaries are in PATH.
   export PATH="${KUBE_GOPATH}/bin:${PATH}"

-  # Change directories so that we are within the GOPATH.  Some tools get really
-  # upset if this is not true.  We use a whole fake GOPATH here to collect the
-  # resultant binaries.
-  local subdir
-  subdir=$(kube::realpath . | sed "s|${KUBE_ROOT}||")
-  cd "${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}/${subdir}" || return 1
-
-  # Set GOROOT so binaries that parse code can work properly.
-  GOROOT=$(go env GOROOT)
-  export GOROOT
-
   # Unset GOBIN in case it already exists in the current session.
   # Cross-compiles will not work with it set.
   unset GOBIN

-  # This seems to matter to some tools
-  export GO15VENDOREXPERIMENT=1
-
-  # Disable workspaces
-  export GOWORK=off
+  # Explicitly turn on modules.
+  export GO111MODULE=on
 }
```

Result: `make` works for k/k:

```
$ make kubectl
+++ [1211 11:07:31] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)

$ make WHAT=./cmd/kubectl/
+++ [1211 11:08:19] Building go targets for linux/amd64
    k8s.io/kubernetes/./cmd/kubectl/ (non-static)

$ make WHAT=k8s.io/kubernetes/cmd/kubectl
+++ [1211 11:08:52] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
```

Result: `make` works for staging by package:

```
$ make WHAT=k8s.io/api
+++ [1211 11:11:37] Building go targets for linux/amd64
    k8s.io/api (non-static)
```

Result: `make` fails for staging by path:

```
$ make WHAT=./staging/src/k8s.io/api
+++ [1211 11:12:44] Building go targets for linux/amd64
    k8s.io/kubernetes/./staging/src/k8s.io/api (non-static)
cannot find module providing package k8s.io/kubernetes/staging/src/k8s.io/api: import lookup disabled by -mod=vendor
	(Go version in go.work is at least 1.14 and vendor directory exists.)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44]  1: /home/thockin/src/kubernetes/hack/lib/golang.sh:850 kube::golang::build_some_binaries(...)
!!! [1211 11:12:44]  2: /home/thockin/src/kubernetes/hack/lib/golang.sh:1012 kube::golang::build_binaries_for_platform(...)
!!! [1211 11:12:44]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:96: all] Error 1
```

Result: `make test` fails:

```
$ make test WHAT=./cmd/kubectl
+++ [1211 11:13:38] Set GOMAXPROCS automatically to 6
+++ [1211 11:13:38] Running tests without code coverage and with -race
cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of:
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree)
	/home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT)
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH)
cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of:
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree)
	/home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT)
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH)
cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of:
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree)
	/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT)
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH)
cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of:
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree)
	/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT)
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH)
make: *** [Makefile:191: test] Error 1
```
2024-02-29 00:22:07 -08:00
Wei Huang
01db4ae9e7 Graduate PodSchedulingReadiness to stable 2024-02-28 23:18:44 -08:00
Jiahui Feng
64ee859aa8 make ValidatingAdmissionPolicy ignore excluded resources. 2024-02-28 15:31:44 -08:00
Jiahui Feng
5b1fffa3e4 add resource filter to admission initializer. 2024-02-28 15:31:18 -08:00
Michał Woźniak
e568a77a93 Support for the Job managedBy field (alpha) (#123273)
* support for the managed-by label in Job

* Use managedBy field instead of managed-by label

* Additional review remarks

* Review remarks 2

* review remarks 3

* Skip cleanup of finalizers for job with custom managedBy

* Drop the performance optimization

* imrpove logs
2024-03-05 09:25:15 -08:00
Kensei Nakada
58a826a59a graduate MinDomainsInPodTopologySpread to stable 2024-02-28 10:42:29 +00:00
Patrick Ohly
4d5264fb0b logging benchmark: support slog
Replacing zapr with slog as JSON implementation can be benchmarked by replacing
"false" with "true" and rerunning. However, slog is slower:

    Encoding/error-value/JSON-36                        171k ± 3%   148k ± 5%  -13.64%  (p=0.008 n=5+5)
    Encoding/error/JSON-36                              207k ± 9%   149k ± 4%  -27.87%  (p=0.008 n=5+5)
    Encoding/kind-worker-kubelet/JSON-36                348k ± 2%   334k ± 1%   -4.13%  (p=0.008 n=5+5)
    Encoding/kube-apiserver/JSON-36                     210k ± 2%   186k ± 1%  -11.53%  (p=0.008 n=5+5)
    Encoding/kube-controller-manager/JSON-36            291k ± 1%   249k ± 2%  -14.26%  (p=0.008 n=5+5)
    Encoding/kube-scheduler/JSON-36                     262k ± 4%   223k ± 2%  -14.91%  (p=0.008 n=5+5)
    Encoding/simple/JSON-36                             161k ± 4%   132k ± 2%  -17.60%  (p=0.008 n=5+5)
    Encoding/split/JSON-36                              159k ± 4%   132k ± 1%  -17.23%  (p=0.008 n=5+5)
    Encoding/v3/kind-worker-kubelet/JSON-36             255k ± 1%   256k ± 1%     ~     (p=0.222 n=5+5)
    Encoding/v3/kube-apiserver/JSON-36                  131k ± 4%   105k ± 4%  -20.07%  (p=0.008 n=5+5)
    Encoding/v3/kube-controller-manager/JSON-36         285k ± 2%   250k ± 2%  -12.13%  (p=0.008 n=5+5)
    Encoding/v3/kube-scheduler/JSON-36                  216k ± 3%   196k ± 4%   -9.63%  (p=0.008 n=5+5)
2024-02-27 15:36:41 +01:00
Kubernetes Prow Robot
98bd90fbe2 Merge pull request #114672 from pohly/log-text-split-streams
log: split streams also for text output
2024-02-26 01:44:58 -08:00
Antonio Ojea
099f26296b integration test for cloud provider
very simple tests to check that the node controllers removes the taint.

Test will be used later for add more coverage without having to depend
on e2e tests for validating complex scenarios and combinations of flags
and controllers.

Change-Id: I55ce01488352cce9d673585e7959678c2bfb5924
2024-02-26 09:01:21 +00:00
Mengjiao Liu
b584b87a94 kube-controller-manager: readjust log verbosity
- Increase the global level for broadcaster's logging to 3 so that users can ignore event messages by lowering the logging level. It reduces information noise.
- Making sure the context is properly injected into the broadcaster, this will allow the -v flag value to be used also in that broadcaster, rather than the above global value.
- test: use cancellation from ktesting
- golangci-hints: checked error return value
2024-02-26 14:51:56 +08:00
Kubernetes Prow Robot
38638adaee Merge pull request #123418 from rikatz/kep-1860-to-beta
Promote LoadBalancerIPMode to Beta
2024-02-24 12:48:07 -08:00
Kubernetes Prow Robot
31a482a149 Merge pull request #120344 from rohitssingh/disable_force_detach
Add a flag to disable force detach behavior in kube-controller-manager
2024-02-22 13:02:38 -08:00
Rohit Singh
13dddca6a2 Add "disable-force-detach-on-timeout" flag to kube-controller-manager 2024-02-22 18:31:52 +00:00
Ricardo Katz
d816ab7657 Promote LoadBalancerIPMode to Beta 2024-02-22 08:35:12 -03:00
Kubernetes Prow Robot
3d3d705a6d Merge pull request #122883 from Jefftree/agg-discovery-block-stale
Aggregated Discovery, mark APIServices stale before initial health check
2024-02-21 12:02:41 -08:00
Jefftree
6726e65619 Aggregated Discovery, mark stale before initial health check 2024-02-20 18:15:13 -05:00
Kubernetes Prow Robot
64386c5aba Merge pull request #122419 from pohly/scheduler-perf-create-any
scheduler-perf: create any object from YAML
2024-02-19 07:44:40 -08:00
Jordan Liggitt
d5d3eddb95 Add allowed/denied metrics for authorizers 2024-02-16 08:20:59 -05:00
Kubernetes Prow Robot
66d038d84d Merge pull request #121946 from liggitt/reload-authz
KEP-3221: Implement authorization configuration file reloading
2024-02-15 18:37:13 -08:00
Kubernetes Prow Robot
50b4cfc414 Merge pull request #123321 from mimowo/job-reset-metrics
Reset metrics in the Job integration test before testing
2024-02-15 10:57:58 -08:00
Kubernetes Prow Robot
47f92ce231 Merge pull request #123242 from mimowo/fast-backoff-for-replacment-policy-tests
Improve accuracy of the PodsCreationTotal metric and use fast pod failure backoff for ReplacementPolicy integration tests
2024-02-15 09:54:53 -08:00
Michal Wozniak
ab8cdae3b1 Reset metrics in the Job integration test before testing 2024-02-15 14:50:24 +01:00
Michal Wozniak
115dc90633 Increase accuracy of the pods_creation_total metric and improve test exec time 2024-02-15 10:59:01 +01:00
Anish Ramasekar
0feb1d5173 Add integration test for multiple audience in structured authn
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
2024-02-14 18:50:46 -08:00
Kubernetes Prow Robot
72c3c7c924 Merge pull request #123282 from enj/enj/i/authn_config_algs
Support all key algs with structured authn config
2024-02-14 18:08:32 -08:00
Kubernetes Prow Robot
ba450636a4 Merge pull request #123165 from aramase/aramase/f/kep_3331_audience_match_policy
Add `AudienceMatchPolicy` and support multiple audiences in AuthenticationConfiguration
2024-02-14 17:00:04 -08:00
Jordan Liggitt
5dc92ada06 Implement authz config file reloading 2024-02-14 18:09:15 -05:00
Anish Ramasekar
18c563546a support multiple audiences with jwt authenticator
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
2024-02-14 10:47:07 -08:00
Monis Khan
b5e0068325 Support all key algs with structured authn config
Signed-off-by: Monis Khan <mok@microsoft.com>
2024-02-14 09:40:25 -05:00
Monis Khan
b8a59346fe test/oidc: extract key type to allow testing different algs
Signed-off-by: Monis Khan <mok@microsoft.com>
2024-02-13 17:05:33 -05:00
Anish Ramasekar
77241d3125 Add apiserver_encryption_config_controller_automatic_reloads_total
metric

- Adds `apiserver_encryption_config_controller_automatic_reloads_total`
  metric with status label for encryption config reload success/failure.
- Deprecated `apiserver_encryption_config_controller_automatic_reload_failures_total` and `apiserver_encryption_config_controller_automatic_reload_success_total`

Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
2024-02-12 21:47:46 -08:00
Kubernetes Prow Robot
3bf9477b52 Merge pull request #122472 from pacoxu/flowcontrolv1
flowcontrol: set the serialization version to v1
2024-02-12 13:17:48 -08:00
Patrick Ohly
da0c9a93ae scheduler_perf: use dynamic client to create arbitrary objects
With a dynamic client and a rest mapper it is possible to load arbitrary YAML
files and create the object defined by it. This is simpler than adding specific
Go code for each supported type.

Because the version now matters, the incorrect version in the DRA YAMLs were
found and fixed.
2024-02-11 10:51:38 +01:00
Patrick Ohly
c46ae1b26a scheduler_perf: use ktesting.TContext + staging StartTestServer
ktesting.TContext combines several different interfaces. This makes the code
simpler because less parameters need to be passed around.

An intentional side effect is that the apiextensions client interface becomes
available, which makes it possible to use CRDs. This will be needed for future
DRA tests.

Support for CRDs depends on starting the apiserver via
k8s.io/kubernetes/cmd/kube-apiserver/app/testing because only that enables the
CRD extensions. As discussed on Slack, the long-term goal is to replace the
in-tree StartTestServer with the one in staging, so this is going in the right
direction.
2024-02-11 10:51:38 +01:00
James Munnelly
105ec3d48f fix linter failures 2024-02-07 12:57:05 +00:00
James Munnelly
852c03a49b check key is set before deleting from map 2024-02-07 12:31:42 +00:00
James Munnelly
4d8c3530f5 fix regular bound service account token test 2024-02-07 12:30:33 +00:00
James Munnelly
dfc20d19c8 fix integration tests now JTI feature is enabled by default 2024-02-05 18:11:50 +00:00
Kubernetes Prow Robot
42941cb88a Merge pull request #122635 from carlory/KEP-2799
Promote LegacyServiceAccountTokenCleanUp to GA
2024-02-02 12:47:23 -08:00
Alexander Zielenski
b636984004 test: remove unnecessary skip healthz check from test
snuck in there while I was working on the test, but is ultimately not necessary to test the functionality.

skipping healthz check resulted in leaking goroutines from poststarthooks.
2024-02-01 14:50:54 -08:00
Alexander Zielenski
71559bd026 bugfix: dont skip reconcile for unchanged policy if last sync failed 2024-01-29 11:16:59 -08:00
Lukasz Szaszkiewicz
41e706600a client-go/reflector: make UseWatchList a pointer
until #115478(use streaming against the etcd storage)
is resolved the cacher need a way to disable the streaming.
2024-01-19 14:40:18 +01:00
Patrick Ohly
8f4c9c7605 k8s.io/component-base/logs: replace klog text implementation
This replaces the klog formatting and message routing with a simpler
implementation that uses less code. The main difference is that we skip the
entire unused message routing.

Instead, the same split output streams as for JSON gets implemented in the
io.Writer implementation that gets passed to the textlogger.
2024-01-17 13:50:03 +01:00
Kensei Nakada
f29d6970c9 doc(scheduler_perf): enrich the documentation 2024-01-15 08:50:08 +00:00
Patrick Ohly
04b772c661 logs benchmark: really write through pipe
While the benchmark is focused on encoding, it becomes a bit more realistic
when actually passing the encoded data to the Linux kernel. Features like
output buffering are more likely to have a visible effect when invoking
syscalls.
2024-01-12 07:49:17 +01:00
Patrick Ohly
7f1a30f8d5 logs benchmark: fix config handling
The logs config must be reset explicitly now when changing it multiple times
per process.
2024-01-12 07:49:17 +01:00
carlory
0fc1b9c9aa Promote LegacyServiceAccountTokenCleanUp to GA 2024-01-08 10:58:49 +08:00
Davanum Srinivas
7a236bd7e0 Add providerless tags appropriately in test/
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
2024-01-07 08:16:05 -05:00
armstrongli
5c196ddaef add patch integration test to disable nodeport for LB type service 2024-01-07 20:04:09 +08:00