What happens at the moment in e.g. pull-kubernetes-e2e-kind in case of a
timeout is that ginkgo-e2e.sh gets killed with SIGTERM. This is not propagated
to the E2E test suite processes, therefore there is no "Interrupted by User"
report and no JUnit file, depending on timing during the process shutdown.
Running the Ginkgo CLI with job control enabled creates a new process group,
which then can be used to kill the Ginko CLI and the E2E test suite
processes. With these changes, more information is produced. Some of it seems
a bit redundant, but it's better than none:
*** hack/ginkgo-e2e.sh: received termination signal -> asking Ginkgo to stop.
***
*** Beware that a timeout may have been caused by some earlier test,
*** not necessarily the one which gets interrupted now.
*** See the "Spec runtime" for information about how long the
*** interrupted test was running.
------------------------------
Interrupted by User
First interrupt received; Ginkgo will run any cleanup and reporting nodes but will skip all remaining specs. Interrupt again to skip cleanup.
Here's a current progress report:
[sig-node] DRA [Feature:DynamicResourceAllocation] [FeatureGate:DynamicResourceAllocation] [Beta] ResourceSlice Controller creates slices (Spec Runtime: 9
.065s)
k8s.io/kubernetes/test/e2e/dra/dra.go:812
In [It] (Node Runtime: 9.044s)
k8s.io/kubernetes/test/e2e/dra/dra.go:812
At [By Step] Creating slices (Step Runtime: 8.884s)
k8s.io/kubernetes/test/e2e/dra/dra.go:847
...
Begin Additional Progress Reports >>
There is no failure as the matcher passed to Consistently has not yet failed
<< End Additional Progress Reports
------------------------------
• [INTERRUPTED] [11.955 seconds]
[sig-node] DRA [Feature:DynamicResourceAllocation] [FeatureGate:DynamicResourceAllocation] [Beta] ResourceSlice Controller [It] creates slices [sig-node, Feature:DynamicResourceAllocation, FeatureGate:DynamicResourceAllocation, Feature:Beta]
k8s.io/kubernetes/test/e2e/dra/dra.go:812
Timeline >>
STEP: Creating a kubernetes client @ 01/09/25 17:18:59.769
...
[FAILED] in [It] - k8s.io/kubernetes/test/e2e/dra/dra.go:881 @ 01/09/25 17:19:08.835
I0109 17:19:11.703212 302727 helper.go:125] Waiting up to 7m0s for all (but 0) nodes to be ready
STEP: dump namespace information after failure @ 01/09/25 17:19:11.706
STEP: Collecting events from namespace "dra-7998". @ 01/09/25 17:19:11.706
STEP: Found 0 events. @ 01/09/25 17:19:11.708
...
STEP: Destroying namespace "dra-7998" for this suite. @ 01/09/25 17:19:11.72
<< Timeline
[INTERRUPTED] Interrupted by User
In [It] at: k8s.io/kubernetes/test/e2e/dra/dra.go:812 @ 01/09/25 17:19:08.833
This is the Progress Report generated when the interrupt was received:
[sig-node] DRA [Feature:DynamicResourceAllocation] [FeatureGate:DynamicResourceAllocation] [Beta] ResourceSlice Controller creates slices (Spec Runtime: 9
.065s)
...
[FAILED] An interrupt occurred and then the following failure was recorded in the interrupted node before it exited:
Context was cancelled (cause: Interrupted by User) after 0.329s.
There is no failure as the matcher passed to Consistently has not yet failed
In [It] at: k8s.io/kubernetes/test/e2e/dra/dra.go:881 @ 01/09/25 17:19:08.835
------------------------------
Checking for custom logdump instances, if any
----------------------------------------------------------------------------------------------------
k/k version of the log-dump.sh script is deprecated!
Please migrate your test job to use test-infra's repo version of log-dump.sh!
Migration steps can be found in the readme file.
----------------------------------------------------------------------------------------------------
Sourcing kube-util.sh
Detecting project
Skeleton Provider: detect-project not implemented
Dumping logs from master locally to '/tmp/test'
Master SSH not supported for local
Dumping logs from nodes locally to '/tmp/test'
Node SSH not supported for local
Summarizing 1 Failure:
[INTERRUPTED] [sig-node] DRA [Feature:DynamicResourceAllocation] [FeatureGate:DynamicResourceAllocation] [Beta] ResourceSlice Controller [It] creates slices [sig-node, Feature:DynamicResourceAllocation, FeatureGate:DynamicResourceAllocation, Feature:Beta]
k8s.io/kubernetes/test/e2e/dra/dra.go:812
Ran 1 of 6644 Specs in 12.208 seconds
FAIL! - Interrupted by User -- 0 Passed | 1 Failed | 0 Pending | 6643 Skipped
--- FAIL: TestE2E (12.74s)
FAIL
Ginkgo ran 1 suite in 13.379078611s
* accept GINKGO_FLAGS for test-e2e-node.sh
* Update Makefile to document LABEL_FILTER
* --label-filter part of ginkgoflags
* Update Makefile
* if label-filter then avoid skip and focus defaults
* LABEL_FILTER can be empty
* Update build/root/Makefile
Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
* skip defaults only if label-filter not set
Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
* focus and label_fiter can live together
Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
* skip and label_filter can live together
Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
---------
Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
The context is used for cancellation and to support contextual logging.
In most cases, alternative *WithContext APIs get added, except for
NewIntegerResourceVersionMutationCache where code searches indicate that the
API is not used downstream.
An API break around SharedInformer couldn't be avoided because the
alternative (keeping the interface unchanged and adding a second one with
the new method) would have been worse. controller-runtime needs to be updated
because it implements that interface in a test package. Downstream consumers of
controller-runtime will work unless they use those test package.
Converting Kubernetes to use the other new alternatives will follow. In the
meantime, usage of the new alternatives cannot be enforced via logcheck
yet (see https://github.com/kubernetes/kubernetes/issues/126379 for the
process).
Passing context through and checking it for cancellation is tricky for event
handlers. A better approach is to map the context cancellation to the normal
removal of an event handler via a helper goroutine. Thanks to the new
HandleErrorWithLogr and HandleCrashWithLogr, remembering the logger is
sufficient for handling problems at runtime.
The kubernetes repository contains some internal golang modules that are
not part of the golang global workspace. Because apidiff is currently
run from the root of the repository, it does not work against this
internal modules.
Instead of executing apidiff from the root we can just cd into the
passed path of the module to avoid this limitation.
We have zero flake policy for a long time now (> 1 year) https://github.com/kubernetes/community/pull/7538, however , there are some places that are still tolerating flakes and retrying
Flakes does not help, to the point that when we have to take a hard decision it creates more iuncertainty.
It does not matter how, we should be always able to deal with flakes:
- if the software or algorithm is racy, we need to work to make deterministic
- if is deterministic, the test must be deterministic
- if the test is determinist but it depends on the environment, then we work on making the environment deterministci
Pruning of tests to the top-level test was added for jobs like
pull-kubernetes-unit which run many tests. For other, more focused jobs like
scheduler-perf benchmarking it would be nice to keep the more detailed
information, in particular because it includes the duration per test case.
This becomes relevant once DynamicResourceAllocation becomes beta with
"disabled" as default. Otherwise AllAlpha=true enables DRAAdminAccess which
depends on DynamicResourceAllocation, which is disabled.
For one thing, this release decouples device management from
libcontainer/cgroups. You can see the result of this in a dropped
cilium/ebpf dependency (which is only needed for device management).
NOTE that due to an issue with go mod / go list,
github.com/opencontainers/runc had to be added to
hack/unwanted-dependencies.json under x/exp. This is bogus because
opencontainers/runc does not use x/exp directly, only via cilium/ebpf
dependency (which is not vendored here).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
