* Mount both /etc/ssl/certs and /etc/pki into control plane static
pods and kube-proxy, rather than choosing one based a variable
(set based on Flatcar Linux or Fedora CoreOS)
* Remove `trusted_certs_dir` variable
* Remove deprecated `--port` from `kube-scheduler` static Pod
* Update minimum required versions for `tls`, `template`,
and `random`. Older versions have some differing behaviors
(e.g. `random` may be missing sensitive fields) and we'd
prefer to shake loose any setups still using very old
providers than continue allowing them
* Remove the upper bound version constraint since providers
are more regularly updated these days and require less
manual vetting to allow use
* Change `enable_aggregation` default from false to true
* These days, Kubernetes control plane components emit annoying
messages related to assumptions baked into the Kubernetes API
Aggregation Layer if you don't enable it. Further the conformance
tests force you to remember to enable it if you care about passing
those
* This change is motivated by eliminating annoyances, rather than
any enthusiasm for Kubernetes' aggregation features
* https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/
* Kubernetes v1.22.0 disables kube-controller-manager insecure
port which was used internally for Prometheus metrics scraping
In Typhoon, we'll switch to using the https port which requires
Prometheus present a bearer token
* Go ahead and disable the insecure port for kube-scheduler too,
we'll configure Prometheus to scrape it with a bearer token as
well
* Remove unused kube-apiserver `--port` flag
Rel:
* https://github.com/kubernetes/kubernetes/pull/96216
* Add init container to auto-mount /sys/fs/cgroup cgroup2
at /run/cilium/cgroupv2 for the Cilium agent
* Enable CNI exclusive mode, to disable other configs
found in /etc/cni/net.d/
* https://github.com/cilium/cilium/pull/16259
* Set `kube-apiserver` `service-account-jwks-uri` because conformance
ServiceAccountIssuerDiscovery OIDC discovery will access a JWT endpoint
using the kube-apiserver's advertise address by default, instead of
using the intended in-cluster service (10.3.0.1) resolved by cluster DNS
`kubernetes.default.svc.cluster.local`, which causes a cert SAN error
* Set the authentication and authorization kubeconfig for kube-scheduler
and kube-controller-manager. Here, authn/z refer to aggregated API
use cases only, so its not strictly neccessary and warnings about
missing `extension-apiserver-authentication` when enable_aggregation
is false can be ignored
* Mount `/var/lib/kubelet/volumeplugins` to to the default location
expected within kube-controller-manager to remove the need for a flag
* Enable `tokencleaner` controller to automatically delete expired
bootstrap tokens (default node token is good 1 year, so cleanup won't
really matter at that point, but enable regardless)
* Remove unused `cloud-provider` flag, we never intend to use in-tree
cloud providers or support custom providers
