github-personal/kubernetes
1
0
Fork 0
mirror of https://github.com/outbackdingo/kubernetes.git synced 2026-02-28 22:00:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove cluster/gce/container-linux dir.

Browse Source
This commit is contained in:
Jing Ai
2018-01-16 14:12:39 -08:00
parent 305656c706 268555a30a
commit df8bbc611f
4277 changed files with 167990 additions and 55622 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
CHANGELOG-1.10.md Normal file
View File

@@ -0,0 +1,104 @@
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [v1.10.0-alpha.1](#v1100-alpha1)
- [Downloads for v1.10.0-alpha.1](#downloads-for-v1100-alpha1)
- [Client Binaries](#client-binaries)
- [Server Binaries](#server-binaries)
- [Node Binaries](#node-binaries)
- [Changelog since v1.9.0](#changelog-since-v190)
- [Action Required](#action-required)
- [Other notable changes](#other-notable-changes)
<!-- END MUNGE: GENERATED_TOC -->
<!-- NEW RELEASE NOTES ENTRY -->
# v1.10.0-alpha.1
[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/master/examples)
## Downloads for v1.10.0-alpha.1
filename | sha256 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes.tar.gz) | `403b90bfa32f7669b326045a629bd15941c533addcaf0c49d3c3c561da0542f2`
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-src.tar.gz) | `266da065e9eddf19d36df5ad325f2f854101a0e712766148e87d998e789b80cf`
### Client Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-darwin-386.tar.gz) | `5aaa8e294ae4060d34828239e37f37b45fa5a69508374be668965102848626be`
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | `40a8e3bab11b88a2bb8e748f0b29da806d89b55775508039abe9c38c5f4ab97d`
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-linux-386.tar.gz) | `e08dde0b561529f0b2bb39c141f4d7b1c943749ef7c1f9779facf5fb5b385d6a`
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | `76a05d31acaab932ef45c67e1d6c9273933b8bc06dd5ce9bad3c7345d5267702`
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | `4b833c9e80f3e4ac4958ea0ffb5ae564b31d2a524f6a14e58802937b2b936d73`
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | `f1484ab75010a2258ed7717b1284d0c139d17e194ac9e391b8f1c0999eec3c2d`
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | `da884f09ec753925b2c1f27ea0a1f6c3da2056855fc88f47929bb3d6c2a09312`
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | `c486f760c6707fc92d1659d3cbe33d68c03190760b73ac215957ee52f9c19195`
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-windows-386.tar.gz) | `514c550b7ff85ac33e6ed333bcc06461651fe4004d8b7c12ca67f5dc1d2198bf`
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | `ddad59222f6a8cb4e88c4330c2a967c4126cb22ac5e0d7126f9f65cca0fb9f45`
### Server Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | `514efd798ce1d7fe4233127f3334a3238faad6c26372a2d457eff02cbe72d756`
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | `f71f75fb96221f65891fc3e04fd52ae4e5628da8b7b4fbedece3fab4cb650afa`
[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-server-linux-arm.tar.gz) | `a9d8c2386813fd690e60623a6ee1968fe8f0a1a8e13bc5cc12b2caf8e8a862e1`
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | `21336a5e40aead4e2ec7e744a99d72bf8cb552341f3141abf8f235beb250cd93`
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | `257e44d38fef83f08990b6b9b5e985118e867c0c33f0e869f0900397b9d30498`
### Node Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | `97bf1210f0595ebf496ca7b000c4367f8a459d97ef72459efc6d0e07a072398f`
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | `eebcd3c14fb4faeb82ab047a2152db528adc2d9f7b20eef6f5dc58202ebe3124`
[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-node-linux-arm.tar.gz) | `3d4428416c775a0a6463f623286bd2ecdf9240ce901e1fbae180dfb564c53ea1`
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | `5cc96b24fad0ac1779a66f9b136d90e975b07bf619fea905e6c26ac5a4c41168`
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | `134c13338edf4efcd511f4161742fbaa6dc232965d3d926c3de435e8a080fcbb`
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.10.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | `ae54bf2bbcb99cdcde959140460d0f83c0ecb187d060b594ae9c5349960ab055`
## Changelog since v1.9.0
### Action Required
* [action required] Remove the kubelet's `--cloud-provider=auto-detect` feature ([#56287](https://github.com/kubernetes/kubernetes/pull/56287), [@stewart-yu](https://github.com/stewart-yu))
### Other notable changes
* Fix Heapster configuration and Metrics Server configuration to enable overriding default resource requirements. ([#56965](https://github.com/kubernetes/kubernetes/pull/56965), [@kawych](https://github.com/kawych))
* YAMLDecoder Read now returns the number of bytes read ([#57000](https://github.com/kubernetes/kubernetes/pull/57000), [@sel](https://github.com/sel))
* Retry 'connection refused' errors when setting up clusters on GCE. ([#57324](https://github.com/kubernetes/kubernetes/pull/57324), [@mborsz](https://github.com/mborsz))
* Update kubeadm's minimum supported Kubernetes version in v1.10.x to v1.9.0 ([#57233](https://github.com/kubernetes/kubernetes/pull/57233), [@xiangpengzhao](https://github.com/xiangpengzhao))
* Graduate CPU Manager feature from alpha to beta. ([#55977](https://github.com/kubernetes/kubernetes/pull/55977), [@ConnorDoyle](https://github.com/ConnorDoyle))
* Drop hacks used for Mesos integration that was already removed from main kubernetes repository ([#56754](https://github.com/kubernetes/kubernetes/pull/56754), [@dims](https://github.com/dims))
* Compare correct file names for volume detach operation ([#57053](https://github.com/kubernetes/kubernetes/pull/57053), [@prashima](https://github.com/prashima))
* Improved event generation in volume mount, attach, and extend operations ([#56872](https://github.com/kubernetes/kubernetes/pull/56872), [@davidz627](https://github.com/davidz627))
* GCE: bump COS image version to cos-stable-63-10032-71-0 ([#57204](https://github.com/kubernetes/kubernetes/pull/57204), [@yujuhong](https://github.com/yujuhong))
* fluentd-gcp updated to version 2.0.11. ([#56927](https://github.com/kubernetes/kubernetes/pull/56927), [@x13n](https://github.com/x13n))
* calico-node addon tolerates all NoExecute and NoSchedule taints by default. ([#57122](https://github.com/kubernetes/kubernetes/pull/57122), [@caseydavenport](https://github.com/caseydavenport))
* Support LoadBalancer for Azure Virtual Machine Scale Sets ([#57131](https://github.com/kubernetes/kubernetes/pull/57131), [@feiskyer](https://github.com/feiskyer))
* Makes the kube-dns addon optional so that users can deploy their own DNS solution. ([#57113](https://github.com/kubernetes/kubernetes/pull/57113), [@wwwtyro](https://github.com/wwwtyro))
* Enabled log rotation for load balancer's api logs to prevent running out of disk space. ([#56979](https://github.com/kubernetes/kubernetes/pull/56979), [@hyperbolic2346](https://github.com/hyperbolic2346))
* Remove ScrubDNS interface from cloudprovider. ([#56955](https://github.com/kubernetes/kubernetes/pull/56955), [@feiskyer](https://github.com/feiskyer))
* Fix `etcd-version-monitor` to backward compatibly support etcd 3.1 [go-grpc-prometheus](https://github.com/grpc-ecosystem/go-grpc-prometheus) metrics format. ([#56871](https://github.com/kubernetes/kubernetes/pull/56871), [@jpbetz](https://github.com/jpbetz))
* enable flexvolume on Windows node ([#56921](https://github.com/kubernetes/kubernetes/pull/56921), [@andyzhangx](https://github.com/andyzhangx))
* When using Role-Based Access Control, the "admin", "edit", and "view" roles now have the expected permissions on NetworkPolicy resources. ([#56650](https://github.com/kubernetes/kubernetes/pull/56650), [@danwinship](https://github.com/danwinship))
* Fix the PersistentVolumeLabel controller from initializing the PV labels when it's not the next pending initializer. ([#56831](https://github.com/kubernetes/kubernetes/pull/56831), [@jhorwit2](https://github.com/jhorwit2))
* kube-apiserver: The external hostname no longer longer use the cloud provider API to select a default. It can be set explicitly using --external-hostname, if needed. ([#56812](https://github.com/kubernetes/kubernetes/pull/56812), [@dims](https://github.com/dims))
* Use GiB unit for creating and resizing volumes for Glusterfs ([#56581](https://github.com/kubernetes/kubernetes/pull/56581), [@gnufied](https://github.com/gnufied))
* PersistentVolume flexVolume sources can now reference secrets in a namespace other than the PersistentVolumeClaim's namespace. ([#56460](https://github.com/kubernetes/kubernetes/pull/56460), [@liggitt](https://github.com/liggitt))
* Scheduler skips pods that use a PVC that either does not exist or is being deleted. ([#55957](https://github.com/kubernetes/kubernetes/pull/55957), [@jsafrane](https://github.com/jsafrane))
* Fixed a garbage collection race condition where objects with ownerRefs pointing to cluster-scoped objects could be deleted incorrectly. ([#57211](https://github.com/kubernetes/kubernetes/pull/57211), [@liggitt](https://github.com/liggitt))
* Kubectl explain now prints out the Kind and API version of the resource being explained ([#55689](https://github.com/kubernetes/kubernetes/pull/55689), [@luksa](https://github.com/luksa))
* api-server provides specific events when unable to repair a service cluster ip or node port ([#54304](https://github.com/kubernetes/kubernetes/pull/54304), [@frodenas](https://github.com/frodenas))
* Added docker-logins config to kubernetes-worker charm ([#56217](https://github.com/kubernetes/kubernetes/pull/56217), [@Cynerva](https://github.com/Cynerva))
* delete useless params containerized ([#56146](https://github.com/kubernetes/kubernetes/pull/56146), [@jiulongzaitian](https://github.com/jiulongzaitian))
* add mount options support for azure disk ([#56147](https://github.com/kubernetes/kubernetes/pull/56147), [@andyzhangx](https://github.com/andyzhangx))
* Use structured generator for kubectl autoscale ([#55913](https://github.com/kubernetes/kubernetes/pull/55913), [@wackxu](https://github.com/wackxu))
* K8s supports cephfs fuse mount. ([#55866](https://github.com/kubernetes/kubernetes/pull/55866), [@zhangxiaoyu-zidif](https://github.com/zhangxiaoyu-zidif))
* COS: Keep the docker network checkpoint ([#54805](https://github.com/kubernetes/kubernetes/pull/54805), [@yujuhong](https://github.com/yujuhong))
* Fixed documentation typo in IPVS README. ([#56578](https://github.com/kubernetes/kubernetes/pull/56578), [@shift](https://github.com/shift))

208
CHANGELOG-1.7.md
View File

@@ -1,86 +1,93 @@
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [v1.7.11](#v1711)
- [Downloads for v1.7.11](#downloads-for-v1711)
- [v1.7.12](#v1712)
- [Downloads for v1.7.12](#downloads-for-v1712)
- [Client Binaries](#client-binaries)
- [Server Binaries](#server-binaries)
- [Node Binaries](#node-binaries)
- [Changelog since v1.7.10](#changelog-since-v1710)
- [Changelog since v1.7.11](#changelog-since-v1711)
- [Other notable changes](#other-notable-changes)
- [v1.7.10](#v1710)
- [Downloads for v1.7.10](#downloads-for-v1710)
- [v1.7.11](#v1711)
- [Downloads for v1.7.11](#downloads-for-v1711)
- [Client Binaries](#client-binaries-1)
- [Server Binaries](#server-binaries-1)
- [Node Binaries](#node-binaries-1)
- [Changelog since v1.7.9](#changelog-since-v179)
- [Changelog since v1.7.10](#changelog-since-v1710)
- [Other notable changes](#other-notable-changes-1)
- [v1.7.9](#v179)
- [Downloads for v1.7.9](#downloads-for-v179)
- [v1.7.10](#v1710)
- [Downloads for v1.7.10](#downloads-for-v1710)
- [Client Binaries](#client-binaries-2)
- [Server Binaries](#server-binaries-2)
- [Node Binaries](#node-binaries-2)
- [Changelog since v1.7.8](#changelog-since-v178)
- [Changelog since v1.7.9](#changelog-since-v179)
- [Other notable changes](#other-notable-changes-2)
- [v1.7.8](#v178)
- [Downloads for v1.7.8](#downloads-for-v178)
- [v1.7.9](#v179)
- [Downloads for v1.7.9](#downloads-for-v179)
- [Client Binaries](#client-binaries-3)
- [Server Binaries](#server-binaries-3)
- [Node Binaries](#node-binaries-3)
- [Changelog since v1.7.7](#changelog-since-v177)
- [Changelog since v1.7.8](#changelog-since-v178)
- [Other notable changes](#other-notable-changes-3)
- [v1.7.7](#v177)
- [Downloads for v1.7.7](#downloads-for-v177)
- [v1.7.8](#v178)
- [Downloads for v1.7.8](#downloads-for-v178)
- [Client Binaries](#client-binaries-4)
- [Server Binaries](#server-binaries-4)
- [Node Binaries](#node-binaries-4)
- [Changelog since v1.7.6](#changelog-since-v176)
- [Changelog since v1.7.7](#changelog-since-v177)
- [Other notable changes](#other-notable-changes-4)
- [v1.7.6](#v176)
- [Downloads for v1.7.6](#downloads-for-v176)
- [v1.7.7](#v177)
- [Downloads for v1.7.7](#downloads-for-v177)
- [Client Binaries](#client-binaries-5)
- [Server Binaries](#server-binaries-5)
- [Node Binaries](#node-binaries-5)
- [Changelog since v1.7.5](#changelog-since-v175)
- [Changelog since v1.7.6](#changelog-since-v176)
- [Other notable changes](#other-notable-changes-5)
- [v1.7.5](#v175)
- [Downloads for v1.7.5](#downloads-for-v175)
- [v1.7.6](#v176)
- [Downloads for v1.7.6](#downloads-for-v176)
- [Client Binaries](#client-binaries-6)
- [Server Binaries](#server-binaries-6)
- [Node Binaries](#node-binaries-6)
- [Changelog since v1.7.4](#changelog-since-v174)
- [Changelog since v1.7.5](#changelog-since-v175)
- [Other notable changes](#other-notable-changes-6)
- [v1.7.4](#v174)
- [Downloads for v1.7.4](#downloads-for-v174)
- [v1.7.5](#v175)
- [Downloads for v1.7.5](#downloads-for-v175)
- [Client Binaries](#client-binaries-7)
- [Server Binaries](#server-binaries-7)
- [Node Binaries](#node-binaries-7)
- [Changelog since v1.7.3](#changelog-since-v173)
- [Changelog since v1.7.4](#changelog-since-v174)
- [Other notable changes](#other-notable-changes-7)
- [v1.7.3](#v173)
- [Downloads for v1.7.3](#downloads-for-v173)
- [v1.7.4](#v174)
- [Downloads for v1.7.4](#downloads-for-v174)
- [Client Binaries](#client-binaries-8)
- [Server Binaries](#server-binaries-8)
- [Node Binaries](#node-binaries-8)
- [Changelog since v1.7.2](#changelog-since-v172)
- [Changelog since v1.7.3](#changelog-since-v173)
- [Other notable changes](#other-notable-changes-8)
- [v1.7.2](#v172)
- [Downloads for v1.7.2](#downloads-for-v172)
- [v1.7.3](#v173)
- [Downloads for v1.7.3](#downloads-for-v173)
- [Client Binaries](#client-binaries-9)
- [Server Binaries](#server-binaries-9)
- [Node Binaries](#node-binaries-9)
- [Changelog since v1.7.1](#changelog-since-v171)
- [Changelog since v1.7.2](#changelog-since-v172)
- [Other notable changes](#other-notable-changes-9)
- [v1.7.1](#v171)
- [Downloads for v1.7.1](#downloads-for-v171)
- [v1.7.2](#v172)
- [Downloads for v1.7.2](#downloads-for-v172)
- [Client Binaries](#client-binaries-10)
- [Server Binaries](#server-binaries-10)
- [Node Binaries](#node-binaries-10)
- [Changelog since v1.7.0](#changelog-since-v170)
- [Changelog since v1.7.1](#changelog-since-v171)
- [Other notable changes](#other-notable-changes-10)
- [v1.7.0](#v170)
- [Downloads for v1.7.0](#downloads-for-v170)
- [v1.7.1](#v171)
- [Downloads for v1.7.1](#downloads-for-v171)
- [Client Binaries](#client-binaries-11)
- [Server Binaries](#server-binaries-11)
- [Node Binaries](#node-binaries-11)
- [Changelog since v1.7.0](#changelog-since-v170)
- [Other notable changes](#other-notable-changes-11)
- [v1.7.0](#v170)
- [Downloads for v1.7.0](#downloads-for-v170)
- [Client Binaries](#client-binaries-12)
- [Server Binaries](#server-binaries-12)
- [Node Binaries](#node-binaries-12)
- [**Major Themes**](#major-themes)
- [**Action Required Before Upgrading**](#action-required-before-upgrading)
- [Network](#network)
@@ -136,7 +143,7 @@
- [Local Storage](#local-storage)
- [Volume Plugins](#volume-plugins)
- [Metrics](#metrics)
- [**Other notable changes**](#other-notable-changes-11)
- [**Other notable changes**](#other-notable-changes-12)
- [Admission plugin](#admission-plugin)
- [API Machinery](#api-machinery-1)
- [Application autoscaling](#application-autoscaling-1)
@@ -164,62 +171,133 @@
- [Previous Releases Included in v1.7.0](#previous-releases-included-in-v170)
- [v1.7.0-rc.1](#v170-rc1)
- [Downloads for v1.7.0-rc.1](#downloads-for-v170-rc1)
- [Client Binaries](#client-binaries-12)
- [Server Binaries](#server-binaries-12)
- [Node Binaries](#node-binaries-12)
- [Changelog since v1.7.0-beta.2](#changelog-since-v170-beta2)
- [Action Required](#action-required)
- [Other notable changes](#other-notable-changes-12)
- [v1.7.0-beta.2](#v170-beta2)
- [Downloads for v1.7.0-beta.2](#downloads-for-v170-beta2)
- [Client Binaries](#client-binaries-13)
- [Server Binaries](#server-binaries-13)
- [Node Binaries](#node-binaries-13)
- [Changelog since v1.7.0-beta.1](#changelog-since-v170-beta1)
- [Action Required](#action-required-1)
- [Changelog since v1.7.0-beta.2](#changelog-since-v170-beta2)
- [Action Required](#action-required)
- [Other notable changes](#other-notable-changes-13)
- [v1.7.0-beta.1](#v170-beta1)
- [Downloads for v1.7.0-beta.1](#downloads-for-v170-beta1)
- [v1.7.0-beta.2](#v170-beta2)
- [Downloads for v1.7.0-beta.2](#downloads-for-v170-beta2)
- [Client Binaries](#client-binaries-14)
- [Server Binaries](#server-binaries-14)
- [Node Binaries](#node-binaries-14)
- [Changelog since v1.7.0-alpha.4](#changelog-since-v170-alpha4)
- [Action Required](#action-required-2)
- [Changelog since v1.7.0-beta.1](#changelog-since-v170-beta1)
- [Action Required](#action-required-1)
- [Other notable changes](#other-notable-changes-14)
- [v1.7.0-alpha.4](#v170-alpha4)
- [Downloads for v1.7.0-alpha.4](#downloads-for-v170-alpha4)
- [v1.7.0-beta.1](#v170-beta1)
- [Downloads for v1.7.0-beta.1](#downloads-for-v170-beta1)
- [Client Binaries](#client-binaries-15)
- [Server Binaries](#server-binaries-15)
- [Node Binaries](#node-binaries-15)
- [Changelog since v1.7.0-alpha.3](#changelog-since-v170-alpha3)
- [Action Required](#action-required-3)
- [Changelog since v1.7.0-alpha.4](#changelog-since-v170-alpha4)
- [Action Required](#action-required-2)
- [Other notable changes](#other-notable-changes-15)
- [v1.7.0-alpha.3](#v170-alpha3)
- [Downloads for v1.7.0-alpha.3](#downloads-for-v170-alpha3)
- [v1.7.0-alpha.4](#v170-alpha4)
- [Downloads for v1.7.0-alpha.4](#downloads-for-v170-alpha4)
- [Client Binaries](#client-binaries-16)
- [Server Binaries](#server-binaries-16)
- [Node Binaries](#node-binaries-16)
- [Changelog since v1.7.0-alpha.2](#changelog-since-v170-alpha2)
- [Action Required](#action-required-4)
- [Changelog since v1.7.0-alpha.3](#changelog-since-v170-alpha3)
- [Action Required](#action-required-3)
- [Other notable changes](#other-notable-changes-16)
- [v1.7.0-alpha.2](#v170-alpha2)
- [Downloads for v1.7.0-alpha.2](#downloads-for-v170-alpha2)
- [v1.7.0-alpha.3](#v170-alpha3)
- [Downloads for v1.7.0-alpha.3](#downloads-for-v170-alpha3)
- [Client Binaries](#client-binaries-17)
- [Server Binaries](#server-binaries-17)
- [Changelog since v1.7.0-alpha.1](#changelog-since-v170-alpha1)
- [Action Required](#action-required-5)
- [Node Binaries](#node-binaries-17)
- [Changelog since v1.7.0-alpha.2](#changelog-since-v170-alpha2)
- [Action Required](#action-required-4)
- [Other notable changes](#other-notable-changes-17)
- [v1.7.0-alpha.1](#v170-alpha1)
- [Downloads for v1.7.0-alpha.1](#downloads-for-v170-alpha1)
- [v1.7.0-alpha.2](#v170-alpha2)
- [Downloads for v1.7.0-alpha.2](#downloads-for-v170-alpha2)
- [Client Binaries](#client-binaries-18)
- [Server Binaries](#server-binaries-18)
- [Changelog since v1.6.0](#changelog-since-v160)
- [Changelog since v1.7.0-alpha.1](#changelog-since-v170-alpha1)
- [Action Required](#action-required-5)
- [Other notable changes](#other-notable-changes-18)
- [v1.7.0-alpha.1](#v170-alpha1)
- [Downloads for v1.7.0-alpha.1](#downloads-for-v170-alpha1)
- [Client Binaries](#client-binaries-19)
- [Server Binaries](#server-binaries-19)
- [Changelog since v1.6.0](#changelog-since-v160)
- [Other notable changes](#other-notable-changes-19)
<!-- END MUNGE: GENERATED_TOC -->
<!-- NEW RELEASE NOTES ENTRY -->
# v1.7.12
[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.7/examples)
## Downloads for v1.7.12
filename | sha256 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes.tar.gz) | `749f811fb77daca197ecce2eacfea13f28e9fa69748d1b9fa7521850a5e77b93`
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-src.tar.gz) | `86804d5a20a929429f1a8ed4aecba78d391a0dbaee7ffca914724b37e56eeebe`
### Client Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-darwin-386.tar.gz) | `7fa3e25fa63a31955de12f1cfa67bb94bcc09ccd3e90e5c5ad090b2ea9d90f94`
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-darwin-amd64.tar.gz) | `107fa0f038b3530f57a6b04512262cbde04c888b771a1b931c6ff0a98adc1bc9`
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-linux-386.tar.gz) | `22827bee712441a57dfa2c6d87182128c82a0f0ded34970910d1aebdb968d4db`
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-linux-amd64.tar.gz) | `01e87c03e4c928a105ac64618a8923d9d5afa321f9ce2c4d739dad5aa564da72`
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-linux-arm64.tar.gz) | `5d44328b0f2070885102fd15e9bb142d53b8b0c431cc5bfc5018fe07642c0380`
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-linux-arm.tar.gz) | `30986808b540706a88855e87bd997103b506635dcc62b02e34e6d6ac507301ef`
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-linux-ppc64le.tar.gz) | `d577a244e0f09f47d926fbcbd097e149a53488406952089225545f591f2c1945`
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-linux-s390x.tar.gz) | `2f5eab8cb47eb467727649ef2683abe72232f9b6f481384244c535507d15a3d7`
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-windows-386.tar.gz) | `e0c060c5fa1fa61ff6477485fb40329d57e6dd20cc6a1bbc50a5f98f54f61d1a`
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-client-windows-amd64.tar.gz) | `bc824cf320dc94a96998665fad5925fb1b6c66569aa9bb34b12e7dfa7d437c73`
### Server Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-server-linux-amd64.tar.gz) | `2bf0fee82996eaae55547852c5082ecbc2389356b4c929294ed3bc198f80ec33`
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-server-linux-arm64.tar.gz) | `b7b193a53650bac279fed535fa6e5a0cb4cff6376731ef4ca3a383af97b94486`
[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-server-linux-arm.tar.gz) | `ecee8f65c62f4a79c423b585bf0f78e3c64ed4bb1afc7a87f0ac6dfcfb262908`
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-server-linux-ppc64le.tar.gz) | `eb9058d726fd48eb6797e99ba2d9353ab2bae4dec21836deaafb2ded0b412acc`
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-server-linux-s390x.tar.gz) | `b6eb522fb1aac7ea82ae2d04b456e4e69740ce40dd48eb205c5d071f4aa49d76`
### Node Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-node-linux-amd64.tar.gz) | `1ab49460eb34ebab60a9109479e2f43194c763ae24a1922889e301d8c1b0644e`
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-node-linux-arm64.tar.gz) | `16bf9e50d74d8b66e791ee9d23498e7b4a6e49f499df02f84baaf277128da9c2`
[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-node-linux-arm.tar.gz) | `c64fe4901f94076f6df2d464e13799f6399f68bc439ad966357ea3790e73a22e`
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-node-linux-ppc64le.tar.gz) | `4c641014245741fd0835e430c6cc61bae0c1f30526ec07313343d59eee462a01`
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-node-linux-s390x.tar.gz) | `9262f3821d02ac6a6d3d5fe51fc56cb264e2bf1adaa4b63b8b87612f1e01411d`
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.7.12/kubernetes-node-windows-amd64.tar.gz) | `266b57c417190621ee9583fa556336dfe447ce8847f8be64d383fa48a81b22e2`
## Changelog since v1.7.11
### Other notable changes
* fix azure disk storage account init issue ([#55927](https://github.com/kubernetes/kubernetes/pull/55927), [@andyzhangx](https://github.com/andyzhangx))
* Fixes a bug where if an error was returned that was not an `autorest.DetailedError` we would return `"not found", nil` which caused nodes to go to `NotReady` state. ([#57484](https://github.com/kubernetes/kubernetes/pull/57484), [@brendandburns](https://github.com/brendandburns))
* Retry 'connection refused' errors when setting up clusters on GCE. ([#57394](https://github.com/kubernetes/kubernetes/pull/57394), [@mborsz](https://github.com/mborsz))
* Retry 'connection refused' errors when setting up clusters on GCE. ([#57394](https://github.com/kubernetes/kubernetes/pull/57394), [@mborsz](https://github.com/mborsz))
* Retry 'connection refused' errors when setting up clusters on GCE. ([#57394](https://github.com/kubernetes/kubernetes/pull/57394), [@mborsz](https://github.com/mborsz))
* Fix a problem of not respecting TerminationGracePeriodSeconds of the Pods created by DaemonSet controller. ([#51279](https://github.com/kubernetes/kubernetes/pull/51279), [@kow3ns](https://github.com/kow3ns))
* BUG FIX: Check both name and ports for azure health probes ([#56918](https://github.com/kubernetes/kubernetes/pull/56918), [@feiskyer](https://github.com/feiskyer))
* Provides compatibility of fields SizeLimit in types.EmptyDirVolumeSource since v1.7.8 ([#56505](https://github.com/kubernetes/kubernetes/pull/56505), [@yue9944882](https://github.com/yue9944882))
* Fixes issue where masquerade rules are flushed in GCE k8s clusters. ([#56728](https://github.com/kubernetes/kubernetes/pull/56728), [@dnardo](https://github.com/dnardo))
* kubelet: fix bug where `runAsUser: MustRunAsNonRoot` strategy didn't reject a pod with a non-numeric `USER`. ([#56711](https://github.com/kubernetes/kubernetes/pull/56711), [@php-coder](https://github.com/php-coder))
* Fix a bug in GCE multizonal clusters where PersistentVolumes were sometimes created in zones without nodes. ([#52322](https://github.com/kubernetes/kubernetes/pull/52322), [@davidz627](https://github.com/davidz627))
* Fix validation of NetworkPolicy ([#56223](https://github.com/kubernetes/kubernetes/pull/56223), [@deads2k](https://github.com/deads2k))
* add GRS, RAGRS storage account type support for azure disk ([#55931](https://github.com/kubernetes/kubernetes/pull/55931), [@andyzhangx](https://github.com/andyzhangx))
* Fixes server name verification of aggregated API servers and webhook admission endpoints ([#56415](https://github.com/kubernetes/kubernetes/pull/56415), [@liggitt](https://github.com/liggitt))
* Fix a typo in prometheus-to-sd configuration, that drops some stackdriver metrics. ([#56473](https://github.com/kubernetes/kubernetes/pull/56473), [@loburm](https://github.com/loburm))
* Update jquery and bootstrap dependencies ([#56447](https://github.com/kubernetes/kubernetes/pull/56447), [@dashpole](https://github.com/dashpole))
# v1.7.11
[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.7/examples)

150
CHANGELOG-1.8.md
View File

@@ -1,45 +1,52 @@
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [v1.8.5](#v185)
- [Downloads for v1.8.5](#downloads-for-v185)
- [v1.8.6](#v186)
- [Downloads for v1.8.6](#downloads-for-v186)
- [Client Binaries](#client-binaries)
- [Server Binaries](#server-binaries)
- [Node Binaries](#node-binaries)
- [Changelog since v1.8.4](#changelog-since-v184)
- [Changelog since v1.8.5](#changelog-since-v185)
- [Other notable changes](#other-notable-changes)
- [v1.8.4](#v184)
- [Downloads for v1.8.4](#downloads-for-v184)
- [v1.8.5](#v185)
- [Downloads for v1.8.5](#downloads-for-v185)
- [Client Binaries](#client-binaries-1)
- [Server Binaries](#server-binaries-1)
- [Node Binaries](#node-binaries-1)
- [Changelog since v1.8.3](#changelog-since-v183)
- [Changelog since v1.8.4](#changelog-since-v184)
- [Other notable changes](#other-notable-changes-1)
- [v1.8.3](#v183)
- [Downloads for v1.8.3](#downloads-for-v183)
- [v1.8.4](#v184)
- [Downloads for v1.8.4](#downloads-for-v184)
- [Client Binaries](#client-binaries-2)
- [Server Binaries](#server-binaries-2)
- [Node Binaries](#node-binaries-2)
- [Changelog since v1.8.2](#changelog-since-v182)
- [Changelog since v1.8.3](#changelog-since-v183)
- [Other notable changes](#other-notable-changes-2)
- [v1.8.2](#v182)
- [Downloads for v1.8.2](#downloads-for-v182)
- [v1.8.3](#v183)
- [Downloads for v1.8.3](#downloads-for-v183)
- [Client Binaries](#client-binaries-3)
- [Server Binaries](#server-binaries-3)
- [Node Binaries](#node-binaries-3)
- [Changelog since v1.8.1](#changelog-since-v181)
- [Changelog since v1.8.2](#changelog-since-v182)
- [Other notable changes](#other-notable-changes-3)
- [v1.8.1](#v181)
- [Downloads for v1.8.1](#downloads-for-v181)
- [v1.8.2](#v182)
- [Downloads for v1.8.2](#downloads-for-v182)
- [Client Binaries](#client-binaries-4)
- [Server Binaries](#server-binaries-4)
- [Node Binaries](#node-binaries-4)
- [Changelog since v1.8.0](#changelog-since-v180)
- [Action Required](#action-required)
- [Changelog since v1.8.1](#changelog-since-v181)
- [Other notable changes](#other-notable-changes-4)
- [v1.8.0](#v180)
- [Downloads for v1.8.0](#downloads-for-v180)
- [v1.8.1](#v181)
- [Downloads for v1.8.1](#downloads-for-v181)
- [Client Binaries](#client-binaries-5)
- [Server Binaries](#server-binaries-5)
- [Node Binaries](#node-binaries-5)
- [Changelog since v1.8.0](#changelog-since-v180)
- [Action Required](#action-required)
- [Other notable changes](#other-notable-changes-5)
- [v1.8.0](#v180)
- [Downloads for v1.8.0](#downloads-for-v180)
- [Client Binaries](#client-binaries-6)
- [Server Binaries](#server-binaries-6)
- [Node Binaries](#node-binaries-6)
- [Introduction to v1.8.0](#introduction-to-v180)
- [Major Themes](#major-themes)
- [SIG API Machinery](#sig-api-machinery)
@@ -100,49 +107,114 @@
- [External Dependencies](#external-dependencies)
- [v1.8.0-rc.1](#v180-rc1)
- [Downloads for v1.8.0-rc.1](#downloads-for-v180-rc1)
- [Client Binaries](#client-binaries-6)
- [Server Binaries](#server-binaries-6)
- [Node Binaries](#node-binaries-6)
- [Changelog since v1.8.0-beta.1](#changelog-since-v180-beta1)
- [Action Required](#action-required-1)
- [Other notable changes](#other-notable-changes-5)
- [v1.8.0-beta.1](#v180-beta1)
- [Downloads for v1.8.0-beta.1](#downloads-for-v180-beta1)
- [Client Binaries](#client-binaries-7)
- [Server Binaries](#server-binaries-7)
- [Node Binaries](#node-binaries-7)
- [Changelog since v1.8.0-alpha.3](#changelog-since-v180-alpha3)
- [Action Required](#action-required-2)
- [Changelog since v1.8.0-beta.1](#changelog-since-v180-beta1)
- [Action Required](#action-required-1)
- [Other notable changes](#other-notable-changes-6)
- [v1.8.0-alpha.3](#v180-alpha3)
- [Downloads for v1.8.0-alpha.3](#downloads-for-v180-alpha3)
- [v1.8.0-beta.1](#v180-beta1)
- [Downloads for v1.8.0-beta.1](#downloads-for-v180-beta1)
- [Client Binaries](#client-binaries-8)
- [Server Binaries](#server-binaries-8)
- [Node Binaries](#node-binaries-8)
- [Changelog since v1.8.0-alpha.2](#changelog-since-v180-alpha2)
- [Action Required](#action-required-3)
- [Changelog since v1.8.0-alpha.3](#changelog-since-v180-alpha3)
- [Action Required](#action-required-2)
- [Other notable changes](#other-notable-changes-7)
- [v1.8.0-alpha.2](#v180-alpha2)
- [Downloads for v1.8.0-alpha.2](#downloads-for-v180-alpha2)
- [v1.8.0-alpha.3](#v180-alpha3)
- [Downloads for v1.8.0-alpha.3](#downloads-for-v180-alpha3)
- [Client Binaries](#client-binaries-9)
- [Server Binaries](#server-binaries-9)
- [Node Binaries](#node-binaries-9)
- [Changelog since v1.7.0](#changelog-since-v170)
- [Action Required](#action-required-4)
- [Changelog since v1.8.0-alpha.2](#changelog-since-v180-alpha2)
- [Action Required](#action-required-3)
- [Other notable changes](#other-notable-changes-8)
- [v1.8.0-alpha.1](#v180-alpha1)
- [Downloads for v1.8.0-alpha.1](#downloads-for-v180-alpha1)
- [v1.8.0-alpha.2](#v180-alpha2)
- [Downloads for v1.8.0-alpha.2](#downloads-for-v180-alpha2)
- [Client Binaries](#client-binaries-10)
- [Server Binaries](#server-binaries-10)
- [Node Binaries](#node-binaries-10)
- [Changelog since v1.7.0](#changelog-since-v170)
- [Action Required](#action-required-4)
- [Other notable changes](#other-notable-changes-9)
- [v1.8.0-alpha.1](#v180-alpha1)
- [Downloads for v1.8.0-alpha.1](#downloads-for-v180-alpha1)
- [Client Binaries](#client-binaries-11)
- [Server Binaries](#server-binaries-11)
- [Node Binaries](#node-binaries-11)
- [Changelog since v1.7.0-alpha.4](#changelog-since-v170-alpha4)
- [Action Required](#action-required-5)
- [Other notable changes](#other-notable-changes-9)
- [Other notable changes](#other-notable-changes-10)
<!-- END MUNGE: GENERATED_TOC -->
<!-- NEW RELEASE NOTES ENTRY -->
# v1.8.6
[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.8/examples)
## Downloads for v1.8.6
filename | sha256 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes.tar.gz) | `8289c42b5d6da1dbf910585fca3a9d909195e540cc81bace61ec1d06b2366c1b`
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-src.tar.gz) | `8a9d5d890c44137527fe3976d71d4f7cb18db21ba34262ce587cd979a88bb2fe`
### Client Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-darwin-386.tar.gz) | `0e282477bfed6b534f2fbbd125e6e3e065bf72d15ac3532acef405e6717d8fb7`
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-darwin-amd64.tar.gz) | `767c7bfbc6c1d01120e11726b9e33e184d32294e07c69a299b229329c5b98eba`
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-linux-386.tar.gz) | `088b40c343fecb83b514bf9af0ad1c359c98ae7aa3b62d2a078c1363f50901c9`
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-linux-amd64.tar.gz) | `47541706e4d27da55d32372344d7a4038ed389ba0be1e6fe15c651c574aac97a`
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-linux-arm64.tar.gz) | `4be0b7a01c28c1f85d4f01f86def03dd3d49ef88cb43bf7be641d9d16b6aabc2`
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-linux-arm.tar.gz) | `2d70384262cbdfb0958542bc5a71d926c49557fc8cc3000a2592571a945ad119`
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-linux-ppc64le.tar.gz) | `c3be3a125ac77aa809da3495ad38456059a89cccfdfad0babaf95896fb958adc`
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-linux-s390x.tar.gz) | `2b9831c2dd65c9669b335e3623e6a7001173b9ddf203f52f37b350659d9f1102`
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-windows-386.tar.gz) | `9d14a96372cdcecbbb28717aff305fcd68beb540066a27f1b5e84e208a25405f`
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-client-windows-amd64.tar.gz) | `0fbe358ff305188fe00793284e22c9c5b2ec0e0213882f0bfe0e4bf9685075f0`
### Server Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-server-linux-amd64.tar.gz) | `9c8ff48343e5314638965407358d1e91d510c72a1c7dd7cde0c3be12790fdb98`
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-server-linux-arm64.tar.gz) | `dd35c1b7572ab383eb2ff60f3b039053afa124836db6d044ab14afdafbe5ca74`
[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-server-linux-arm.tar.gz) | `5f4637d309eb47f4f97db8d2978b0b37b271339feb5952b216a9d09ad7e67c32`
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-server-linux-ppc64le.tar.gz) | `6d3ea43edd53253e9e3b9ceb49e61b6d2c093e55be35f7b1a8f798cde842a562`
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-server-linux-s390x.tar.gz) | `dfe89b91399977cee291d57b446625f01cf76ebecce696e2e889863bd3c8d3b1`
### Node Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-node-linux-amd64.tar.gz) | `f8f3e7bb07db540f4b88fa5818c46efb918e795e5e89e389b9048f2f7f37674d`
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-node-linux-arm64.tar.gz) | `1754b8a20d9176317fea3b77b5c48ad5565b922820adcbca4017bf210168dc6e`
[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-node-linux-arm.tar.gz) | `0a8255effff1d5b3ad7c84c3d6f6b8cfb5beb71606bfedaef0bb45f170b806d6`
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-node-linux-ppc64le.tar.gz) | `fef465c9f66eda35479e152619b6c91e2432e92736646a898c5917098a10a1b4`
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-node-linux-s390x.tar.gz) | `ff024e59d52afdee003f11c65f7de428915f7e28f9b8be4b3ebf117422ae5d67`
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.8.6/kubernetes-node-windows-amd64.tar.gz) | `19a673b714c02322c544ec3a972e011410b69a7aed016ecf7ba09eccb175a1de`
## Changelog since v1.8.5
### Other notable changes
* change default azure file/dir mode to 0755 ([#56551](https://github.com/kubernetes/kubernetes/pull/56551), [@andyzhangx](https://github.com/andyzhangx))
* Retry 'connection refused' errors when setting up clusters on GCE. ([#57394](https://github.com/kubernetes/kubernetes/pull/57394), [@mborsz](https://github.com/mborsz))
* enable flexvolume on Windows node ([#56921](https://github.com/kubernetes/kubernetes/pull/56921), [@andyzhangx](https://github.com/andyzhangx))
* Add prometheus metrics for the PodSecurityPolicy admission controller ([#57346](https://github.com/kubernetes/kubernetes/pull/57346), [@tallclair](https://github.com/tallclair))
* fix CreateVolume func: use search mode instead ([#54687](https://github.com/kubernetes/kubernetes/pull/54687), [@andyzhangx](https://github.com/andyzhangx))
* remove time waiting after create storage account (save 25s) ([#56679](https://github.com/kubernetes/kubernetes/pull/56679), [@andyzhangx](https://github.com/andyzhangx))
* Add pvc as part of equivalence hash ([#56577](https://github.com/kubernetes/kubernetes/pull/56577), [@resouer](https://github.com/resouer))
* fix azure disk storage account init issue ([#55927](https://github.com/kubernetes/kubernetes/pull/55927), [@andyzhangx](https://github.com/andyzhangx))
* falls back to parse Docker runtime version as generic if not semver ([#54040](https://github.com/kubernetes/kubernetes/pull/54040), [@dixudx](https://github.com/dixudx))
* BUG FIX: Check both name and ports for azure health probes ([#56918](https://github.com/kubernetes/kubernetes/pull/56918), [@feiskyer](https://github.com/feiskyer))
# v1.8.5
[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.8/examples)

911
CHANGELOG-1.9.md
View File

@@ -1,48 +1,917 @@
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [v1.9.0-beta.2](#v190-beta2)
- [Downloads for v1.9.0-beta.2](#downloads-for-v190-beta2)
- [v1.9.1](#v191)
- [Downloads for v1.9.1](#downloads-for-v191)
- [Client Binaries](#client-binaries)
- [Server Binaries](#server-binaries)
- [Node Binaries](#node-binaries)
- [Changelog since v1.9.0-beta.1](#changelog-since-v190-beta1)
- [Changelog since v1.9.0](#changelog-since-v190)
- [Other notable changes](#other-notable-changes)
- [v1.9.0-beta.1](#v190-beta1)
- [Downloads for v1.9.0-beta.1](#downloads-for-v190-beta1)
- [v1.9.0](#v190)
- [Downloads for v1.9.0](#downloads-for-v190)
- [Client Binaries](#client-binaries-1)
- [Server Binaries](#server-binaries-1)
- [Node Binaries](#node-binaries-1)
- [Changelog since v1.9.0-alpha.3](#changelog-since-v190-alpha3)
- [Action Required](#action-required)
- [Other notable changes](#other-notable-changes-1)
- [v1.9.0-alpha.3](#v190-alpha3)
- [Downloads for v1.9.0-alpha.3](#downloads-for-v190-alpha3)
- [1.9 Release Notes](#19-release-notes)
- [WARNING: etcd backup strongly recommended](#warning-etcd-backup-strongly-recommended)
- [Introduction to 1.9.0](#introduction-to-190)
- [Major themes](#major-themes)
- [API Machinery](#api-machinery)
- [Apps](#apps)
- [Auth](#auth)
- [AWS](#aws)
- [Azure](#azure)
- [Cluster Lifecycle](#cluster-lifecycle)
- [Instrumentation](#instrumentation)
- [Network](#network)
- [Node](#node)
- [OpenStack](#openstack)
- [Storage](#storage)
- [Windows](#windows)
- [Before Upgrading](#before-upgrading)
- [**API Machinery**](#api-machinery-1)
- [**Auth**](#auth-1)
- [**CLI**](#cli)
- [**Cluster Lifecycle**](#cluster-lifecycle-1)
- [**Multicluster**](#multicluster)
- [**Node**](#node-1)
- [**Network**](#network-1)
- [**Scheduling**](#scheduling)
- [**Storage**](#storage-1)
- [**OpenStack**](#openstack-1)
- [Known Issues](#known-issues)
- [Deprecations](#deprecations)
- [**API Machinery**](#api-machinery-2)
- [**Auth**](#auth-2)
- [**Cluster Lifecycle**](#cluster-lifecycle-2)
- [**Network**](#network-2)
- [**Storage**](#storage-2)
- [**Scheduling**](#scheduling-1)
- [**Node**](#node-2)
- [Notable Changes](#notable-changes)
- [**Workloads API (apps/v1)**](#workloads-api-appsv1)
- [**API Machinery**](#api-machinery-3)
- [**Admission Control**](#admission-control)
- [**API & API server**](#api-&-api-server)
- [**Audit**](#audit)
- [**Custom Resources**](#custom-resources)
- [**Other**](#other)
- [**Apps**](#apps-1)
- [**Auth**](#auth-3)
- [**Audit**](#audit-1)
- [**RBAC**](#rbac)
- [**Other**](#other-1)
- [**GCE**](#gce)
- [**Autoscaling**](#autoscaling)
- [**AWS**](#aws-1)
- [**Azure**](#azure-1)
- [**CLI**](#cli-1)
- [**Kubectl**](#kubectl)
- [**Cluster Lifecycle**](#cluster-lifecycle-3)
- [**API Server**](#api-server)
- [**Cloud Provider Integration**](#cloud-provider-integration)
- [**Kubeadm**](#kubeadm)
- [**Juju**](#juju)
- [**Other**](#other-2)
- [**GCP**](#gcp)
- [**Instrumentation**](#instrumentation-1)
- [**Audit**](#audit-2)
- [**Other**](#other-3)
- [**Multicluster**](#multicluster-1)
- [**Federation**](#federation)
- [**Network**](#network-3)
- [**IPv6**](#ipv6)
- [**IPVS**](#ipvs)
- [**Kube-Proxy**](#kube-proxy)
- [**CoreDNS**](#coredns)
- [**Other**](#other-4)
- [**Node**](#node-3)
- [**Pod API**](#pod-api)
- [**Hardware Accelerators**](#hardware-accelerators)
- [**Container Runtime**](#container-runtime)
- [**Kubelet**](#kubelet)
- [**Other**](#other-5)
- [**OpenStack**](#openstack-2)
- [**Scheduling**](#scheduling-2)
- [**Hardware Accelerators**](#hardware-accelerators-1)
- [**Other**](#other-6)
- [**Storage**](#storage-3)
- [External Dependencies](#external-dependencies)
- [v1.9.0-beta.2](#v190-beta2)
- [Downloads for v1.9.0-beta.2](#downloads-for-v190-beta2)
- [Client Binaries](#client-binaries-2)
- [Server Binaries](#server-binaries-2)
- [Node Binaries](#node-binaries-2)
- [Changelog since v1.9.0-alpha.2](#changelog-since-v190-alpha2)
- [Action Required](#action-required-1)
- [Other notable changes](#other-notable-changes-2)
- [v1.9.0-alpha.2](#v190-alpha2)
- [Downloads for v1.9.0-alpha.2](#downloads-for-v190-alpha2)
- [Changelog since v1.9.0-beta.1](#changelog-since-v190-beta1)
- [Other notable changes](#other-notable-changes-1)
- [v1.9.0-beta.1](#v190-beta1)
- [Downloads for v1.9.0-beta.1](#downloads-for-v190-beta1)
- [Client Binaries](#client-binaries-3)
- [Server Binaries](#server-binaries-3)
- [Node Binaries](#node-binaries-3)
- [Changelog since v1.8.0](#changelog-since-v180)
- [Action Required](#action-required-2)
- [Other notable changes](#other-notable-changes-3)
- [v1.9.0-alpha.1](#v190-alpha1)
- [Downloads for v1.9.0-alpha.1](#downloads-for-v190-alpha1)
- [Changelog since v1.9.0-alpha.3](#changelog-since-v190-alpha3)
- [Action Required](#action-required)
- [Other notable changes](#other-notable-changes-2)
- [v1.9.0-alpha.3](#v190-alpha3)
- [Downloads for v1.9.0-alpha.3](#downloads-for-v190-alpha3)
- [Client Binaries](#client-binaries-4)
- [Server Binaries](#server-binaries-4)
- [Node Binaries](#node-binaries-4)
- [Changelog since v1.9.0-alpha.2](#changelog-since-v190-alpha2)
- [Action Required](#action-required-1)
- [Other notable changes](#other-notable-changes-3)
- [v1.9.0-alpha.2](#v190-alpha2)
- [Downloads for v1.9.0-alpha.2](#downloads-for-v190-alpha2)
- [Client Binaries](#client-binaries-5)
- [Server Binaries](#server-binaries-5)
- [Node Binaries](#node-binaries-5)
- [Changelog since v1.8.0](#changelog-since-v180)
- [Action Required](#action-required-2)
- [Other notable changes](#other-notable-changes-4)
- [v1.9.0-alpha.1](#v190-alpha1)
- [Downloads for v1.9.0-alpha.1](#downloads-for-v190-alpha1)
- [Client Binaries](#client-binaries-6)
- [Server Binaries](#server-binaries-6)
- [Node Binaries](#node-binaries-6)
- [Changelog since v1.8.0-alpha.3](#changelog-since-v180-alpha3)
- [Action Required](#action-required-3)
- [Other notable changes](#other-notable-changes-4)
- [Other notable changes](#other-notable-changes-5)
<!-- END MUNGE: GENERATED_TOC -->
<!-- NEW RELEASE NOTES ENTRY -->
# v1.9.1
[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.9/examples)
## Downloads for v1.9.1
filename | sha256 hash
-------- | -----------
[kubernetes.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes.tar.gz) | `0eece0e6c1f68535ea71b58b87e239019bb57fdd61118f3d7defa6bbf4fad5ee`
[kubernetes-src.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-src.tar.gz) | `625ebb79412bd12feccf12e8b6a15d9c71ea681b571f34deaa59fe6c9ba55935`
### Client Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-client-darwin-386.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-darwin-386.tar.gz) | `909556ed9b8445703d0124f2d8c1901b00afaba63a9123a4296be8663c3a2b2d`
[kubernetes-client-darwin-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-darwin-amd64.tar.gz) | `71e191d99d3ac1426e23e087b8d0875e793e5615d3aa7ac1e175b250f9707c48`
[kubernetes-client-linux-386.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-386.tar.gz) | `1c4e60c0c056a3300c7fcc9faccd1b1ea2b337e1360c20c5b1c25fdc47923cf0`
[kubernetes-client-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-amd64.tar.gz) | `fe8fe40148df404b33069931ea30937699758ed4611ef6baddb4c21b7b19db5e`
[kubernetes-client-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-arm64.tar.gz) | `921f5711b97f0b4de69784d9c79f95e80f75a550f28fc1f26597aa0ef6faa471`
[kubernetes-client-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-arm.tar.gz) | `77b010cadef98dc832a2f560afe15e57a675ed9fbc59ffad5e19878510997874`
[kubernetes-client-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-ppc64le.tar.gz) | `02aa71ddcbe8b711814af7287aac79de5d99c1c143c0d3af5e14b1ff195b8bdc`
[kubernetes-client-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-s390x.tar.gz) | `7e315024267306a620045d003785ecc8d7f2e763a6108ae806d5d384aa7552cc`
[kubernetes-client-windows-386.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-windows-386.tar.gz) | `99b2a81b7876498e119db4cb34c434b3790bc41cd882384037c1c1b18cba9f99`
[kubernetes-client-windows-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-windows-amd64.tar.gz) | `d89d303cbbf9e57e5a540277158e4d83ad18ca7402b5b54665f1378bb4528599`
### Server Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-amd64.tar.gz) | `5acf2527461419ba883ac352f7c36c3fa0b86a618dbede187054ad90fa233b0e`
[kubernetes-server-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-arm64.tar.gz) | `e1f61b4dc6e0c9986e95ec25f876f9a89966215ee8cc7f4a3539ec391b217587`
[kubernetes-server-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-arm.tar.gz) | `441c45e16e63e9bdf99887a896a99b3a376af778cb778cc1d0e6afc505237200`
[kubernetes-server-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-ppc64le.tar.gz) | `c0175f02180d9c88028ee5ad4e3ea04af8a6741a97f4900b02615f7f83c4d1c5`
[kubernetes-server-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-s390x.tar.gz) | `2178150d31197ad7f59d44ffea37d682c2675b3a4ea2fc3fa1eaa0e768b993f7`
### Node Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-amd64.tar.gz) | `b8ff0ae693ecca4d55669c66786d6c585f8c77b41a270d65f8175eba8729663a`
[kubernetes-node-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-arm64.tar.gz) | `f0f63baaace463dc663c98cbc9a41e52233d1ef33410571ce3f3e78bd485787e`
[kubernetes-node-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-arm.tar.gz) | `554bdd11deaf390de85830c7c888dfd4d75d9de8ac147799df12993f27bde905`
[kubernetes-node-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-ppc64le.tar.gz) | `913af8ca8b258930e76fd3368acc83608e36e7e270638fa01a6e3be4f682d8bd`
[kubernetes-node-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-s390x.tar.gz) | `8192c1c80563230d727fab71514105571afa52cde8520b3d90af58e6daf0e19c`
[kubernetes-node-windows-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-windows-amd64.tar.gz) | `4408e6d741c6008044584c0d7235e608c596e836d51346ee773589d9b4589fdc`
## Changelog since v1.9.0
### Other notable changes
* Compare correct file names for volume detach operation ([#57053](https://github.com/kubernetes/kubernetes/pull/57053), [@prashima](https://github.com/prashima))
* Fixed a garbage collection race condition where objects with ownerRefs pointing to cluster-scoped objects could be deleted incorrectly. ([#57211](https://github.com/kubernetes/kubernetes/pull/57211), [@liggitt](https://github.com/liggitt))
* Free up CPU and memory requested but unused by Metrics Server Pod Nanny. ([#57252](https://github.com/kubernetes/kubernetes/pull/57252), [@kawych](https://github.com/kawych))
* Configurable liveness probe initial delays for etcd and kube-apiserver in GCE ([#57749](https://github.com/kubernetes/kubernetes/pull/57749), [@wojtek-t](https://github.com/wojtek-t))
* Fixed garbage collection hang ([#57503](https://github.com/kubernetes/kubernetes/pull/57503), [@liggitt](https://github.com/liggitt))
* GCE: Fixes ILB creation on automatic networks with manually created subnetworks. ([#57351](https://github.com/kubernetes/kubernetes/pull/57351), [@nicksardo](https://github.com/nicksardo))
* Check for known manifests during preflight instead of only checking for non-empty manifests directory. ([#57287](https://github.com/kubernetes/kubernetes/pull/57287), [@mattkelly](https://github.com/mattkelly))
* enable flexvolume on Windows node ([#56921](https://github.com/kubernetes/kubernetes/pull/56921), [@andyzhangx](https://github.com/andyzhangx))
* change default azure file/dir mode to 0755 ([#56551](https://github.com/kubernetes/kubernetes/pull/56551), [@andyzhangx](https://github.com/andyzhangx))
* fix incorrect error info when creating an azure file PVC failed ([#56550](https://github.com/kubernetes/kubernetes/pull/56550), [@andyzhangx](https://github.com/andyzhangx))
* Retry 'connection refused' errors when setting up clusters on GCE. ([#57394](https://github.com/kubernetes/kubernetes/pull/57394), [@mborsz](https://github.com/mborsz))
* Fixes issue creating docker secrets with kubectl 1.9 for accessing docker private registries. ([#57463](https://github.com/kubernetes/kubernetes/pull/57463), [@dims](https://github.com/dims))
* Fixes a bug where if an error was returned that was not an `autorest.DetailedError` we would return `"not found", nil` which caused nodes to go to `NotReady` state. ([#57484](https://github.com/kubernetes/kubernetes/pull/57484), [@brendandburns](https://github.com/brendandburns))
* Fix Heapster configuration and Metrics Server configuration to enable overriding default resource requirements. ([#56965](https://github.com/kubernetes/kubernetes/pull/56965), [@kawych](https://github.com/kawych))
# v1.9.0
[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.9/examples)
## Downloads for v1.9.0
filename | sha256 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes.tar.gz) | `d8a52a97382a418b69d46a8b3946bd95c404e03a2d50489d16b36517c9dbc7f4`
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-src.tar.gz) | `95d35ad7d274e5ed207674983c3e8ec28d8190c17e635ee922e2af8349fb031b`
### Client Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-darwin-386.tar.gz) | `2646aa4badf9281b42b921c1e9e2ed235e1305d331423f252a3380396e0c383f`
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-darwin-amd64.tar.gz) | `e76e69cf58399c10908afce8bb8d1f12cb8811de7b24e657e5f9fc80e7b9b6fb`
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-linux-386.tar.gz) | `bcd5ca428eb78fdaadbcf9ff78d9cbcbf70585a2d2582342a4460e55f3bbad13`
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-linux-amd64.tar.gz) | `ba96c8e71dba68b1b3abcad769392fb4df53e402cb65ef25cd176346ee2c39e8`
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-linux-arm64.tar.gz) | `80ceae744fbbfc7759c3d95999075f98e5d86d80e53ea83d16fa8e849da4073d`
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-linux-arm.tar.gz) | `86b271e2518230f3502708cbe8f188a3a68b913c812247b8cc6fbb4c9f35f6c8`
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-linux-ppc64le.tar.gz) | `8b7506ab64ceb2ff470120432d7a6a93adf14e14e612b3c53b3c238d334b55e2`
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-linux-s390x.tar.gz) | `c066aa75a99c141410f9b9a78d230aff4a14dee472fe2b17729e902739798831`
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-windows-386.tar.gz) | `a315535d6a64842a7c2efbf2bb876c0b73db7efd4c848812af07956c2446f526`
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-client-windows-amd64.tar.gz) | `5d2ba1f008253da1a784c8bb5266d026fb6fdac5d22133b51e86d348dbaff49b`
### Server Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-server-linux-amd64.tar.gz) | `a8d7be19e3b662681dc50dc0085ca12045979530a27d0200cf986ada3eff4d32`
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-server-linux-arm64.tar.gz) | `8ef6ad23c60a50b4255ff41db044b2f5922e2a4b0332303065d9e66688a0b026`
[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-server-linux-arm.tar.gz) | `7cb99cf65553c9637ee6f55821ea3f778873a9912917ebbd6203e06d5effb055`
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-server-linux-ppc64le.tar.gz) | `529b0f45a0fc688aa624aa2b850f28807ce2be3ac1660189f20cd3ae864ac064`
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-server-linux-s390x.tar.gz) | `692f0c198da712f15ff93a4634c67f9105e3ec603240b50b51a84480ed63e987`
### Node Binaries
filename | sha256 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-node-linux-amd64.tar.gz) | `7ff3f526d1c4ec23516a65ecec3b947fd8f52d8c0605473b1a87159399dfeab1`
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-node-linux-arm64.tar.gz) | `fada290471467c341734a3cfff63cd0f867aad95623b67096029d76c459bde06`
[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-node-linux-arm.tar.gz) | `ded3640bef5f9701f7f622de4ed162cd2e5a968e80a6a56b843ba84a0b146fac`
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-node-linux-ppc64le.tar.gz) | `a83ebe3b360d33c2190bffd5bf0e2c68268ca2c85e3b5295c1a71ddb517a4f90`
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-node-linux-s390x.tar.gz) | `1210efdf35ec5e0b2e96ff7e456e340684ff12dbea36aa255ac592ca7195e168`
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.9.0/kubernetes-node-windows-amd64.tar.gz) | `9961ad142abc7e769bbe962aeb30a014065fae83291a2d65bc2da91f04fbf185`
## 1.9 Release Notes
## WARNING: etcd backup strongly recommended
Before updating to 1.9, you are strongly recommended to back up your etcd data. Consult the installation procedure you are using (kargo, kops, kube-up, kube-aws, kubeadm etc) for specific advice.
Some upgrade methods might upgrade etcd from 3.0 to 3.1 automatically when you upgrade from Kubernetes 1.8, unless you specify otherwise. Because [etcd does not support downgrading](https://coreos.com/etcd/docs/latest/upgrades/upgrade_3_1.html), you'll need to either remain on etcd 3.1 or restore from a backup if you want to downgrade back to Kubernetes 1.8.
## Introduction to 1.9.0
Kubernetes version 1.9 includes new features and enhancements, as well as fixes to identified issues. The release notes contain a brief overview of the important changes introduced in this release. The content is organized by Special Interest Group ([SIG](https://github.com/kubernetes/community/blob/master/sig-list.md)).
For initial installations, see the [Setup topics](https://kubernetes.io/docs/setup/pick-right-solution/) in the Kubernetes documentation.
To upgrade to this release from a previous version, first take any actions required [Before Upgrading](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#before-upgrading).
For more information about this release and for the latest documentation, see the [Kubernetes documentation](https://kubernetes.io/docs/home/).
## Major themes
Kubernetes is developed by community members whose work is organized into
[Special Interest Groups](https://github.com/kubernetes/community/blob/master/sig-list.md), which provide the themes that guide their work. For the 1.9 release, these themes included:
### API Machinery
Extensibility. SIG API Machinery added a new class of admission control webhooks (mutating), and brought the admission control webhooks to beta.
### Apps
The core workloads API, which is composed of the DaemonSet, Deployment, ReplicaSet, and StatefulSet kinds, has been promoted to GA stability in the apps/v1 group version. As such, the apps/v1beta2 group version is deprecated, and all new code should use the kinds in the apps/v1 group version.
### Auth
SIG Auth focused on extension-related authorization improvements. Permissions can now be added to the built-in RBAC admin/edit/view roles using [cluster role aggregation](https://kubernetes.io/docs/admin/authorization/rbac/#aggregated-clusterroles). [Webhook authorizers](https://kubernetes.io/docs/admin/authorization/webhook/) can now deny requests and short-circuit checking subsequent authorizers. Performance and usability of the beta [PodSecurityPolicy](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) feature was also improved.
### AWS
In v1.9 SIG AWS has improved stability of EBS support across the board. If a Volume is “stuck” in the attaching state to a node for too long a unschedulable taint will be applied to the node, so a Kubernetes admin can [take manual steps to correct the error](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-attaching-volume.html). Users are encouraged to ensure they are monitoring for the taint, and should consider automatically terminating instances in this state.
In addition, support for NVMe disks has been added to Kubernetes, and a service of type LoadBalancer can now be backed with an NLB instead of an ELB (alpha).
### Azure
SIG Azure worked on improvements in the cloud provider, including significant work on the Azure Load Balancer implementation.
### Cluster Lifecycle
SIG Cluster Lifecycle has been focusing on improving kubeadm in order to bring it to GA in a future release, as well as developing the [Cluster API](https://github.com/kubernetes/kube-deploy/tree/master/cluster-api). For kubeadm, most new features, such as support for CoreDNS, IPv6 and Dynamic Kubelet Configuration, have gone in as alpha features. We expect to graduate these features to beta and beyond in the next release. The initial Cluster API spec and GCE sample implementation were developed from scratch during this cycle, and we look forward to stabilizing them into something production-grade during 2018.
### Instrumentation
In v1.9 we focused on improving stability of the components owned by the SIG, including Heapster, Custom Metrics API adapters for Prometheus, and Stackdriver.
### Network
In v1.9 SIG Network has implemented alpha support for IPv6, and alpha support for CoreDNS as a drop-in replacement for kube-dns. Additionally, SIG Network has begun the deprecation process for the extensions/v1beta1 NetworkPolicy API in favor of the networking.k8s.io/v1 equivalent.
### Node
SIG Node iterated on the ability to support more workloads with better performance and improved reliability. Alpha features were improved around hardware accelerator support, device plugins enablement, and cpu pinning policies to enable us to graduate these features to beta in a future release. In addition, a number of reliability and performance enhancements were made across the node to help operators in production.
### OpenStack
In this cycle, SIG OpenStack focused on configuration simplification through smarter defaults and the use of auto-detection wherever feasible (Block Storage API versions, Security Groups) as well as updating API support, including:
* Block Storage (Cinder) V3 is now supported.
* Load Balancer (Octavia) V2 is now supported, in addition to Neutron LBaaS V2.
* Neutron LBaas V1 support has been removed.
This work enables Kubernetes to take full advantage of the relevant services as exposed by OpenStack clouds. Refer to the [Cloud Providers](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#openstack) documentation for more information.
### Storage
[SIG Storage](https://github.com/kubernetes/community/tree/master/sig-storage) is responsible for storage and volume plugin components.
For the 1.9 release, SIG Storage made Kubernetes more pluggable and modular by introducing an alpha implementation of the Container Storage Interface (CSI). CSI will make installing new volume plugins as easy as deploying a pod, and enable third-party storage providers to develop their plugins without the need to add code to the core Kubernetes codebase.
The SIG also focused on adding functionality to the Kubernetes volume subsystem, such as alpha support for exposing volumes as block devices inside containers, extending the alpha volume-resizing support to more volume plugins, and topology-aware volume scheduling.
### Windows
We are advancing support for Windows Server and Windows Server Containers to beta along with continued feature and functional advancements on both the Kubernetes and Windows platforms. This opens the door for many Windows-specific applications and workloads to run on Kubernetes, significantly expanding the implementation scenarios and the enterprise reach of Kubernetes.
## Before Upgrading
Consider the following changes, limitations, and guidelines before you upgrade:
### **API Machinery**
* The admission API, which is used when the API server calls admission control webhooks, is moved from `admission.v1alpha1` to `admission.v1beta1`. You must **delete any existing webhooks before you upgrade** your cluster, and update them to use the latest API. This change is not backward compatible.
* The admission webhook configurations API, part of the admissionregistration API, is now at v1beta1. Delete any existing webhook configurations before you upgrade, and update your configuration files to use the latest API. For this and the previous change, see also [the documentation]([https://kubernetes.io/docs/admin/extensible-admission-controllers/#external-admission-webhooks](https://kubernetes.io/docs/admin/extensible-admission-controllers/#external-admission-webhooks)).
* A new `ValidatingAdmissionWebhook` is added (replacing `GenericAdmissionWebhook`) and is available in the generic API server. You must update your API server configuration file to pass the webhook to the `--admission-control` flag. ([#55988](https://github.com/kubernetes/kubernetes/pull/55988),[ @caesarxuchao](https://github.com/caesarxuchao)) ([#54513](https://github.com/kubernetes/kubernetes/pull/54513),[ @deads2k](https://github.com/deads2k))
* The deprecated options `--portal-net` and `--service-node-ports` for the API server are removed. ([#52547](https://github.com/kubernetes/kubernetes/pull/52547),[ @xiangpengzhao](https://github.com/xiangpengzhao))
### **Auth**
* PodSecurityPolicy: A compatibility issue with the allowPrivilegeEscalation field that caused policies to start denying pods they previously allowed was fixed. If you defined PodSecurityPolicy objects using a 1.8.0 client or server and set allowPrivilegeEscalation to false, these objects must be reapplied after you upgrade. ([#53443](https://github.com/kubernetes/kubernetes/pull/53443),[ @liggitt](https://github.com/liggitt))
* KMS: Alpha integration with GCP KMS was removed in favor of a future out-of-process extension point. Discontinue use of the GCP KMS integration and ensure [data has been decrypted](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#decrypting-all-data) (or reencrypted with a different provider) before upgrading ([#54759](https://github.com/kubernetes/kubernetes/pull/54759),[ @sakshamsharma](https://github.com/sakshamsharma))
### **CLI**
* Swagger 1.2 validation is removed for kubectl. The options `--use-openapi` and `--schema-cache-dir` are also removed because they are no longer needed. ([#53232](https://github.com/kubernetes/kubernetes/pull/53232),[ @apelisse](https://github.com/apelisse))
### **Cluster Lifecycle**
* You must either specify the `--discovery-token-ca-cert-hash` flag to `kubeadm join`, or opt out of the CA pinning feature using `--discovery-token-unsafe-skip-ca-verification`.
* The default `auto-detect` behavior of the kubelet's `--cloud-provider` flag is removed.
* You can manually set `--cloud-provider=auto-detect`, but be aware that this behavior will be removed completely in a future version.
* Best practice for version 1.9 and future versions is to explicitly set a cloud-provider. See [the documentation](https://kubernetes.io/docs/getting-started-guides/scratch/#cloud-providers)
* The kubeadm `--skip-preflight-checks` flag is now deprecated and will be removed in a future release.
* If you are using the cloud provider API to determine the external host address of the apiserver, set `--external-hostname` explicitly instead. The cloud provider detection has been deprecated and will be removed in the future ([#54516](https://github.com/kubernetes/kubernetes/pull/54516),[ @dims](https://github.com/dims))
### **Multicluster**
* Development of Kubernetes Federation has moved to [github.com/kubernetes/federation](github.com/kubernetes/federation). This move out of tree also means that Federation will begin releasing separately from Kubernetes. Impact:
* Federation-specific behavior will no longer be included in kubectl
* kubefed will no longer be released as part of Kubernetes
* The Federation servers will no longer be included in the hyperkube binary and image. ([#53816](https://github.com/kubernetes/kubernetes/pull/53816),[ @marun](https://github.com/marun))
### **Node**
* The kubelet `--network-plugin-dir` flag is removed. This flag was deprecated in version 1.7, and is replaced with `--cni-bin-dir`. ([#53564](https://github.com/kubernetes/kubernetes/pull/53564),[ @supereagle](https://github.com/supereagle))
* kubelet's `--cloud-provider` flag no longer defaults to "auto-detect". If you want cloud-provider support in kubelet, you must set a specific cloud-provider explicitly. ([#53573](https://github.com/kubernetes/kubernetes/pull/53573),[ @dims](https://github.com/dims))
### **Network**
* NetworkPolicy objects are now stored in etcd in v1 format. After you upgrade to version 1.9, make sure that all NetworkPolicy objects are migrated to v1. ([#51955](https://github.com/kubernetes/kubernetes/pull/51955), [@danwinship](https://github.com/danwinship))
* The API group/version for the kube-proxy configuration has changed from `componentconfig/v1alpha1` to `kubeproxy.config.k8s.io/v1alpha1`. If you are using a config file for kube-proxy instead of the command line flags, you must change its apiVersion to `kubeproxy.config.k8s.io/v1alpha1`. ([#53645](https://github.com/kubernetes/kubernetes/pull/53645), [@xiangpengzhao](https://github.com/xiangpengzhao))
* The "ServiceNodeExclusion" feature gate must now be enabled for the `alpha.service-controller.kubernetes.io/exclude-balancer` annotation on nodes to be honored. ([#54644](https://github.com/kubernetes/kubernetes/pull/54644),[ @brendandburns](https://github.com/brendandburns))
### **Scheduling**
* Taint key `unreachable` is now in GA.
* Taint key `notReady` is changed to `not-ready`, and is also now in GA.
* These changes are automatically updated for taints. Tolerations for these taints must be updated manually. Specifically, you must:
* Change `node.alpha.kubernetes.io/notReady` to `node.kubernetes.io/not-ready`
* Change `node.alpha.kubernetes.io/unreachable` to `node.kubernetes.io/unreachable`
* The `node.kubernetes.io/memory-pressure` taint now respects the configured whitelist. To use it, you must add it to the whitelist.([#55251](https://github.com/kubernetes/kubernetes/pull/55251),[ @deads2k](https://github.com/deads2k))
* Refactor kube-scheduler configuration ([#52428](https://github.com/kubernetes/kubernetes/pull/52428))
* The kube-scheduler command now supports a --config flag which is the location of a file containing a serialized scheduler configuration. Most other kube-scheduler flags are now deprecated. ([#52562](https://github.com/kubernetes/kubernetes/pull/52562),[ @ironcladlou](https://github.com/ironcladlou))
* Opaque integer resources (OIR), which were (deprecated in v1.8.), have been removed. ([#55103](https://github.com/kubernetes/kubernetes/pull/55103),[ @ConnorDoyle](https://github.com/ConnorDoyle))
### **Storage**
* [alpha] The LocalPersistentVolumes alpha feature now also requires the VolumeScheduling alpha feature. This is a breaking change, and the following changes are required:
* The VolumeScheduling feature gate must also be enabled on kube-scheduler and kube-controller-manager components.
* The NoVolumeNodeConflict predicate has been removed. For non-default schedulers, update your scheduler policy.
* The CheckVolumeBinding predicate must be enabled in non-default schedulers. ([#55039](https://github.com/kubernetes/kubernetes/pull/55039),[ @msau42](https://github.com/msau42))
### **OpenStack**
* Remove the LbaasV1 of OpenStack cloud provider, currently only support LbaasV2. ([#52717](https://github.com/kubernetes/kubernetes/pull/52717),[ @FengyunPan](https://github.com/FengyunPan))
## Known Issues
This section contains a list of known issues reported in Kubernetes 1.9 release. The content is populated from the [v1.9.x known issues and FAQ accumulator](https://github.com/kubernetes/kubernetes/issues/57159](https://github.com/kubernetes/kubernetes/issues/57159).
* If you are adding Windows Server Virtual Machines as nodes to your Kubernetes environment, there is a compatibility issue with certain virtualization products. Specifically the Windows version of the kubelet.exe calls `GetPhysicallyInstalledSystemMemory` to get the physical memory installed on Windows machines and reports it as part of node metrics to heapster. This API call fails for VMware and VirtualBox virtualization environments. This issue is not present in bare metal Windows deployments, in Hyper-V, or on some of the popular public cloud providers.
* If you run `kubectl get po` while the API server in unreachable, a misleading error is returned: `the server doesn't have a resource type "po"`. To work around this issue, specify the full resource name in the command instead of the abbreviation: `kubectl get pods`. This issue will be fixed in a future release.
For more information, see [#57198](https://github.com/kubernetes/kubernetes/issues/57198).
* Mutating and validating webhook configurations are continuously polled by the API server (once per second). This issue will be fixed in a future release.
For more information, see [#56357](https://github.com/kubernetes/kubernetes/issues/56357).
* Audit logging is slow because writes to the log are performed synchronously with requests to the log. This issue will be fixed in a future release.
For more information, see [#53006](https://github.com/kubernetes/kubernetes/issues/53006).
* Custom Resource Definitions (CRDs) are not properly deleted under certain conditions. This issue will be fixed in a future release.
For more information, see [#56348](https://github.com/kubernetes/kubernetes/issues/56348).
* API server times out after performing a rolling update of the etcd cluster. This issue will be fixed in a future release.
For more information, see [#47131](https://github.com/kubernetes/kubernetes/issues/47131)
* If a namespaced resource is owned by a cluster scoped resource, and the namespaced dependent is processed before the cluster scoped owner has ever been observed by the garbage collector, the dependent will be erroneously deleted.
For more information, see [#54940](https://github.com/kubernetes/kubernetes/issues/54940)
## Deprecations
This section provides an overview of deprecated API versions, options, flags, and arguments. Deprecated means that we intend to remove the capability from a future release. After removal, the capability will no longer work. The sections are organized by SIGs.
### **API Machinery**
* The kube-apiserver `--etcd-quorum-read` flag is deprecated and the ability to switch off quorum read will be removed in a future release. ([#53795](https://github.com/kubernetes/kubernetes/pull/53795),[ @xiangpengzhao](https://github.com/xiangpengzhao))
* The `/ui` redirect in kube-apiserver is deprecated and will be removed in Kubernetes 1.10. ([#53046](https://github.com/kubernetes/kubernetes/pull/53046), [@maciaszczykm](https://github.com/maciaszczykm))
* `etcd2` as a backend is deprecated and support will be removed in Kubernetes 1.13 or 1.14.
### **Auth**
* Default controller-manager options for `--cluster-signing-cert-file` and `--cluster-signing-key-file` are deprecated and will be removed in a future release. ([#54495](https://github.com/kubernetes/kubernetes/pull/54495),[ @mikedanese](https://github.com/mikedanese))
* RBAC objects are now stored in etcd in v1 format. After upgrading to 1.9, ensure all RBAC objects (Roles, RoleBindings, ClusterRoles, ClusterRoleBindings) are at v1. v1alpha1 support is deprecated and will be removed in a future release. ([#52950](https://github.com/kubernetes/kubernetes/pull/52950),[ @liggitt](https://github.com/liggitt))
### **Cluster Lifecycle**
* kube-apiserver: `--ssh-user` and `--ssh-keyfile` are now deprecated and will be removed in a future release. Users of SSH tunnel functionality in Google Container Engine for the Master -> Cluster communication should plan alternate methods for bridging master and node networks. ([#54433](https://github.com/kubernetes/kubernetes/pull/54433),[ @dims](https://github.com/dims))
* The kubeadm `--skip-preflight-checks` flag is now deprecated and will be removed in a future release.
* If you are using the cloud provider API to determine the external host address of the apiserver, set `--external-hostname` explicitly instead. The cloud provider detection has been deprecated and will be removed in the future ([#54516](https://github.com/kubernetes/kubernetes/pull/54516),[ @dims](https://github.com/dims))
### **Network**
* The NetworkPolicy extensions/v1beta1 API is now deprecated and will be removed in a future release. This functionality has been migrated to a dedicated v1 API - networking.k8s.io/v1. v1beta1 Network Policies can be upgraded to the v1 API with the [cluster/update-storage-objects.sh script](https://github.com/danwinship/kubernetes/blob/master/cluster/update-storage-objects.sh). Documentation can be found [here](https://kubernetes.io/docs/concepts/services-networking/network-policies/). ([#56425](https://github.com/kubernetes/kubernetes/pull/56425), [@cmluciano](https://github.com/cmluciano))
### **Storage**
* The `volume.beta.kubernetes.io/storage-class` annotation is deprecated. It will be removed in a future release. For the StorageClass API object, use v1, and in place of the annotation use `v1.PersistentVolumeClaim.Spec.StorageClassName` and `v1.PersistentVolume.Spec.StorageClassName` instead. ([#53580](https://github.com/kubernetes/kubernetes/pull/53580),[ @xiangpengzhao](https://github.com/xiangpengzhao))
### **Scheduling**
* The kube-scheduler command now supports a `--config` flag, which is the location of a file containing a serialized scheduler configuration. Most other kube-scheduler flags are now deprecated. ([#52562](https://github.com/kubernetes/kubernetes/pull/52562),[ @ironcladlou](https://github.com/ironcladlou))
### **Node**
* The kubelet's `--enable-custom-metrics` flag is now deprecated. ([#54154](https://github.com/kubernetes/kubernetes/pull/54154),[ @mtaufen](https://github.com/mtaufen))
## Notable Changes
### **Workloads API (apps/v1)**
As announced with the release of version 1.8, the Kubernetes Workloads API is at v1 in version 1.9. This API consists of the DaemonSet, Deployment, ReplicaSet and StatefulSet kinds.
### **API Machinery**
#### **Admission Control**
* Admission webhooks are now in beta, and include the following:
* Mutation support for admission webhooks. ([#54892](https://github.com/kubernetes/kubernetes/pull/54892),[ @caesarxuchao](https://github.com/caesarxuchao))
* Webhook admission now takes a config file that describes how to authenticate to webhook servers ([#54414](https://github.com/kubernetes/kubernetes/pull/54414),[ @deads2k](https://github.com/deads2k))
* The dynamic admission webhook now supports a URL in addition to a service reference, to accommodate out-of-cluster webhooks. ([#54889](https://github.com/kubernetes/kubernetes/pull/54889),[ @lavalamp](https://github.com/lavalamp))
* Added `namespaceSelector` to `externalAdmissionWebhook` configuration to allow applying webhooks only to objects in the namespaces that have matching labels. ([#54727](https://github.com/kubernetes/kubernetes/pull/54727),[ @caesarxuchao](https://github.com/caesarxuchao))
* Metrics are added for monitoring admission plugins, including the new dynamic (webhook-based) ones. ([#55183](https://github.com/kubernetes/kubernetes/pull/55183),[ @jpbetz](https://github.com/jpbetz))
* The PodSecurityPolicy annotation kubernetes.io/psp on pods is set only once on create. ([#55486](https://github.com/kubernetes/kubernetes/pull/55486),[ @sttts](https://github.com/sttts))
#### **API & API server**
* Fixed a bug related to discovery information for scale subresources in the apps API group ([#54683](https://github.com/kubernetes/kubernetes/pull/54683),[ @liggitt](https://github.com/liggitt))
* Fixed a bug that prevented client-go metrics from being registered in Prometheus. This bug affected multiple components. ([#53434](https://github.com/kubernetes/kubernetes/pull/53434),[ @crassirostris](https://github.com/crassirostris))
#### **Audit**
* Fixed a bug so that `kube-apiserver` now waits for open connections to finish before exiting. This fix provides graceful shutdown and ensures that the audit backend no longer drops events on shutdown. ([#53695](https://github.com/kubernetes/kubernetes/pull/53695),[ @hzxuzhonghu](https://github.com/hzxuzhonghu))
* Webhooks now always retry sending if a connection reset error is returned. ([#53947](https://github.com/kubernetes/kubernetes/pull/53947),[ @crassirostris](https://github.com/crassirostris))
#### **Custom Resources**
* Validation of resources defined by a Custom Resource Definition (CRD) is now in beta ([#54647](https://github.com/kubernetes/kubernetes/pull/54647),[ @colemickens](https://github.com/colemickens))
* An example CRD controller has been added, at [github.com/kubernetes/sample-controller](github.com/kubernetes/sample-controller). ([#52753](https://github.com/kubernetes/kubernetes/pull/52753),[ @munnerz](https://github.com/munnerz))
* Custom resources served by CustomResourceDefinition objects now support field selectors for `metadata.name` and `metadata.namespace`. Also fixed an issue with watching a single object; earlier versions could watch only a collection, and so a watch on an instance would fail. ([#53345](https://github.com/kubernetes/kubernetes/pull/53345),[ @ncdc](https://github.com/ncdc))
#### **Other**
* `kube-apiserver` now runs with the default value for `service-cluster-ip-range` ([#52870](https://github.com/kubernetes/kubernetes/pull/52870),[ @jennybuckley](https://github.com/jennybuckley))
* Add `--etcd-compaction-interval` to apiserver for controlling request of compaction to etcd3 from apiserver. ([#51765](https://github.com/kubernetes/kubernetes/pull/51765),[ @mitake](https://github.com/mitake))
* The httpstream/spdy calls now support CIDR notation for NO_PROXY ([#54413](https://github.com/kubernetes/kubernetes/pull/54413),[ @kad](https://github.com/kad))
* Code generation for CRD and User API server types is improved with the addition of two new scripts to k8s.io/code-generator: `generate-groups.sh` and `generate-internal-groups.sh`. ([#52186](https://github.com/kubernetes/kubernetes/pull/52186),[ @sttts](https://github.com/sttts))
* [beta] Flag `--chunk-size={SIZE}` is added to `kubectl get` to customize the number of results returned in large lists of resources. This reduces the perceived latency of managing large clusters because the server returns the first set of results to the client much more quickly. Pass 0 to disable this feature.([#53768](https://github.com/kubernetes/kubernetes/pull/53768),[ @smarterclayton](https://github.com/smarterclayton))
* [beta] API chunking via the limit and continue request parameters is promoted to beta in this release. Client libraries using the Informer or ListWatch types will automatically opt in to chunking. ([#52949](https://github.com/kubernetes/kubernetes/pull/52949),[ @smarterclayton](https://github.com/smarterclayton))
* The `--etcd-quorum-read` flag now defaults to true to ensure correct operation with HA etcd clusters. This flag is deprecated and the flag will be removed in future versions, as well as the ability to turn off this functionality. ([#53717](https://github.com/kubernetes/kubernetes/pull/53717),[ @liggitt](https://github.com/liggitt))
* Add events.k8s.io api group with v1beta1 API containing redesigned event type. ([#49112](https://github.com/kubernetes/kubernetes/pull/49112),[ @gmarek](https://github.com/gmarek))
* Fixed a bug where API discovery failures were crashing the kube controller manager via the garbage collector. ([#55259](https://github.com/kubernetes/kubernetes/pull/55259),[ @ironcladlou](https://github.com/ironcladlou))
* `conversion-gen` is now usable in a context without a vendored k8s.io/kubernetes. The Kubernetes core API is removed from `default extra-peer-dirs`. ([#54394](https://github.com/kubernetes/kubernetes/pull/54394),[ @sttts](https://github.com/sttts))
* Fixed a bug where the `client-gen` tag for code-generator required a newline between a comment block and a statement. tag shortcomings when newline is omitted ([#53893](https://github.com/kubernetes/kubernetes/pull/53893)) ([#55233](https://github.com/kubernetes/kubernetes/pull/55233),[ @sttts](https://github.com/sttts))
* The Apiserver proxy now rewrites the URL when a service returns an absolute path with the request's host. ([#52556](https://github.com/kubernetes/kubernetes/pull/52556),[ @roycaihw](https://github.com/roycaihw))
* The gRPC library is updated to pick up data race fix ([#53124](https://github.com/kubernetes/kubernetes/pull/53124)) ([#53128](https://github.com/kubernetes/kubernetes/pull/53128),[ @dixudx](https://github.com/dixudx))
* Fixed server name verification of aggregated API servers and webhook admission endpoints ([#56415](https://github.com/kubernetes/kubernetes/pull/56415),[ @liggitt](https://github.com/liggitt))
### **Apps**
* The `kubernetes.io/created-by` annotation is no longer added to controller-created objects. Use the `metadata.ownerReferences` item with controller set to `true` to determine which controller, if any, owns an object. ([#54445](https://github.com/kubernetes/kubernetes/pull/54445),[ @crimsonfaith91](https://github.com/crimsonfaith91))
* StatefulSet controller now creates a label for each Pod in a StatefulSet. The label is `statefulset.kubernetes.io/pod-name`, where `pod-name` = the name of the Pod. This allows users to create a Service per Pod to expose a connection to individual Pods. ([#55329](https://github.com/kubernetes/kubernetes/pull/55329),[ @kow3ns](https://github.com/kow3ns))
* DaemonSet status includes a new field named `conditions`, making it consistent with other workloads controllers. ([#55272](https://github.com/kubernetes/kubernetes/pull/55272),[ @janetkuo](https://github.com/janetkuo))
* StatefulSet status now supports conditions, making it consistent with other core controllers in v1 ([#55268](https://github.com/kubernetes/kubernetes/pull/55268),[ @foxish](https://github.com/foxish))
* The default garbage collection policy for Deployment, DaemonSet, StatefulSet, and ReplicaSet has changed from OrphanDependents to DeleteDependents when the deletion is requested through an `apps/v1` endpoint. ([#55148](https://github.com/kubernetes/kubernetes/pull/55148),[ @dixudx](https://github.com/dixudx))
* Clients using older endpoints will be unaffected. This change is only at the REST API level and is independent of the default behavior of particular clients (e.g. this does not affect the default for the kubectl `--cascade` flag).
* If you upgrade your client-go libs and use the `AppsV1()` interface, please note that the default garbage collection behavior is changed.
### **Auth**
#### **Audit**
* RequestReceivedTimestamp and StageTimestamp are added to audit events ([#52981](https://github.com/kubernetes/kubernetes/pull/52981),[ @CaoShuFeng](https://github.com/CaoShuFeng))
* Advanced audit policy now supports a policy wide omitStage ([#54634](https://github.com/kubernetes/kubernetes/pull/54634),[ @CaoShuFeng](https://github.com/CaoShuFeng))
#### **RBAC**
* New permissions have been added to default RBAC roles ([#52654](https://github.com/kubernetes/kubernetes/pull/52654),[ @liggitt](https://github.com/liggitt)):
* The default admin and edit roles now include read/write permissions
* The view role includes read permissions on poddisruptionbudget.policy resources.
* RBAC rules can now match the same subresource on any resource using the form `*/(subresource)`. For example, `*/scale` matches requests to `replicationcontroller/scale`. ([#53722](https://github.com/kubernetes/kubernetes/pull/53722),[ @deads2k](https://github.com/deads2k))
* The RBAC bootstrapping policy now allows authenticated users to create selfsubjectrulesreviews. ([#56095](https://github.com/kubernetes/kubernetes/pull/56095),[ @ericchiang](https://github.com/ericchiang))
* RBAC ClusterRoles can now select other roles to aggregate. ([#54005](https://github.com/kubernetes/kubernetes/pull/54005),[ @deads2k](https://github.com/deads2k))
* Fixed an issue with RBAC reconciliation that caused duplicated subjects in some bootstrapped RoleBinding objects on each restart of the API server. ([#53239](https://github.com/kubernetes/kubernetes/pull/53239),[ @enj](https://github.com/enj))
#### **Other**
* Pod Security Policy can now manage access to specific FlexVolume drivers ([#53179](https://github.com/kubernetes/kubernetes/pull/53179),[ @wanghaoran1988](https://github.com/wanghaoran1988))
* Audit policy files without apiVersion and kind are treated as invalid. ([#54267](https://github.com/kubernetes/kubernetes/pull/54267),[ @ericchiang](https://github.com/ericchiang))
* Fixed a bug that where forbidden errors were encountered when accessing ReplicaSet and DaemonSets objects via the apps API group. ([#54309](https://github.com/kubernetes/kubernetes/pull/54309),[ @liggitt](https://github.com/liggitt))
* Improved PodSecurityPolicy admission latency. ([#55643](https://github.com/kubernetes/kubernetes/pull/55643),[ @tallclair](https://github.com/tallclair))
* kube-apiserver: `--oidc-username-prefix` and `--oidc-group-prefix` flags are now correctly enabled. ([#56175](https://github.com/kubernetes/kubernetes/pull/56175),[ @ericchiang](https://github.com/ericchiang))
* If multiple PodSecurityPolicy objects allow a submitted pod, priority is given to policies that do not require default values for any fields in the pod spec. If default values are required, the first policy ordered by name that allows the pod is used. ([#52849](https://github.com/kubernetes/kubernetes/pull/52849),[ @liggitt](https://github.com/liggitt))
* A new controller automatically cleans up Certificate Signing Requests that are Approved and Issued, or Denied. ([#51840](https://github.com/kubernetes/kubernetes/pull/51840),[ @jcbsmpsn](https://github.com/jcbsmpsn))
* PodSecurityPolicies have been added for all in-tree cluster addons ([#55509](https://github.com/kubernetes/kubernetes/pull/55509),[ @tallclair](https://github.com/tallclair))
#### **GCE**
* Added support for PodSecurityPolicy on GCE: `ENABLE_POD_SECURITY_POLICY=true` enables the admission controller, and installs policies for default addons. ([#52367](https://github.com/kubernetes/kubernetes/pull/52367),[ @tallclair](https://github.com/tallclair))
### **Autoscaling**
* HorizontalPodAutoscaler objects now properly functions on scalable resources in any API group. Fixed by adding a polymorphic scale client. ([#53743](https://github.com/kubernetes/kubernetes/pull/53743),[ @DirectXMan12](https://github.com/DirectXMan12))
* Fixed a set of minor issues with Cluster Autoscaler 1.0.1 ([#54298](https://github.com/kubernetes/kubernetes/pull/54298),[ @mwielgus](https://github.com/mwielgus))
* HPA tolerance is now configurable by setting the `horizontal-pod-autoscaler-tolerance` flag. ([#52275](https://github.com/kubernetes/kubernetes/pull/52275),[ @mattjmcnaughton](https://github.com/mattjmcnaughton))
* Fixed a bug that allowed the horizontal pod autoscaler to allocate more `desiredReplica` objects than `maxReplica` objects in certain instances. ([#53690](https://github.com/kubernetes/kubernetes/pull/53690),[ @mattjmcnaughton](https://github.com/mattjmcnaughton))
### **AWS**
* Nodes can now use instance types (such as C5) that use NVMe. ([#56607](https://github.com/kubernetes/kubernetes/pull/56607), [@justinsb](https://github.com/justinsb))
* Nodes are now unreachable if volumes are stuck in the attaching state. Implemented by applying a taint to the node. ([#55558](https://github.com/kubernetes/kubernetes/pull/55558),[ @gnufied](https://github.com/gnufied))
* Volumes are now checked for available state before attempting to attach or delete a volume in EBS. ([#55008](https://github.com/kubernetes/kubernetes/pull/55008),[ @gnufied](https://github.com/gnufied))
* Fixed a bug where error log messages were breaking into two lines. ([#49826](https://github.com/kubernetes/kubernetes/pull/49826),[ @dixudx](https://github.com/dixudx))
* Fixed a bug so that volumes are now detached from stopped nodes. ([#55893](https://github.com/kubernetes/kubernetes/pull/55893),[ @gnufied](https://github.com/gnufied))
* You can now override the health check parameters for AWS ELBs by specifying annotations on the corresponding service. The new annotations are: `healthy-threshold`, `unhealthy-threshold`, `timeout`, `interval`. The prefix for all annotations is `service.beta.kubernetes.io/aws-load-balancer-healthcheck-`. ([#56024](https://github.com/kubernetes/kubernetes/pull/56024),[ @dimpavloff](https://github.com/dimpavloff))
* Fixed a bug so that AWS ECR credentials are now supported in the China region. ([#50108](https://github.com/kubernetes/kubernetes/pull/50108),[ @zzq889](https://github.com/zzq889))
* Added Amazon NLB support ([#53400](https://github.com/kubernetes/kubernetes/pull/53400),[ @micahhausler](https://github.com/micahhausler))
* Additional annotations are now properly set or updated for AWS load balancers ([#55731](https://github.com/kubernetes/kubernetes/pull/55731),[ @georgebuckerfield](https://github.com/georgebuckerfield))
* AWS SDK is updated to version 1.12.7 ([#53561](https://github.com/kubernetes/kubernetes/pull/53561),[ @justinsb](https://github.com/justinsb))
### **Azure**
* Fixed several issues with properly provisioning Azure disk storage ([#55927](https://github.com/kubernetes/kubernetes/pull/55927),[ @andyzhangx](https://github.com/andyzhangx))
* A new service annotation `service.beta.kubernetes.io/azure-dns-label-name` now sets the Azure DNS label for a public IP address. ([#47849](https://github.com/kubernetes/kubernetes/pull/47849),[ @tomerf](https://github.com/tomerf))
* Support for GetMountRefs function added; warning messages no longer displayed. ([#54670](https://github.com/kubernetes/kubernetes/pull/54670), [#52401](https://github.com/kubernetes/kubernetes/pull/52401),[ @andyzhangx](https://github.com/andyzhangx))
* Fixed an issue where an Azure PersistentVolume object would crash because the value of `volumeSource.ReadOnly` was set to nil. ([#54607](https://github.com/kubernetes/kubernetes/pull/54607),[ @andyzhangx](https://github.com/andyzhangx))
* Fixed an issue with Azure disk mount failures on CoreOS and some other distros ([#54334](https://github.com/kubernetes/kubernetes/pull/54334),[ @andyzhangx](https://github.com/andyzhangx))
* GRS, RAGRS storage account types are now supported for Azure disks. ([#55931](https://github.com/kubernetes/kubernetes/pull/55931),[ @andyzhangx](https://github.com/andyzhangx))
* Azure NSG rules are now restricted so that external access is allowed only to the load balancer IP. ([#54177](https://github.com/kubernetes/kubernetes/pull/54177),[ @itowlson](https://github.com/itowlson))
* Azure NSG rules can be consolidated to reduce the likelihood of hitting Azure resource limits (available only in regions where the Augmented Security Groups preview is available). ([#55740](https://github.com/kubernetes/kubernetes/pull/55740), [@itowlson](https://github.com/itowlson))
* The Azure SDK is upgraded to v11.1.1. ([#54971](https://github.com/kubernetes/kubernetes/pull/54971),[ @itowlson](https://github.com/itowlson))
* You can now create Windows mount paths ([#51240](https://github.com/kubernetes/kubernetes/pull/51240),[ @andyzhangx](https://github.com/andyzhangx))
* Fixed a controller manager crash issue on a manually created k8s cluster. ([#53694](https://github.com/kubernetes/kubernetes/pull/53694),[ @andyzhangx](https://github.com/andyzhangx))
* Azure-based clusters now support unlimited mount points. ([#54668](https://github.com/kubernetes/kubernetes/pull/54668)) ([#53629](https://github.com/kubernetes/kubernetes/pull/53629),[ @andyzhangx](https://github.com/andyzhangx))
* Load balancer reconciliation now considers NSG rules based not only on Name, but also on Protocol, SourcePortRange, DestinationPortRange, SourceAddressPrefix, DestinationAddressPrefix, Access, and Direction. This change makes it possible to update NSG rules under more conditions. ([#55752](https://github.com/kubernetes/kubernetes/pull/55752),[ @kevinkim9264](https://github.com/kevinkim9264))
* Custom mountOptions for the azurefile StorageClass object are now respected. Specifically, `dir_mode` and `file_mode` can now be customized. ([#54674](https://github.com/kubernetes/kubernetes/pull/54674),[ @andyzhangx](https://github.com/andyzhangx))
* Azure Load Balancer Auto Mode: Services can be annotated to allow auto selection of available load balancers and to provide specific availability sets that host the load balancers (for example, `service.beta.kubernetes.io/azure-load-balancer-mode=auto|as1,as2...`)
### **CLI**
#### **Kubectl**
* `kubectl cp` can now copy a remote file into a local directory. ([#46762](https://github.com/kubernetes/kubernetes/pull/46762),[ @bruceauyeung](https://github.com/bruceauyeung))
* `kubectl cp` now honors destination names for directories. A complete directory is now copied; in previous versions only the file contents were copied. ([#51215](https://github.com/kubernetes/kubernetes/pull/51215),[ @juanvallejo](https://github.com/juanvallejo))
* You can now use `kubectl get` with a fieldSelector. ([#50140](https://github.com/kubernetes/kubernetes/pull/50140),[ @dixudx](https://github.com/dixudx))
* Secret data containing Docker registry auth objects is now generated using the config.json format ([#53916](https://github.com/kubernetes/kubernetes/pull/53916),[ @juanvallejo](https://github.com/juanvallejo))
* `kubectl apply` now calculates the diff between the current and new configurations based on the OpenAPI spec. If the OpenAPI spec is not available, it falls back to baked-in types. ([#51321](https://github.com/kubernetes/kubernetes/pull/51321),[ @mengqiy](https://github.com/mengqiy))
* `kubectl explain` now explains `apiservices` and `customresourcedefinition`. (Updated to use OpenAPI instead of Swagger 1.2.) ([#53228](https://github.com/kubernetes/kubernetes/pull/53228),[ @apelisse](https://github.com/apelisse))
* `kubectl get` now uses OpenAPI schema extensions by default to select columns for custom types. ([#53483](https://github.com/kubernetes/kubernetes/pull/53483),[ @apelisse](https://github.com/apelisse))
* kubectl `top node` now sorts by name and `top pod` sorts by namespace. Fixed a bug where results were inconsistently sorted. ([#53560](https://github.com/kubernetes/kubernetes/pull/53560),[ @dixudx](https://github.com/dixudx))
* Added --dry-run option to kubectl drain. ([#52440](https://github.com/kubernetes/kubernetes/pull/52440),[ @juanvallejo](https://github.com/juanvallejo))
* Kubectl now outputs <none> for columns specified by -o custom-columns but not found in object, rather than "xxx is not found" ([#51750](https://github.com/kubernetes/kubernetes/pull/51750),[ @jianhuiz](https://github.com/jianhuiz))
* `kubectl create pdb` no longer sets the min-available field by default. ([#53047](https://github.com/kubernetes/kubernetes/pull/53047),[ @yuexiao-wang](https://github.com/yuexiao-wang))
* The canonical pronunciation of kubectl is "cube control".
* Added --raw to kubectl create to POST using the normal transport. ([#54245](https://github.com/kubernetes/kubernetes/pull/54245),[ @deads2k](https://github.com/deads2k))
* Added kubectl `create priorityclass` subcommand ([#54858](https://github.com/kubernetes/kubernetes/pull/54858),[ @wackxu](https://github.com/wackxu))
* Fixed an issue where `kubectl set` commands occasionally encountered conversion errors for ReplicaSet and DaemonSet objects ([#53158](https://github.com/kubernetes/kubernetes/pull/53158),[ @liggitt](https://github.com/liggitt))
### **Cluster Lifecycle**
#### **API Server**
* [alpha] Added an `--endpoint-reconciler-type` command-line argument to select the endpoint reconciler to use. The default is to use the 'master-count' reconciler which is the default for 1.9 and in use prior to 1.9. The 'lease' reconciler stores endpoints within the storage api for better cleanup of deleted (or removed) API servers. The 'none' reconciler is a no-op reconciler, which can be used in self-hosted environments. ([#51698](https://github.com/kubernetes/kubernetes/pull/51698), [@rphillips](https://github.com/rphillips))
#### **Cloud Provider Integration**
* Added `cloud-controller-manager` to `hyperkube`. This is useful as a number of deployment tools run all of the kubernetes components from the `hyperkube `image/binary. It also makes testing easier as a single binary/image can be built and pushed quickly. ([#54197](https://github.com/kubernetes/kubernetes/pull/54197),[ @colemickens](https://github.com/colemickens))
* Added the concurrent service sync flag to the Cloud Controller Manager to allow changing the number of workers. (`--concurrent-service-syncs`) ([#55561](https://github.com/kubernetes/kubernetes/pull/55561),[ @jhorwit2](https://github.com/jhorwit2))
* kubelet's --cloud-provider flag no longer defaults to "auto-detect". If you want cloud-provider support in kubelet, you must set a specific cloud-provider explicitly. ([#53573](https://github.com/kubernetes/kubernetes/pull/53573),[ @dims](https://github.com/dims))
#### **Kubeadm**
* kubeadm health checks can now be skipped with `--ignore-preflight-errors`; the `--skip-preflight-checks` flag is now deprecated and will be removed in a future release. ([#56130](https://github.com/kubernetes/kubernetes/pull/56130),[ @anguslees](https://github.com/anguslees)) ([#56072](https://github.com/kubernetes/kubernetes/pull/56072),[ @kad](https://github.com/kad))
* You now have the option to use CoreDNS instead of KubeDNS. To install CoreDNS instead of kube-dns, set CLUSTER_DNS_CORE_DNS to 'true'. This support is experimental. ([#52501](https://github.com/kubernetes/kubernetes/pull/52501),[ @rajansandeep](https://github.com/rajansandeep)) ([#55728](https://github.com/kubernetes/kubernetes/pull/55728),[ @rajansandeep](https://github.com/rajansandeep))
* Added --print-join-command flag for kubeadm token create. ([#56185](https://github.com/kubernetes/kubernetes/pull/56185),[ @mattmoyer](https://github.com/mattmoyer))
* Added a new --etcd-upgrade keyword to kubeadm upgrade apply. When this keyword is specified, etcd's static pod gets upgraded to the etcd version officially recommended for a target kubernetes release. ([#55010](https://github.com/kubernetes/kubernetes/pull/55010),[ @sbezverk](https://github.com/sbezverk))
* Kubeadm now supports Kubelet Dynamic Configuration on an alpha level. ([#55803](https://github.com/kubernetes/kubernetes/pull/55803),[ @xiangpengzhao](https://github.com/xiangpengzhao))
* Added support for adding a Windows node ([#53553](https://github.com/kubernetes/kubernetes/pull/53553),[ @bsteciuk](https://github.com/bsteciuk))
#### **Juju**
* Added support for SAN entries in the master node certificate. ([#54234](https://github.com/kubernetes/kubernetes/pull/54234),[ @hyperbolic2346](https://github.com/hyperbolic2346))
* Add extra-args configs for scheduler and controller-manager to kubernetes-master charm ([#55185](https://github.com/kubernetes/kubernetes/pull/55185),[ @Cynerva](https://github.com/Cynerva))
* Add support for RBAC ([#53820](https://github.com/kubernetes/kubernetes/pull/53820),[ @ktsakalozos](https://github.com/ktsakalozos))
* Fixed iptables FORWARD policy for Docker 1.13 in kubernetes-worker charm ([#54796](https://github.com/kubernetes/kubernetes/pull/54796),[ @Cynerva](https://github.com/Cynerva))
* Upgrading the kubernetes-master units now results in staged upgrades just like the kubernetes-worker nodes. Use the upgrade action in order to continue the upgrade process on each unit such as juju run-action kubernetes-master/0 upgrade ([#55990](https://github.com/kubernetes/kubernetes/pull/55990),[ @hyperbolic2346](https://github.com/hyperbolic2346))
* Added extra_sans config option to kubeapi-load-balancer charm. This allows the user to specify extra SAN entries on the certificate generated for the load balancer. ([#54947](https://github.com/kubernetes/kubernetes/pull/54947),[ @hyperbolic2346](https://github.com/hyperbolic2346))
* Added extra-args configs to kubernetes-worker charm ([#55334](https://github.com/kubernetes/kubernetes/pull/55334),[ @Cynerva](https://github.com/Cynerva))
#### **Other**
* Base images have been bumped to Debian Stretch (9) ([#52744](https://github.com/kubernetes/kubernetes/pull/52744),[ @rphillips](https://github.com/rphillips))
* Upgraded to go1.9. ([#51375](https://github.com/kubernetes/kubernetes/pull/51375),[ @cblecker](https://github.com/cblecker))
* Add-on manager now supports HA masters. ([#55466](https://github.com/kubernetes/kubernetes/pull/55466),[ #55782](https://github.com/x13n),[ @x13n](https://github.com/x13n))
* Hyperkube can now run from a non-standard path. ([#54570](https://github.com/kubernetes/kubernetes/pull/54570))
#### **GCP**
* The service account made available on your nodes is now configurable. ([#52868](https://github.com/kubernetes/kubernetes/pull/52868),[ @ihmccreery](https://github.com/ihmccreery))
* GCE nodes with NVIDIA GPUs attached now expose nvidia.com/gpu as a resource instead of alpha.kubernetes.io/nvidia-gpu. ([#54826](https://github.com/kubernetes/kubernetes/pull/54826),[ @mindprince](https://github.com/mindprince))
* Docker's live-restore on COS/ubuntu can now be disabled ([#55260](https://github.com/kubernetes/kubernetes/pull/55260),[ @yujuhong](https://github.com/yujuhong))
* Metadata concealment is now controlled by the ENABLE_METADATA_CONCEALMENT env var. See cluster/gce/config-default.sh for more info. ([#54150](https://github.com/kubernetes/kubernetes/pull/54150),[ @ihmccreery](https://github.com/ihmccreery))
* Masquerading rules are now added by default to GCE/GKE ([#55178](https://github.com/kubernetes/kubernetes/pull/55178),[ @dnardo](https://github.com/dnardo))
* Fixed master startup issues with concurrent iptables invocations. ([#55945](https://github.com/kubernetes/kubernetes/pull/55945),[ @x13n](https://github.com/x13n))
* Fixed issue deleting internal load balancers when the firewall resource may not exist. ([#53450](https://github.com/kubernetes/kubernetes/pull/53450),[ @nicksardo](https://github.com/nicksardo))
### **Instrumentation**
#### **Audit**
* Adjust batching audit webhook default parameters: increase queue size, batch size, and initial backoff. Add throttling to the batching audit webhook. Default rate limit is 10 QPS. ([#53417](https://github.com/kubernetes/kubernetes/pull/53417),[ @crassirostris](https://github.com/crassirostris))
* These parameters are also now configurable. ([#56638](https://github.com/kubernetes/kubernetes/pull/56638), [@crassirostris](https://github.com/crassirostris))
#### **Other**
* Fix a typo in prometheus-to-sd configuration, that drops some stackdriver metrics. ([#56473](https://github.com/kubernetes/kubernetes/pull/56473),[ @loburm](https://github.com/loburm))
* [fluentd-elasticsearch addon] Elasticsearch and Kibana are updated to version 5.6.4 ([#55400](https://github.com/kubernetes/kubernetes/pull/55400),[ @mrahbar](https://github.com/mrahbar))
* fluentd now supports CRI log format. ([#54777](https://github.com/kubernetes/kubernetes/pull/54777),[ @Random-Liu](https://github.com/Random-Liu))
* Bring all prom-to-sd container to the same image version ([#54583](https://github.com/kubernetes/kubernetes/pull/54583))
* Reduce log noise produced by prometheus-to-sd, by bumping it to version 0.2.2. ([#54635](https://github.com/kubernetes/kubernetes/pull/54635),[ @loburm](https://github.com/loburm))
* [fluentd-elasticsearch addon] Elasticsearch service name can be overridden via env variable ELASTICSEARCH_SERVICE_NAME ([#54215](https://github.com/kubernetes/kubernetes/pull/54215),[ @mrahbar](https://github.com/mrahbar))
### **Multicluster**
#### **Federation**
* Kubefed init now supports --imagePullSecrets and --imagePullPolicy, making it possible to use private registries. ([#50740](https://github.com/kubernetes/kubernetes/pull/50740),[ @dixudx](https://github.com/dixudx))
* Updated cluster printer to enable --show-labels ([#53771](https://github.com/kubernetes/kubernetes/pull/53771),[ @dixudx](https://github.com/dixudx))
* Kubefed init now supports --nodeSelector, enabling you to determine on what node the controller will be installed. ([#50749](https://github.com/kubernetes/kubernetes/pull/50749),[ @dixudx](https://github.com/dixudx))
### **Network**
#### **IPv6**
* [alpha] IPv6 support has been added. Notable IPv6 support details include:
* Support for IPv6-only Kubernetes cluster deployments. **<span style="text-decoration:underline;">Note:</span>** This feature does not provide dual-stack support.
* Support for IPv6 Kubernetes control and data planes.
* Support for Kubernetes IPv6 cluster deployments using kubeadm.
* Support for the iptables kube-proxy backend using ip6tables.
* Relies on CNI 0.6.0 binaries for IPv6 pod networking.
* Adds IPv6 support for kube-dns using SRV records.
* Caveats
* Only the CNI bridge and local-ipam plugins have been tested for the alpha release, although other CNI plugins do support IPv6.
* HostPorts are not supported.
* An IPv6 network mask for pod or cluster cidr network must be /66 or longer. For example: 2001:db1::/66, 2001:dead:beef::/76, 2001:cafe::/118 are supported. 2001:db1::/64 is not supported
* For details, see [the complete list of merged pull requests for IPv6 support](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Amerged+label%3Aarea%2Fipv6).
#### **IPVS**
* You can now use the --cleanup-ipvs flag to tell kube-proxy whether to flush all existing ipvs rules in on startup ([#56036](https://github.com/kubernetes/kubernetes/pull/56036),[ @m1093782566](https://github.com/m1093782566))
* Graduate kube-proxy IPVS mode to beta. ([#56623](https://github.com/kubernetes/kubernetes/pull/56623), [@m1093782566](https://github.com/m1093782566))
#### **Kube-Proxy**
* Added iptables rules to allow Pod traffic even when default iptables policy is to reject. ([#52569](https://github.com/kubernetes/kubernetes/pull/52569),[ @tmjd](https://github.com/tmjd))
* You can once again use 0 values for conntrack min, max, max per core, tcp close wait timeout, and tcp established timeout; this functionality was broken in 1.8. ([#55261](https://github.com/kubernetes/kubernetes/pull/55261),[ @ncdc](https://github.com/ncdc))
#### **CoreDNS**
* You now have the option to use CoreDNS instead of KubeDNS. To install CoreDNS instead of kube-dns, set CLUSTER_DNS_CORE_DNS to 'true'. This support is experimental. ([#52501](https://github.com/kubernetes/kubernetes/pull/52501),[ @rajansandeep](https://github.com/rajansandeep)) ([#55728](https://github.com/kubernetes/kubernetes/pull/55728),[ @rajansandeep](https://github.com/rajansandeep))
#### **Other**
* Pod addresses will now be removed from the list of endpoints when the pod is in graceful termination. ([#54828](https://github.com/kubernetes/kubernetes/pull/54828),[ @freehan](https://github.com/freehan))
* You can now use a new supported service annotation for AWS clusters, `service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy`, which lets you specify which [predefined AWS SSL policy](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html) you would like to use. ([#54507](https://github.com/kubernetes/kubernetes/pull/54507),[ @micahhausler](https://github.com/micahhausler))
* Termination grace period for the calico/node add-on DaemonSet has been eliminated, reducing downtime during a rolling upgrade or deletion. ([#55015](https://github.com/kubernetes/kubernetes/pull/55015),[ @fasaxc](https://github.com/fasaxc))
* Fixed bad conversion in host port chain name generating func which led to some unreachable host ports. ([#55153](https://github.com/kubernetes/kubernetes/pull/55153),[ @chenchun](https://github.com/chenchun))
* Fixed IPVS availability check ([#51874](https://github.com/kubernetes/kubernetes/pull/51874),[ @vfreex](https://github.com/vfreex))
* The output for kubectl describe networkpolicy * has been enhanced to be more useful. ([#46951](https://github.com/kubernetes/kubernetes/pull/46951),[ @aanm](https://github.com/aanm))
* Kernel modules are now loaded automatically inside a kube-proxy pod ([#52003](https://github.com/kubernetes/kubernetes/pull/52003),[ @vfreex](https://github.com/vfreex))
* Improve resilience by annotating kube-dns addon with podAntiAffinity to prefer scheduling on different nodes. ([#52193](https://github.com/kubernetes/kubernetes/pull/52193),[ @StevenACoffman](https://github.com/StevenACoffman))
* [alpha] Added DNSConfig field to PodSpec. "None" mode for DNSPolicy is now supported. ([#55848](https://github.com/kubernetes/kubernetes/pull/55848),[ @MrHohn](https://github.com/MrHohn))
* You can now add "options" to the host's /etc/resolv.conf (or --resolv-conf), and they will be copied into pod's resolv.conf when dnsPolicy is Default. Being able to customize options is important because it is common to leverage options to fine-tune the behavior of DNS client. ([#54773](https://github.com/kubernetes/kubernetes/pull/54773),[ @phsiao](https://github.com/phsiao))
* Fixed a bug so that the service controller no longer retries if doNotRetry service update fails. ([#54184](https://github.com/kubernetes/kubernetes/pull/54184),[ @MrHohn](https://github.com/MrHohn))
* Added --no-negcache flag to kube-dns to prevent caching of NXDOMAIN responses. ([#53604](https://github.com/kubernetes/kubernetes/pull/53604),[ @cblecker](https://github.com/cblecker))
### **Node**
#### **Pod API**
* A single value in metadata.annotations/metadata.labels can now be passed into the containers via the Downward API. ([#55902](https://github.com/kubernetes/kubernetes/pull/55902),[ @yguo0905](https://github.com/yguo0905))
* Pods will no longer briefly transition to a "Pending" state during the deletion process. ([#54593](https://github.com/kubernetes/kubernetes/pull/54593),[ @dashpole](https://github.com/dashpole))
* Added pod-level local ephemeral storage metric to the Summary API. Pod-level ephemeral storage reports the total filesystem usage for the containers and emptyDir volumes in the measured Pod. ([#55447](https://github.com/kubernetes/kubernetes/pull/55447),[ @jingxu97](https://github.com/jingxu97))
#### **Hardware Accelerators**
* Kubelet now exposes metrics for NVIDIA GPUs attached to the containers. ([#55188](https://github.com/kubernetes/kubernetes/pull/55188),[ @mindprince](https://github.com/mindprince))
* The device plugin Alpha API no longer supports returning artifacts per device as part of AllocateResponse. ([#53031](https://github.com/kubernetes/kubernetes/pull/53031),[ @vishh](https://github.com/vishh))
* Fix to ignore extended resources that are not registered with kubelet during container resource allocation. ([#53547](https://github.com/kubernetes/kubernetes/pull/53547),[ @jiayingz](https://github.com/jiayingz))
#### **Container Runtime**
* [alpha] [cri-tools](https://github.com/kubernetes-incubator/cri-tools): CLI and validation tools for CRI is now v1.0.0-alpha.0. This release mainly focuses on UX improvements. [[@feiskyer](https://github.com/feiskyer)]
* Make crictl command more user friendly and add more subcommands.
* Integrate with CRI verbose option to provide extra debug information.
* Update CRI to kubernetes v1.9.
* Bug fixes in validation test suites.
* [beta] [cri-containerd](https://github.com/kubernetes-incubator/cri-containerd): CRI implementation for containerd is now v1.0.0-beta.0, [[@Random-Liu](https://github.com/Random-Liu)]
* This release supports Kubernetes 1.9+ and containerd v1.0.0+.
* Pass all Kubernetes 1.9 e2e test, node e2e test and CRI validation tests.
* [Kube-up.sh integration](https://github.com/kubernetes-incubator/cri-containerd/blob/master/docs/kube-up.md).
* [Full crictl integration including CRI verbose option.](https://github.com/kubernetes-incubator/cri-containerd/blob/master/docs/crictl.md)
* Integration with cadvisor to provide better summary api support.
* [stable] [cri-o](https://github.com/kubernetes-incubator/cri-o): CRI implementation for OCI-based runtimes is now v1.9. [[@mrunalp](https://github.com/mrunalp)]
* Pass all the Kubernetes 1.9 end-to-end test suites and now gating PRs as well
* Pass all the CRI validation tests
* Release has been focused on bug fixes, stability and performance with runc and Clear Containers
* Minikube integration
* [stable] [frakti](https://github.com/kubernetes/frakti): CRI implementation for hypervisor-based runtimes is now v1.9. [[@resouer](https://github.com/resouer)]
* Added ARM64 release. Upgraded to CNI 0.6.0, added block device as Pod volume mode. Fixed CNI plugin compatibility.
* Passed all CRI validation conformance tests and node end-to-end conformance tests.
* [alpha] [rktlet](https://github.com/kubernetes-incubator/rktlet): CRI implementation for the rkt runtime is now v0.1.0. [[@iaguis](https://github.com/iaguis)]
* This is the first release of rktlet and it implements support for the CRI including fetching images, running pods, CNI networking, logging and exec.
This release passes 129/145 Kubernetes e2e conformance tests.
* Container Runtime Interface API change. [[@yujuhong](https://github.com/yujuhong)]
* A new field is added to CRI container log format to support splitting a long log line into multiple lines. ([#55922](https://github.com/kubernetes/kubernetes/pull/55922), [@Random-Liu](https://github.com/Random-Liu))
* CRI now supports debugging via a verbose option for status functions. ([#53965](https://github.com/kubernetes/kubernetes/pull/53965), [@Random-Liu](https://github.com/Random-Liu))
* Kubelet can now provide full summary api support for the CRI container runtime, with the exception of container log stats. ([#55810](https://github.com/kubernetes/kubernetes/pull/55810), [@abhi](https://github.com/abhi))
* CRI now uses the correct localhost seccomp path when provided with input in the format of localhost//profileRoot/profileName. ([#55450](https://github.com/kubernetes/kubernetes/pull/55450), [@feiskyer](https://github.com/feiskyer))
#### **Kubelet**
* The EvictionHard, EvictionSoft, EvictionSoftGracePeriod, EvictionMinimumReclaim, SystemReserved, and KubeReserved fields in the KubeletConfiguration object (`kubeletconfig/v1alpha1`) are now of type map[string]string, which facilitates writing JSON and YAML files. ([#54823](https://github.com/kubernetes/kubernetes/pull/54823),[ @mtaufen](https://github.com/mtaufen))
* Relative paths in the Kubelet's local config files (`--init-config-dir`) will now be resolved relative to the location of the containing files. ([#55648](https://github.com/kubernetes/kubernetes/pull/55648),[ @mtaufen](https://github.com/mtaufen))
* It is now possible to set multiple manifest URL headers with the kubelet's `--manifest-url-header` flag. Multiple headers for the same key will be added in the order provided. The ManifestURLHeader field in KubeletConfiguration object (kubeletconfig/v1alpha1) is now a map[string][]string, which facilitates writing JSON and YAML files. ([#54643](https://github.com/kubernetes/kubernetes/pull/54643),[ @mtaufen](https://github.com/mtaufen))
* The Kubelet's feature gates are now specified as a map when provided via a JSON or YAML KubeletConfiguration, rather than as a string of key-value pairs, making them less awkward for users. ([#53025](https://github.com/kubernetes/kubernetes/pull/53025),[ @mtaufen](https://github.com/mtaufen))
##### **Other**
* Fixed a performance issue ([#51899](https://github.com/kubernetes/kubernetes/pull/51899)) identified in large-scale clusters when deleting thousands of pods simultaneously across hundreds of nodes, by actively removing containers of deleted pods, rather than waiting for periodic garbage collection and batching resulting pod API deletion requests. ([#53233](https://github.com/kubernetes/kubernetes/pull/53233),[ @dashpole](https://github.com/dashpole))
* Problems deleting local static pods have been resolved. ([#48339](https://github.com/kubernetes/kubernetes/pull/48339),[ @dixudx](https://github.com/dixudx))
* CRI now only calls UpdateContainerResources when cpuset is set. ([#53122](https://github.com/kubernetes/kubernetes/pull/53122),[ @resouer](https://github.com/resouer))
* Containerd monitoring is now supported. ([#56109](https://github.com/kubernetes/kubernetes/pull/56109),[ @dashpole](https://github.com/dashpole))
* deviceplugin has been extended to more gracefully handle the full device plugin lifecycle, including: ([#55088](https://github.com/kubernetes/kubernetes/pull/55088),[ @jiayingz](https://github.com/jiayingz))
* Kubelet now uses an explicit cm.GetDevicePluginResourceCapacity() function that makes it possible to more accurately determine what resources are inactive and return a more accurate view of available resources.
* Extends the device plugin checkpoint data to record registered resources so that we can finish resource removing devices even upon kubelet restarts.
* Passes sourcesReady from kubelet to the device plugin to avoid removing inactive pods during the grace period of kubelet restart.
* Extends the gpu_device_plugin e2e_node test to verify that scheduled pods can continue to run even after a device plugin deletion and kubelet restart.
* The NodeController no longer supports kubelet 1.2. ([#48996](https://github.com/kubernetes/kubernetes/pull/48996),[ @k82cn](https://github.com/k82cn))
* Kubelet now provides more specific events via FailedSync when unable to sync a pod. ([#53857](https://github.com/kubernetes/kubernetes/pull/53857),[ @derekwaynecarr](https://github.com/derekwaynecarr))
* You can now disable AppArmor by setting the AppArmor profile to unconfined. ([#52395](https://github.com/kubernetes/kubernetes/pull/52395),[ @dixudx](https://github.com/dixudx))
* ImageGCManage now consumes ImageFS stats from StatsProvider rather than cadvisor. ([#53094](https://github.com/kubernetes/kubernetes/pull/53094),[ @yguo0905](https://github.com/yguo0905))
* Hyperkube now supports the support --experimental-dockershim kubelet flag. ([#54508](https://github.com/kubernetes/kubernetes/pull/54508),[ @ivan4th](https://github.com/ivan4th))
* Kubelet no longer removes default labels from Node API objects on startup ([#54073](https://github.com/kubernetes/kubernetes/pull/54073),[ @liggitt](https://github.com/liggitt))
* The overlay2 container disk metrics for Docker and CRI-O now work properly. ([#54827](https://github.com/kubernetes/kubernetes/pull/54827),[ @dashpole](https://github.com/dashpole))
* Removed docker dependency during kubelet start up. ([#54405](https://github.com/kubernetes/kubernetes/pull/54405),[ @resouer](https://github.com/resouer))
* Added Windows support to the system verification check. ([#53730](https://github.com/kubernetes/kubernetes/pull/53730),[ @bsteciuk](https://github.com/bsteciuk))
* Kubelet no longer removes unregistered extended resource capacities from node status; cluster admins will have to manually remove extended resources exposed via device plugins when they the remove plugins themselves. ([#53353](https://github.com/kubernetes/kubernetes/pull/53353),[ @jiayingz](https://github.com/jiayingz))
* The stats summary network value now takes into account multiple network interfaces, and not just eth0. ([#52144](https://github.com/kubernetes/kubernetes/pull/52144),[ @andyxning](https://github.com/andyxning))
* Base images have been bumped to Debian Stretch (9). ([#52744](https://github.com/kubernetes/kubernetes/pull/52744),[ @rphillips](https://github.com/rphillips))
### **OpenStack**
* OpenStack Cinder support has been improved:
* Cinder version detection now works properly. ([#53115](https://github.com/kubernetes/kubernetes/pull/53115),[ @FengyunPan](https://github.com/FengyunPan))
* The OpenStack cloud provider now supports Cinder v3 API. ([#52910](https://github.com/kubernetes/kubernetes/pull/52910),[ @FengyunPan](https://github.com/FengyunPan))
* Load balancing is now more flexible:
* The OpenStack LBaaS v2 Provider is now [configurable](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#openstack). ([#54176](https://github.com/kubernetes/kubernetes/pull/54176),[ @gonzolino](https://github.com/gonzolino))
* OpenStack Octavia v2 is now supported as a load balancer provider in addition to the existing support for the Neutron LBaaS V2 implementation. Neutron LBaaS V1 support has been removed. ([#55393](https://github.com/kubernetes/kubernetes/pull/55393),[ @jamiehannaford](https://github.com/jamiehannaford))
* OpenStack security group support has been beefed up ([#50836](https://github.com/kubernetes/kubernetes/pull/50836),[ @FengyunPan](https://github.com/FengyunPan)):
* Kubernetes will now automatically determine the security group for the node
* Nodes can now belong to multiple security groups
### **Scheduling**
#### **Hardware Accelerators**
* Add ExtendedResourceToleration admission controller. This facilitates creation of dedicated nodes with extended resources. If operators want to create dedicated nodes with extended resources (such as GPUs, FPGAs, and so on), they are expected to taint the node with extended resource name as the key. This admission controller, if enabled, automatically adds tolerations for such taints to pods requesting extended resources, so users don't have to manually add these tolerations. ([#55839](https://github.com/kubernetes/kubernetes/pull/55839),[ @mindprince](https://github.com/mindprince))
#### **Other**
* Scheduler cache ignores updates to an assumed pod if updates are limited to pod annotations. ([#54008](https://github.com/kubernetes/kubernetes/pull/54008),[ @yguo0905](https://github.com/yguo0905))
* Issues with namespace deletion have been resolved. ([#53720](https://github.com/kubernetes/kubernetes/pull/53720),[ @shyamjvs](https://github.com/shyamjvs)) ([#53793](https://github.com/kubernetes/kubernetes/pull/53793),[ @wojtek-t](https://github.com/wojtek-t))
* Pod preemption has been improved.
* Now takes PodDisruptionBudget into account. ([#56178](https://github.com/kubernetes/kubernetes/pull/56178),[ @bsalamat](https://github.com/bsalamat))
* Nominated pods are taken into account during scheduling to avoid starvation of higher priority pods. ([#55933](https://github.com/kubernetes/kubernetes/pull/55933),[ @bsalamat](https://github.com/bsalamat))
* Fixed 'Schedulercache is corrupted' error in kube-scheduler ([#55262](https://github.com/kubernetes/kubernetes/pull/55262),[ @liggitt](https://github.com/liggitt))
* The kube-scheduler command now supports a --config flag which is the location of a file containing a serialized scheduler configuration. Most other kube-scheduler flags are now deprecated. ([#52562](https://github.com/kubernetes/kubernetes/pull/52562),[ @ironcladlou](https://github.com/ironcladlou))
* A new scheduling queue helps schedule the highest priority pending pod first. ([#55109](https://github.com/kubernetes/kubernetes/pull/55109),[ @bsalamat](https://github.com/bsalamat))
* A Pod can now listen to the same port on multiple IP addresses. ([#52421](https://github.com/kubernetes/kubernetes/pull/52421),[ @WIZARD-CXY](https://github.com/WIZARD-CXY))
* Object count quotas supported on all standard resources using count/<resource>.<group> syntax ([#54320](https://github.com/kubernetes/kubernetes/pull/54320),[ @derekwaynecarr](https://github.com/derekwaynecarr))
* Apply algorithm in scheduler by feature gates. ([#52723](https://github.com/kubernetes/kubernetes/pull/52723),[ @k82cn](https://github.com/k82cn))
* A new priority function ResourceLimitsPriorityMap (disabled by default and behind alpha feature gate and not part of the scheduler's default priority functions list) that assigns a lowest possible score of 1 to a node that satisfies one or both of input pod's cpu and memory limits, mainly to break ties between nodes with same scores. ([#55906](https://github.com/kubernetes/kubernetes/pull/55906),[ @aveshagarwal](https://github.com/aveshagarwal))
* Kubelet evictions now take pod priority into account ([#53542](https://github.com/kubernetes/kubernetes/pull/53542),[ @dashpole](https://github.com/dashpole))
* PodTolerationRestriction admisson plugin: if namespace level tolerations are empty, now they override cluster level tolerations. ([#54812](https://github.com/kubernetes/kubernetes/pull/54812),[ @aveshagarwal](https://github.com/aveshagarwal))
### **Storage**
* [stable] `PersistentVolume` and `PersistentVolumeClaim` objects must now have a capacity greater than zero.
* [stable] Mutation of `PersistentVolumeSource` after creation is no longer allowed
* [alpha] Deletion of `PersistentVolumeClaim` objects that are in use by a pod no longer permitted (if alpha feature is enabled).
* [alpha] Container Storage Interface
* New CSIVolumeSource enables Kubernetes to use external CSI drivers to provision, attach, and mount volumes.
* [alpha] Raw block volumes
* Support for surfacing volumes as raw block devices added to Kubernetes storage system.
* Only Fibre Channel volume plugin supports exposes this functionality, in this release.
* [alpha] Volume resizing
* Added file system resizing for the following volume plugins: GCE PD, Ceph RBD, AWS EBS, OpenStack Cinder
* [alpha] Topology Aware Volume Scheduling
* Improved volume scheduling for Local PersistentVolumes, by allowing the scheduler to make PersistentVolume binding decisions while respecting the Pod's scheduling requirements.
* Dynamic provisioning is not supported with this feature yet.
* [alpha] Containerized mount utilities
* Allow mount utilities, used to mount volumes, to run inside a container instead of on the host.
* Bug Fixes
* ScaleIO volume plugin is no longer dependent on the drv_cfg binary, so a Kubernetes cluster can easily run a containerized kubelet. ([#54956](https://github.com/kubernetes/kubernetes/pull/54956),[ @vladimirvivien](https://github.com/vladimirvivien))
* AWS EBS Volumes are detached from stopped AWS nodes. ([#55893](https://github.com/kubernetes/kubernetes/pull/55893),[ @gnufied](https://github.com/gnufied))
* AWS EBS volumes are detached if attached to a different node than expected. ([#55491](https://github.com/kubernetes/kubernetes/pull/55491),[ @gnufied](https://github.com/gnufied))
* PV Recycle now works in environments that use architectures other than x86. ([#53958](https://github.com/kubernetes/kubernetes/pull/53958),[ @dixudx](https://github.com/dixudx))
* Pod Security Policy can now manage access to specific FlexVolume drivers.([#53179](https://github.com/kubernetes/kubernetes/pull/53179),[ @wanghaoran1988](https://github.com/wanghaoran1988))
* To prevent unauthorized access to CHAP Secrets, you can now set the secretNamespace storage class parameters for the following volume types:
* ScaleIO; StoragePool and ProtectionDomain attributes no longer default to the value default. ([#54013](https://github.com/kubernetes/kubernetes/pull/54013),[ @vladimirvivien](https://github.com/vladimirvivien))
* RBD Persistent Volume Sources ([#54302](https://github.com/kubernetes/kubernetes/pull/54302),[ @sbezverk](https://github.com/sbezverk))
* iSCSI Persistent Volume Sources ([#51530](https://github.com/kubernetes/kubernetes/pull/51530),[ @rootfs](https://github.com/rootfs))
* In GCE multizonal clusters, `PersistentVolume` objects will no longer be dynamically provisioned in zones without nodes. ([#52322](https://github.com/kubernetes/kubernetes/pull/52322),[ @davidz627](https://github.com/davidz627))
* Multi Attach PVC errors and events are now more useful and less noisy. ([#53401](https://github.com/kubernetes/kubernetes/pull/53401),[ @gnufied](https://github.com/gnufied))
* The compute-rw scope has been removed from GCE nodes ([#53266](https://github.com/kubernetes/kubernetes/pull/53266),[ @mikedanese](https://github.com/mikedanese))
* Updated vSphere cloud provider to support k8s cluster spread across multiple vCenters ([#55845](https://github.com/kubernetes/kubernetes/pull/55845),[ @rohitjogvmw](https://github.com/rohitjogvmw))
* vSphere: Fix disk is not getting detached when PV is provisioned on clustered datastore. ([#54438](https://github.com/kubernetes/kubernetes/pull/54438),[ @pshahzeb](https://github.com/pshahzeb))
* If a non-absolute mountPath is passed to the kubelet, it must now be prefixed with the appropriate root path. ([#55665](https://github.com/kubernetes/kubernetes/pull/55665),[ @brendandburns](https://github.com/brendandburns))
## External Dependencies
* The supported etcd server version is **3.1.10**, as compared to 3.0.17 in v1.8 ([#49393](https://github.com/kubernetes/kubernetes/pull/49393),[ @hongchaodeng](https://github.com/hongchaodeng))
* The validated docker versions are the same as for v1.8: **1.11.2 to 1.13.1 and 17.03.x**
* The Go version was upgraded from go1.8.3 to **go1.9.2** ([#51375](https://github.com/kubernetes/kubernetes/pull/51375),[ @cblecker](https://github.com/cblecker))
* The minimum supported go version bumps to 1.9.1. ([#55301](https://github.com/kubernetes/kubernetes/pull/55301),[ @xiangpengzhao](https://github.com/xiangpengzhao))
* Kubernetes has been upgraded to go1.9.2 ([#55420](https://github.com/kubernetes/kubernetes/pull/55420),[ @cblecker](https://github.com/cblecker))
* CNI was upgraded to **v0.6.0** ([#51250](https://github.com/kubernetes/kubernetes/pull/51250),[ @dixudx](https://github.com/dixudx))
* The dashboard add-on has been updated to [v1.8.0](https://github.com/kubernetes/dashboard/releases/tag/v1.8.0). ([#53046](https://github.com/kubernetes/kubernetes/pull/53046), [@maciaszczykm](https://github.com/maciaszczykm))
* Heapster has been updated to [v1.5.0](https://github.com/kubernetes/heapster/releases/tag/v1.5.0). ([#57046](https://github.com/kubernetes/kubernetes/pull/57046), [@piosz](https://github.com/piosz))
* Cluster Autoscaler has been updated to [v1.1.0](https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.1.0). ([#56969](https://github.com/kubernetes/kubernetes/pull/56969), [@mwielgus](https://github.com/mwielgus))
* Update kube-dns 1.14.7 ([#54443](https://github.com/kubernetes/kubernetes/pull/54443),[ @bowei](https://github.com/bowei))
* Update influxdb to v1.3.3 and grafana to v4.4.3 ([#53319](https://github.com/kubernetes/kubernetes/pull/53319),[ @kairen](https://github.com/kairen))
- [v1.9.0-beta.2](#v190-beta2)
- [v1.9.0-beta.1](#v190-beta1)
- [v1.9.0-alpha.3](#v190-alpha3)
- [v1.9.0-alpha.2](#v190-alpha2)
- [v1.9.0-alpha.1](#v190-alpha1)
# v1.9.0-beta.2
[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.9/examples)

6
CONTRIBUTING.md
View File

@@ -1,9 +1,7 @@
# Contributing
Information about contributing to the
[kubernetes code repo](README.md) lives in the
[kubernetes community repo](https://github.com/kubernetes/community)
(it's a big topic).
Welcome to Kubernetes! If you are interested in contributing to the [Kubernetes code repo](README.md) then checkout the [Contributor's Guide](https://git.k8s.io/community/contributors/guide/)
The [Kubernetes community repo](https://github.com/kubernetes/community) contains information on how the community is organized and other information that is pertinent to contributing.
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/CONTRIBUTING.md?pixel)]()

904
Godeps/Godeps.json generated
View File

File diff suppressed because it is too large Load Diff

10701
Godeps/LICENSES generated
View File

File diff suppressed because it is too large Load Diff

9
OWNERS_ALIASES
View File

@@ -17,7 +17,6 @@ aliases:
- adohe
- brendandburns
- deads2k
- fabianofranz
- janetkuo
- liggitt
- pwittrock
@@ -30,7 +29,6 @@ aliases:
- dshulyak
- eparis
- ericchiang
- fabianofranz
- ghodss
- mengqiy
- rootfs
@@ -101,6 +99,12 @@ aliases:
sig-apps-api-approvers:
- erictune
- smarterclayton
sig-autoscaling-maintainers:
- aleksandra-malinowska
- bskiba
- DirectXMan12
- MaciekPytel
- mwielgus
milestone-maintainers:
- lavalamp
- deads2k
@@ -121,7 +125,6 @@ aliases:
- slack
- colemickens
- foxish
- fabianofranz
- pwittrock
- AdoHe
- lukemarsden

6
README.md
View File

@@ -55,7 +55,7 @@ $ cd kubernetes
$ make quick-release
```
If you are less impatient, head over to the [developer's documentation].
For the full story, head over to the [developer's documentation].
## Support
@@ -71,7 +71,7 @@ That said, if you have questions, reach out to us
[communication]: https://github.com/kubernetes/community/blob/master/communication.md
[community repository]: https://github.com/kubernetes/community
[containerized applications]: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
[developer's documentation]: https://github.com/kubernetes/community/tree/master/contributors/devel
[developer's documentation]: https://github.com/kubernetes/community/tree/master/contributors/devel#readme
[Docker environment]: https://docs.docker.com/engine
[Go environment]: https://golang.org/doc/install
[GoDoc]: https://godoc.org/k8s.io/kubernetes
@@ -81,6 +81,6 @@ That said, if you have questions, reach out to us
[Scalable Microservices with Kubernetes]: https://www.udacity.com/course/scalable-microservices-with-kubernetes--ud615
[Submit Queue]: http://submit-queue.k8s.io/#/ci
[Submit Queue Widget]: http://submit-queue.k8s.io/health.svg?v=1
[troubleshooting guide]: https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/
[troubleshooting guide]: https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/README.md?pixel)]()

325
Vagrantfile vendored
View File

@@ -1,325 +0,0 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
# Require a recent version of vagrant otherwise some have reported errors setting host names on boxes
Vagrant.require_version ">= 1.7.4"
if ARGV.first == "up" && ENV['USING_KUBE_SCRIPTS'] != 'true'
raise Vagrant::Errors::VagrantError.new, <<END
Calling 'vagrant up' directly is not supported. Instead, please run the following:
export KUBERNETES_PROVIDER=vagrant
export VAGRANT_DEFAULT_PROVIDER=providername
./cluster/kube-up.sh
END
end
# The number of nodes to provision
$num_node = (ENV['NUM_NODES'] || 1).to_i
# ip configuration
$master_ip = ENV['MASTER_IP']
$node_ip_base = ENV['NODE_IP_BASE'] || ""
$node_ips = $num_node.times.collect { |n| $node_ip_base + "#{n+3}" }
# Determine the OS platform to use
$kube_os = ENV['KUBERNETES_OS'] || "fedora"
# Determine whether vagrant should use nfs to sync folders
$use_nfs = ENV['KUBERNETES_VAGRANT_USE_NFS'] == 'true'
# Determine whether vagrant should use rsync to sync folders
$use_rsync = ENV['KUBERNETES_VAGRANT_USE_RSYNC'] == 'true'
# To override the vagrant provider, use (e.g.):
# KUBERNETES_PROVIDER=vagrant VAGRANT_DEFAULT_PROVIDER=... .../cluster/kube-up.sh
# To override the box, use (e.g.):
# KUBERNETES_PROVIDER=vagrant KUBERNETES_BOX_NAME=... .../cluster/kube-up.sh
# You can specify a box version:
# KUBERNETES_PROVIDER=vagrant KUBERNETES_BOX_NAME=... KUBERNETES_BOX_VERSION=... .../cluster/kube-up.sh
# You can specify a box location:
# KUBERNETES_PROVIDER=vagrant KUBERNETES_BOX_NAME=... KUBERNETES_BOX_URL=... .../cluster/kube-up.sh
# KUBERNETES_BOX_URL and KUBERNETES_BOX_VERSION will be ignored unless
# KUBERNETES_BOX_NAME is set
# Default OS platform to provider/box information
$kube_provider_boxes = {
:parallels => {
'fedora' => {
# :box_url and :box_version are optional (and mutually exclusive);
# if :box_url is omitted the box will be retrieved by :box_name (and
# :box_version if provided) from
# http://atlas.hashicorp.com/boxes/search (formerly
# http://vagrantcloud.com/); this allows you override :box_name with
# your own value so long as you provide :box_url; for example, the
# "official" name of this box is "rickard-von-essen/
# opscode_fedora-20", but by providing the URL and our own name, we
# make it appear as yet another provider under the "kube-fedora22"
# box
:box_name => 'kube-fedora23',
:box_url => 'https://opscode-vm-bento.s3.amazonaws.com/vagrant/parallels/opscode_fedora-23_chef-provisionerless.box'
}
},
:virtualbox => {
'fedora' => {
:box_name => 'kube-fedora23',
:box_url => 'https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_fedora-23_chef-provisionerless.box'
}
},
:libvirt => {
'fedora' => {
:box_name => 'kube-fedora23',
:box_url => 'https://dl.fedoraproject.org/pub/fedora/linux/releases/23/Cloud/x86_64/Images/Fedora-Cloud-Base-Vagrant-23-20151030.x86_64.vagrant-libvirt.box'
}
},
:vmware_desktop => {
'fedora' => {
:box_name => 'kube-fedora23',
:box_url => 'https://opscode-vm-bento.s3.amazonaws.com/vagrant/vmware/opscode_fedora-23_chef-provisionerless.box'
}
},
:vsphere => {
'fedora' => {
:box_name => 'vsphere-dummy',
:box_url => 'https://github.com/deromka/vagrant-vsphere/blob/master/vsphere-dummy.box?raw=true'
}
}
}
# Give access to all physical cpu cores
# Previously cargo-culted from here:
# http://www.stefanwrobel.com/how-to-make-vagrant-performance-not-suck
# Rewritten to actually determine the number of hardware cores instead of assuming
# that the host has hyperthreading enabled.
host = RbConfig::CONFIG['host_os']
if host =~ /darwin/
$vm_cpus = `sysctl -n hw.physicalcpu`.to_i
elsif host =~ /linux/
#This should work on most processors, however it will fail on ones without the core id field.
#So far i have only seen this on a raspberry pi. which you probably don't want to run vagrant on anyhow...
#But just in case we'll default to the result of nproc if we get 0 just to be safe.
$vm_cpus = `cat /proc/cpuinfo | grep 'core id' | sort -u | wc -l`.to_i
if $vm_cpus < 1
$vm_cpus = `nproc`.to_i
end
else # sorry Windows folks, I can't help you
$vm_cpus = 2
end
# Give VM 1024MB of RAM by default
# In Fedora VM, tmpfs device is mapped to /tmp. tmpfs is given 50% of RAM allocation.
# When doing Salt provisioning, we copy approximately 200MB of content in /tmp before anything else happens.
# This causes problems if anything else was in /tmp or the other directories that are bound to tmpfs device (i.e /run, etc.)
$vm_master_mem = (ENV['KUBERNETES_MASTER_MEMORY'] || ENV['KUBERNETES_MEMORY'] || 1280).to_i
$vm_node_mem = (ENV['KUBERNETES_NODE_MEMORY'] || ENV['KUBERNETES_MEMORY'] || 2048).to_i
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
if Vagrant.has_plugin?("vagrant-proxyconf")
$http_proxy = ENV['KUBERNETES_HTTP_PROXY'] || ""
$https_proxy = ENV['KUBERNETES_HTTPS_PROXY'] || ""
$no_proxy = ENV['KUBERNETES_NO_PROXY'] || "127.0.0.1"
config.proxy.http = $http_proxy
config.proxy.https = $https_proxy
config.proxy.no_proxy = $no_proxy
end
# this corrects a bug in 1.8.5 where an invalid SSH key is inserted.
if Vagrant::VERSION == "1.8.5"
config.ssh.insert_key = false
end
def setvmboxandurl(config, provider)
if ENV['KUBERNETES_BOX_NAME'] then
config.vm.box = ENV['KUBERNETES_BOX_NAME']
if ENV['KUBERNETES_BOX_URL'] then
config.vm.box_url = ENV['KUBERNETES_BOX_URL']
end
if ENV['KUBERNETES_BOX_VERSION'] then
config.vm.box_version = ENV['KUBERNETES_BOX_VERSION']
end
else
config.vm.box = $kube_provider_boxes[provider][$kube_os][:box_name]
if $kube_provider_boxes[provider][$kube_os][:box_url] then
config.vm.box_url = $kube_provider_boxes[provider][$kube_os][:box_url]
end
if $kube_provider_boxes[provider][$kube_os][:box_version] then
config.vm.box_version = $kube_provider_boxes[provider][$kube_os][:box_version]
end
end
end
def customize_vm(config, vm_mem)
if $use_nfs then
config.vm.synced_folder ".", "/vagrant", nfs: true
elsif $use_rsync then
opts = {}
if ENV['KUBERNETES_VAGRANT_RSYNC_ARGS'] then
opts[:rsync__args] = ENV['KUBERNETES_VAGRANT_RSYNC_ARGS'].split(" ")
end
if ENV['KUBERNETES_VAGRANT_RSYNC_EXCLUDE'] then
opts[:rsync__exclude] = ENV['KUBERNETES_VAGRANT_RSYNC_EXCLUDE'].split(" ")
end
config.vm.synced_folder ".", "/vagrant", opts
end
# Try VMWare Fusion first (see
# https://docs.vagrantup.com/v2/providers/basic_usage.html)
config.vm.provider :vmware_fusion do |v, override|
setvmboxandurl(override, :vmware_desktop)
v.vmx['memsize'] = vm_mem
v.vmx['numvcpus'] = $vm_cpus
end
# configure libvirt provider
config.vm.provider :libvirt do |v, override|
setvmboxandurl(override, :libvirt)
v.memory = vm_mem
v.cpus = $vm_cpus
v.nested = true
v.volume_cache = 'none'
end
# Then try VMWare Workstation
config.vm.provider :vmware_workstation do |v, override|
setvmboxandurl(override, :vmware_desktop)
v.vmx['memsize'] = vm_mem
v.vmx['numvcpus'] = $vm_cpus
end
# Then try Parallels
config.vm.provider :parallels do |v, override|
setvmboxandurl(override, :parallels)
v.memory = vm_mem # v.customize ['set', :id, '--memsize', vm_mem]
v.cpus = $vm_cpus # v.customize ['set', :id, '--cpus', $vm_cpus]
# Don't attempt to update the Parallels tools on the image (this can
# be done manually if necessary)
v.update_guest_tools = false # v.customize ['set', :id, '--tools-autoupdate', 'off']
# Set up Parallels folder sharing to behave like VirtualBox (i.e.,
# mount the current directory as /vagrant and that's it)
v.customize ['set', :id, '--shf-guest', 'off']
v.customize ['set', :id, '--shf-guest-automount', 'off']
v.customize ['set', :id, '--shf-host', 'on']
# Synchronize VM clocks to host clock (Avoid certificate invalid issue)
v.customize ['set', :id, '--time-sync', 'on']
# Remove all auto-mounted "shared folders"; the result seems to
# persist between runs (i.e., vagrant halt && vagrant up)
override.vm.provision :shell, :inline => (%q{
set -ex
if [ -d /media/psf ]; then
for i in /media/psf/*; do
if [ -d "${i}" ]; then
umount "${i}" || true
rmdir -v "${i}"
fi
done
rmdir -v /media/psf
fi
exit
}).strip
end
# Then try vsphere
config.vm.provider :vsphere do |vsphere, override|
setvmboxandurl(override, :vsphere)
#config.vm.hostname = ENV['MASTER_NAME']
config.ssh.username = ENV['MASTER_USER']
config.ssh.password = ENV['MASTER_PASSWD']
config.ssh.pty = true
config.ssh.insert_key = true
#config.ssh.private_key_path = '~/.ssh/id_rsa_vsphere'
# Don't attempt to update the tools on the image (this can
# be done manually if necessary)
# vsphere.update_guest_tools = false # v.customize ['set', :id, '--tools-autoupdate', 'off']
# The vSphere host we're going to connect to
vsphere.host = ENV['VAGRANT_VSPHERE_URL']
# The ESX host for the new VM
vsphere.compute_resource_name = ENV['VAGRANT_VSPHERE_RESOURCE_POOL']
# The resource pool for the new VM
#vsphere.resource_pool_name = 'Comp'
# path to folder where new VM should be created, if not specified template's parent folder will be used
vsphere.vm_base_path = ENV['VAGRANT_VSPHERE_BASE_PATH']
# The template we're going to clone
vsphere.template_name = ENV['VAGRANT_VSPHERE_TEMPLATE_NAME']
# The name of the new machine
#vsphere.name = ENV['MASTER_NAME']
# vSphere login
vsphere.user = ENV['VAGRANT_VSPHERE_USERNAME']
# vSphere password
vsphere.password = ENV['VAGRANT_VSPHERE_PASSWORD']
# cpu count
vsphere.cpu_count = $vm_cpus
# memory in MB
vsphere.memory_mb = vm_mem
# If you don't have SSL configured correctly, set this to 'true'
vsphere.insecure = ENV['VAGRANT_VSPHERE_INSECURE']
end
# Don't attempt to update Virtualbox Guest Additions (requires gcc)
if Vagrant.has_plugin?("vagrant-vbguest") then
config.vbguest.auto_update = false
end
# Finally, fall back to VirtualBox
config.vm.provider :virtualbox do |v, override|
setvmboxandurl(override, :virtualbox)
v.memory = vm_mem # v.customize ["modifyvm", :id, "--memory", vm_mem]
v.cpus = $vm_cpus # v.customize ["modifyvm", :id, "--cpus", $vm_cpus]
# Use faster paravirtualized networking
v.customize ["modifyvm", :id, "--nictype1", "virtio"]
v.customize ["modifyvm", :id, "--nictype2", "virtio"]
end
end
# Kubernetes master
config.vm.define "master" do |c|
customize_vm c, $vm_master_mem
if ENV['KUBE_TEMP'] then
script = "#{ENV['KUBE_TEMP']}/master-start.sh"
c.vm.provision "shell", run: "always", path: script
end
c.vm.network "private_network", ip: "#{$master_ip}"
end
# Kubernetes node
$num_node.times do |n|
node_vm_name = "node-#{n+1}"
config.vm.define node_vm_name do |node|
customize_vm node, $vm_node_mem
node_ip = $node_ips[n]
if ENV['KUBE_TEMP'] then
script = "#{ENV['KUBE_TEMP']}/node-start-#{n}.sh"
node.vm.provision "shell", run: "always", path: script
end
node.vm.network "private_network", ip: "#{node_ip}"
end
end
end

2
api/openapi-spec/README.md
View File

@@ -4,7 +4,7 @@ This folder contains an [OpenAPI specification][openapi] for Kubernetes API.
## Vendor Extensions
Kuberntes extends OpenAPI using these extensions. Note the version that
Kubernetes extends OpenAPI using these extensions. Note the version that
extensions has been added.
### `x-kubernetes-group-version-kind`

45
api/openapi-spec/swagger.json generated
View File

@@ -70501,7 +70501,7 @@
"description": "Spec to control the desired behavior of rolling update.",
"properties": {
"maxSurge": {
"description": "The maximum number of pods that can be scheduled above the desired number of pods. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up. Defaults to 25%. Example: when this is set to 30%, the new RC can be scaled up immediately when the rolling update starts, such that the total number of old and new pods do not exceed 130% of desired pods. Once old pods have been killed, new RC can be scaled up further, ensuring that total number of pods running at any time during the update is atmost 130% of desired pods.",
"description": "The maximum number of pods that can be scheduled above the desired number of pods. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up. Defaults to 25%. Example: when this is set to 30%, the new RC can be scaled up immediately when the rolling update starts, such that the total number of old and new pods do not exceed 130% of desired pods. Once old pods have been killed, new RC can be scaled up further, ensuring that total number of pods running at any time during the update is at most 130% of desired pods.",
"$ref": "#/definitions/io.k8s.apimachinery.pkg.util.intstr.IntOrString"
},
"maxUnavailable": {
@@ -73604,7 +73604,7 @@
"$ref": "#/definitions/io.k8s.api.autoscaling.v2beta1.ResourceMetricSource"
},
"type": {
"description": "type is the type of metric source. It should match one of the fields below.",
"description": "type is the type of metric source. It should be one of \"Object\", \"Pods\" or \"Resource\", each mapping to a matching field in the object.",
"type": "string"
}
}
@@ -73628,7 +73628,7 @@
"$ref": "#/definitions/io.k8s.api.autoscaling.v2beta1.ResourceMetricStatus"
},
"type": {
"description": "type is the type of metric source. It will match one of the fields below.",
"description": "type is the type of metric source. It will be one of \"Object\", \"Pods\" or \"Resource\", each corresponds to a matching field in the object.",
"type": "string"
}
}
@@ -75654,8 +75654,39 @@
}
}
},
"io.k8s.api.core.v1.FlexPersistentVolumeSource": {
"description": "FlexPersistentVolumeSource represents a generic persistent volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],
"properties": {
"driver": {
"description": "Driver is the name of the driver to use for this volume.",
"type": "string"
},
"fsType": {
"description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default filesystem depends on FlexVolume script.",
"type": "string"
},
"options": {
"description": "Optional: Extra command options if any.",
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"readOnly": {
"description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.",
"type": "boolean"
},
"secretRef": {
"description": "Optional: SecretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts.",
"$ref": "#/definitions/io.k8s.api.core.v1.SecretReference"
}
}
},
"io.k8s.api.core.v1.FlexVolumeSource": {
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],
@@ -77006,8 +77037,8 @@
"$ref": "#/definitions/io.k8s.api.core.v1.FCVolumeSource"
},
"flexVolume": {
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"$ref": "#/definitions/io.k8s.api.core.v1.FlexVolumeSource"
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"$ref": "#/definitions/io.k8s.api.core.v1.FlexPersistentVolumeSource"
},
"flocker": {
"description": "Flocker represents a Flocker volume attached to a kubelet's host machine and exposed to the pod for its usage. This depends on the Flocker control service being running",
@@ -78895,7 +78926,7 @@
"$ref": "#/definitions/io.k8s.api.core.v1.FCVolumeSource"
},
"flexVolume": {
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"$ref": "#/definitions/io.k8s.api.core.v1.FlexVolumeSource"
},
"flocker": {

6
api/swagger-spec/apps_v1.json generated
View File

@@ -6799,7 +6799,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -7197,7 +7197,7 @@
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],
@@ -8893,7 +8893,7 @@
},
"maxSurge": {
"type": "string",
"description": "The maximum number of pods that can be scheduled above the desired number of pods. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up. Defaults to 25%. Example: when this is set to 30%, the new RC can be scaled up immediately when the rolling update starts, such that the total number of old and new pods do not exceed 130% of desired pods. Once old pods have been killed, new RC can be scaled up further, ensuring that total number of pods running at any time during the update is atmost 130% of desired pods."
"description": "The maximum number of pods that can be scheduled above the desired number of pods. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up. Defaults to 25%. Example: when this is set to 30%, the new RC can be scaled up immediately when the rolling update starts, such that the total number of old and new pods do not exceed 130% of desired pods. Once old pods have been killed, new RC can be scaled up further, ensuring that total number of pods running at any time during the update is at most 130% of desired pods."
}
}
},

2
api/swagger-spec/apps_v1alpha1.json generated
View File

@@ -1311,7 +1311,7 @@
},
"serviceAccountName": {
"type": "string",
"description": "ServiceAccountName is the name of the ServiceAccount to use to run this pod. More info: http://releases.k8s.io/HEAD/docs/design/service_accounts.md"
"description": "ServiceAccountName is the name of the ServiceAccount to use to run this pod. More info: https://git.k8s.io/community/contributors/design-proposals/auth/service_accounts.md"
},
"serviceAccount": {
"type": "string",

4
api/swagger-spec/apps_v1beta1.json generated
View File

@@ -4433,7 +4433,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -4831,7 +4831,7 @@
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],

4
api/swagger-spec/apps_v1beta2.json generated
View File

@@ -6799,7 +6799,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -7197,7 +7197,7 @@
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],

4
api/swagger-spec/autoscaling_v2beta1.json generated
View File

@@ -1547,7 +1547,7 @@
"properties": {
"type": {
"type": "string",
"description": "type is the type of metric source. It should match one of the fields below."
"description": "type is the type of metric source. It should be one of \"Object\", \"Pods\" or \"Resource\", each mapping to a matching field in the object."
},
"object": {
"$ref": "v2beta1.ObjectMetricSource",
@@ -1680,7 +1680,7 @@
"properties": {
"type": {
"type": "string",
"description": "type is the type of metric source. It will match one of the fields below."
"description": "type is the type of metric source. It will be one of \"Object\", \"Pods\" or \"Resource\", each corresponds to a matching field in the object."
},
"object": {
"$ref": "v2beta1.ObjectMetricStatus",

4
api/swagger-spec/batch_v1.json generated
View File

@@ -1773,7 +1773,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -2171,7 +2171,7 @@
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],

4
api/swagger-spec/batch_v1beta1.json generated
View File

@@ -1828,7 +1828,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -2226,7 +2226,7 @@
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],

4
api/swagger-spec/batch_v2alpha1.json generated
View File

@@ -1828,7 +1828,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -2226,7 +2226,7 @@
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],

4
api/swagger-spec/extensions_v1beta1.json generated
View File

@@ -7441,7 +7441,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -7839,7 +7839,7 @@
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],

4
api/swagger-spec/settings.k8s.io_v1alpha1.json generated
View File

@@ -1615,7 +1615,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -2013,7 +2013,7 @@
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],

63
api/swagger-spec/v1.json generated
View File

@@ -20606,8 +20606,8 @@
"description": "Flocker represents a Flocker volume attached to a kubelet's host machine and exposed to the pod for its usage. This depends on the Flocker control service being running"
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"$ref": "v1.FlexPersistentVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"azureFile": {
"$ref": "v1.AzureFilePersistentVolumeSource",
@@ -21020,9 +21020,9 @@
}
}
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future.",
"v1.FlexPersistentVolumeSource": {
"id": "v1.FlexPersistentVolumeSource",
"description": "FlexPersistentVolumeSource represents a generic persistent volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],
@@ -21036,7 +21036,7 @@
"description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default filesystem depends on FlexVolume script."
},
"secretRef": {
"$ref": "v1.LocalObjectReference",
"$ref": "v1.SecretReference",
"description": "Optional: SecretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts."
},
"readOnly": {
@@ -21049,16 +21049,6 @@
}
}
},
"v1.LocalObjectReference": {
"id": "v1.LocalObjectReference",
"description": "LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.",
"properties": {
"name": {
"type": "string",
"description": "Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names"
}
}
},
"v1.AzureFilePersistentVolumeSource": {
"id": "v1.AzureFilePersistentVolumeSource",
"description": "AzureFile represents an Azure File Service mount on the host and bind mount to the pod.",
@@ -21596,7 +21586,7 @@
},
"flexVolume": {
"$ref": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. This is an alpha feature and may change in future."
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin."
},
"cinder": {
"$ref": "v1.CinderVolumeSource",
@@ -21803,6 +21793,16 @@
}
}
},
"v1.LocalObjectReference": {
"id": "v1.LocalObjectReference",
"description": "LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.",
"properties": {
"name": {
"type": "string",
"description": "Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names"
}
}
},
"v1.PersistentVolumeClaimVolumeSource": {
"id": "v1.PersistentVolumeClaimVolumeSource",
"description": "PersistentVolumeClaimVolumeSource references the user's PVC in the same namespace. This volume finds the bound PV and mounts that volume for the pod. A PersistentVolumeClaimVolumeSource is, essentially, a wrapper around another type of volume that is owned by someone else (the system).",
@@ -21865,6 +21865,35 @@
}
}
},
"v1.FlexVolumeSource": {
"id": "v1.FlexVolumeSource",
"description": "FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.",
"required": [
"driver"
],
"properties": {
"driver": {
"type": "string",
"description": "Driver is the name of the driver to use for this volume."
},
"fsType": {
"type": "string",
"description": "Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. \"ext4\", \"xfs\", \"ntfs\". The default filesystem depends on FlexVolume script."
},
"secretRef": {
"$ref": "v1.LocalObjectReference",
"description": "Optional: SecretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts."
},
"readOnly": {
"type": "boolean",
"description": "Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts."
},
"options": {
"type": "object",
"description": "Optional: Extra command options if any."
}
}
},
"v1.CephFSVolumeSource": {
"id": "v1.CephFSVolumeSource",
"description": "Represents a Ceph Filesystem mount that lasts the lifetime of a pod Cephfs volumes do not support ownership management or SELinux relabeling.",

4
build/BUILD
View File

@@ -38,7 +38,7 @@ DOCKERIZED_BINARIES = {
},
"kube-scheduler": {
"base": "@official_busybox//image",
"target": "//plugin/cmd/kube-scheduler:kube-scheduler",
"target": "//cmd/kube-scheduler:kube-scheduler",
},
"kube-proxy": {
"base": "@debian-iptables-amd64//image",
@@ -127,7 +127,7 @@ release_filegroup(
"//cmd/hyperkube",
"//cmd/kube-apiserver",
"//cmd/kube-controller-manager",
"//plugin/cmd/kube-scheduler",
"//cmd/kube-scheduler",
"//vendor/k8s.io/kube-aggregator",
],
)

2
build/build-image/cross/Makefile
View File

@@ -24,4 +24,4 @@ build:
docker build --pull -t gcr.io/google_containers/$(IMAGE):$(TAG) .
push: build
gcloud docker --server=gcr.io -- push gcr.io/google_containers/$(IMAGE):$(TAG)
gcloud docker -- push gcr.io/google_containers/$(IMAGE):$(TAG)

4
build/common.sh
View File

@@ -451,8 +451,8 @@ function kube::build::build_image() {
cp /etc/localtime "${LOCAL_OUTPUT_BUILD_CONTEXT}/"
cp build/build-image/Dockerfile "${LOCAL_OUTPUT_BUILD_CONTEXT}/Dockerfile"
cp build/build-image/rsyncd.sh "${LOCAL_OUTPUT_BUILD_CONTEXT}/"
cp ${KUBE_ROOT}/build/build-image/Dockerfile "${LOCAL_OUTPUT_BUILD_CONTEXT}/Dockerfile"
cp ${KUBE_ROOT}/build/build-image/rsyncd.sh "${LOCAL_OUTPUT_BUILD_CONTEXT}/"
dd if=/dev/urandom bs=512 count=1 2>/dev/null | LC_ALL=C tr -dc 'A-Za-z0-9' | dd bs=32 count=1 2>/dev/null > "${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password"
chmod go= "${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password"

2
build/debs/BUILD
View File

@@ -41,7 +41,7 @@ deb_data(
name = "kube-scheduler",
data = [
{
"files": ["//plugin/cmd/kube-scheduler"],
"files": ["//cmd/kube-scheduler"],
"mode": "0755",
"dir": "/usr/bin",
},

8
build/lib/release.sh
View File

@@ -28,11 +28,7 @@ readonly RELEASE_STAGE="${LOCAL_OUTPUT_ROOT}/release-stage"
readonly RELEASE_TARS="${LOCAL_OUTPUT_ROOT}/release-tars"
readonly RELEASE_IMAGES="${LOCAL_OUTPUT_ROOT}/release-images"
KUBE_BUILD_HYPERKUBE=${KUBE_BUILD_HYPERKUBE:-n}
if [[ -n "${KUBE_DOCKER_IMAGE_TAG-}" && -n "${KUBE_DOCKER_REGISTRY-}" ]]; then
# retain legacy behavior of automatically building hyperkube during releases
KUBE_BUILD_HYPERKUBE=y
fi
KUBE_BUILD_HYPERKUBE=${KUBE_BUILD_HYPERKUBE:-y}
# Validate a ci version
#
@@ -418,7 +414,6 @@ function kube::release::package_kube_manifests_tarball() {
cp "${salt_dir}/e2e-image-puller/e2e-image-puller.manifest" "${gci_dst_dir}/"
cp "${KUBE_ROOT}/cluster/gce/gci/configure-helper.sh" "${gci_dst_dir}/gci-configure-helper.sh"
cp "${KUBE_ROOT}/cluster/gce/gci/health-monitor.sh" "${gci_dst_dir}/health-monitor.sh"
cp "${KUBE_ROOT}/cluster/gce/container-linux/configure-helper.sh" "${gci_dst_dir}/container-linux-configure-helper.sh"
cp -r "${salt_dir}/kube-admission-controls/limit-range" "${gci_dst_dir}"
local objects
objects=$(cd "${KUBE_ROOT}/cluster/addons" && find . \( -name \*.yaml -or -name \*.yaml.in -or -name \*.json \) | grep -v demo)
@@ -521,7 +516,6 @@ EOF
cp -R "${KUBE_ROOT}/docs" "${release_stage}/"
cp "${KUBE_ROOT}/README.md" "${release_stage}/"
cp "${KUBE_ROOT}/Godeps/LICENSES" "${release_stage}/"
cp "${KUBE_ROOT}/Vagrantfile" "${release_stage}/"
echo "${KUBE_GIT_VERSION}" > "${release_stage}/version"

8
build/pause/CHANGELOG.md Normal file
View File

@@ -0,0 +1,8 @@
# 3.1
* The pause container gains a signal handler to clean up orphaned zombie processes. ([#36853](https://prs.k8s.io/36853), [@verb](https://github.com/verb))
* `pause -v` will return build information for the pause binary. ([#56762](https://prs.k8s.io/56762), [@verb](https://github.com/verb))
# 3.0
* The pause container was rewritten entirely in C. ([#23009](https://prs.k8s.io/23009), [@uluyol](https://github.com/uluyol))

7
build/pause/Makefile
View File

@@ -18,14 +18,15 @@ REGISTRY ?= gcr.io/google_containers
IMAGE = $(REGISTRY)/pause-$(ARCH)
LEGACY_AMD64_IMAGE = $(REGISTRY)/pause
TAG = 3.0
TAG = 3.1
REV = $(shell git describe --contains --always --match='v*')
# Architectures supported: amd64, arm, arm64, ppc64le and s390x
ARCH ?= amd64
ALL_ARCH = amd64 arm arm64 ppc64le s390x
CFLAGS = -Os -Wall -Werror -static
CFLAGS = -Os -Wall -Werror -static -DVERSION=v$(TAG)-$(REV)
KUBE_CROSS_IMAGE ?= gcr.io/google_containers/kube-cross
KUBE_CROSS_VERSION ?= $(shell cat ../build-image/cross/VERSION)
@@ -37,7 +38,7 @@ ifeq ($(ARCH),amd64)
endif
ifeq ($(ARCH),arm)
TRIPLE ?= arm-linux-gnueabi
TRIPLE ?= arm-linux-gnueabihf
endif
ifeq ($(ARCH),arm64)

21
build/pause/pause.c
View File

@@ -17,20 +17,37 @@ limitations under the License.
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#define STRINGIFY(x) #x
#define VERSION_STRING(x) STRINGIFY(x)
#ifndef VERSION
#define VERSION HEAD
#endif
static void sigdown(int signo) {
psignal(signo, "Shutting down, got signal");
exit(0);
}
static void sigreap(int signo) {
while (waitpid(-1, NULL, WNOHANG) > 0);
while (waitpid(-1, NULL, WNOHANG) > 0)
;
}
int main() {
int main(int argc, char **argv) {
int i;
for (i = 1; i < argc; ++i) {
if (!strcasecmp(argv[i], "-v")) {
printf("pause.c %s\n", VERSION_STRING(VERSION));
return 0;
}
}
if (getpid() != 1)
/* Not an error because pause sees use outside of infra containers. */
fprintf(stderr, "Warning: pause should be the first process\n");

1
build/release-tars/BUILD
View File

@@ -193,7 +193,6 @@ pkg_tar(
files = [
"//:Godeps/LICENSES",
"//:README.md",
"//:Vagrantfile",
"//:version",
"//cluster:all-srcs",
"//docs:all-srcs",

17
build/root/Makefile
View File

@@ -389,7 +389,7 @@ define RELEASE_SKIP_TESTS_HELP_INFO
#
# Args:
# KUBE_RELEASE_RUN_TESTS: Whether to run tests. Set to 'y' to run tests anyways.
# KUBE_FASTBUILD: Whether to cross-compile for other architectures. Set to 'true' to do so.
# KUBE_FASTBUILD: Whether to cross-compile for other architectures. Set to 'false' to do so.
#
# Example:
# make release-skip-tests
@@ -466,21 +466,6 @@ $(filter-out %$(EXCLUDE_TARGET),$(notdir $(abspath $(wildcard cmd/*/)))): genera
hack/make-rules/build.sh cmd/$@
endif
define PLUGIN_CMD_HELP_INFO
# Add rules for all directories in plugin/cmd/
#
# Example:
# make kube-scheduler
endef
.PHONY: $(notdir $(abspath $(wildcard plugin/cmd/*/)))
ifeq ($(PRINT_HELP),y)
$(notdir $(abspath $(wildcard plugin/cmd/*/))):
@echo "$$PLUGIN_CMD_HELP_INFO"
else
$(notdir $(abspath $(wildcard plugin/cmd/*/))): generated_files
hack/make-rules/build.sh plugin/cmd/$@
endif
define GENERATED_FILES_HELP_INFO
# Produce auto-generated files needed for the build.
#

43
build/root/WORKSPACE
View File

@@ -1,15 +1,22 @@
http_archive(
name = "io_bazel_rules_go",
sha256 = "441e560e947d8011f064bd7348d86940d6b6131ae7d7c4425a538e8d9f884274",
strip_prefix = "rules_go-c72631a220406c4fae276861ee286aaec82c5af2",
urls = ["https://github.com/bazelbuild/rules_go/archive/c72631a220406c4fae276861ee286aaec82c5af2.tar.gz"],
sha256 = "e8c7f1fda9ee482745a5b35e8314ac3ae744d4ba30f3e6de28148fd166044306",
strip_prefix = "rules_go-737df20c53499fd84b67f04c6ca9ccdee2e77089",
urls = ["https://github.com/bazelbuild/rules_go/archive/737df20c53499fd84b67f04c6ca9ccdee2e77089.tar.gz"],
)
http_archive(
name = "io_kubernetes_build",
sha256 = "89788eb30f10258ae0c6ab8b8625a28cb4c101fba93a8a6725ba227bb778ff27",
strip_prefix = "repo-infra-653485c1a6d554513266d55683da451bd41f7d65",
urls = ["https://github.com/kubernetes/repo-infra/archive/653485c1a6d554513266d55683da451bd41f7d65.tar.gz"],
sha256 = "cf138e48871629345548b4aaf23101314b5621c1bdbe45c4e75edb45b08891f0",
strip_prefix = "repo-infra-1fb0a3ff0cc5308a6d8e2f3f9c57d1f2f940354e",
urls = ["https://github.com/kubernetes/repo-infra/archive/1fb0a3ff0cc5308a6d8e2f3f9c57d1f2f940354e.tar.gz"],
)
http_archive(
name = "bazel_skylib",
sha256 = "bbccf674aa441c266df9894182d80de104cabd19be98be002f6d478aaa31574d",
strip_prefix = "bazel-skylib-2169ae1c374aab4a09aa90e65efe1a3aad4e279b",
urls = ["https://github.com/bazelbuild/bazel-skylib/archive/2169ae1c374aab4a09aa90e65efe1a3aad4e279b.tar.gz"],
)
ETCD_VERSION = "3.1.10"
@@ -39,35 +46,17 @@ http_archive(
urls = ["https://github.com/bazelbuild/rules_docker/archive/8bbe2a8abd382641e65ff7127a3700a8530f02ce.tar.gz"],
)
load("@io_kubernetes_build//defs:bazel_version.bzl", "check_version")
load("@bazel_skylib//:lib.bzl", "versions")
check_version("0.6.0")
versions.check(minimum_bazel_version = "0.8.0")
load("@io_bazel_rules_go//go:def.bzl", "go_rules_dependencies", "go_register_toolchains", "go_download_sdk")
load("@io_bazel_rules_docker//docker:docker.bzl", "docker_repositories", "docker_pull")
go_rules_dependencies()
# The upstream version of rules_go is broken in a number of ways. Until it's fixed, explicitly download and use go1.9.2 ourselves.
go_download_sdk(
name = "go_sdk",
sdks = {
"darwin_amd64": ("go1.9.2.darwin-amd64.tar.gz", "73fd5840d55f5566d8db6c0ffdd187577e8ebe650c783f68bd27cbf95bde6743"),
"linux_386": ("go1.9.2.linux-386.tar.gz", "574b2c4b1a248e58ef7d1f825beda15429610a2316d9cbd3096d8d3fa8c0bc1a"),
"linux_amd64": ("go1.9.2.linux-amd64.tar.gz", "de874549d9a8d8d8062be05808509c09a88a248e77ec14eb77453530829ac02b"),
"linux_armv6l": ("go1.9.2.linux-armv6l.tar.gz", "8a6758c8d390e28ef2bcea511f62dcb43056f38c1addc06a8bc996741987e7bb"),
"windows_386": ("go1.9.2.windows-386.zip", "35d3be5d7b97c6d11ffb76c1b19e20a824e427805ee918e82c08a2e5793eda20"),
"windows_amd64": ("go1.9.2.windows-amd64.zip", "682ec3626a9c45b657c2456e35cadad119057408d37f334c6c24d88389c2164c"),
"freebsd_386": ("go1.9.2.freebsd-386.tar.gz", "809dcb0a8457c8d0abf954f20311a1ee353486d0ae3f921e9478189721d37677"),
"freebsd_amd64": ("go1.9.2.freebsd-amd64.tar.gz", "8be985c3e251c8e007fa6ecd0189bc53e65cc519f4464ddf19fa11f7ed251134"),
"linux_arm64": ("go1.9.2.linux-arm64.tar.gz", "0016ac65ad8340c84f51bc11dbb24ee8265b0a4597dbfdf8d91776fc187456fa"),
"linux_ppc64le": ("go1.9.2.linux-ppc64le.tar.gz", "adb440b2b6ae9e448c253a20836d8e8aa4236f731d87717d9c7b241998dc7f9d"),
"linux_s390x": ("go1.9.2.linux-s390x.tar.gz", "a7137b4fbdec126823a12a4b696eeee2f04ec616e9fb8a54654c51d5884c1345"),
},
)
go_register_toolchains(
go_version = "overridden by go_download_sdk",
go_version = "1.9.2",
)
docker_repositories()

2
build/visible_to/BUILD
View File

@@ -210,7 +210,7 @@ package_group(
"//cmd/clicheck",
"//cmd/hyperkube",
"//cmd/kube-proxy/app",
"//plugin/cmd/kube-scheduler/app",
"//cmd/kube-scheduler/app",
],
)

4
cluster/BUILD
View File

@@ -20,7 +20,6 @@ filegroup(
"//cluster/images/etcd/rollback:all-srcs",
"//cluster/images/hyperkube:all-srcs",
"//cluster/images/kubemark:all-srcs",
"//cluster/lib:all-srcs",
"//cluster/saltbase:all-srcs",
],
tags = ["automanaged"],
@@ -55,7 +54,6 @@ sh_test(
name = "common_test",
srcs = ["common.sh"],
deps = [
"//cluster/lib",
"//hack/lib",
],
)
@@ -64,7 +62,6 @@ sh_test(
name = "clientbin_test",
srcs = ["clientbin.sh"],
deps = [
"//cluster/lib",
"//hack/lib",
],
)
@@ -73,7 +70,6 @@ sh_test(
name = "kube-util_test",
srcs = ["kube-util.sh"],
deps = [
"//cluster/lib",
"//hack/lib",
],
)

3
cluster/addons/addon-manager/CHANGELOG.md
View File

@@ -1,6 +1,9 @@
### Version 8.4 (Thu November 30 2017 zou nengren @zouyee)
- Update kubectl to v1.8.4.
### Version 6.5 (Wed October 15 2017 Daniel Kłobuszewski <danielmk@google.com>)
- Support for HA masters.
### Version 6.4-beta.2 (Mon June 12 2017 Jeff Grafton <jgrafton@google.com>)
- Update kubectl to v1.6.4.
- Refresh base images.

9
cluster/addons/calico-policy-controller/calico-node-daemonset.yaml
View File

@@ -149,5 +149,10 @@ spec:
hostPath:
path: /etc/cni/net.d
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
# Make sure calico/node gets scheduled on all nodes.
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
- key: CriticalAddonsOnly
operator: Exists

4
cluster/addons/cluster-loadbalancing/glbc/default-svc-controller.yaml
View File

@@ -1,4 +1,4 @@
apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: Deployment
metadata:
name: l7-default-backend
@@ -24,7 +24,7 @@ spec:
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: gcr.io/google_containers/defaultbackend:1.3
image: gcr.io/google_containers/defaultbackend:1.4
livenessProbe:
httpGet:
path: /healthz

61
cluster/addons/cluster-monitoring/google/heapster-controller.yaml
View File

@@ -20,32 +20,58 @@ metadata:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
name: heapster-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: v1
kind: ConfigMap
metadata:
name: eventer-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster-v1.5.0-beta.3
name: heapster-v1.5.0
namespace: kube-system
labels:
k8s-app: heapster
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v1.5.0-beta.3
version: v1.5.0
spec:
replicas: 1
selector:
matchLabels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
template:
metadata:
labels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
containers:
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.3
- image: gcr.io/google_containers/heapster-amd64:v1.5.0
name: heapster
livenessProbe:
httpGet:
@@ -58,13 +84,13 @@ spec:
- /heapster
- --source=kubernetes.summary_api:''
- --sink=gcm
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.3
- image: gcr.io/google_containers/heapster-amd64:v1.5.0
name: eventer
command:
- /eventer
- --source=kubernetes:''
- --sink=gcl
- image: gcr.io/google_containers/addon-resizer:1.7
- image: gcr.io/google_containers/addon-resizer:1.8.1
name: heapster-nanny
resources:
limits:
@@ -73,6 +99,9 @@ spec:
requests:
cpu: 50m
memory: {{ nanny_memory }}
volumeMounts:
- name: heapster-config-volume
mountPath: /etc/config
env:
- name: MY_POD_NAME
valueFrom:
@@ -84,16 +113,17 @@ spec:
fieldPath: metadata.namespace
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu={{ base_metrics_cpu }}
- --extra-cpu={{ metrics_cpu_per_node }}m
- --memory={{ base_metrics_memory }}
- --extra-memory={{metrics_memory_per_node}}Mi
- --threshold=5
- --deployment=heapster-v1.5.0-beta.3
- --deployment=heapster-v1.5.0
- --container=heapster
- --poll-period=300000
- --estimator=exponential
- image: gcr.io/google_containers/addon-resizer:1.7
- image: gcr.io/google_containers/addon-resizer:1.8.1
name: eventer-nanny
resources:
limits:
@@ -111,17 +141,28 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: eventer-config-volume
mountPath: /etc/config
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu=100m
- --extra-cpu=0m
- --memory={{base_eventer_memory}}
- --extra-memory={{eventer_memory_per_node}}Ki
- --threshold=5
- --deployment=heapster-v1.5.0-beta.3
- --deployment=heapster-v1.5.0
- --container=eventer
- --poll-period=300000
- --estimator=exponential
volumes:
- name: heapster-config-volume
configMap:
name: heapster-config
- name: eventer-config-volume
configMap:
name: eventer-config
serviceAccountName: heapster
tolerations:
- key: "CriticalAddonsOnly"

61
cluster/addons/cluster-monitoring/googleinfluxdb/heapster-controller-combined.yaml
View File

@@ -20,32 +20,58 @@ metadata:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
name: heapster-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: v1
kind: ConfigMap
metadata:
name: eventer-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster-v1.5.0-beta.3
name: heapster-v1.5.0
namespace: kube-system
labels:
k8s-app: heapster
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v1.5.0-beta.3
version: v1.5.0
spec:
replicas: 1
selector:
matchLabels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
template:
metadata:
labels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
containers:
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.3
- image: gcr.io/google_containers/heapster-amd64:v1.5.0
name: heapster
livenessProbe:
@@ -60,13 +86,13 @@ spec:
- --source=kubernetes.summary_api:''
- --sink=influxdb:http://monitoring-influxdb:8086
- --sink=gcm:?metrics=autoscaling
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.3
- image: gcr.io/google_containers/heapster-amd64:v1.5.0
name: eventer
command:
- /eventer
- --source=kubernetes:''
- --sink=gcl
- image: gcr.io/google_containers/addon-resizer:1.7
- image: gcr.io/google_containers/addon-resizer:1.8.1
name: heapster-nanny
resources:
limits:
@@ -75,6 +101,9 @@ spec:
requests:
cpu: 50m
memory: {{ nanny_memory }}
volumeMounts:
- name: heapster-config-volume
mountPath: /etc/config
env:
- name: MY_POD_NAME
valueFrom:
@@ -86,16 +115,17 @@ spec:
fieldPath: metadata.namespace
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu={{ base_metrics_cpu }}
- --extra-cpu={{ metrics_cpu_per_node }}m
- --memory={{ base_metrics_memory }}
- --extra-memory={{ metrics_memory_per_node }}Mi
- --threshold=5
- --deployment=heapster-v1.5.0-beta.3
- --deployment=heapster-v1.5.0
- --container=heapster
- --poll-period=300000
- --estimator=exponential
- image: gcr.io/google_containers/addon-resizer:1.7
- image: gcr.io/google_containers/addon-resizer:1.8.1
name: eventer-nanny
resources:
limits:
@@ -104,6 +134,9 @@ spec:
requests:
cpu: 50m
memory: {{ nanny_memory }}
volumeMounts:
- name: eventer-config-volume
mountPath: /etc/config
env:
- name: MY_POD_NAME
valueFrom:
@@ -115,15 +148,23 @@ spec:
fieldPath: metadata.namespace
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu=100m
- --extra-cpu=0m
- --memory={{ base_eventer_memory }}
- --extra-memory={{ eventer_memory_per_node }}Ki
- --threshold=5
- --deployment=heapster-v1.5.0-beta.3
- --deployment=heapster-v1.5.0
- --container=eventer
- --poll-period=300000
- --estimator=exponential
volumes:
- name: heapster-config-volume
configMap:
name: heapster-config
- name: eventer-config-volume
configMap:
name: eventer-config
serviceAccountName: heapster
tolerations:
- key: "CriticalAddonsOnly"

61
cluster/addons/cluster-monitoring/influxdb/heapster-controller.yaml
View File

@@ -20,32 +20,58 @@ metadata:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
name: heapster-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: v1
kind: ConfigMap
metadata:
name: eventer-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster-v1.5.0-beta.3
name: heapster-v1.5.0
namespace: kube-system
labels:
k8s-app: heapster
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v1.5.0-beta.3
version: v1.5.0
spec:
replicas: 1
selector:
matchLabels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
template:
metadata:
labels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
containers:
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.3
- image: gcr.io/google_containers/heapster-amd64:v1.5.0
name: heapster
livenessProbe:
httpGet:
@@ -58,13 +84,13 @@ spec:
- /heapster
- --source=kubernetes.summary_api:''
- --sink=influxdb:http://monitoring-influxdb:8086
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.3
- image: gcr.io/google_containers/heapster-amd64:v1.5.0
name: eventer
command:
- /eventer
- --source=kubernetes:''
- --sink=influxdb:http://monitoring-influxdb:8086
- image: gcr.io/google_containers/addon-resizer:1.7
- image: gcr.io/google_containers/addon-resizer:1.8.1
name: heapster-nanny
resources:
limits:
@@ -82,18 +108,22 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: heapster-config-volume
mountPath: /etc/config
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu={{ base_metrics_cpu }}
- --extra-cpu={{ metrics_cpu_per_node }}m
- --memory={{ base_metrics_memory }}
- --extra-memory={{ metrics_memory_per_node }}Mi
- --threshold=5
- --deployment=heapster-v1.5.0-beta.3
- --deployment=heapster-v1.5.0
- --container=heapster
- --poll-period=300000
- --estimator=exponential
- image: gcr.io/google_containers/addon-resizer:1.7
- image: gcr.io/google_containers/addon-resizer:1.8.1
name: eventer-nanny
resources:
limits:
@@ -111,17 +141,28 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: eventer-config-volume
mountPath: /etc/config
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu=100m
- --extra-cpu=0m
- --memory={{ base_eventer_memory }}
- --extra-memory={{ eventer_memory_per_node }}Ki
- --threshold=5
- --deployment=heapster-v1.5.0-beta.3
- --deployment=heapster-v1.5.0
- --container=eventer
- --poll-period=300000
- --estimator=exponential
volumes:
- name: heapster-config-volume
configMap:
name: heapster-config
- name: eventer-config-volume
configMap:
name: eventer-config
serviceAccountName: heapster
tolerations:
- key: "CriticalAddonsOnly"

35
cluster/addons/cluster-monitoring/stackdriver/heapster-controller.yaml
View File

@@ -18,32 +18,45 @@ metadata:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
name: heapster-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster-v1.5.0-beta.3
name: heapster-v1.5.0
namespace: kube-system
labels:
k8s-app: heapster
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v1.5.0-beta.3
version: v1.5.0
spec:
replicas: 1
selector:
matchLabels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
template:
metadata:
labels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
containers:
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.3
- image: gcr.io/google_containers/heapster-amd64:v1.5.0
name: heapster
livenessProbe:
httpGet:
@@ -76,7 +89,7 @@ spec:
fieldRef:
fieldPath: metadata.namespace
# END_PROMETHEUS_TO_SD
- image: gcr.io/google_containers/addon-resizer:1.7
- image: gcr.io/google_containers/addon-resizer:1.8.1
name: heapster-nanny
resources:
limits:
@@ -85,6 +98,9 @@ spec:
requests:
cpu: 50m
memory: {{ nanny_memory }}
volumeMounts:
- name: heapster-config-volume
mountPath: /etc/config
env:
- name: MY_POD_NAME
valueFrom:
@@ -96,15 +112,20 @@ spec:
fieldPath: metadata.namespace
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu={{ base_metrics_cpu }}
- --extra-cpu={{ metrics_cpu_per_node }}m
- --memory={{ base_metrics_memory }}
- --extra-memory={{metrics_memory_per_node}}Mi
- --threshold=5
- --deployment=heapster-v1.5.0-beta.3
- --deployment=heapster-v1.5.0
- --container=heapster
- --poll-period=300000
- --estimator=exponential
volumes:
- name: heapster-config-volume
configMap:
name: heapster-config
serviceAccountName: heapster
tolerations:
- key: "CriticalAddonsOnly"

35
cluster/addons/cluster-monitoring/standalone/heapster-controller.yaml
View File

@@ -18,32 +18,45 @@ metadata:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
name: heapster-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster-v1.5.0-beta.3
name: heapster-v1.5.0
namespace: kube-system
labels:
k8s-app: heapster
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v1.5.0-beta.3
version: v1.5.0
spec:
replicas: 1
selector:
matchLabels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
template:
metadata:
labels:
k8s-app: heapster
version: v1.5.0-beta.3
version: v1.5.0
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
containers:
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.3
- image: gcr.io/google_containers/heapster-amd64:v1.5.0
name: heapster
livenessProbe:
httpGet:
@@ -55,7 +68,7 @@ spec:
command:
- /heapster
- --source=kubernetes.summary_api:''
- image: gcr.io/google_containers/addon-resizer:1.7
- image: gcr.io/google_containers/addon-resizer:1.8.1
name: heapster-nanny
resources:
limits:
@@ -73,17 +86,25 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: heapster-config-volume
mountPath: /etc/config
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu={{ base_metrics_cpu }}
- --extra-cpu={{ metrics_cpu_per_node }}m
- --memory={{ base_metrics_memory }}
- --extra-memory={{ metrics_memory_per_node }}Mi
- --threshold=5
- --deployment=heapster-v1.5.0-beta.3
- --deployment=heapster-v1.5.0
- --container=heapster
- --poll-period=300000
- --estimator=exponential
volumes:
- name: heapster-config-volume
configMap:
name: heapster-config
serviceAccountName: heapster
tolerations:
- key: "CriticalAddonsOnly"

2
cluster/addons/dashboard/dashboard-controller.yaml
View File

@@ -35,7 +35,7 @@ spec:
cpu: 100m
memory: 300Mi
requests:
cpu: 100m
cpu: 50m
memory: 100Mi
ports:
- containerPort: 8443

2
cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml
View File

@@ -36,7 +36,7 @@ spec:
hostPath:
path: /dev
containers:
- image: "gcr.io/google-containers/nvidia-gpu-device-plugin@sha256:5e3837c3ab99e90d4c19053998ad86239591de4264bc177faad75642b64b723d"
- image: "gcr.io/google-containers/nvidia-gpu-device-plugin@sha256:0e79da6998a61257585e0d3fb5848240129f0fa5b4ad972dfed4049448093c33"
command: ["/usr/bin/nvidia-gpu-device-plugin", "-logtostderr"]
name: nvidia-gpu-device-plugin
resources:

8
cluster/addons/dns/coredns.yaml.base
View File

@@ -57,9 +57,11 @@ data:
Corefile: |
.:53 {
errors
log stdout
log
health
kubernetes __PILLAR__DNS__DOMAIN__ __PILLAR__CLUSTER_CIDR__
kubernetes __PILLAR__DNS__DOMAIN__ __PILLAR__CLUSTER_CIDR__ {
pods insecure
}
prometheus
proxy . /etc/resolv.conf
cache 30
@@ -93,7 +95,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: coredns/coredns:0.9.10
image: coredns/coredns:1.0.1
imagePullPolicy: IfNotPresent
resources:
limits:

8
cluster/addons/dns/coredns.yaml.in
View File

@@ -57,9 +57,11 @@ data:
Corefile: |
.:53 {
errors
log stdout
log
health
kubernetes {{ pillar['dns_domain'] }} {{ pillar['service_cluster_ip_range'] }}
kubernetes {{ pillar['dns_domain'] }} {{ pillar['service_cluster_ip_range'] }} {
pods insecure
}
prometheus
proxy . /etc/resolv.conf
cache 30
@@ -93,7 +95,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: coredns/coredns:0.9.10
image: coredns/coredns:1.0.1
imagePullPolicy: IfNotPresent
resources:
limits:

8
cluster/addons/dns/coredns.yaml.sed
View File

@@ -57,9 +57,11 @@ data:
Corefile: |
.:53 {
errors
log stdout
log
health
kubernetes $DNS_DOMAIN $SERVICE_CLUSTER_IP_RANGE
kubernetes $DNS_DOMAIN $SERVICE_CLUSTER_IP_RANGE {
pods insecure
}
prometheus
proxy . /etc/resolv.conf
cache 30
@@ -93,7 +95,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: coredns/coredns:0.9.10
image: coredns/coredns:1.0.1
imagePullPolicy: IfNotPresent
resources:
limits:

6
cluster/addons/dns/kube-dns.yaml.base
View File

@@ -94,7 +94,7 @@ spec:
optional: true
containers:
- name: kubedns
image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.8
resources:
# TODO: Set memory limits when we've profiled the container for large
# clusters, then set request = limit to keep this container in
@@ -145,7 +145,7 @@ spec:
- name: kube-dns-config
mountPath: /kube-dns-config
- name: dnsmasq
image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.8
livenessProbe:
httpGet:
path: /healthcheck/dnsmasq
@@ -184,7 +184,7 @@ spec:
- name: kube-dns-config
mountPath: /etc/k8s/dns/dnsmasq-nanny
- name: sidecar
image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.8
livenessProbe:
httpGet:
path: /metrics

6
cluster/addons/dns/kube-dns.yaml.in
View File

@@ -94,7 +94,7 @@ spec:
optional: true
containers:
- name: kubedns
image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.8
resources:
# TODO: Set memory limits when we've profiled the container for large
# clusters, then set request = limit to keep this container in
@@ -145,7 +145,7 @@ spec:
- name: kube-dns-config
mountPath: /kube-dns-config
- name: dnsmasq
image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.8
livenessProbe:
httpGet:
path: /healthcheck/dnsmasq
@@ -184,7 +184,7 @@ spec:
- name: kube-dns-config
mountPath: /etc/k8s/dns/dnsmasq-nanny
- name: sidecar
image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.8
livenessProbe:
httpGet:
path: /metrics

6
cluster/addons/dns/kube-dns.yaml.sed
View File

@@ -94,7 +94,7 @@ spec:
optional: true
containers:
- name: kubedns
image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.8
resources:
# TODO: Set memory limits when we've profiled the container for large
# clusters, then set request = limit to keep this container in
@@ -145,7 +145,7 @@ spec:
- name: kube-dns-config
mountPath: /kube-dns-config
- name: dnsmasq
image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.8
livenessProbe:
httpGet:
path: /healthcheck/dnsmasq
@@ -184,7 +184,7 @@ spec:
- name: kube-dns-config
mountPath: /etc/k8s/dns/dnsmasq-nanny
- name: sidecar
image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.8
livenessProbe:
httpGet:
path: /metrics

12
cluster/addons/fluentd-elasticsearch/README.md
View File

@@ -8,7 +8,7 @@ is a graphical interface for viewing and querying the logs stored in
Elasticsearch.
**Note:** this addon should **not** be used as-is in production. This is
an example and you should treat is as such. Please see at least the
an example and you should treat it as such. Please see at least the
[Security](#security) and the [Storage](#storage) sections for more
information.
@@ -19,9 +19,9 @@ a Deployment, but allows for maintaining state on storage volumes.
### Security
Elasticsearch has capabilities to enable authorization using
Elasticsearch has capabilities to enable authorization using the
[X-Pack plugin][xPack]. See configuration parameter `xpack.security.enabled`
in Elasticsearch and Kibana configurations. It can also be set via
in Elasticsearch and Kibana configurations. It can also be set via the
`XPACK_SECURITY_ENABLED` env variable. After enabling the feature,
follow [official documentation][setupCreds] to set up credentials in
Elasticsearch and Kibana. Don't forget to propagate those credentials also to
@@ -31,7 +31,7 @@ and [Secrets][secret] to store credentials in the Kubernetes apiserver.
### Initialization
The Elasticsearch Statefulset manifest specifies that there shall be an
The Elasticsearch StatefulSet manifest specifies that there shall be an
[init container][initContainer] executing before Elasticsearch containers
themselves, in order to ensure that the kernel state variable
`vm.max_map_count` is at least 262144, since this is a requirement of
@@ -61,7 +61,7 @@ Learn more in the [official Kubernetes documentation][k8sElasticsearchDocs].
Since Fluentd talks to the Elasticsearch service inside the cluster, instances
on masters won't work, because masters have no kube-proxy. Don't mark masters
with a label mentioned in the previous paragraph or add a taint on them to
with the label mentioned in the previous paragraph or add a taint on them to
avoid Fluentd pods scheduling there.
[fluentd]: http://www.fluentd.org/
@@ -71,7 +71,7 @@ avoid Fluentd pods scheduling there.
[setupCreds]: https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html#reset-built-in-user-passwords
[fluentdCreds]: https://github.com/uken/fluent-plugin-elasticsearch#user-password-path-scheme-ssl_verify
[fluentdEnvVar]: https://docs.fluentd.org/v0.12/articles/faq#how-can-i-use-environment-variables-to-configure-parameters-dynamically
[configMap]: https://kubernetes.io/docs/tasks/configure-pod-container/configmap/
[configMap]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/
[secret]: https://kubernetes.io/docs/concepts/configuration/secret/
[statefulSet]: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset
[initContainer]: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

2
cluster/addons/fluentd-elasticsearch/es-image/BUILD
View File

@@ -8,8 +8,8 @@ load(
go_binary(
name = "es-image",
embed = [":go_default_library"],
importpath = "k8s.io/kubernetes/cluster/addons/fluentd-elasticsearch/es-image",
library = ":go_default_library",
)
go_library(

14
cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml
View File

@@ -105,7 +105,7 @@ data:
path /var/log/containers/*.log
pos_file /var/log/es-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag kubernetes.*
tag raw.kubernetes.*
read_from_head true
format multi_format
<pattern>
@@ -118,6 +118,16 @@ data:
time_format %Y-%m-%dT%H:%M:%S.%N%:z
</pattern>
</source>
# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
@type detect_exceptions
remove_tag_prefix raw
message log
stream stream
multiline_flush_interval 5
max_bytes 500000
max_lines 1000
</match>
system.input.conf: |-
# Example:
# 2015-12-21 23:17:22,066 [salt.state ][INFO ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081
@@ -367,7 +377,7 @@ data:
num_threads 2
</match>
metadata:
name: fluentd-es-config-v0.1.1
name: fluentd-es-config-v0.1.2
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile

12
cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml
View File

@@ -48,24 +48,24 @@ roleRef:
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
name: fluentd-es-v2.0.2
name: fluentd-es-v2.0.3
namespace: kube-system
labels:
k8s-app: fluentd-es
version: v2.0.2
version: v2.0.3
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
k8s-app: fluentd-es
version: v2.0.2
version: v2.0.3
template:
metadata:
labels:
k8s-app: fluentd-es
kubernetes.io/cluster-service: "true"
version: v2.0.2
version: v2.0.3
# This annotation ensures that fluentd does not get evicted if the node
# supports critical pod annotation based priority scheme.
# Note that this does not guarantee admission on the nodes (#40573).
@@ -75,7 +75,7 @@ spec:
serviceAccountName: fluentd-es
containers:
- name: fluentd-es
image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.2
image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.3
env:
- name: FLUENTD_ARGS
value: --no-supervisor -q
@@ -112,4 +112,4 @@ spec:
path: /usr/lib64
- name: config-volume
configMap:
name: fluentd-es-config-v0.1.1
name: fluentd-es-config-v0.1.2

1
cluster/addons/fluentd-elasticsearch/fluentd-es-image/Gemfile
View File

@@ -5,6 +5,7 @@ gem 'activesupport', '~>4.2.6'
gem 'fluent-plugin-kubernetes_metadata_filter', '~>0.27.0'
gem 'fluent-plugin-elasticsearch', '~>1.9.5'
gem 'fluent-plugin-systemd', '~>0.0.8'
gem 'fluent-plugin-detect-exceptions', '~>0.0.8'
gem 'fluent-plugin-prometheus', '~>0.3.0'
gem 'fluent-plugin-multi-format-parser', '~>0.1.1'
gem 'oj', '~>2.18.1'

2
cluster/addons/fluentd-elasticsearch/fluentd-es-image/Makefile
View File

@@ -16,7 +16,7 @@
PREFIX = gcr.io/google-containers
IMAGE = fluentd-elasticsearch
TAG = v2.0.2
TAG = v2.0.3
build:
docker build --pull -t $(PREFIX)/$(IMAGE):$(TAG) .

8
cluster/addons/fluentd-gcp/fluentd-gcp-ds.yaml
View File

@@ -1,13 +1,13 @@
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd-gcp-v2.0.10
name: fluentd-gcp-v2.0.14
namespace: kube-system
labels:
k8s-app: fluentd-gcp
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v2.0.10
version: v2.0.14
spec:
updateStrategy:
type: RollingUpdate
@@ -16,7 +16,7 @@ spec:
labels:
k8s-app: fluentd-gcp
kubernetes.io/cluster-service: "true"
version: v2.0.10
version: v2.0.14
# This annotation ensures that fluentd does not get evicted if the node
# supports critical pod annotation based priority scheme.
# Note that this does not guarantee admission on the nodes (#40573).
@@ -27,7 +27,7 @@ spec:
dnsPolicy: Default
containers:
- name: fluentd-gcp
image: gcr.io/google-containers/fluentd-gcp:2.0.10
image: gcr.io/google-containers/fluentd-gcp:2.0.14
env:
- name: FLUENTD_ARGS
value: --no-supervisor -q

15
cluster/addons/metadata-proxy/gce/metadata-proxy.yaml
View File

@@ -38,19 +38,28 @@ spec:
dnsPolicy: Default
containers:
- name: metadata-proxy
image: gcr.io/google_containers/metadata-proxy:v0.1.5
image: gcr.io/google_containers/metadata-proxy:v0.1.7
securityContext:
privileged: true
# Request and limit resources to get guaranteed QoS.
resources:
requests:
memory: "32Mi"
memory: "25Mi"
cpu: "30m"
limits:
memory: "32Mi"
memory: "25Mi"
cpu: "30m"
# BEGIN_PROMETHEUS_TO_SD
- name: prometheus-to-sd-exporter
image: gcr.io/google_containers/prometheus-to-sd:v0.2.2
# Request and limit resources to get guaranteed QoS.
resources:
requests:
memory: "20Mi"
cpu: "2m"
limits:
memory: "20Mi"
cpu: "2m"
command:
- /monitor
- --stackdriver-prefix={{ prometheus_to_sd_prefix }}/addons

39
cluster/addons/metrics-server/metrics-server-deployment.yaml
View File

@@ -7,34 +7,47 @@ metadata:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metrics-server-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: metrics-server-v0.2.0
name: metrics-server-v0.2.1
namespace: kube-system
labels:
k8s-app: metrics-server
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v0.2.0
version: v0.2.1
spec:
selector:
matchLabels:
k8s-app: metrics-server
version: v0.2.0
version: v0.2.1
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
version: v0.2.0
version: v0.2.1
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: metrics-server
containers:
- name: metrics-server
image: gcr.io/google_containers/metrics-server-amd64:v0.2.0
image: gcr.io/google_containers/metrics-server-amd64:v0.2.1
command:
- /metrics-server
- --source=kubernetes.summary_api:''
@@ -43,14 +56,14 @@ spec:
name: https
protocol: TCP
- name: metrics-server-nanny
image: gcr.io/google_containers/addon-resizer:1.7
image: gcr.io/google_containers/addon-resizer:1.8.1
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 50m
memory: 100Mi
cpu: 5m
memory: 50Mi
env:
- name: MY_POD_NAME
valueFrom:
@@ -60,17 +73,25 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: metrics-server-config-volume
mountPath: /etc/config
command:
- /pod_nanny
- --config-dir=/etc/config
- --cpu=40m
- --extra-cpu=0.5m
- --memory=140Mi
- --extra-memory=4Mi
- --threshold=5
- --deployment=metrics-server-v0.2.0
- --deployment=metrics-server-v0.2.1
- --container=metrics-server
- --poll-period=300000
- --estimator=exponential
volumes:
- name: metrics-server-config-volume
configMap:
name: metrics-server-config
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"

14
cluster/addons/rbac/legacy-kubelet-user-disable/kubelet-binding.yaml
View File

@@ -7,6 +7,20 @@ metadata:
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects: []
---
# This is required so that new clusters still have bootstrap permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubelet-bootstrap
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole

26
cluster/aws/util.sh
View File

@@ -1,26 +0,0 @@
#!/bin/bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
source "${KUBE_ROOT}/hack/lib/util.sh"
echo -e "${color_red}WARNING${color_norm}: The bash deployment for AWS is obsolete. The" >&2
echo -e "v1.5.x releases are the last to support cluster/kube-up.sh with AWS." >&2
echo "For a list of viable alternatives, see:" >&2
echo >&2
echo " http://kubernetes.io/docs/getting-started-guides/aws/" >&2
echo >&2
exit 1

1
pkg/auth/handlers/OWNERS → cluster/centos/OWNERS Executable file → Normal file
View File

@@ -1 +1,2 @@
reviewers:
- zouyee

7
cluster/clientbin.sh
View File

@@ -84,9 +84,14 @@ function get_bin() {
"${KUBE_ROOT}/_output/bin/${bin}"
"${KUBE_ROOT}/_output/dockerized/bin/${host_os}/${host_arch}/${bin}"
"${KUBE_ROOT}/_output/local/bin/${host_os}/${host_arch}/${bin}"
"${KUBE_ROOT}/bazel-bin/${srcdir}/${bin}"
"${KUBE_ROOT}/platforms/${host_os}/${host_arch}/${bin}"
)
# Also search for binary in bazel build tree.
# The bazel go rules place binaries in subtrees like
# "bazel-bin/source/path/linux_amd64_pure_stripped/binaryname", so make sure
# the platform name is matched in the path.
locations+=($(find "${KUBE_ROOT}/bazel-bin/${srcdir}" -type f -executable \
-path "*/${host_os}_${host_arch}*/${bin}" 2>/dev/null || true) )
echo $( (ls -t "${locations[@]}" 2>/dev/null || true) | head -1 )
}

36
cluster/common.sh
View File

@@ -25,7 +25,6 @@ KUBE_ROOT=$(cd $(dirname "${BASH_SOURCE}")/.. && pwd)
DEFAULT_KUBECONFIG="${HOME:-.}/.kube/config"
source "${KUBE_ROOT}/hack/lib/util.sh"
source "${KUBE_ROOT}/cluster/lib/logging.sh"
# KUBE_RELEASE_VERSION_REGEX matches things like "v1.2.3" or "v1.2.3-alpha.4"
#
# NOTE This must match the version_regex in build/common.sh
@@ -436,8 +435,8 @@ function find-release-tars() {
# This tarball is used by GCI, Ubuntu Trusty, and Container Linux.
KUBE_MANIFESTS_TAR=
if [[ "${MASTER_OS_DISTRIBUTION:-}" == "trusty" || "${MASTER_OS_DISTRIBUTION:-}" == "gci" || "${MASTER_OS_DISTRIBUTION:-}" == "container-linux" || "${MASTER_OS_DISTRIBUTION:-}" == "ubuntu" ]] || \
[[ "${NODE_OS_DISTRIBUTION:-}" == "trusty" || "${NODE_OS_DISTRIBUTION:-}" == "gci" || "${NODE_OS_DISTRIBUTION:-}" == "container-linux" || "${NODE_OS_DISTRIBUTION:-}" == "ubuntu" ]] ; then
if [[ "${MASTER_OS_DISTRIBUTION:-}" == "trusty" || "${MASTER_OS_DISTRIBUTION:-}" == "gci" || "${MASTER_OS_DISTRIBUTION:-}" == "ubuntu" ]] || \
[[ "${NODE_OS_DISTRIBUTION:-}" == "trusty" || "${NODE_OS_DISTRIBUTION:-}" == "gci" || "${NODE_OS_DISTRIBUTION:-}" == "ubuntu" ]] ; then
KUBE_MANIFESTS_TAR=$(find-tar kubernetes-manifests.tar.gz)
fi
}
@@ -499,7 +498,7 @@ function stage-images() {
done
kube::util::wait-for-jobs || {
kube::log::error "unable to push images. See ${temp_dir}/*.log for more info."
echo "!!! unable to push images. See ${temp_dir}/*.log for more info." 1>&2
return 1
}
@@ -576,9 +575,7 @@ function build-kube-env {
local server_binary_tar_url=$SERVER_BINARY_TAR_URL
local salt_tar_url=$SALT_TAR_URL
local kube_manifests_tar_url="${KUBE_MANIFESTS_TAR_URL:-}"
if [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "container-linux" ]] || \
[[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "container-linux" ]] || \
[[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]] || \
if [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]] || \
[[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "ubuntu" ]] ; then
# TODO: Support fallback .tar.gz settings on Container Linux
server_binary_tar_url=$(split_csv "${SERVER_BINARY_TAR_URL}")
@@ -671,6 +668,7 @@ ADVANCED_AUDIT_WEBHOOK_THROTTLE_QPS: $(yaml-quote ${ADVANCED_AUDIT_WEBHOOK_THROT
ADVANCED_AUDIT_WEBHOOK_THROTTLE_BURST: $(yaml-quote ${ADVANCED_AUDIT_WEBHOOK_THROTTLE_BURST:-})
ADVANCED_AUDIT_WEBHOOK_INITIAL_BACKOFF: $(yaml-quote ${ADVANCED_AUDIT_WEBHOOK_INITIAL_BACKOFF:-})
GCE_API_ENDPOINT: $(yaml-quote ${GCE_API_ENDPOINT:-})
GCE_GLBC_IMAGE: $(yaml-quote ${GCE_GLBC_IMAGE:-})
PROMETHEUS_TO_SD_ENDPOINT: $(yaml-quote ${PROMETHEUS_TO_SD_ENDPOINT:-})
PROMETHEUS_TO_SD_PREFIX: $(yaml-quote ${PROMETHEUS_TO_SD_PREFIX:-})
ENABLE_PROMETHEUS_TO_SD: $(yaml-quote ${ENABLE_PROMETHEUS_TO_SD:-false})
@@ -695,8 +693,8 @@ EOF
TERMINATED_POD_GC_THRESHOLD: $(yaml-quote ${TERMINATED_POD_GC_THRESHOLD})
EOF
fi
if [[ "${master}" == "true" && ("${MASTER_OS_DISTRIBUTION}" == "trusty" || "${MASTER_OS_DISTRIBUTION}" == "gci" || "${MASTER_OS_DISTRIBUTION}" == "container-linux") || "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]] || \
[[ "${master}" == "false" && ("${NODE_OS_DISTRIBUTION}" == "trusty" || "${NODE_OS_DISTRIBUTION}" == "gci" || "${NODE_OS_DISTRIBUTION}" == "container-linux") || "${NODE_OS_DISTRIBUTION}" = "ubuntu" ]] ; then
if [[ "${master}" == "true" && ("${MASTER_OS_DISTRIBUTION}" == "trusty" || "${MASTER_OS_DISTRIBUTION}" == "gci") || "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]] || \
[[ "${master}" == "false" && ("${NODE_OS_DISTRIBUTION}" == "trusty" || "${NODE_OS_DISTRIBUTION}" == "gci") || "${NODE_OS_DISTRIBUTION}" = "ubuntu" ]] ; then
cat >>$file <<EOF
KUBE_MANIFESTS_TAR_URL: $(yaml-quote ${kube_manifests_tar_url})
KUBE_MANIFESTS_TAR_HASH: $(yaml-quote ${KUBE_MANIFESTS_TAR_HASH})
@@ -837,6 +835,16 @@ EOF
if [ -n "${ETCD_HOSTNAME:-}" ]; then
cat >>$file <<EOF
ETCD_HOSTNAME: $(yaml-quote ${ETCD_HOSTNAME})
EOF
fi
if [ -n "${ETCD_LIVENESS_PROBE_INITIAL_DELAY_SEC:-}" ]; then
cat >>$file <<EOF
ETCD_LIVENESS_PROBE_INITIAL_DELAY_SEC: $(yaml-quote ${ETCD_LIVENESS_PROBE_INITIAL_DELAY_SEC})
EOF
fi
if [ -n "${KUBE_APISERVER_LIVENESS_PROBE_INITIAL_DELAY_SEC:-}" ]; then
cat >>$file <<EOF
KUBE_APISERVER_LIVENESS_PROBE_INITIAL_DELAY_SEC: $(yaml-quote ${KUBE_APISERVER_LIVENESS_PROBE_INITIAL_DELAY_SEC})
EOF
fi
if [ -n "${APISERVER_TEST_ARGS:-}" ]; then
@@ -931,16 +939,6 @@ EOF
if [ -n "${EVICTION_HARD:-}" ]; then
cat >>$file <<EOF
EVICTION_HARD: $(yaml-quote ${EVICTION_HARD})
EOF
fi
if [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "container-linux" ]] || \
[[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "container-linux" ]]; then
# Container-Linux-only env vars. TODO(yifan): Make them available on other distros.
cat >>$file <<EOF
KUBERNETES_CONTAINER_RUNTIME: $(yaml-quote ${CONTAINER_RUNTIME:-rkt})
RKT_VERSION: $(yaml-quote ${RKT_VERSION:-})
RKT_PATH: $(yaml-quote ${RKT_PATH:-})
RKT_STAGE1_IMAGE: $(yaml-quote ${RKT_STAGE1_IMAGE:-})
EOF
fi
if [[ "${ENABLE_CLUSTER_AUTOSCALER}" == "true" ]]; then

9
cluster/gce/BUILD
View File

@@ -6,7 +6,6 @@ load("@io_kubernetes_build//defs:build.bzl", "release_filegroup")
pkg_tar(
name = "gci-trusty-manifests",
files = [
"container-linux/configure-helper.sh",
"gci/configure-helper.sh",
"gci/health-monitor.sh",
"//cluster/gce/gci/mounter",
@@ -15,7 +14,6 @@ pkg_tar(
strip_prefix = ".",
# pkg_tar doesn't support renaming the files we add, so instead create symlinks.
symlinks = {
"container-linux-configure-helper.sh": "container-linux/configure-helper.sh",
"gci-configure-helper.sh": "gci/configure-helper.sh",
"health-monitor.sh": "gci/health-monitor.sh",
"gci-mounter": "gci/mounter/mounter",
@@ -40,13 +38,12 @@ filegroup(
tags = ["automanaged"],
)
# Having the configure-vm.sh script and and trusty code from the GCE cluster
# deploy hosted with the release is useful for GKE.
# This list should match the list in kubernetes/release/lib/releaselib.sh.
# Having the COS code from the GCE cluster deploy hosted with the release is
# useful for GKE. This list should match the list in
# kubernetes/release/lib/releaselib.sh.
release_filegroup(
name = "gcs-release-artifacts",
srcs = [
"configure-vm.sh",
"gci/configure.sh",
"gci/master.yaml",
"gci/node.yaml",

2
cluster/gce/config-common.sh
View File

@@ -98,4 +98,6 @@ function get-cluster-ip-range {
echo "${suggested_range}"
}
# NOTE: Avoid giving nodes empty scopes, because kubelet needs a service account
# in order to initialize properly.
NODE_SCOPES="${NODE_SCOPES:-monitoring,logging-write,storage-ro}"

10
cluster/gce/config-default.sh
View File

@@ -54,12 +54,6 @@ CREATE_CUSTOM_NETWORK=${CREATE_CUSTOM_NETWORK:-false}
MASTER_OS_DISTRIBUTION=${KUBE_MASTER_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
if [[ "${MASTER_OS_DISTRIBUTION}" == "coreos" ]]; then
MASTER_OS_DISTRIBUTION="container-linux"
fi
if [[ "${NODE_OS_DISTRIBUTION}" == "coreos" ]]; then
NODE_OS_DISTRIBUTION="container-linux"
fi
if [[ "${MASTER_OS_DISTRIBUTION}" == "cos" ]]; then
MASTER_OS_DISTRIBUTION="gci"
@@ -80,7 +74,7 @@ fi
# Also please update corresponding image for node e2e at:
# https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/jenkins/image-config.yaml
CVM_VERSION=${CVM_VERSION:-container-vm-v20170627}
GCI_VERSION=${KUBE_GCI_VERSION:-cos-stable-60-9592-90-0}
GCI_VERSION=${KUBE_GCI_VERSION:-cos-stable-63-10032-71-0}
MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-}
MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud}
NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}
@@ -121,6 +115,8 @@ MASTER_IP_RANGE="${MASTER_IP_RANGE:-10.246.0.0/24}"
# It is the primary range in the subnet and is the range used for node instance IPs.
NODE_IP_RANGE="$(get-node-ip-range)"
# NOTE: Avoid giving nodes empty scopes, because kubelet needs a service account
# in order to initialize properly.
NODE_SCOPES="${NODE_SCOPES:-monitoring,logging-write,storage-ro}"
# Extra docker options for nodes.

16
cluster/gce/config-test.sh
View File

@@ -37,6 +37,11 @@ MASTER_ROOT_DISK_SIZE=${MASTER_ROOT_DISK_SIZE:-$(get-master-root-disk-size)}
NODE_DISK_TYPE=${NODE_DISK_TYPE:-pd-standard}
NODE_DISK_SIZE=${NODE_DISK_SIZE:-100GB}
NODE_LOCAL_SSDS=${NODE_LOCAL_SSDS:-0}
# An extension to local SSDs allowing users to specify block/fs and SCSI/NVMe devices
# Format of this variable will be "#,scsi/nvme,block/fs" you can specify multiple
# configurations by seperating them by a semi-colon ex. "2,scsi,fs;1,nvme,block"
# is a request for 2 SCSI formatted and mounted SSDs and 1 NVMe block device SSD.
NODE_LOCAL_SSDS_EXT=${NODE_LOCAL_SSDS_EXT:-}
NODE_ACCELERATORS=${NODE_ACCELERATORS:-""}
REGISTER_MASTER_KUBELET=${REGISTER_MASTER:-true}
KUBE_APISERVER_REQUEST_TIMEOUT=300
@@ -48,13 +53,6 @@ CREATE_CUSTOM_NETWORK=${CREATE_CUSTOM_NETWORK:-false}
MASTER_OS_DISTRIBUTION=${KUBE_MASTER_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
if [[ "${MASTER_OS_DISTRIBUTION}" == "coreos" ]]; then
MASTER_OS_DISTRIBUTION="container-linux"
fi
if [[ "${NODE_OS_DISTRIBUTION}" == "coreos" ]]; then
NODE_OS_DISTRIBUTION="container-linux"
fi
if [[ "${MASTER_OS_DISTRIBUTION}" == "cos" ]]; then
MASTER_OS_DISTRIBUTION="gci"
fi
@@ -74,7 +72,7 @@ fi
# Also please update corresponding image for node e2e at:
# https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/jenkins/image-config.yaml
CVM_VERSION=${CVM_VERSION:-container-vm-v20170627}
GCI_VERSION=${KUBE_GCI_VERSION:-cos-stable-60-9592-90-0}
GCI_VERSION=${KUBE_GCI_VERSION:-cos-stable-63-10032-71-0}
MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-}
MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud}
NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}
@@ -352,6 +350,8 @@ OPENCONTRAIL_PUBLIC_SUBNET="${OPENCONTRAIL_PUBLIC_SUBNET:-10.1.0.0/16}"
# Network Policy plugin specific settings.
NETWORK_POLICY_PROVIDER="${NETWORK_POLICY_PROVIDER:-none}" # calico
NON_MASQUERADE_CIDR="0.0.0.0/0"
# How should the kubelet configure hairpin mode?
HAIRPIN_MODE="${HAIRPIN_MODE:-promiscuous-bridge}" # promiscuous-bridge, hairpin-veth, none

889
cluster/gce/configure-vm.sh
View File

@@ -1,889 +0,0 @@
#!/bin/bash
# Copyright 2015 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
# If we have any arguments at all, this is a push and not just setup.
is_push=$@
function ensure-basic-networking() {
# Deal with GCE networking bring-up race. (We rely on DNS for a lot,
# and it's just not worth doing a whole lot of startup work if this
# isn't ready yet.)
until getent hosts metadata.google.internal &>/dev/null; do
echo 'Waiting for functional DNS (trying to resolve metadata.google.internal)...'
sleep 3
done
until getent hosts $(hostname -f || echo _error_) &>/dev/null; do
echo 'Waiting for functional DNS (trying to resolve my own FQDN)...'
sleep 3
done
until getent hosts $(hostname -i || echo _error_) &>/dev/null; do
echo 'Waiting for functional DNS (trying to resolve my own IP)...'
sleep 3
done
echo "Networking functional on $(hostname) ($(hostname -i))"
}
# A hookpoint for installing any needed packages
ensure-packages() {
:
}
function create-node-pki {
echo "Creating node pki files"
local -r pki_dir="/etc/kubernetes/pki"
mkdir -p "${pki_dir}"
if [[ -z "${CA_CERT_BUNDLE:-}" ]]; then
CA_CERT_BUNDLE="${CA_CERT}"
fi
CA_CERT_BUNDLE_PATH="${pki_dir}/ca-certificates.crt"
echo "${CA_CERT_BUNDLE}" | base64 --decode > "${CA_CERT_BUNDLE_PATH}"
if [[ ! -z "${KUBELET_CERT:-}" && ! -z "${KUBELET_KEY:-}" ]]; then
KUBELET_CERT_PATH="${pki_dir}/kubelet.crt"
echo "${KUBELET_CERT}" | base64 --decode > "${KUBELET_CERT_PATH}"
KUBELET_KEY_PATH="${pki_dir}/kubelet.key"
echo "${KUBELET_KEY}" | base64 --decode > "${KUBELET_KEY_PATH}"
fi
}
# A hookpoint for setting up local devices
ensure-local-disks() {
for ssd in /dev/disk/by-id/google-local-ssd-*; do
if [ -e "$ssd" ]; then
ssdnum=`echo $ssd | sed -e 's/\/dev\/disk\/by-id\/google-local-ssd-\([0-9]*\)/\1/'`
echo "Formatting and mounting local SSD $ssd to /mnt/disks/ssd$ssdnum"
mkdir -p /mnt/disks/ssd$ssdnum
/usr/share/google/safe_format_and_mount -m "mkfs.ext4 -F" "${ssd}" /mnt/disks/ssd$ssdnum &>/var/log/local-ssd-$ssdnum-mount.log || \
{ echo "Local SSD $ssdnum mount failed, review /var/log/local-ssd-$ssdnum-mount.log"; return 1; }
else
echo "No local SSD disks found."
fi
done
}
function config-ip-firewall {
echo "Configuring IP firewall rules"
if [[ "${ENABLE_METADATA_CONCEALMENT:-}" == "true" ]]; then
echo "Add rule for metadata concealment"
iptables -w -t nat -I PREROUTING -p tcp -d 169.254.169.254 --dport 80 -m comment --comment "metadata-concealment: bridge traffic to metadata server goes to metadata proxy" -j DNAT --to-destination 127.0.0.1:988
fi
}
function ensure-install-dir() {
INSTALL_DIR="/var/cache/kubernetes-install"
mkdir -p ${INSTALL_DIR}
cd ${INSTALL_DIR}
}
function salt-apiserver-timeout-grain() {
cat <<EOF >>/etc/salt/minion.d/grains.conf
minRequestTimeout: '$1'
EOF
}
function set-broken-motd() {
echo -e '\nBroken (or in progress) Kubernetes node setup! Suggested first step:\n tail /var/log/startupscript.log\n' > /etc/motd
}
function reset-motd() {
# kubelet is installed both on the master and nodes, and the version is easy to parse (unlike kubectl)
local -r version="$(/usr/local/bin/kubelet --version=true | cut -f2 -d " ")"
# This logic grabs either a release tag (v1.2.1 or v1.2.1-alpha.1),
# or the git hash that's in the build info.
local gitref="$(echo "${version}" | sed -r "s/(v[0-9]+\.[0-9]+\.[0-9]+)(-[a-z]+\.[0-9]+)?.*/\1\2/g")"
local devel=""
if [[ "${gitref}" != "${version}" ]]; then
devel="
Note: This looks like a development version, which might not be present on GitHub.
If it isn't, the closest tag is at:
https://github.com/kubernetes/kubernetes/tree/${gitref}
"
gitref="${version//*+/}"
fi
cat > /etc/motd <<EOF
Welcome to Kubernetes ${version}!
You can find documentation for Kubernetes at:
http://docs.kubernetes.io/
The source for this release can be found at:
/usr/local/share/doc/kubernetes/kubernetes-src.tar.gz
Or you can download it at:
https://storage.googleapis.com/kubernetes-release/release/${version}/kubernetes-src.tar.gz
It is based on the Kubernetes source at:
https://github.com/kubernetes/kubernetes/tree/${gitref}
${devel}
For Kubernetes copyright and licensing information, see:
/usr/local/share/doc/kubernetes/LICENSES
EOF
}
function curl-metadata() {
curl --fail --retry 5 --silent -H 'Metadata-Flavor: Google' "http://metadata/computeMetadata/v1/instance/attributes/${1}"
}
function set-kube-env() {
local kube_env_yaml="${INSTALL_DIR}/kube_env.yaml"
until curl-metadata kube-env > "${kube_env_yaml}"; do
echo 'Waiting for kube-env...'
sleep 3
done
# kube-env has all the environment variables we care about, in a flat yaml format
eval "$(python -c '
import pipes,sys,yaml
for k,v in yaml.load(sys.stdin).iteritems():
print("""readonly {var}={value}""".format(var = k, value = pipes.quote(str(v))))
print("""export {var}""".format(var = k))
' < """${kube_env_yaml}""")"
}
function remove-docker-artifacts() {
echo "== Deleting docker0 =="
apt-get-install bridge-utils
# Remove docker artifacts on minion nodes, if present
iptables -t nat -F || true
ifconfig docker0 down || true
brctl delbr docker0 || true
echo "== Finished deleting docker0 =="
}
# Retry a download until we get it. Takes a hash and a set of URLs.
#
# $1 is the sha1 of the URL. Can be "" if the sha1 is unknown.
# $2+ are the URLs to download.
download-or-bust() {
local -r hash="$1"
shift 1
urls=( $* )
while true; do
for url in "${urls[@]}"; do
local file="${url##*/}"
rm -f "${file}"
if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --max-time 300 --retry 6 --retry-delay 10 "${url}"; then
echo "== Failed to download ${url}. Retrying. =="
elif [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
echo "== Hash validation of ${url} failed. Retrying. =="
else
if [[ -n "${hash}" ]]; then
echo "== Downloaded ${url} (SHA1 = ${hash}) =="
else
echo "== Downloaded ${url} =="
fi
return
fi
done
done
}
validate-hash() {
local -r file="$1"
local -r expected="$2"
local actual
actual=$(sha1sum ${file} | awk '{ print $1 }') || true
if [[ "${actual}" != "${expected}" ]]; then
echo "== ${file} corrupted, sha1 ${actual} doesn't match expected ${expected} =="
return 1
fi
}
apt-get-install() {
local -r packages=( $@ )
installed=true
for package in "${packages[@]}"; do
if ! dpkg -s "${package}" &>/dev/null; then
installed=false
break
fi
done
if [[ "${installed}" == "true" ]]; then
echo "== ${packages[@]} already installed, skipped apt-get install ${packages[@]} =="
return
fi
apt-get-update
# Forcibly install packages (options borrowed from Salt logs).
until apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install $@; do
echo "== install of packages $@ failed, retrying =="
sleep 5
done
}
apt-get-update() {
echo "== Refreshing package database =="
until apt-get update; do
echo "== apt-get update failed, retrying =="
sleep 5
done
}
# Restart any services that need restarting due to a library upgrade
# Uses needrestart
restart-updated-services() {
# We default to restarting services, because this is only done as part of an update
if [[ "${AUTO_RESTART_SERVICES:-true}" != "true" ]]; then
echo "Auto restart of services prevented by AUTO_RESTART_SERVICES=${AUTO_RESTART_SERVICES}"
return
fi
echo "Restarting services with updated libraries (needrestart -r a)"
# The pipes make sure that needrestart doesn't think it is running with a TTY
# Debian bug #803249; fixed but not necessarily in package repos yet
echo "" | needrestart -r a 2>&1 | tee /dev/null
}
# Reboot the machine if /var/run/reboot-required exists
reboot-if-required() {
if [[ ! -e "/var/run/reboot-required" ]]; then
return
fi
echo "Reboot is required (/var/run/reboot-required detected)"
if [[ -e "/var/run/reboot-required.pkgs" ]]; then
echo "Packages that triggered reboot:"
cat /var/run/reboot-required.pkgs
fi
# We default to rebooting the machine because this is only done as part of an update
if [[ "${AUTO_REBOOT:-true}" != "true" ]]; then
echo "Reboot prevented by AUTO_REBOOT=${AUTO_REBOOT}"
return
fi
rm -f /var/run/reboot-required
rm -f /var/run/reboot-required.pkgs
echo "Triggering reboot"
init 6
}
# Install upgrades using unattended-upgrades, then reboot or restart services
auto-upgrade() {
# We default to not installing upgrades
if [[ "${AUTO_UPGRADE:-false}" != "true" ]]; then
echo "AUTO_UPGRADE not set to true; won't auto-upgrade"
return
fi
apt-get-install unattended-upgrades needrestart
unattended-upgrade --debug
reboot-if-required # We may reboot the machine right here
restart-updated-services
}
#
# Install salt from GCS. See README.md for instructions on how to update these
# debs.
install-salt() {
if dpkg -s salt-minion &>/dev/null; then
echo "== SaltStack already installed, skipping install step =="
return
fi
echo "== Refreshing package database =="
until apt-get update; do
echo "== apt-get update failed, retrying =="
sleep 5
done
mkdir -p /var/cache/salt-install
cd /var/cache/salt-install
DEBS=(
libzmq3_3.2.3+dfsg-1~bpo70~dst+1_amd64.deb
python-zmq_13.1.0-1~bpo70~dst+1_amd64.deb
salt-common_2014.1.13+ds-1~bpo70+1_all.deb
salt-minion_2014.1.13+ds-1~bpo70+1_all.deb
)
URL_BASE="https://storage.googleapis.com/kubernetes-release/salt"
for deb in "${DEBS[@]}"; do
if [ ! -e "${deb}" ]; then
download-or-bust "" "${URL_BASE}/${deb}"
fi
done
# Based on
# https://major.io/2014/06/26/install-debian-packages-without-starting-daemons/
# We do this to prevent Salt from starting the salt-minion
# daemon. The other packages don't have relevant daemons. (If you
# add a package that needs a daemon started, add it to a different
# list.)
cat > /usr/sbin/policy-rc.d <<EOF
#!/bin/sh
echo "Salt shall not start." >&2
exit 101
EOF
chmod 0755 /usr/sbin/policy-rc.d
for deb in "${DEBS[@]}"; do
echo "== Installing ${deb}, ignore dependency complaints (will fix later) =="
dpkg --skip-same-version --force-depends -i "${deb}"
done
# This will install any of the unmet dependencies from above.
echo "== Installing unmet dependencies =="
until apt-get install -f -y; do
echo "== apt-get install failed, retrying =="
sleep 5
done
rm /usr/sbin/policy-rc.d
# Log a timestamp
echo "== Finished installing Salt =="
}
# Ensure salt-minion isn't running and never runs
stop-salt-minion() {
if [[ -e /etc/init/salt-minion.override ]]; then
# Assume this has already run (upgrade, or baked into containervm)
return
fi
# This ensures it on next reboot
echo manual > /etc/init/salt-minion.override
update-rc.d salt-minion disable
while service salt-minion status >/dev/null; do
echo "salt-minion found running, stopping"
service salt-minion stop
sleep 1
done
}
# Finds the master PD device; returns it in MASTER_PD_DEVICE
find-master-pd() {
MASTER_PD_DEVICE=""
if [[ ! -e /dev/disk/by-id/google-master-pd ]]; then
return
fi
device_info=$(ls -l /dev/disk/by-id/google-master-pd)
relative_path=${device_info##* }
MASTER_PD_DEVICE="/dev/disk/by-id/${relative_path}"
}
# Create the overlay files for the salt tree. We create these in a separate
# place so that we can blow away the rest of the salt configs on a kube-push and
# re-apply these.
function create-salt-pillar() {
# Always overwrite the cluster-params.sls (even on a push, we have
# these variables)
mkdir -p /srv/salt-overlay/pillar
cat <<EOF >/srv/salt-overlay/pillar/cluster-params.sls
instance_prefix: '$(echo "$INSTANCE_PREFIX" | sed -e "s/'/''/g")'
node_tags: '$(echo "$NODE_TAGS" | sed -e "s/'/''/g")'
node_instance_prefix: '$(echo "$NODE_INSTANCE_PREFIX" | sed -e "s/'/''/g")'
cluster_cidr: '$(echo "$CLUSTER_IP_RANGE" | sed -e "s/'/''/g")'
allocate_node_cidrs: '$(echo "$ALLOCATE_NODE_CIDRS" | sed -e "s/'/''/g")'
non_masquerade_cidr: '$(echo "$NON_MASQUERADE_CIDR" | sed -e "s/'/''/g")'
service_cluster_ip_range: '$(echo "$SERVICE_CLUSTER_IP_RANGE" | sed -e "s/'/''/g")'
enable_cluster_monitoring: '$(echo "$ENABLE_CLUSTER_MONITORING" | sed -e "s/'/''/g")'
enable_cluster_logging: '$(echo "$ENABLE_CLUSTER_LOGGING" | sed -e "s/'/''/g")'
enable_cluster_ui: '$(echo "$ENABLE_CLUSTER_UI" | sed -e "s/'/''/g")'
enable_node_problem_detector: '$(echo "$ENABLE_NODE_PROBLEM_DETECTOR" | sed -e "s/'/''/g")'
enable_l7_loadbalancing: '$(echo "$ENABLE_L7_LOADBALANCING" | sed -e "s/'/''/g")'
enable_node_logging: '$(echo "$ENABLE_NODE_LOGGING" | sed -e "s/'/''/g")'
enable_metadata_proxy: '$(echo "$ENABLE_METADATA_CONCEALMENT" | sed -e "s/'/''/g")'
enable_metrics_server: '$(echo "$ENABLE_METRICS_SERVER" | sed -e "s/'/''/g")'
enable_pod_security_policy: '$(echo "$ENABLE_POD_SECURITY_POLICY" | sed -e "s/'/''/g")'
enable_rescheduler: '$(echo "$ENABLE_RESCHEDULER" | sed -e "s/'/''/g")'
logging_destination: '$(echo "$LOGGING_DESTINATION" | sed -e "s/'/''/g")'
elasticsearch_replicas: '$(echo "$ELASTICSEARCH_LOGGING_REPLICAS" | sed -e "s/'/''/g")'
enable_cluster_dns: '$(echo "$ENABLE_CLUSTER_DNS" | sed -e "s/'/''/g")'
cluster_dns_core_dns: '$(echo "$CLUSTER_DNS_CORE_DNS" | sed -e "s/'/''/g")'
enable_cluster_registry: '$(echo "$ENABLE_CLUSTER_REGISTRY" | sed -e "s/'/''/g")'
dns_server: '$(echo "$DNS_SERVER_IP" | sed -e "s/'/''/g")'
dns_domain: '$(echo "$DNS_DOMAIN" | sed -e "s/'/''/g")'
enable_dns_horizontal_autoscaler: '$(echo "$ENABLE_DNS_HORIZONTAL_AUTOSCALER" | sed -e "s/'/''/g")'
admission_control: '$(echo "$ADMISSION_CONTROL" | sed -e "s/'/''/g")'
network_provider: '$(echo "$NETWORK_PROVIDER" | sed -e "s/'/''/g")'
prepull_e2e_images: '$(echo "$PREPULL_E2E_IMAGES" | sed -e "s/'/''/g")'
hairpin_mode: '$(echo "$HAIRPIN_MODE" | sed -e "s/'/''/g")'
softlockup_panic: '$(echo "$SOFTLOCKUP_PANIC" | sed -e "s/'/''/g")'
opencontrail_tag: '$(echo "$OPENCONTRAIL_TAG" | sed -e "s/'/''/g")'
opencontrail_kubernetes_tag: '$(echo "$OPENCONTRAIL_KUBERNETES_TAG")'
opencontrail_public_subnet: '$(echo "$OPENCONTRAIL_PUBLIC_SUBNET")'
network_policy_provider: '$(echo "$NETWORK_POLICY_PROVIDER" | sed -e "s/'/''/g")'
enable_manifest_url: '$(echo "${ENABLE_MANIFEST_URL:-}" | sed -e "s/'/''/g")'
manifest_url: '$(echo "${MANIFEST_URL:-}" | sed -e "s/'/''/g")'
manifest_url_header: '$(echo "${MANIFEST_URL_HEADER:-}" | sed -e "s/'/''/g")'
num_nodes: $(echo "${NUM_NODES:-}" | sed -e "s/'/''/g")
e2e_storage_test_environment: '$(echo "$E2E_STORAGE_TEST_ENVIRONMENT" | sed -e "s/'/''/g")'
kube_uid: '$(echo "${KUBE_UID}" | sed -e "s/'/''/g")'
initial_etcd_cluster: '$(echo "${INITIAL_ETCD_CLUSTER:-}" | sed -e "s/'/''/g")'
initial_etcd_cluster_state: '$(echo "${INITIAL_ETCD_CLUSTER_STATE:-}" | sed -e "s/'/''/g")'
ca_cert_bundle_path: '$(echo "${CA_CERT_BUNDLE_PATH:-}" | sed -e "s/'/''/g")'
hostname: '$(echo "${ETCD_HOSTNAME:-$(hostname -s)}" | sed -e "s/'/''/g")'
enable_pod_priority: '$(echo "${ENABLE_POD_PRIORITY:-}" | sed -e "s/'/''/g")'
enable_default_storage_class: '$(echo "$ENABLE_DEFAULT_STORAGE_CLASS" | sed -e "s/'/''/g")'
kube_proxy_daemonset: '$(echo "$KUBE_PROXY_DAEMONSET" | sed -e "s/'/''/g")'
EOF
if [ -n "${STORAGE_BACKEND:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
storage_backend: '$(echo "$STORAGE_BACKEND" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${STORAGE_MEDIA_TYPE:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
storage_media_type: '$(echo "$STORAGE_MEDIA_TYPE" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${KUBE_APISERVER_REQUEST_TIMEOUT_SEC:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
kube_apiserver_request_timeout_sec: '$(echo "$KUBE_APISERVER_REQUEST_TIMEOUT_SEC" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${ADMISSION_CONTROL:-}" ] && [ ${ADMISSION_CONTROL} == *"ImagePolicyWebhook"* ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
admission-control-config-file: /etc/admission_controller.config
EOF
fi
if [ -n "${KUBELET_PORT:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
kubelet_port: '$(echo "$KUBELET_PORT" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${ETCD_IMAGE:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
etcd_docker_tag: '$(echo "$ETCD_IMAGE" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${ETCD_DOCKER_REPOSITORY:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
etcd_docker_repository: '$(echo "$ETCD_DOCKER_REPOSITORY" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${ETCD_VERSION:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
etcd_version: '$(echo "$ETCD_VERSION" | sed -e "s/'/''/g")'
EOF
fi
if [[ -n "${ETCD_CA_KEY:-}" && -n "${ETCD_CA_CERT:-}" && -n "${ETCD_PEER_KEY:-}" && -n "${ETCD_PEER_CERT:-}" ]]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
etcd_over_ssl: 'true'
EOF
else
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
etcd_over_ssl: 'false'
EOF
fi
if [ -n "${ETCD_QUORUM_READ:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
etcd_quorum_read: '$(echo "${ETCD_QUORUM_READ}" | sed -e "s/'/''/g")'
EOF
fi
# Configuration changes for test clusters
if [ -n "${APISERVER_TEST_ARGS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
apiserver_test_args: '$(echo "$APISERVER_TEST_ARGS" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${API_SERVER_TEST_LOG_LEVEL:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
api_server_test_log_level: '$(echo "$API_SERVER_TEST_LOG_LEVEL" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${KUBELET_TEST_ARGS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
kubelet_test_args: '$(echo "$KUBELET_TEST_ARGS" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${KUBELET_TEST_LOG_LEVEL:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
kubelet_test_log_level: '$(echo "$KUBELET_TEST_LOG_LEVEL" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${DOCKER_TEST_LOG_LEVEL:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
docker_test_log_level: '$(echo "$DOCKER_TEST_LOG_LEVEL" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${CONTROLLER_MANAGER_TEST_ARGS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
controller_manager_test_args: '$(echo "$CONTROLLER_MANAGER_TEST_ARGS" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${CONTROLLER_MANAGER_TEST_LOG_LEVEL:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
controller_manager_test_log_level: '$(echo "$CONTROLLER_MANAGER_TEST_LOG_LEVEL" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${SCHEDULER_TEST_ARGS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
scheduler_test_args: '$(echo "$SCHEDULER_TEST_ARGS" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${SCHEDULER_TEST_LOG_LEVEL:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
scheduler_test_log_level: '$(echo "$SCHEDULER_TEST_LOG_LEVEL" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${KUBEPROXY_TEST_ARGS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
kubeproxy_test_args: '$(echo "$KUBEPROXY_TEST_ARGS" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${KUBEPROXY_TEST_LOG_LEVEL:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
kubeproxy_test_log_level: '$(echo "$KUBEPROXY_TEST_LOG_LEVEL" | sed -e "s/'/''/g")'
EOF
fi
# TODO: Replace this with a persistent volume (and create it).
if [[ "${ENABLE_CLUSTER_REGISTRY}" == true && -n "${CLUSTER_REGISTRY_DISK}" ]]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
cluster_registry_disk_type: gce
cluster_registry_disk_size: $(echo $(convert-bytes-gce-kube ${CLUSTER_REGISTRY_DISK_SIZE}) | sed -e "s/'/''/g")
cluster_registry_disk_name: $(echo ${CLUSTER_REGISTRY_DISK} | sed -e "s/'/''/g")
EOF
fi
if [ -n "${TERMINATED_POD_GC_THRESHOLD:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
terminated_pod_gc_threshold: '$(echo "${TERMINATED_POD_GC_THRESHOLD}" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${ENABLE_CUSTOM_METRICS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
enable_custom_metrics: '$(echo "${ENABLE_CUSTOM_METRICS}" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${NODE_LABELS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
node_labels: '$(echo "${NODE_LABELS}" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${NON_MASTER_NODE_LABELS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
non_master_node_labels: '$(echo "${NON_MASTER_NODE_LABELS}" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${NODE_TAINTS:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
node_taints: '$(echo "${NODE_TAINTS}" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${EVICTION_HARD:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
eviction_hard: '$(echo "${EVICTION_HARD}" | sed -e "s/'/''/g")'
EOF
fi
if [[ "${ENABLE_CLUSTER_AUTOSCALER:-false}" == "true" ]]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
enable_cluster_autoscaler: '$(echo "${ENABLE_CLUSTER_AUTOSCALER}" | sed -e "s/'/''/g")'
autoscaler_mig_config: '$(echo "${AUTOSCALER_MIG_CONFIG}" | sed -e "s/'/''/g")'
autoscaler_expander_config: '$(echo "${AUTOSCALER_EXPANDER_CONFIG}" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${SCHEDULING_ALGORITHM_PROVIDER:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
scheduling_algorithm_provider: '$(echo "${SCHEDULING_ALGORITHM_PROVIDER}" | sed -e "s/'/''/g")'
EOF
fi
if [ -n "${ENABLE_IP_ALIASES:-}" ]; then
cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls
enable_ip_aliases: '$(echo "$ENABLE_IP_ALIASES" | sed -e "s/'/''/g")'
EOF
fi
}
# The job of this function is simple, but the basic regular expression syntax makes
# this difficult to read. What we want to do is convert from [0-9]+B, KB, KiB, MB, etc
# into [0-9]+, Ki, Mi, Gi, etc.
# This is done in two steps:
# 1. Convert from [0-9]+X?i?B into [0-9]X? (X denotes the prefix, ? means the field
# is optional.
# 2. Attach an 'i' to the end of the string if we find a letter.
# The two step process is needed to handle the edge case in which we want to convert
# a raw byte count, as the result should be a simple number (e.g. 5B -> 5).
function convert-bytes-gce-kube() {
local -r storage_space=$1
echo "${storage_space}" | sed -e 's/^\([0-9]\+\)\([A-Z]\)\?i\?B$/\1\2/g' -e 's/\([A-Z]\)$/\1i/'
}
# This should happen both on cluster initialization and node upgrades.
#
# - Uses KUBELET_CA_CERT (falling back to CA_CERT), KUBELET_CERT, and
# KUBELET_KEY to generate a kubeconfig file for the kubelet to securely
# connect to the apiserver.
function create-salt-kubelet-auth() {
local -r kubelet_kubeconfig_file="/srv/salt-overlay/salt/kubelet/bootstrap-kubeconfig"
if [ ! -e "${kubelet_kubeconfig_file}" ]; then
mkdir -p /srv/salt-overlay/salt/kubelet
(umask 077;
cat > "${kubelet_kubeconfig_file}" <<EOF
apiVersion: v1
kind: Config
users:
- name: kubelet
user:
client-certificate: ${KUBELET_CERT_PATH}
client-key: ${KUBELET_KEY_PATH}
clusters:
- name: local
cluster:
server: https://${KUBERNETES_MASTER_NAME}
certificate-authority: ${CA_CERT_BUNDLE_PATH}
contexts:
- context:
cluster: local
user: kubelet
name: service-account-context
current-context: service-account-context
EOF
)
fi
}
# This should happen both on cluster initialization and node upgrades.
#
# - When run as static pods, use the CA_CERT and KUBE_PROXY_TOKEN to generate a
# kubeconfig file for the kube-proxy to securely connect to the apiserver.
function create-salt-kubeproxy-auth() {
local -r kube_proxy_kubeconfig_file="/srv/salt-overlay/salt/kube-proxy/kubeconfig"
if [ ! -e "${kube_proxy_kubeconfig_file}" ]; then
mkdir -p /srv/salt-overlay/salt/kube-proxy
(umask 077;
cat > "${kube_proxy_kubeconfig_file}" <<EOF
apiVersion: v1
kind: Config
users:
- name: kube-proxy
user:
token: ${KUBE_PROXY_TOKEN}
clusters:
- name: local
cluster:
certificate-authority-data: ${CA_CERT_BUNDLE}
contexts:
- context:
cluster: local
user: kube-proxy
name: service-account-context
current-context: service-account-context
EOF
)
fi
}
function split-commas() {
echo $1 | tr "," "\n"
}
function try-download-release() {
# TODO(zmerlynn): Now we REALLy have no excuse not to do the reboot
# optimization.
local -r server_binary_tar_urls=( $(split-commas "${SERVER_BINARY_TAR_URL}") )
local -r server_binary_tar="${server_binary_tar_urls[0]##*/}"
if [[ -n "${SERVER_BINARY_TAR_HASH:-}" ]]; then
local -r server_binary_tar_hash="${SERVER_BINARY_TAR_HASH}"
else
echo "Downloading binary release sha1 (not found in env)"
download-or-bust "" "${server_binary_tar_urls[@]/.tar.gz/.tar.gz.sha1}"
local -r server_binary_tar_hash=$(cat "${server_binary_tar}.sha1")
fi
echo "Downloading binary release tar (${server_binary_tar_urls[@]})"
download-or-bust "${server_binary_tar_hash}" "${server_binary_tar_urls[@]}"
local -r salt_tar_urls=( $(split-commas "${SALT_TAR_URL}") )
local -r salt_tar="${salt_tar_urls[0]##*/}"
if [[ -n "${SALT_TAR_HASH:-}" ]]; then
local -r salt_tar_hash="${SALT_TAR_HASH}"
else
echo "Downloading Salt tar sha1 (not found in env)"
download-or-bust "" "${salt_tar_urls[@]/.tar.gz/.tar.gz.sha1}"
local -r salt_tar_hash=$(cat "${salt_tar}.sha1")
fi
echo "Downloading Salt tar (${salt_tar_urls[@]})"
download-or-bust "${salt_tar_hash}" "${salt_tar_urls[@]}"
echo "Unpacking Salt tree and checking integrity of binary release tar"
rm -rf kubernetes
tar xzf "${salt_tar}" && tar tzf "${server_binary_tar}" > /dev/null
}
function download-release() {
# In case of failure checking integrity of release, retry.
until try-download-release; do
sleep 15
echo "Couldn't download release. Retrying..."
done
echo "Running release install script"
kubernetes/saltbase/install.sh "${SERVER_BINARY_TAR_URL##*/}"
}
function fix-apt-sources() {
sed -i -e "\|^deb.*http://http.debian.net/debian| s/^/#/" /etc/apt/sources.list
sed -i -e "\|^deb.*http://ftp.debian.org/debian| s/^/#/" /etc/apt/sources.list.d/backports.list
}
function salt-run-local() {
cat <<EOF >/etc/salt/minion.d/local.conf
file_client: local
file_roots:
base:
- /srv/salt
EOF
}
function salt-debug-log() {
cat <<EOF >/etc/salt/minion.d/log-level-debug.conf
log_level: debug
log_level_logfile: debug
EOF
}
function salt-node-role() {
local -r kubelet_bootstrap_kubeconfig="/srv/salt-overlay/salt/kubelet/bootstrap-kubeconfig"
local -r kubelet_kubeconfig="/srv/salt-overlay/salt/kubelet/kubeconfig"
cat <<EOF >/etc/salt/minion.d/grains.conf
grains:
roles:
- kubernetes-pool
cloud: gce
api_servers: '${KUBERNETES_MASTER_NAME}'
kubelet_bootstrap_kubeconfig: /var/lib/kubelet/bootstrap-kubeconfig
kubelet_kubeconfig: /var/lib/kubelet/kubeconfig
EOF
}
function env-to-grains {
local key=$1
local env_key=`echo $key | tr '[:lower:]' '[:upper:]'`
local value=${!env_key:-}
if [[ -n "${value}" ]]; then
# Note this is yaml, so indentation matters
cat <<EOF >>/etc/salt/minion.d/grains.conf
${key}: '$(echo "${value}" | sed -e "s/'/''/g")'
EOF
fi
}
function node-docker-opts() {
if [[ -n "${EXTRA_DOCKER_OPTS-}" ]]; then
DOCKER_OPTS="${DOCKER_OPTS:-} ${EXTRA_DOCKER_OPTS}"
fi
# Decide whether to enable a docker registry mirror. This is taken from
# the "kube-env" metadata value.
if [[ -n "${DOCKER_REGISTRY_MIRROR_URL:-}" ]]; then
echo "Enable docker registry mirror at: ${DOCKER_REGISTRY_MIRROR_URL}"
DOCKER_OPTS="${DOCKER_OPTS:-} --registry-mirror=${DOCKER_REGISTRY_MIRROR_URL}"
fi
}
function salt-grains() {
env-to-grains "docker_opts"
env-to-grains "docker_root"
env-to-grains "kubelet_root"
env-to-grains "feature_gates"
}
function configure-salt() {
mkdir -p /etc/salt/minion.d
salt-run-local
salt-node-role
node-docker-opts
salt-grains
install-salt
stop-salt-minion
}
function run-salt() {
echo "== Calling Salt =="
local rc=0
for i in {0..6}; do
salt-call --retcode-passthrough --local state.highstate && rc=0 || rc=$?
if [[ "${rc}" == 0 ]]; then
return 0
fi
done
echo "Salt failed to run repeatedly" >&2
return "${rc}"
}
function run-user-script() {
if curl-metadata k8s-user-startup-script > "${INSTALL_DIR}/k8s-user-script.sh"; then
user_script=$(cat "${INSTALL_DIR}/k8s-user-script.sh")
fi
if [[ ! -z ${user_script:-} ]]; then
chmod u+x "${INSTALL_DIR}/k8s-user-script.sh"
echo "== running user startup script =="
"${INSTALL_DIR}/k8s-user-script.sh"
fi
}
if [[ "${KUBERNETES_MASTER:-}" == "true" ]]; then
echo "Support for debian master has been removed"
exit 1
fi
if [[ -z "${is_push}" ]]; then
echo "== kube-up node config starting =="
set-broken-motd
ensure-basic-networking
fix-apt-sources
ensure-install-dir
ensure-packages
set-kube-env
auto-upgrade
ensure-local-disks
create-node-pki
create-salt-pillar
create-salt-kubelet-auth
if [[ "${KUBE_PROXY_DAEMONSET:-}" != "true" ]]; then
create-salt-kubeproxy-auth
fi
download-release
configure-salt
remove-docker-artifacts
config-ip-firewall
run-salt
reset-motd
run-user-script
echo "== kube-up node config done =="
else
echo "== kube-push node config starting =="
ensure-basic-networking
ensure-install-dir
set-kube-env
create-salt-pillar
download-release
reset-motd
run-salt
echo "== kube-push node config done =="
fi

8
cluster/gce/container-linux/OWNERS
View File

@@ -1,8 +0,0 @@
approvers:
- euank
- yifan-gu
- ethernetdan
reviewers:
- euank
- yifan-gu
- ethernetdan

8
cluster/gce/container-linux/README.md
View File

@@ -1,8 +0,0 @@
# Container Linux image
The [Container Linux Operating System](https://coreos.com/why/) is a Linux distribution optimized for running containers securely at scale.
CoreOS provides [a Container Linux image](https://coreos.com/os/docs/latest/booting-on-google-compute-engine.html) for Google Cloud Platform (GCP).
This folder contains configuration and tooling to allow kube-up to create a Kubernetes cluster on Google Cloud Platform running on the official Container Linux image.
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/cluster/gce/container-linux/README.md?pixel)]()

1598
cluster/gce/container-linux/configure-helper.sh
View File

File diff suppressed because it is too large Load Diff

176
cluster/gce/container-linux/configure.sh
View File

@@ -1,176 +0,0 @@
#!/bin/bash
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
function download-kube-env {
# Fetch kube-env from GCE metadata server.
local -r tmp_kube_env="/tmp/kube-env.yaml"
curl --fail --retry 5 --retry-delay 3 --silent --show-error \
-H "X-Google-Metadata-Request: True" \
-o "${tmp_kube_env}" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-env
# Convert the yaml format file into a shell-style file.
sed 's/: /=/' < "${tmp_kube_env}" > "${KUBE_HOME}/kube-env"
rm -f "${tmp_kube_env}"
}
function validate-hash {
local -r file="$1"
local -r expected="$2"
actual=$(sha1sum ${file} | awk '{ print $1 }') || true
if [[ "${actual}" != "${expected}" ]]; then
echo "== ${file} corrupted, sha1 ${actual} doesn't match expected ${expected} =="
return 1
fi
}
# Retry a download until we get it. Takes a hash and a set of URLs.
#
# $1 is the sha1 of the URL. Can be "" if the sha1 is unknown.
# $2+ are the URLs to download.
function download-or-bust {
local -r hash="$1"
shift 1
local -r urls=( $* )
while true; do
for url in "${urls[@]}"; do
local file="${url##*/}"
rm -f "${file}"
if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --max-time 300 --retry 6 --retry-delay 10 "${url}"; then
echo "== Failed to download ${url}. Retrying. =="
elif [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
echo "== Hash validation of ${url} failed. Retrying. =="
else
if [[ -n "${hash}" ]]; then
echo "== Downloaded ${url} (SHA1 = ${hash}) =="
else
echo "== Downloaded ${url} =="
fi
return
fi
done
done
}
function split-commas {
echo $1 | tr "," "\n"
}
# Downloads kubernetes binaries and kube-system manifest tarball, unpacks them,
# and places them into suitable directories. Files are placed in /opt/kubernetes.
function install-kube-binary-config {
cd "${KUBE_HOME}"
local -r server_binary_tar_urls=( $(split-commas "${SERVER_BINARY_TAR_URL}") )
local -r server_binary_tar="${server_binary_tar_urls[0]##*/}"
if [[ -n "${SERVER_BINARY_TAR_HASH:-}" ]]; then
local -r server_binary_tar_hash="${SERVER_BINARY_TAR_HASH}"
else
echo "Downloading binary release sha1 (not found in env)"
download-or-bust "" "${server_binary_tar_urls[@]/.tar.gz/.tar.gz.sha1}"
local -r server_binary_tar_hash=$(cat "${server_binary_tar}.sha1")
fi
echo "Downloading binary release tar"
download-or-bust "${server_binary_tar_hash}" "${server_binary_tar_urls[@]}"
tar xzf "${KUBE_HOME}/${server_binary_tar}" -C "${KUBE_HOME}" --overwrite
# Copy docker_tag and image files to ${KUBE_HOME}/kube-docker-files.
src_dir="${KUBE_HOME}/kubernetes/server/bin"
dst_dir="${KUBE_HOME}/kube-docker-files"
mkdir -p "${dst_dir}"
cp "${src_dir}/"*.docker_tag "${dst_dir}"
if [[ "${KUBERNETES_MASTER:-}" == "false" ]]; then
cp "${src_dir}/kube-proxy.tar" "${dst_dir}"
else
cp "${src_dir}/kube-apiserver.tar" "${dst_dir}"
cp "${src_dir}/kube-controller-manager.tar" "${dst_dir}"
cp "${src_dir}/kube-scheduler.tar" "${dst_dir}"
cp -r "${KUBE_HOME}/kubernetes/addons" "${dst_dir}"
fi
local -r kube_bin="${KUBE_HOME}/bin"
mv "${src_dir}/kubelet" "${kube_bin}"
mv "${src_dir}/kubectl" "${kube_bin}"
if [[ "${NETWORK_PROVIDER:-}" == "kubenet" ]] || \
[[ "${NETWORK_PROVIDER:-}" == "cni" ]]; then
local -r cni_version="v0.6.0"
local -r cni_tar="cni-plugins-amd64-${cni_version}.tgz"
local -r cni_sha1="d595d3ded6499a64e8dac02466e2f5f2ce257c9f"
download-or-bust "${cni_sha1}" "https://storage.googleapis.com/kubernetes-release/network-plugins/${cni_tar}"
local -r cni_dir="${KUBE_HOME}/cni"
mkdir -p "${cni_dir}/bin"
tar xzf "${KUBE_HOME}/${cni_tar}" -C "${cni_dir}/bin" --overwrite
mv "${cni_dir}/bin"/* "${kube_bin}"
rmdir "${cni_dir}/bin"
rm -f "${KUBE_HOME}/${cni_tar}"
fi
mv "${KUBE_HOME}/kubernetes/LICENSES" "${KUBE_HOME}"
mv "${KUBE_HOME}/kubernetes/kubernetes-src.tar.gz" "${KUBE_HOME}"
# Put kube-system pods manifests in ${KUBE_HOME}/kube-manifests/.
dst_dir="${KUBE_HOME}/kube-manifests"
mkdir -p "${dst_dir}"
local -r manifests_tar_urls=( $(split-commas "${KUBE_MANIFESTS_TAR_URL}") )
local -r manifests_tar="${manifests_tar_urls[0]##*/}"
if [ -n "${KUBE_MANIFESTS_TAR_HASH:-}" ]; then
local -r manifests_tar_hash="${KUBE_MANIFESTS_TAR_HASH}"
else
echo "Downloading k8s manifests sha1 (not found in env)"
download-or-bust "" "${manifests_tar_urls[@]/.tar.gz/.tar.gz.sha1}"
local -r manifests_tar_hash=$(cat "${manifests_tar}.sha1")
fi
echo "Downloading k8s manifests tar"
download-or-bust "${manifests_tar_hash}" "${manifests_tar_urls[@]}"
tar xzf "${KUBE_HOME}/${manifests_tar}" -C "${dst_dir}" --overwrite
local -r kube_addon_registry="${KUBE_ADDON_REGISTRY:-gcr.io/google_containers}"
if [[ "${kube_addon_registry}" != "gcr.io/google_containers" ]]; then
find "${dst_dir}" -name \*.yaml -or -name \*.yaml.in | \
xargs sed -ri "s@(image:\s.*)gcr.io/google_containers@\1${kube_addon_registry}@"
find "${dst_dir}" -name \*.manifest -or -name \*.json | \
xargs sed -ri "s@(image\":\s+\")gcr.io/google_containers@\1${kube_addon_registry}@"
fi
cp "${dst_dir}/kubernetes/gci-trusty/container-linux-configure-helper.sh" "${KUBE_HOME}/bin/configure-helper.sh"
chmod -R 755 "${kube_bin}"
# Clean up.
rm -rf "${KUBE_HOME}/kubernetes"
rm -f "${KUBE_HOME}/${server_binary_tar}"
rm -f "${KUBE_HOME}/${server_binary_tar}.sha1"
rm -f "${KUBE_HOME}/${manifests_tar}"
rm -f "${KUBE_HOME}/${manifests_tar}.sha1"
}
######### Main Function ##########
echo "Start to install kubernetes files"
KUBE_HOME="/opt/kubernetes"
mkdir -p "${KUBE_HOME}"
download-kube-env
source "${KUBE_HOME}/kube-env"
install-kube-binary-config
echo "Done for installing kubernetes files"
# On Container Linux, the hosts is in /usr/share/baselayout/hosts
# So we need to manually populdate the hosts file here on gce.
echo "127.0.0.1 localhost" >> /etc/hosts
echo "::1 localhost" >> /etc/hosts
echo "Configuring hostname"
hostnamectl set-hostname $(hostname | cut -f1 -d.)

83
cluster/gce/container-linux/health-monitor.sh
View File

@@ -1,83 +0,0 @@
#!/bin/bash
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This script is for master and node instance health monitoring, which is
# packed in kube-manifest tarball. It is executed through a systemd service
# in cluster/gce/gci/<master/node>.yaml. The env variables come from an env
# file provided by the systemd service.
set -o nounset
set -o pipefail
# We simply kill the process when there is a failure. Another systemd service will
# automatically restart the process.
function docker_monitoring {
while [ 1 ]; do
if ! timeout 60 docker ps > /dev/null; then
echo "Docker daemon failed!"
pkill docker
# Wait for a while, as we don't want to kill it again before it is really up.
sleep 30
else
sleep "${SLEEP_SECONDS}"
fi
done
}
function kubelet_monitoring {
echo "Wait for 2 minutes for kubelet to be fuctional"
# TODO(andyzheng0831): replace it with a more reliable method if possible.
sleep 120
local -r max_seconds=10
local output=""
while [ 1 ]; do
if ! output=$(curl --insecure -m "${max_seconds}" -f -s -S https://127.0.0.1:${KUBELET_PORT:-10250}/healthz 2>&1); then
# Print the response and/or errors.
echo $output
echo "Kubelet is unhealthy!"
pkill kubelet
# Wait for a while, as we don't want to kill it again before it is really up.
sleep 60
else
sleep "${SLEEP_SECONDS}"
fi
done
}
############## Main Function ################
if [[ "$#" -ne 1 ]]; then
echo "Usage: health-monitor.sh <docker/kubelet>"
exit 1
fi
KUBE_ENV="/home/kubernetes/kube-env"
if [[ ! -e "${KUBE_ENV}" ]]; then
echo "The ${KUBE_ENV} file does not exist!! Terminate health monitoring"
exit 1
fi
SLEEP_SECONDS=10
component=$1
echo "Start kubernetes health monitoring for ${component}"
source "${KUBE_ENV}"
if [[ "${component}" == "docker" ]]; then
docker_monitoring
elif [[ "${component}" == "kubelet" ]]; then
kubelet_monitoring
else
echo "Health monitoring for component "${component}" is not supported!"
fi

19
cluster/gce/container-linux/helper.sh
View File

@@ -1,19 +0,0 @@
#!/bin/bash
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# A library of helper functions and constants for the Container Linux distro.
# This file intentionally left blank

146
cluster/gce/container-linux/master-helper.sh
View File

@@ -1,146 +0,0 @@
#!/bin/bash
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# A library of helper functions and constant for the Container Linux distro.
source "${KUBE_ROOT}/cluster/gce/container-linux/helper.sh"
# create-master-instance creates the master instance. If called with
# an argument, the argument is used as the name to a reserved IP
# address for the master. (In the case of upgrade/repair, we re-use
# the same IP.)
#
# It requires a whole slew of assumed variables, partially due to to
# the call to write-master-env. Listing them would be rather
# futile. Instead, we list the required calls to ensure any additional
#
# variables are set:
# ensure-temp-dir
# detect-project
# get-bearer-token
function create-master-instance {
local address=""
[[ -n ${1:-} ]] && address="${1}"
write-master-env
create-master-instance-internal "${MASTER_NAME}" "${address}"
}
function replicate-master-instance() {
local existing_master_zone="${1}"
local existing_master_name="${2}"
local existing_master_replicas="${3}"
local kube_env="$(get-metadata "${existing_master_zone}" "${existing_master_name}" kube-env)"
# Substitute INITIAL_ETCD_CLUSTER to enable etcd clustering.
kube_env="$(echo "${kube_env}" | grep -v "INITIAL_ETCD_CLUSTER")"
kube_env="$(echo -e "${kube_env}\nINITIAL_ETCD_CLUSTER: '${existing_master_replicas},${REPLICA_NAME}'")"
ETCD_CA_KEY="$(echo "${kube_env}" | grep "ETCD_CA_KEY" | sed "s/^.*: '//" | sed "s/'$//")"
ETCD_CA_CERT="$(echo "${kube_env}" | grep "ETCD_CA_CERT" | sed "s/^.*: '//" | sed "s/'$//")"
create-etcd-certs "${REPLICA_NAME}" "${ETCD_CA_CERT}" "${ETCD_CA_KEY}"
kube_env="$(echo "${kube_env}" | grep -v "ETCD_PEER_KEY")"
kube_env="$(echo -e "${kube_env}\nETCD_PEER_KEY: '${ETCD_PEER_KEY_BASE64}'")"
kube_env="$(echo "${kube_env}" | grep -v "ETCD_PEER_CERT")"
kube_env="$(echo -e "${kube_env}\nETCD_PEER_CERT: '${ETCD_PEER_CERT_BASE64}'")"
echo "${kube_env}" > ${KUBE_TEMP}/master-kube-env.yaml
get-metadata "${existing_master_zone}" "${existing_master_name}" cluster-name > "${KUBE_TEMP}/cluster-name.txt"
create-master-instance-internal "${REPLICA_NAME}"
}
function create-master-instance-internal() {
local gcloud="gcloud"
local retries=5
if [[ "${ENABLE_IP_ALIASES:-}" == 'true' ]]; then
gcloud="gcloud beta"
fi
local -r master_name="${1}"
local -r address="${2:-}"
local preemptible_master=""
if [[ "${PREEMPTIBLE_MASTER:-}" == "true" ]]; then
preemptible_master="--preemptible --maintenance-policy TERMINATE"
fi
local enable_ip_aliases
if [[ "${NODE_IPAM_MODE:-}" == "CloudAllocator" ]]; then
enable_ip_aliases=true
else
enable_ip_aliases=false
fi
local network=$(make-gcloud-network-argument \
"${NETWORK_PROJECT}" "${REGION}" "${NETWORK}" "${SUBNETWORK:-}" \
"${address:-}" "${enable_ip_aliases:-}" "${IP_ALIAS_SIZE:-}")
local metadata="kube-env=${KUBE_TEMP}/master-kube-env.yaml"
metadata="${metadata},user-data=${KUBE_ROOT}/cluster/gce/container-linux/master.yaml"
metadata="${metadata},configure-sh=${KUBE_ROOT}/cluster/gce/container-linux/configure.sh"
metadata="${metadata},cluster-name=${KUBE_TEMP}/cluster-name.txt"
local disk="name=${master_name}-pd"
disk="${disk},device-name=master-pd"
disk="${disk},mode=rw"
disk="${disk},boot=no"
disk="${disk},auto-delete=no"
for attempt in $(seq 1 ${retries}); do
if result=$(${gcloud} compute instances create "${master_name}" \
--project "${PROJECT}" \
--zone "${ZONE}" \
--machine-type "${MASTER_SIZE}" \
--image-project="${MASTER_IMAGE_PROJECT}" \
--image "${MASTER_IMAGE}" \
--tags "${MASTER_TAG}" \
--scopes "storage-ro,compute-rw,monitoring,logging-write" \
--metadata-from-file "${metadata}" \
--disk "${disk}" \
--boot-disk-size "${MASTER_ROOT_DISK_SIZE}" \
${MASTER_MIN_CPU_ARCHITECTURE:+"--min-cpu-platform=${MASTER_MIN_CPU_ARCHITECTURE}"} \
${preemptible_master} \
${network} 2>&1); then
echo "${result}" >&2
return 0
else
echo "${result}" >&2
if [[ ! "${result}" =~ "try again later" ]]; then
echo "Failed to create master instance due to non-retryable error" >&2
return 1
fi
sleep 10
fi
done
echo "Failed to create master instance despite ${retries} attempts" >&2
return 1
}
function get-metadata() {
local zone="${1}"
local name="${2}"
local key="${3}"
local metadata_url="http://metadata.google.internal/computeMetadata/v1/instance/attributes/${key}"
gcloud compute ssh "${name}" \
--project "${PROJECT}" \
--zone "${zone}" \
--command "curl '${metadata_url}' -H 'Metadata-Flavor: Google'" 2>/dev/null
}

56
cluster/gce/container-linux/master.yaml
View File

@@ -1,56 +0,0 @@
#cloud-config
coreos:
update:
reboot-strategy: off
units:
- name: locksmithd.service
mask: true
- name: kube-master-installation.service
command: start
content: |
[Unit]
Description=Download and install k8s binaries and configurations
After=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/bin/mkdir -p /opt/kubernetes/bin
ExecStartPre=/usr/bin/curl --fail --retry 5 --retry-delay 3 --silent --show-error -H "X-Google-Metadata-Request: True" -o /opt/kubernetes/bin/configure.sh http://metadata.google.internal/computeMetadata/v1/instance/attributes/configure-sh
ExecStartPre=/bin/chmod 544 /opt/kubernetes/bin/configure.sh
ExecStart=/opt/kubernetes/bin/configure.sh
[Install]
WantedBy=kubernetes.target
- name: kube-master-configuration.service
command: start
content: |
[Unit]
Description=Configure kubernetes master
After=kube-master-installation.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/bin/chmod 544 /opt/kubernetes/bin/configure-helper.sh
ExecStart=/opt/kubernetes/bin/configure-helper.sh
[Install]
WantedBy=kubernetes.target
- name: kubernetes.target
enable: true
command: start
content: |
[Unit]
Description=Kubernetes
[Install]
WantedBy=multi-user.target
- name: docker.service
drop-ins:
- name: "use-cgroupfs-driver.conf"
# This is required for setting cgroup parent in the current ~1.4 per-pod cgroup impl
content: |
[Service]
Environment="DOCKER_CGROUPS=--exec-opt native.cgroupdriver="

35
cluster/gce/container-linux/node-helper.sh
View File

@@ -1,35 +0,0 @@
#!/bin/bash
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# A library of helper functions and constant for the Container Linux distro.
source "${KUBE_ROOT}/cluster/gce/container-linux/helper.sh"
function get-node-instance-metadata {
local metadata=""
metadata+="kube-env=${KUBE_TEMP}/node-kube-env.yaml,"
metadata+="user-data=${KUBE_ROOT}/cluster/gce/container-linux/node.yaml,"
metadata+="configure-sh=${KUBE_ROOT}/cluster/gce/container-linux/configure.sh,"
metadata+="cluster-name=${KUBE_TEMP}/cluster-name.txt"
echo "${metadata}"
}
# $1: template name (required).
function create-node-instance-template {
local template_name="$1"
create-node-template "$template_name" "${scope_flags[*]}" "$(get-node-instance-metadata)"
# TODO(euank): We should include update-strategy here. We should also switch to ignition
}

56
cluster/gce/container-linux/node.yaml
View File

@@ -1,56 +0,0 @@
#cloud-config
coreos:
update:
reboot-strategy: off
units:
- name: locksmithd.service
mask: true
- name: kube-node-installation.service
command: start
content: |
[Unit]
Description=Download and install k8s binaries and configurations
After=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/bin/mkdir -p /opt/kubernetes/bin
ExecStartPre=/usr/bin/curl --fail --retry 5 --retry-delay 3 --silent --show-error -H "X-Google-Metadata-Request: True" -o /opt/kubernetes/bin/configure.sh http://metadata.google.internal/computeMetadata/v1/instance/attributes/configure-sh
ExecStartPre=/bin/chmod 544 /opt/kubernetes/bin/configure.sh
ExecStart=/opt/kubernetes/bin/configure.sh
[Install]
WantedBy=kubernetes.target
- name: kube-node-configuration.service
command: start
content: |
[Unit]
Description=Configure kubernetes master
After=kube-node-installation.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/bin/chmod 544 /opt/kubernetes/bin/configure-helper.sh
ExecStart=/opt/kubernetes/bin/configure-helper.sh
[Install]
WantedBy=kubernetes.target
- name: kubernetes.target
enable: true
command: start
content: |
[Unit]
Description=Kubernetes
[Install]
WantedBy=multi-user.target
- name: docker.service
drop-ins:
- name: "use-cgroupfs-driver.conf"
# This is required for setting cgroup parent in the current ~1.4 per-pod cgroup impl
content: |
[Service]
Environment="DOCKER_CGROUPS=--exec-opt native.cgroupdriver="

1
cluster/gce/custom Symbolic link
View File

@@ -0,0 +1 @@
gci

32
cluster/gce/debian/node-helper.sh
View File

@@ -1,32 +0,0 @@
#!/bin/bash
# Copyright 2015 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# A library of helper functions and constant for debian os distro
function get-node-instance-metadata {
local metadata=""
metadata+="startup-script=${KUBE_TEMP}/configure-vm.sh,"
metadata+="kube-env=${KUBE_TEMP}/node-kube-env.yaml,"
metadata+="cluster-name=${KUBE_TEMP}/cluster-name.txt"
echo "${metadata}"
}
# $1: template name (required)
function create-node-instance-template {
local template_name="$1"
prepare-startup-script
create-node-template "$template_name" "${scope_flags}" "$(get-node-instance-metadata)"
}

49
cluster/gce/gci/configure-helper.sh
View File

@@ -28,6 +28,12 @@ set -o pipefail
readonly UUID_MNT_PREFIX="/mnt/disks/by-uuid/google-local-ssds"
readonly UUID_BLOCK_PREFIX="/dev/disk/by-uuid/google-local-ssds"
# Use --retry-connrefused opt only if it's supported by curl.
CURL_RETRY_CONNREFUSED=""
if curl --help | grep -q -- '--retry-connrefused'; then
CURL_RETRY_CONNREFUSED='--retry-connrefused'
fi
function setup-os-params {
# Reset core_pattern. On GCI, the default core_pattern pipes the core dumps to
# /sbin/crash_reporter which is more restrictive in saving crash dumps. So for
@@ -986,6 +992,14 @@ current-context: kube-scheduler
EOF
}
function create-kubescheduler-policy-config {
echo "Creating kube-scheduler policy config file"
mkdir -p /etc/srv/kubernetes/kube-scheduler
cat <<EOF >/etc/srv/kubernetes/kube-scheduler/policy-config
${SCHEDULER_POLICY_CONFIG}
EOF
}
function create-node-problem-detector-kubeconfig {
echo "Creating node-problem-detector kubeconfig file"
mkdir -p /var/lib/node-problem-detector
@@ -1055,23 +1069,11 @@ function assemble-docker-flags {
echo "DOCKER_OPTS=\"${docker_opts} ${EXTRA_DOCKER_OPTS:-}\"" > /etc/default/docker
if [[ "${use_net_plugin}" == "true" ]]; then
# If using a network plugin, extend the docker configuration to always remove
# the network checkpoint to avoid corrupt checkpoints.
# (https://github.com/docker/docker/issues/18283).
echo "Extend the docker.service configuration to remove the network checkpiont"
mkdir -p /etc/systemd/system/docker.service.d
cat <<EOF >/etc/systemd/system/docker.service.d/01network.conf
[Service]
ExecStartPre=/bin/sh -x -c "rm -rf /var/lib/docker/network"
EOF
fi
# Ensure TasksMax is sufficient for docker.
# (https://github.com/kubernetes/kubernetes/issues/51977)
echo "Extend the docker.service configuration to set a higher pids limit"
mkdir -p /etc/systemd/system/docker.service.d
cat <<EOF >/etc/systemd/system/docker.service.d/02tasksmax.conf
cat <<EOF >/etc/systemd/system/docker.service.d/01tasksmax.conf
[Service]
TasksMax=infinity
EOF
@@ -1378,6 +1380,7 @@ function prepare-etcd-manifest {
sed -i -e "s@{{ *hostname *}}@$host_name@g" "${temp_file}"
sed -i -e "s@{{ *srv_kube_path *}}@/etc/srv/kubernetes@g" "${temp_file}"
sed -i -e "s@{{ *etcd_cluster *}}@$etcd_cluster@g" "${temp_file}"
sed -i -e "s@{{ *liveness_probe_initial_delay *}}@${ETCD_LIVENESS_PROBE_INITIAL_DELAY_SEC:-15}@g" "${temp_file}"
# Get default storage backend from manifest file.
local -r default_storage_backend=$(cat "${temp_file}" | \
grep -o "{{ *pillar\.get('storage_backend', '\(.*\)') *}}" | \
@@ -1449,6 +1452,8 @@ function start-etcd-servers {
# CLOUD_CONFIG_VOLUME
# CLOUD_CONFIG_MOUNT
# DOCKER_REGISTRY
# FLEXVOLUME_HOSTPATH_MOUNT
# FLEXVOLUME_HOSTPATH_VOLUME
function compute-master-manifest-variables {
CLOUD_CONFIG_OPT=""
CLOUD_CONFIG_VOLUME=""
@@ -1462,6 +1467,13 @@ function compute-master-manifest-variables {
if [[ -n "${KUBE_DOCKER_REGISTRY:-}" ]]; then
DOCKER_REGISTRY="${KUBE_DOCKER_REGISTRY}"
fi
FLEXVOLUME_HOSTPATH_MOUNT=""
FLEXVOLUME_HOSTPATH_VOLUME=""
if [[ -n "${VOLUME_PLUGIN_DIR:-}" ]]; then
FLEXVOLUME_HOSTPATH_MOUNT="{ \"name\": \"flexvolumedir\", \"mountPath\": \"${VOLUME_PLUGIN_DIR}\", \"readOnly\": true},"
FLEXVOLUME_HOSTPATH_VOLUME="{ \"name\": \"flexvolumedir\", \"hostPath\": {\"path\": \"${VOLUME_PLUGIN_DIR}\"}},"
fi
}
# A helper function that bind mounts kubelet dirs for running mount in a chroot
@@ -1667,7 +1679,7 @@ function start-kube-apiserver {
params+=" --feature-gates=${FEATURE_GATES}"
fi
if [[ -n "${PROJECT_ID:-}" && -n "${TOKEN_URL:-}" && -n "${TOKEN_BODY:-}" && -n "${NODE_NETWORK:-}" ]]; then
local -r vm_external_ip=$(curl --retry 5 --retry-delay 3 --fail --silent -H 'Metadata-Flavor: Google' "http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip")
local -r vm_external_ip=$(curl --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --fail --silent -H 'Metadata-Flavor: Google' "http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip")
if [[ -n "${PROXY_SSH_USER:-}" ]]; then
params+=" --advertise-address=${vm_external_ip}"
params+=" --ssh-user=${PROXY_SSH_USER}"
@@ -1753,6 +1765,7 @@ function start-kube-apiserver {
sed -i -e "s@{{pillar\['kube_docker_registry'\]}}@${DOCKER_REGISTRY}@g" "${src_file}"
sed -i -e "s@{{pillar\['kube-apiserver_docker_tag'\]}}@${kube_apiserver_docker_tag}@g" "${src_file}"
sed -i -e "s@{{pillar\['allow_privileged'\]}}@true@g" "${src_file}"
sed -i -e "s@{{liveness_probe_initial_delay}}@${KUBE_APISERVER_LIVENESS_PROBE_INITIAL_DELAY_SEC:-15}@g" "${src_file}"
sed -i -e "s@{{secure_port}}@443@g" "${src_file}"
sed -i -e "s@{{secure_port}}@8080@g" "${src_file}"
sed -i -e "s@{{additional_cloud_config_mount}}@@g" "${src_file}"
@@ -1863,6 +1876,9 @@ function start-kube-controller-manager {
sed -i -e "s@{{additional_cloud_config_volume}}@@g" "${src_file}"
sed -i -e "s@{{pv_recycler_mount}}@${PV_RECYCLER_MOUNT}@g" "${src_file}"
sed -i -e "s@{{pv_recycler_volume}}@${PV_RECYCLER_VOLUME}@g" "${src_file}"
sed -i -e "s@{{flexvolume_hostpath_mount}}@${FLEXVOLUME_HOSTPATH_MOUNT}@g" "${src_file}"
sed -i -e "s@{{flexvolume_hostpath}}@${FLEXVOLUME_HOSTPATH_VOLUME}@g" "${src_file}"
cp "${src_file}" /etc/kubernetes/manifests
}
@@ -1886,6 +1902,11 @@ function start-kube-scheduler {
if [[ -n "${SCHEDULING_ALGORITHM_PROVIDER:-}" ]]; then
params+=" --algorithm-provider=${SCHEDULING_ALGORITHM_PROVIDER}"
fi
if [[ -n "${SCHEDULER_POLICY_CONFIG:-}" ]]; then
create-kubescheduler-policy-config
params+=" --use-legacy-policy-config"
params+=" --policy-config-file=/etc/srv/kubernetes/kube-scheduler/policy-config"
fi
local -r kube_scheduler_docker_tag=$(cat "${KUBE_HOME}/kube-docker-files/kube-scheduler.docker_tag")
# Remove salt comments and replace variables with values.

28
cluster/gce/gci/configure.sh
View File

@@ -31,6 +31,12 @@ DEFAULT_NPD_SHA1="a57a3fe64cab8a18ec654f5cef0aec59dae62568"
DEFAULT_MOUNTER_TAR_SHA="8003b798cf33c7f91320cd6ee5cec4fa22244571"
###
# Use --retry-connrefused opt only if it's supported by curl.
CURL_RETRY_CONNREFUSED=""
if curl --help | grep -q -- '--retry-connrefused'; then
CURL_RETRY_CONNREFUSED='--retry-connrefused'
fi
function set-broken-motd {
cat > /etc/motd <<EOF
Broken (or in progress) Kubernetes node setup! Check the cluster initialization status
@@ -48,8 +54,9 @@ EOF
function download-kube-env {
# Fetch kube-env from GCE metadata server.
(umask 700;
local -r tmp_kube_env="/tmp/kube-env.yaml"
curl --fail --retry 5 --retry-delay 3 --silent --show-error \
curl --fail --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --silent --show-error \
-H "X-Google-Metadata-Request: True" \
-o "${tmp_kube_env}" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-env
@@ -60,12 +67,14 @@ for k,v in yaml.load(sys.stdin).iteritems():
print("readonly {var}={value}".format(var = k, value = pipes.quote(str(v))))
''' < "${tmp_kube_env}" > "${KUBE_HOME}/kube-env")
rm -f "${tmp_kube_env}"
)
}
function download-kube-master-certs {
# Fetch kube-env from GCE metadata server.
(umask 700;
local -r tmp_kube_master_certs="/tmp/kube-master-certs.yaml"
curl --fail --retry 5 --retry-delay 3 --silent --show-error \
curl --fail --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --silent --show-error \
-H "X-Google-Metadata-Request: True" \
-o "${tmp_kube_master_certs}" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-master-certs
@@ -76,6 +85,7 @@ for k,v in yaml.load(sys.stdin).iteritems():
print("readonly {var}={value}".format(var = k, value = pipes.quote(str(v))))
''' < "${tmp_kube_master_certs}" > "${KUBE_HOME}/kube-master-certs")
rm -f "${tmp_kube_master_certs}"
)
}
function validate-hash {
@@ -102,7 +112,7 @@ function download-or-bust {
for url in "${urls[@]}"; do
local file="${url##*/}"
rm -f "${file}"
if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --max-time 300 --retry 6 --retry-delay 10 "${url}"; then
if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --max-time 300 --retry 6 --retry-delay 10 ${CURL_RETRY_CONNREFUSED} "${url}"; then
echo "== Failed to download ${url}. Retrying. =="
elif [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
echo "== Hash validation of ${url} failed. Retrying. =="
@@ -128,6 +138,13 @@ function split-commas {
echo $1 | tr "," "\n"
}
function remount-flexvolume-directory {
local -r flexvolume_plugin_dir=$1
mkdir -p $flexvolume_plugin_dir
mount --bind $flexvolume_plugin_dir $flexvolume_plugin_dir
mount -o remount,exec $flexvolume_plugin_dir
}
function install-gci-mounter-tools {
CONTAINERIZED_MOUNTER_HOME="${KUBE_HOME}/containerized_mounter"
local -r mounter_tar_sha="${DEFAULT_MOUNTER_TAR_SHA}"
@@ -326,6 +343,11 @@ function install-kube-binary-config {
# Install gci mounter related artifacts to allow mounting storage volumes in GCI
install-gci-mounter-tools
# Remount the Flexvolume directory with the "exec" option, if needed.
if [[ "${REMOUNT_VOLUME_PLUGIN_DIR:-}" == "true" && -n "${VOLUME_PLUGIN_DIR:-}" ]]; then
remount-flexvolume-directory "${VOLUME_PLUGIN_DIR}"
fi
# Clean up.
rm -rf "${KUBE_HOME}/kubernetes"
rm -f "${KUBE_HOME}/${server_binary_tar}"

3
cluster/gce/gci/master.yaml
View File

@@ -15,7 +15,8 @@ write_files:
ExecStartPre=/bin/mkdir -p /home/kubernetes/bin
ExecStartPre=/bin/mount --bind /home/kubernetes/bin /home/kubernetes/bin
ExecStartPre=/bin/mount -o remount,exec /home/kubernetes/bin
ExecStartPre=/usr/bin/curl --fail --retry 5 --retry-delay 3 --silent --show-error -H "X-Google-Metadata-Request: True" -o /home/kubernetes/bin/configure.sh http://metadata.google.internal/computeMetadata/v1/instance/attributes/configure-sh
# Use --retry-connrefused opt only if it's supported by curl.
ExecStartPre=/bin/bash -c 'OPT=""; if curl --help | grep -q -- "--retry-connrefused"; then OPT="--retry-connrefused"; fi; /usr/bin/curl --fail --retry 5 --retry-delay 3 $OPT --silent --show-error -H "X-Google-Metadata-Request: True" -o /home/kubernetes/bin/configure.sh http://metadata.google.internal/computeMetadata/v1/instance/attributes/configure-sh'
ExecStartPre=/bin/chmod 544 /home/kubernetes/bin/configure.sh
ExecStart=/home/kubernetes/bin/configure.sh

2
cluster/gce/gci/mounter/BUILD
View File

@@ -8,8 +8,8 @@ load(
go_binary(
name = "mounter",
embed = [":go_default_library"],
importpath = "k8s.io/kubernetes/cluster/gce/gci/mounter",
library = ":go_default_library",
)
go_library(

3
cluster/gce/gci/node.yaml
View File

@@ -15,7 +15,8 @@ write_files:
ExecStartPre=/bin/mkdir -p /home/kubernetes/bin
ExecStartPre=/bin/mount --bind /home/kubernetes/bin /home/kubernetes/bin
ExecStartPre=/bin/mount -o remount,exec /home/kubernetes/bin
ExecStartPre=/usr/bin/curl --fail --retry 5 --retry-delay 3 --silent --show-error -H "X-Google-Metadata-Request: True" -o /home/kubernetes/bin/configure.sh http://metadata.google.internal/computeMetadata/v1/instance/attributes/configure-sh
# Use --retry-connrefused opt only if it's supported by curl.
ExecStartPre=/bin/bash -c 'OPT=""; if curl --help | grep -q -- "--retry-connrefused"; then OPT="--retry-connrefused"; fi; /usr/bin/curl --fail --retry 5 --retry-delay 3 $OPT --silent --show-error -H "X-Google-Metadata-Request: True" -o /home/kubernetes/bin/configure.sh http://metadata.google.internal/computeMetadata/v1/instance/attributes/configure-sh'
ExecStartPre=/bin/chmod 544 /home/kubernetes/bin/configure.sh
ExecStart=/home/kubernetes/bin/configure.sh

33
cluster/gce/upgrade.sh
View File

@@ -537,6 +537,39 @@ if [[ -z "${STORAGE_MEDIA_TYPE:-}" ]] && [[ "${STORAGE_BACKEND:-}" != "etcd2" ]]
fi
fi
# Prompt if etcd image/version is unspecified when doing master upgrade.
# In e2e tests, we use TEST_ALLOW_IMPLICIT_ETCD_UPGRADE=true to skip this
# prompt, simulating the behavior when the user confirms interactively.
# All other automated use of this script should explicitly specify a version.
if [[ "${master_upgrade}" == "true" ]]; then
if [[ -z "${ETCD_IMAGE:-}" && -z "${TEST_ETCD_IMAGE:-}" ]] || [[ -z "${ETCD_VERSION:-}" && -z "${TEST_ETCD_VERSION:-}" ]]; then
echo
echo "***WARNING***"
echo "Upgrading Kubernetes with this script might result in an upgrade to a new etcd version."
echo "Some etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path."
echo "To pin the etcd version to your current one (e.g. v3.0.17), set the following variables"
echo "before running this script:"
echo
echo "# example: pin to etcd v3.0.17"
echo "export ETCD_IMAGE=3.0.17"
echo "export ETCD_VERSION=3.0.17"
echo
echo "Alternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,"
echo "you might still be able to downgrade Kubernetes by pinning to the newer etcd version."
echo "In all cases, it is strongly recommended to have an etcd backup before upgrading."
echo
if [ -t 0 ] && [ -t 1 ]; then
read -p "Continue with default etcd version, which might upgrade etcd? [y/N] " confirm
if [[ "${confirm}" != "y" ]]; then
exit 1
fi
elif [[ "${TEST_ALLOW_IMPLICIT_ETCD_UPGRADE:-}" != "true" ]]; then
echo "ETCD_IMAGE and ETCD_VERSION must be specified when run non-interactively." >&2
exit 1
fi
fi
fi
print-node-version-info "Pre-Upgrade"
if [[ "${local_binaries}" == "false" ]]; then

83
cluster/gce/util.sh
View File

@@ -25,14 +25,14 @@ source "${KUBE_ROOT}/cluster/gce/${KUBE_CONFIG_FILE-"config-default.sh"}"
source "${KUBE_ROOT}/cluster/common.sh"
source "${KUBE_ROOT}/hack/lib/util.sh"
if [[ "${NODE_OS_DISTRIBUTION}" == "debian" || "${NODE_OS_DISTRIBUTION}" == "container-linux" || "${NODE_OS_DISTRIBUTION}" == "trusty" || "${NODE_OS_DISTRIBUTION}" == "gci" || "${NODE_OS_DISTRIBUTION}" == "ubuntu" ]]; then
if [[ "${NODE_OS_DISTRIBUTION}" == "gci" || "${NODE_OS_DISTRIBUTION}" == "ubuntu" ]]; then
source "${KUBE_ROOT}/cluster/gce/${NODE_OS_DISTRIBUTION}/node-helper.sh"
else
echo "Cannot operate on cluster using node os distro: ${NODE_OS_DISTRIBUTION}" >&2
exit 1
fi
if [[ "${MASTER_OS_DISTRIBUTION}" == "container-linux" || "${MASTER_OS_DISTRIBUTION}" == "trusty" || "${MASTER_OS_DISTRIBUTION}" == "gci" || "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]]; then
if [[ "${MASTER_OS_DISTRIBUTION}" == "trusty" || "${MASTER_OS_DISTRIBUTION}" == "gci" || "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]]; then
source "${KUBE_ROOT}/cluster/gce/${MASTER_OS_DISTRIBUTION}/master-helper.sh"
else
echo "Cannot operate on cluster using master os distro: ${MASTER_OS_DISTRIBUTION}" >&2
@@ -243,12 +243,12 @@ function set-preferred-region() {
else
KUBE_ADDON_REGISTRY="gcr.io/google_containers"
fi
if [[ "${ENABLE_DOCKER_REGISTRY_CACHE:-}" == "true" ]]; then
DOCKER_REGISTRY_MIRROR_URL="https://${preferred}-mirror.gcr.io"
fi
}
if [[ "${ENABLE_DOCKER_REGISTRY_CACHE:-}" == "true" ]]; then
DOCKER_REGISTRY_MIRROR_URL="https://mirror.gcr.io"
fi
# Take the local tar files and upload them to Google Storage. They will then be
# downloaded by the master as part of the start up script for the master.
#
@@ -1532,7 +1532,7 @@ function check-cluster() {
-H "Authorization: Bearer ${KUBE_BEARER_TOKEN}" \
${secure} \
--max-time 5 --fail \
"https://${KUBE_MASTER_IP}/api/v1/pods" > "${curl_out}" 2>&1; do
"https://${KUBE_MASTER_IP}/api/v1/pods?limit=100" > "${curl_out}" 2>&1; do
local elapsed=$(($(date +%s) - ${start_time}))
if [[ ${elapsed} -gt ${KUBE_CLUSTER_INITIALIZATION_TIMEOUT} ]]; then
echo -e "${color_red}Cluster failed to initialize within ${KUBE_CLUSTER_INITIALIZATION_TIMEOUT} seconds.${color_norm}" >&2
@@ -2065,66 +2065,6 @@ function prepare-push() {
fi
}
# Push binaries to kubernetes master
function push-master() {
echo "Updating master metadata ..."
write-master-env
prepare-startup-script
add-instance-metadata-from-file "${KUBE_MASTER}" "kube-env=${KUBE_TEMP}/master-kube-env.yaml" "startup-script=${KUBE_TEMP}/configure-vm.sh"
echo "Pushing to master (log at ${OUTPUT}/push-${KUBE_MASTER}.log) ..."
cat ${KUBE_TEMP}/configure-vm.sh | gcloud compute ssh --ssh-flag="-o LogLevel=quiet" --project "${PROJECT}" --zone "${ZONE}" "${KUBE_MASTER}" --command "sudo bash -s -- --push" &> ${OUTPUT}/push-"${KUBE_MASTER}".log
}
# Push binaries to kubernetes node
function push-node() {
node=${1}
echo "Updating node ${node} metadata... "
prepare-startup-script
add-instance-metadata-from-file "${node}" "kube-env=${KUBE_TEMP}/node-kube-env.yaml" "startup-script=${KUBE_TEMP}/configure-vm.sh"
echo "Start upgrading node ${node} (log at ${OUTPUT}/push-${node}.log) ..."
cat ${KUBE_TEMP}/configure-vm.sh | gcloud compute ssh --ssh-flag="-o LogLevel=quiet" --project "${PROJECT}" --zone "${ZONE}" "${node}" --command "sudo bash -s -- --push" &> ${OUTPUT}/push-"${node}".log
}
# Push binaries to kubernetes cluster
function kube-push() {
# Disable this until it's fixed.
# See https://github.com/kubernetes/kubernetes/issues/17397
echo "./cluster/kube-push.sh is currently not supported in GCE."
echo "Please use ./cluster/gce/upgrade.sh."
exit 1
prepare-push true
push-master
for (( i=0; i<${#NODE_NAMES[@]}; i++)); do
push-node "${NODE_NAMES[$i]}" &
done
kube::util::wait-for-jobs || {
echo -e "${color_red}Some commands failed.${color_norm}" >&2
}
# TODO(zmerlynn): Re-create instance-template with the new
# node-kube-env. This isn't important until the node-ip-range issue
# is solved (because that's blocking automatic dynamic nodes from
# working). The node-kube-env has to be composed with the KUBELET_TOKEN
# and KUBE_PROXY_TOKEN. Ideally we would have
# http://issue.k8s.io/3168
# implemented before then, though, so avoiding this mess until then.
echo
echo "Kubernetes cluster is running. The master is running at:"
echo
echo " https://${KUBE_MASTER_IP}"
echo
echo "The user name and password to use is located in ~/.kube/config"
echo
}
# -----------------------------------------------------------------------------
# Cluster specific test helpers used from hack/e2e.go
@@ -2233,12 +2173,3 @@ function ssh-to-node() {
function prepare-e2e() {
detect-project
}
# Writes configure-vm.sh to a temporary location with comments stripped. GCE
# limits the size of metadata fields to 32K, and stripping comments is the
# easiest way to buy us a little more room.
function prepare-startup-script() {
# Find a standard sed instance (and ensure that the command works as expected on a Mac).
kube::util::ensure-gnu-sed
${SED} '/^\s*#\([^!].*\)*$/ d' ${KUBE_ROOT}/cluster/gce/configure-vm.sh > ${KUBE_TEMP}/configure-vm.sh
}

14
cluster/get-kube.sh
View File

@@ -24,20 +24,6 @@
# Set KUBERNETES_PROVIDER to choose between different providers:
# Google Compute Engine [default]
# * export KUBERNETES_PROVIDER=gce; wget -q -O - https://get.k8s.io | bash
# Google Container Engine
# * export KUBERNETES_PROVIDER=gke; wget -q -O - https://get.k8s.io | bash
# Amazon EC2
# * export KUBERNETES_PROVIDER=aws; wget -q -O - https://get.k8s.io | bash
# Libvirt (with CoreOS as a guest operating system)
# * export KUBERNETES_PROVIDER=libvirt-coreos; wget -q -O - https://get.k8s.io | bash
# Microsoft Azure
# * export KUBERNETES_PROVIDER=azure-legacy; wget -q -O - https://get.k8s.io | bash
# Vagrant (local virtual machines)
# * export KUBERNETES_PROVIDER=vagrant; wget -q -O - https://get.k8s.io | bash
# VMWare Photon Controller
# * export KUBERNETES_PROVIDER=photon-controller; wget -q -O - https://get.k8s.io | bash
# OpenStack-Heat
# * export KUBERNETES_PROVIDER=openstack-heat; wget -q -O - https://get.k8s.io | bash
#
# Set KUBERNETES_RELEASE to choose a specific release instead of the current
# stable release, (e.g. 'v1.3.7').

4
cluster/images/etcd-version-monitor/BUILD
View File

@@ -8,8 +8,8 @@ load(
go_binary(
name = "etcd-version-monitor",
embed = [":go_default_library"],
importpath = "k8s.io/kubernetes/cluster/images/etcd-version-monitor",
library = ":go_default_library",
)
go_library(
@@ -19,6 +19,8 @@ go_library(
deps = [
"//vendor/github.com/golang/glog:go_default_library",
"//vendor/github.com/prometheus/client_golang/prometheus:go_default_library",
"//vendor/github.com/prometheus/client_golang/prometheus/promhttp:go_default_library",
"//vendor/github.com/prometheus/client_model/go:go_default_library",
"//vendor/github.com/prometheus/common/expfmt:go_default_library",
"//vendor/github.com/spf13/pflag:go_default_library",
],

2
cluster/images/etcd-version-monitor/Makefile
View File

@@ -20,7 +20,7 @@
ARCH:=amd64
GOLANG_VERSION?=1.8.3
REGISTRY?=gcr.io/google-containers
TAG?=0.1.0
TAG?=0.1.1
IMAGE:=$(REGISTRY)/etcd-version-monitor:$(TAG)
CURRENT_DIR:=$(pwd)
TEMP_DIR:=$(shell mktemp -d)

14
cluster/images/etcd-version-monitor/README.md
View File

@@ -1,11 +1,19 @@
# etcd-version-monitor
This is a tool for exporting metrics related to etcd version, like etcd
server's binary version, cluster version, and counts of different kinds of
gRPC calls (which is a characteristic of v3), etc. These metrics are in
This is a tool for exporting etcd metrics and supplementing them with etcd
server binary version and cluster version. These metrics are in
prometheus format and can be scraped by a prometheus server.
The metrics are exposed at the http://localhost:9101/metrics endpoint.
For etcd 3.1+, the
[go-grpc-prometheus](https://github.com/grpc-ecosystem/go-grpc-prometheus)
metrics format, which backward incompatibly replaces the 3.0 legacy grpc metric
format, is exposed in both the 3.1 format and in the 3.0. This preserves
backward compatiblity.
For etcd 3.1+, the `--metrics=extensive` must be set on etcd for grpc request
latency metrics (`etcd_grpc_unary_requests_duration_seconds`) to be exposed.
**RUNNING THE TOOL**
To run this tool as a docker container:

245
cluster/images/etcd-version-monitor/etcd-version-monitor.go
View File

@@ -25,6 +25,8 @@ import (
"github.com/golang/glog"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
"github.com/spf13/pflag"
)
@@ -52,6 +54,11 @@ const (
// Initialize prometheus metrics to be exported.
var (
// Register all custom metrics with a dedicated registry to keep them separate.
customMetricRegistry = prometheus.NewRegistry()
// Custom etcd version metric since etcd 3.2- does not export one.
// This will be replaced by https://github.com/coreos/etcd/pull/8960 in etcd 3.3.
etcdVersion = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: namespace,
@@ -59,15 +66,122 @@ var (
Help: "Etcd server's binary version",
},
[]string{"binary_version"})
etcdGRPCRequestsTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Namespace: namespace,
Name: "grpc_requests_total",
Help: "Counter of received grpc requests, labeled by the grpc method and service names",
gatherer = &monitorGatherer{
// Rewrite rules for etcd metrics that are exported by default.
exported: map[string]*exportedMetric{
// etcd 3.0 metric format for total grpc requests with renamed method and service labels.
"etcd_grpc_requests_total": {
rewriters: []rewriteFunc{
func(mf *dto.MetricFamily) (*dto.MetricFamily, error) {
mf = deepCopyMetricFamily(mf)
renameLabels(mf, map[string]string{
"grpc_method": "method",
"grpc_service": "service",
})
return mf, nil
},
},
},
// etcd 3.1+ metric format for total grpc requests.
"grpc_server_handled_total": {
rewriters: []rewriteFunc{
// Export the metric exactly as-is. For 3.1+ metrics, we will
// pass all metrics directly through.
identity,
// Write to the etcd 3.0 metric format for backward compatibility.
func(mf *dto.MetricFamily) (*dto.MetricFamily, error) {
mf = deepCopyMetricFamily(mf)
renameMetric(mf, "etcd_grpc_requests_total")
renameLabels(mf, map[string]string{
"grpc_method": "method",
"grpc_service": "service",
})
return mf, nil
},
},
},
// etcd 3.0 metric format for grpc request latencies,
// rewritten to the etcd 3.1+ format.
"etcd_grpc_unary_requests_duration_seconds": {
rewriters: []rewriteFunc{
func(mf *dto.MetricFamily) (*dto.MetricFamily, error) {
mf = deepCopyMetricFamily(mf)
renameMetric(mf, "grpc_server_handling_seconds")
tpeName := "grpc_type"
tpeVal := "unary"
for _, m := range mf.Metric {
m.Label = append(m.Label, &dto.LabelPair{Name: &tpeName, Value: &tpeVal})
}
return mf, nil
},
},
},
// etcd 3.1+ metric format for total grpc requests.
"grpc_server_handling_seconds": {},
},
[]string{"method", "service"})
}
)
// monitorGatherer is a custom metric gatherer for prometheus that exports custom metrics
// defined by this monitor as well as rewritten etcd metrics.
type monitorGatherer struct {
exported map[string]*exportedMetric
}
// exportedMetric identifies a metric that is exported and defines how it is rewritten before
// it is exported.
type exportedMetric struct {
rewriters []rewriteFunc
}
// rewriteFunc rewrites metrics before they are exported.
type rewriteFunc func(mf *dto.MetricFamily) (*dto.MetricFamily, error)
func (m *monitorGatherer) Gather() ([]*dto.MetricFamily, error) {
etcdMetrics, err := scrapeMetrics()
if err != nil {
return nil, err
}
exported, err := m.rewriteExportedMetrics(etcdMetrics)
if err != nil {
return nil, err
}
custom, err := customMetricRegistry.Gather()
if err != nil {
return nil, err
}
result := make([]*dto.MetricFamily, 0, len(exported)+len(custom))
result = append(result, exported...)
result = append(result, custom...)
return result, nil
}
func (m *monitorGatherer) rewriteExportedMetrics(metrics map[string]*dto.MetricFamily) ([]*dto.MetricFamily, error) {
results := make([]*dto.MetricFamily, 0, len(metrics))
for n, mf := range metrics {
if e, ok := m.exported[n]; ok {
// Apply rewrite rules for metrics that have them.
if e.rewriters == nil {
results = append(results, mf)
} else {
for _, rewriter := range e.rewriters {
new, err := rewriter(mf)
if err != nil {
return nil, err
}
results = append(results, new)
}
}
} else {
// Proxy all metrics without any rewrite rules directly.
results = append(results, mf)
}
}
return results, nil
}
// Struct for unmarshalling the json response from etcd's /version endpoint.
type EtcdVersion struct {
BinaryVersion string `json:"etcdserver"`
@@ -132,83 +246,78 @@ func getVersionPeriodically(stopCh <-chan struct{}) {
}
}
// Struct for storing labels for gRPC request types.
type GRPCRequestLabels struct {
Method string
Service string
}
// Function for fetching etcd grpc request counts and feeding it to the prometheus metric.
func getGRPCRequestCount(lastRecordedCount *map[GRPCRequestLabels]float64) error {
// Create the get request for the etcd metrics endpoint.
// scrapeMetrics scrapes the prometheus metrics from the etcd metrics URI.
func scrapeMetrics() (map[string]*dto.MetricFamily, error) {
req, err := http.NewRequest("GET", etcdMetricsScrapeURI, nil)
if err != nil {
return fmt.Errorf("Failed to create GET request for etcd metrics: %v", err)
return nil, fmt.Errorf("Failed to create GET request for etcd metrics: %v", err)
}
// Send the get request and receive a response.
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("Failed to receive GET response for etcd metrics: %v", err)
return nil, fmt.Errorf("Failed to receive GET response for etcd metrics: %v", err)
}
defer resp.Body.Close()
// Parse the metrics in text format to a MetricFamily struct.
var textParser expfmt.TextParser
metricFamilies, err := textParser.TextToMetricFamilies(resp.Body)
if err != nil {
return fmt.Errorf("Failed to parse etcd metrics: %v", err)
}
// Look through the grpc requests metric family and update our promotheus metric.
for _, metric := range metricFamilies["etcd_grpc_requests_total"].GetMetric() {
var grpcRequestLabels GRPCRequestLabels
for _, label := range metric.GetLabel() {
if label.GetName() == "grpc_method" {
grpcRequestLabels.Method = label.GetValue()
}
if label.GetName() == "grpc_service" {
grpcRequestLabels.Service = label.GetValue()
}
}
if grpcRequestLabels.Method == "" || grpcRequestLabels.Service == "" {
return fmt.Errorf("Could not get value for grpc_method and/or grpc_service label")
}
// Get last recorded value and new value of the metric and update it suitably.
previousMetricValue := 0.0
if value, ok := (*lastRecordedCount)[grpcRequestLabels]; ok {
previousMetricValue = value
}
newMetricValue := metric.GetCounter().GetValue()
(*lastRecordedCount)[grpcRequestLabels] = newMetricValue
if newMetricValue >= previousMetricValue {
etcdGRPCRequestsTotal.With(prometheus.Labels{
"method": grpcRequestLabels.Method,
"service": grpcRequestLabels.Service,
}).Add(newMetricValue - previousMetricValue)
}
}
return nil
return textParser.TextToMetricFamilies(resp.Body)
}
// Function for periodically fetching etcd GRPC request counts.
func getGRPCRequestCountPeriodically(stopCh <-chan struct{}) {
// This map stores last recorded count for a given grpc request type.
lastRecordedCount := make(map[GRPCRequestLabels]float64)
for {
if err := getGRPCRequestCount(&lastRecordedCount); err != nil {
glog.Errorf("Failed to fetch etcd grpc request counts: %v", err)
}
select {
case <-stopCh:
break
case <-time.After(scrapeTimeout):
func renameMetric(mf *dto.MetricFamily, name string) {
mf.Name = &name
}
func renameLabels(mf *dto.MetricFamily, nameMapping map[string]string) {
for _, m := range mf.Metric {
for _, lbl := range m.Label {
if alias, ok := nameMapping[*lbl.Name]; ok {
lbl.Name = &alias
}
}
}
}
func identity(mf *dto.MetricFamily) (*dto.MetricFamily, error) {
return mf, nil
}
func deepCopyMetricFamily(mf *dto.MetricFamily) *dto.MetricFamily {
r := &dto.MetricFamily{}
r.Name = mf.Name
r.Help = mf.Help
r.Type = mf.Type
r.Metric = make([]*dto.Metric, len(mf.Metric))
for i, m := range mf.Metric {
r.Metric[i] = deepCopyMetric(m)
}
return r
}
func deepCopyMetric(m *dto.Metric) *dto.Metric {
r := &dto.Metric{}
r.Label = make([]*dto.LabelPair, len(m.Label))
for i, lp := range m.Label {
r.Label[i] = deepCopyLabelPair(lp)
}
r.Gauge = m.Gauge
r.Counter = m.Counter
r.Summary = m.Summary
r.Untyped = m.Untyped
r.Histogram = m.Histogram
r.TimestampMs = m.TimestampMs
return r
}
func deepCopyLabelPair(lp *dto.LabelPair) *dto.LabelPair {
r := &dto.LabelPair{}
r.Name = lp.Name
r.Value = lp.Value
return r
}
func main() {
// Register the commandline flags passed to the tool.
registerFlags(pflag.CommandLine)
@@ -216,18 +325,16 @@ func main() {
pflag.Parse()
// Register the metrics we defined above with prometheus.
prometheus.MustRegister(etcdVersion)
prometheus.MustRegister(etcdGRPCRequestsTotal)
prometheus.Unregister(prometheus.NewGoCollector())
customMetricRegistry.MustRegister(etcdVersion)
customMetricRegistry.Unregister(prometheus.NewGoCollector())
// Spawn threads for periodically scraping etcd version metrics.
stopCh := make(chan struct{})
defer close(stopCh)
go getVersionPeriodically(stopCh)
go getGRPCRequestCountPeriodically(stopCh)
// Serve our metrics on listenAddress/metricsPath.
glog.Infof("Listening on: %v", listenAddress)
http.Handle(metricsPath, prometheus.UninstrumentedHandler())
http.Handle(metricsPath, promhttp.HandlerFor(gatherer, promhttp.HandlerOpts{}))
glog.Errorf("Stopped listening/serving metrics: %v", http.ListenAndServe(listenAddress, nil))
}

8
cluster/images/etcd/Makefile
View File

@@ -15,7 +15,7 @@
# Build the etcd image
#
# Usage:
# [TAGS=2.2.1 2.3.7 3.0.17 3.1.10] [REGISTRY=gcr.io/google_containers] [ARCH=amd64] [BASEIMAGE=busybox] make (build|push)
# [TAGS=2.2.1 2.3.7 3.0.17 3.1.11] [REGISTRY=gcr.io/google_containers] [ARCH=amd64] [BASEIMAGE=busybox] make (build|push)
# The image contains different etcd versions to simplify
# upgrades. Thus be careful when removing any tag from here.
@@ -26,8 +26,8 @@
# Except from etcd-$(tag) and etcdctl-$(tag) binaries, we also
# need etcd and etcdctl binaries for backward compatibility reasons.
# That binary will be set to the last tag from $(TAGS).
TAGS?=2.2.1 2.3.7 3.0.17 3.1.10
REGISTRY_TAG?=3.1.10
TAGS?=2.2.1 2.3.7 3.0.17 3.1.11
REGISTRY_TAG?=3.1.11
ARCH?=amd64
REGISTRY?=gcr.io/google_containers
GOLANG_VERSION?=1.7.6
@@ -53,7 +53,7 @@ endif
build:
# Copy the content in this dir to the temp dir,
# without copying the subdirectories.
find ./ -maxdepth 1 -type f | xargs cp -t $(TEMP_DIR)
find ./ -maxdepth 1 -type f | xargs -I {} cp {} $(TEMP_DIR)
# Compile attachlease
docker run -i -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes -v $(TEMP_DIR):/build -e GOARCH=$(ARCH) golang:$(GOLANG_VERSION) \

2
cluster/images/etcd/attachlease/BUILD
View File

@@ -8,8 +8,8 @@ load(
go_binary(
name = "attachlease",
embed = [":go_default_library"],
importpath = "k8s.io/kubernetes/cluster/images/etcd/attachlease",
library = ":go_default_library",
)
go_library(

2
cluster/images/etcd/rollback/BUILD
View File

@@ -8,8 +8,8 @@ load(
go_binary(
name = "rollback",
embed = [":go_default_library"],
importpath = "k8s.io/kubernetes/cluster/images/etcd/rollback",
library = ":go_default_library",
)
go_library(

3
cluster/images/hyperkube/Makefile
View File

@@ -19,7 +19,8 @@
REGISTRY?=gcr.io/google-containers
ARCH?=amd64
HYPERKUBE_BIN?=_output/dockerized/bin/linux/$(ARCH)/hyperkube
OUT_DIR?=_output
HYPERKUBE_BIN?=$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/hyperkube
BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.8
TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX)

4
cluster/juju/layers/kubeapi-load-balancer/config.yaml
View File

@@ -9,3 +9,7 @@ options:
description: |
Space-separated list of extra SAN entries to add to the x509 certificate
created for the load balancers.
proxy_read_timeout:
type: int
default: 90
description: Timeout in seconds for reading a response from proxy server.

Some files were not shown because too many files have changed in this diff Show More