mirror of
https://github.com/outbackdingo/kubernetes.git
synced 2026-02-14 15:00:30 +00:00
feb4d2004a
Automatic merge from submit-queue Filter internal Kubernetes labels from Prometheus metrics **What this PR does / why we need it**: Kubernetes uses Docker labels as storage for some internal labels. The majority of these labels are not meaningful metric labels and a few of them are even harmful as they're not static and cause wrong aggregation results. This change provides a custom labels func to only attach meaningful labels to cAdvisor exported metrics. **Which issue this PR fixes** google/cadvisor#1312 **Special notes for your reviewer**: Depends on google/cadvisor#1429. Once that is merged, I'll update the vendor update commit. **Release note**: ```release-note Remove environment variables and internal Kubernetes Docker labels from cAdvisor Prometheus metric labels. Old behavior: - environment variables explicitly whitelisted via --docker-env-metadata-whitelist were exported as `container_env_*=*`. Default is zero so by default non were exported - all docker labels were exported as `container_label_*=*` New behavior: - Only `container_name`, `pod_name`, `namespace`, `id`, `image`, and `name` labels are exposed - no environment variables will be exposed ever via /metrics, even if whitelisted ``` --- Given that we have full control over the exported label set, I shortened the pod_name, pod_namespace and container_name label names. Below an example of the change (reformatted for readability). ``` # BEFORE container_cpu_cfs_periods_total{ container_label_io_kubernetes_container_hash="5af8c3b4", container_label_io_kubernetes_container_name="sync", container_label_io_kubernetes_container_restartCount="1", container_label_io_kubernetes_container_terminationMessagePath="/dev/termination-log", container_label_io_kubernetes_pod_name="popularsearches-web-3165456836-2bfey", container_label_io_kubernetes_pod_namespace="popularsearches", container_label_io_kubernetes_pod_terminationGracePeriod="30", container_label_io_kubernetes_pod_uid="6a291e48-47c4-11e6-84a4-c81f66bdf8bd", id="/docker/68e1f15353921f4d6d4d998fa7293306c4ac828d04d1284e410ddaa75cf8cf25", image="redacted.com/popularsearches:42-16-ba6bd88", name="k8s_sync.5af8c3b4_popularsearches-web-3165456836-2bfey_popularsearches_6a291e48-47c4-11e6-84a4-c81f66bdf8bd_c02d3775" } 72819 # AFTER container_cpu_cfs_periods_total{ container_name="sync", pod_name="popularsearches-web-3165456836-2bfey", namespace="popularsearches", id="/docker/68e1f15353921f4d6d4d998fa7293306c4ac828d04d1284e410ddaa75cf8cf25", image="redacted.com/popularsearches:42-16-ba6bd88", name="k8s_sync.5af8c3b4_popularsearches-web-3165456836-2bfey_popularsearches_6a291e48-47c4-11e6-84a4-c81f66bdf8bd_c02d3775" } 72819 ``` Feedback requested on: * Label names. Other suggestions? Should we keep these very long ones? * Do we need to export io.kubernetes.pod.uid? It makes working with the metrics a bit more complicated and the pod name is already unique at any time (but not over time). The UID is aslo part of `name`. As discussed with @timstclair, this should be added to v1.4 as the current labels are harmful. PTAL @jimmidyson @fabxc @vishh