This change allows us to substitute values into our rules files.
Example:
- alert: my_region_is_down
expr: up{region="{{ $my_region }}"} == 0
To support this change, rule annotations that used the expansion
{{ $labels.foo }} had to be surrounded with "{{` ... `}}" to render
correctly.
Change-Id: Ia7ac891de8261acca62105a3e2636bd747a5fbea
Some scrape targets require the use of TLS client certificates, which
are specified as filenames as part of the tls_config.
This change allows these client certs and keys to be provided, stores
them in a secret, and mounts them in the pod under /tls_configs.
Example:
tls_configs:
kubernetes-etcd:
ca.pem: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
crt.pem: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
key.pem: |
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
conf:
prometheus:
scrape_configs:
template: |
scrape_configs:
- job_name: kubernetes-etcd
scheme: https
tls_config:
ca_file: /tls_configs/kubernetes-etcd.ca.pem
cert_file: /tls_configs/kubernetes-etcd.cert.pem
key_file: /tls_configs/kubernetes-etcd.key.pem
Change-Id: I963c65dc39f1b5110b091296b93e2de9cdd980a4
- Update alertmanger and prometheus discovery port from 6783 to 9094
- Update to support fqdn for discovery hostname
- Add one test alert to Prometheus to test alert pipeline
- update container name from alertmanger to prometheus-alertmanager
Change-Id: Iec5e758e4b576dff01e84591a2440d030d5ff3c4
This updates the chart to include the pod security context
on the pod template.
This also adds the container security context to set
readOnlyRootFilesystem flag to true
Change-Id: Icb7a9de4d98bac1f0bcf6181b6e88695f4b09709
Unrestrict octal values rule since benefits of file modes readability
exceed possible issues with yaml 1.2 adoption in future k8s versions.
These issues will be addressed when/if they occur.
Also ensure osh-infra is a required project for lint job, that matters
when running job against another project.
Change-Id: Ic5e327cf40c4b09c90738baff56419a6cef132da
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
This commit rewrites lint job to make template linting available.
Currently yamllint is run in warning mode against all templates
rendered with default values. Duplicates detected and issues will be
addressed in subsequent commits.
Also all y*ml files are added for linting and corresponding code changes
are made. For non-templates warning rules are disabled to improve
readability. Chart and requirements yamls are also modified in the name
of consistency.
Change-Id: Ife6727c5721a00c65902340d95b7edb0a9c77365
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.
This change removes all references to this copyright by the
non-existent group and any blank lines underneath.
Change-Id: I1882738cf9757c5350a8533876fd37b5920b5235
This change converts alert expressions which relied on instant vectors
to use range aggregate functions instead - For just the 'basic_linux'
rules.
Change-Id: I30d6ab71d747b297f522bbeb12b8f4dbfce1eefe
Co-Authored-By: Meghan Heisler <mkheisler93@gmail.com>
This change updates the prometheus alerting rules to use ranged vectors
in their expressions, to avoid situations wher missed scrapes would
cause scalar metrics to "go stale" - resetting the alert timer.
Only the ceph alerts are affected by this change.
Change-Id: Ib47866d12616aaa808e6a09c58aa4352e338a152
Co-Authored-By: Meghan Heisler <mkheisler93@gmail.com>
This change converts alert expressions which relied on instant vectors
to use range aggregate functions instead.
Change-Id: I4df757f961524bed23b6a6ad361779c1749ca2c5
Co-Authored-By: Meghan Heisler <mkheisler93@gmail.com>
This change adds a means of introducing new storage classes
and local persistent volumes.
Change-Id: I340c75f3d0a1678f3149f3cf62e4ab104823cc49
Co-Authored-By: Steven Fitzpatrick <steven.fitzpatrick@att.com>
This patch set updates and tests the apiVersion for rbac.authorization.k8s.io
from v1beta1 to v1 in preparation for its removal in k8s 1.20.
Change-Id: I4e68db1f75ff72eee55ecec93bd59c68c179c627
Signed-off-by: Tin Lam <tin@irrational.io>
This updates the deployment scripts for Prometheus to leverage the
feature gate functionality rather than bash generation of the list
of override files to use for alerting rules
Change-Id: Ie497ae930f7cc4db690a4ddc812a92e4491cde93
Signed-off-by: Steve Wilkerson <sw5822@att.com>
I noticed a some nagios service checks were checking prometheus
alerts which did not exist in our default prometheus configuration.
In one case a prometheus alert did not match the naming convention
of similar alerts.
One nagios service check, ceph_monitor_clock_skew_high, does not
have a corresponding alert at all, so I've changed it to check the
node_ntmp_clock_skew_high
alert, where a node has the label ceph-mon="enabled".
Change-Id: I2ebf9a4954190b8e2caefc8a61270e28bf24d9fa
This updates the Prometheus chart to support federation. This
moves to defining the Prometheus configuration file via a template
in the values.yaml file instead of through raw yaml. This allows
for overriding the chart's default configuration wholesale, as
this would be required for a hierarchical federated setup. This
also strips out all of the default rules defined in the chart for
the same reason. There are example rules defined for the various
aspects of OSH's infrastructure in the prometheus/values_overrides
directory that are executed as part of the normal CI jobs. This
also adds a nonvoting federated-monitoring job that vets out the
ability to federate prometheus in a hierarchical fashion with
extremely basic overrides
Change-Id: I0f121ad5e4f80be4c790dc869955c6b299ca9f26
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the podManagementPolicy to 'Parallel' for Prometheus
and Alertmanager, as there's no need to handle deploying these
two services in a sequential manner
Change-Id: I2f33b9651bed20c4cb2e0c477ae2227cbf9310cf
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the Prometheus version deployed by default from
2.3.2 to 2.12.0
Change-Id: Ic10e02a6b136a7f65fb686f5ef1adf1bcf6a9a9d
Signed-off-by: Steve Wilkerson <sw5822@att.com>
It was observed in some charts' values.yaml that the values defining
lifecycle upgrade parameters were incorrectly placed.
This change aims to correct these instances by adding a deployment-
type subkey corresponding with the deployment types identified in
the chart's templates dir, and indenting the values appropriately.
Change-Id: Id5437b1eeaf6e71472520f1fee91028c9b6bfdd3
This updates the kubernetes-entrypoint image reference to consume
the publicly available kubernetes-entrypoint image that is built
and maintained under the airshipit namespace, as the stackanetes
image is no longer actively maintained
Change-Id: I5bfdc156ae228ab16da57569ac6b05a9a125cb6a
Signed-off-by: Steve Wilkerson <sw5822@att.com>
Added the reblabeling config lines to the kubernetes_sd_config key, to
replace the node_name with hostname for Node-exporter. This must now
display the hostname also as one of the labels of the Node-exporter
metrics.
Change-Id: Ic96a890552a1cd2f5e595c37330de048f31a0e75
There were some false alerts about volume_claim_capacity_high_utilization
due to wrong formula used to determine the percentage of used capacity.
Change-Id: I24afed7946f915e5e13f0ba759eca252c2598af9
We now have a process for OSH-images image building,
using Zuul, so we should point the images by default to those
images, instead of pointing to stale images.
Without this, the osh-images build process is completely not
in use (and completely opaque to deployers), and updating the
osh-images process or patching its code has no impact on OSH.
This should fix it.
Change-Id: Ic00bd98c151669dc2485cd88e0e8c2ab05445959
This ps exposes the anti-affinity weight value, including
default, that will be consumed by the updated htk function.
Change-Id: Id8eb303674764ef8b0664f62040723aaf77e0a54
This adds a basic egress policy to the charts run by the
network-policy check. A change was recently merged requiring
the eggress tag to be in the chart but did not add it, this
addresses that
Change-Id: I60669c9351db7854cba8c69723eb783a966d2a56
This PS adds emptydirs backing the /tmp directory in pods, which
is required in most cases for full operation when using a read only
filesystem backing the container.
Additionally some yaml indent issues are resolved.
Change-Id: I8b7f1614da059783254aa6efc09facf23fca3cad
Signed-off-by: Pete Birley <pete@port.direct>
