By tying the fluent condfiguration to the release, it will be re-rendered
if the release is upgraded. This is useful in combination with [0], allowing
powerful configuration updates using helm upgrade. For example
Values:
.Values.pod.env.fluentd.vars.OUTPUT_ENABLED: true
fluent.conf:
...
{{- if .Values.pod.env.fluentd.vars.OUTPUT_ENABLED }}
<match **>
# Output Configuration here
</match>
{{- end }}
To disable this output section, issue a helm upgrade command and set the
apprpriate value to false.
helm upgrade fluentd ./fluentd --set pod.env.fluentd.vars.OUTPUT_ENABLED=false
[0] https://review.opendev.org/#/c/726880/
Change-Id: I3dce9e5c4eaf588569e8cc3e1ea3cf3bebd0c3c5
This change updates the fluentd configmap-etc to render
.Values.conf.fluentd.template as a template, allowing for greater
flexibility in configuration.
Change-Id: I8809767c679c377e319ecc53960c55ae18e1b558
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.
This change removes all references to this copyright by the
non-existent group and any blank lines underneath.
Change-Id: I1882738cf9757c5350a8533876fd37b5920b5235
This change updates the fluentd chart to use the native
fluent-plugin-prometheus for metric production. This plugin
provides more detailed metrics about fluentd's operations,
specifically regarding input and output statistics.
https://github.com/fluent/fluent-plugin-prometheus
Using the plugin, each fluentd pod produces metrics, so
scape annotations have been added to the pod spec. The
zuul check on metric producers has been updated to account
for this.
Depends-On: https://review.opendev.org/714167
Change-Id: I809356d92b0cff1e31cb2062102bbedefd4843fd
This patch set updates and tests the apiVersion for rbac.authorization.k8s.io
from v1beta1 to v1 in preparation for its removal in k8s 1.20.
Change-Id: I4e68db1f75ff72eee55ecec93bd59c68c179c627
Signed-off-by: Tin Lam <tin@irrational.io>
This change adds in missing network policy overrides for
fluent-daemonset and prometheus-exporter, as well as removes
existing mariadb network policies overrides that were causing
the network policy check job to fail.
Change-Id: Ib7a33f3d14617f9a9fda264f32cde7729a923193
This adds a helm-toolkit util for consuming arbitrary secret env
variables via pod env variables. It also updates the Fluentd chart
to add a release secret that is used to house the secret env
variables defined in the chart's values.yaml. This can be used as
an example to expand to other charts where this functionality is
desired
Change-Id: I9ef606840af92e54b2204e637c58442085e2c748
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This proposes adding a kafka chart to osh-infra that aligns
with the design patterns laid out by the other charts in osh-infra
and osh.
danielqsj's kafka-exporter image is leveraged to deploy a prometheus
exporter for kafka alongside the main application if enabled in
values.yaml
Change-Id: I5997b0994fc3aef9bd1b222c373cc3a013112566
Co-Authored-By: Meghan Heisler <mh783g@att.com>
This updates charts that consume images built from osh-images to
use tags other than the :latest tags. This will be followed up
with the definition of jobs to allow for vetting out of updated
images, as reliance on :latest tags assumes any change merged into
osh-images will result in functionally correct behavior (which has
shown to not be the case traditionally)
Change-Id: I181aa56ed187604dc7583d8081e53cc69eb27310
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the kubernetes-entrypoint image reference to consume
the publicly available kubernetes-entrypoint image that is built
and maintained under the airshipit namespace, as the stackanetes
image is no longer actively maintained
Change-Id: I5bfdc156ae228ab16da57569ac6b05a9a125cb6a
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the kubeadm and minikube Kubernetes deployments to
deploy version 1.16.2
Change-Id: I324f9665a24c9383c59376fb77cdb853facd0f18
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This adds support for configuring fluentd's update strategy when
deployed as a daemonset, as this was previously missed when
the changes to support both daemonsets and deployments were made
Change-Id: I5ac4fbfc0e64caaf207de42cd71c893f8d0f6ff1
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the Fluentd deployment template to use the helm
toolkit util for generating environment variables through the
chart's values.yaml. This adds flexibility in defining fluentd
outputs, as arbitrary environment variables can be injected and
consumed in fluentd's filters and outputs
Change-Id: I72a2c476378cc555bde1387781b4a06f13b51bc6
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the fluentd chart to add support for leveraging a
Kafka output. This required adding a kafka endpoint entry to the
chart's values.yaml, as well as the required template updates to
the fluentd deployment template and the addition of a secret for
kafka credentials
Depends-On: https://review.opendev.org/#/c/679297/
Change-Id: I80a487a0538f0b3704fb598da38c07feedaccb0e
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the Fluentd clusterrole to allow for getting
namespaces, as this is required for the fluentd kubernetes
plugin to function correctly
Change-Id: Id9d735310c53a922a62c6a82121edd332e7df724
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This fixes the whitespace chomps for adding extra volumes and
volume mounts via values.yaml for the Fluentd chart, as currently
too much whitespace is removed and the extra volumes and mounts
are not added correctly
Change-Id: I9cf67c3321339078ac795a7290f441b16cc41d41
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This adds a conditional check on the deployment type of the
Fluentd chart to determine whether to enable the current liveness
and readiness probes or not. The current probes are designed
around using fluentd as an aggregator and do not function properly
when fluentd is deployed as a daemonset. When run as a daemonset
and configured to tail files via the tail input plugin, fluentd
will prioritize reading the entirety of those files before
processing other input types, including opening the forward source
socket required for the current probes to function correctly. This
results in scenarios where the current probes will fail when in
fact fluentd is functioning correctly.
Daemonset focused probes to come as a follow on once a proper path
forward has been determined
Change-Id: I8a164bd47ce1950e0bd6c5043713f4cde9f85d79
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This removes the readOnly flag from the /var/log mount for the
fluentd pod to allow for using the file buffer mechanism when
desired
Change-Id: I23f0f03824eec5b142d3f2e8e42e8d07cddfe618
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the default fluentd configuration to include
recommended settings for preventing the elasticsearch plugin from
reloading the connection after 10000 requests (default for the
ruby gem). This also updates the configuration overrides for the
fluentd-daemonset deployment to provide input parity with the
default fluentbit configuration by adding inputs for the docker
and kubelet systemd units, inputs for ceph, libvirt, kernel logs,
and auth logs on the host. Finally, this updates the fluentd
template to include environment variables for the host name and
the fluentd pod name so they can be added to logged events through
fluentd filter plugins
Change-Id: I21f7a89a325c44f8b058ff01a20191bea1a210b4
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This moves Fluentd to use the helm-toolkit endpoint lookup for
using the fqdn for the Elasticsearch hostname instead of the
standard short host name
Change-Id: Ibe640979002331693f0a9b6155c9014572294664
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the default fluentd image to use the fluentd image
built with the systemd input plugin from the openstack-helm-images
repository
Change-Id: I7c75cd19d62f3dbc3fa4708642119f1781e58677
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This adds required changes to the Fluentd chart to allow for
deploying Fluentd as either a deployment or a daemonset. This
follows the pattern laid out by the ingress chart. This also
updates the single and multinode jobs to deploy fluentd as both
a daemonset and a deployment for validation
Change-Id: I84353a2daa2ce56ff59882a8d33203286ed27e06
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This begins to split the fluent-logging chart into two separate
charts, one for fluentbit and one for fluentd. This is to help
isolate each chart and its dependencies better, and to treat each
service as its own entity.
This also moves the job for creating Elasticsearch templates to
the Elasticsearch chart, as the elasticsearch chart should have
ownership of creating the templates for its indices.
This also performs some general cleanup of values keys that are
not currently used
Change-Id: I827277d5faa62b8b59c5960330703d23c297ca47
Signed-off-by: Steve Wilkerson <sw5822@att.com>
