Commit Graph

371 Commits

Author SHA1 Message Date
Zuul
ce2c0e08d2 Merge "Update helm version to 2.14.1" 2019-07-04 18:36:57 +00:00
chengli3
13c89f9721 Update ldap overrides values file reference
The ldap overrides values file had been moved to
keystone/values_overrides[1]. This patch is to update the reference.

[1]
cede6c0d48 (diff-89208df3c46570cf56141a9353ce27a7)

Change-Id: Ib03bb979dc681a647abd36df77f55fd82e0d4df6
2019-07-02 11:02:06 +08:00
Kabanov, Dmitrii (dk370c)
1db1ddf0ba [Kibana] Add improvements to Selenium tests for Kibana
This PS adds several fixes to Selenium tests (for Kibana) and adds
role which allows to collect the results.

Change-Id: If9fb5f50e395379fdd3ccc46e945a93606dcbabe
2019-06-25 20:51:01 -07:00
Steve Wilkerson
b117b14c3a Update helm version to 2.14.1
This updates the helm version from 2.13.1 to 2.14.1

Change-Id: I619351d846253bf17caa922ad7f7b0ff19c778a2
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-06-25 21:33:20 +00:00
Zuul
28d3cb538b Merge "Remove fluent-loggin from network-policy" 2019-06-25 03:28:05 +00:00
Kabanov, Dmitrii (dk370c)
44c1dcc4c9 [etcd] Move etcd storage to tmpfs
The PS allows to use tmpfs for etcd during the gates.
There is an assumption that it will improve the performance and
will allow to get rid of weird issues.

Change-Id: Id68645b6535c9b1d87c133431b7cd6eb50fb030e
2019-06-24 11:14:21 -07:00
MegHeisler
d5af61b751 Remove fluent-loggin from network-policy
This removes the old fluent-logging chart from network
policy and replaces it with the new fluentbit and fluentd
charts. This will return the network policy gate back to
passing

Change-Id: I060c6c3034fa798a131a053b9d496e5d8781c55d
2019-06-24 16:14:05 +00:00
Randeep Jalli
1c4084bdc0 add docker-default apparmor profile for prometheus-alertmanager
Add in prometheus-alertmanager gate script as a script

Change-Id: I3c10f9a9d4403fd91da292a50d204f73a9295611
2019-06-22 10:13:18 +00:00
Steve Wilkerson
f2db36862c Add openstack-exporter to aio-monitoring & openstack-support jobs
This deploys the openstack-exporter as part of the aio-monitoring
single node and openstack-support jobs, to ensure the exporter is
functional and working as expected

Change-Id: If31046769fd9032647af47ce8fce0cde5b8f7d7d
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-06-19 16:38:10 +00:00
Renis Makadia
c7f5c9979c Add helm tests for Ceph Provisioners chart
- Adding helm tests for Ceph provisioner chart
- Helm test should only executed when deploying chart with
client_secrets: true.

Co-Authored-By: Chinasubbareddy Mallavarapu <cr3938@att.com>

Change-Id: I33421249246dfaf6ea4f835e76a74813dfb3b595
2019-06-12 12:32:30 -05:00
Steve Wilkerson
40769d5a60 Nagios: Add Selenium tests via helm test pod
This adds selenium tests for the Nagios chart via a helm test
pod to help ensure the Nagios deployment is functional and
accessible

Change-Id: I44f30fbac274546abadba0290de029ed2b9d1958
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-06-06 19:35:59 +00:00
Steve Wilkerson
0d2ac10fc2 Update apparmor job after fluent-logging split
This updates the apparmor job to account for the splitting of the
fluent-logging chart, as it was missed during that change.  Now,
the apparmor job will deploy fluentbit as well as fluentd deployed
as a daemonset running as a collecting agent

Change-Id: Iefa50f474b57a10c5e7e5a9032c7b23d26d97640
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-06-04 12:58:33 +00:00
Steve Wilkerson
aee9708b06 Fluentd: Update configuration defaults and overrides
This updates the default fluentd configuration to include
recommended settings for preventing the elasticsearch plugin from
reloading the connection after 10000 requests (default for the
ruby gem). This also updates the configuration overrides for the
fluentd-daemonset deployment to provide input parity with the
default fluentbit configuration by adding inputs for the docker
and kubelet systemd units, inputs for ceph, libvirt, kernel logs,
and auth logs on the host.  Finally, this updates the fluentd
template to include environment variables for the host name and
the fluentd pod name so they can be added to logged events through
fluentd filter plugins

Change-Id: I21f7a89a325c44f8b058ff01a20191bea1a210b4
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-06-03 16:00:44 +00:00
Steve Wilkerson
0a8b710083 Elasticsearch: Add job history to Curator, update schedule key
This updates the Elastic Curator cron job to include configuration
for successful and failed job history limits, similar to the other
cron jobs we deploy. This also moves the key for configuring the
cron schedule from under .Values.conf.curator to a new top level
jobs key to maintain consistency

This also fixes an indentation issue with the deployment overrides
for Curator as well as adds the overrides for the Armada job

Change-Id: I9c720df9677215bdd2bf18be77959bd5f671c0ca
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-05-30 15:28:30 +00:00
Doug Aaser
9efb353b83 Patroni inclusion work for HA Postgres
This patchset aims to add HA Clustering support for Postgres. HA Clustering
provides automatic failover in the event of the database going down in addition
to keeping replicas of the database for rebuilding in the event of a node
going down. To achieve this clustering we use
[Patroni](https://github.com/zalando/patroni) which offers HA clustering
support for Postgres.

Patroni is a daemon that runs in the background and keeps track of which
node in your cluster is currently the leader node and routes all traffic
on the Postgresql endpoint to that node. If the leader node goes down,
Patroni holds an election to chose a new leader and updates the endpoint
to route traffic accordingly. All communication between nodes is done by
a Patroni created endpoint, seperate from the externally facing Postgres
endpoint.

Note that, although the postgresql helm chart can be upgraded from
non-patroni to patroni clustering, the previous `postgresql`
endpoints object (which is not directly managed by helm) must be
deleted via an out-of-band mechanism so that it may be replaced by the
patroni-managed endpoints.  If Postgres itself is leveraged for the
deployment process, this must be done with careful timing.  Note that
the old endpoints had a port named "db", and the new endpoints has
a port named "postgresql".

- Picking up patchset: https://review.openstack.org/#/c/591663

Co-authored-by: Tony Sorrentino <as1413@att.com>
Co-authored-by: Randeep Jalli <rj2083@att.com>
Co-authored-by: Pete Birley <pete@port.direct>
Co-authored-by: Matt McEuen <mm9745@att.com>

Change-Id: I721b745017dc1ea7ae05dfd9f8d5dd08d0965985
2019-05-28 19:13:13 +00:00
Steve Wilkerson
bdaf866a4e Fluentd: Support Daemonset deployment
This adds required changes to the Fluentd chart to allow for
deploying Fluentd as either a deployment or a daemonset. This
follows the pattern laid out by the ingress chart. This also
updates the single and multinode jobs to deploy fluentd as both
a daemonset and a deployment for validation

Change-Id: I84353a2daa2ce56ff59882a8d33203286ed27e06
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-05-28 08:23:44 -05:00
Jean-Philippe Evrard
ed200cb3b0 Wait for pods for openstack-support scenario
The wait for pods is not consistently used in the
openstack-support scenario.

This is a problem, as some helm charts deploys are
basically masking issues that can arise.

This should fix it.

Change-Id: Ib3e8f16bea701bf20375d4deec7c7869e7bf85c2
2019-05-28 07:51:34 +00:00
Zuul
5cb47aed80 Merge "Separate fluentbit and fluentd charts" 2019-05-24 17:34:27 +00:00
Tin Lam
752bbf48c4 Migrate ubuntu jobs from Xenial to Bionic
This patch set is based on [0] that also fixed up a handful of Bionic-
specific changes required for the gates to pass.

[0] https://review.openstack.org/#/c/649698/

Co-Authored-By: ghanshyam <gmann@ghanshyammann.com>

Change-Id: I217a27c53eec2a51ddbea7226a23042558c5946b
2019-05-24 13:52:08 +00:00
Steve Wilkerson
abb5e0f713 Separate fluentbit and fluentd charts
This begins to split the fluent-logging chart into two separate
charts, one for fluentbit and one for fluentd. This is to help
isolate each chart and its dependencies better, and to treat each
service as its own entity.

This also moves the job for creating Elasticsearch templates to
the Elasticsearch chart, as the elasticsearch chart should have
ownership of creating the templates for its indices.

This also performs some general cleanup of values keys that are
not currently used

Change-Id: I827277d5faa62b8b59c5960330703d23c297ca47
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-05-24 06:31:09 -05:00
Steve Wilkerson
b44d99b36a Fluent-logging: Remove utils for generating configuration files
This removes the utilities for generating the fluentd, fluentbit,
and parser configuration files from yaml and moves to instead
consume the configuration files as strings from the values.yaml.

This allows for easier creation and maintenance of configuration
files for fluentd and fluentbit, as the utilities became unwieldy
with complex configuration files.

This necessitated the removal of the core test executed by the
charts helm tests, but this would be required as we move to split
the charts regardless

Change-Id: Ied4a76bbdf58b54a6d702db04a7120b64f54dcac
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-05-23 13:06:19 -05:00
Zuul
85c03c422e Merge "Add docker-default (enforce) AppArmor profile to openvswitch" 2019-05-23 15:05:53 +00:00
Renis Makadia
5985b61286 Ceph-Client: Update, Enable and Cleanup helm tests
- Update ceph-client chart to
1) By default, enable ceph-client helm test. Update enabler
key in values.yaml to follow pattern as in other charts
2) Add needed dependancy for ceph-client helm tests
3) Update helm test script to reduce output and update
error msgs
4) Removed unwanted ENV variables SPECS and EXPECTED_POOLMINSIZE
- Update gate scripts to run helm test command

Change-Id: I6a0e4f5107e49dac081ac2037bcc0f9c0864793f
2019-05-18 03:09:45 +00:00
Steve Wilkerson
7c093716ca Enable fluentd monitoring in single and multinode jobs
This updates the scripts for deploying fluentd to include
overrides for enabling prometheus monitoring. Despite not
deploying prometheus in the osh-infra-logging job, we can still
leverage the post run job to gather metrics from the exporters
service. This gives us the means for verifying the functionality
of the exporter

Change-Id: Id98474de89d86419157635007e2f114f0947498e
2019-05-10 01:18:13 +00:00
Zuul
ead842d1d8 Merge "Elasticsearch: Heap configuration and ingest node updates" 2019-05-09 21:10:26 +00:00
Steve Wilkerson
4c0f82eb9a Add missing password to armada-lma manifest
This adds the password for the mariadb sst user to the armada-lma
manifest, as it was previously missed

Change-Id: I8768569fff96bf15cb4b2a577a0f667972fda886
2019-05-08 08:34:34 -05:00
Steve Wilkerson
031ee3e6af Elasticsearch: Heap configuration and ingest node updates
This updates the Elasticsearch chart to allow for setting the
heap size per node type instead of for all nodes equally. This
also adds the required environment variable to configure whether
a node is an ingest node. This is set to false, as suggested for
elasticsearch versions <= 6.x

This also removes the ES_PLUGINS_INSTALL environment variable as
it is not used for anything in the current charts

Change-Id: I9096774db46dcbcd48b8a5448f0510984bf4108f
2019-05-06 14:55:45 -05:00
Zuul
9771b16e87 Merge "Define test specific timeouts for Armada LMA components" 2019-05-03 15:07:43 +00:00
Jagan Kavva
34d691c9b1 Add docker-default (enforce) AppArmor profile to openvswitch
Change-Id: I7b091f668d9293d7eafd9c1b54c4eab715bbd93c
2019-04-29 09:24:41 -05:00
diwakarthyagaraj
0ed4f0de5e Add Docker default AppArmor profile to Fluentbit and Elasticsearch
Change-Id: I21efbf8d434f6245eef04308973af4e7ec0b2380
Co-authored-by: ld366r@att.com
2019-04-25 13:55:52 -05:00
Zuul
6d0a9c21b0 Merge "Fix Ceph deployment in apparmor job" 2019-04-25 15:35:10 +00:00
Zuul
56bfb4a02d Merge "Update airship-divingbell check job" 2019-04-25 02:54:14 +00:00
Meg Heisler
a600471cb0 Define test specific timeouts for Armada LMA components
This adds test specific timeout to all LMA components in
Armada. This also removes test enabled true because
Armada enables tests by default

Change-Id: I893342b36ba27cbe7d3ce8767f73795e84a11732
2019-04-24 11:07:18 -05:00
Meg Heisler
010faee9d5 Add wait.resource for LMA services to armada manifest
This adds the wait.resource.type to each LMA service in
the armada manifest

Change-Id: I86adbb1a5325ce2beef8506a406865e8db53876b
2019-04-24 00:39:05 +00:00
Steve Wilkerson
880f32f059 Fix Ceph deployment in apparmor job
This fixes the ceph deployment in the apparmor job as the previous
overrides weren't entirely correct.  This also reorders the
deployment steps in the apparmor job to enforce the sequential
naming scheme used

Change-Id: I161bae649d4ff67307abeadc12b3c7d321af31c7
2019-04-23 10:25:54 -05:00
Steve Wilkerson
2c11798c65 Update airship-divingbell check job
This updates the airship-divingbell repository defined in the
single node check job to use the correct repository namespace.

This also updates the divingbell check job to use the standard
osh-infra-gate-runner playbook as well as the minikube based
kubernetes deployment

Change-Id: Iff53279b3e09058deb323d092955cbf87230b5e5
2019-04-23 10:09:27 -05:00
Pete Birley
a6558281a1 Gate: Remove stable helm repo is present to improve build time
This PS remvoes the stable helm repo, if present, to improve the
build time of patches.

Change-Id: Id6ec86e5ff426994b12adf4ca8e80eda2e52f147
Signed-off-by: Pete Birley <pete@port.direct>
2019-04-22 07:27:12 -05:00
Randeep Jalli
8ed0c1dd2b This commit adds docker-default apparmor profile for prometheus-openstack-exporter.
Change-Id: I1461246505f42c41051cebd981217c0064d39928
2019-04-21 03:17:18 +00:00
Randeep Jalli
ccfd614141 This commit adds docker-default apparmor profile for prometheus-node-exporter.
Change-Id: Ie8660e206280184eb5f4c03b7dd54047436c16ba
2019-04-20 19:01:20 +00:00
Zuul
f9404949d8 Merge "Add Docker default AppArmor profile to Ceph-osd Change apparmor for ceph-osd into gate script Change-Id: I587c98dd6e55b8eb7af0c8100f2cb3cb1be96438" 2019-04-18 20:45:10 +00:00
Steve Wilkerson
2e8c96a623 Elasticsearch, Fluent-logging, Kibana Ingress Policy
This adds ingress network policy for the fluent-logging, kibana
and Elasticsearch charts. This leverages the helm-toolkit template
that was used in openstack-helm for the openstack services

Change-Id: I2a89b62f1002851346e9a25de40113078e9c518f
2019-04-16 19:44:46 +00:00
Zuul
6b17525b93 Merge "Fix YAML indentation" 2019-04-16 17:34:26 +00:00
Zuul
5da1320e3f Merge "Update ceph armada manifest" 2019-04-15 20:33:11 +00:00
Steve Wilkerson
4c85854246 Update ceph armada manifest
This updates the ceph armada manifest with new resource
waits where applicable

Change-Id: I6a622fe13dfa7317a3b5a7ed3f57b713b441fab0
2019-04-15 13:03:37 -05:00
dt241s
7e868649bb Add Docker default AppArmor profile to Ceph-osd
Change apparmor for ceph-osd into gate script
Change-Id: I587c98dd6e55b8eb7af0c8100f2cb3cb1be96438
2019-04-15 11:08:47 -05:00
Tin Lam
36108b1db8 Fix YAML indentation
Trivial fix. This patch set fixes inconsistent indentations in YAML file.

Change-Id: I98ed9680d93f9c21e44b7da8462c9ce3607350bd
Signed-off-by: Tin Lam <tin@irrational.io>
2019-04-13 00:47:22 +00:00
Randeep Jalli
26faa8a66d Add docker-default apparmor profile for prometheus process exporter.
Change-Id: Iaa1eb80074c280ff00cc599105b598130132cc26
2019-04-11 17:16:49 +00:00
kranthikirang
6c45566132 ceph-mon, tenat-ceph storageclass schema
In order to align with ceph-provisioner storageclass
.Values we need to update ceph-mon .Values and tenant-
ceph scripts

Change-Id: I36fb07de9e791ac33cf0b4c38b3e4d63337d3e72
Story: storageclass schema changes
Signed-off-by: kranthi guttikonda <kranthi.guttikonda@att.com>
Signed-off-by: kranthikirang <kranthi.guttikonda@b-yond.com>
2019-04-09 22:10:23 +00:00
Steve Wilkerson
a63f3b3662 MariaDB: Add network policy template for mysql exporter
This adds the network policy template and overrides for the mysql
prometheus exporter.

Change-Id: I83b526d11fe25f45b02a7e797b785b8ad59c7848
2019-04-09 12:16:22 -05:00
Zuul
1a81952929 Merge "MariaDB: add basic sanity test" 2019-04-07 08:20:21 +00:00