The postgresql chart currently fails to run when deployed with
helm v3 due to invalid fields defined in values.yaml that are
more strictly enforced. This change removes these invalid values
to allow deploying the postgresql chart with helm v3.
Change-Id: Iabd3cfa77da618026ceb2dfdffd5d2a0b1519d93
Pick up the helm-toolkit DB backup enhancement in postgresql
to add capability to retry uploading backup to remote server.
Change-Id: I041d83211f08a8d0c9c22a66e16e6b7652bfc7d9
This will ease mirroring capabilities for the docker official images.
Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: I0f9177b0b83e4fad599ae0c3f3820202bf1d450d
This change adds cleanup mechanism to archive by following steps:
1) add archive_cleanup.sh under /tmp directory
2) through the start.sh this script will be triggered
3) It runs every hour, checking utilization of archive dir
4) If it is above threshold it deletes half of old files
Change-Id: I918284b0aa5a698a6028b9807fcbf6559ef0ff45
This reverts commit 982e3754a5.
"Add default reject rule end in Postgres pg_hba.conf to ensure all
connections must be explicitly allowed."
The original commit introduced a breaking change when installing with
the chart defaults - before, all remote connections with md5 auth were
allowed, and after the change, only explicit users are allowed.
This is fully overridable, but the original defaults are more
conservative.
Change-Id: Ib297e480bccd3ac7c0cf15985b3def2c8b3e889e
* add preStop hook to trigger Fast Shutdown
* disable readiness probe by default
When Kubernetes terminates a pod, the container runtime typically sends
a SIGTERM signal to pid 1 in each container [0]. PostgreSQL interprets
SIGTERM as a request to do a "Smart Shutdown" [1]. This can take minutes
(often exhausting the termination grace period), and during this time,
new connections are not being serviced.
Now that postgresql has a single replica, this behavior is undesirable.
If we kill the pod (e.g. in an upgrade), we probably want it to come
back as soon as possible.
This change adds a preStop hook that sends a SIGINT to postgresql in
order to trigger a "Fast Shutdown". In addition, the readiness probe is
disabled by default, since it adds no value in a single-replica
scenario.
0: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
1: https://www.postgresql.org/docs/9.6/server-shutdown.html
Change-Id: Ib5f3d2a49e55332604c91f9a011e87d78947dbef
Uses the standard helm-toolkit macros for liveness and readiness probes,
allowing them to be enabled or disabled, and params to be overridden.
The existing hard-coded settings are preserved as the chart defaults.
Change-Id: Idd063e6b8721126c88fa22c459f93812151d7b64
To safeguard postgres from clogging up wal files
in pg_xlog directory, This change does the following:
1) adding postgres archiving to move the WAL file to different directory
2) Makes sure that archive is in different Persistent volume.
Change-Id: I59bc76f27384d4f3836ef609855afcc33a7b99d0
default reject at the end of pg_hba.conf is added to ensure all connections must be explicitly allowed.
List of dependant users are added to allow connections are:
1. postgresql-admin
2. postgres
3. psql_exporter
Change-Id: Ic7bd19e5eb4745b91d94d5a88851280054459547
There are a couple of issues that need fixing:
1) "backoffLimit" and "activeDeadlineSeconds" attributes are placed in
the CronJob part of the cron-job-backup-postgres.yaml, but should be
placed in the Job template part.
2) The backup cronjob had two names in the values.yaml
"backup_postgresql" and "postgresql_backup" in various places. It should
be "postgresql_backup" in all of those places so that the CronJob can be
deployed correctly.
Change-Id: Ifd1c7c03ee947763ac073e55c6d74c211615c343
This updates the chart to include the pod security context
on the pod template.
This also adds the container security context to set
readOnlyRootFilesystem flag to true
Change-Id: Icb7a9de4d98bac1f0bcf6181b6e88695f4b09709
This PS fixes:
1) Removes printing of the word "Done" after the restore/list command
executes, which is not needed and clutters the output.
2) Fixes problem with list_tables related to command output.
3) Fixes parameter ordering problem with list_rows and list_schema
4) Adds the missing menu/parameter parsing code for list_schema
5) Fixes backup-restore secret and handling of PD_DUMPALL_OPTIONS.
6) Fixes single db restore, which wasn't dropping the database, and
ended up adding duplicate rows.
7) Fixes cronjob deficiencies - added security context and init containers,
fixed backup related service account related typos.
8) Fixes get_schema so that it only finds the table requested, rather
than other tables that also start with the same substring.
9) Fixes swift endpoint issue where it sometimes returns the wrong
endpoint, due to bad grep command.
Change-Id: I0e3ab81732db031cb6e162b622efaf77bbc7ec25
This patchset adds a secret containing the backup/restore configuration
for Postgresql, in case it is needed for invoking a backup/restore
operation from a different application or from a different namespace
(like from a utility container). Default is to not produce the secret.
Change-Id: I273fe169e7ee533c3fe04ad33c97af64b29bc16f
This patchset introduces the framework by which all OSH-based database
systems can use to backup and restore their databases. The framework
is refactored from the Postgresql backup and restore logic. This will
prevent alot of code duplication in the backup restore scripts across
each cluster.
In the process, some improvements needed to be made:
1) Removing the need for 2 separate containers to do the backup
and restore work to a remote gateway. This simplifies the design
and enables a higher level of robustness.
2) Adding separate "days to keep" config value for remote backup files,
as there may be different requirements for the remote files than the
local backup files.
3) Adding capability to send Storage_Policy when creating the remote
RGW swift container.
4) Making coding style improvement for readability and maintainability.
5) Fixing a deployment bug that occurs when remote backup is disabled.
Change-Id: I3a3482ad67320e89f04305b17da79abf7ad6eb45
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.
This change removes all references to this copyright by the
non-existent group and any blank lines underneath.
Change-Id: I1882738cf9757c5350a8533876fd37b5920b5235
1) Added a new backup container for accessing RGW via Openstack Swift API.
2) Modified the backup script so that tarballed databases can be sent to the RGW.
3) Added new script to send the database backup to the RGW.
4) Modified the restore script so that databases can be retrieved from the RGW.
5) Added new script to retrieve the database backups from the RGW.
Change-Id: Id17a8fcb63f5614ea038c58acdc256fb4e05f434
This change must enable postgresql-exporter to push additional metrics
(like replication_lag) which are derived using a SQL query against Postgres DB.
(Co-Author: Steven Fitzpatrick)
Change-Id: I78dc433a3782b48155ab293cb5afe90b3bc0ef1f
Currently postgresql database backup job will fail due to not having
correct permissions on the mounted PVC. This patchset corrects the
permissions on the PVC mount so that the backup pods can write to the
/var/backup directory structure.
Another problem was that pg_dumpall was not able to get the correct
password from the admin_user.conf. This may be due to the extra lines
in the file, so this patchset reads it differently in order to find
the password. This was a change to the backup and restore scripts.
Also there are a number of small corrections made to the error handling
for both backup and restore scripts, to be consistent with the MariaDB
backup/restore scripts.
Change-Id: Ica361764c591099e16d03a0988f73c6976583ceb
This commit adds an audit user to the postgresql database which
will have only SELECT privileges on the postgresql database tables.
This is accomplished by setting up audit user creation parameters
in the Patroni bootstrap environment settings, according to (1).
(1) https://patroni.readthedocs.io/en/latest/ENVIRONMENT.html
Change-Id: Idf1cd90b5d093f12fa4a3c5c794d4b5bbc6c8831
In this PS we explicitly define the admin user rather than letting
patroni use the default username and password.
Change-Id: I9885314902c3a60e709f96e2850a719ff9586b3d
This patch set places in a default kubernetes egress network
policy for postgresql database chart.
Change-Id: I6caa917faf23becc3a1c09b47f457b8b2db996e4
Signed-off-by: Tin Lam <tin@irrational.io>
This updates the kubernetes-entrypoint image reference to consume
the publicly available kubernetes-entrypoint image that is built
and maintained under the airshipit namespace, as the stackanetes
image is no longer actively maintained
Change-Id: I5bfdc156ae228ab16da57569ac6b05a9a125cb6a
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This commit fixes a small issue with Patroni where sometimes pg_rewind
would fail due to limitations in Postgres 9.5. To combat pg_rewind
failures, we can enable remove_data_directory_on_rewind_failure which
will cleanup the data directory on the pod and recreates it as a
replica so that the pod can restart from fresh, rather than churning in
an error state. This commit also sets
remove_data_directory_on_diverged_timelines to give Patroni a greater
ability to combat timeline divergence errors.
Change-Id: Ic9f75dbfa0dd990e2b215ed204e55cd67a5d1159
- Allow configuration of the termination grace period
for the Patroni pod with a default of 180s to ensure
the database has time to gracefully spin down, even
on slow disk.
Change-Id: I420cbd601bbffa50217b717bd4a636d48d324617
- Change the Postgres configuration to use x509 client
certs for authenticating the connections for replicating
between Patroni nodes. This is a straightforward solution
for support credential rotation for the replication user.
Password authentication is problematic due to the declartive
nature of helm charts and requiring an existing replication
connection to replicate the rotated password.
Change-Id: I0c5456a01b3a36fee8ee4c986d25c4a1d807cb77
This patch is part of an effort to cleanup the values.yaml file for
Postgres, which has gotten messy since the introduction of Patroni. This
patch specifically removes unused configuration values which were
causing unnecessary bloat and complexity.
Change-Id: I96180fd9c91200ba7558e58bd503b4ef9ebc183e
This patch fixes an issue with Postgres HA where
the PVC which stores the database was filling up with
WAL records and not deleting them due to some
misconfigurations with Postgres. Once the PVC
would fill up, replication would fail across the node
and the database would not be able to start, crashing
the system.
Specifically, archive_mode was turned on, but was not
supplied with a function through which to archive the
logs. When WAL archiving is turned on, old WAL files
cannot be removed until the system has archived them first.
However, since we never told the system how to archive the
files, it would repeatedly fail so the WAL files would
never be cleaned up.
Also in this patch are some small house keeping items:
- Lowered the wal_keep_segments drastically so Postgres
can't keep as many WAL segments around to minimize the
chance of PVC fill issues
- Turned the wal_level from 'logical' to 'hot_standby'
to keep it consistent with the fact that Patroni uses
streaming replication and not logical replication
- Removed the autovaccuum configurations as they are not
needed
Change-Id: Id48c3ee9976823b2bdb4395a029fe75476bdaa62
This patchset aims to add HA Clustering support for Postgres. HA Clustering
provides automatic failover in the event of the database going down in addition
to keeping replicas of the database for rebuilding in the event of a node
going down. To achieve this clustering we use
[Patroni](https://github.com/zalando/patroni) which offers HA clustering
support for Postgres.
Patroni is a daemon that runs in the background and keeps track of which
node in your cluster is currently the leader node and routes all traffic
on the Postgresql endpoint to that node. If the leader node goes down,
Patroni holds an election to chose a new leader and updates the endpoint
to route traffic accordingly. All communication between nodes is done by
a Patroni created endpoint, seperate from the externally facing Postgres
endpoint.
Note that, although the postgresql helm chart can be upgraded from
non-patroni to patroni clustering, the previous `postgresql`
endpoints object (which is not directly managed by helm) must be
deleted via an out-of-band mechanism so that it may be replaced by the
patroni-managed endpoints. If Postgres itself is leveraged for the
deployment process, this must be done with careful timing. Note that
the old endpoints had a port named "db", and the new endpoints has
a port named "postgresql".
- Picking up patchset: https://review.openstack.org/#/c/591663
Co-authored-by: Tony Sorrentino <as1413@att.com>
Co-authored-by: Randeep Jalli <rj2083@att.com>
Co-authored-by: Pete Birley <pete@port.direct>
Co-authored-by: Matt McEuen <mm9745@att.com>
Change-Id: I721b745017dc1ea7ae05dfd9f8d5dd08d0965985
This ps exposes the anti-affinity weight value, including
default, that will be consumed by the updated htk function.
Change-Id: Id8eb303674764ef8b0664f62040723aaf77e0a54
This adds a security context to the postgresql exporter, which
changes the pod's user from root to the nobody user instead
This also adds the container security context to set
allowPrivilegeEscalation to false and readOnlyRootFilesystem to true
Change-Id: Ibe49f77ed2d0a588b5abe175318edd1c82a57cca
* backup script for postgresql
* restore script for postgresql
* cronjob to control backup automation for postgresql
* add parameters to values.yaml
Change-Id: I5eaa82e824c9f361aa667c431cd93058391f2e60
- Postgres initdb fails running as non-root as it cannot
change the ownership or permission on the PVC mounted
to the container. Update the chart to use a uid 0 init
container for setting ownership before the postgres
container starts.
Change-Id: I648fe7ca3dbc1f6ca6f4513360de2278be7c1ce4
- Make the default to run the postgres database as the uid 999 which
the default image maps to the 'postgres' user
- If the database is already initialized, before starting postgres
set the 'postgres' database user password to match the declared
intended password
Change-Id: I7b0ea7a86246b098f38ef4c03dd157731f61e066
This removes the auth credentials from the postgresql exporter's
endpoint configuration, as the secret and job for creating that
user come from the auth credentials defined in the postgresql
endpoint instead
Change-Id: Id17578b8e22a1808d7c3323bda9cd005c2584c97
This PS implements the helm toolkit function to generate the
Egress in kubernetes network policy manifest based on overrideable values.
It also enbale the K8s network policy at Osh-infra gate.
Change-Id: Icbe2a18c98dba795d15398dcdcac64228f6a7b4c
This moves the postgresql chart to openstack-helm-infra as part of
the effort to move charts to the appropriate repositories
Change-Id: I25c026e5d4c4abe4dd0805047051281911632739
Story: 2002204
Task: 21729
