The set -x has produced 6 identical log strings every time the
log_backup_error_exit function is called. Prometheus is using
the occurrence and number of some logs over a period of time to
evaluate database backup failure or not. Only one log should be
generated when a particular database backup scenario failed.
Upon discussion with database backup and restore SME, it is
recommended to remove the set -x once and for all.
Change-Id: I846b5c16908f04ac40ee8f4d87d3b7df86036512
Pick up the helm-toolkit DB backup enhancement in postgresql
to add capability to retry uploading backup to remote server.
Change-Id: I041d83211f08a8d0c9c22a66e16e6b7652bfc7d9
This change adds cleanup mechanism to archive by following steps:
1) add archive_cleanup.sh under /tmp directory
2) through the start.sh this script will be triggered
3) It runs every hour, checking utilization of archive dir
4) If it is above threshold it deletes half of old files
Change-Id: I918284b0aa5a698a6028b9807fcbf6559ef0ff45
This PS adds the capability to Mariadb and Postgresql to backup a
single database (as an optional parameter to the backup script).
Change-Id: I9bc1eb0173063638b2cf58465c063f602ed20bc1
Recently, the Postgresql backups were modified to generate drop database
commands (--clean pgdumpall option). Also for single database restore,
a DROP DATABASE command was added before the restore so that the
database could be restored without duplicate rows. However, if there are
existing database connections (by the applications or other users), then
the drop database commands will fail. So for the duration of the restore
database operation, the databases being restored need to have their
existing connections dropped and new connections prevented until the
database(s) restored, then connections should be re-allowed.
Also found a problem with psql returning 0 (success code) even though
there were errors during its execution. The solution is to check the
output for errors and if there are any, dump out the log file for the
user to see and let the user know there are errors.
Lastly, a problem was found with the single database restortion, where
the database dump for a single database was being incorrectly extracted
from the psql dump file, resulting in the database not being restored
correctly (most of the db being wiped out). This patchset fixes that
issue as well.
Change-Id: I4db3f6ac7e9fe7cce6a432dfba056e17ad1e3f06
This PS fixes:
1) Removes printing of the word "Done" after the restore/list command
executes, which is not needed and clutters the output.
2) Fixes problem with list_tables related to command output.
3) Fixes parameter ordering problem with list_rows and list_schema
4) Adds the missing menu/parameter parsing code for list_schema
5) Fixes backup-restore secret and handling of PD_DUMPALL_OPTIONS.
6) Fixes single db restore, which wasn't dropping the database, and
ended up adding duplicate rows.
7) Fixes cronjob deficiencies - added security context and init containers,
fixed backup related service account related typos.
8) Fixes get_schema so that it only finds the table requested, rather
than other tables that also start with the same substring.
9) Fixes swift endpoint issue where it sometimes returns the wrong
endpoint, due to bad grep command.
Change-Id: I0e3ab81732db031cb6e162b622efaf77bbc7ec25
Adding the capability to retrieve a list of tables, list of rows,
and the table schema information from a given database backup
archive file, for the purpose of manual database table/row
restoration and also for just viewing.
This is added to the HTK _restore_main.sh.tpl and is integrated
into the Postgresql restore script (Mariadb will be done later).
Change-Id: I729ecf7a720f1847a431de7e149cec6841ec67b8
This patchset introduces the framework by which all OSH-based database
systems can use to backup and restore their databases. The framework
is refactored from the Postgresql backup and restore logic. This will
prevent alot of code duplication in the backup restore scripts across
each cluster.
In the process, some improvements needed to be made:
1) Removing the need for 2 separate containers to do the backup
and restore work to a remote gateway. This simplifies the design
and enables a higher level of robustness.
2) Adding separate "days to keep" config value for remote backup files,
as there may be different requirements for the remote files than the
local backup files.
3) Adding capability to send Storage_Policy when creating the remote
RGW swift container.
4) Making coding style improvement for readability and maintainability.
5) Fixing a deployment bug that occurs when remote backup is disabled.
Change-Id: I3a3482ad67320e89f04305b17da79abf7ad6eb45
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.
This change removes all references to this copyright by the
non-existent group and any blank lines underneath.
Change-Id: I1882738cf9757c5350a8533876fd37b5920b5235
1) Added a new backup container for accessing RGW via Openstack Swift API.
2) Modified the backup script so that tarballed databases can be sent to the RGW.
3) Added new script to send the database backup to the RGW.
4) Modified the restore script so that databases can be retrieved from the RGW.
5) Added new script to retrieve the database backups from the RGW.
Change-Id: Id17a8fcb63f5614ea038c58acdc256fb4e05f434
Currently postgresql database backup job will fail due to not having
correct permissions on the mounted PVC. This patchset corrects the
permissions on the PVC mount so that the backup pods can write to the
/var/backup directory structure.
Another problem was that pg_dumpall was not able to get the correct
password from the admin_user.conf. This may be due to the extra lines
in the file, so this patchset reads it differently in order to find
the password. This was a change to the backup and restore scripts.
Also there are a number of small corrections made to the error handling
for both backup and restore scripts, to be consistent with the MariaDB
backup/restore scripts.
Change-Id: Ica361764c591099e16d03a0988f73c6976583ceb
This patch set fixes the calculation of how long a database backup
has be taken. In the existing code, the time difference is rounded
to days, even a second less than 4 days will be rounded to 3 days.
This effectively allows archives to be kept for one additional day.
The new calculation and comparison is based on seconds.
Change-Id: I5547e923538ddb83f409b1e7df936baf664e717a
- Change the Postgres configuration to use x509 client
certs for authenticating the connections for replicating
between Patroni nodes. This is a straightforward solution
for support credential rotation for the replication user.
Password authentication is problematic due to the declartive
nature of helm charts and requiring an existing replication
connection to replicate the rotated password.
Change-Id: I0c5456a01b3a36fee8ee4c986d25c4a1d807cb77
This patchset aims to add HA Clustering support for Postgres. HA Clustering
provides automatic failover in the event of the database going down in addition
to keeping replicas of the database for rebuilding in the event of a node
going down. To achieve this clustering we use
[Patroni](https://github.com/zalando/patroni) which offers HA clustering
support for Postgres.
Patroni is a daemon that runs in the background and keeps track of which
node in your cluster is currently the leader node and routes all traffic
on the Postgresql endpoint to that node. If the leader node goes down,
Patroni holds an election to chose a new leader and updates the endpoint
to route traffic accordingly. All communication between nodes is done by
a Patroni created endpoint, seperate from the externally facing Postgres
endpoint.
Note that, although the postgresql helm chart can be upgraded from
non-patroni to patroni clustering, the previous `postgresql`
endpoints object (which is not directly managed by helm) must be
deleted via an out-of-band mechanism so that it may be replaced by the
patroni-managed endpoints. If Postgres itself is leveraged for the
deployment process, this must be done with careful timing. Note that
the old endpoints had a port named "db", and the new endpoints has
a port named "postgresql".
- Picking up patchset: https://review.openstack.org/#/c/591663
Co-authored-by: Tony Sorrentino <as1413@att.com>
Co-authored-by: Randeep Jalli <rj2083@att.com>
Co-authored-by: Pete Birley <pete@port.direct>
Co-authored-by: Matt McEuen <mm9745@att.com>
Change-Id: I721b745017dc1ea7ae05dfd9f8d5dd08d0965985
* Postgresql lack the feature of creating databases if not exist
* Add a function to workaround in the restore script.
Change-Id: If451204e3cec573a0dbfe91549a47ee569799a22
* backup script for postgresql
* restore script for postgresql
* cronjob to control backup automation for postgresql
* add parameters to values.yaml
Change-Id: I5eaa82e824c9f361aa667c431cd93058391f2e60
- Make the default to run the postgres database as the uid 999 which
the default image maps to the 'postgres' user
- If the database is already initialized, before starting postgres
set the 'postgres' database user password to match the declared
intended password
Change-Id: I7b0ea7a86246b098f38ef4c03dd157731f61e066
This moves the postgresql chart to openstack-helm-infra as part of
the effort to move charts to the appropriate repositories
Change-Id: I25c026e5d4c4abe4dd0805047051281911632739
Story: 2002204
Task: 21729
