From fb89e75ce48c2e2aebe307c5db9e2089e69ec4da Mon Sep 17 00:00:00 2001 From: Oleksii Kliukin Date: Tue, 20 Dec 2016 18:22:57 +0100 Subject: [PATCH] Make patroni documentation available on patroni.readthedocs.io. (#373) Run sphnix-quickstart and some workarounds. Sphinx is a logical choice because our docs is already in .rst. --- .gitignore | 5 + CONTRIBUTING.md | 19 --- README.rst | 2 +- docs/CONTRIBUTING.rst | 24 +++ docs/ENVIRONMENT.rst | 2 + docs/Makefile | 20 +++ docs/README.rst | 90 ++++++++++ docs/SETTINGS.rst | 4 + docs/conf.py | 190 +++++++++++++++++++++ docs/dynamic_configuration.rst | 6 +- docs/index.rst | 36 ++++ docs/pause.rst | 2 + docs/releases.rst | 299 +++++++++++++++++++++++++++++++++ docs/replication_modes.rst | 2 + 14 files changed, 679 insertions(+), 22 deletions(-) delete mode 100644 CONTRIBUTING.md create mode 100644 docs/CONTRIBUTING.rst create mode 100644 docs/Makefile create mode 100644 docs/README.rst create mode 100644 docs/conf.py create mode 100644 docs/index.rst create mode 100644 docs/releases.rst diff --git a/.gitignore b/.gitignore index 236f02ea..4e614bb2 100644 --- a/.gitignore +++ b/.gitignore @@ -46,3 +46,8 @@ dummy pgpass scm-source.json + +# Sphinx-generated documentation +docs/build/ +docs/source/_static/ +docs/source/_templates/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index cda1e856..00000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,19 +0,0 @@ -# Contributing guidelines - -Wanna contribute to Patroni? Yay - here is how! - -## Filing issues - -If you have a question about patroni or have a problem using it, please read the [readme]( README.rst) before filing an issue. -Also double check with the current issues on our [Issues Tracker](https://github.com/zalando/patroni/issues). - -## Contributing a pull request - -1. Submit a comment to the relevant issue or create a new issue describing your proposed change. -1. Do a fork, develop and test your code changes. -1. Include documentation -1. Submit a pull request. - -You'll get feedback about your pull request as soon as possible. - -Happy Patroni hacking ;-) diff --git a/README.rst b/README.rst index 4ca9839d..c152a555 100644 --- a/README.rst +++ b/README.rst @@ -32,7 +32,7 @@ For additional background info, see: Development Status ================ -Patroni is in active development and accepts contributions. See our `Contributing `__ section below for more details. +Patroni is in active development and accepts contributions. See our `Contributing `__ section below for more details. We report new releases information `here `__. diff --git a/docs/CONTRIBUTING.rst b/docs/CONTRIBUTING.rst new file mode 100644 index 00000000..09409d8a --- /dev/null +++ b/docs/CONTRIBUTING.rst @@ -0,0 +1,24 @@ +.. _contributing: + +Contributing guidelines +======================= + +Wanna contribute to Patroni? Yay - here is how! + +Reporting issues +---------------- + +If you have a question about patroni or have a problem using it, please read the :ref:`README ` before filing an issue. +Also double check with the current issues on our `Issues Tracker `__. + +Contributing a pull request +--------------------------- + +1) Submit a comment to the relevant issue or create a new issue describing your proposed change. +2) Do a fork, develop and test your code changes. +3) Include documentation +4) Submit a pull request. + +You'll get feedback about your pull request as soon as possible. + +Happy Patroni hacking ;-) diff --git a/docs/ENVIRONMENT.rst b/docs/ENVIRONMENT.rst index 68ec7343..c446b725 100644 --- a/docs/ENVIRONMENT.rst +++ b/docs/ENVIRONMENT.rst @@ -1,3 +1,5 @@ +.. _environment: + ================================== Environment Configuration Settings ================================== diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 00000000..0b252c85 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +SPHINXPROJ = Patroni +SOURCEDIR = . +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/README.rst b/docs/README.rst new file mode 100644 index 00000000..1787af23 --- /dev/null +++ b/docs/README.rst @@ -0,0 +1,90 @@ +.. _readme: + +================= +How Patroni Works +================= + +Patroni originated as a fork of `Governor `__, the project from Compose. It includes plenty of new features. + +For an example of a Docker-based deployment with Patroni, see `Spilo `__, currently in use at Zalando. + +For additional background info, see: + +* `PostgreSQL HA with Kubernetes and Patroni `__, talk by Josh Berkus at KubeCon 2016 (video) +* `Feb. 2016 Zalando Tech blog post `__ + +================== +Development Status +================== + +Patroni is in active development and accepts contributions. See our :ref:`Contributing ` section below for more details. + +We report new releases information :ref:`here `. + +=================================== +Technical Requirements/Installation +=================================== + +**For Mac** + +To install requirements on a Mac, run the following: + +:: + + brew install postgresql etcd haproxy libyaml python + pip install psycopg2 pyyaml + +======================= +Running and Configuring +======================= + +To get started, do the following from different terminals: +:: + + > etcd --data-dir=data/etcd + > ./patroni.py postgres0.yml + > ./patroni.py postgres1.yml + +You will then see a high-availability cluster start up. Test different settings in the YAML files to see how the cluster's behavior changes. Kill some of the components to see how the system behaves. + +Add more ``postgres*.yml`` files to create an even larger cluster. + +Patroni provides an `HAProxy `__ configuration, which will give your application a single endpoint for connecting to the cluster's leader. To configure, +run: + +:: + + > haproxy -f haproxy.cfg + +:: + + > psql --host 127.0.0.1 --port 5000 postgres + +================== +YAML Configuration +================== + +Go :ref:`here ` for comprehensive information about settings for etcd, consul, and ZooKeeper. And for an example, see `postgres0.yml `__. + +========================= +Environment Configuration +========================= + +Go :ref:`here ` for comprehensive information about configuring(overriding) settings via environment variables. + +=================== +Replication Choices +=================== + +Patroni uses Postgres' streaming replication, which is asynchronous by default. Patroni's asynchronous replication configuration allows for ``maximum_lag_on_failover`` settings. This setting ensures failover will not occur if a follower is more than a certain number of bytes behind the leader. This setting should be increased or decreased based on business requirements. It's also possible to use synchronous replication for better durability guarantees. See :ref:`replication modes documentation ` for details. + +====================================== +Applications Should Not Use Superusers +====================================== + +When connecting from an application, always use a non-superuser. Patroni requires access to the database to function properly. By using a superuser from an application, you can potentially use the entire connection pool, including the connections reserved for superusers, with the ``superuser_reserved_connections`` setting. If Patroni cannot access the Primary because the connection pool is full, behavior will be undesirable. + +.. |Build Status| image:: https://travis-ci.org/zalando/patroni.svg?branch=master + :target: https://travis-ci.org/zalando/patroni +.. |Coverage Status| image:: https://coveralls.io/repos/zalando/patroni/badge.svg?branch=master + :target: https://coveralls.io/r/zalando/patroni?branch=master diff --git a/docs/SETTINGS.rst b/docs/SETTINGS.rst index 0632bd21..8911df57 100644 --- a/docs/SETTINGS.rst +++ b/docs/SETTINGS.rst @@ -1,3 +1,5 @@ +.. _settings: + =========================== YAML Configuration Settings =========================== @@ -60,6 +62,8 @@ Exhibitor - **poll\_interval**: how often the list of ZooKeeper and Exhibitor nodes should be updated from Exhibitor - **port**: Exhibitor port. +.. _postgresql_settings: + PostgreSQL ---------- - **authentication**: diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 00000000..cb3ad729 --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# +# Patroni documentation build configuration file, created by +# sphinx-quickstart on Mon Dec 19 16:54:09 2016. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = ['sphinx.ext.intersphinx', + 'sphinx.ext.todo', + 'sphinx.ext.mathjax', + 'sphinx.ext.ifconfig', + 'sphinx.ext.viewcode'] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = 'Patroni' +copyright = '2016, Zalando SE' +author = 'Zalando SE' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = '1.2' +# The full version, including alpha/beta/rc tags. +release = '1.2.2' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = True + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# + +on_rtd = os.environ.get('READTHEDOCS', None) == 'True' +if not on_rtd: # only import and set the theme if we're building docs locally + import sphinx_rtd_theme + html_theme = 'sphinx_rtd_theme' + html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'Patronidoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, 'Patroni.tex', 'Patroni Documentation', + 'Zalando SE', 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, 'patroni', 'Patroni Documentation', + [author], 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, 'Patroni', 'Patroni Documentation', + author, 'Patroni', 'One line description of project.', + 'Miscellaneous'), +] + + + +# -- Options for Epub output ---------------------------------------------- + +# Bibliographic Dublin Core info. +epub_title = project +epub_author = author +epub_publisher = author +epub_copyright = copyright + +# The unique identifier of the text. This can be a ISBN number +# or the project homepage. +# +# epub_identifier = '' + +# A unique identification for the text. +# +# epub_uid = '' + +# A list of files that should not be packed into the epub file. +epub_exclude_files = ['search.html'] + + + +# Example configuration for intersphinx: refer to the Python standard library. +intersphinx_mapping = {'https://docs.python.org/': None} diff --git a/docs/dynamic_configuration.rst b/docs/dynamic_configuration.rst index ce773b60..07cdb981 100644 --- a/docs/dynamic_configuration.rst +++ b/docs/dynamic_configuration.rst @@ -1,3 +1,5 @@ +.. _dynamic_configuration: + Patroni configuration ===================== @@ -10,11 +12,11 @@ Patroni configuration is stored in the DCS (Distributed Configuration Store). Th have changed), a special flag, ``pending_restart`` indicating this, is set in the members.data JSON. Additionally, the node status also indicates this, by showing ``"restart_pending": true``. -- Local `configuration `__ (patroni.yml). +- Local :ref:`configuration ` (patroni.yml). These options are defined in the configuration file and take precedence over dynamic configuration. patroni.yml could be changed and reload in runtime (without restart of Patroni) by sending SIGHUP to the Patroni process or by performing ``POST /reload`` REST-API request. -- Environment `configuration `__ . +- Environment :ref:`configuration ` . It is possible to set/override some of the "Local" configuration parameters with environment variables. Environment configuration is very useful when you are running in a dynamic environment and you don't know some of the parameters in advance (for example it's not possible to know you external IP address when you are running inside ``docker``). diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 00000000..6764b30d --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,36 @@ +.. Patroni documentation master file, created by + sphinx-quickstart on Mon Dec 19 16:54:09 2016. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Introduction +============ + +Patroni is a template for you to create your own customized, high-availability solution using Python and - for maximum accessibility - a distributed configuration store like `ZooKeeper `__, `etcd `__ or `Consul `__. Database engineers, DBAs, DevOps engineers, and SREs who are looking to quickly deploy HA PostgreSQL in the datacenter-or anywhere else-will hopefully find it useful. + +We call Patroni a "template" because it is far from being a one-size-fits-all or plug-and-play replication system. It will have its own caveats. Use wisely. There are many ways to run high availability with PostgreSQL; for a list, see the `PostgreSQL Documentation `__. + +**Note to Kubernetes users**: We're currently developing Patroni to be as useful as possible for teams running Kubernetes on top of Google Compute Engine; Patroni can be the HA solution for Postgres in such an environment. To this end, we've created a `Helm Chart `__ that enables you to deploy a five-node Patroni cluster using a Kubernetes PetSet. + + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + README + dynamic_configuration + ENVIRONMENT + SETTINGS + replication_modes + pause + releases + CONTRIBUTING + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` + + diff --git a/docs/pause.rst b/docs/pause.rst index fb049347..325505e6 100644 --- a/docs/pause.rst +++ b/docs/pause.rst @@ -1,3 +1,5 @@ +.. _pause: + Pause/Resume mode for the cluster ================================= diff --git a/docs/releases.rst b/docs/releases.rst new file mode 100644 index 00000000..72e3c986 --- /dev/null +++ b/docs/releases.rst @@ -0,0 +1,299 @@ +.. _releases: + +Release notes +============= + +Version 1.1 +----------- + +This release improves management of Patroni cluster by bring in pause mode, improves maintenance with scheduled and conditional restarts, makes Patroni interaction with Etcd or Zookeeper more resilient and greatly enhances patronictl. + +**Upgrade notice** + +When upgrading from releases below 1.0 read about changing of credentials and configuration format at 1.0 release notes. + +**Pause mode** + +- Introduce pause mode to temporary detach Patroni from managing PostgreSQL instance (Murat Kabilov, Alexander Kukushkin, Oleksii Kliukin). + +Previously, one had to send SIGKILL signal to Patroni to stop it without terminating PostgreSQL. The new pause mode detaches Patroni from PostgreSQL cluster-wide without terminating Patroni. It is similar to the maintenance mode in Pacemaker. Patroni is still responsible for updating member and leader keys in DCS, but it will not start, stop or restart PostgreSQL server in the process. There are a few exceptions, for instance, manual failovers, reinitializes and restarts are still allowed. You can read :ref:`a detailed description of this feature `. + +In addition, patronictl supports new ``pause`` and ``resume`` commands to toggle the pause mode. + +**Scheduled and conditional restarts** + +- Add conditions to the restart API command (Oleksii) + +This change enhances Patroni restarts by adding a couple of conditions that can be verified in order to do the restart. Among the conditions are restarting when PostgreSQL role is either a master or a replica, checking the PostgreSQL version number or restarting only when restart is necessary in order to apply configuration changes. + +- Add scheduled restarts (Oleksii) + +It is now possible to schedule a restart in the future. Only one scheduled restart per node is supported. It is possible to clear the scheduled restart if it is not needed anymore. A combination of scheduled and conditional restarts is supported, making it possible, for instance, to scheduled minor PostgreSQL upgrades in the night, restarting only the instances that are running the outdated minor version without adding postgres-specific logic to administration scripts. + +- Add support for conditional and scheduled restarts to patronictl (Murat). + +patronictl restart supports several new options. There is also patronictl flush command to clean the scheduled actions. + +**Robust DCS interaction** + +- Set Kazoo timeouts depending on the loop_wait (Alexander) + +Originally, ping_timeout and connect_timeout values were calculated from the negotiated session timeout. Patroni loop_wait was not taken into account. As +a result, a single retry could take more time than the session timeout, forcing Patroni to release the lock and demote. + +This change set ping and connect timeout to half of the value of loop_wait, speeding up detection of connection issues and leaving enough time to retry the connection attempt before loosing the lock. + +- Update Etcd topology only after original request succeed (Alexander) + +Postpone updating the Etcd topology known to the client until after the original request. When retrieving the cluster topology, implement the retry timeouts depending on the known number of nodes in the Etcd cluster. This makes our client prefer to get the results of the request to having the up-to-date list of nodes. + +Both changes make Patroni connections to DCS more robust in the face of network issues. + +**Patronictl, monitoring and configuration** + +- Return information about streaming replicas via the API (Feike Steenbergen) + +Previously, there was no reliable way to query Patroni about PostgreSQL instances that fail to stream changes (for instance, due to connection issues). This change exposes the contents of pg_stat_replication via the /patroni endpoint. + +- Add patronictl scaffold command (Oleksii) + +Add a command to create cluster structure in Etcd. The cluster is created with user-specified sysid and leader, and both leader and member keys are made persistent. This command is useful to create so-called master-less configurations, where Patroni cluster consisting of only replicas replicate from the external master node that is unaware of Patroni. Subsequently, one +may remove the leader key, promoting one of the Patroni nodes and replacing +the original master with the Patroni-based HA cluster. + +- Add configuration option ``bin_dir`` to locate PostgreSQL binaries (Ants Aasma) + +It is useful to be able to specify the location of PostgreSQL binaries explicitly when Linux distros that support installing multiple PostgreSQL versions at the same time. + +- Allow configuration file path to be overridden using ``custom_conf`` of (Alejandro Martínez) + +Allows for custom configuration file paths, which will be unmanaged by Patroni, :ref:`details `. + +**Bug fixes and code improvements** + +- Make Patroni compatible with new version schema in PostgreSQL 10 and above (Feike) + +Make sure that Patroni understand 2-digits version numbers when doing conditional restarts based on the PostgreSQL version. + +- Use pkgutil to find DCS modules (Alexander) + +Use the dedicated python module instead of traversing directories manually in order to find DCS modules. + +- Always call on_start callback when starting Patroni (Alexander) + +Previously, Patroni did not call any callbacks when attaching to the already running node with the correct role. Since callbacks are often used to route +client connections that could result in the failure to register the running +node in the connection routing scheme. With this fix, Patroni calls on_start +callback even when attaching to the already running node. + +- Do not drop active replication slots (Murat, Oleksii) + +Avoid dropping active physical replication slots on master. PostgreSQL cannot +drop such slots anyway. This change makes possible to run non-Patroni managed +replicas/consumers on the master. + +- Close Patroni connections during start of the PostgreSQL instance (Alexander) + +Forces Patroni to close all former connections when PostgreSQL node is started. Avoids the trap of reusing former connections if postmaster was killed with SIGKILL. + +- Replace invalid characters when constructing slot names from member names (Ants) + +Make sure that standby names that do not comply with the slot naming rules don't cause the slot creation and standby startup to fail. Replace the dashes in the slot names with underscores and all other characters not allowed in slot names with their unicode codepoints. + +Version 1.0 +----------- + +This release introduces the global dynamic configuration that allows dynamic changes of the PostgreSQL and Patroni configuration parameters for the entire HA cluster. It also delivers numerous bugfixes. + +**Upgrade notice** + +When upgrading from v0.90 or below, always upgrade all replicas before the master. Since we don't store replication credentials in DCS anymore, an old replica won't be able to connect to the new master. + +**Dynamic Configuration** + +- Implement the dynamic global configuration (Alexander Kukushkin) + +Introduce new REST API endpoint /config to provide PostgreSQL and Patroni configuration parameters that should be set globally for the entire HA cluster (master and all the replicas). Those parameters are set in DCS and in many cases can be applied without disrupting PostgreSQL or Patroni. Patroni sets a special flag called "pending restart" visible via the API when some of the values require the PostgreSQL restart. In that case, restart should be issued manually via the API. + +Patroni SIGHUP or POST to /reload will make it re-read the configuration file. + +See the :ref:`dynamic configuration ` for the details on which parameters can be changed and the order of processing difference configuration sources. + +The configuration file format *has changed* since the v0.90. Patroni is still compatible with the old configuration files, but in order to take advantage of the bootstrap parameters one needs to change it. Users are encourage to update them by referring to the :ref:`dynamic configuraton documentation page `. + +**More flexible configuration*** + +- Make postgresql configuration and database name Patroni connects to configurable (Misja Hoebe) + +Introduce `database` and `config_base_name` configuration parameters. Among others, it makes possible to run Patroni with PipelineDB and other PostgreSQL forks. + +- Implement possibility to configure some Patroni configuration parameters via environment (Alexander) + +Those include the scope, the node name and the namespace, as well as the secrets and makes it easier to run Patroni in a dynamic environment, i.e. Kubernetes Please, refer to the :ref:`supported environment variables ` for further details. + +- Update the built-in Patroni docker container to take advantage of environment-based configuration (Feike Steenbergen). + +- Add Zookeeper support to Patroni docker image (Alexander) + +- Split the Zookeeper and Exhibitor configuration options (Alexander) + +- Make patronictl reuse the code from Patroni to read configuration (Alexander) + +This allows patronictl to take advantage of environment-based configuration. + +- Set application name to node name in primary_conninfo (Alexander) + +This simplifies identification and configuration of synchronous replication for a given node. + +**Stability, security and usability improvements** + +- Reset sysid and do not call pg_controldata when restore of backup in progress (Alexander) + +This change reduces the amount of noise generated by Patroni API health checks during the lengthy initialization of this node from the backup. + +- Fix a bunch of pg_rewind corner-cases (Alexander) + +Avoid running pg_rewind if the source cluster is not the master. + +In addition, avoid removing the data directory on an unsuccessful rewind, unless the new parameter *remove_data_directory_on_rewind_failure* is set to true. By default it is false. + +- Remove passwords from the replication connection string in DCS (Alexander) + +Previously, Patroni always used the replication credentials from the Postgres URL in DCS. That is now changed to take the credentials from the patroni configuration. The secrets (replication username and password) and no longer exposed in DCS. + +- Fix the asynchronous machinery around the demote call (Alexander) + +Demote now runs totally asynchronously without blocking the DCS interactions. + +- Make patronictl always send the authorization header if it is configured (Alexander) + +This allows patronictl to issue "protected" requests, i.e. restart or reinitialize, when Patroni is configured to require authorization on those. + +- Handle the SystemExit exception correctly (Alexander) + +Avoids the issues of Patroni not stopping properly when receiving the SIGTERM + +- Sample haproxy templates for confd (Alexander) + +Generates and dynamically changes haproxy configuration from the patroni state in the DCS using confide + +- Improve and restructure the documentation to make it more friendly to the new users (Lauri Apple) + +- API must report role=master during pg_ctl stop (Alexander) + +Makes the callback calls more reliable, particularly in the cluster stop case. In addition, introduce the `pg_ctl_timeout` option to set the timeout for the start, stop and restart calls via the `pg_ctl`. + +- Fix the retry logic in etcd (Alexander) + +Make retries more predictable and robust. + +- Make Zookeeper code more resilient against short network hiccups (Alexander) + +Reduce the connection timeouts to make Zookeeper connection attempts more frequent. + +Version 0.90 +------------ + +This releases adds support for Consul, includes a new *noloadbalance* tag, changes the behavior of the *clonefrom* tag, improves *pg_rewind* handling and improves *patronictl* control program. + +**Consul support** + +- Implement Consul support (Alexander Kukushkin) + +Patroni runs against Consul, in addition to Etcd and Zookeeper. the connection parameters can be configured in the YAML file. + +**New and improved tags** + +- Implement *noloadbalance* tag (Alexander) + +This tag makes Patroni always return that the replica is not available to the load balancer. + +- Change the implementation of the *clonefrom* tag (Alexander) + +Previously, a node name had to be supplied to the *clonefrom*, forcing a tagged replica to clone from the specific node. The new implementation makes *clonefrom* a boolean tag: if it is set to true, the replica becomes a candidate for other replicas to clone from it. When multiple candidates are present, the replicas picks one randomly. + +**Stability and security improvements** + +- Numerous reliability improvements (Alexander) + +Removes some spurious error messages, improves the stability of the failover, addresses some corner cases with reading data from DCS, shutdown, demote and reattaching of the former leader. + +- Improve systems script to avoid killing Patroni children on stop (Jan Keirse, Alexander Kukushkin) + +Previously, when stopping Patroni, *systemd* also sent a signal to PostgreSQL. Since Patroni also tried to stop PostgreSQL by itself, it resulted in sending to different shutdown requests (the smart shutdown, followed by the fast shutdown). That resulted in replicas disconnecting too early and a former master not being able to rejoin after demote. Fix by Jan with prior research by Alexander. + +- Eliminate some cases where the former master was unable to call pg_rewind before rejoining as a replica (Oleksii Kliukin) + +Previously, we only called *pg_rewind* if the former master had crashed. Change this to always run pg_rewind for the former master as long as pg_rewind is present in the system. This fixes the case when the master is shut down before the replicas managed to get the latest changes (i.e. during the "smart" shutdown). + +- Numerous improvements to unit- and acceptance- tests, in particular, enable support for Zookeeper and Consul (Alexander). + +- Make Travis CI faster and implement support for running tests against Zookeeper (Exhibitor) and Consul (Alexander) + +Both unit and acceptance tests run automatically against Etcd, Zookeeper and Consul on each commit or pull-request. + +- Clear environment variables before calling PostgreSQL commands from Patroni (Feike Steenbergen) + +This prevents a possibility of reading system environment variables by connecting to the PostgreSQL cluster managed by Patroni. + +**Configuration and control changes** + +- Unify patronictl and Patroni configuration (Feike) + +patronictl can use the same configuration file as Patroni itself. + +- Enable Patroni to read the configuration from the environment variables (Oleksii) + +This simplifies generating configuration for Patroni automatically, or merging a single configuration from different sources. + +- Include database system identifier in the information returned by the API (Feike) + +- Implement *delete_cluster* for all available DCSs (Alexander) + +Enables support for DCSs other than Etcd in patronictl. + + +Version 0.80 +------------ + +This release adds support for *cascading replication* and simplifies Patroni management by providing *scheduled failovers*. One may use older versions of Patroni (in particular, 0.78) combined with this one in order to migrate to the new release. Note that the scheduled failover and cascading replication related features will only work with Patroni 0.80 and above. + +**Cascading replication** + + - Add support for the *replicatefrom* and *clonefrom* tags for the patroni node (Oleksii Kliukin). + + The tag *replicatefrom* allows a replica to use an arbitrary node a source, not necessary the master. The *clonefrom* does the same for the initial backup. Together, they enable Patroni to fully support cascading replication. + +- Add support for running replication methods to initialize the replica even without a running replication connection (Oleksii). + + This is useful in order to create replicas from the snapshots stored on S3 or FTP. A replication method that does not require a running replication connection should supply *no_master: true* in the yaml configuration. Those scripts will still be called in order if the replication connection is present. + +**Patronictl, API and DCS improvements** + +- Implement scheduled failovers (Feike Steenbergen). + + Failovers can be scheduled to happen at a certain time in the future, using either patronictl, or API calls. + +- Add support for *dbuser* and *password* parameters in patronictl (Feike). + +- Add PostgreSQL version to the health check output (Feike). + +- Improve Zookeeper support in patronictl (Oleksandr Shulgin) + +- Migrate to python-etcd 0.43 (Alexander Kukushkin) + +**Configuration** + +- Add a sample systems configuration script for Patroni (Jan Keirse). + +- Fix the problem of Patroni ignoring the superuser name specified in the configuration file for DB connections (Alexander). + +- Fix the handling of CTRL-C by creating a separate session ID and process group for the postmaster launched by Patroni (Alexander). + +**Tests** + +- Add acceptance tests with *behave* in order to check real-world scenarios of running Patroni (Alexander, Oleksii). + + The tests can be launched manually using the *behave* command. They are also launched automatically for pull requests and after commits. + +Releases notes for some older versions can be found on `project's github page `__. \ No newline at end of file diff --git a/docs/replication_modes.rst b/docs/replication_modes.rst index 80a47805..e7d48db8 100644 --- a/docs/replication_modes.rst +++ b/docs/replication_modes.rst @@ -1,3 +1,5 @@ +.. _replication_modes: + ================= Replication modes =================