1<chapter id="repmgrd-operation" xreflabel="repmgrd operation"> 2 <title>repmgrd operation</title> 3 4 <indexterm> 5 <primary>repmgrd</primary> 6 <secondary>operation</secondary> 7 </indexterm> 8 9 <sect1 id="repmgrd-pausing" xreflabel="pausing the repmgrd service"> 10 11 <title>Pausing the repmgrd service</title> 12 13 <indexterm> 14 <primary>repmgrd</primary> 15 <secondary>pausing</secondary> 16 </indexterm> 17 18 <indexterm> 19 <primary>pausing repmgrd</primary> 20 </indexterm> 21 22 23 <para> 24 In normal operation, &repmgrd; monitors the state of the 25 PostgreSQL node it is running on, and will take appropriate action if problems 26 are detected, e.g. (if so configured) promote the node to primary, if the existing 27 primary has been determined as failed. 28 </para> 29 30 <para> 31 However, &repmgrd; is unable to distinguish between 32 planned outages (such as performing a <link linkend="performing-switchover">switchover</link> 33 or installing PostgreSQL maintenance released), and an actual server outage. In versions prior to 34 &repmgr; 4.2 it was necessary to stop &repmgrd; on all nodes (or at least 35 on all nodes where &repmgrd; is 36 <link linkend="repmgrd-automatic-failover">configured for automatic failover</link>) 37 to prevent &repmgrd; from making unintentional changes to the 38 replication cluster. 39 </para> 40 41 <para> 42 From <link linkend="release-4.2">&repmgr; 4.2</link>, &repmgrd; 43 can now be "paused", i.e. instructed not to take any action such as performing a failover. 44 This can be done from any node in the cluster, removing the need to stop/restart 45 each &repmgrd; individually. 46 </para> 47 48 <note> 49 <para> 50 For major PostgreSQL upgrades, e.g. from PostgreSQL 11 to PostgreSQL 12, 51 &repmgrd; should be shut down completely and only started up 52 once the &repmgr; packages for the new PostgreSQL major version have been installed. 53 </para> 54 </note> 55 56 <sect2 id="repmgrd-pausing-prerequisites"> 57 <title>Prerequisites for pausing &repmgrd;</title> 58 <para> 59 In order to be able to pause/unpause &repmgrd;, following 60 prerequisites must be met: 61 <itemizedlist spacing="compact" mark="bullet"> 62 63 <listitem> 64 <simpara><link linkend="release-4.2">&repmgr; 4.2</link> or later must be installed on all nodes.</simpara> 65 </listitem> 66 67 <listitem> 68 <simpara>The same major &repmgr; version (e.g. 4.2) must be installed on all nodes (and preferably the same minor version).</simpara> 69 </listitem> 70 71 <listitem> 72 <simpara> 73 PostgreSQL on all nodes must be accessible from the node where the 74 <literal>pause</literal>/<literal>unpause</literal> operation is executed, using the 75 <varname>conninfo</varname> string shown by <link linkend="repmgr-cluster-show"><command>repmgr cluster show</command></link>. 76 </simpara> 77 </listitem> 78 </itemizedlist> 79 </para> 80 <note> 81 <para> 82 These conditions are required for normal &repmgr; operation in any case. 83 </para> 84 </note> 85 86 </sect2> 87 88 <sect2 id="repmgrd-pausing-execution"> 89 <title>Pausing/unpausing &repmgrd;</title> 90 <para> 91 To pause &repmgrd;, execute <link linkend="repmgr-service-pause"><command>repmgr service pause</command></link> 92 (&repmgr; 4.2 - 4.4: <link linkend="repmgr-service-pause"><command>repmgr daemon pause</command></link>), 93 e.g.: 94 <programlisting> 95$ repmgr -f /etc/repmgr.conf service pause 96NOTICE: node 1 (node1) paused 97NOTICE: node 2 (node2) paused 98NOTICE: node 3 (node3) paused</programlisting> 99 </para> 100 <para> 101 The state of &repmgrd; on each node can be checked with 102 <link linkend="repmgr-service-status"><command>repmgr service status</command></link> 103 (&repmgr; 4.2 - 4.4: <link linkend="repmgr-service-status"><command>repmgr daemon status</command></link>), 104 e.g.: 105 <programlisting>$ repmgr -f /etc/repmgr.conf service status 106 ID | Name | Role | Status | repmgrd | PID | Paused? 107----+-------+---------+---------+---------+------+--------- 108 1 | node1 | primary | running | running | 7851 | yes 109 2 | node2 | standby | running | running | 7889 | yes 110 3 | node3 | standby | running | running | 7918 | yes</programlisting> 111 </para> 112 113 <note> 114 <para> 115 If executing a switchover with <link linkend="repmgr-standby-switchover"><command>repmgr standby switchover</command></link>, 116 &repmgr; will automatically pause/unpause the &repmgrd; service as part of the switchover process. 117 </para> 118 </note> 119 120 <para> 121 If the primary (in this example, <literal>node1</literal>) is stopped, &repmgrd; 122 running on one of the standbys (here: <literal>node2</literal>) will react like this: 123 <programlisting> 124[2019-08-28 12:22:21] [WARNING] unable to connect to upstream node "node1" (node ID: 1) 125[2019-08-28 12:22:21] [INFO] checking state of node 1, 1 of 5 attempts 126[2019-08-28 12:22:21] [INFO] sleeping 1 seconds until next reconnection attempt 127... 128[2019-08-28 12:22:24] [INFO] sleeping 1 seconds until next reconnection attempt 129[2019-08-28 12:22:25] [INFO] checking state of node 1, 5 of 5 attempts 130[2019-08-28 12:22:25] [WARNING] unable to reconnect to node 1 after 5 attempts 131[2019-08-28 12:22:25] [NOTICE] node is paused 132[2019-08-28 12:22:33] [INFO] node "node2" (ID: 2) monitoring upstream node "node1" (node ID: 1) in degraded state 133[2019-08-28 12:22:33] [DETAIL] repmgrd paused by administrator 134[2019-08-28 12:22:33] [HINT] execute "repmgr service unpause" to resume normal failover mode</programlisting> 135 </para> 136 <para> 137 If the primary becomes available again (e.g. following a software upgrade), &repmgrd; 138 will automatically reconnect, e.g.: 139 <programlisting> 140[2019-08-28 12:25:41] [NOTICE] reconnected to upstream node "node1" (ID: 1) after 8 seconds, resuming monitoring</programlisting> 141 </para> 142 143 <para> 144 To unpause the &repmgrd; service, execute 145 <link linkend="repmgr-service-unpause"><command>repmgr service unpause</command></link> 146 ((&repmgr; 4.2 - 4.4: <link linkend="repmgr-service-unpause"><command>repmgr daemon unpause</command></link>), 147 e.g.: 148 <programlisting> 149$ repmgr -f /etc/repmgr.conf service unpause 150NOTICE: node 1 (node1) unpaused 151NOTICE: node 2 (node2) unpaused 152NOTICE: node 3 (node3) unpaused</programlisting> 153 </para> 154 155 <note> 156 <para> 157 If the previous primary is no longer accessible when &repmgrd; 158 is unpaused, no failover action will be taken. Instead, a new primary must be manually promoted using 159 <link linkend="repmgr-standby-promote"><command>repmgr standby promote</command></link>, 160 and any standbys attached to the new primary with 161 <link linkend="repmgr-standby-follow"><command>repmgr standby follow</command></link>. 162 </para> 163 <para> 164 This is to prevent execution of <link linkend="repmgr-service-unpause"><command>repmgr service unpause</command></link> 165 resulting in the automatic promotion of a new primary, which may be a problem particularly 166 in larger clusters, where &repmgrd; could select a different promotion 167 candidate to the one intended by the administrator. 168 </para> 169 </note> 170 </sect2> 171 <sect2 id="repmgrd-pausing-details"> 172 <title>Details on the &repmgrd; pausing mechanism</title> 173 174 <para> 175 The pause state of each node will be stored over a PostgreSQL restart. 176 </para> 177 178 <para> 179 <link linkend="repmgr-service-pause"><command>repmgr service pause</command></link> and 180 <link linkend="repmgr-service-unpause"><command>repmgr service unpause</command></link> can be 181 executed even if &repmgrd; is not running; in this case, 182 &repmgrd; will start up in whichever pause state has been set. 183 </para> 184 <note> 185 <para> 186 <link linkend="repmgr-service-pause"><command>repmgr service pause</command></link> and 187 <link linkend="repmgr-service-unpause"><command>repmgr service unpause</command></link> 188 <emphasis>do not</emphasis> start/stop &repmgrd;. 189 </para> 190 <para> 191 The commands <link linkend="repmgr-daemon-start"><command>repmgr daemon start</command></link> 192 and <link linkend="repmgr-daemon-stop"><command>repmgr daemon stop</command></link> 193 (<link linkend="repmgrd-service-configuration">if correctly configured</link>) can be used to start/stop 194 &repmgrd; on individual nodes. 195 </para> 196 </note> 197 </sect2> 198 </sect1> 199 200 <sect1 id="repmgrd-wal-replay-pause"> 201 <title>repmgrd and paused WAL replay</title> 202 203 <indexterm> 204 <primary>repmgrd</primary> 205 <secondary>paused WAL replay</secondary> 206 </indexterm> 207 208 <para> 209 If WAL replay has been paused (using <command>pg_wal_replay_pause()</command>, 210 on PostgreSQL 9.6 and earlier <command>pg_xlog_replay_pause()</command>), 211 in a failover situation &repmgrd; will 212 automatically resume WAL replay. 213 </para> 214 <para> 215 This is because if WAL replay is paused, but WAL is pending replay, 216 PostgreSQL cannot be promoted until WAL replay is resumed. 217 </para> 218 <note> 219 <para> 220 <command><link linkend="repmgr-standby-promote">repmgr standby promote</link></command> 221 will refuse to promote a node in this state, as the PostgreSQL 222 <command>promote</command> command will not be acted on until 223 WAL replay is resumed, leaving the cluster in a potentially 224 unstable state. In this case it is up to the user to 225 decide whether to resume WAL replay. 226 </para> 227 </note> 228 </sect1> 229 230<sect1 id="repmgrd-degraded-monitoring" xreflabel="repmgrd degraded monitoring"> 231 <title>"degraded monitoring" mode</title> 232 233 <indexterm> 234 <primary>repmgrd</primary> 235 <secondary>degraded monitoring</secondary> 236 </indexterm> 237 238 <indexterm> 239 <primary>degraded monitoring</primary> 240 </indexterm> 241 242 <para> 243 In certain circumstances, &repmgrd; is not able to fulfill its primary mission 244 of monitoring the node's upstream server. In these cases it enters "degraded monitoring" 245 mode, where &repmgrd; remains active but is waiting for the situation 246 to be resolved. 247 </para> 248 <para> 249 Situations where this happens are: 250 <itemizedlist spacing="compact" mark="bullet"> 251 252 <listitem> 253 <simpara>a failover situation has occurred, no nodes in the primary node's location are visible</simpara> 254 </listitem> 255 256 <listitem> 257 <simpara>a failover situation has occurred, but no promotion candidate is available</simpara> 258 </listitem> 259 260 <listitem> 261 <simpara>a failover situation has occurred, but the promotion candidate could not be promoted</simpara> 262 </listitem> 263 264 <listitem> 265 <simpara>a failover situation has occurred, but the node was unable to follow the new primary</simpara> 266 </listitem> 267 268 <listitem> 269 <simpara>a failover situation has occurred, but no primary has become available</simpara> 270 </listitem> 271 272 <listitem> 273 <simpara>a failover situation has occurred, but automatic failover is not enabled for the node</simpara> 274 </listitem> 275 276 <listitem> 277 <simpara>repmgrd is monitoring the primary node, but it is not available (and no other node has been promoted as primary)</simpara> 278 </listitem> 279 </itemizedlist> 280 </para> 281 282 <para> 283 Example output in a situation where there is only one standby with <literal>failover=manual</literal>, 284 and the primary node is unavailable (but is later restarted): 285 <programlisting> 286 [2017-08-29 10:59:19] [INFO] node "node2" (ID: 2) monitoring upstream node "node1" (ID: 1) in normal state (automatic failover disabled) 287 [2017-08-29 10:59:33] [WARNING] unable to connect to upstream node "node1" (ID: 1) 288 [2017-08-29 10:59:33] [INFO] checking state of node 1, 1 of 5 attempts 289 [2017-08-29 10:59:33] [INFO] sleeping 1 seconds until next reconnection attempt 290 (...) 291 [2017-08-29 10:59:37] [INFO] checking state of node 1, 5 of 5 attempts 292 [2017-08-29 10:59:37] [WARNING] unable to reconnect to node 1 after 5 attempts 293 [2017-08-29 10:59:37] [NOTICE] this node is not configured for automatic failover so will not be considered as promotion candidate 294 [2017-08-29 10:59:37] [NOTICE] no other nodes are available as promotion candidate 295 [2017-08-29 10:59:37] [HINT] use "repmgr standby promote" to manually promote this node 296 [2017-08-29 10:59:37] [INFO] node "node2" (ID: 2) monitoring upstream node "node1" (ID: 1) in degraded state (automatic failover disabled) 297 [2017-08-29 10:59:53] [INFO] node "node2" (ID: 2) monitoring upstream node "node1" (ID: 1) in degraded state (automatic failover disabled) 298 [2017-08-29 11:00:45] [NOTICE] reconnected to upstream node "node1" (ID: 1) after 68 seconds, resuming monitoring 299 [2017-08-29 11:00:57] [INFO] node "node2" (ID: 2) monitoring upstream node "node1" (ID: 1) in normal state (automatic failover disabled)</programlisting> 300 301 </para> 302 <para> 303 By default, <literal>repmgrd</literal> will continue in degraded monitoring mode indefinitely. 304 However a timeout (in seconds) can be set with <varname>degraded_monitoring_timeout</varname>, 305 after which &repmgrd; will terminate. 306 </para> 307 308 <note> 309 <para> 310 If &repmgrd; is monitoring a primary mode which has been stopped 311 and manually restarted as a standby attached to a new primary, it will automatically detect 312 the status change and update the node record to reflect the node's new status 313 as an active standby. It will then resume monitoring the node as a standby. 314 </para> 315 </note> 316</sect1> 317 318 319<sect1 id="repmgrd-monitoring" xreflabel="Storing monitoring data"> 320 <title>Storing monitoring data</title> 321 <indexterm> 322 <primary>repmgrd</primary> 323 <secondary>monitoring</secondary> 324 </indexterm> 325 <indexterm> 326 <primary>monitoring</primary> 327 <secondary>with repmgrd</secondary> 328 </indexterm> 329 330 <para> 331 When &repmgrd; is running with the option <literal>monitoring_history=true</literal>, 332 it will constantly write standby node status information to the 333 <varname>monitoring_history</varname> table, providing a near-real time 334 overview of replication status on all nodes 335 in the cluster. 336 </para> 337 <para> 338 The view <literal>replication_status</literal> shows the most recent state 339 for each node, e.g.: 340 <programlisting> 341 repmgr=# select * from repmgr.replication_status; 342 -[ RECORD 1 ]-------------+------------------------------ 343 primary_node_id | 1 344 standby_node_id | 2 345 standby_name | node2 346 node_type | standby 347 active | t 348 last_monitor_time | 2017-08-24 16:28:41.260478+09 349 last_wal_primary_location | 0/6D57A00 350 last_wal_standby_location | 0/5000000 351 replication_lag | 29 MB 352 replication_time_lag | 00:00:11.736163 353 apply_lag | 15 MB 354 communication_time_lag | 00:00:01.365643</programlisting> 355 </para> 356 <para> 357 The interval in which monitoring history is written is controlled by the 358 configuration parameter <varname>monitor_interval_secs</varname>; 359 default is 2. 360 </para> 361 <para> 362 As this can generate a large amount of monitoring data in the table 363 <literal>repmgr.monitoring_history</literal>. it's advisable to regularly 364 purge historical data using the <xref linkend="repmgr-cluster-cleanup"/> 365 command; use the <literal>-k/--keep-history</literal> option to 366 specify how many day's worth of data should be retained. 367 </para> 368 <para> 369 It's possible to use &repmgrd; to run in monitoring 370 mode only (without automatic failover capability) for some or all 371 nodes by setting <literal>failover=manual</literal> in the node's 372 <filename>repmgr.conf</filename> file. In the event of the node's upstream failing, 373 no failover action will be taken and the node will require manual intervention to 374 be reattached to replication. If this occurs, an 375 <link linkend="event-notifications">event notification</link> 376 <varname>standby_disconnect_manual</varname> will be created. 377 </para> 378 <para> 379 Note that when a standby node is not streaming directly from its upstream 380 node, e.g. recovering WAL from an archive, <varname>apply_lag</varname> will always appear as 381 <literal>0 bytes</literal>. 382 </para> 383 <tip> 384 <para> 385 If monitoring history is enabled, the contents of the <literal>repmgr.monitoring_history</literal> 386 table will be replicated to attached standbys. This means there will be a small but 387 constant stream of replication activity which may not be desirable. To prevent 388 this, convert the table to an <literal>UNLOGGED</literal> one with: 389 <programlisting> 390 ALTER TABLE repmgr.monitoring_history SET UNLOGGED;</programlisting> 391 </para> 392 <para> 393 This will however mean that monitoring history will not be available on 394 another node following a failover, and the view <literal>repmgr.replication_status</literal> 395 will not work on standbys. 396 </para> 397 </tip> 398</sect1> 399 400 401</chapter> 402