The OCF RA shared code directory

If an RA is too big to be comfortably maintained, split it into
several source files. Obviosuly, if two or more RAs share some
code, move that code out to a file which can be shared.

These files will be installed in $OCF_ROOT/lib/heartbeat with
permissions 644.

Naming practice

Use names such as <RA>.sh or <RA>-check.sh or anything-else.sh
where "anything-else" should be related to both the RA and the
code it contains. By adding extension (.sh) it is going to be
easier to notice that these files are not complete resource
agents.

For instance, oracle and oralsnr RA can both use code in
ora-common.sh.

Of course, if the RA is implemented in another programming
language, use the appropriate extension.

RA tracing

RA tracing may be turned on by setting OCF_TRACE_RA. The trace
output will be saved to OCF_TRACE_FILE, if set, or by default to

  $HA_VARLIB/trace_ra/<type>/<id>.<action>.<timestamp>

e.g. $HA_VARLIB/trace_ra/oracle/db.start.2012-11-27.08:37:08

HA_VARLIB is typically set to /var/lib/heartbeat.

OCF_TRACE_FILE can be set to a path or file descriptor:

- FD (small integer [3-9]) in that case it is up to the callers
  to capture output; the FD _must_ be open for writing

- absolute path

NB: FD 9 may be used for tracing with bash >= v4 in case
OCF_TRACE_FILE is set to a path.

Notes regarding the Galera resource agent
---

In the resource agent, the action of bootstrapping a Galera cluster is
implemented into a series of small steps, by using:

  * Two CIB attributes `last-committed` and `bootstrap` to elect a
    bootstrap node that will restart the cluster.

  * One CIB attribute `sync-needed` that will identify that joining
    nodes are in the process of synchronizing their local database
    via SST.

  * A Master/Slave pacemaker resource which helps splitting the boot
    into steps, up to a point where a galera node is available.

  * the recurring monitor action to coordinate switch from one
    state to another.

How boot works
====

There are two things to know to understand how the resource agent
restart a Galera cluster.

### Bootstrap the cluster with the right node

When synced, the nodes of a galera cluster have in common a last seqno,
which identifies the last transaction considered successful by a
majority of nodes in the cluster (think quorum).

To restart a cluster, the resource agent must ensure that it will
bootstrap the cluster from an node which is up-to-date, i.e which has
the highest seqno of all nodes.

As a result, if the resource agent cannot retrieve the seqno on all
nodes, it won't be able to safely identify a bootstrap node, and
will simply refuse to start the galera cluster.

### synchronizing nodes can be a long operation

Starting a bootstrap node is relatively fast, so it's performed
during the "promote" operation, which is a one-off, time-bounded
operation.

Subsequent nodes will need to synchronize via SST, which consists
in "pushing" an entire Galera DB from one node to another.

There is no perfect time-out, as time spent during synchronization
depends on the size of the DB. Thus, joiner nodes are started during
the "monitor" operation, which is a recurring operation that can
better track the progress of the SST.


State flow
====

General idea for starting Galera:

  * Before starting the Galera cluster each node needs to go in Slave
    state so that the agent records its last seqno into the CIB.
    __ This uses attribute last-committed __

  * When all node went in Slave, the agent can safely determine the
    last seqno and elect a bootstrap node (`detect_first_master()`).
    __ This uses attribute bootstrap __

  * The agent then sets the score of the elected bootstrap node to
    Master so that pacemaker promote it and start the first Galera
    server.

  * Once the first Master is running, the agent can start joiner
    nodes during the "monitor" operation, and starts monitoring
    their SST sync.
    __ This uses attribute sync-needed __

  * Only when SST is over on joiner nodes, the agent promotes them
    to Master. At this point, the entire Galera cluster is up.


Attribute usage and liveness
====

Here is how attributes are created on a per-node basis. If you
modify the resource agent make sure those properties still hold.

### last-committed

It is just a temporary hint for the resource agent to help
elect a bootstrap node. Once the bootstrap attribute is set on one
of the nodes, we can get rid of last-committed.

 - Used   : during Slave state to compare seqno
 - Created: before entering Slave state:
              . at startup in `galera_start()`
              . or when a Galera node is stopped in `galera_demote()`
 - Deleted: just before node starts in `galera_start_local_node()`;
            cleaned-up during `galera_demote()` and `galera_stop()`

We delete last-committed before starting Galera, to avoid race
conditions that could arise due to discrepancies between the CIB and
Galera.

### bootstrap

Attribute set on the node that is elected to bootstrap Galera.

- Used   : during promotion in `galera_start_local_node()`
- Created: at startup once all nodes have `last-committed`;
           or during monitor if all nodes have failed
- Deleted: in `galera_start_local_node()`, just after the bootstrap
           node started and is ready;
           cleaned-up during `galera_demote()` and `galera_stop()`

There cannot be more than one bootstrap node at any time, otherwise
the Galera cluster would stop replicating properly.

### sync-needed

While this attribute is set on a node, the Galera node is in JOIN
state, i.e. SST is in progress and the node cannot serve queries.

The resource agent relies on the underlying SST method to monitor
the progress of the SST. For instance, with `wsrep_sst_rsync`,
timeout would be reported by rsync, the Galera node would go in
Non-primary state, which would make `galera_monitor()` fail.

- Used   : during recurring slave monitor in `check_sync_status()`
- Created: in `galera_start_local_node()`, just after the joiner
           node started and entered the Galera cluster
- Deleted: during recurring slave monitor in `check_sync_status()`
           as soon as the Galera code reports to be SYNC-ed.

### no-grastate

If a galera node was unexpectedly killed in a middle of a replication,
InnoDB can retain the equivalent of a XA transaction in prepared state
in its redo log. If so, mysqld cannot recover state (nor last seqno)
automatically, and special recovery heuristic has to be used to
unblock the node.

This transient attribute is used to keep track of forced recoveries to
prevent bootstrapping a cluster from a recovered node when possible.

- Used   : during `detect_first_master()` to elect the bootstrap node
- Created: in `detect_last_commit()` if the node has a pending XA
           transaction to recover in the redo log
- Deleted: when a node is promoted to Master.

Setting up the MariaDB resource agent
=====================================

This resource agent requires corosync version >= 2 and mariadb version > 10.2 .

Before embarking on this quest one should read the MariaDB pages on replication
and global transaction IDs, GTID. This will greatly help in understanding what
is going on and why.

Replication: https://mariadb.com/kb/en/mariadb/setting-up-replication/
GTID: https://mariadb.com/kb/en/mariadb/gtid/
semi-sync: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/

Some reading on failures under enhanced semi-sync can be found here:
https://jira.mariadb.org/browse/MDEV-162

Part 1: MariaDB Setup
---------------------

It is best to initialize your MariaDB and do a failover before trying to use
Pacemaker to manage MariaDB. This will both verify the MariaDB configuration
and help you understand what is going on.

###Configuration Files

In your MariaDB config file for the server on node 1, place the following
entry (replacing my_database and other names as needed):
```
[mariadb]
log-bin
server_id=1
log-basename=master
binlog_do_db=my_database
```

Then for each other node create the same entry, but increment the server_id.

###Replication User

Now create the replication user (be sure to change the password!):
```
GRANT ALL PRIVILEGES ON *.* TO 'slave_user'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'slave_user'@'localhost' IDENTIFIED BY 'password';
```

The second entry may not be necessary, but simplified other steps. Change
user name and password as needed.


###Intialize from a database backup

Initialize all nodes from an existing backup, or create a backup from the
first node if needed:

On the current database:
```
mysqldump -u root --master-data --databases my_database1 my_database2 > backup.sql
```

At the top of this file is a commented out line:
SET GLOBAL gtid_slave_pos='XXXX...'

uncomment his line.

On all new nodes:
```
mysqldump -u root < backup.sql
```

###Initialize replication

Choose a node as master, in this example node1.

On all slaves, execute:
```
RESET MASTER;

CHANGE MASTER TO master_host="node1", master_port=3306, \
       master_user="slave_user", master_password="password", \
       master_use_gtid=current_pos;

SET GLOBAL rpl_semi_sync_master_enabled='ON', rpl_semi_sync_slave_enabled='ON';

START SLAVE;

SHOW SLAVE STATUS\G
```

In an ideal world this will show that replication is now fully working.

Once replication is working, verify the configuration by doing some updates
and verifying that they are replicated.

Now try changing the master. On each slave perform:
```
STOP SLAVE
```

Choose a new master, node2 in our example. On all slave nodes execute:
```
CHANGE MASTER TO  master_host="node2", master_port=3306, \
       master_user="slave_user", master_password="password", \
       master_use_gtid=current_pos;

START SLAVE;
```

And again, check that replication is working and changes are synchronized.


Part 2: Pacemaker Setup
-----------------------

This is pretty straightforward. Example is using pcs.

```
# Dump the cib
pcs cluster cib mariadb_cfg

# Create the mariadb_server resource
pcs -f mariadb_cfg resource create mariadb_server mariadb \
   binary="/usr/sbin/mysqld" \
   replication_user="slave_user" \
   replication_passwd="password" \
   node_list="node1 node2 node3" \
   op start timeout=120 interval=0 \
   op stop timeout=120 interval=0 \
   op promote timeout=120 interval=0 \
   op demote timeout=120 interval=0 \
   op monitor role=Master timeout=30 interval=10 \
   op monitor role=Slave timeout=30 interval=20 \
   op notify  timeout="60s" interval="0s"

# Create the master slave resource
pcs -f mariadb_cfg resource master msMariadb mariadb_server \
    master-max=1 master-node-max=1 clone-max=3 clone-node-max=1 notify=true

# Avoid running this on some nodes, only if needed
pcs -f mariadb_cfg constraint location msMariadb avoids \
    node4=INFINITY node5=INFINITY

# Push the cib
pcs cluster cib-push mariadb_cfg
```

You should now have a running MariaDB cluster:
```
pcs status

...
 Master/Slave Set: msMariadb [mariadb_server]
      Masters: [ node1 ]
      Slaves: [ node2 node3 ]
...
```