1 2Current cluster status: 3RemoteNode database-0: UNCLEAN (offline) 4RemoteNode database-2: UNCLEAN (offline) 5Online: [ controller-0 controller-1 controller-2 ] 6RemoteOnline: [ database-1 messaging-0 messaging-1 messaging-2 ] 7Containers: [ galera-bundle-1:galera-bundle-docker-1 rabbitmq-bundle-0:rabbitmq-bundle-docker-0 rabbitmq-bundle-1:rabbitmq-bundle-docker-1 rabbitmq-bundle-2:rabbitmq-bundle-docker-2 redis-bundle-0:redis-bundle-docker-0 redis-bundle-2:redis-bundle-docker-2 ] 8 9 database-0 (ocf::pacemaker:remote): Stopped 10 database-1 (ocf::pacemaker:remote): Started controller-2 11 database-2 (ocf::pacemaker:remote): Stopped 12 messaging-0 (ocf::pacemaker:remote): Started controller-2 13 messaging-1 (ocf::pacemaker:remote): Started controller-2 14 messaging-2 (ocf::pacemaker:remote): Started controller-2 15 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] 16 rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started messaging-0 17 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started messaging-1 18 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started messaging-2 19 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] 20 galera-bundle-0 (ocf::heartbeat:galera): FAILED Master database-0 (UNCLEAN) 21 galera-bundle-1 (ocf::heartbeat:galera): Master database-1 22 galera-bundle-2 (ocf::heartbeat:galera): FAILED Master database-2 (UNCLEAN) 23 Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest] 24 redis-bundle-0 (ocf::heartbeat:redis): Slave controller-0 25 redis-bundle-1 (ocf::heartbeat:redis): Stopped 26 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 27 ip-192.168.24.11 (ocf::heartbeat:IPaddr2): Stopped 28 ip-10.0.0.104 (ocf::heartbeat:IPaddr2): Stopped 29 ip-172.17.1.19 (ocf::heartbeat:IPaddr2): Started controller-2 30 ip-172.17.1.11 (ocf::heartbeat:IPaddr2): Stopped 31 ip-172.17.3.13 (ocf::heartbeat:IPaddr2): Stopped 32 ip-172.17.4.19 (ocf::heartbeat:IPaddr2): Started controller-2 33 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] 34 haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 35 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Stopped 36 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-2 37 openstack-cinder-volume (systemd:openstack-cinder-volume): Stopped 38 stonith-fence_ipmilan-525400244e09 (stonith:fence_ipmilan): Started controller-2 39 stonith-fence_ipmilan-525400cdec10 (stonith:fence_ipmilan): Started controller-2 40 stonith-fence_ipmilan-525400c709f7 (stonith:fence_ipmilan): Stopped 41 stonith-fence_ipmilan-525400a7f9e0 (stonith:fence_ipmilan): Started controller-0 42 stonith-fence_ipmilan-525400a25787 (stonith:fence_ipmilan): Started controller-2 43 stonith-fence_ipmilan-5254005ea387 (stonith:fence_ipmilan): Stopped 44 stonith-fence_ipmilan-525400542c06 (stonith:fence_ipmilan): Stopped 45 stonith-fence_ipmilan-525400aac413 (stonith:fence_ipmilan): Started controller-2 46 stonith-fence_ipmilan-525400498d34 (stonith:fence_ipmilan): Stopped 47 48Transition Summary: 49 * Fence (reboot) galera-bundle-2 (resource: galera-bundle-docker-2) 'guest is unclean' 50 * Fence (reboot) galera-bundle-0 (resource: galera-bundle-docker-0) 'guest is unclean' 51 * Start database-0 ( controller-0 ) 52 * Start database-2 ( controller-1 ) 53 * Recover galera-bundle-docker-0 ( database-0 ) 54 * Start galera-bundle-0 ( controller-0 ) 55 * Recover galera:0 ( Master galera-bundle-0 ) 56 * Recover galera-bundle-docker-2 ( database-2 ) 57 * Start galera-bundle-2 ( controller-1 ) 58 * Recover galera:2 ( Master galera-bundle-2 ) 59 * Promote redis:0 ( Slave -> Master redis-bundle-0 ) 60 * Start redis-bundle-docker-1 ( controller-1 ) 61 * Start redis-bundle-1 ( controller-1 ) 62 * Start redis:1 ( redis-bundle-1 ) 63 * Start ip-192.168.24.11 ( controller-0 ) 64 * Start ip-10.0.0.104 ( controller-1 ) 65 * Start ip-172.17.1.11 ( controller-0 ) 66 * Start ip-172.17.3.13 ( controller-1 ) 67 * Start haproxy-bundle-docker-1 ( controller-1 ) 68 * Start openstack-cinder-volume ( controller-0 ) 69 * Start stonith-fence_ipmilan-525400c709f7 ( controller-1 ) 70 * Start stonith-fence_ipmilan-5254005ea387 ( controller-1 ) 71 * Start stonith-fence_ipmilan-525400542c06 ( controller-0 ) 72 * Start stonith-fence_ipmilan-525400498d34 ( controller-1 ) 73 74Executing cluster transition: 75 * Resource action: database-0 start on controller-0 76 * Resource action: database-2 start on controller-1 77 * Pseudo action: redis-bundle-master_pre_notify_start_0 78 * Resource action: stonith-fence_ipmilan-525400c709f7 start on controller-1 79 * Resource action: stonith-fence_ipmilan-5254005ea387 start on controller-1 80 * Resource action: stonith-fence_ipmilan-525400542c06 start on controller-0 81 * Resource action: stonith-fence_ipmilan-525400498d34 start on controller-1 82 * Pseudo action: redis-bundle_start_0 83 * Pseudo action: galera-bundle_demote_0 84 * Resource action: database-0 monitor=20000 on controller-0 85 * Resource action: database-2 monitor=20000 on controller-1 86 * Pseudo action: galera-bundle-master_demote_0 87 * Resource action: redis notify on redis-bundle-0 88 * Resource action: redis notify on redis-bundle-2 89 * Pseudo action: redis-bundle-master_confirmed-pre_notify_start_0 90 * Pseudo action: redis-bundle-master_start_0 91 * Resource action: stonith-fence_ipmilan-525400c709f7 monitor=60000 on controller-1 92 * Resource action: stonith-fence_ipmilan-5254005ea387 monitor=60000 on controller-1 93 * Resource action: stonith-fence_ipmilan-525400542c06 monitor=60000 on controller-0 94 * Resource action: stonith-fence_ipmilan-525400498d34 monitor=60000 on controller-1 95 * Pseudo action: galera_demote_0 96 * Pseudo action: galera_demote_0 97 * Pseudo action: galera-bundle-master_demoted_0 98 * Pseudo action: galera-bundle_demoted_0 99 * Pseudo action: galera-bundle_stop_0 100 * Resource action: galera-bundle-docker-0 stop on database-0 101 * Resource action: galera-bundle-docker-2 stop on database-2 102 * Pseudo action: stonith-galera-bundle-2-reboot on galera-bundle-2 103 * Pseudo action: stonith-galera-bundle-0-reboot on galera-bundle-0 104 * Pseudo action: galera-bundle-master_stop_0 105 * Resource action: redis-bundle-docker-1 start on controller-1 106 * Resource action: redis-bundle-1 monitor on controller-1 107 * Resource action: ip-192.168.24.11 start on controller-0 108 * Resource action: ip-10.0.0.104 start on controller-1 109 * Resource action: ip-172.17.1.11 start on controller-0 110 * Resource action: ip-172.17.3.13 start on controller-1 111 * Resource action: openstack-cinder-volume start on controller-0 112 * Pseudo action: haproxy-bundle_start_0 113 * Pseudo action: galera_stop_0 114 * Resource action: redis-bundle-docker-1 monitor=60000 on controller-1 115 * Resource action: redis-bundle-1 start on controller-1 116 * Resource action: ip-192.168.24.11 monitor=10000 on controller-0 117 * Resource action: ip-10.0.0.104 monitor=10000 on controller-1 118 * Resource action: ip-172.17.1.11 monitor=10000 on controller-0 119 * Resource action: ip-172.17.3.13 monitor=10000 on controller-1 120 * Resource action: haproxy-bundle-docker-1 start on controller-1 121 * Resource action: openstack-cinder-volume monitor=60000 on controller-0 122 * Pseudo action: haproxy-bundle_running_0 123 * Pseudo action: galera_stop_0 124 * Pseudo action: galera-bundle-master_stopped_0 125 * Resource action: redis start on redis-bundle-1 126 * Pseudo action: redis-bundle-master_running_0 127 * Resource action: redis-bundle-1 monitor=60000 on controller-1 128 * Resource action: haproxy-bundle-docker-1 monitor=60000 on controller-1 129 * Pseudo action: galera-bundle_stopped_0 130 * Pseudo action: galera-bundle_start_0 131 * Pseudo action: galera-bundle-master_start_0 132 * Resource action: galera-bundle-docker-0 start on database-0 133 * Resource action: galera-bundle-0 monitor on controller-1 134 * Resource action: galera-bundle-docker-2 start on database-2 135 * Resource action: galera-bundle-2 monitor on controller-1 136 * Pseudo action: redis-bundle-master_post_notify_running_0 137 * Resource action: galera-bundle-docker-0 monitor=60000 on database-0 138 * Resource action: galera-bundle-0 start on controller-0 139 * Resource action: galera-bundle-docker-2 monitor=60000 on database-2 140 * Resource action: galera-bundle-2 start on controller-1 141 * Resource action: redis notify on redis-bundle-0 142 * Resource action: redis notify on redis-bundle-1 143 * Resource action: redis notify on redis-bundle-2 144 * Pseudo action: redis-bundle-master_confirmed-post_notify_running_0 145 * Pseudo action: redis-bundle_running_0 146 * Resource action: galera start on galera-bundle-0 147 * Resource action: galera start on galera-bundle-2 148 * Pseudo action: galera-bundle-master_running_0 149 * Resource action: galera-bundle-0 monitor=60000 on controller-0 150 * Resource action: galera-bundle-2 monitor=60000 on controller-1 151 * Pseudo action: redis-bundle-master_pre_notify_promote_0 152 * Pseudo action: redis-bundle_promote_0 153 * Pseudo action: galera-bundle_running_0 154 * Resource action: redis notify on redis-bundle-0 155 * Resource action: redis notify on redis-bundle-1 156 * Resource action: redis notify on redis-bundle-2 157 * Pseudo action: redis-bundle-master_confirmed-pre_notify_promote_0 158 * Pseudo action: redis-bundle-master_promote_0 159 * Pseudo action: galera-bundle_promote_0 160 * Pseudo action: galera-bundle-master_promote_0 161 * Resource action: redis promote on redis-bundle-0 162 * Pseudo action: redis-bundle-master_promoted_0 163 * Resource action: galera promote on galera-bundle-0 164 * Resource action: galera promote on galera-bundle-2 165 * Pseudo action: galera-bundle-master_promoted_0 166 * Pseudo action: redis-bundle-master_post_notify_promoted_0 167 * Pseudo action: galera-bundle_promoted_0 168 * Resource action: galera monitor=10000 on galera-bundle-0 169 * Resource action: galera monitor=10000 on galera-bundle-2 170 * Resource action: redis notify on redis-bundle-0 171 * Resource action: redis notify on redis-bundle-1 172 * Resource action: redis notify on redis-bundle-2 173 * Pseudo action: redis-bundle-master_confirmed-post_notify_promoted_0 174 * Pseudo action: redis-bundle_promoted_0 175 * Resource action: redis monitor=20000 on redis-bundle-0 176 * Resource action: redis monitor=60000 on redis-bundle-1 177 * Resource action: redis monitor=45000 on redis-bundle-1 178 179Revised cluster status: 180Online: [ controller-0 controller-1 controller-2 ] 181RemoteOnline: [ database-0 database-1 database-2 messaging-0 messaging-1 messaging-2 ] 182Containers: [ galera-bundle-0:galera-bundle-docker-0 galera-bundle-1:galera-bundle-docker-1 galera-bundle-2:galera-bundle-docker-2 rabbitmq-bundle-0:rabbitmq-bundle-docker-0 rabbitmq-bundle-1:rabbitmq-bundle-docker-1 rabbitmq-bundle-2:rabbitmq-bundle-docker-2 redis-bundle-0:redis-bundle-docker-0 redis-bundle-1:redis-bundle-docker-1 redis-bundle-2:redis-bundle-docker-2 ] 183 184 database-0 (ocf::pacemaker:remote): Started controller-0 185 database-1 (ocf::pacemaker:remote): Started controller-2 186 database-2 (ocf::pacemaker:remote): Started controller-1 187 messaging-0 (ocf::pacemaker:remote): Started controller-2 188 messaging-1 (ocf::pacemaker:remote): Started controller-2 189 messaging-2 (ocf::pacemaker:remote): Started controller-2 190 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] 191 rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started messaging-0 192 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started messaging-1 193 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started messaging-2 194 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] 195 galera-bundle-0 (ocf::heartbeat:galera): Master database-0 196 galera-bundle-1 (ocf::heartbeat:galera): Master database-1 197 galera-bundle-2 (ocf::heartbeat:galera): Master database-2 198 Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest] 199 redis-bundle-0 (ocf::heartbeat:redis): Master controller-0 200 redis-bundle-1 (ocf::heartbeat:redis): Slave controller-1 201 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 202 ip-192.168.24.11 (ocf::heartbeat:IPaddr2): Started controller-0 203 ip-10.0.0.104 (ocf::heartbeat:IPaddr2): Started controller-1 204 ip-172.17.1.19 (ocf::heartbeat:IPaddr2): Started controller-2 205 ip-172.17.1.11 (ocf::heartbeat:IPaddr2): Started controller-0 206 ip-172.17.3.13 (ocf::heartbeat:IPaddr2): Started controller-1 207 ip-172.17.4.19 (ocf::heartbeat:IPaddr2): Started controller-2 208 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] 209 haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 210 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-1 211 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-2 212 openstack-cinder-volume (systemd:openstack-cinder-volume): Started controller-0 213 stonith-fence_ipmilan-525400244e09 (stonith:fence_ipmilan): Started controller-2 214 stonith-fence_ipmilan-525400cdec10 (stonith:fence_ipmilan): Started controller-2 215 stonith-fence_ipmilan-525400c709f7 (stonith:fence_ipmilan): Started controller-1 216 stonith-fence_ipmilan-525400a7f9e0 (stonith:fence_ipmilan): Started controller-0 217 stonith-fence_ipmilan-525400a25787 (stonith:fence_ipmilan): Started controller-2 218 stonith-fence_ipmilan-5254005ea387 (stonith:fence_ipmilan): Started controller-1 219 stonith-fence_ipmilan-525400542c06 (stonith:fence_ipmilan): Started controller-0 220 stonith-fence_ipmilan-525400aac413 (stonith:fence_ipmilan): Started controller-2 221 stonith-fence_ipmilan-525400498d34 (stonith:fence_ipmilan): Started controller-1 222 223