1Using the original execution date of: 2017-09-12 10:51:59Z 2 3Current cluster status: 4Node controller-0 (1): UNCLEAN (offline) 5Online: [ controller-1 controller-2 ] 6Containers: [ galera-bundle-1:galera-bundle-docker-1 galera-bundle-2:galera-bundle-docker-2 rabbitmq-bundle-1:rabbitmq-bundle-docker-1 rabbitmq-bundle-2:rabbitmq-bundle-docker-2 redis-bundle-1:redis-bundle-docker-1 redis-bundle-2:redis-bundle-docker-2 ] 7 8 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] 9 rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): FAILED controller-0 (UNCLEAN) 10 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started controller-1 11 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started controller-2 12 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] 13 galera-bundle-0 (ocf::heartbeat:galera): FAILED Master controller-0 (UNCLEAN) 14 galera-bundle-1 (ocf::heartbeat:galera): Master controller-1 15 galera-bundle-2 (ocf::heartbeat:galera): Master controller-2 16 Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest] 17 redis-bundle-0 (ocf::heartbeat:redis): FAILED Master controller-0 (UNCLEAN) 18 redis-bundle-1 (ocf::heartbeat:redis): Slave controller-1 19 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 20 ip-192.168.24.7 (ocf::heartbeat:IPaddr2): Started controller-0 (UNCLEAN) 21 ip-10.0.0.109 (ocf::heartbeat:IPaddr2): Started controller-0 (UNCLEAN) 22 ip-172.17.1.14 (ocf::heartbeat:IPaddr2): Started controller-2 23 ip-172.17.1.19 (ocf::heartbeat:IPaddr2): Started controller-2 24 ip-172.17.3.19 (ocf::heartbeat:IPaddr2): Started controller-2 25 ip-172.17.4.11 (ocf::heartbeat:IPaddr2): Started controller-0 (UNCLEAN) 26 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] 27 haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 (UNCLEAN) 28 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-2 29 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-1 30 openstack-cinder-volume (systemd:openstack-cinder-volume): Started controller-2 31 stonith-fence_ipmilan-525400efba5c (stonith:fence_ipmilan): Started controller-2 32 stonith-fence_ipmilan-5254003e8e97 (stonith:fence_ipmilan): Started controller-0 (UNCLEAN) 33 stonith-fence_ipmilan-5254000dcb3f (stonith:fence_ipmilan): Started controller-0 (UNCLEAN) 34 35Transition Summary: 36 * Fence (off) redis-bundle-0 (resource: redis-bundle-docker-0) 'guest is unclean' 37 * Fence (off) rabbitmq-bundle-0 (resource: rabbitmq-bundle-docker-0) 'guest is unclean' 38 * Fence (off) galera-bundle-0 (resource: galera-bundle-docker-0) 'guest is unclean' 39 * Fence (reboot) controller-0 'peer is no longer part of the cluster' 40 * Stop rabbitmq-bundle-docker-0 ( controller-0 ) due to node availability 41 * Stop rabbitmq-bundle-0 ( controller-0 ) due to unrunnable rabbitmq-bundle-docker-0 start 42 * Stop rabbitmq:0 ( rabbitmq-bundle-0 ) due to unrunnable rabbitmq-bundle-docker-0 start 43 * Stop galera-bundle-docker-0 ( controller-0 ) due to node availability 44 * Stop galera-bundle-0 ( controller-0 ) due to unrunnable galera-bundle-docker-0 start 45 * Stop galera:0 ( Master galera-bundle-0 ) due to unrunnable galera-bundle-docker-0 start 46 * Stop redis-bundle-docker-0 ( controller-0 ) due to node availability 47 * Stop redis-bundle-0 ( controller-0 ) due to unrunnable redis-bundle-docker-0 start 48 * Stop redis:0 ( Master redis-bundle-0 ) due to unrunnable redis-bundle-docker-0 start 49 * Promote redis:1 ( Slave -> Master redis-bundle-1 ) 50 * Move ip-192.168.24.7 ( controller-0 -> controller-2 ) 51 * Move ip-10.0.0.109 ( controller-0 -> controller-1 ) 52 * Move ip-172.17.4.11 ( controller-0 -> controller-1 ) 53 * Stop haproxy-bundle-docker-0 ( controller-0 ) due to node availability 54 * Move stonith-fence_ipmilan-5254003e8e97 ( controller-0 -> controller-1 ) 55 * Move stonith-fence_ipmilan-5254000dcb3f ( controller-0 -> controller-2 ) 56 57Executing cluster transition: 58 * Pseudo action: rabbitmq-bundle-clone_pre_notify_stop_0 59 * Pseudo action: rabbitmq-bundle-0_stop_0 60 * Resource action: rabbitmq-bundle-0 monitor on controller-2 61 * Resource action: rabbitmq-bundle-0 monitor on controller-1 62 * Resource action: rabbitmq-bundle-1 monitor on controller-2 63 * Resource action: rabbitmq-bundle-2 monitor on controller-1 64 * Pseudo action: galera-bundle-0_stop_0 65 * Resource action: galera-bundle-0 monitor on controller-2 66 * Resource action: galera-bundle-0 monitor on controller-1 67 * Resource action: galera-bundle-1 monitor on controller-2 68 * Resource action: galera-bundle-2 monitor on controller-1 69 * Resource action: redis cancel=45000 on redis-bundle-1 70 * Resource action: redis cancel=60000 on redis-bundle-1 71 * Pseudo action: redis-bundle-master_pre_notify_demote_0 72 * Pseudo action: redis-bundle-0_stop_0 73 * Resource action: redis-bundle-0 monitor on controller-2 74 * Resource action: redis-bundle-0 monitor on controller-1 75 * Resource action: redis-bundle-1 monitor on controller-2 76 * Resource action: redis-bundle-2 monitor on controller-1 77 * Pseudo action: stonith-fence_ipmilan-5254003e8e97_stop_0 78 * Pseudo action: stonith-fence_ipmilan-5254000dcb3f_stop_0 79 * Pseudo action: haproxy-bundle_stop_0 80 * Pseudo action: redis-bundle_demote_0 81 * Pseudo action: galera-bundle_demote_0 82 * Pseudo action: rabbitmq-bundle_stop_0 83 * Pseudo action: rabbitmq-bundle_start_0 84 * Fencing controller-0 (reboot) 85 * Resource action: rabbitmq notify on rabbitmq-bundle-1 86 * Resource action: rabbitmq notify on rabbitmq-bundle-2 87 * Pseudo action: rabbitmq-bundle-clone_confirmed-pre_notify_stop_0 88 * Pseudo action: rabbitmq-bundle-docker-0_stop_0 89 * Pseudo action: galera-bundle-master_demote_0 90 * Resource action: redis notify on redis-bundle-1 91 * Resource action: redis notify on redis-bundle-2 92 * Pseudo action: redis-bundle-master_confirmed-pre_notify_demote_0 93 * Pseudo action: redis-bundle-master_demote_0 94 * Pseudo action: haproxy-bundle-docker-0_stop_0 95 * Resource action: stonith-fence_ipmilan-5254003e8e97 start on controller-1 96 * Resource action: stonith-fence_ipmilan-5254000dcb3f start on controller-2 97 * Pseudo action: stonith-redis-bundle-0-off on redis-bundle-0 98 * Pseudo action: stonith-rabbitmq-bundle-0-off on rabbitmq-bundle-0 99 * Pseudo action: stonith-galera-bundle-0-off on galera-bundle-0 100 * Pseudo action: haproxy-bundle_stopped_0 101 * Pseudo action: rabbitmq_post_notify_stop_0 102 * Pseudo action: rabbitmq-bundle-clone_stop_0 103 * Pseudo action: galera_demote_0 104 * Pseudo action: galera-bundle-master_demoted_0 105 * Pseudo action: redis_post_notify_stop_0 106 * Pseudo action: redis_demote_0 107 * Pseudo action: redis-bundle-master_demoted_0 108 * Pseudo action: ip-192.168.24.7_stop_0 109 * Pseudo action: ip-10.0.0.109_stop_0 110 * Pseudo action: ip-172.17.4.11_stop_0 111 * Resource action: stonith-fence_ipmilan-5254003e8e97 monitor=60000 on controller-1 112 * Resource action: stonith-fence_ipmilan-5254000dcb3f monitor=60000 on controller-2 113 * Pseudo action: galera-bundle_demoted_0 114 * Pseudo action: galera-bundle_stop_0 115 * Pseudo action: rabbitmq_stop_0 116 * Pseudo action: rabbitmq-bundle-clone_stopped_0 117 * Pseudo action: galera-bundle-master_stop_0 118 * Pseudo action: galera-bundle-docker-0_stop_0 119 * Pseudo action: redis-bundle-master_post_notify_demoted_0 120 * Resource action: ip-192.168.24.7 start on controller-2 121 * Resource action: ip-10.0.0.109 start on controller-1 122 * Resource action: ip-172.17.4.11 start on controller-1 123 * Pseudo action: rabbitmq-bundle-clone_post_notify_stopped_0 124 * Pseudo action: galera_stop_0 125 * Pseudo action: galera-bundle-master_stopped_0 126 * Pseudo action: galera-bundle-master_start_0 127 * Resource action: redis notify on redis-bundle-1 128 * Resource action: redis notify on redis-bundle-2 129 * Pseudo action: redis-bundle-master_confirmed-post_notify_demoted_0 130 * Pseudo action: redis-bundle-master_pre_notify_stop_0 131 * Resource action: ip-192.168.24.7 monitor=10000 on controller-2 132 * Resource action: ip-10.0.0.109 monitor=10000 on controller-1 133 * Resource action: ip-172.17.4.11 monitor=10000 on controller-1 134 * Pseudo action: redis-bundle_demoted_0 135 * Pseudo action: redis-bundle_stop_0 136 * Pseudo action: galera-bundle_stopped_0 137 * Resource action: rabbitmq notify on rabbitmq-bundle-1 138 * Resource action: rabbitmq notify on rabbitmq-bundle-2 139 * Pseudo action: rabbitmq-bundle-clone_confirmed-post_notify_stopped_0 140 * Pseudo action: rabbitmq-bundle-clone_pre_notify_start_0 141 * Pseudo action: galera-bundle-master_running_0 142 * Resource action: redis notify on redis-bundle-1 143 * Resource action: redis notify on redis-bundle-2 144 * Pseudo action: redis-bundle-master_confirmed-pre_notify_stop_0 145 * Pseudo action: redis-bundle-master_stop_0 146 * Pseudo action: redis-bundle-docker-0_stop_0 147 * Pseudo action: galera-bundle_running_0 148 * Pseudo action: rabbitmq-bundle_stopped_0 149 * Pseudo action: rabbitmq_notified_0 150 * Pseudo action: rabbitmq-bundle-clone_confirmed-pre_notify_start_0 151 * Pseudo action: rabbitmq-bundle-clone_start_0 152 * Pseudo action: redis_stop_0 153 * Pseudo action: redis-bundle-master_stopped_0 154 * Pseudo action: rabbitmq-bundle-clone_running_0 155 * Pseudo action: redis-bundle-master_post_notify_stopped_0 156 * Pseudo action: rabbitmq-bundle-clone_post_notify_running_0 157 * Resource action: redis notify on redis-bundle-1 158 * Resource action: redis notify on redis-bundle-2 159 * Pseudo action: redis-bundle-master_confirmed-post_notify_stopped_0 160 * Pseudo action: redis-bundle-master_pre_notify_start_0 161 * Pseudo action: redis-bundle_stopped_0 162 * Pseudo action: rabbitmq-bundle-clone_confirmed-post_notify_running_0 163 * Pseudo action: redis_notified_0 164 * Pseudo action: redis-bundle-master_confirmed-pre_notify_start_0 165 * Pseudo action: redis-bundle-master_start_0 166 * Pseudo action: rabbitmq-bundle_running_0 167 * Pseudo action: redis-bundle-master_running_0 168 * Pseudo action: redis-bundle-master_post_notify_running_0 169 * Pseudo action: redis-bundle-master_confirmed-post_notify_running_0 170 * Pseudo action: redis-bundle_running_0 171 * Pseudo action: redis-bundle-master_pre_notify_promote_0 172 * Pseudo action: redis-bundle_promote_0 173 * Resource action: redis notify on redis-bundle-1 174 * Resource action: redis notify on redis-bundle-2 175 * Pseudo action: redis-bundle-master_confirmed-pre_notify_promote_0 176 * Pseudo action: redis-bundle-master_promote_0 177 * Resource action: redis promote on redis-bundle-1 178 * Pseudo action: redis-bundle-master_promoted_0 179 * Pseudo action: redis-bundle-master_post_notify_promoted_0 180 * Resource action: redis notify on redis-bundle-1 181 * Resource action: redis notify on redis-bundle-2 182 * Pseudo action: redis-bundle-master_confirmed-post_notify_promoted_0 183 * Pseudo action: redis-bundle_promoted_0 184 * Resource action: redis monitor=20000 on redis-bundle-1 185Using the original execution date of: 2017-09-12 10:51:59Z 186 187Revised cluster status: 188Online: [ controller-1 controller-2 ] 189OFFLINE: [ controller-0 ] 190Containers: [ galera-bundle-1:galera-bundle-docker-1 galera-bundle-2:galera-bundle-docker-2 rabbitmq-bundle-1:rabbitmq-bundle-docker-1 rabbitmq-bundle-2:rabbitmq-bundle-docker-2 redis-bundle-1:redis-bundle-docker-1 redis-bundle-2:redis-bundle-docker-2 ] 191 192 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] 193 rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): FAILED 194 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started controller-1 195 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started controller-2 196 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] 197 galera-bundle-0 (ocf::heartbeat:galera): FAILED Master 198 galera-bundle-1 (ocf::heartbeat:galera): Master controller-1 199 galera-bundle-2 (ocf::heartbeat:galera): Master controller-2 200 Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest] 201 redis-bundle-0 (ocf::heartbeat:redis): FAILED Master 202 redis-bundle-1 (ocf::heartbeat:redis): Master controller-1 203 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 204 ip-192.168.24.7 (ocf::heartbeat:IPaddr2): Started controller-2 205 ip-10.0.0.109 (ocf::heartbeat:IPaddr2): Started controller-1 206 ip-172.17.1.14 (ocf::heartbeat:IPaddr2): Started controller-2 207 ip-172.17.1.19 (ocf::heartbeat:IPaddr2): Started controller-2 208 ip-172.17.3.19 (ocf::heartbeat:IPaddr2): Started controller-2 209 ip-172.17.4.11 (ocf::heartbeat:IPaddr2): Started controller-1 210 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] 211 haproxy-bundle-docker-0 (ocf::heartbeat:docker): Stopped 212 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-2 213 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-1 214 openstack-cinder-volume (systemd:openstack-cinder-volume): Started controller-2 215 stonith-fence_ipmilan-525400efba5c (stonith:fence_ipmilan): Started controller-2 216 stonith-fence_ipmilan-5254003e8e97 (stonith:fence_ipmilan): Started controller-1 217 stonith-fence_ipmilan-5254000dcb3f (stonith:fence_ipmilan): Started controller-2 218 219