1Using the original execution date of: 2018-09-11 21:23:25Z 2 3Current cluster status: 4Online: [ controller-0 controller-1 controller-2 ] 5RemoteOnline: [ database-0 database-1 database-2 messaging-0 messaging-1 messaging-2 ] 6Containers: [ galera-bundle-1:galera-bundle-docker-1 galera-bundle-2:galera-bundle-docker-2 rabbitmq-bundle-0:rabbitmq-bundle-docker-0 rabbitmq-bundle-1:rabbitmq-bundle-docker-1 rabbitmq-bundle-2:rabbitmq-bundle-docker-2 redis-bundle-0:redis-bundle-docker-0 redis-bundle-1:redis-bundle-docker-1 redis-bundle-2:redis-bundle-docker-2 ] 7 8 database-0 (ocf::pacemaker:remote): Started controller-0 9 database-1 (ocf::pacemaker:remote): Started controller-1 10 database-2 (ocf::pacemaker:remote): Started controller-2 11 messaging-0 (ocf::pacemaker:remote): Started controller-2 12 messaging-1 (ocf::pacemaker:remote): Started controller-1 13 messaging-2 (ocf::pacemaker:remote): Started controller-1 14 Docker container set: galera-bundle [192.168.24.1:8787/rhosp13/openstack-mariadb:pcmklatest] 15 galera-bundle-0 (ocf::heartbeat:galera): FAILED Master database-0 16 galera-bundle-1 (ocf::heartbeat:galera): Master database-1 17 galera-bundle-2 (ocf::heartbeat:galera): Master database-2 18 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp13/openstack-rabbitmq:pcmklatest] 19 rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started messaging-0 20 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started messaging-1 21 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started messaging-2 22 Docker container set: redis-bundle [192.168.24.1:8787/rhosp13/openstack-redis:pcmklatest] 23 redis-bundle-0 (ocf::heartbeat:redis): Slave controller-0 24 redis-bundle-1 (ocf::heartbeat:redis): Master controller-1 25 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 26 ip-192.168.24.12 (ocf::heartbeat:IPaddr2): Started controller-1 27 ip-10.0.0.109 (ocf::heartbeat:IPaddr2): Started controller-2 28 ip-172.17.1.18 (ocf::heartbeat:IPaddr2): Started controller-1 29 ip-172.17.1.12 (ocf::heartbeat:IPaddr2): Started controller-2 30 ip-172.17.3.18 (ocf::heartbeat:IPaddr2): Started controller-1 31 ip-172.17.4.14 (ocf::heartbeat:IPaddr2): Started controller-1 32 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp13/openstack-haproxy:pcmklatest] 33 haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 34 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-1 35 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-2 36 Docker container: openstack-cinder-volume [192.168.24.1:8787/rhosp13/openstack-cinder-volume:pcmklatest] 37 openstack-cinder-volume-docker-0 (ocf::heartbeat:docker): Started controller-0 38 stonith-fence_ipmilan-5254005f9a33 (stonith:fence_ipmilan): Started controller-2 39 stonith-fence_ipmilan-52540098c9ff (stonith:fence_ipmilan): Started controller-1 40 stonith-fence_ipmilan-5254000203a2 (stonith:fence_ipmilan): Started controller-2 41 stonith-fence_ipmilan-5254003296a5 (stonith:fence_ipmilan): Started controller-1 42 stonith-fence_ipmilan-52540066e27e (stonith:fence_ipmilan): Started controller-1 43 stonith-fence_ipmilan-52540065418e (stonith:fence_ipmilan): Started controller-2 44 stonith-fence_ipmilan-525400aab9d9 (stonith:fence_ipmilan): Started controller-2 45 stonith-fence_ipmilan-525400a16c0d (stonith:fence_ipmilan): Started controller-1 46 stonith-fence_ipmilan-5254002f6d57 (stonith:fence_ipmilan): Started controller-1 47 48Transition Summary: 49 * Fence (reboot) galera-bundle-0 (resource: galera-bundle-docker-0) 'guest is unclean' 50 * Recover galera-bundle-docker-0 ( database-0 ) 51 * Recover galera-bundle-0 ( controller-0 ) 52 * Recover galera:0 ( Master galera-bundle-0 ) 53 * Restart stonith-fence_ipmilan-5254005f9a33 ( controller-2 ) due to resource definition change 54 * Restart stonith-fence_ipmilan-52540098c9ff ( controller-1 ) due to resource definition change 55 * Restart stonith-fence_ipmilan-5254000203a2 ( controller-2 ) due to resource definition change 56 * Restart stonith-fence_ipmilan-5254003296a5 ( controller-1 ) due to resource definition change 57 * Restart stonith-fence_ipmilan-52540066e27e ( controller-1 ) due to resource definition change 58 * Restart stonith-fence_ipmilan-52540065418e ( controller-2 ) due to resource definition change 59 * Restart stonith-fence_ipmilan-525400aab9d9 ( controller-2 ) due to resource definition change 60 * Restart stonith-fence_ipmilan-525400a16c0d ( controller-1 ) due to resource definition change 61 * Restart stonith-fence_ipmilan-5254002f6d57 ( controller-1 ) due to resource definition change 62 63Executing cluster transition: 64 * Resource action: galera-bundle-0 stop on controller-0 65 * Cluster action: clear_failcount for galera-bundle-0 on controller-0 66 * Resource action: galera-bundle-0 cancel=30000 on controller-0 67 * Resource action: galera-bundle-1 monitor=60000 on controller-1 68 * Resource action: galera-bundle-1 cancel=30000 on controller-1 69 * Resource action: galera-bundle-2 monitor=60000 on controller-2 70 * Resource action: galera-bundle-2 cancel=30000 on controller-2 71 * Resource action: rabbitmq-bundle-0 monitor=60000 on controller-2 72 * Resource action: rabbitmq-bundle-0 cancel=30000 on controller-2 73 * Resource action: rabbitmq-bundle-1 monitor=60000 on controller-1 74 * Resource action: rabbitmq-bundle-1 cancel=30000 on controller-1 75 * Resource action: rabbitmq-bundle-2 monitor=60000 on controller-1 76 * Resource action: rabbitmq-bundle-2 cancel=30000 on controller-1 77 * Resource action: redis-bundle-0 monitor=60000 on controller-0 78 * Resource action: redis-bundle-0 cancel=30000 on controller-0 79 * Resource action: redis-bundle-1 monitor=60000 on controller-1 80 * Resource action: redis-bundle-1 cancel=30000 on controller-1 81 * Resource action: redis-bundle-2 monitor=60000 on controller-2 82 * Resource action: redis-bundle-2 cancel=30000 on controller-2 83 * Resource action: stonith-fence_ipmilan-5254005f9a33 stop on controller-2 84 * Resource action: stonith-fence_ipmilan-5254005f9a33 start on controller-2 85 * Resource action: stonith-fence_ipmilan-5254005f9a33 monitor=60000 on controller-2 86 * Resource action: stonith-fence_ipmilan-52540098c9ff stop on controller-1 87 * Resource action: stonith-fence_ipmilan-52540098c9ff start on controller-1 88 * Resource action: stonith-fence_ipmilan-52540098c9ff monitor=60000 on controller-1 89 * Resource action: stonith-fence_ipmilan-5254000203a2 stop on controller-2 90 * Resource action: stonith-fence_ipmilan-5254000203a2 start on controller-2 91 * Resource action: stonith-fence_ipmilan-5254000203a2 monitor=60000 on controller-2 92 * Resource action: stonith-fence_ipmilan-5254003296a5 stop on controller-1 93 * Resource action: stonith-fence_ipmilan-5254003296a5 start on controller-1 94 * Resource action: stonith-fence_ipmilan-5254003296a5 monitor=60000 on controller-1 95 * Resource action: stonith-fence_ipmilan-52540066e27e stop on controller-1 96 * Resource action: stonith-fence_ipmilan-52540066e27e start on controller-1 97 * Resource action: stonith-fence_ipmilan-52540066e27e monitor=60000 on controller-1 98 * Resource action: stonith-fence_ipmilan-52540065418e stop on controller-2 99 * Resource action: stonith-fence_ipmilan-52540065418e start on controller-2 100 * Resource action: stonith-fence_ipmilan-52540065418e monitor=60000 on controller-2 101 * Resource action: stonith-fence_ipmilan-525400aab9d9 stop on controller-2 102 * Resource action: stonith-fence_ipmilan-525400aab9d9 start on controller-2 103 * Resource action: stonith-fence_ipmilan-525400aab9d9 monitor=60000 on controller-2 104 * Resource action: stonith-fence_ipmilan-525400a16c0d stop on controller-1 105 * Resource action: stonith-fence_ipmilan-525400a16c0d start on controller-1 106 * Resource action: stonith-fence_ipmilan-525400a16c0d monitor=60000 on controller-1 107 * Resource action: stonith-fence_ipmilan-5254002f6d57 stop on controller-1 108 * Resource action: stonith-fence_ipmilan-5254002f6d57 start on controller-1 109 * Resource action: stonith-fence_ipmilan-5254002f6d57 monitor=60000 on controller-1 110 * Pseudo action: galera-bundle_demote_0 111 * Pseudo action: galera-bundle-master_demote_0 112 * Pseudo action: galera_demote_0 113 * Pseudo action: galera-bundle-master_demoted_0 114 * Pseudo action: galera-bundle_demoted_0 115 * Pseudo action: galera-bundle_stop_0 116 * Resource action: galera-bundle-docker-0 stop on database-0 117 * Pseudo action: stonith-galera-bundle-0-reboot on galera-bundle-0 118 * Pseudo action: galera-bundle-master_stop_0 119 * Pseudo action: galera_stop_0 120 * Pseudo action: galera-bundle-master_stopped_0 121 * Pseudo action: galera-bundle_stopped_0 122 * Pseudo action: galera-bundle_start_0 123 * Pseudo action: galera-bundle-master_start_0 124 * Resource action: galera-bundle-docker-0 start on database-0 125 * Resource action: galera-bundle-docker-0 monitor=60000 on database-0 126 * Resource action: galera-bundle-0 start on controller-0 127 * Resource action: galera start on galera-bundle-0 128 * Pseudo action: galera-bundle-master_running_0 129 * Resource action: galera-bundle-0 monitor=60000 on controller-0 130 * Pseudo action: galera-bundle_running_0 131 * Pseudo action: galera-bundle_promote_0 132 * Pseudo action: galera-bundle-master_promote_0 133 * Resource action: galera promote on galera-bundle-0 134 * Pseudo action: galera-bundle-master_promoted_0 135 * Pseudo action: galera-bundle_promoted_0 136 * Resource action: galera monitor=10000 on galera-bundle-0 137Using the original execution date of: 2018-09-11 21:23:25Z 138 139Revised cluster status: 140Online: [ controller-0 controller-1 controller-2 ] 141RemoteOnline: [ database-0 database-1 database-2 messaging-0 messaging-1 messaging-2 ] 142Containers: [ galera-bundle-0:galera-bundle-docker-0 galera-bundle-1:galera-bundle-docker-1 galera-bundle-2:galera-bundle-docker-2 rabbitmq-bundle-0:rabbitmq-bundle-docker-0 rabbitmq-bundle-1:rabbitmq-bundle-docker-1 rabbitmq-bundle-2:rabbitmq-bundle-docker-2 redis-bundle-0:redis-bundle-docker-0 redis-bundle-1:redis-bundle-docker-1 redis-bundle-2:redis-bundle-docker-2 ] 143 144 database-0 (ocf::pacemaker:remote): Started controller-0 145 database-1 (ocf::pacemaker:remote): Started controller-1 146 database-2 (ocf::pacemaker:remote): Started controller-2 147 messaging-0 (ocf::pacemaker:remote): Started controller-2 148 messaging-1 (ocf::pacemaker:remote): Started controller-1 149 messaging-2 (ocf::pacemaker:remote): Started controller-1 150 Docker container set: galera-bundle [192.168.24.1:8787/rhosp13/openstack-mariadb:pcmklatest] 151 galera-bundle-0 (ocf::heartbeat:galera): Master database-0 152 galera-bundle-1 (ocf::heartbeat:galera): Master database-1 153 galera-bundle-2 (ocf::heartbeat:galera): Master database-2 154 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp13/openstack-rabbitmq:pcmklatest] 155 rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started messaging-0 156 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started messaging-1 157 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started messaging-2 158 Docker container set: redis-bundle [192.168.24.1:8787/rhosp13/openstack-redis:pcmklatest] 159 redis-bundle-0 (ocf::heartbeat:redis): Slave controller-0 160 redis-bundle-1 (ocf::heartbeat:redis): Master controller-1 161 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 162 ip-192.168.24.12 (ocf::heartbeat:IPaddr2): Started controller-1 163 ip-10.0.0.109 (ocf::heartbeat:IPaddr2): Started controller-2 164 ip-172.17.1.18 (ocf::heartbeat:IPaddr2): Started controller-1 165 ip-172.17.1.12 (ocf::heartbeat:IPaddr2): Started controller-2 166 ip-172.17.3.18 (ocf::heartbeat:IPaddr2): Started controller-1 167 ip-172.17.4.14 (ocf::heartbeat:IPaddr2): Started controller-1 168 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp13/openstack-haproxy:pcmklatest] 169 haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 170 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-1 171 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-2 172 Docker container: openstack-cinder-volume [192.168.24.1:8787/rhosp13/openstack-cinder-volume:pcmklatest] 173 openstack-cinder-volume-docker-0 (ocf::heartbeat:docker): Started controller-0 174 stonith-fence_ipmilan-5254005f9a33 (stonith:fence_ipmilan): Started controller-2 175 stonith-fence_ipmilan-52540098c9ff (stonith:fence_ipmilan): Started controller-1 176 stonith-fence_ipmilan-5254000203a2 (stonith:fence_ipmilan): Started controller-2 177 stonith-fence_ipmilan-5254003296a5 (stonith:fence_ipmilan): Started controller-1 178 stonith-fence_ipmilan-52540066e27e (stonith:fence_ipmilan): Started controller-1 179 stonith-fence_ipmilan-52540065418e (stonith:fence_ipmilan): Started controller-2 180 stonith-fence_ipmilan-525400aab9d9 (stonith:fence_ipmilan): Started controller-2 181 stonith-fence_ipmilan-525400a16c0d (stonith:fence_ipmilan): Started controller-1 182 stonith-fence_ipmilan-5254002f6d57 (stonith:fence_ipmilan): Started controller-1 183 184