1.. index:: command-line tool 2 3Using Pacemaker Command-Line Tools 4---------------------------------- 5 6.. index:: 7 single: command-line tool; output format 8 9.. _cmdline_output: 10 11Controlling Command Line Output 12############################### 13 14Some of the pacemaker command line utilities have been converted to a new 15output system. Among these tools are ``crm_mon`` and ``stonith_admin``. This 16is an ongoing project, and more tools will be converted over time. This system 17lets you control the formatting of output with ``--output-as=`` and the 18destination of output with ``--output-to=``. 19 20The available formats vary by tool, but at least plain text and XML are 21supported by all tools that use the new system. The default format is plain 22text. The default destination is stdout but can be redirected to any file. 23Some formats support command line options for changing the style of the output. 24For instance: 25 26.. code-block:: none 27 28 # crm_mon --help-output 29 Usage: 30 crm_mon [OPTION?] 31 32 Provides a summary of cluster's current state. 33 34 Outputs varying levels of detail in a number of different formats. 35 36 Output Options: 37 --output-as=FORMAT Specify output format as one of: console (default), html, text, xml 38 --output-to=DEST Specify file name for output (or "-" for stdout) 39 --html-cgi Add text needed to use output in a CGI program 40 --html-stylesheet=URI Link to an external CSS stylesheet 41 --html-title=TITLE Page title 42 --text-fancy Use more highly formatted output 43 44.. index:: 45 single: crm_mon 46 single: command-line tool; crm_mon 47 48.. _crm_mon: 49 50Monitor a Cluster with crm_mon 51############################## 52 53The ``crm_mon`` utility displays the current state of an active cluster. It can 54show the cluster status organized by node or by resource, and can be used in 55either single-shot or dynamically updating mode. It can also display operations 56performed and information about failures. 57 58Using this tool, you can examine the state of the cluster for irregularities, 59and see how it responds when you cause or simulate failures. 60 61See the manual page or the output of ``crm_mon --help`` for a full description 62of its many options. 63 64.. topic:: Sample output from crm_mon -1 65 66 .. code-block:: none 67 68 Cluster Summary: 69 * Stack: corosync 70 * Current DC: node2 (version 2.0.0-1) - partition with quorum 71 * Last updated: Mon Jan 29 12:18:42 2018 72 * Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 73 * 5 nodes configured 74 * 2 resources configured 75 76 Node List: 77 * Online: [ node1 node2 node3 node4 node5 ] 78 79 * Active resources: 80 * Fencing (stonith:fence_xvm): Started node1 81 * IP (ocf:heartbeat:IPaddr2): Started node2 82 83.. topic:: Sample output from crm_mon -n -1 84 85 .. code-block:: none 86 87 Cluster Summary: 88 * Stack: corosync 89 * Current DC: node2 (version 2.0.0-1) - partition with quorum 90 * Last updated: Mon Jan 29 12:21:48 2018 91 * Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 92 * 5 nodes configured 93 * 2 resources configured 94 95 * Node List: 96 * Node node1: online 97 * Fencing (stonith:fence_xvm): Started 98 * Node node2: online 99 * IP (ocf:heartbeat:IPaddr2): Started 100 * Node node3: online 101 * Node node4: online 102 * Node node5: online 103 104As mentioned in an earlier chapter, the DC is the node is where decisions are 105made. The cluster elects a node to be DC as needed. The only significance of 106the choice of DC to an administrator is the fact that its logs will have the 107most information about why decisions were made. 108 109.. index:: 110 pair: crm_mon; CSS 111 112.. _crm_mon_css: 113 114Styling crm_mon HTML output 115___________________________ 116 117Various parts of ``crm_mon``'s HTML output have a CSS class associated with 118them. Not everything does, but some of the most interesting portions do. In 119the following example, the status of each node has an ``online`` class and the 120details of each resource have an ``rsc-ok`` class. 121 122.. code-block:: html 123 124 <h2>Node List</h2> 125 <ul> 126 <li> 127 <span>Node: cluster01</span><span class="online"> online</span> 128 </li> 129 <li><ul><li><span class="rsc-ok">ping (ocf::pacemaker:ping): Started</span></li></ul></li> 130 <li> 131 <span>Node: cluster02</span><span class="online"> online</span> 132 </li> 133 <li><ul><li><span class="rsc-ok">ping (ocf::pacemaker:ping): Started</span></li></ul></li> 134 </ul> 135 136By default, a stylesheet for styling these classes is included in the head of 137the HTML output. The relevant portions of this stylesheet that would be used 138in the above example is: 139 140.. code-block:: css 141 142 <style> 143 .online { color: green } 144 .rsc-ok { color: green } 145 </style> 146 147If you want to override some or all of the styling, simply create your own 148stylesheet, place it on a web server, and pass ``--html-stylesheet=<URL>`` 149to ``crm_mon``. The link is added after the default stylesheet, so your 150changes take precedence. You don't need to duplicate the entire default. 151Only include what you want to change. 152 153.. index:: 154 single: cibadmin 155 single: command-line tool; cibadmin 156 157.. _cibadmin: 158 159Edit the CIB XML with cibadmin 160############################## 161 162The most flexible tool for modifying the configuration is Pacemaker's 163``cibadmin`` command. With ``cibadmin``, you can query, add, remove, update 164or replace any part of the configuration. All changes take effect immediately, 165so there is no need to perform a reload-like operation. 166 167The simplest way of using ``cibadmin`` is to use it to save the current 168configuration to a temporary file, edit that file with your favorite 169text or XML editor, and then upload the revised configuration. 170 171.. topic:: Safely using an editor to modify the cluster configuration 172 173 .. code-block:: none 174 175 # cibadmin --query > tmp.xml 176 # vi tmp.xml 177 # cibadmin --replace --xml-file tmp.xml 178 179Some of the better XML editors can make use of a RELAX NG schema to 180help make sure any changes you make are valid. The schema describing 181the configuration can be found in ``pacemaker.rng``, which may be 182deployed in a location such as ``/usr/share/pacemaker`` depending on your 183operating system distribution and how you installed the software. 184 185If you want to modify just one section of the configuration, you can 186query and replace just that section to avoid modifying any others. 187 188.. topic:: Safely using an editor to modify only the resources section 189 190 .. code-block:: none 191 192 # cibadmin --query --scope resources > tmp.xml 193 # vi tmp.xml 194 # cibadmin --replace --scope resources --xml-file tmp.xml 195 196To quickly delete a part of the configuration, identify the object you wish to 197delete by XML tag and id. For example, you might search the CIB for all 198STONITH-related configuration: 199 200.. topic:: Searching for STONITH-related configuration items 201 202 .. code-block:: none 203 204 # cibadmin --query | grep stonith 205 <nvpair id="cib-bootstrap-options-stonith-action" name="stonith-action" value="reboot"/> 206 <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="1"/> 207 <primitive id="child_DoFencing" class="stonith" type="external/vmware"> 208 <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith"> 209 <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith"> 210 <lrm_resource id="child_DoFencing:1" type="external/vmware" class="stonith"> 211 <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith"> 212 <lrm_resource id="child_DoFencing:2" type="external/vmware" class="stonith"> 213 <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith"> 214 <lrm_resource id="child_DoFencing:3" type="external/vmware" class="stonith"> 215 216If you wanted to delete the ``primitive`` tag with id ``child_DoFencing``, 217you would run: 218 219.. code-block:: none 220 221 # cibadmin --delete --xml-text '<primitive id="child_DoFencing"/>' 222 223See the cibadmin man page for more options. 224 225.. warning:: 226 227 Never edit the live ``cib.xml`` file directly. Pacemaker will detect such 228 changes and refuse to use the configuration. 229 230 231.. index:: 232 single: crm_shadow 233 single: command-line tool; crm_shadow 234 235.. _crm_shadow: 236 237Batch Configuration Changes with crm_shadow 238########################################### 239 240Often, it is desirable to preview the effects of a series of configuration 241changes before updating the live configuration all at once. For this purpose, 242``crm_shadow`` creates a "shadow" copy of the configuration and arranges for 243all the command-line tools to use it. 244 245To begin, simply invoke ``crm_shadow --create`` with a name of your choice, 246and follow the simple on-screen instructions. Shadow copies are identified with 247a name to make it possible to have more than one. 248 249.. warning:: 250 251 Read this section and the on-screen instructions carefully; failure to do so 252 could result in destroying the cluster's active configuration! 253 254.. topic:: Creating and displaying the active sandbox 255 256 .. code-block:: none 257 258 # crm_shadow --create test 259 Setting up shadow instance 260 Type Ctrl-D to exit the crm_shadow shell 261 shadow[test]: 262 shadow[test] # crm_shadow --which 263 test 264 265From this point on, all cluster commands will automatically use the shadow copy 266instead of talking to the cluster's active configuration. Once you have 267finished experimenting, you can either make the changes active via the 268``--commit`` option, or discard them using the ``--delete`` option. Again, be 269sure to follow the on-screen instructions carefully! 270 271For a full list of ``crm_shadow`` options and commands, invoke it with the 272``--help`` option. 273 274.. topic:: Use sandbox to make multiple changes all at once, discard them, and verify real configuration is untouched 275 276 .. code-block:: none 277 278 shadow[test] # crm_failcount -r rsc_c001n01 -G 279 scope=status name=fail-count-rsc_c001n01 value=0 280 shadow[test] # crm_standby --node c001n02 -v on 281 shadow[test] # crm_standby --node c001n02 -G 282 scope=nodes name=standby value=on 283 284 shadow[test] # cibadmin --erase --force 285 shadow[test] # cibadmin --query 286 <cib crm_feature_set="3.0.14" validate-with="pacemaker-3.0" epoch="112" num_updates="2" admin_epoch="0" cib-last-written="Mon Jan 8 23:26:47 2018" update-origin="rhel7-1" update-client="crm_node" update-user="root" have-quorum="1" dc-uuid="1"> 287 <configuration> 288 <crm_config/> 289 <nodes/> 290 <resources/> 291 <constraints/> 292 </configuration> 293 <status/> 294 </cib> 295 shadow[test] # crm_shadow --delete test --force 296 Now type Ctrl-D to exit the crm_shadow shell 297 shadow[test] # exit 298 # crm_shadow --which 299 No active shadow configuration defined 300 # cibadmin -Q 301 <cib crm_feature_set="3.0.14" validate-with="pacemaker-3.0" epoch="110" num_updates="2" admin_epoch="0" cib-last-written="Mon Jan 8 23:26:47 2018" update-origin="rhel7-1" update-client="crm_node" update-user="root" have-quorum="1"> 302 <configuration> 303 <crm_config> 304 <cluster_property_set id="cib-bootstrap-options"> 305 <nvpair id="cib-bootstrap-1" name="stonith-enabled" value="1"/> 306 <nvpair id="cib-bootstrap-2" name="pe-input-series-max" value="30000"/> 307 308See the next section, :ref:`crm_simulate`, for how to test your changes before 309committing them to the live cluster. 310 311 312.. index:: 313 single: crm_simulate 314 single: command-line tool; crm_simulate 315 316.. _crm_simulate: 317 318Simulate Cluster Activity with crm_simulate 319########################################### 320 321The command-line tool `crm_simulate` shows the results of the same logic 322the cluster itself uses to respond to a particular cluster configuration and 323status. 324 325As always, the man page is the primary documentation, and should be consulted 326for further details. This section aims for a better conceptual explanation and 327practical examples. 328 329Replaying cluster decision-making logic 330_______________________________________ 331 332At any given time, one node in a Pacemaker cluster will be elected DC, and that 333node will run Pacemaker's scheduler to make decisions. 334 335Each time decisions need to be made (a "transition"), the DC will have log 336messages like "Calculated transition ... saving inputs in ..." with a file 337name. You can grab the named file and replay the cluster logic to see why 338particular decisions were made. The file contains the live cluster 339configuration at that moment, so you can also look at it directly to see the 340value of node attributes, etc., at that time. 341 342The simplest usage is (replacing $FILENAME with the actual file name): 343 344.. topic:: Simulate cluster response to a given CIB 345 346 .. code-block:: none 347 348 # crm_simulate --simulate --xml-file $FILENAME 349 350That will show the cluster state when the process started, the actions that 351need to be taken ("Transition Summary"), and the resulting cluster state if the 352actions succeed. Most actions will have a brief description of why they were 353required. 354 355The transition inputs may be compressed. ``crm_simulate`` can handle these 356compressed files directly, though if you want to edit the file, you'll need to 357uncompress it first. 358 359You can do the same simulation for the live cluster configuration at the 360current moment. This is useful mainly when using ``crm_shadow`` to create a 361sandbox version of the CIB; the ``--live-check`` option will use the shadow CIB 362if one is in effect. 363 364.. topic:: Simulate cluster response to current live CIB or shadow CIB 365 366 .. code-block:: none 367 368 # crm_simulate --simulate --live-check 369 370 371Why decisions were made 372_______________________ 373 374To get further insight into the "why", it gets user-unfriendly very quickly. If 375you add the ``--show-scores`` option, you will also see all the scores that 376went into the decision-making. The node with the highest cumulative score for a 377resource will run it. You can look for ``-INFINITY`` scores in particular to 378see where complete bans came into effect. 379 380You can also add ``-VVVV`` to get more detailed messages about what's happening 381under the hood. You can add up to two more V's even, but that's usually useful 382only if you're a masochist or tracing through the source code. 383 384 385Visualizing the action sequence 386_______________________________ 387 388Another handy feature is the ability to generate a visual graph of the actions 389needed, using the ``--dot-file`` option. This relies on the separate 390Graphviz [#]_ project. 391 392.. topic:: Generate a visual graph of cluster actions from a saved CIB 393 394 .. code-block:: none 395 396 # crm_simulate --simulate --xml-file $FILENAME --dot-file $FILENAME.dot 397 # dot $FILENAME.dot -Tsvg > $FILENAME.svg 398 399``$FILENAME.dot`` will contain a GraphViz representation of the cluster's 400response to your changes, including all actions with their ordering 401dependencies. 402 403``$FILENAME.svg`` will be the same information in a standard graphical format 404that you can view in your browser or other app of choice. You could, of course, 405use other ``dot`` options to generate other formats. 406 407How to interpret the graphical output: 408 409 * Bubbles indicate actions, and arrows indicate ordering dependencies 410 * Resource actions have text of the form 411 ``<RESOURCE>_<ACTION>_<INTERVAL_IN_MS> <NODE>`` indicating that the 412 specified action will be executed for the specified resource on the 413 specified node, once if interval is 0 or at specified recurring interval 414 otherwise 415 * Actions with black text will be sent to the executor (that is, the 416 appropriate agent will be invoked) 417 * Actions with orange text are "pseudo" actions that the cluster uses 418 internally for ordering but require no real activity 419 * Actions with a solid green border are part of the transition (that is, the 420 cluster will attempt to execute them in the given order -- though a 421 transition can be interrupted by action failure or new events) 422 * Dashed arrows indicate dependencies that are not present in the transition 423 graph 424 * Actions with a dashed border will not be executed. If the dashed border is 425 blue, the cluster does not feel the action needs to be executed. If the 426 dashed border is red, the cluster would like to execute the action but 427 cannot. Any actions depending on an action with a dashed border will not be 428 able to execute. 429 * Loops should not happen, and should be reported as a bug if found. 430 431.. topic:: Small Cluster Transition 432 433 .. image:: ../shared/images/Policy-Engine-small.png 434 :alt: An example transition graph as represented by Graphviz 435 :align: center 436 437In the above example, it appears that a new node, ``pcmk-2``, has come online 438and that the cluster is checking to make sure ``rsc1``, ``rsc2`` and ``rsc3`` 439are not already running there (indicated by the ``rscN_monitor_0`` entries). 440Once it did that, and assuming the resources were not active there, it would 441have liked to stop ``rsc1`` and ``rsc2`` on ``pcmk-1`` and move them to 442``pcmk-2``. However, there appears to be some problem and the cluster cannot or 443is not permitted to perform the stop actions which implies it also cannot 444perform the start actions. For some reason, the cluster does not want to start 445``rsc3`` anywhere. 446 447.. topic:: Complex Cluster Transition 448 449 .. image:: ../shared/images/Policy-Engine-big.png 450 :alt: Complex transition graph that you're not expected to be able to read 451 :align: center 452 453 454What-if scenarios 455_________________ 456 457You can make changes to the saved or shadow CIB and simulate it again, to see 458how Pacemaker would react differently. You can edit the XML by hand, use 459command-line tools such as ``cibadmin`` with either a shadow CIB or the 460``CIB_file`` environment variable set to the filename, or use higher-level tool 461support (see the man pages of the specific tool you're using for how to perform 462actions on a saved CIB file rather than the live CIB). 463 464You can also inject node failures and/or action failures into the simulation; 465see the ``crm_simulate`` man page for more details. 466 467This capability is useful when using a shadow CIB to edit the configuration. 468Before committing the changes to the live cluster with ``crm_shadow --commit``, 469you can use ``crm_simulate`` to see how the cluster will react to the changes. 470 471.. _crm_attribute: 472 473.. index:: 474 single: attrd_updater 475 single: command-line tool; attrd_updater 476 single: crm_attribute 477 single: command-line tool; crm_attribute 478 479Manage Node Attributes, Cluster Options and Defaults with crm_attribute and attrd_updater 480######################################################################################### 481 482``crm_attribute`` and ``attrd_updater`` are confusingly similar tools with subtle 483differences. 484 485``attrd_updater`` can query and update node attributes. ``crm_attribute`` can query 486and update not only node attributes, but also cluster options, resource 487defaults, and operation defaults. 488 489To understand the differences, it helps to understand the various types of node 490attribute. 491 492.. table:: **Types of Node Attributes** 493 494 +-----------+----------+-------------------+------------------+----------------+----------------+ 495 | Type | Recorded | Recorded in | Survive full | Manageable by | Manageable by | 496 | | in CIB? | attribute manager | cluster restart? | crm_attribute? | attrd_updater? | 497 | | | memory? | | | | 498 +===========+==========+===================+==================+================+================+ 499 | permanent | yes | no | yes | yes | no | 500 +-----------+----------+-------------------+------------------+----------------+----------------+ 501 | transient | yes | yes | no | yes | yes | 502 +-----------+----------+-------------------+------------------+----------------+----------------+ 503 | private | no | yes | no | no | yes | 504 +-----------+----------+-------------------+------------------+----------------+----------------+ 505 506As you can see from the table above, ``crm_attribute`` can manage permanent and 507transient node attributes, while ``attrd_updater`` can manage transient and 508private node attributes. 509 510The difference between the two tools lies mainly in *how* they update node 511attributes: ``attrd_updater`` always contacts the Pacemaker attribute manager 512directly, while ``crm_attribute`` will contact the attribute manager only for 513transient node attributes, and will instead modify the CIB directly for 514permanent node attributes (and for transient node attributes when unable to 515contact the attribute manager). 516 517By contacting the attribute manager directly, ``attrd_updater`` can change 518an attribute's "dampening" (whether changes are immediately flushed to the CIB 519or after a specified amount of time, to minimize disk writes for frequent 520changes), set private node attributes (which are never written to the CIB), and 521set attributes for nodes that don't yet exist. 522 523By modifying the CIB directly, ``crm_attribute`` can set permanent node 524attributes (which are only in the CIB and not managed by the attribute 525manager), and can be used with saved CIB files and shadow CIBs. 526 527However a transient node attribute is set, it is synchronized between the CIB 528and the attribute manager, on all nodes. 529 530 531.. index:: 532 single: crm_failcount 533 single: command-line tool; crm_failcount 534 single: crm_node 535 single: command-line tool; crm_node 536 single: crm_report 537 single: command-line tool; crm_report 538 single: crm_standby 539 single: command-line tool; crm_standby 540 single: crm_verify 541 single: command-line tool; crm_verify 542 single: stonith_admin 543 single: command-line tool; stonith_admin 544 545Other Commonly Used Tools 546######################### 547 548Other command-line tools include: 549 550* ``crm_failcount``: query or delete resource fail counts 551* ``crm_node``: manage cluster nodes 552* ``crm_report``: generate a detailed cluster report for bug submissions 553* ``crm_resource``: manage cluster resources 554* ``crm_standby``: manage standby status of nodes 555* ``crm_verify``: validate a CIB 556* ``stonith_admin``: manage fencing devices 557 558See the manual pages for details. 559 560.. rubric:: Footnotes 561 562.. [#] Graph visualization software. See http://www.graphviz.org/ for details. 563