1.. _network_debug_troubleshooting: 2 3*************************************** 4Network Debug and Troubleshooting Guide 5*************************************** 6 7.. contents:: 8 :local: 9 10 11Introduction 12============ 13 14Starting with Ansible version 2.1, you can now use the familiar Ansible models of playbook authoring and module development to manage heterogeneous networking devices. Ansible supports a growing number of network devices using both CLI over SSH and API (when available) transports. 15 16This section discusses how to debug and troubleshoot network modules in Ansible 2.3. 17 18 19 20How to troubleshoot 21=================== 22 23This section covers troubleshooting issues with Network Modules. 24 25Errors generally fall into one of the following categories: 26 27:Authentication issues: 28 * Not correctly specifying credentials 29 * Remote device (network switch/router) not falling back to other other authentication methods 30 * SSH key issues 31:Timeout issues: 32 * Can occur when trying to pull a large amount of data 33 * May actually be masking a authentication issue 34:Playbook issues: 35 * Use of ``delegate_to``, instead of ``ProxyCommand``. See :ref:`network proxy guide <network_delegate_to_vs_ProxyCommand>` for more information. 36 37.. warning:: ``unable to open shell`` 38 39 The ``unable to open shell`` message is new in Ansible 2.3, it means that the ``ansible-connection`` daemon has not been able to successfully 40 talk to the remote network device. This generally means that there is an authentication issue. See the "Authentication and connection issues" section 41 in this document for more information. 42 43.. _enable_network_logging: 44 45Enabling Networking logging and how to read the logfile 46------------------------------------------------------- 47 48**Platforms:** Any 49 50Ansible 2.3 features improved logging to help diagnose and troubleshoot issues regarding Ansible Networking modules. 51 52Because logging is very verbose it is disabled by default. It can be enabled via the :envvar:`ANSIBLE_LOG_PATH` and :envvar:`ANSIBLE_DEBUG` options on the ansible-controller, that is the machine running ansible-playbook. 53 54Before running ``ansible-playbook`` run the following commands to enable logging:: 55 56 # Specify the location for the log file 57 export ANSIBLE_LOG_PATH=~/ansible.log 58 # Enable Debug 59 export ANSIBLE_DEBUG=True 60 61 # Run with 4*v for connection level verbosity 62 ansible-playbook -vvvv ... 63 64After Ansible has finished running you can inspect the log file which has been created on the ansible-controller: 65 66.. code:: 67 68 less $ANSIBLE_LOG_PATH 69 70 2017-03-30 13:19:52,740 p=28990 u=fred | creating new control socket for host veos01:22 as user admin 71 2017-03-30 13:19:52,741 p=28990 u=fred | control socket path is /home/fred/.ansible/pc/ca5960d27a 72 2017-03-30 13:19:52,741 p=28990 u=fred | current working directory is /home/fred/ansible/test/integration 73 2017-03-30 13:19:52,741 p=28990 u=fred | using connection plugin network_cli 74 ... 75 2017-03-30 13:20:14,771 paramiko.transport userauth is OK 76 2017-03-30 13:20:15,283 paramiko.transport Authentication (keyboard-interactive) successful! 77 2017-03-30 13:20:15,302 p=28990 u=fred | ssh connection done, setting terminal 78 2017-03-30 13:20:15,321 p=28990 u=fred | ssh connection has completed successfully 79 2017-03-30 13:20:15,322 p=28990 u=fred | connection established to veos01 in 0:00:22.580626 80 81 82From the log notice: 83 84* ``p=28990`` Is the PID (Process ID) of the ``ansible-connection`` process 85* ``u=fred`` Is the user `running` ansible, not the remote-user you are attempting to connect as 86* ``creating new control socket for host veos01:22 as user admin`` host:port as user 87* ``control socket path is`` location on disk where the persistent connection socket is created 88* ``using connection plugin network_cli`` Informs you that persistent connection is being used 89* ``connection established to veos01 in 0:00:22.580626`` Time taken to obtain a shell on the remote device 90 91 92.. note: Port None ``creating new control socket for host veos01:None`` 93 94 If the log reports the port as ``None`` this means that the default port is being used. 95 A future Ansible release will improve this message so that the port is always logged. 96 97Because the log files are verbose, you can use grep to look for specific information. For example, once you have identified the ``pid`` from the ``creating new control socket for host`` line you can search for other connection log entries:: 98 99 grep "p=28990" $ANSIBLE_LOG_PATH 100 101 102Enabling Networking device interaction logging 103---------------------------------------------- 104 105**Platforms:** Any 106 107Ansible 2.8 features added logging of device interaction in log file to help diagnose and troubleshoot 108issues regarding Ansible Networking modules. The messages are logged in file pointed by ``log_path`` configuration 109option in Ansible configuration file or by set :envvar:`ANSIBLE_LOG_PATH` as mentioned in above section. 110 111.. warning:: 112 The device interaction messages consist of command executed on target device and the returned response, as this 113 log data can contain sensitive information including passwords in plain text it is disabled by default. 114 Additionally, in order to prevent accidental leakage of data, a warning will be shown on every task with this 115 setting enabled specifying which host has it enabled and where the data is being logged. 116 117Be sure to fully understand the security implications of enabling this option. The device interaction logging can be enabled either globally by setting in configuration file or by setting environment or enabled on per task basis by passing special variable to task. 118 119Before running ``ansible-playbook`` run the following commands to enable logging:: 120 121 # Specify the location for the log file 122 export ANSIBLE_LOG_PATH=~/ansible.log 123 124 125Enable device interaction logging for a given task 126 127.. code-block:: yaml 128 129 - name: get version information 130 ios_command: 131 commands: 132 - show version 133 vars: 134 ansible_persistent_log_messages: True 135 136 137To make this a global setting, add the following to your ``ansible.cfg`` file: 138 139.. code-block:: ini 140 141 [persistent_connection] 142 log_messages = True 143 144or enable environment variable `ANSIBLE_PERSISTENT_LOG_MESSAGES` 145 146 # Enable device interaction logging 147 export ANSIBLE_PERSISTENT_LOG_MESSAGES=True 148 149If the task is failing at the time on connection initialization itself it is recommended to enable this option 150globally else if an individual task is failing intermittently this option can be enabled for that task itself to 151find the root cause. 152 153After Ansible has finished running you can inspect the log file which has been created on the ansible-controller 154 155.. note:: Be sure to fully understand the security implications of enabling this option as it can log sensitive 156 information in log file thus creating security vulnerability. 157 158 159Isolating an error 160------------------ 161 162**Platforms:** Any 163 164As with any effort to troubleshoot it's important to simplify the test case as much as possible. 165 166For Ansible this can be done by ensuring you are only running against one remote device: 167 168* Using ``ansible-playbook --limit switch1.example.net...`` 169* Using an ad-hoc ``ansible`` command 170 171`ad-hoc` refers to running Ansible to perform some quick command using ``/usr/bin/ansible``, rather than the orchestration language, which is ``/usr/bin/ansible-playbook``. In this case we can ensure connectivity by attempting to execute a single command on the remote device:: 172 173 ansible -m eos_command -a 'commands=?' -i inventory switch1.example.net -e 'ansible_connection=local' -u admin -k 174 175In the above example, we: 176 177* connect to ``switch1.example.net`` specified in the inventory file ``inventory`` 178* use the module ``eos_command`` 179* run the command ``?`` 180* connect using the username ``admin`` 181* inform ansible to prompt for the ssh password by specifying ``-k`` 182 183If you have SSH keys configured correctly, you don't need to specify the ``-k`` parameter 184 185If the connection still fails you can combine it with the enable_network_logging parameter. For example:: 186 187 # Specify the location for the log file 188 export ANSIBLE_LOG_PATH=~/ansible.log 189 # Enable Debug 190 export ANSIBLE_DEBUG=True 191 # Run with 4*v for connection level verbosity 192 ansible -m eos_command -a 'commands=?' -i inventory switch1.example.net -e 'ansible_connection=local' -u admin -k 193 194Then review the log file and find the relevant error message in the rest of this document. 195 196.. For details on other ways to authenticate, see LINKTOAUTHHOWTODOCS. 197 198.. _socket_path_issue: 199 200Category "socket_path issue" 201============================ 202 203**Platforms:** Any 204 205The ``socket_path does not exist or cannot be found`` and ``unable to connect to socket`` messages are new in Ansible 2.5. These messages indicate that the socket used to communicate with the remote network device is unavailable or does not exist. 206 207 208For example: 209 210.. code-block:: none 211 212 fatal: [spine02]: FAILED! => { 213 "changed": false, 214 "failed": true, 215 "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_TSqk5J/ansible_modlib.zip/ansible/module_utils/connection.py\", line 115, in _exec_jsonrpc\nansible.module_utils.connection.ConnectionError: socket_path does not exist or cannot be found\n", 216 "module_stdout": "", 217 "msg": "MODULE FAILURE", 218 "rc": 1 219 } 220 221or 222 223.. code-block:: none 224 225 fatal: [spine02]: FAILED! => { 226 "changed": false, 227 "failed": true, 228 "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_TSqk5J/ansible_modlib.zip/ansible/module_utils/connection.py\", line 123, in _exec_jsonrpc\nansible.module_utils.connection.ConnectionError: unable to connect to socket\n", 229 "module_stdout": "", 230 "msg": "MODULE FAILURE", 231 "rc": 1 232 } 233 234Suggestions to resolve: 235 236Follow the steps detailed in :ref:`enable network logging <enable_network_logging>`. 237 238If the identified error message from the log file is: 239 240.. code-block:: yaml 241 242 2017-04-04 12:19:05,670 p=18591 u=fred | command timeout triggered, timeout value is 30 secs 243 244or 245 246.. code-block:: yaml 247 248 2017-04-04 12:19:05,670 p=18591 u=fred | persistent connection idle timeout triggered, timeout value is 30 secs 249 250Follow the steps detailed in :ref:`timeout issues <timeout_issues>` 251 252 253.. _unable_to_open_shell: 254 255Category "Unable to open shell" 256=============================== 257 258 259**Platforms:** Any 260 261The ``unable to open shell`` message is new in Ansible 2.3. This message means that the ``ansible-connection`` daemon has not been able to successfully talk to the remote network device. This generally means that there is an authentication issue. It is a "catch all" message, meaning you need to enable :ref:logging`a_note_about_logging` to find the underlying issues. 262 263 264 265For example: 266 267.. code-block:: none 268 269 TASK [prepare_eos_tests : enable cli on remote device] ************************************************** 270 fatal: [veos01]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell"} 271 272 273or: 274 275 276.. code-block:: none 277 278 TASK [ios_system : configure name_servers] ************************************************************* 279 task path: 280 fatal: [ios-csr1000v]: FAILED! => { 281 "changed": false, 282 "failed": true, 283 "msg": "unable to open shell", 284 } 285 286Suggestions to resolve: 287 288Follow the steps detailed in enable_network_logging_. 289 290Once you've identified the error message from the log file, the specific solution can be found in the rest of this document. 291 292 293 294Error: "[Errno -2] Name or service not known" 295--------------------------------------------- 296 297**Platforms:** Any 298 299Indicates that the remote host you are trying to connect to can not be reached 300 301For example: 302 303.. code-block:: yaml 304 305 2017-04-04 11:39:48,147 p=15299 u=fred | control socket path is /home/fred/.ansible/pc/ca5960d27a 306 2017-04-04 11:39:48,147 p=15299 u=fred | current working directory is /home/fred/git/ansible-inc/stable-2.3/test/integration 307 2017-04-04 11:39:48,147 p=15299 u=fred | using connection plugin network_cli 308 2017-04-04 11:39:48,340 p=15299 u=fred | connecting to host veos01 returned an error 309 2017-04-04 11:39:48,340 p=15299 u=fred | [Errno -2] Name or service not known 310 311 312Suggestions to resolve: 313 314* If you are using the ``provider:`` options ensure that its suboption ``host:`` is set correctly. 315* If you are not using ``provider:`` nor top-level arguments ensure your inventory file is correct. 316 317 318 319 320 321Error: "Authentication failed" 322------------------------------ 323 324**Platforms:** Any 325 326Occurs if the credentials (username, passwords, or ssh keys) passed to ``ansible-connection`` (via ``ansible`` or ``ansible-playbook``) can not be used to connect to the remote device. 327 328 329 330For example: 331 332.. code-block:: yaml 333 334 <ios01> ESTABLISH CONNECTION FOR USER: cisco on PORT 22 TO ios01 335 <ios01> Authentication failed. 336 337 338Suggestions to resolve: 339 340If you are specifying credentials via ``password:`` (either directly or via ``provider:``) or the environment variable `ANSIBLE_NET_PASSWORD` it is possible that ``paramiko`` (the Python SSH library that Ansible uses) is using ssh keys, and therefore the credentials you are specifying are being ignored. To find out if this is the case, disable "look for keys". This can be done like this: 341 342.. code-block:: yaml 343 344 export ANSIBLE_PARAMIKO_LOOK_FOR_KEYS=False 345 346To make this a permanent change, add the following to your ``ansible.cfg`` file: 347 348.. code-block:: ini 349 350 [paramiko_connection] 351 look_for_keys = False 352 353 354Error: "connecting to host <hostname> returned an error" or "Bad address" 355------------------------------------------------------------------------- 356 357This may occur if the SSH fingerprint hasn't been added to Paramiko's (the Python SSH library) know hosts file. 358 359When using persistent connections with Paramiko, the connection runs in a background process. If the host doesn't already have a valid SSH key, by default Ansible will prompt to add the host key. This will cause connections running in background processes to fail. 360 361For example: 362 363.. code-block:: yaml 364 365 2017-04-04 12:06:03,486 p=17981 u=fred | using connection plugin network_cli 366 2017-04-04 12:06:04,680 p=17981 u=fred | connecting to host veos01 returned an error 367 2017-04-04 12:06:04,682 p=17981 u=fred | (14, 'Bad address') 368 2017-04-04 12:06:33,519 p=17981 u=fred | number of connection attempts exceeded, unable to connect to control socket 369 2017-04-04 12:06:33,520 p=17981 u=fred | persistent_connect_interval=1, persistent_connect_retries=30 370 371 372Suggestions to resolve: 373 374Use ``ssh-keyscan`` to pre-populate the known_hosts. You need to ensure the keys are correct. 375 376.. code-block:: shell 377 378 ssh-keyscan veos01 379 380 381or 382 383You can tell Ansible to automatically accept the keys 384 385Environment variable method:: 386 387 export ANSIBLE_PARAMIKO_HOST_KEY_AUTO_ADD=True 388 ansible-playbook ... 389 390``ansible.cfg`` method: 391 392ansible.cfg 393 394.. code-block:: ini 395 396 [paramiko_connection] 397 host_key_auto_add = True 398 399 400 401.. warning: Security warning 402 403 Care should be taken before accepting keys. 404 405Error: "No authentication methods available" 406-------------------------------------------- 407 408For example: 409 410.. code-block:: yaml 411 412 2017-04-04 12:19:05,670 p=18591 u=fred | creating new control socket for host veos01:None as user admin 413 2017-04-04 12:19:05,670 p=18591 u=fred | control socket path is /home/fred/.ansible/pc/ca5960d27a 414 2017-04-04 12:19:05,670 p=18591 u=fred | current working directory is /home/fred/git/ansible-inc/ansible-workspace-2/test/integration 415 2017-04-04 12:19:05,670 p=18591 u=fred | using connection plugin network_cli 416 2017-04-04 12:19:06,606 p=18591 u=fred | connecting to host veos01 returned an error 417 2017-04-04 12:19:06,606 p=18591 u=fred | No authentication methods available 418 2017-04-04 12:19:35,708 p=18591 u=fred | connect retry timeout expired, unable to connect to control socket 419 2017-04-04 12:19:35,709 p=18591 u=fred | persistent_connect_retry_timeout is 15 secs 420 421 422Suggestions to resolve: 423 424No password or SSH key supplied 425 426Clearing Out Persistent Connections 427----------------------------------- 428 429**Platforms:** Any 430 431In Ansible 2.3, persistent connection sockets are stored in ``~/.ansible/pc`` for all network devices. When an Ansible playbook runs, the persistent socket connection is displayed when verbose output is specified. 432 433``<switch> socket_path: /home/fred/.ansible/pc/f64ddfa760`` 434 435To clear out a persistent connection before it times out (the default timeout is 30 seconds 436of inactivity), simple delete the socket file. 437 438 439.. _timeout_issues: 440 441Timeout issues 442============== 443 444Persistent connection idle timeout 445---------------------------------- 446 447By default, ``ANSIBLE_PERSISTENT_CONNECT_TIMEOUT`` is set to 30 (seconds). You may see the following error if this value is too low: 448 449.. code-block:: yaml 450 451 2017-04-04 12:19:05,670 p=18591 u=fred | persistent connection idle timeout triggered, timeout value is 30 secs 452 453Suggestions to resolve: 454 455Increase value of persistent connection idle timeout: 456 457.. code-block:: sh 458 459 export ANSIBLE_PERSISTENT_CONNECT_TIMEOUT=60 460 461To make this a permanent change, add the following to your ``ansible.cfg`` file: 462 463.. code-block:: ini 464 465 [persistent_connection] 466 connect_timeout = 60 467 468Command timeout 469--------------- 470 471By default, ``ANSIBLE_PERSISTENT_COMMAND_TIMEOUT`` is set to 30 (seconds). Prior versions of Ansible had this value set to 10 seconds by default. 472You may see the following error if this value is too low: 473 474 475.. code-block:: yaml 476 477 2017-04-04 12:19:05,670 p=18591 u=fred | command timeout triggered, timeout value is 30 secs 478 479Suggestions to resolve: 480 481* Option 1 (Global command timeout setting): 482 Increase value of command timeout in configuration file or by setting environment variable. 483 484 .. code-block:: yaml 485 486 export ANSIBLE_PERSISTENT_COMMAND_TIMEOUT=60 487 488 To make this a permanent change, add the following to your ``ansible.cfg`` file: 489 490 .. code-block:: ini 491 492 [persistent_connection] 493 command_timeout = 60 494 495* Option 2 (Per task command timeout setting): 496 Increase command timeout per task basis. All network modules support a 497 timeout value that can be set on a per task basis. 498 The timeout value controls the amount of time in seconds before the 499 task will fail if the command has not returned. 500 501 For local connection type: 502 503 .. FIXME: Detail error here 504 505 Suggestions to resolve: 506 507 .. code-block:: yaml 508 509 - name: save running-config 510 ios_command: 511 commands: copy running-config startup-config 512 provider: "{{ cli }}" 513 timeout: 30 514 515 For network_cli, netconf connection type (applicable from 2.7 onwards): 516 517 .. FIXME: Detail error here 518 519 Suggestions to resolve: 520 521 .. code-block:: yaml 522 523 - name: save running-config 524 ios_command: 525 commands: copy running-config startup-config 526 vars: 527 ansible_command_timeout: 60 528 529Some operations take longer than the default 30 seconds to complete. One good 530example is saving the current running config on IOS devices to startup config. 531In this case, changing the timeout value from the default 30 seconds to 60 532seconds will prevent the task from failing before the command completes 533successfully. 534 535Persistent connection retry timeout 536----------------------------------- 537 538By default, ``ANSIBLE_PERSISTENT_CONNECT_RETRY_TIMEOUT`` is set to 15 (seconds). You may see the following error if this value is too low: 539 540.. code-block:: yaml 541 542 2017-04-04 12:19:35,708 p=18591 u=fred | connect retry timeout expired, unable to connect to control socket 543 2017-04-04 12:19:35,709 p=18591 u=fred | persistent_connect_retry_timeout is 15 secs 544 545Suggestions to resolve: 546 547Increase the value of the persistent connection idle timeout. 548Note: This value should be greater than the SSH timeout value (the timeout value under the defaults 549section in the configuration file) and less than the value of the persistent 550connection idle timeout (connect_timeout). 551 552.. code-block:: yaml 553 554 export ANSIBLE_PERSISTENT_CONNECT_RETRY_TIMEOUT=30 555 556To make this a permanent change, add the following to your ``ansible.cfg`` file: 557 558.. code-block:: ini 559 560 [persistent_connection] 561 connect_retry_timeout = 30 562 563 564Timeout issue due to platform specific login menu with ``network_cli`` connection type 565-------------------------------------------------------------------------------------- 566 567In Ansible 2.9 and later, the network_cli connection plugin configuration options are added 568to handle the platform specific login menu. These options can be set as group/host or tasks 569variables. 570 571Example: Handle single login menu prompts with host variables 572 573.. code-block:: console 574 575 $cat host_vars/<hostname>.yaml 576 --- 577 ansible_terminal_initial_prompt: 578 - "Connect to a host" 579 ansible_terminal_initial_answer: 580 - "3" 581 582Example: Handle remote host multiple login menu prompts with host variables 583 584.. code-block:: console 585 586 $cat host_vars/<inventory-hostname>.yaml 587 --- 588 ansible_terminal_initial_prompt: 589 - "Press any key to enter main menu" 590 - "Connect to a host" 591 ansible_terminal_initial_answer: 592 - "\\r" 593 - "3" 594 ansible_terminal_initial_prompt_checkall: True 595 596To handle multiple login menu prompts: 597 598* The values of ``ansible_terminal_initial_prompt`` and ``ansible_terminal_initial_answer`` should be a list. 599* The prompt sequence should match the answer sequence. 600* The value of ``ansible_terminal_initial_prompt_checkall`` should be set to ``True``. 601 602.. note:: If all the prompts in sequence are not received from remote host at the time connection initialization it will result in a timeout. 603 604 605Playbook issues 606=============== 607 608This section details issues are caused by issues with the Playbook itself. 609 610Error: "Unable to enter configuration mode" 611------------------------------------------- 612 613**Platforms:** eos and ios 614 615This occurs when you attempt to run a task that requires privileged mode in a user mode shell. 616 617For example: 618 619.. code-block:: console 620 621 TASK [ios_system : configure name_servers] ***************************************************************************** 622 task path: 623 fatal: [ios-csr1000v]: FAILED! => { 624 "changed": false, 625 "failed": true, 626 "msg": "unable to enter configuration mode", 627 } 628 629Suggestions to resolve: 630 631In Ansible prior to 2.5 : 632Add ``authorize: yes`` to the task. For example: 633 634.. code-block:: yaml 635 636 - name: configure hostname 637 ios_system: 638 provider: 639 hostname: foo 640 authorize: yes 641 register: result 642 643If the user requires a password to go into privileged mode, this can be specified with ``auth_pass``; if ``auth_pass`` isn't set, the environment variable `ANSIBLE_NET_AUTHORIZE` will be used instead. 644 645 646Add ``authorize: yes`` to the task. For example: 647 648.. code-block:: yaml 649 650 - name: configure hostname 651 ios_system: 652 provider: 653 hostname: foo 654 authorize: yes 655 auth_pass: "{{ mypasswordvar }}" 656 register: result 657 658 659.. note:: Starting with Ansible 2.5 we recommend using ``connection: network_cli`` and ``become: yes`` 660 661 662Proxy Issues 663============ 664 665 .. _network_delegate_to_vs_ProxyCommand: 666 667delegate_to vs ProxyCommand 668--------------------------- 669 670The new connection framework for Network Modules in Ansible 2.3 that uses ``cli`` transport 671no longer supports the use of the ``delegate_to`` directive. 672In order to use a bastion or intermediate jump host to connect to network devices over ``cli`` 673transport, network modules now support the use of ``ProxyCommand``. 674 675To use ``ProxyCommand``, configure the proxy settings in the Ansible inventory 676file to specify the proxy host. 677 678.. code-block:: ini 679 680 [nxos] 681 nxos01 682 nxos02 683 684 [nxos:vars] 685 ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q bastion01"' 686 687 688With the configuration above, simply build and run the playbook as normal with 689no additional changes necessary. The network module will now connect to the 690network device by first connecting to the host specified in 691``ansible_ssh_common_args``, which is ``bastion01`` in the above example. 692 693You can also set the proxy target for all hosts by using environment variables. 694 695.. code-block:: sh 696 697 export ANSIBLE_SSH_ARGS='-o ProxyCommand="ssh -W %h:%p -q bastion01"' 698 699Using bastion/jump host with netconf connection 700----------------------------------------------- 701 702Enabling jump host setting 703-------------------------- 704 705 706Bastion/jump host with netconf connection can be enabled by: 707 - Setting Ansible variable ``ansible_netconf_ssh_config`` either to ``True`` or custom ssh config file path 708 - Setting environment variable ``ANSIBLE_NETCONF_SSH_CONFIG`` to ``True`` or custom ssh config file path 709 - Setting ``ssh_config = 1`` or ``ssh_config = <ssh-file-path>`` under ``netconf_connection`` section 710 711If the configuration variable is set to 1 the proxycommand and other ssh variables are read from 712default ssh config file (~/.ssh/config). 713 714If the configuration variable is set to file path the proxycommand and other ssh variables are read 715from the given custom ssh file path 716 717Example ssh config file (~/.ssh/config) 718--------------------------------------- 719 720.. code-block:: ini 721 722 Host jumphost 723 HostName jumphost.domain.name.com 724 User jumphost-user 725 IdentityFile "/path/to/ssh-key.pem" 726 Port 22 727 728 # Note: Due to the way that Paramiko reads the SSH Config file, 729 # you need to specify the NETCONF port that the host uses. 730 # i.e. It does not automatically use ansible_port 731 # As a result you need either: 732 733 Host junos01 734 HostName junos01 735 ProxyCommand ssh -W %h:22 jumphost 736 737 # OR 738 739 Host junos01 740 HostName junos01 741 ProxyCommand ssh -W %h:830 jumphost 742 743 # Depending on the netconf port used. 744 745Example Ansible inventory file 746 747.. code-block:: ini 748 749 [junos] 750 junos01 751 752 [junos:vars] 753 ansible_connection=netconf 754 ansible_network_os=junos 755 ansible_user=myuser 756 ansible_password=!vault... 757 758 759.. note:: Using ``ProxyCommand`` with passwords via variables 760 761 By design, SSH doesn't support providing passwords via environment variables. 762 This is done to prevent secrets from leaking out, for example in ``ps`` output. 763 764 We recommend using SSH Keys, and if needed an ssh-agent, rather than passwords, where ever possible. 765 766Miscellaneous Issues 767==================== 768 769 770Intermittent failure while using ``network_cli`` connection type 771---------------------------------------------------------------- 772 773If the command prompt received in response is not matched correctly within 774the ``network_cli`` connection plugin the task might fail intermittently with truncated 775response or with the error message ``operation requires privilege escalation``. 776Starting in 2.7.1 a new buffer read timer is added to ensure prompts are matched properly 777and a complete response is send in output. The timer default value is 0.2 seconds and 778can be adjusted on a per task basis or can be set globally in seconds. 779 780Example Per task timer setting 781 782.. code-block:: yaml 783 784 - name: gather ios facts 785 ios_facts: 786 gather_subset: all 787 register: result 788 vars: 789 ansible_buffer_read_timeout: 2 790 791 792To make this a global setting, add the following to your ``ansible.cfg`` file: 793 794.. code-block:: ini 795 796 [persistent_connection] 797 buffer_read_timeout = 2 798 799This timer delay per command executed on remote host can be disabled by setting the value to zero. 800 801 802Task failure due to mismatched error regex within command response using ``network_cli`` connection type 803-------------------------------------------------------------------------------------------------------- 804 805In Ansible 2.9 and later, the network_cli connection plugin configuration options are added 806to handle the stdout and stderr regex to identify if the command execution response consist 807of a normal response or an error response. These options can be set group/host variables or as 808tasks variables. 809 810Example: For mismatched error response 811 812.. code-block:: yaml 813 814 - name: fetch logs from remote host 815 ios_command: 816 commands: 817 - show logging 818 819 820Playbook run output: 821 822.. code-block:: console 823 824 TASK [first fetch logs] ******************************************************** 825 fatal: [ios01]: FAILED! => { 826 "changed": false, 827 "msg": "RF Name:\r\n\r\n <--nsip--> 828 \"IPSEC-3-REPLAY_ERROR: Test log\"\r\n*Aug 1 08:36:18.483: %SYS-7-USERLOG_DEBUG: 829 Message from tty578(user id: ansible): test\r\nan-ios-02#"} 830 831Suggestions to resolve: 832 833Modify the error regex for individual task. 834 835.. code-block:: yaml 836 837 - name: fetch logs from remote host 838 ios_command: 839 commands: 840 - show logging 841 vars: 842 ansible_terminal_stderr_re: 843 - pattern: 'connection timed out' 844 flags: 're.I' 845 846The terminal plugin regex options ``ansible_terminal_stderr_re`` and ``ansible_terminal_stdout_re`` have 847``pattern`` and ``flags`` as keys. The value of the ``flags`` key should be a value that is accepted by 848the ``re.compile`` python method. 849 850 851Intermittent failure while using ``network_cli`` connection type due to slower network or remote target host 852------------------------------------------------------------------------------------------------------------ 853 854In Ansible 2.9 and later, the ``network_cli`` connection plugin configuration option is added to control 855the number of attempts to connect to a remote host. The default number of attempts is three. 856After every retry attempt the delay between retries is increased by power of 2 in seconds until either the 857maximum attempts are exhausted or either the ``persistent_command_timeout`` or ``persistent_connect_timeout`` timers are triggered. 858 859To make this a global setting, add the following to your ``ansible.cfg`` file: 860 861.. code-block:: ini 862 863 [persistent_connection] 864 network_cli_retries = 5 865