1Debugging 2========= 3 4:index:`\ <single: Crash>`\ :index:`\ <single: Debug; crash>`\ 5 6This chapter describes, how to debug Bareos, when the program crashes. If you are just interested about how to get more information about a running Bareos daemon, please read :ref:`section-debug-messages`. 7 8If you are running on a Linux system, and you have a set of working configuration files, it is very unlikely that Bareos will crash. As with all software, however, it is inevitable that someday, it may crash. 9 10This chapter explains what you should do if one of the three Bareos daemons (Director, File, Storage) crashes. When we speak of crashing, we mean that the daemon terminates abnormally because of an error. There are many cases where Bareos detects errors (such as PIPE errors) and will fail a job. These are not considered crashes. In addition, under certain conditions, Bareos will detect a fatal in the configuration, such as lack of permission to read/write the working directory. In that case, 11Bareos will force itself to crash with a SEGFAULT. However, before crashing, Bareos will normally display a message indicating why. For more details, please read on. 12 13Traceback 14--------- 15 16:index:`\ <single: Traceback>`\ 17 18Each of the three Bareos daemons has a built-in exception handler which, in case of an error, will attempt to produce a traceback. If successful the traceback will be emailed to you. 19 20For this to work, you need to ensure that a few things are setup correctly on your system: 21 22#. You must have a version of Bareos with debug information and not stripped of debugging symbols. When using a packaged version of Bareos, this requires to install the Bareos debug packages (**bareos-debug** on RPM based systems, **bareos-dbg** on Debian based systems). 23 24#. On Linux, :command:`gdb` (the GNU debugger) must be installed. On some systems such as Solaris, :command:`gdb` may be replaced by :command:`dbx`. 25 26#. By default, btraceback uses :command:`bsmtp` to send the traceback via email. Therefore it expects a local mail transfer daemon running. It send the traceback to root@localhost via :strong:`localhost`. 27 28#. Some Linux distributions, e.g. Ubuntu:index:`\ <single: Platform; Ubuntu; Debug>`\ , disable the possibility to examine the memory of other processes. While this is a good idea for hardening a system, our debug mechanismen will fail. To disable this feature, run (as root): 29 30 .. code-block:: shell-session 31 :caption: disable ptrace protection to enable debugging (required on Ubuntu Linux) 32 33 <command> </command><parameter>test -e /proc/sys/kernel/yama/ptrace_scope && echo 0 > /proc/sys/kernel/yama/ptrace_scope</parameter> 34 35If all the above conditions are met, the daemon that crashes will produce a traceback report and email it. If the above conditions are not true, you can run the debugger by hand as described below. 36 37Testing The Traceback 38--------------------- 39 40:index:`\ <single: Traceback; Test>`\ 41 42To "manually" test the traceback feature, you simply start Bareos then obtain the PID of the main daemon thread (there are multiple threads). The output produced here will look different depending on what OS and what version of the kernel you are running. 43 44.. code-block:: shell-session 45 :caption: get the process ID of a running Bareos daemon 46 47 <command> </command><parameter>ps fax | grep bareos-dir</parameter> 48 2103 ? S 0:00 /usr/sbin/bareos-dir 49 50which in this case is 2103. Then while Bareos is running, you call the program giving it the path to the Bareos executable and the PID. In this case, it is: 51 52.. code-block:: shell-session 53 :caption: get traceback of running Bareos director daemon 54 55 <command> </command><parameter>btraceback /usr/sbin/bareos-dir 2103</parameter> 56 57It should produce an email showing you the current state of the daemon (in this case the Director), and then exit leaving Bareos running as if nothing happened. If this is not the case, you will need to correct the problem by modifying the :command:`btraceback` script. 58 59Getting A Traceback On Other Systems 60~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 61 62It should be possible to produce a similar traceback on systems other than Linux, either using :command:`gdb` or some other debugger. Solaris:index:`\ <single: Platform; Solaris; Debug>`\ with :command:`dbx` loaded works quite fine. On other systems, you will need to modify the :command:`btraceback` program to invoke the correct debugger, and possibly correct the :file:`btraceback.gdb` script to have appropriate commands for your debugger. 63Please keep in mind that for any debugger to work, it will most likely need to run as root. 64 65Manually Running Bareos Under The Debugger 66------------------------------------------ 67 68If for some reason you cannot get the automatic traceback, or if you want to interactively examine the variable contents after a crash, you can run Bareos under the debugger. Assuming you want to run the Storage daemon under the debugger (the technique is the same for the other daemons, only the name changes), you would do the following: 69 70#. The Director and the File daemon should be running but the Storage daemon should not. 71 72#. Start the Storage daemon under the debugger: 73 74 .. code-block:: shell-session 75 :caption: run the Bareos Storage daemon in the debugger 76 77 <command>gdb</command><parameter> --args /usr/sbin/bareos-sd -f -s -d 200</parameter> 78 (gdb) <input>run</input> 79 80 Parameter: 81 82 -f 83 foreground 84 85 -s 86 no signals 87 88 -d nnn 89 debug level 90 91 See section :ref:`daemon command line options <section-daemon-command-line-options>` for a detailed list of options. 92 93#. At this point, Bareos will be fully operational. 94 95#. In another shell command window, start the Console program and do what is necessary to cause Bareos to die. 96 97#. When Bareos crashes, the gdb shell window will become active and gdb will show you the error that occurred. 98 99#. To get a general traceback of all threads, issue the following command: 100 101 .. code-block:: shell-session 102 :caption: run the Bareos Storage daemon in the debugger 103 104 (gdb) <input>thread apply all bt</input> 105 106 After that you can issue any debugging command. 107 108 109 110 111