1# Linux Minidump to Core 2 3On Linux, Chromium can use Breakpad to generate minidump files for crashes. It 4is possible to convert the minidump files to core files, and examine the core 5file in gdb, cgdb, or Qtcreator. In the examples below cgdb is assumed but any 6gdb based debugger can be used. 7 8[TOC] 9 10## Creating the core file 11 12Use `minidump-2-core` to convert the minidump file to a core file. On Linux, one 13can build the minidump-2-core target in a Chromium checkout, or alternatively, 14build it in a Google Breakpad checkout. 15 16```shell 17$ ninja -C out/Release minidump-2-core 18$ ./out/Release/minidump-2-core foo.dmp > foo.core 19``` 20 21## Retrieving Chrome binaries 22 23If the minidump is from a public build then Googlers can find Google Chrome 24Linux binaries and debugging symbols via https://goto.google.com/chromesymbols. 25Otherwise, use the locally built chrome files. Google Chrome uses the 26_debug link_ method to specify the debugging file. Either way be sure to put 27`chrome` and `chrome.debug` (the stripped debug information) in the same 28directory as the core file so that the debuggers can find them. 29 30For Chrome OS release binaries look for `debug-*.tgz` files on 31GoldenEye. 32 33## Loading the core file into gdb/cgdb 34 35The recommended syntax for loading a core file into gdb/cgdb is as follows, 36specifying both the executable and the core file: 37 38```shell 39$ cgdb chrome foo.core 40``` 41 42If the executable is not available then the core file can be loaded on its own 43but debugging options will be limited: 44 45```shell 46$ cgdb -c foo.core 47``` 48 49## Loading the core file into Qtcreator 50 51Qtcreator is a full GUI wrapper for gdb and it can also load Chrome's core 52files. From Qtcreator select the Debug menu, Start Debugging, Load Core File... 53and then enter the paths to the core file and executable. Qtcreator has windows 54to display the call stack, locals, registers, etc. For more information on 55debugging with Qtcreator see 56[Getting Started Debugging on Linux.](https://www.youtube.com/watch?v=xTmAknUbpB0) 57 58## Source debugging 59 60If you have a Chromium repo that is synchronized to exactly (or even 61approximately) when the Chrome build was created then you can tell 62`gdb/cgdb/Qtcreator` to load source code. Since all source paths in Chrome are 63relative to the out/Release directory you just need to add that directory to 64your debugger search path, by adding a line similar to this to `~/.gdbinit`: 65 66``` 67(gdb) directory /usr/local/chromium/src/out/Release/ 68``` 69 70## Notes 71 72* Since the core file is created from a minidump, it is incomplete and the 73 debugger may not know values for variables in memory. Minidump files contain 74 thread stacks so local variables and function parameters should be 75 available, subject to the limitations of optimized builds. 76* For gdb's `add-symbol-file` command to work, the file must have debugging 77 symbols. 78 * In case of separate debug files, 79 [the gdb manual](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html) 80 explains how gdb looks for them. 81* If the stack trace involve system libraries, the Advanced module loading 82 steps shown below need to be repeated for each library. 83 84## Advanced module loading 85 86If gdb doesn't find shared objects that are needed you can force it to load 87them. In gdb, the `add-symbol-file` command takes a filename and an address. To 88figure out the address, look near the end of `foo.dmp`, which contains a copy of 89`/proc/pid/maps` from the process that crashed. 90 91One quick way to do this is with `grep`. For instance, if the executable is 92`/path/to/chrome`, one can simply run: 93 94```shell 95$ grep -a /path/to/chrome$ foo.dmp 96 977fe749a90000-7fe74d28f000 r-xp 00000000 08:07 289158 /path/to/chrome 987fe74d290000-7fe74d4b7000 r--p 037ff000 08:07 289158 /path/to/chrome 997fe74d4b7000-7fe74d4e0000 rw-p 03a26000 08:07 289158 /path/to/chrome 100``` 101 102 103In this case, `7fe749a90000` is the base address for `/path/to/chrome`, but gdb 104takes the start address of the file's text section. To calculate this, one will 105need a copy of `/path/to/chrome`, and run: 106 107```shell 108$ objdump -x /path/to/chrome | grep '\.text' | head -n 1 | tr -s ' ' | \ 109 cut -d' ' -f 7 110 111005282c0 112``` 113 114 115Now add the two addresses: `7fe749a90000 + 005282c0 = 7fe749fb82c0` and in gdb, run: 116 117``` 118(gdb) add-symbol-file /path/to/chrome 0x7fe749fb82c0 119``` 120 121Then use gdb as normal. 122 123## Other resources 124 125For more discussion on this process see 126[Debugging a Minidump]. 127This page discusses the same process in the context of Chrome OS and many of the 128concepts and techniques overlap. 129 130[Debugging a Minidump]( 131https://www.chromium.org/chromium-os/packages/crash-reporting/debugging-a-minidump) 132