1# Tips for debugging on Linux 2 3This page is for Chromium-specific debugging tips; learning how to run gdb is 4out of scope. 5 6[TOC] 7 8## Symbolized stack trace 9 10The sandbox can interfere with the internal symbolizer. Use `--no-sandbox` (but 11keep this temporary) or an external symbolizer (see 12`tools/valgrind/asan/asan_symbolize.py`). 13 14Generally, do not use `--no-sandbox` on waterfall bots, sandbox testing is 15needed. Talk to security@chromium.org. 16 17## GDB 18 19*** promo 20GDB-7.7 is required in order to debug Chrome on Linux. 21*** 22 23Any prior version will fail to resolve symbols or segfault. 24 25### Basic browser process debugging 26 27 gdb -tui -ex=r --args out/Debug/chrome --disable-seccomp-sandbox \ 28 http://google.com 29 30### Allowing attaching to foreign processes 31 32On distributions that use the 33[Yama LSM](https://www.kernel.org/doc/Documentation/security/Yama.txt) (that 34includes Ubuntu and Chrome OS), process A can attach to process B only if A is 35an ancestor of B. 36 37You will probably want to disable this feature by using 38 39 echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope 40 41If you don't you'll get an error message such as "Could not attach to process". 42 43Note that you'll also probably want to use `--no-sandbox`, as explained below. 44 45### Multiprocess Tricks 46 47#### Getting renderer subprocesses into gdb 48 49Since Chromium itself spawns the renderers, it can be tricky to grab a 50particular with gdb. This command does the trick: 51 52``` 53chrome --no-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb --args' 54``` 55 56The `--no-sandbox` flag is needed because otherwise the seccomp sandbox will 57kill the renderer process on startup, or the setuid sandbox will prevent xterm's 58execution. The "xterm" is necessary or gdb will run in the current terminal, 59which can get particularly confusing since it's running in the background, and 60if you're also running the main process in gdb, won't work at all (the two 61instances will fight over the terminal). To auto-start the renderers in the 62debugger, send the "run" command to the debugger: 63 64 chrome --no-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb \ 65 -ex run --args 66 67If you're using Emacs and `M-x gdb`, you can do 68 69 chrome "--renderer-cmd-prefix=gdb --args" 70 71*** note 72Note: using the `--renderer-cmd-prefix` option bypasses the zygote launcher, so 73the renderers won't be sandboxed. It is generally not an issue, except when you 74are trying to debug interactions with the sandbox. If that's what you are doing, 75you will need to attach your debugger to a running renderer process (see below). 76*** 77 78You may also want to pass `--disable-hang-monitor` to suppress the hang monitor, 79which is rather annoying. 80 81You can also use `--renderer-startup-dialog` and attach to the process in order 82to debug the renderer code. Go to 83https://www.chromium.org/blink/getting-started-with-blink-debugging for more 84information on how this can be done. 85 86#### Choosing which renderers to debug 87 88If you are starting multiple renderers then the above means that multiple gdb's 89start and fight over the console. Instead, you can set the prefix to point to 90this shell script: 91 92```sh 93#!/bin/sh 94 95echo "**** Child $$ starting: y to debug" 96read input 97if [ "$input" = "y" ] ; then 98 gdb --args $* 99else 100 $* 101fi 102``` 103 104#### Selective breakpoints 105 106When debugging both the browser and renderer process, you might want to have 107separate set of breakpoints to hit. You can use gdb's command files to 108accomplish this by putting breakpoints in separate files and instructing gdb to 109load them. 110 111``` 112gdb -x ~/debug/browser --args chrome --no-sandbox --disable-hang-monitor \ 113 --renderer-cmd-prefix='xterm -title renderer -e gdb -x ~/debug/renderer \ 114 --args ' 115``` 116 117Also, instead of running gdb, you can use the script above, which let's you 118select which renderer process to debug. Note: you might need to use the full 119path to the script and avoid `$HOME` or `~/.` 120 121#### Connecting to a running renderer 122 123Usually `ps aux | grep chrome` will not give very helpful output. Try 124`pstree -p | grep chrome` to get something like 125 126``` 127 | |-bash(21969)---chrome(672)-+-chrome(694) 128 | | |-chrome(695)---chrome(696)-+-{chrome}(697) 129 | | | \-{chrome}(709) 130 | | |-{chrome}(675) 131 | | |-{chrome}(678) 132 | | |-{chrome}(679) 133 | | |-{chrome}(680) 134 | | |-{chrome}(681) 135 | | |-{chrome}(682) 136 | | |-{chrome}(684) 137 | | |-{chrome}(685) 138 | | |-{chrome}(705) 139 | | \-{chrome}(717) 140``` 141 142Most of those are threads. In this case the browser process would be 672 and the 143(sole) renderer process is 696. You can use `gdb -p 696` to attach. 144Alternatively, you might find out the process ID from Chrome's built-in Task 145Manager (under the Tools menu). Right-click on the Task Manager, and enable 146"Process ID" in the list of columns. 147 148Note: by default, sandboxed processes can't be attached by a debugger. To be 149able to do so, you will need to pass the `--allow-sandbox-debugging` option. 150 151If the problem only occurs with the seccomp sandbox enabled (and the previous 152tricks don't help), you could try enabling core-dumps (see the **Core files** 153section). That would allow you to get a backtrace and see some local variables, 154though you won't be able to step through the running program. 155 156Note: If you're interested in debugging LinuxSandboxIPC process, you can attach 157to 694 in the above diagram. The LinuxSandboxIPC process has the same command 158line flag as the browser process so that it's easy to identify it if you run 159`pstree -pa`. 160 161#### Getting GPU subprocesses into gdb 162 163Use `--gpu-launcher` flag instead of `--renderer-cmd-prefix` in the instructions 164for renderer above. 165 166#### Getting `browser_tests` launched browsers into gdb 167 168Use environment variable `BROWSER_WRAPPER` instead of `--renderer-cmd-prefix` 169switch in the instructions above. 170 171Example: 172 173```shell 174BROWSER_WRAPPER='xterm -title renderer -e gdb --eval-command=run \ 175 --eval-command=quit --args' out/Debug/browser_tests --gtest_filter=Print 176``` 177 178#### Plugin Processes 179 180Same strategies as renderers above, but the flag is called `--plugin-launcher`: 181 182 chrome --plugin-launcher='xterm -e gdb --args' 183 184*** note 185Note: For now, this does not currently apply to PPAPI plugins because they 186currently run in the renderer process. 187*** 188 189#### Single-Process mode 190 191Depending on whether it's relevant to the problem, it's often easier to just run 192in "single process" mode where the renderer threads are in-process. Then you can 193just run gdb on the main process. 194 195 gdb --args chrome --single-process 196 197Currently, the `--disable-gpu` flag is also required, as there are known crashes 198that occur under TextureImageTransportSurface without it. The crash described in 199https://crbug.com/361689 can also sometimes occur, but that crash can be 200continued from without harm. 201 202Note that for technical reasons plugins cannot be in-process, so 203`--single-process` only puts the renderers in the browser process. The flag is 204still useful for debugging plugins (since it's only two processes instead of 205three) but you'll still need to use `--plugin-launcher` or another approach. 206 207### Printing Chromium types 208 209gdb 7 lets us use Python to write pretty-printers for Chromium types. See 210[gdbinit](https://chromium.googlesource.com/chromium/src/+/master/docs/gdbinit.md) 211to enable pretty-printing of Chromium types. This will import Blink 212pretty-printers as well. 213 214Pretty printers for std types shouldn't be necessary in gdb 7, but they're 215provided here in case you're using an older gdb. Put the following into 216`~/.gdbinit`: 217 218``` 219# Print a C++ string. 220define ps 221 print $arg0.c_str() 222end 223 224# Print a C++ wstring or wchar_t*. 225define pws 226 printf "\"" 227 set $c = (wchar_t*)$arg0 228 while ( *$c ) 229 if ( *$c > 0x7f ) 230 printf "[%x]", *$c 231 else 232 printf "%c", *$c 233 end 234 set $c++ 235 end 236 printf "\"\n" 237end 238``` 239 240[More STL GDB macros](http://www.yolinux.com/TUTORIALS/src/dbinit_stl_views-1.01.txt) 241 242### JsDbg -- visualize data structures in the browser 243 244JsDbg is a debugger plugin to display various Chrome data structures in a 245browser window, such as the accessibility tree, layout object tree, DOM tree, 246and others. 247[Installation instructions are here](https://github.com/MicrosoftEdge/JsDbg), 248and see [here](https://github.com/MicrosoftEdge/JsDbg/blob/master/docs/FEATURES.md) 249for screenshots and an introduction. 250 251For Googlers, please see [go/jsdbg](https://goto.google.com/jsdbg) for 252installation instructions. 253 254### Time travel debugging with rr 255 256You can use [rr](https://rr-project.org) for time travel debugging, so you 257can also step or execute backwards. This works by first recording a trace 258and then debugging based on that. I recommend installing it by compiling 259[from source](https://github.com/mozilla/rr/wiki/Building-And-Installing). 260 261As of May 2020, you must build from source for [`MADV_WIPEONFORK` 262support](https://bugs.chromium.org/p/chromium/issues/detail?id=1082304). If you 263get the following error, rr is too old: 264``` 265Expected EINVAL for 'madvise' but got result 0 (errno SUCCESS); unknown madvise(18) 266``` 267 268Once installed, you can use it like this: 269``` 270rr record out/Debug/content_shell --single-process --no-sandbox --disable-hang-monitor --single-process --disable-seccomp-sandbox --disable-setuid-sandbox 271rr replay 272(gdb) c 273(gdb) break blink::NGBlockNode::Layout 274(gdb) rc # reverse-continue to the last Layout call 275(gdb) jsdbg # run JsDbg as described above to find the interesting object 276(gdb) watch -l box_->frame_rect_.size_.width_.value_ 277(gdb) rc # reverse-continue to the last time the width was changed 278(gdb) rn # reverse-next to the previous line 279(gdb) reverse-fin # run to where this function was called from 280``` 281 282You can debug multi-process chrome using `rr -f [PID]`. To find the process 283id you can either run `rr ps` after recording, or a convenient way 284to find the correct process id is to run with `--vmodule=render_frame_impl=1` 285which will log a message on navigations. e.g. 286 287``` 288$ rr record out/Debug/content_shell --disable-hang-monitor --no-sandbox --disable-seccomp-sandbox --disable-setuid-sandbox --vmodule=render_frame_impl=1 https://google.com/ 289rr: Saving execution to trace directory `...'. 290... 291[128515:128515:0320/164124.768687:VERBOSE1:render_frame_impl.cc(4244)] Committed provisional load: https://www.google.com/ 292``` 293 294From the log message we can see that the site was loaded into process 128515 295and can set a breakpoint for when that process is forked. 296 297``` 298rr replay -f 128515 299``` 300 301### Graphical Debugging Aid for Chromium Views 302 303The following link describes a tool that can be used on Linux, Windows and Mac under GDB. 304 305[graphical_debugging_aid_chromium_views](graphical_debugging_aid_chromium_views.md) 306 307### Faster startup 308 309Use the `gdb-add-index` script (e.g. 310`build/gdb-add-index out/Debug/browser_tests`) 311 312Only makes sense if you run the binary multiple times or maybe if you use the 313component build since most `.so` files won't require reindexing on a rebuild. 314 315See 316https://groups.google.com/a/chromium.org/forum/#!searchin/chromium-dev/gdb-add-index/chromium-dev/ELRuj1BDCL4/5Ki4LGx41CcJ 317for more info. 318 319You can improve GDB load time significantly at the cost of link time by not 320splitting symbols from the object files. In GN, set `use_debug_fission=false` in 321your "gn args". 322 323### Source level debug with -fdebug-compilation-dir 324 325When `strip_absolute_paths_from_debug_symbols` is enabled (which is the 326default), gdb may not be able to find debug files, making source-level debugging 327impossible. See 328[gdbinit](https://chromium.googlesource.com/chromium/src/+/master/docs/gdbinit.md) 329to configure gdb to be able to find debug files. 330 331## Core files 332 333`ulimit -c unlimited` should cause all Chrome processes (run from that shell) to 334dump cores, with the possible exception of some sandboxed processes. 335 336Some sandboxed subprocesses might not dump cores unless you pass the 337`--allow-sandbox-debugging` flag. 338 339If the problem is a freeze rather than a crash, you may be able to trigger a 340core-dump by sending SIGABRT to the relevant process: 341 342 kill -6 [process id] 343 344## Breakpad minidump files 345 346See [minidump_to_core.md](minidump_to_core.md) 347 348## Running Tests 349 350Many of our tests bring up windows on screen. This can be annoying (they steal 351your focus) and hard to debug (they receive extra events as you mouse over them). 352Instead, use `Xvfb` or `Xephyr` to run a nested X session to debug them, as 353outlined on [testing/web_tests_linux.md](testing/web_tests_linux.md). 354 355### Browser tests 356 357By default the `browser_tests` forks a new browser for each test. To debug the 358browser side of a single test, use a command like 359 360``` 361gdb --args out/Debug/browser_tests --single-process-tests --gtest_filter=MyTestName 362``` 363 364**note the use of `single-process-tests`** -- this makes the test harness and 365browser process share the outermost process. 366 367 368To debug a renderer process in this case, use the tips above about renderers. 369 370### Web tests 371 372See [testing/web_tests_linux.md](testing/web_tests_linux.md) for some tips. In particular, 373note that it's possible to debug a web test via `ssh`ing to a Linux box; you 374don't need anything on screen if you use `Xvfb`. 375 376### UI tests 377 378UI tests are run in forked browsers. Unlike browser tests, you cannot do any 379single process tricks here to debug the browser. See below about 380`BROWSER_WRAPPER`. 381 382To pass flags to the browser, use a command line like 383`--extra-chrome-flags="--foo --bar"`. 384 385### Timeouts 386 387UI tests have a confusing array of timeouts in place. (Pawel is working on 388reducing the number of timeouts.) To disable them while you debug, set the 389timeout flags to a large value: 390 391* `--test-timeout=100000000` 392* `--ui-test-action-timeout=100000000` 393* `--ui-test-terminate-timeout=100000000` 394 395### To replicate Window Manager setup on the bots 396 397Chromium try bots and main waterfall's bots run tests under Xvfb&openbox 398combination. Xvfb is an X11 server that redirects the graphical output to the 399memory, and openbox is a simple window manager that is running on top of Xvfb. 400The behavior of openbox is markedly different when it comes to focus management 401and other window tasks, so test that runs fine locally may fail or be flaky on 402try bots. To run the tests on a local machine as on a bot, follow these steps: 403 404Make sure you have openbox: 405 406 apt-get install openbox 407 408Start Xvfb and openbox on a particular display: 409 410 Xvfb :6.0 -screen 0 1280x1024x24 & DISPLAY=:6.0 openbox & 411 412Run your tests with graphics output redirected to that display: 413 414 DISPLAY=:6.0 out/Debug/browser_tests --gtest_filter="MyBrowserTest.MyActivateWindowTest" 415 416You can look at a snapshot of the output by: 417 418 xwd -display :6.0 -root | xwud 419 420Alternatively, you can use testing/xvfb.py to set up your environment for you: 421 422 testing/xvfb.py out/Debug/browser_tests \ 423 --gtest_filter="MyBrowserTest.MyActivateWindowTest" 424 425### BROWSER_WRAPPER 426 427You can also get the browser under a debugger by setting the `BROWSER_WRAPPER` 428environment variable. (You can use this for `browser_tests` too, but see above 429for discussion of a simpler way.) 430 431 BROWSER_WRAPPER='xterm -e gdb --args' out/Debug/browser_tests 432 433### Replicating try bot Slowness 434 435Try bots are pretty stressed, and can sometimes expose timing issues you can't 436normally reproduce locally. 437 438You can simulate this by shutting down all but one of the CPUs 439(http://www.cyberciti.biz/faq/debian-rhel-centos-redhat-suse-hotplug-cpu/) and 440running a CPU loading tool (e.g., http://www.devin.com/lookbusy/). Now run your 441test. It will run slowly, but any flakiness found by the try bot should replicate 442locally now - and often nearly 100% of the time. 443 444## Logging 445 446### Seeing all LOG(foo) messages 447 448Default log level hides `LOG(INFO)`. Run with `--log-level=0` and 449`--enable-logging=stderr` flags. 450 451Newer versions of Chromium with VLOG may need --v=1 too. For more VLOG tips, see 452[the chromium-dev thread](https://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/dcd0cd7752b35de6?pli=1). 453 454### Seeing IPC debug messages 455 456Run with `CHROME_IPC_LOGGING=1` eg. 457 458 CHROME_IPC_LOGGING=1 out/Debug/chrome 459 460or within gdb: 461 462 set environment CHROME_IPC_LOGGING 1 463 464If some messages show as unknown, check if the list of IPC message headers in 465[chrome/common/logging_chrome.cc](/chrome/common/logging_chrome.cc) is 466up to date. In case this file reference goes out of date, try looking for usage 467of macros like `IPC_MESSAGE_LOG_ENABLED` or `IPC_MESSAGE_MACROS_LOG_ENABLED`. 468 469## Profiling 470 471See 472https://sites.google.com/a/chromium.org/dev/developers/profiling-chromium-and-webkit 473and [Linux Profiling](profiling.md). 474 475## i18n 476 477We obey your system locale. Try something like: 478 479 LANG=ja_JP.UTF-8 out/Debug/chrome 480 481If this doesn't work, make sure that the `LANGUAGE`, `LC_ALL` and `LC_MESSAGE` 482environment variables aren't set -- they have higher priority than LANG in the 483order listed. Alternatively, just do this: 484 485 LANGUAGE=fr out/Debug/chrome 486 487Note that because we use GTK, some locale data comes from the system -- for 488example, file save boxes and whether the current language is considered RTL. 489Without all the language data available, Chrome will use a mixture of your 490system language and the language you run Chrome in. 491 492Here's how to install the Arabic (ar) and Hebrew (he) language packs: 493 494 sudo apt-get install language-pack-ar language-pack-he \ 495 language-pack-gnome-ar language-pack-gnome-he 496 497Note that the `--lang` flag does **not** work properly for this. 498 499On non-Debian systems, you need the `gtk30.mo` files. (Please update these docs 500with the appropriate instructions if you know what they are.) 501 502## Breakpad 503 504See the last section of [Linux Crash Dumping](crash_dumping.md). 505 506## Drag and Drop 507 508If you break in a debugger during a drag, Chrome will have grabbed your mouse 509and keyboard so you won't be able to interact with the debugger! To work around 510this, run via `Xephyr`. Instructions for how to use `Xephyr` are on the 511[Running web tests on Linux](testing/web_tests_linux.md) page. 512 513## Tracking Down Bugs 514 515### Isolating Regressions 516 517Old builds are archived here: 518https://build.chromium.org/buildbot/snapshots/chromium-rel-linux/ 519(TODO: does not exist). 520 521`tools/bisect-builds.py` in the tree automates bisecting through the archived 522builds. Despite a computer science education, I am still amazed how quickly 523binary search will find its target. 524 525### Screen recording for bug reports 526 527 sudo apt-get install gtk-recordmydesktop 528 529## Version-specific issues 530 531### Google Chrome 532 533Google Chrome binaries don't include symbols. Googlers can read where to get 534symbols from 535[the Google-internal wiki](http://wiki/Main/ChromeOfficialBuildLinux#The_Build_Archive). 536 537### Ubuntu Chromium 538 539Since we don't build the Ubuntu packages (Ubuntu does) we can't get useful 540backtraces from them. Direct users to https://wiki.ubuntu.com/Chromium/Debugging 541 542### Fedora's Chromium 543 544Like Ubuntu, but direct users to 545https://fedoraproject.org/wiki/TomCallaway/Chromium_Debug 546 547### Xlib 548 549If you're trying to track down X errors like: 550 551``` 552The program 'chrome' received an X Window System error. 553This probably reflects a bug in the program. 554The error was 'BadDrawable (invalid Pixmap or Window parameter)'. 555``` 556 557Some strategies are: 558 559* pass `--sync` on the command line to make all X calls synchronous 560* run chrome via [xtrace](http://xtrace.alioth.debian.org/) 561* turn on IPC debugging (see above section) 562 563### Window Managers 564 565To test on various window managers, you can use a nested X server like `Xephyr`. 566Instructions for how to use `Xephyr` are on the 567[Running web tests on Linux](web_tests_linux.md) page. 568 569If you need to test something with hardware accelerated compositing 570(e.g., compiz), you can use `Xgl` (`sudo apt-get install xserver-xgl`). E.g.: 571 572 Xgl :1 -ac -accel glx:pbuffer -accel xv:pbuffer -screen 1024x768 573 574## Mozilla Tips 575 576https://developer.mozilla.org/en/Debugging_Mozilla_on_Linux_FAQ 577