1# DMD 2 3DMD (short for "dark matter detector") is a heap profiler within 4Firefox. It has four modes. 5 6- "Dark Matter" mode. In this mode, DMD tracks the contents of the 7 heap, including which heap blocks have been reported by memory 8 reporters. It helps us reduce the "heap-unclassified" value in 9 Firefox's about:memory page, and also detects if any heap blocks 10 are reported twice. Originally, this was the only mode that DMD had, 11 which explains DMD's name. This is the default mode. 12- "Live" mode. In this mode, DMD tracks the current contents of the 13 heap. You can dump that information to file, giving a profile of the 14 live heap blocks at that point in time. This is good for 15 understanding how memory is used at an interesting point in time, 16 such as peak memory usage. 17- "Cumulative" mode. In this mode, DMD tracks both the past and 18 current contents of the heap. You can dump that information to file, 19 giving a profile of the heap usage for the entire session. This is 20 good for finding parts of the code that cause high heap churn, e.g. 21 by allocating many short-lived allocations. 22- "Heap scanning" mode. This mode is like live mode, but it also 23 records the contents of every live block in the log. This can be 24 used to investigate leaks by figuring out which objects might be 25 holding references to other objects. 26 27## Building and Running 28 29### Nightly Firefox 30 31The easiest way to use DMD is with the normal Nightly Firefox build, 32which has DMD already enabled in the build. To have DMD active while 33running it, you just need to set the environment variable `DMD=1` when 34running. For instance, on OSX, you can run something like: 35 36 DMD=1 /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox 37 38You can tell it is working by going to about:memory and looking for 39"Save DMD Output". If DMD has been properly enabled, the "Save" 40button won't be grayed out. Look at the "Trigger" section below to 41see the full list of ways to get a DMD report once you have it 42activated. Note that stack information you get will likely be less 43detailed, due to being unable to symbolicate. You will be able to get 44function names, but not line numbers. 45 46### Desktop Firefox (Linux) 47 48#### Build 49 50Build Firefox with these options: 51 52 ac_add_options --enable-dmd 53 54If building via try server, modify 55`browser/config/mozconfigs/linux64/common-opt` or a similar file before 56pushing. 57 58#### Launch 59 60Use `mach run --dmd`; use `--mode` to choose the mode. Add `--debug` to 61run under gdb. 62 63#### Trigger 64 65There are three ways to trigger a DMD snapshot. 66 671. Visit about:memory and click the DMD button (depending on how old 68 your build is, it might be labelled "Save" or "Analyze reports" 69 or "DMD"). The button won't be present in non-DMD builds, and 70 will be grayed out in DMD builds if DMD isn't enabled at start-up. 71 722. If you wish to trigger DMD dumps from within C++ or JavaScript code, 73 you can use `nsIMemoryInfoDumper.dumpMemoryToTempDir`. For example, 74 from JavaScript code you can do the following. 75 76 const Cc = Components.classes; 77 let mydumper = Cc["@mozilla.org/memory-info-dumper;1"] 78 .getService(Ci.nsIMemoryInfoDumper); 79 mydumper.dumpMemoryInfoToTempDir(identifier, anonymize, minimize); 80 81 This will dump memory reports and DMD output to the temporary 82 directory. `identifier` is a string that will be used for part of 83 the filename (or a timestamp will be used if it is an empty string); 84 `anonymize` is a boolean that indicates if the memory reports should 85 be anonymized; and `minimize` is a boolean that indicates if memory 86 usage should be minimized first. 87 883. (Linux only) You can send signal 34 to the firefox process, e.g. 89 with the following command. 90 91 $ killall -34 firefox 92 93Each one of these steps triggers all the memory reporters and then DMD 94analyzes the reports, printing commentary like this: 95 96 DMD[5222] opened /tmp/dmd-1414556492-5222.json.gz for writing 97 DMD[5222] Dump 1 { 98 DMD[5222] Constructing the heap block list... 99 DMD[5222] Constructing the stack trace table... 100 DMD[5222] Constructing the stack frame table... 101 DMD[5222] } 102 103In an e10s-enabled build, you'll see separate output for each process. 104This step can take 10 or more seconds and may make Firefox freeze 105temporarily. 106 107If you see the "opened" line, it tells you where the file was saved. 108It's always in a temp directory, and the filenames are always of the 109form dmd-<pid>. 110 111### Desktop Firefox (Mac) 112 113#### Build 114 115Build with these options: 116 117 ac_add_options --enable-dmd 118 119If building via try server, modify 120`browser/config/mozconfigs/macosx64/common-opt` or a similar file before 121pushing. 122 123#### Launch 124 125Use `mach run --dmd; `use `--mode` to choose the mode. Add `--debug` to 126run under lldb. 127 128#### Trigger 129 130Follow the [Trigger instructions for Linux](#Trigger_7). Note that on 131Mac this step can take 30+ seconds. 132 133### Desktop Firefox (Windows) 134 135#### Build 136 137Build with these options: 138 139 ac_add_options --enable-dmd 140 141If building via try server, modify 142`browser/config/mozconfigs/win32/common-opt`. Also, add this line to 143`build/mozconfig.common`: 144 145 MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1 146 147#### Launch 148 149On a local build, use `mach run --dmd`; use `--mode` to choose the mode. 150 151On a build done by the try server, follow [these 152instructions](https://bugzilla.mozilla.org/show_bug.cgi?id=936784#c69){.external 153.text} instead. 154 155#### Trigger 156 157Follow the [Trigger instructions for Linux] 158 159### Fennec 160 161::: {.note} 162In order to use DMD on Fennec you will need root access on the Android 163device. Instructions on how to root your device is outside the scope of 164this document. 165::: 166 167#### Build 168 169Build with these options: 170 171 ac_add_options --enable-dmd 172 173#### Prep 174 175In order to prepare your device for running Fennec with DMD enabled, you 176will need to do a few things. First, you will need to push the libdmd.so 177library to the device so that it can by dynamically loaded by Fennec. 178You can do this by running: 179 180 adb push $OBJDIR/dist/bin/libdmd.so /sdcard/ 181 182Second, you will need to make an executable wrapper for Fennec which 183sets an environment variable before launching it. (If you are familiar 184with the recommended "--es env0" method for setting environment 185variables when launching Fennec, note that you cannot use this method 186here because those are processed too late in the startup process. If you 187are not familiar with that method, you can ignore this parenthetical 188note.) First make the executable wrapper on your host machine using the 189editor of your choice. Name the file dmd_fennec and enter this as the 190contents: 191 192 #!/system/bin/sh 193 export MOZ_REPLACE_MALLOC_LIB=/sdcard/libdmd.so 194 exec "$@" 195 196If you want to use other DMD options, you can enter additional 197environment variables above. You will need to push this to the device 198and make it executable. Since you cannot mark files in /sdcard/ as 199executable, we will use /data/local/tmp for this purpose: 200 201 adb push dmd_fennec /data/local/tmp 202 adb shell 203 cd /data/local/tmp 204 chmod 755 dmd_fennec 205 206The final step is to make Android use the above wrapper script while 207launching Fennec, so that the environment variable is present when 208Fennec starts up. Assuming you have done a local build, the app 209identifier will be `org.mozilla.fennec_$USERNAME` (`$USERNAME` is your 210username on the host machine) and so we do this as shown below. If you 211are using a DMD-enabled try build, or build from other source, adjust 212the app identifier as necessary. 213 214 adb shell 215 su # You need root access for the setprop command to take effect 216 setprop wrap.org.mozilla.fennec_$USERNAME "/data/local/tmp/dmd_fennec" 217 218Once this is set up, starting the `org.mozilla.fennec_$USERNAME` app 219will use the wrapper script. 220 221#### Launch 222 223Launch Fennec either by tapping on the icon as usual, or from the 224command line (as before, be sure to replace 225`org.mozilla.fennec_$USERNAME` with the app identifier as appropriate). 226 227 adb shell am start -n org.mozilla.fennec_$USERNAME/.App 228 229#### Trigger 230 231Use the existing memory-report dumping hook: 232 233 adb shell am broadcast -a org.mozilla.gecko.MEMORY_DUMP 234 235In logcat, you should see output similar to this: 236 237 I/DMD (20731): opened /storage/emulated/0/Download/memory-reports/dmd-default-20731.json.gz for writing 238 ... 239 I/GeckoConsole(20731): nsIMemoryInfoDumper dumped reports to /storage/emulated/0/Download/memory-reports/unified-memory-report-default-20731.json.gz 240 241The path is where the memory reports and DMD reports get dumped to. You 242can pull them like so: 243 244 adb pull /sdcard/Download/memory-reports/dmd-default-20731.json.gz 245 adb pull /sdcard/Download/memory-reports/unified-memory-report-default-20731.json.gz 246 247## Processing the output 248 249DMD outputs one gzipped JSON file per process that contains a 250description of that process's heap. You can analyze these files (either 251gzipped or not) using `dmd.py`. On Nightly Firefox, `dmd.py` is included 252in the distribution. For instance on OS X, it is located in the 253directory `/Applications/Firefox Nightly.app/Contents/Resources/`. For 254Nightly, symbolication will fail, but you can at least get some 255information. In a local build, `dmd.py` will be located in the directory 256`$OBJDIR/dist/bin/`. 257 258Some platforms (Linux, Mac, Android) require stack fixing, which adds 259missing filename, function name and line number information. This will 260occur automatically the first time you run `dmd.py` on the output file. 261This can take 10s of seconds or more to complete. (This will fail if 262your build does not contain symbols. However, if you have crash reporter 263symbols for your build -- as tryserver builds do -- you can use [this 264script](https://github.com/mstange/analyze-tryserver-profiles/blob/master/resymbolicate_dmd.py) 265instead: clone the whole repo, edit the paths at the top of 266`resymbolicate_dmd.py` and run it.) The simplest way to do this is to 267just run the `dmd.py` script on your DMD report while your working 268directory is `$OBJDIR/dist/bin`. This will allow the local libraries to 269be found and used. 270 271If you invoke `dmd.py` without arguments you will get output appropriate 272for the mode in which DMD was invoked. 273 274### "Dark matter" mode output 275 276For "dark matter" mode, `dmd.py`'s output describes how the live heap 277blocks are covered by memory reports. This output is broken into 278multiple sections. 279 2801. "Invocation". This tells you how DMD was invoked, i.e. what 281 options were used. 2822. "Twice-reported stack trace records". This tells you which heap 283 blocks were reported twice or more. The presence of any such records 284 indicates bugs in one or more memory reporters. 2853. "Unreported stack trace records". This tells you which heap blocks 286 were not reported, which indicate where additional memory reporters 287 would be most helpful. 2884. "Once-reported stack trace records": like the "Unreported stack 289 trace records" section, but for blocks reported once. 2905. "Summary": gives measurements of the total heap, and the 291 unreported/once-reported/twice-reported portions of it. 292 293The "Twice-reported stack trace records" and "Unreported stack trace 294records" sections are the most important, because they indicate ways in 295which the memory reporters can be improved. 296 297Here's an example stack trace record from the "Unreported stack trace 298records" section. 299 300 Unreported { 301 150 blocks in heap block record 283 of 5,495 302 21,600 bytes (20,400 requested / 1,200 slop) 303 Individual block sizes: 144 x 150 304 0.00% of the heap (16.85% cumulative) 305 0.02% of unreported (94.68% cumulative) 306 Allocated at { 307 #01: replace_malloc (/home/njn/moz/mi5/go64dmd/memory/replace/dmd/../../../../memory/replace/dmd/DMD.cpp:1286) 308 #02: malloc (/home/njn/moz/mi5/go64dmd/memory/build/../../../memory/build/replace_malloc.c:153) 309 #03: moz_xmalloc (/home/njn/moz/mi5/memory/mozalloc/mozalloc.cpp:84) 310 #04: nsCycleCollectingAutoRefCnt::incr(void*, nsCycleCollectionParticipant*) (/home/njn/moz/mi5/go64dmd/dom/xul/../../dist/include/nsISupportsImpl.h:250) 311 #05: nsXULElement::Create(nsXULPrototypeElement*, nsIDocument*, bool, bool,mozilla::dom::Element**) (/home/njn/moz/mi5/dom/xul/nsXULElement.cpp:287) 312 #06: nsXBLContentSink::CreateElement(char16_t const**, unsigned int, mozilla::dom::NodeInfo*, unsigned int, nsIContent**, bool*, mozilla::dom::FromParser) (/home/njn/moz/mi5/dom/xbl/nsXBLContentSink.cpp:874) 313 #07: nsCOMPtr<nsIContent>::StartAssignment() (/home/njn/moz/mi5/go64dmd/dom/xml/../../dist/include/nsCOMPtr.h:753) 314 #08: nsXMLContentSink::HandleStartElement(char16_t const*, char16_t const**, unsigned int, unsigned int, bool) (/home/njn/moz/mi5/dom/xml/nsXMLContentSink.cpp:1007) 315 } 316 } 317 318It tells you that there were 150 heap blocks that were allocated from 319the program point indicated by the "Allocated at" stack trace, that 320these blocks took up 21,600 bytes, that all 150 blocks had a size of 144 321bytes, and that 1,200 of those bytes were "slop" (wasted space caused 322by the heap allocator rounding up request sizes). It also indicates what 323percentage of the total heap size and the unreported portion of the heap 324these blocks represent. 325 326Within each section, records are listed from largest to smallest. 327 328Once-reported and twice-reported stack trace records also have stack 329traces for the report point(s). For example: 330 331 Reported at { 332 #01: mozilla::dmd::Report(void const*) (/home/njn/moz/mi2/memory/replace/dmd/DMD.cpp:1740) 0x7f68652581ca 333 #02: CycleCollectorMallocSizeOf(void const*) (/home/njn/moz/mi2/xpcom/base/nsCycleCollector.cpp:3008) 0x7f6860fdfe02 334 #03: nsPurpleBuffer::SizeOfExcludingThis(unsigned long (*)(void const*)) const (/home/njn/moz/mi2/xpcom/base/nsCycleCollector.cpp:933) 0x7f6860fdb7af 335 #04: nsCycleCollector::SizeOfIncludingThis(unsigned long (*)(void const*), unsigned long*, unsigned long*, unsigned long*, unsigned long*, unsigned long*) const (/home/njn/moz/mi2/xpcom/base/nsCycleCollector.cpp:3029) 0x7f6860fdb6b1 336 #05: CycleCollectorMultiReporter::CollectReports(nsIMemoryMultiReporterCallback*, nsISupports*) (/home/njn/moz/mi2/xpcom/base/nsCycleCollector.cpp:3075) 0x7f6860fde432 337 #06: nsMemoryInfoDumper::DumpMemoryReportsToFileImpl(nsAString_internal const&) (/home/njn/moz/mi2/xpcom/base/nsMemoryInfoDumper.cpp:626) 0x7f6860fece79 338 #07: nsMemoryInfoDumper::DumpMemoryReportsToFile(nsAString_internal const&, bool, bool) (/home/njn/moz/mi2/xpcom/base/nsMemoryInfoDumper.cpp:344) 0x7f6860febaf9 339 #08: mozilla::(anonymous namespace)::DumpMemoryReportsRunnable::Run() (/home/njn/moz/mi2/xpcom/base/nsMemoryInfoDumper.cpp:58) 0x7f6860fefe03 340 } 341 342You can tell which memory reporter made the report by the name of the 343`MallocSizeOf` function near the top of the stack trace. In this case it 344was the cycle collector's reporter. 345 346By default, DMD does not record an allocation stack trace for most 347blocks, to make it run faster. The decision on whether to record is done 348probabilistically, and larger blocks are more likely to have an 349allocation stack trace recorded. All unreported blocks that lack an 350allocation stack trace will end up in a single record. For example: 351 352 Unreported { 353 420,010 blocks in heap block record 2 of 5,495 354 29,203,408 bytes (27,777,288 requested / 1,426,120 slop) 355 Individual block sizes: 2,048 x 3; 1,024 x 103; 512 x 147; 496 x 7; 480 x 31; 464 x 6; 448 x 50; 432 x 41; 416 x 28; 400 x 53; 384 x 43; 368 x 216; 352 x 141; 336 x 58; 320 x 104; 304 x 5,130; 288 x 150; 272 x 591; 256 x 6,017; 240 x 1,372; 224 x 93; 208 x 488; 192 x 1,919; 176 x 18,903; 160 x 1,754; 144 x 5,041; 128 x 36,709; 112 x 5,571; 96 x 6,280; 80 x 40,738; 64 x 37,925; 48 x 78,392; 32 x 136,199; 16 x 31,001; 8 x 4,706 356 3.78% of the heap (10.24% cumulative) 357 21.24% of unreported (57.53% cumulative) 358 Allocated at { 359 #01: (no stack trace recorded due to --stacks=partial) 360 } 361 } 362 363In contrast, stack traces are always recorded when a block is reported, 364which means you can end up with records like this where the allocation 365point is unknown but the reporting point *is* known: 366 367 Once-reported { 368 104,491 blocks in heap block record 13 of 4,689 369 10,392,000 bytes (10,392,000 requested / 0 slop) 370 Individual block sizes: 512 x 124; 256 x 242; 192 x 813; 128 x 54,664; 64 x 48,648 371 1.35% of the heap (48.65% cumulative) 372 1.64% of once-reported (59.18% cumulative) 373 Allocated at { 374 #01: (no stack trace recorded due to --stacks=partial) 375 } 376 Reported at { 377 #01: mozilla::dmd::DMDFuncs::Report(void const*) (/home/njn/moz/mi5/go64dmd/memory/replace/dmd/../../../../memory/replace/dmd/DMD.cpp:1646) 378 #02: WindowsMallocSizeOf(void const*) (/home/njn/moz/mi5/dom/base/nsWindowMemoryReporter.cpp:189) 379 #03: nsAttrAndChildArray::SizeOfExcludingThis(unsigned long (*)(void const*)) const (/home/njn/moz/mi5/dom/base/nsAttrAndChildArray.cpp:880) 380 #04: mozilla::dom::FragmentOrElement::SizeOfExcludingThis(unsigned long (*)(void const*)) const (/home/njn/moz/mi5/dom/base/FragmentOrElement.cpp:2337) 381 #05: nsINode::SizeOfIncludingThis(unsigned long (*)(void const*)) const (/home/njn/moz/mi5/go64dmd/parser/html/../../../dom/base/nsINode.h:307) 382 #06: mozilla::dom::NodeInfo::NodeType() const (/home/njn/moz/mi5/go64dmd/dom/base/../../dist/include/mozilla/dom/NodeInfo.h:127) 383 #07: nsHTMLDocument::DocAddSizeOfExcludingThis(nsWindowSizes*) const (/home/njn/moz/mi5/dom/html/nsHTMLDocument.cpp:3710) 384 #08: nsIDocument::DocAddSizeOfIncludingThis(nsWindowSizes*) const (/home/njn/moz/mi5/dom/base/nsDocument.cpp:12820) 385 } 386 } 387 388The choice of whether to record an allocation stack trace for all blocks 389is controlled by an option (see below). 390 391### "Live" mode output 392 393 394For "live" mode, dmd.py's output describes what live heap blocks are 395present. This output is broken into multiple sections. 396 3971. "Invocation". This tells you how DMD was invoked, i.e. what 398 options were used. 3992. "Live stack trace records". This tells you which heap blocks were 400 present. 4013. "Summary": gives measurements of the total heap. 402 403The individual records are similar to those output in "dark matter" 404mode. 405 406### "Cumulative" mode output 407 408For "cumulative" mode, dmd.py's output describes how the live heap 409blocks are covered by memory reports. This output is broken into 410multiple sections. 411 4121. "Invocation". This tells you how DMD was invoked, i.e. what 413 options were used. 4142. "Cumulative stack trace records". This tells you which heap blocks 415 were allocated during the session. 4163. "Summary": gives measurements of the total (cumulative) heap. 417 418The individual records are similar to those output in "dark matter" 419mode. 420 421### "Scan" mode output 422 423For "scan" mode, the output of `dmd.py` is the same as "live" mode. 424A separate script, `block_analyzer.py`, can be used to find out 425information about which blocks refer to a particular block. 426`dmd.py --clamp-contents` needs to be run on the log first. See [this 427other page](heap_scan_mode.md) for an 428overview of how to use heap scan mode to fix a leak involving refcounted 429objects. 430 431## Options 432 433### Runtime 434 435When you run `mach run --dmd` you can specify additional options to 436control how DMD runs. Run `mach help run` for documentation on these. 437 438The most interesting one is `--mode`. Acceptable values are 439`dark-matter` (the default), `live`, `cumulative`, and `scan`. 440 441Another interesting one is `--stacks`. Acceptable values are `partial` 442(the default) and `full`. In the former case most blocks will not have 443an allocation stack trace recorded. However, because larger blocks are 444more likely to have one recorded, most allocated bytes should have an 445allocation stack trace even though most allocated blocks do not. Use 446`--stacks=full` if you want complete information, but note that DMD will 447run substantially slower in that case. 448 449The options may also be put in the environment variable DMD, or set DMD 450to 1 to enable DMD with default options (dark-matter and partial 451stacks). 452 453The `MOZ_DMD_SHUTDOWN_LOG` environment variable, if set, triggers a DMD 454run at shutdown; its value must be a directory where the logs will be 455placed. Which processes get logged is controlled by the 456`MOZ_DMD_LOG_PROCESS` environment variable, which can take the following 457values. 458 459- Unset: log all processes. 460- "`default`": log the parent process only. 461- "`tab`": log content processes only. 462 463For example, if you have 464 465 MOZ_DMD_SHUTDOWN_LOG=~/dmdlogs/ MOZ_DMD_LOG_PROCESS=tab 466 467then DMD logs for content processes will be saved to `~/dmdlogs/`. 468 469**NOTE:** 470 471- To dump DMD data from Content processes, you'll need to disable the 472 sandbox 473- MOZ_DMD_SHUTDOWN_LOG must (currently) include the trailing separator 474 (\'\'/\") 475 476### Post-processing 477 478`dmd.py` also takes options that control how it works. Run `dmd.py -h` 479for documentation. The following options are the most interesting ones. 480 481- `-f` / `--max-frames`. By default, records show up to 8 stack 482 frames. You can choose a smaller number, in which case more 483 allocations will be aggregated into each record, but you'll have 484 less context. Or you can choose a larger number, in which cases 485 allocations will be split across more records, but you will have 486 more context. There is no single best value, but values in the range 487 2..10 are often good. The maximum is 24. 488 489- `-a` / `--ignore-alloc-frames`. Many allocation stack traces start 490 with multiple frames that mention allocation wrapper functions, e.g. 491 `js_calloc()` calls `replace_calloc()`. This option filters these 492 out. It often helps improve the quality of the output when using a 493 small `--max-frames` value. 494 495- `-s` / `--sort-by`. This controls how records are sorted. Acceptable 496 values are `usable` (the default), `req`, `slop` and `num-blocks`. 497 498- `--clamp-contents`. For a heap scan log, this performs a 499 conservative pointer analysis on the contents of each block, 500 changing any value that is a pointer into the middle of a live block 501 into a pointer to the start of that block. All other values are 502 changes to null. In addition, all trailing nulls are removed from 503 the block contents. 504 505As an example that combines multiple options, if you apply the following 506command to a profile obtained in "live" mode: 507 508 dmd.py -r -f 2 -a -s slop 509 510it will give you a good idea of where the major sources of slop are. 511 512`dmd.py` can also compute the difference between two DMD output files, 513so long as those files were produced in the same mode. Simply pass it 514two filenames instead of one to get the difference. 515 516## Which heap blocks are reported? 517 518At this stage you might wonder how DMD knows, in "dark matter" mode, 519which allocations have been reported and which haven't. DMD only knows 520about heap blocks that are measured via a function created with one of 521the following two macros: 522 523 MOZ_DEFINE_MALLOC_SIZE_OF 524 MOZ_DEFINE_MALLOC_SIZE_OF_ON_ALLOC 525 526Fortunately, most of the existing memory reporters do this. See 527[Performance/Memory_Reporting](https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Memory_reporting "Platform/Memory Reporting") 528for more details about how memory reporters are written. 529