1Dark Matter Detector (DMD) 2========================== 3 4.. note:: 5 6 This section was copied with minimal modification from 7 https://developer.mozilla.org/en-US/docs/Mozilla/Performance/DMD. Some 8 parts of the documentation may be out-of-date. 9 10DMD (short for "dark matter detector") is a heap profiler within Firefox. It 11has four modes. 12 13* "Dark Matter" mode. In this mode, DMD tracks the contents of the heap, 14 including which heap blocks have been reported by memory reporters. It helps 15 us reduce the "heap-unclassified" value in Firefox's about:memory page, and 16 also detects if any heap blocks are reported twice. Originally, this was the 17 only mode that DMD had, which explains DMD's name. This is the default mode. 18 19* "Live" mode. In this mode, DMD tracks the current contents of the heap. You 20 can dump that information to file, giving a profile of the live heap blocks 21 at that point in time. This is good for understanding how memory is used at 22 an interesting point in time, such as peak memory usage. 23 24* "Cumulative" mode. In this mode, DMD tracks both the past and current 25 contents of the heap. You can dump that information to file, giving a profile 26 of the heap usage for the entire session. This is good for finding parts of 27 the code that cause high heap churn, e.g. by allocating many short-lived 28 allocations. 29 30* "Heap scanning" mode. This mode is like live mode, but it also records the 31 contents of every live block in the log. This can be used to investigate 32 leaks by figuring out which objects might be holding references to other 33 objects. 34 35 36Building and Running 37-------------------- 38 39Nightly Firefox 40~~~~~~~~~~~~~~~ 41 42The easiest way to use DMD is with the normal Nightly Firefox build, which 43has DMD already enabled in the build. To have DMD active while running it, 44you just need to set the environment variable ``DMD=1`` when running. For 45instance, on OSX, you can run something like: 46 47.. code-block: bash 48 49 DMD=1 /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox 50 51You can tell it is working by going to ``about:memory`` and looking for "Save 52DMD Output". If DMD has been properly enabled, the "Save" button won't be 53grayed out. Look at the "Trigger" section below to see the full list of ways 54to get a DMD report once you have it activated. Note that stack information 55you get will likely be less detailed, due to being unable to symbolicate. You 56will be able to get function names, but not line numbers. 57 58Processing the output 59--------------------- 60 61DMD outputs one gzipped JSON file per process that contains a description of 62that process's heap. You can analyze these files (either gzipped or not) 63using ``dmd.py``. On Nightly Firefox, ``dmd.py`` is included in the 64distribution. For instance on OS X, it is located in the directory 65``/Applications/Firefox Nightly.app/Contents/Resources/``. For Nightly, 66symbolication will fail, but you can at least get some information. In a 67local build, ``dmd.py`` will be located in the directory 68``$OBJDIR/dist/bin/``. 69 70Some platforms (Linux, Mac, Android) require stack fixing, which adds missing 71filename, function name and line number information. This will occur 72automatically the first time you run ``dmd.py`` on the output file. This can 73take 10s of seconds or more to complete. (This will fail if your build does 74not contain symbols. However, if you have crash reporter symbols for your 75build (as tryserver builds do) you can use this script instead: clone the 76whole repo, edit the paths at the top of ``resymbolicate_dmd.py`` and run 77it.) The simplest way to do this is to just run the ``dmd.py`` script on your 78DMD report while your working directory is ``$OBJDIR/dist/bin``. This will 79allow the local libraries to be found and used. 80 81If you invoke ``dmd.py`` without arguments you will get output appropriate 82for the mode in which DMD was invoked. 83 84"Dark matter" mode output 85~~~~~~~~~~~~~~~~~~~~~~~~~ 86 87For "dark matter" mode, ``dmd.py``'s output describes how the live heap blocks 88are covered by memory reports. This output is broken into multiple sections. 89 901. "Invocation". This tells you how DMD was invoked, i.e. what options were used. 91 922. "Twice-reported stack trace records". This tells you which heap blocks 93 were reported twice or more. The presence of any such records indicates bugs 94 in one or more memory reporters. 95 963. "Unreported stack trace records". This tells you which heap blocks were 97 not reported, which indicate where additional memory reporters would be most 98 helpful. 99 1004. "Once-reported stack trace records": like the "Unreported stack trace 101 records" section, but for blocks reported once. 102 1035. "Summary": gives measurements of the total heap, and the 104 unreported/once-reported/twice-reported portions of it. 105 106The "Twice-reported stack trace records" and "Unreported stack trace records" 107sections are the most important, because they indicate ways in which the 108memory reporters can be improved. 109 110Here's an example stack trace record from the "Unreported stack trace 111records" section. 112 113.. code-block:: 114 115 Unreported { 116 150 blocks in heap block record 283 of 5,495 117 21,600 bytes (20,400 requested / 1,200 slop) 118 Individual block sizes: 144 x 150 119 0.00% of the heap (16.85% cumulative) 120 0.02% of unreported (94.68% cumulative) 121 Allocated at { 122 #01: replace_malloc (/home/njn/moz/mi5/go64dmd/memory/replace/dmd/../../../../memory/replace/dmd/DMD.cpp:1286) 123 #02: malloc (/home/njn/moz/mi5/go64dmd/memory/build/../../../memory/build/replace_malloc.c:153) 124 #03: moz_xmalloc (/home/njn/moz/mi5/memory/mozalloc/mozalloc.cpp:84) 125 #04: nsCycleCollectingAutoRefCnt::incr(void*, nsCycleCollectionParticipant*) (/home/njn/moz/mi5/go64dmd/dom/xul/../../dist/include/nsISupportsImpl.h:250) 126 #05: nsXULElement::Create(nsXULPrototypeElement*, nsIDocument*, bool, bool,mozilla::dom::Element**) (/home/njn/moz/mi5/dom/xul/nsXULElement.cpp:287) 127 #06: nsXBLContentSink::CreateElement(char16_t const**, unsigned int, mozilla::dom::NodeInfo*, unsigned int, nsIContent**, bool*, mozilla::dom::FromParser) (/home/njn/moz/mi5/dom/xbl/nsXBLContentSink.cpp:874) 128 #07: nsCOMPtr<nsIContent>::StartAssignment() (/home/njn/moz/mi5/go64dmd/dom/xml/../../dist/include/nsCOMPtr.h:753) 129 #08: nsXMLContentSink::HandleStartElement(char16_t const*, char16_t const**, unsigned int, unsigned int, bool) (/home/njn/moz/mi5/dom/xml/nsXMLContentSink.cpp:1007) 130 } 131 } 132 133It tells you that there were 150 heap blocks that were allocated from the 134program point indicated by the "Allocated at" stack trace, that these blocks 135took up 21,600 bytes, that all 150 blocks had a size of 144 bytes, and that 1361,200 of those bytes were "slop" (wasted space caused by the heap allocator 137rounding up request sizes). It also indicates what percentage of the total 138heap size and the unreported portion of the heap these blocks represent. 139 140Within each section, records are listed from largest to smallest. 141 142Once-reported and twice-reported stack trace records also have stack traces for the report point(s). For example: 143 144.. code-block:: 145 146 Reported at { 147 #01: mozilla::dmd::Report(void const*) (/home/njn/moz/mi2/memory/replace/dmd/DMD.cpp:1740) 0x7f68652581ca 148 #02: CycleCollectorMallocSizeOf(void const*) (/home/njn/moz/mi2/xpcom/base/nsCycleCollector.cpp:3008) 0x7f6860fdfe02 149 #03: nsPurpleBuffer::SizeOfExcludingThis(unsigned long (*)(void const*)) const (/home/njn/moz/mi2/xpcom/base/nsCycleCollector.cpp:933) 0x7f6860fdb7af 150 #04: nsCycleCollector::SizeOfIncludingThis(unsigned long (*)(void const*), unsigned long*, unsigned long*, unsigned long*, unsigned long*, unsigned long*) const (/home/njn/moz/mi2/xpcom/base/nsCycleCollector.cpp:3029) 0x7f6860fdb6b1 151 #05: CycleCollectorMultiReporter::CollectReports(nsIMemoryMultiReporterCallback*, nsISupports*) (/home/njn/moz/mi2/xpcom/base/nsCycleCollector.cpp:3075) 0x7f6860fde432 152 #06: nsMemoryInfoDumper::DumpMemoryReportsToFileImpl(nsAString_internal const&) (/home/njn/moz/mi2/xpcom/base/nsMemoryInfoDumper.cpp:626) 0x7f6860fece79 153 #07: nsMemoryInfoDumper::DumpMemoryReportsToFile(nsAString_internal const&, bool, bool) (/home/njn/moz/mi2/xpcom/base/nsMemoryInfoDumper.cpp:344) 0x7f6860febaf9 154 #08: mozilla::(anonymous namespace)::DumpMemoryReportsRunnable::Run() (/home/njn/moz/mi2/xpcom/base/nsMemoryInfoDumper.cpp:58) 0x7f6860fefe03 155 } 156 157You can tell which memory reporter made the report by the name of the 158``MallocSizeOf`` function near the top of the stack trace. In this case it 159was the cycle collector's reporter. 160 161By default, DMD does not record an allocation stack trace for most blocks, to 162make it run faster. The decision on whether to record is done 163probabilistically, and larger blocks are more likely to have an allocation 164stack trace recorded. All unreported blocks that lack an allocation stack 165trace will end up in a single record. For example: 166 167.. code-block:: 168 169 Unreported { 170 420,010 blocks in heap block record 2 of 5,495 171 29,203,408 bytes (27,777,288 requested / 1,426,120 slop) 172 Individual block sizes: 2,048 x 3; 1,024 x 103; 512 x 147; 496 x 7; 480 x 31; 464 x 6; 448 x 50; 432 x 41; 416 x 28; 400 x 53; 384 x 43; 368 x 216; 352 x 141; 336 x 58; 320 x 104; 304 x 5,130; 288 x 150; 272 x 591; 256 x 6,017; 240 x 1,372; 224 x 93; 208 x 488; 192 x 1,919; 176 x 18,903; 160 x 1,754; 144 x 5,041; 128 x 36,709; 112 x 5,571; 96 x 6,280; 80 x 40,738; 64 x 37,925; 48 x 78,392; 32 x 136,199; 16 x 31,001; 8 x 4,706 173 3.78% of the heap (10.24% cumulative) 174 21.24% of unreported (57.53% cumulative) 175 Allocated at { 176 #01: (no stack trace recorded due to --stacks=partial) 177 } 178 } 179 180In contrast, stack traces are always recorded when a block is reported, which 181means you can end up with records like this where the allocation point is 182unknown but the reporting point is known: 183 184.. code-block:: 185 186 Once-reported { 187 104,491 blocks in heap block record 13 of 4,689 188 10,392,000 bytes (10,392,000 requested / 0 slop) 189 Individual block sizes: 512 x 124; 256 x 242; 192 x 813; 128 x 54,664; 64 x 48,648 190 1.35% of the heap (48.65% cumulative) 191 1.64% of once-reported (59.18% cumulative) 192 Allocated at { 193 #01: (no stack trace recorded due to --stacks=partial) 194 } 195 Reported at { 196 #01: mozilla::dmd::DMDFuncs::Report(void const*) (/home/njn/moz/mi5/go64dmd/memory/replace/dmd/../../../../memory/replace/dmd/DMD.cpp:1646) 197 #02: WindowsMallocSizeOf(void const*) (/home/njn/moz/mi5/dom/base/nsWindowMemoryReporter.cpp:189) 198 #03: nsAttrAndChildArray::SizeOfExcludingThis(unsigned long (*)(void const*)) const (/home/njn/moz/mi5/dom/base/nsAttrAndChildArray.cpp:880) 199 #04: mozilla::dom::FragmentOrElement::SizeOfExcludingThis(unsigned long (*)(void const*)) const (/home/njn/moz/mi5/dom/base/FragmentOrElement.cpp:2337) 200 #05: nsINode::SizeOfIncludingThis(unsigned long (*)(void const*)) const (/home/njn/moz/mi5/go64dmd/parser/html/../../../dom/base/nsINode.h:307) 201 #06: mozilla::dom::NodeInfo::NodeType() const (/home/njn/moz/mi5/go64dmd/dom/base/../../dist/include/mozilla/dom/NodeInfo.h:127) 202 #07: nsHTMLDocument::DocAddSizeOfExcludingThis(nsWindowSizes*) const (/home/njn/moz/mi5/dom/html/nsHTMLDocument.cpp:3710) 203 #08: nsIDocument::DocAddSizeOfIncludingThis(nsWindowSizes*) const (/home/njn/moz/mi5/dom/base/nsDocument.cpp:12820) 204 } 205 } 206 207The choice of whether to record an allocation stack trace for all blocks is controlled by an option (see below). 208 209"Live" mode output 210~~~~~~~~~~~~~~~~~~ 211 212For "live" mode, dmd.py's output describes what live heap blocks are present. 213This output is broken into multiple sections. 214 2151. "Invocation". This tells you how DMD was invoked, i.e. what options were used. 2162. "Live stack trace records". This tells you which heap blocks were present. 2173. "Summary": gives measurements of the total heap. 218 219The individual records are similar to those output in "dark matter" mode. 220 221 222"Cumulative" mode output 223~~~~~~~~~~~~~~~~~~~~~~~~ 224 225For "cumulative" mode, dmd.py's output describes how the live heap blocks are 226covered by memory reports. This output is broken into multiple sections. 227 2281. "Invocation". This tells you how DMD was invoked, i.e. what options were used. 2292. "Cumulative stack trace records". This tells you which heap blocks were allocated during the session. 2303. "Summary": gives measurements of the total (cumulative) heap. 231 232The individual records are similar to those output in "dark matter" mode. 233 234 235"Scan" mode output 236~~~~~~~~~~~~~~~~~~ 237 238For "scan" mode, the output of ``dmd.py`` is the same as "live" mode. A 239separate script, ``block_analyzer.py``, can be used to find out information 240about which blocks refer to a particular block. ``dmd.py --clamp-contents`` 241needs to be run on the log first. See this other page for an overview of how 242to use heap scan mode to fix a leak involving refcounted objects. 243 244Options 245------- 246 247Runtime 248~~~~~~~ 249 250When you run ``mach run --dmd`` you can specify additional options to control 251how DMD runs. Run ``mach help run`` for documentation on these. 252 253The most interesting one is ``--mode``. Acceptable values are ``dark-matter`` 254(the default), ``live``, ``cumulative``, and ``scan``. 255 256Another interesting one is ``--stacks``. Acceptable values are ``partial`` 257(the default) and ``full``. In the former case most blocks will not have an 258allocation stack trace recorded. However, because larger blocks are more 259likely to have one recorded, most allocated bytes should have an allocation 260stack trace even though most allocated blocks do not. Use ``--stacks=full`` 261if you want complete information, but note that DMD will run substantially 262slower in that case. 263 264The options may also be put in the environment variable DMD, or set DMD to 1 265to enable DMD with default options (dark-matter and partial stacks). 266 267The ``MOZ_DMD_SHUTDOWN_LOG`` environment variable, if set, triggers a DMD run 268at shutdown; its value must be a directory where the logs will be placed. 269Which processes get logged is controlled by the ``MOZ_DMD_LOG_PROCESS`` 270environment variable, which can take the following values. 271 272* Unset: log all processes. 273* "default": log the parent process only. 274* "tab": log content processes only. 275 276For example, if you have 277 278.. code-block:: 279 280 MOZ_DMD_SHUTDOWN_LOG=~/dmdlogs/ MOZ_DMD_LOG_PROCESS=tab 281 282then DMD logs for content processes will be saved to ~/dmdlogs/. 283 284.. note:: 285 286 To dump DMD data from Content processes, you'll need to disable the sandbox 287 288.. note:: 289 290 ``MOZ_DMD_SHUTDOWN_LOG`` must (currently) include the trailing separator (''/") 291 292 293Post-processing 294~~~~~~~~~~~~~~~ 295 296``dmd.py`` also takes options that control how it works. Run ``dmd.py -h`` 297for documentation. The following options are the most interesting ones. 298 299* ``-f`` / ``--max-frames``. By default, records show up to 8 stack frames. You 300 can choose a smaller number, in which case more allocations will be 301 aggregated into each record, but you'll have less context. Or you can choose 302 a larger number, in which cases allocations will be split across more 303 records, but you will have more context. There is no single best value, but 304 values in the range 2..10 are often good. The maximum is 24. 305 306* ``-a`` / ``--ignore-alloc-frames``. Many allocation stack traces start with 307 multiple frames that mention allocation wrapper functions, e.g. 308 ``js_calloc()`` calls replace_calloc(). This option filters these out. It 309 often helps improve the quality of the output when using a small 310 ``--max-frames`` value. 311 312* ``-s`` / ``--sort-by``. This controls how records are sorted. Acceptable 313 values are usable (the default), ``req``, ``slop`` and ``num-blocks``. 314 315* ``--clamp-contents``. For a heap scan log, this performs a conservative 316 pointer analysis on the contents of each block, changing any value that is a 317 pointer into the middle of a live block into a pointer to the start of that 318 block. All other values are changes to null. In addition, all trailing nulls 319 are removed from the block contents. 320 321As an example that combines multiple options, if you apply the following 322command to a profile obtained in "live" mode: 323 324.. code-block:: 325 326 dmd.py -r -f 2 -a -s slop 327 328it will give you a good idea of where the major sources of slop are. 329 330``dmd.py`` can also compute the difference between two DMD output files, so 331long as those files were produced in the same mode. Simply pass it two 332filenames instead of one to get the difference. 333 334 335Which heap blocks are reported? 336------------------------------- 337 338At this stage you might wonder how DMD knows, in "dark matter" mode, which 339allocations have been reported and which haven't. DMD only knows about heap 340blocks that are measured via a function created with one of the following two 341macros: 342 343.. code-block:: 344 345 MOZ_DEFINE_MALLOC_SIZE_OF 346 MOZ_DEFINE_MALLOC_SIZE_OF_ON_ALLOC 347 348Fortunately, most of the existing memory reporters do this. 349