1:mod:`tracemalloc` --- Trace memory allocations 2=============================================== 3 4.. module:: tracemalloc 5 :synopsis: Trace memory allocations. 6 7.. versionadded:: 3.4 8 9**Source code:** :source:`Lib/tracemalloc.py` 10 11-------------- 12 13The tracemalloc module is a debug tool to trace memory blocks allocated by 14Python. It provides the following information: 15 16* Traceback where an object was allocated 17* Statistics on allocated memory blocks per filename and per line number: 18 total size, number and average size of allocated memory blocks 19* Compute the differences between two snapshots to detect memory leaks 20 21To trace most memory blocks allocated by Python, the module should be started 22as early as possible by setting the :envvar:`PYTHONTRACEMALLOC` environment 23variable to ``1``, or by using :option:`-X` ``tracemalloc`` command line 24option. The :func:`tracemalloc.start` function can be called at runtime to 25start tracing Python memory allocations. 26 27By default, a trace of an allocated memory block only stores the most recent 28frame (1 frame). To store 25 frames at startup: set the 29:envvar:`PYTHONTRACEMALLOC` environment variable to ``25``, or use the 30:option:`-X` ``tracemalloc=25`` command line option. 31 32 33Examples 34-------- 35 36Display the top 10 37^^^^^^^^^^^^^^^^^^ 38 39Display the 10 files allocating the most memory:: 40 41 import tracemalloc 42 43 tracemalloc.start() 44 45 # ... run your application ... 46 47 snapshot = tracemalloc.take_snapshot() 48 top_stats = snapshot.statistics('lineno') 49 50 print("[ Top 10 ]") 51 for stat in top_stats[:10]: 52 print(stat) 53 54 55Example of output of the Python test suite:: 56 57 [ Top 10 ] 58 <frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B 59 <frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B 60 /usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B 61 /usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B 62 /usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B 63 /usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B 64 <frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B 65 <frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B 66 <string>:5: size=49.7 KiB, count=148, average=344 B 67 /usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB 68 69We can see that Python loaded ``4855 KiB`` data (bytecode and constants) from 70modules and that the :mod:`collections` module allocated ``244 KiB`` to build 71:class:`~collections.namedtuple` types. 72 73See :meth:`Snapshot.statistics` for more options. 74 75 76Compute differences 77^^^^^^^^^^^^^^^^^^^ 78 79Take two snapshots and display the differences:: 80 81 import tracemalloc 82 tracemalloc.start() 83 # ... start your application ... 84 85 snapshot1 = tracemalloc.take_snapshot() 86 # ... call the function leaking memory ... 87 snapshot2 = tracemalloc.take_snapshot() 88 89 top_stats = snapshot2.compare_to(snapshot1, 'lineno') 90 91 print("[ Top 10 differences ]") 92 for stat in top_stats[:10]: 93 print(stat) 94 95Example of output before/after running some tests of the Python test suite:: 96 97 [ Top 10 differences ] 98 <frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B 99 /usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B 100 /usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B 101 <frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B 102 /usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B 103 /usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB 104 /usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B 105 /usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B 106 /usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B 107 /usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B 108 109We can see that Python has loaded ``8173 KiB`` of module data (bytecode and 110constants), and that this is ``4428 KiB`` more than had been loaded before the 111tests, when the previous snapshot was taken. Similarly, the :mod:`linecache` 112module has cached ``940 KiB`` of Python source code to format tracebacks, all 113of it since the previous snapshot. 114 115If the system has little free memory, snapshots can be written on disk using 116the :meth:`Snapshot.dump` method to analyze the snapshot offline. Then use the 117:meth:`Snapshot.load` method reload the snapshot. 118 119 120Get the traceback of a memory block 121^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 122 123Code to display the traceback of the biggest memory block:: 124 125 import tracemalloc 126 127 # Store 25 frames 128 tracemalloc.start(25) 129 130 # ... run your application ... 131 132 snapshot = tracemalloc.take_snapshot() 133 top_stats = snapshot.statistics('traceback') 134 135 # pick the biggest memory block 136 stat = top_stats[0] 137 print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024)) 138 for line in stat.traceback.format(): 139 print(line) 140 141Example of output of the Python test suite (traceback limited to 25 frames):: 142 143 903 memory blocks: 870.1 KiB 144 File "<frozen importlib._bootstrap>", line 716 145 File "<frozen importlib._bootstrap>", line 1036 146 File "<frozen importlib._bootstrap>", line 934 147 File "<frozen importlib._bootstrap>", line 1068 148 File "<frozen importlib._bootstrap>", line 619 149 File "<frozen importlib._bootstrap>", line 1581 150 File "<frozen importlib._bootstrap>", line 1614 151 File "/usr/lib/python3.4/doctest.py", line 101 152 import pdb 153 File "<frozen importlib._bootstrap>", line 284 154 File "<frozen importlib._bootstrap>", line 938 155 File "<frozen importlib._bootstrap>", line 1068 156 File "<frozen importlib._bootstrap>", line 619 157 File "<frozen importlib._bootstrap>", line 1581 158 File "<frozen importlib._bootstrap>", line 1614 159 File "/usr/lib/python3.4/test/support/__init__.py", line 1728 160 import doctest 161 File "/usr/lib/python3.4/test/test_pickletools.py", line 21 162 support.run_doctest(pickletools) 163 File "/usr/lib/python3.4/test/regrtest.py", line 1276 164 test_runner() 165 File "/usr/lib/python3.4/test/regrtest.py", line 976 166 display_failure=not verbose) 167 File "/usr/lib/python3.4/test/regrtest.py", line 761 168 match_tests=ns.match_tests) 169 File "/usr/lib/python3.4/test/regrtest.py", line 1563 170 main() 171 File "/usr/lib/python3.4/test/__main__.py", line 3 172 regrtest.main_in_temp_cwd() 173 File "/usr/lib/python3.4/runpy.py", line 73 174 exec(code, run_globals) 175 File "/usr/lib/python3.4/runpy.py", line 160 176 "__main__", fname, loader, pkg_name) 177 178We can see that the most memory was allocated in the :mod:`importlib` module to 179load data (bytecode and constants) from modules: ``870.1 KiB``. The traceback is 180where the :mod:`importlib` loaded data most recently: on the ``import pdb`` 181line of the :mod:`doctest` module. The traceback may change if a new module is 182loaded. 183 184 185Pretty top 186^^^^^^^^^^ 187 188Code to display the 10 lines allocating the most memory with a pretty output, 189ignoring ``<frozen importlib._bootstrap>`` and ``<unknown>`` files:: 190 191 import linecache 192 import os 193 import tracemalloc 194 195 def display_top(snapshot, key_type='lineno', limit=10): 196 snapshot = snapshot.filter_traces(( 197 tracemalloc.Filter(False, "<frozen importlib._bootstrap>"), 198 tracemalloc.Filter(False, "<unknown>"), 199 )) 200 top_stats = snapshot.statistics(key_type) 201 202 print("Top %s lines" % limit) 203 for index, stat in enumerate(top_stats[:limit], 1): 204 frame = stat.traceback[0] 205 print("#%s: %s:%s: %.1f KiB" 206 % (index, frame.filename, frame.lineno, stat.size / 1024)) 207 line = linecache.getline(frame.filename, frame.lineno).strip() 208 if line: 209 print(' %s' % line) 210 211 other = top_stats[limit:] 212 if other: 213 size = sum(stat.size for stat in other) 214 print("%s other: %.1f KiB" % (len(other), size / 1024)) 215 total = sum(stat.size for stat in top_stats) 216 print("Total allocated size: %.1f KiB" % (total / 1024)) 217 218 tracemalloc.start() 219 220 # ... run your application ... 221 222 snapshot = tracemalloc.take_snapshot() 223 display_top(snapshot) 224 225Example of output of the Python test suite:: 226 227 Top 10 lines 228 #1: Lib/base64.py:414: 419.8 KiB 229 _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] 230 #2: Lib/base64.py:306: 419.8 KiB 231 _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] 232 #3: collections/__init__.py:368: 293.6 KiB 233 exec(class_definition, namespace) 234 #4: Lib/abc.py:133: 115.2 KiB 235 cls = super().__new__(mcls, name, bases, namespace) 236 #5: unittest/case.py:574: 103.1 KiB 237 testMethod() 238 #6: Lib/linecache.py:127: 95.4 KiB 239 lines = fp.readlines() 240 #7: urllib/parse.py:476: 71.8 KiB 241 for a in _hexdig for b in _hexdig} 242 #8: <string>:5: 62.0 KiB 243 #9: Lib/_weakrefset.py:37: 60.0 KiB 244 self.data = set() 245 #10: Lib/base64.py:142: 59.8 KiB 246 _b32tab2 = [a + b for a in _b32tab for b in _b32tab] 247 6220 other: 3602.8 KiB 248 Total allocated size: 5303.1 KiB 249 250See :meth:`Snapshot.statistics` for more options. 251 252 253API 254--- 255 256Functions 257^^^^^^^^^ 258 259.. function:: clear_traces() 260 261 Clear traces of memory blocks allocated by Python. 262 263 See also :func:`stop`. 264 265 266.. function:: get_object_traceback(obj) 267 268 Get the traceback where the Python object *obj* was allocated. 269 Return a :class:`Traceback` instance, or ``None`` if the :mod:`tracemalloc` 270 module is not tracing memory allocations or did not trace the allocation of 271 the object. 272 273 See also :func:`gc.get_referrers` and :func:`sys.getsizeof` functions. 274 275 276.. function:: get_traceback_limit() 277 278 Get the maximum number of frames stored in the traceback of a trace. 279 280 The :mod:`tracemalloc` module must be tracing memory allocations to 281 get the limit, otherwise an exception is raised. 282 283 The limit is set by the :func:`start` function. 284 285 286.. function:: get_traced_memory() 287 288 Get the current size and peak size of memory blocks traced by the 289 :mod:`tracemalloc` module as a tuple: ``(current: int, peak: int)``. 290 291 292.. function:: get_tracemalloc_memory() 293 294 Get the memory usage in bytes of the :mod:`tracemalloc` module used to store 295 traces of memory blocks. 296 Return an :class:`int`. 297 298 299.. function:: is_tracing() 300 301 ``True`` if the :mod:`tracemalloc` module is tracing Python memory 302 allocations, ``False`` otherwise. 303 304 See also :func:`start` and :func:`stop` functions. 305 306 307.. function:: start(nframe: int=1) 308 309 Start tracing Python memory allocations: install hooks on Python memory 310 allocators. Collected tracebacks of traces will be limited to *nframe* 311 frames. By default, a trace of a memory block only stores the most recent 312 frame: the limit is ``1``. *nframe* must be greater or equal to ``1``. 313 314 Storing more than ``1`` frame is only useful to compute statistics grouped 315 by ``'traceback'`` or to compute cumulative statistics: see the 316 :meth:`Snapshot.compare_to` and :meth:`Snapshot.statistics` methods. 317 318 Storing more frames increases the memory and CPU overhead of the 319 :mod:`tracemalloc` module. Use the :func:`get_tracemalloc_memory` function 320 to measure how much memory is used by the :mod:`tracemalloc` module. 321 322 The :envvar:`PYTHONTRACEMALLOC` environment variable 323 (``PYTHONTRACEMALLOC=NFRAME``) and the :option:`-X` ``tracemalloc=NFRAME`` 324 command line option can be used to start tracing at startup. 325 326 See also :func:`stop`, :func:`is_tracing` and :func:`get_traceback_limit` 327 functions. 328 329 330.. function:: stop() 331 332 Stop tracing Python memory allocations: uninstall hooks on Python memory 333 allocators. Also clears all previously collected traces of memory blocks 334 allocated by Python. 335 336 Call :func:`take_snapshot` function to take a snapshot of traces before 337 clearing them. 338 339 See also :func:`start`, :func:`is_tracing` and :func:`clear_traces` 340 functions. 341 342 343.. function:: take_snapshot() 344 345 Take a snapshot of traces of memory blocks allocated by Python. Return a new 346 :class:`Snapshot` instance. 347 348 The snapshot does not include memory blocks allocated before the 349 :mod:`tracemalloc` module started to trace memory allocations. 350 351 Tracebacks of traces are limited to :func:`get_traceback_limit` frames. Use 352 the *nframe* parameter of the :func:`start` function to store more frames. 353 354 The :mod:`tracemalloc` module must be tracing memory allocations to take a 355 snapshot, see the :func:`start` function. 356 357 See also the :func:`get_object_traceback` function. 358 359 360DomainFilter 361^^^^^^^^^^^^ 362 363.. class:: DomainFilter(inclusive: bool, domain: int) 364 365 Filter traces of memory blocks by their address space (domain). 366 367 .. versionadded:: 3.6 368 369 .. attribute:: inclusive 370 371 If *inclusive* is ``True`` (include), match memory blocks allocated 372 in the address space :attr:`domain`. 373 374 If *inclusive* is ``False`` (exclude), match memory blocks not allocated 375 in the address space :attr:`domain`. 376 377 .. attribute:: domain 378 379 Address space of a memory block (``int``). Read-only property. 380 381 382Filter 383^^^^^^ 384 385.. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False, domain: int=None) 386 387 Filter on traces of memory blocks. 388 389 See the :func:`fnmatch.fnmatch` function for the syntax of 390 *filename_pattern*. The ``'.pyc'`` file extension is 391 replaced with ``'.py'``. 392 393 Examples: 394 395 * ``Filter(True, subprocess.__file__)`` only includes traces of the 396 :mod:`subprocess` module 397 * ``Filter(False, tracemalloc.__file__)`` excludes traces of the 398 :mod:`tracemalloc` module 399 * ``Filter(False, "<unknown>")`` excludes empty tracebacks 400 401 402 .. versionchanged:: 3.5 403 The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. 404 405 .. versionchanged:: 3.6 406 Added the :attr:`domain` attribute. 407 408 409 .. attribute:: domain 410 411 Address space of a memory block (``int`` or ``None``). 412 413 tracemalloc uses the domain ``0`` to trace memory allocations made by 414 Python. C extensions can use other domains to trace other resources. 415 416 .. attribute:: inclusive 417 418 If *inclusive* is ``True`` (include), only match memory blocks allocated 419 in a file with a name matching :attr:`filename_pattern` at line number 420 :attr:`lineno`. 421 422 If *inclusive* is ``False`` (exclude), ignore memory blocks allocated in 423 a file with a name matching :attr:`filename_pattern` at line number 424 :attr:`lineno`. 425 426 .. attribute:: lineno 427 428 Line number (``int``) of the filter. If *lineno* is ``None``, the filter 429 matches any line number. 430 431 .. attribute:: filename_pattern 432 433 Filename pattern of the filter (``str``). Read-only property. 434 435 .. attribute:: all_frames 436 437 If *all_frames* is ``True``, all frames of the traceback are checked. If 438 *all_frames* is ``False``, only the most recent frame is checked. 439 440 This attribute has no effect if the traceback limit is ``1``. See the 441 :func:`get_traceback_limit` function and :attr:`Snapshot.traceback_limit` 442 attribute. 443 444 445Frame 446^^^^^ 447 448.. class:: Frame 449 450 Frame of a traceback. 451 452 The :class:`Traceback` class is a sequence of :class:`Frame` instances. 453 454 .. attribute:: filename 455 456 Filename (``str``). 457 458 .. attribute:: lineno 459 460 Line number (``int``). 461 462 463Snapshot 464^^^^^^^^ 465 466.. class:: Snapshot 467 468 Snapshot of traces of memory blocks allocated by Python. 469 470 The :func:`take_snapshot` function creates a snapshot instance. 471 472 .. method:: compare_to(old_snapshot: Snapshot, key_type: str, cumulative: bool=False) 473 474 Compute the differences with an old snapshot. Get statistics as a sorted 475 list of :class:`StatisticDiff` instances grouped by *key_type*. 476 477 See the :meth:`Snapshot.statistics` method for *key_type* and *cumulative* 478 parameters. 479 480 The result is sorted from the biggest to the smallest by: absolute value 481 of :attr:`StatisticDiff.size_diff`, :attr:`StatisticDiff.size`, absolute 482 value of :attr:`StatisticDiff.count_diff`, :attr:`Statistic.count` and 483 then by :attr:`StatisticDiff.traceback`. 484 485 486 .. method:: dump(filename) 487 488 Write the snapshot into a file. 489 490 Use :meth:`load` to reload the snapshot. 491 492 493 .. method:: filter_traces(filters) 494 495 Create a new :class:`Snapshot` instance with a filtered :attr:`traces` 496 sequence, *filters* is a list of :class:`DomainFilter` and 497 :class:`Filter` instances. If *filters* is an empty list, return a new 498 :class:`Snapshot` instance with a copy of the traces. 499 500 All inclusive filters are applied at once, a trace is ignored if no 501 inclusive filters match it. A trace is ignored if at least one exclusive 502 filter matches it. 503 504 .. versionchanged:: 3.6 505 :class:`DomainFilter` instances are now also accepted in *filters*. 506 507 508 .. classmethod:: load(filename) 509 510 Load a snapshot from a file. 511 512 See also :meth:`dump`. 513 514 515 .. method:: statistics(key_type: str, cumulative: bool=False) 516 517 Get statistics as a sorted list of :class:`Statistic` instances grouped 518 by *key_type*: 519 520 ===================== ======================== 521 key_type description 522 ===================== ======================== 523 ``'filename'`` filename 524 ``'lineno'`` filename and line number 525 ``'traceback'`` traceback 526 ===================== ======================== 527 528 If *cumulative* is ``True``, cumulate size and count of memory blocks of 529 all frames of the traceback of a trace, not only the most recent frame. 530 The cumulative mode can only be used with *key_type* equals to 531 ``'filename'`` and ``'lineno'``. 532 533 The result is sorted from the biggest to the smallest by: 534 :attr:`Statistic.size`, :attr:`Statistic.count` and then by 535 :attr:`Statistic.traceback`. 536 537 538 .. attribute:: traceback_limit 539 540 Maximum number of frames stored in the traceback of :attr:`traces`: 541 result of the :func:`get_traceback_limit` when the snapshot was taken. 542 543 .. attribute:: traces 544 545 Traces of all memory blocks allocated by Python: sequence of 546 :class:`Trace` instances. 547 548 The sequence has an undefined order. Use the :meth:`Snapshot.statistics` 549 method to get a sorted list of statistics. 550 551 552Statistic 553^^^^^^^^^ 554 555.. class:: Statistic 556 557 Statistic on memory allocations. 558 559 :func:`Snapshot.statistics` returns a list of :class:`Statistic` instances. 560 561 See also the :class:`StatisticDiff` class. 562 563 .. attribute:: count 564 565 Number of memory blocks (``int``). 566 567 .. attribute:: size 568 569 Total size of memory blocks in bytes (``int``). 570 571 .. attribute:: traceback 572 573 Traceback where the memory block was allocated, :class:`Traceback` 574 instance. 575 576 577StatisticDiff 578^^^^^^^^^^^^^ 579 580.. class:: StatisticDiff 581 582 Statistic difference on memory allocations between an old and a new 583 :class:`Snapshot` instance. 584 585 :func:`Snapshot.compare_to` returns a list of :class:`StatisticDiff` 586 instances. See also the :class:`Statistic` class. 587 588 .. attribute:: count 589 590 Number of memory blocks in the new snapshot (``int``): ``0`` if 591 the memory blocks have been released in the new snapshot. 592 593 .. attribute:: count_diff 594 595 Difference of number of memory blocks between the old and the new 596 snapshots (``int``): ``0`` if the memory blocks have been allocated in 597 the new snapshot. 598 599 .. attribute:: size 600 601 Total size of memory blocks in bytes in the new snapshot (``int``): 602 ``0`` if the memory blocks have been released in the new snapshot. 603 604 .. attribute:: size_diff 605 606 Difference of total size of memory blocks in bytes between the old and 607 the new snapshots (``int``): ``0`` if the memory blocks have been 608 allocated in the new snapshot. 609 610 .. attribute:: traceback 611 612 Traceback where the memory blocks were allocated, :class:`Traceback` 613 instance. 614 615 616Trace 617^^^^^ 618 619.. class:: Trace 620 621 Trace of a memory block. 622 623 The :attr:`Snapshot.traces` attribute is a sequence of :class:`Trace` 624 instances. 625 626 .. versionchanged:: 3.6 627 Added the :attr:`domain` attribute. 628 629 .. attribute:: domain 630 631 Address space of a memory block (``int``). Read-only property. 632 633 tracemalloc uses the domain ``0`` to trace memory allocations made by 634 Python. C extensions can use other domains to trace other resources. 635 636 .. attribute:: size 637 638 Size of the memory block in bytes (``int``). 639 640 .. attribute:: traceback 641 642 Traceback where the memory block was allocated, :class:`Traceback` 643 instance. 644 645 646Traceback 647^^^^^^^^^ 648 649.. class:: Traceback 650 651 Sequence of :class:`Frame` instances sorted from the oldest frame to the 652 most recent frame. 653 654 A traceback contains at least ``1`` frame. If the ``tracemalloc`` module 655 failed to get a frame, the filename ``"<unknown>"`` at line number ``0`` is 656 used. 657 658 When a snapshot is taken, tracebacks of traces are limited to 659 :func:`get_traceback_limit` frames. See the :func:`take_snapshot` function. 660 661 The :attr:`Trace.traceback` attribute is an instance of :class:`Traceback` 662 instance. 663 664 .. versionchanged:: 3.7 665 Frames are now sorted from the oldest to the most recent, instead of most recent to oldest. 666 667 .. method:: format(limit=None, most_recent_first=False) 668 669 Format the traceback as a list of lines with newlines. Use the 670 :mod:`linecache` module to retrieve lines from the source code. 671 If *limit* is set, format the *limit* most recent frames if *limit* 672 is positive. Otherwise, format the ``abs(limit)`` oldest frames. 673 If *most_recent_first* is ``True``, the order of the formatted frames 674 is reversed, returning the most recent frame first instead of last. 675 676 Similar to the :func:`traceback.format_tb` function, except that 677 :meth:`.format` does not include newlines. 678 679 Example:: 680 681 print("Traceback (most recent call first):") 682 for line in traceback: 683 print(line) 684 685 Output:: 686 687 Traceback (most recent call first): 688 File "test.py", line 9 689 obj = Object() 690 File "test.py", line 12 691 tb = tracemalloc.get_object_traceback(f()) 692