1# Key Concepts in Chrome Memory 2 3## What's so hard about memory? Isn't it just malloc and free? 4 5Not really. There are lots of differences and subtleties that change per 6operating system and even per operating system configuration. 7 8Fortunately, these differences mostly disappear when a program is running 9with sufficient resources. 10 11Unfortunately, the distinctions end up being very relevant when 12working near out of memory conditions or analyzing overall performance 13when there is any amount of memory pressure; this makes crafting and 14interpreting memory statistics hard. 15 16Fortunately, the point of this doc is to give succinct background that 17will help you ramp up on the subtleties to work in this space. Yes, this 18is complicated stuff...but don't despair. You work on a multi-process 19browser implementing the web platform with high security guarantees. 20Compared to the rest the system, memory is not THAT complicated. 21 22## Can you give specific examples of how it's harder than malloc/free? 23 24Here are some example questions that require a more complex 25view of memory than malloc/free. 26 27 * When Chrome allocates memory, when does it take up swap space? 28 * When memory is `free()`d, when is it made usable by other applications? 29 * Is it always safe to touch the memory returned by malloc()? 30 * How many heaps does Chrome have? 31 * How are memory resources used by the GPU and drivers accounted for? 32 * Is that the same on systems where GPU memory isn't shared with main memory? 33 * How are shared libraries accounted for? How big of a penalty is there for 34 each process that shares the memory? 35 * What types of memory does Task Manager/Activity Monitor/top report? 36 * What about the UMA stats? 37 38In many of the above, the answer actually changes per operating system variant. 39There is at least one major schism between Windows-based machines and more 40unixy systems. For example, it is impossible to return all resources (physical 41ram as well as swap space) to the OS in a way brings them back on demand which 42drastically changes the way one can handle free lists. 43 44However, even in macOS, Android, CrOS, and "standard desktop linux" each 45also have enough divergences (compressed memory, pagefile vs swap partition 46vs no swap, overcommit settings, memory perssure signals etc) that even 47answering "how much memory is Chromium using" is hard to do in a uniform 48manner. 49 50The goal of this document is to give a common set of vocabulary 51and concepts such that Chromium developers can more discuss questions like 52the ones above without misunderstanding each other. 53 54 55## Key gotchas 56 57### Windows allocation uses resources immediately; other OSes use it on first touch. 58 59Arguably the biggest difference for Windows and other OSes is memory granted to 60a process is always "committed" on allocation. Pragmatically this means that in 61Windows, `malloc(10*1024*1024*1024)` will immediately prevent other applications 62from being able to successfully allocate memory thereby causing them to crash 63or not be able to open. In Unix variants, usage usually only consumes system 64resources [TODO(awong): Link to overcommit] when pages are touched. 65 66Not being aware of this difference can cause architecture choices that have a 67larger than expected resource impact on Windows and incorrect interpretation for metrics on Windows 68 69See the following section on "discardable" memory for more info. 70 71 72### Because of the commit guarantee difference, "discarding" memory has completely different meanings across platforms. 73 74In Unix systems, there is an `madvise()` function via which pages that have 75been committed via usage can be returned to the non-resource consuming state. 76Such a page will then be recommitted on demand making it a tempting optimization 77for data structures with freelists. However, there is no such API on Windows. 78The `VirtualAlloc(MEM_RESET)`, `DiscardVirtualMemory()`, and 79`OfferVirtualMemory()` look temptingly similar and on first glance they even 80look like they work because they will immediately reduce the amount of physical 81ram (aka Working Set) a processes uses. However, they do NOT release swap 82meaning they will not help prevent OOM scenarios. 83 84Designing a freelist structure that conflates this behavior (see this 85[PartitionAlloc bug](https://bugs.chromium.org/p/chromium/issues/detail?id=726077)) 86will result in a system that only truly reduces resource usage on Unix-like 87systems. 88 89 90## Terms and definitions 91 92Each platform exposes a different memory model. This section describes a 93consistent set of terminology that will be used by this document. This 94terminology is intentionally Linux-biased, since that is the platform most 95readers are expected to be familiar with. 96 97### Supported platforms 98* Linux 99* Android 100* ChromeOS 101* Windows [kernel: Windows NT] 102* macOS/iOS [kernel: Darwin/XNU/Mach] 103 104### Terminology 105Warning: This terminology is neither complete, nor precise, when compared to the 106terminology used by any specific platform. Any in-depth discussion should occur 107on a per-platform basis, and use terminology specific to that platform. 108 109* **Virtual memory** - A per-process abstraction layer exposed by the kernel. A 110 contiguous region divided into 4kb **virtual pages**. 111* **Physical memory** - A per-machine abstraction layer internal to the kernel. 112 A contiguous region divided into 4kb **physical pages**. Each **physical 113 page** represents 4kb of physical memory. 114* **Resident** - A virtual page whose contents is backed by a physical 115 page. 116* **Swapped/Compressed** - A virtual page whose contents is backed by 117 something other than a physical page. 118* **Swapping/Compression** - [verb] The process of taking Resident pages and 119 making them Swapped/Compressed pages. This frees up physical pages. 120* **Unlocked Discardable/Reusable** - Android [Ashmem] and Darwin specific. A virtual 121 page whose contents is backed by a physical page, but the Kernel is free 122 to reuse the physical page at any point in time. 123* **Private** - A virtual page whose contents will only be modifiable by the 124 current process. 125* **Copy on Write** - A private virtual page owned by the parent process. 126 When either the parent or child process attempts to make a modification, the 127 child is given a private copy of the page. 128* **Shared** - A virtual page whose contents could be shared with other 129 processes. 130* **File-backed** - A virtual page whose contents reflect those of a 131 file. 132* **Anonymous** - A virtual page that is not file-backed. 133 134## Platform Specific Sources of Truth 135Memory is a complex topic, fraught with potential miscommunications. In an 136attempt to forestall disagreement over semantics, these are the sources of truth 137used to determine memory usage for a given process. 138 139* Windows: [SysInternals 140 VMMap](https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap) 141* Darwin: 142 [vmmap](https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/vmmap.1.html) 143* Linux/Derivatives: 144 [/proc/<pid\>/smaps](http://man7.org/linux/man-pages/man5/proc.5.html) 145 146## Shared Memory 147 148Accounting for shared memory is poorly defined. If a memory region is mapped 149into multiple processes [possibly multiple times], which ones should it count 150towards? 151 152On Linux, one common solution is to use proportional set size, which counts 1531/Nth of the resident size, where N is the number of other processes that have 154page faulted the region. This has the nice property of being additive across 155processes. The downside is that it is context dependent. e.g. If a user opens 156more tabs, thus causing a system library to be mapped into more processes, the 157PSS for previous tabs will go down. 158 159File backed shared memory regions are typically not interesting to report, since 160they typically represent shared system resources, libraries, and the browser 161binary itself, all of which are outside of the control of developers. This is 162particularly problematic across different versions of the OS, where the set of 163base libraries that get linked by default into a process highly varies, out of 164Chrome's control. 165 166In Chrome, we have implemented ownership tracking for anonymous shared memory 167regions - each shared memory region counts towards exactly one process, which is 168determined by the type and usage of the shared memory region. 169