1Graphics Overview 2========================= 3 4Work in progress. Possibly incorrect or incomplete. 5--------------------------------------------------- 6 7Jargon 8------ 9 10There's a lot of jargon in the graphics stack. We try to maintain a list 11of common words and acronyms `here <https://wiki.mozilla.org/Platform/GFX/Jargon>`__. 12 13Overview 14-------- 15 16The graphics systems is responsible for rendering (painting, drawing) 17the frame tree (rendering tree) elements as created by the layout 18system. Each leaf in the tree has content, either bounded by a rectangle 19(or perhaps another shape, in the case of SVG.) 20 21The simple approach for producing the result would thus involve 22traversing the frame tree, in a correct order, drawing each frame into 23the resulting buffer and displaying (printing non-withstanding) that 24buffer when the traversal is done. It is worth spending some time on the 25“correct order” note above. If there are no overlapping frames, this is 26fairly simple - any order will do, as long as there is no background. If 27there is background, we just have to worry about drawing that first. 28Since we do not control the content, chances are the page is more 29complicated. There are overlapping frames, likely with transparency, so 30we need to make sure the elements are draw “back to front”, in layers, 31so to speak. Layers are an important concept, and we will revisit them 32shortly, as they are central to fixing a major issue with the above 33simple approach. 34 35While the above simple approach will work, the performance will suffer. 36Each time anything changes in any of the frames, the complete process 37needs to be repeated, everything needs to be redrawn. Further, there is 38very little space to take advantage of the modern graphics (GPU) 39hardware, or multi-core computers. If you recall from the previous 40sections, the frame tree is only accessible from the UI thread, so while 41we’re doing all this work, the UI is basically blocked. 42 43(Retained) Layers 44~~~~~~~~~~~~~~~~~ 45 46Layers framework was introduced to address the above performance issues, 47by having a part of the design address each item. At the high level: 48 491. We create a layer tree. The leaf elements of the tree contain all 50 frames (possibly multiple frames per leaf). 512. We render each layer tree element and cache (retain) the result. 523. We composite (combine) all the leaf elements into the final result. 53 54Let’s examine each of these steps, in reverse order. 55 56Compositing 57~~~~~~~~~~~ 58 59We use the term composite as it implies that the order is important. If 60the elements being composited overlap, whether there is transparency 61involved or not, the order in which they are combined will effect the 62result. Compositing is where we can use some of the power of the modern 63graphics hardware. It is optimal for doing this job. In the scenarios 64where only the position of individual frames changes, without the 65content inside them changing, we see why caching each layer would be 66advantageous - we only need to repeat the final compositing step, 67completely skipping the layer tree creation and the rendering of each 68leaf, thus speeding up the process considerably. 69 70Another benefit is equally apparent in the context of the stated 71deficiencies of the simple approach. We can use the available graphics 72hardware accelerated APIs to do the compositing step. Direct3D, OpenGL 73can be used on different platforms and are well suited to accelerate 74this step. 75 76Finally, we can now envision performing the compositing step on a 77separate thread, unblocking the UI thread for other work, and doing more 78work in parallel. More on this below. 79 80It is important to note that the number of operations in this step is 81proportional to the number of layer tree (leaf) elements, so there is 82additional work and complexity involved, when the layer tree is large. 83 84Render and retain layer elements 85~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 86 87As we saw, the compositing step benefits from caching the intermediate 88result. This does result in the extra memory usage, so needs to be 89considered during the layer tree creation. Beyond the caching, we can 90accelerate the rendering of each element by (indirectly) using the 91available platform APIs (e.g., Direct2D, CoreGraphics, even some of the 923D APIs like OpenGL or Direct3D) as available. This is actually done 93through a platform independent API (see Moz2D) below, but is important 94to realize it does get accelerated appropriately. 95 96Creating the layer tree 97~~~~~~~~~~~~~~~~~~~~~~~ 98 99We need to create a layer tree (from the frames tree), which will give 100us the correct result while striking the right balance between a layer 101per frame element and a single layer for the complete frames tree. As 102was mentioned above, there is an overhead in traversing the whole tree 103and caching each of the elements, balanced by the performance 104improvements. Some of the performance improvements are only noticed when 105something changes (e.g., one element is moving, we only need to redo the 106compositing step). 107 108Refresh Driver 109~~~~~~~~~~~~~~ 110 111Layers 112~~~~~~ 113 114Rendering each layer 115~~~~~~~~~~~~~~~~~~~~ 116 117Tiling vs. Buffer Rotation vs. Full paint 118~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 119 120Compositing for the final result 121~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 122 123Graphics API 124~~~~~~~~~~~~ 125 126Compositing 127~~~~~~~~~~~ 128 129Image Decoding 130~~~~~~~~~~~~~~ 131 132Image Animation 133~~~~~~~~~~~~~~~ 134 135`Historical Documents <http://www.youtube.com/watch?v=lLZQz26-kms>`__ 136--------------------------------------------------------------------- 137 138A number of posts and blogs that will give you more details or more 139background, or reasoning that led to different solutions and approaches. 140 141- 2010-01 `Layers: Cross Platform Acceleration <http://www.basschouten.com/blog1.php/layers-cross-platform-acceleration>`__ 142- 2010-04 `Layers <http://robert.ocallahan.org/2010/04/layers_01.html>`__ 143- 2010-07 `Retained Layers <http://robert.ocallahan.org/2010/07/retained-layers_16.html>`__ 144- 2011-04 `Introduction <https://web.archive.org/web/20140604005804/https://blog.mozilla.org/joe/2011/04/26/introducing-the-azure-project/>`__ 145- 2011-07 `Layers <http://chrislord.net/index.php/2011/07/25/shadow-layers-and-learning-by-failing/%20Shadow>`__ 146- 2011-09 `Graphics API Design <http://robert.ocallahan.org/2011/09/graphics-api-design.html>`__ 147- 2012-04 `Moz2D Canvas on OSX <http://muizelaar.blogspot.ca/2012/04/azure-canvas-on-os-x.html>`__ 148- 2012-05 `Mask Layers <http://featherweightmusings.blogspot.co.uk/2012/05/mask-layers_26.html>`__ 149- 2013-07 `Graphics related <http://www.basschouten.com/blog1.php>`__ 150