1.. taskcluster_dockerimages: 2 3============= 4Docker Images 5============= 6 7TaskCluster Docker images are defined in the source directory under 8``taskcluster/docker``. Each directory therein contains the name of an 9image used as part of the task graph. 10 11Organization 12------------ 13 14Each folder describes a single docker image. We have two types of images that can be defined: 15 161. Task Images (build-on-push) 172. Docker Images (prebuilt) 18 19These images depend on one another, as described in the `FROM 20<https://docs.docker.com/v1.8/reference/builder/#from>`_ line at the top of the 21Dockerfile in each folder. 22 23Images could either be an image intended for pushing to a docker registry, or 24one that is meant either for local testing or being built as an artifact when 25pushed to vcs. 26 27Task Images (build-on-push) 28::::::::::::::::::::::::::: 29 30Images can be uploaded as a task artifact, :ref:`indexed <task-image-index-namespace>` under 31a given namespace, and used in other tasks by referencing the task ID. 32 33Important to note, these images do not require building and pushing to a docker registry, and are 34built per push (if necessary) and uploaded as task artifacts. 35 36The decision task that is run per push will :ref:`determine <context-directory-hashing>` 37if the image needs to be built based on the hash of the context directory and if the image 38exists under the namespace for a given branch. 39 40As an additional convenience, and a precaution to loading images per branch, if an image 41has been indexed with a given context hash for mozilla-central, any tasks requiring that image 42will use that indexed task. This is to ensure there are not multiple images built/used 43that were built from the same context. In summary, if the image has been built for mozilla-central, 44pushes to any branch will use that already built image. 45 46To use within an in-tree task definition, the format is: 47 48.. code-block:: yaml 49 50 image: 51 type: 'task-image' 52 path: 'public/image.tar.zst' 53 taskId: '<task_id_for_image_builder>' 54 55.. _context-directory-hashing: 56 57Context Directory Hashing 58......................... 59 60Decision tasks will calculate the sha256 hash of the contents of the image 61directory and will determine if the image already exists for a given branch and hash 62or if a new image must be built and indexed. 63 64Note: this is the contents of *only* the context directory, not the 65image contents. 66 67The decision task will: 68 691. Recursively collect the paths of all files within the context directory 702. Sort the filenames alphabetically to ensure the hash is consistently calculated 713. Generate a sha256 hash of the contents of each file 724. All file hashes will then be combined with their path and used to update the 73 hash of the context directory 74 75This ensures that the hash is consistently calculated and path changes will result 76in different hashes being generated. 77 78.. _task-image-index-namespace: 79 80Task Image Index Namespace 81.......................... 82 83Images that are built on push and uploaded as an artifact of a task will be indexed under the 84following namespaces. 85 86* gecko.cache.level-{level}.docker.v2.{name}.hash.{digest} 87* gecko.cache.level-{level}.docker.v2.{name}.latest 88* gecko.cache.level-{level}.docker.v2.{name}.pushdate.{year}.{month}-{day}-{pushtime} 89 90Not only can images be browsed by the pushdate and context hash, but the 'latest' namespace 91is meant to view the latest built image. This functions similarly to the 'latest' tag 92for docker images that are pushed to a registry. 93 94Docker Registry Images (prebuilt) 95::::::::::::::::::::::::::::::::: 96 97***Warning: Registry images are only used for ``decision`` and 98``image_builder`` images.*** 99 100These are images that are intended to be pushed to a docker registry and used 101by specifying the docker image name in task definitions. They are generally 102referred to by a ``<repo>@<repodigest>`` string: 103 104Example: 105 106.. code-block:: none 107 108 image: taskcluster/decision:0.1.10@sha256:c5451ee6c655b3d97d4baa3b0e29a5115f23e0991d4f7f36d2a8f793076d6854 109 110Such images must always be referred to with both a version and a repo digest. 111For the decision image, the repo digest is stored in the ``HASH`` file in the 112image directory and used to refer to the image as above. The version for both 113images is in ``VERSION``. 114 115The version file serves to help users identify which image is being used, and makes old 116versions easy to discover in the registry. 117 118The file ``taskcluster/docker/REGISTRY`` specifies the image registry to which 119the completed image should be uploaded. 120 121Docker Hashes and Digests 122......................... 123 124There are several hashes involved in this process: 125 126 * Image Hash -- the long version of the image ID; can be seen with 127 ``docker images --no-trunc`` or in the ``Id`` field in ``docker inspect``. 128 129 * Repo Digest -- hash of the image manifest; seen when running ``docker 130 push`` or ``docker pull``. 131 132 * Context Directory Hash -- see above (not a Docker concept at all) 133 134The use of hashes allows older tasks which were designed to run on an older 135version of the image to be executed in Taskcluster while new tasks use the new 136version. Furthermore, this mitigates attacks against the registry as docker 137will verify the image hash when loading the image. 138 139(Re)-Building images 140-------------------- 141 142Generally, images can be pulled from the Docker registry rather than built 143locally, however, for developing new images it's often helpful to hack on them 144locally. 145 146To build an image, invoke ``mach taskcluster-build-image`` with the name of the 147folder (without a trailing slash): 148 149.. code-block:: none 150 151 ./mach taskcluster-build-image <image-name> 152 153This is a wrapper around ``docker build -t $REGISTRY/$FOLDER:$VERSION``. 154 155It's a good idea to bump the ``VERSION`` early in this process, to avoid 156``docker push``-ing over any old tags. 157 158For task images, test your image locally or push to try. This is all that is 159required. 160 161Docker Registry Images 162:::::::::::::::::::::: 163 164Landing docker registry images takes a little more care. 165 166Begin by bumping the ``VERSION``. Once the new version of the image has been 167built and tested locally, push it to the docker registry and make note of the 168resulting repo digest. Put this value in the ``HASH`` file for the 169``decision`` image and in ``taskcluster/gecko_taskgraph/transforms/docker_image.py`` 170for the ``image_builder`` image. 171 172The change is now safe to use in Try pushes. 173 174Note that ``image_builder`` change can be tested directly in try pushes without 175using a registry, as the in-registry ``image_builder`` image is used to build a 176task image which is then used to build other images. It is referenced by hash 177in ``taskcluster/gecko_taskgraph/transforms/docker_image.py``. 178 179Special Dockerfile Syntax 180------------------------- 181 182Dockerfile syntax has been extended to allow *any* file from the 183source checkout to be added to the image build *context*. (Traditionally 184you can only ``ADD`` files from the same directory as the Dockerfile.) 185 186Simply add the following syntax as a comment in a Dockerfile:: 187 188 # %include <path> 189 190e.g. 191 192 # %include mach 193 # %include testing/mozharness 194 195The argument to ``# %include`` is a relative path from the root level of 196the source directory. It can be a file or a directory. If a file, only that 197file will be added. If a directory, every file under that directory will be 198added (even files that are untracked or ignored by version control). 199 200Files added using ``# %include`` syntax are available inside the build 201context under the ``topsrcdir/`` path. 202 203Files are added as they exist on disk. e.g. executable flags should be 204preserved. However, the file owner/group is changed to ``root`` and the 205``mtime`` of the file is normalized. 206 207Here is an example Dockerfile snippet:: 208 209 # %include mach 210 ADD topsrcdir/mach /builds/worker/mach 211