1# Kokoro Configuration and Build scripts. 2 3This directory contains the build scripts and configuration files 4for Kokoro, Google's internal CI system for open source projects. 5 6We use Kokoro because: 7 8- It has more bandwidth than other CI systems supporting Open Source prjects, 9 such as Travis. 10- We can run Linux, Windows, and macOS builds from a single CI system. 11- We can store secrets, such as key files or access tokens, and run integration 12 builds against production. 13 14The documentation for Kokoro is available to Googlers only (sorry) at go/kokoro, 15there are extensive codelabs and detailed descriptions of the system there. 16 17In brief, the Kokoro configuration is split in two: 18 191. A series of configuration files inside Google define what builds exist, and 20 what resources these builds have access to. 21 * These files are in a hierarchy that mirrors the ci/kokoro directory in this 22 repo. 23 * The `common.cfg` files are parsed first. 24 * The `common.cfg` files are applied according to the directory hierarchy. 25 * Finally any settings in `foobar.cfg` are applied. 261. A series of configuration files in the `ci/kokoro` directory further define 27 the build configuration: 28 * They define the build script for each build, though they are often common. 29 * They define which of the resources *allowed* by the internal configuration 30 are actually *used* by that build. 31 32Somewhat unique to Kokoro one must define a separate *INTEGRATION* vs. 33*PRESUBMIT* thus the duplication of configuration files for essentially the 34same settings. 35 36## Setup the Google Container Registry cache for Kokoro builds. 37 38### Background 39 40Most of our builds run in a Docker container inside the Kokoro build machines. 41We use Docker to: 42 43- Test with different toolchains from the ones provided by Kokoro. 44- Test with the native toolchains in multiple distributions, to ensure the 45 default experience works. 46- Test our "README" and "INSTALL" instructions by running the builds with the 47 minimal set of tools required to complete a build. 48- To create more hermetic builds with respect to the Kokoro configuration. 49 50One of the first steps in the build is to create the Docker image used by that 51build. This ensures that changes made to the definitions of the images are 52tested with any code changes that need them. But this means that the build can 53be slowed down by the image creation step, and they often fail because the 54image creation fails, even nothing has changed with the image itself. Moreover, 55to create the image we need to download packages from multiple sources, this 56can be slow or fail when the servers hosting packages for certain distros 57experience an outage. Such outages can last hours, the usual approach of 58"try N times with a backoff" do not work during these outages. 59 60To minimize the impact of these outages, and to also speed up the builds a 61little bit, we cache these images in Google Container Registry (GCR). Builds 62triggered by GitHub pull requests download the cached image, then use that cache 63to rebuild the image (which usually results in "nothing to rebuild") and then 64run the build using the newly created image. Builds triggered by GitHub commits 65to a branch (typically `master` or a release branch) only use the cache if 66rebuilding the image from scratch fails. Such builds also push new versions of 67the cached image to GCR. 68 69### Instructions 70 71The following instructions configure the GCR cache for a project, and create a 72new service account with permissions to read and write to this cache. Pick a 73project that will host the container image cache: 74 75```console 76$ PROJECT_ID=... # Most likely an existing project for builds. 77``` 78 79We need to grant the service account full storage admin access to the bucket 80that will contain the containers. To this end we need to create the 81[bucket first](https://cloud.google.com/container-registry/docs/access-control): 82 83```console 84$ gcloud auth configure-docker 85$ docker pull hello-world:latest 86$ docker tag hello-world:latest gcr.io/${PROJECT_ID}/hello-world:latest 87$ docker push gcr.io/${PROJECT_ID}/hello-world:latest 88``` 89 90Create a service account to access the Google Container Registry: 91 92```console 93$ SA_NAME=kokoro-gcr-updater-sa 94``` 95 96Then create the service account: 97 98```console 99$ gcloud iam service-accounts create --project ${PROJECT_ID} ${SA_NAME} \ 100 --display-name="Allows Kokoro builds to access GCR images." 101$ FULL_SA_NAME="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" 102``` 103 104Grant the service account `roles/storage.admin` permissions on the container 105registry bucket. 106 107```console 108$ gsutil iam ch serviceAccount:${FULL_SA_NAME}:roles/storage.admin \ 109 gs://artifacts.${PROJECT_ID}.appspot.com/ 110``` 111 112Create new keys for this service account and download then to a temporary place: 113 114```console 115$ gcloud iam service-accounts keys create /dev/shm/gcr-service-account.json \ 116 --iam-account "${FULL_SA_NAME}" 117``` 118 119Create a configuration file for the Kokoro script: 120 121```bash 122$ echo "PROJECT_ID=${PROJECT_ID}" >/dev/shm/gcr-configuration.sh 123``` 124 125Copy the service account keys and configuration file to the location where we 126keep Kokoro secrets (Google's internal KMS), then delete these files from your 127workstation. 128