1# Kokoro Configuration and Build scripts.
2
3This directory contains the build scripts and configuration files
4for Kokoro, Google's internal CI system for open source projects.
5
6We use Kokoro because:
7
8- It has more bandwidth than other CI systems supporting Open Source prjects,
9  such as Travis.
10- We can run Linux, Windows, and macOS builds from a single CI system.
11- We can store secrets, such as key files or access tokens, and run integration
12  builds against production.
13
14The documentation for Kokoro is available to Googlers only (sorry) at go/kokoro,
15there are extensive codelabs and detailed descriptions of the system there.
16
17In brief, the Kokoro configuration is split in two:
18
191. A series of configuration files inside Google define what builds exist, and
20   what resources these builds have access to.
21   * These files are in a hierarchy that mirrors the ci/kokoro directory in this
22     repo.
23   * The `common.cfg` files are parsed first.
24   * The `common.cfg` files are applied according to the directory hierarchy.
25   * Finally any settings in `foobar.cfg` are applied.
261. A series of configuration files in the `ci/kokoro` directory further define
27   the build configuration:
28   * They define the build script for each build, though they are often common.
29   * They define which of the resources *allowed* by the internal configuration
30     are actually *used* by that build.
31
32Somewhat unique to Kokoro one must define a separate *INTEGRATION* vs.
33*PRESUBMIT* thus the duplication of configuration files for essentially the
34same settings.
35
36## Setup the Google Container Registry cache for Kokoro builds.
37
38### Background
39
40Most of our builds run in a Docker container inside the Kokoro build machines.
41We use Docker to:
42
43- Test with different toolchains from the ones provided by Kokoro.
44- Test with the native toolchains in multiple distributions, to ensure the
45  default experience works.
46- Test our "README" and "INSTALL" instructions by running the builds with the
47  minimal set of tools required to complete a build.
48- To create more hermetic builds with respect to the Kokoro configuration.
49
50One of the first steps in the build is to create the Docker image used by that
51build. This ensures that changes made to the definitions of the images are
52tested with any code changes that need them. But this means that the build can
53be slowed down by the image creation step, and they often fail because the
54image creation fails, even nothing has changed with the image itself. Moreover,
55to create the image we need to download packages from multiple sources, this
56can be slow or fail when the servers hosting packages for certain distros
57experience an outage. Such outages can last hours, the usual approach of
58"try N times with a backoff" do not work during these outages.
59
60To minimize the impact of these outages, and to also speed up the builds a
61little bit, we cache these images in Google Container Registry (GCR). Builds
62triggered by GitHub pull requests download the cached image, then use that cache
63to rebuild the image (which usually results in "nothing to rebuild") and then
64run the build using the newly created image. Builds triggered by GitHub commits
65to a branch (typically `master` or a release branch) only use the cache if
66rebuilding the image from scratch fails. Such builds also push new versions of
67the cached image to GCR.
68
69### Instructions
70
71The following instructions configure the GCR cache for a project, and create a
72new service account with permissions to read and write to this cache. Pick a
73project that will host the container image cache:
74
75```console
76$ PROJECT_ID=... # Most likely an existing project for builds.
77```
78
79We need to grant the service account full storage admin access to the bucket
80that will contain the containers. To this end we need to create the
81[bucket first](https://cloud.google.com/container-registry/docs/access-control):
82
83```console
84$ gcloud auth configure-docker
85$ docker pull hello-world:latest
86$ docker tag hello-world:latest gcr.io/${PROJECT_ID}/hello-world:latest
87$ docker push gcr.io/${PROJECT_ID}/hello-world:latest
88```
89
90Create a service account to access the Google Container Registry:
91
92```console
93$ SA_NAME=kokoro-gcr-updater-sa
94```
95
96Then create the service account:
97
98```console
99$ gcloud iam service-accounts create --project ${PROJECT_ID} ${SA_NAME} \
100    --display-name="Allows Kokoro builds to access GCR images."
101$ FULL_SA_NAME="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
102```
103
104Grant the service account `roles/storage.admin` permissions on the container
105registry bucket.
106
107```console
108$ gsutil iam ch serviceAccount:${FULL_SA_NAME}:roles/storage.admin \
109    gs://artifacts.${PROJECT_ID}.appspot.com/
110```
111
112Create new keys for this service account and download then to a temporary place:
113
114```console
115$ gcloud iam service-accounts keys create /dev/shm/gcr-service-account.json \
116      --iam-account "${FULL_SA_NAME}"
117```
118
119Create a configuration file for the Kokoro script:
120
121```bash
122$ echo "PROJECT_ID=${PROJECT_ID}" >/dev/shm/gcr-configuration.sh
123```
124
125Copy the service account keys and configuration file to the location where we
126keep Kokoro secrets (Google's internal KMS), then delete these files from your
127workstation.
128