1# Taskgraph Setup 2 3The taskgraph is built from a YAML file. This file has two top-level 4properties: `components` and `tasks`. The full list of tasks is 5defined by the `tasks` object; each task is an object with a single 6property representing the task with the corresponding value an object 7representing the task properties. Each task requires the following 8top-level properties: 9 10* `provisionerId`: String. Name of Taskcluster provisioner 11* `schedulerId`: String. Name of Taskcluster scheduler 12* `deadline`: String. Time until the task expires 13* `image`: String. Name of docker image to use for task 14* `maxRunTime`: Number. Maximum time in seconds for which the task can 15 run. 16* `artifacts`: Object. List of artifacts and directories to upload; see 17 Taskcluster documentation. 18* `command`: String. Command to run. This is automatically wrapped in a 19 run_tc command 20* `options`: Optional Object. Options to pass into run_tc 21 - xvfb: Boolean. Enable Xvfb for run 22 - oom-killer: Boolean. Enable xvfb for run 23 - hosts: Boolean. Update hosts file with wpt hosts before run 24 - install-certificates: Boolean. Install wpt certs into OS 25 certificate store for run 26 - browser: List. List of browser names for run 27 - channel: String. Browser channel for run 28* `trigger`: Object. Conditions on which to consider task. One or more 29 of following properties: 30 - branch: List. List of branch names on which to trigger. 31 - pull-request: No value. Trigger for pull request actions 32* `schedule-if`: Optional Object. Conditions on which task should be 33 scheduled given it meets the trigger conditions. 34 - `run-job`: List. Job names for which this task should be considered, 35 matching the output from `./wpt test-jobs` 36* `env`: Optional Object. Environment variables to set when running task. 37* `depends-on`: Optional list. List of task names that must be complete 38 before the current task is scheduled. 39* `description`: String. Task description. 40* `name`: Optional String. Name to use for the task overriding the 41 property name. This is useful in combination with substitutions 42 described below. 43* `download-artifacts`: Optional Object. An artifact to download from 44 a task that this task depends on. This has the following properties: 45 - `task` - Name of the task producing the artifact 46 - `glob` - A glob pattern for the filename of the artifact 47 - `dest` - A directory reltive to the home directory in which to place 48 the artifact 49 - `extract` - Optional. A boolean indicating whether an archive artifact 50 should be extracted in-place. 51 52## Task Expansions 53 54Using the above syntax it's possble to describe each task 55directly. But typically in a taskgraph there are many common 56properties between tasks so it's tedious and error prone to repeat 57information that's common to multiple tasks. Therefore the taskgraph 58format provides several mechanisms to reuse partial task definitions 59across multiple tasks. 60 61### Components 62 63The other top-level property in the taskgraph format is 64`components`. The value of this property is an object containing named 65partial task definitions. Each task definition may contain a property called 66`use` which is a list of components to use as the basis for the task 67definition. The components list is evaluated in order. If a property 68is not previously defined in the output it is added to the output. If 69it was previously defined, the value is updated according to the type: 70 * Strings and numbers are replaced with a new value 71 * Lists are extended with the additional values 72 * Objects are updated recursively following the above rules 73This means that types must always match between components and the 74final value. 75 76For example 77``` 78components: 79 example-1: 80 list_prop: 81 - first 82 - second 83 object_prop: 84 key1: value1 85 key2: base_value 86 example-2: 87 list_prop: 88 - third 89 - fourth 90 object_prop: 91 key3: 92 - value3-1 93 94tasks: 95 - example-task: 96 use: 97 - example-1 98 - example-2 99 object_prop: 100 key2: value2 101 key3: 102 - value3-2 103``` 104 105will evaluate to the following task: 106 107``` 108example-task: 109 list_prop: 110 - first 111 - second 112 - third 113 - fourth 114 object_prop: 115 key1: value1 116 key2: value2 117 key3: 118 - value3-1 119 - value3-2 120``` 121 122Note that components cannot currently define `use` properties of their own. 123 124## Substitutions 125 126Components and tasks can define a property `vars` that holds variables 127which are later substituted into the task definition using the syntax 128`${vars.property-name}`. For example: 129 130``` 131components: 132 generic-component: 133 prop: ${vars.value} 134 135tasks: 136 - first: 137 use: 138 - generic-component 139 vars: 140 value: value1 141 - second: 142 use: 143 - generic-component 144 vars: 145 value: value2 146``` 147 148Results in the following tasks: 149 150``` 151first: 152 prop: value1 153second: 154 prop: value2 155``` 156 157## Maps 158 159Instead of defining a task directly, an item in the tasks property may 160be an object with a single property `$map`. This object itself has two 161child properties; `for` and `do`. The value of `for` is a list of 162objects, and the value of `do` is either an object or a list of 163objects. For each object in the `for` property, a set of tasks is 164created by taking a copy of that object for each task in the `do` 165property, updating the object with the properties from the 166corresponding `do` object, using the same rules as for components 167above, and then processing as for a normal task. `$map` rules can also 168be nested. 169 170Note: Although `$map` shares a name with the `$map` used in json-e 171(used. in `.taskcluster.yml`), the semantics are different. 172 173For example 174 175``` 176components: {} 177tasks: 178 $map: 179 for: 180 - vars: 181 example: value1 182 - vars: 183 example: value2 184 do: 185 example-${vars.example} 186 prop: ${vars.example} 187``` 188 189Results in the tasks 190 191``` 192example-value1: 193 prop: value1 194example-value2: 195 prop: value2 196``` 197 198Note that in combination with `$map`, variable substitutions are 199applied *twice*; once after the `$map` is evaluated and once after the 200`use` statements are evaluated. 201 202## Chunks 203 204A common requirements for tasks is that they are "chunked" into N 205partial tasks. This is handled specially in the syntax. A top level 206property `chunks` can be used to define the number of individual 207chunks to create for a specific task. Each chunked task is created 208with a `chunks` property set to an object containing an `id` property 209containing the one-based index of the chunk an a `total` property 210containing the total number of chunks. These can be substituted into 211the task definition using the same syntax as for `vars` above 212e.g. `${chunks.id}`. Note that because task names must be unique, it's 213common to specify a `name` property on the task that will override the 214property name e.g. 215 216``` 217components: {} 218tasks: 219 - chunked-task: 220 chunks:2 221 command: "task-run --chunk=${chunks.id} --totalChunks=${chunks.total}" 222 name: task-chunk-${chunks.id} 223``` 224 225creates tasks: 226 227``` 228task-chunk-1: 229 command: "task-run --chunk=1 --totalChunks=2" 230task-chunk-2: 231 command: "task-run --chunk=2 --totalChunks=2" 232``` 233 234# Overall processing model 235 236The overall processing model for tasks is as follows: 237 * Evaluate maps 238 * Perform subsitutions 239 * Evaluate use statements 240 * Expand chunks 241 * Perform subsitutions 242 243At each point after maps are evaluated tasks must have a unique name. 244