1# Taskgraph Setup
2
3The taskgraph is built from a YAML file. This file has two top-level
4properties: `components` and `tasks`. The full list of tasks is
5defined by the `tasks` object; each task is an object with a single
6property representing the task with the corresponding value an object
7representing the task properties. Each task requires the following
8top-level properties:
9
10* `provisionerId`: String. Name of Taskcluster provisioner
11* `schedulerId`: String. Name of Taskcluster scheduler
12* `deadline`: String. Time until the task expires
13* `image`: String. Name of docker image to use for task
14* `maxRunTime`: Number. Maximum time in seconds for which the task can
15  run.
16* `artifacts`: Object. List of artifacts and directories to upload; see
17  Taskcluster documentation.
18* `command`: String. Command to run. This is automatically wrapped in a
19  run_tc command
20* `options`: Optional Object. Options to pass into run_tc
21  - xvfb: Boolean. Enable Xvfb for run
22  - oom-killer: Boolean. Enable xvfb for run
23  - hosts: Boolean. Update hosts file with wpt hosts before run
24  - install-certificates: Boolean. Install wpt certs into OS
25    certificate store for run
26  - browser: List. List of browser names for run
27  - channel: String. Browser channel for run
28* `trigger`: Object. Conditions on which to consider task. One or more
29  of following properties:
30  - branch: List. List of branch names on which to trigger.
31  - pull-request: No value. Trigger for pull request actions
32* `schedule-if`: Optional Object. Conditions on which task should be
33  scheduled given it meets the trigger conditions.
34  - `run-job`: List. Job names for which this task should be considered,
35    matching the output from `./wpt test-jobs`
36* `env`: Optional Object. Environment variables to set when running task.
37* `depends-on`: Optional list. List of task names that must be complete
38  before the current task is scheduled.
39* `description`: String. Task description.
40* `name`: Optional String. Name to use for the task overriding the
41  property name. This is useful in combination with substitutions
42  described below.
43* `download-artifacts`: Optional Object. An artifact to download from
44  a task that this task depends on. This has the following properties:
45  - `task` - Name of the task producing the artifact
46  - `glob` - A glob pattern for the filename of the artifact
47  - `dest` - A directory reltive to the home directory in which to place
48             the artifact
49  - `extract` - Optional. A boolean indicating whether an archive artifact
50                should be extracted in-place.
51
52## Task Expansions
53
54Using the above syntax it's possble to describe each task
55directly. But typically in a taskgraph there are many common
56properties between tasks so it's tedious and error prone to repeat
57information that's common to multiple tasks. Therefore the taskgraph
58format provides several mechanisms to reuse partial task definitions
59across multiple tasks.
60
61### Components
62
63The other top-level property in the taskgraph format is
64`components`. The value of this property is an object containing named
65partial task definitions. Each task definition may contain a property called
66`use` which is a list of components to use as the basis for the task
67definition. The components list is evaluated in order. If a property
68is not previously defined in the output it is added to the output. If
69it was previously defined, the value is updated according to the type:
70 * Strings and numbers are replaced with a new value
71 * Lists are extended with the additional values
72 * Objects are updated recursively following the above rules
73This means that types must always match between components and the
74final value.
75
76For example
77```
78components:
79  example-1:
80    list_prop:
81      - first
82      - second
83    object_prop:
84      key1: value1
85      key2: base_value
86  example-2:
87    list_prop:
88      - third
89      - fourth
90    object_prop:
91      key3:
92        - value3-1
93
94tasks:
95  - example-task:
96      use:
97        - example-1
98        - example-2
99      object_prop:
100        key2: value2
101        key3:
102          - value3-2
103```
104
105will evaluate to the following task:
106
107```
108example-task:
109  list_prop:
110    - first
111    - second
112    - third
113    - fourth
114  object_prop:
115    key1: value1
116    key2: value2
117    key3:
118      - value3-1
119      - value3-2
120```
121
122Note that components cannot currently define `use` properties of their own.
123
124## Substitutions
125
126Components and tasks can define a property `vars` that holds variables
127which are later substituted into the task definition using the syntax
128`${vars.property-name}`. For example:
129
130```
131components:
132  generic-component:
133    prop: ${vars.value}
134
135tasks:
136  - first:
137      use:
138        - generic-component
139      vars:
140        value: value1
141  - second:
142      use:
143        - generic-component
144      vars:
145        value: value2
146```
147
148Results in the following tasks:
149
150```
151first:
152  prop: value1
153second:
154  prop: value2
155```
156
157## Maps
158
159Instead of defining a task directly, an item in the tasks property may
160be an object with a single property `$map`. This object itself has two
161child properties; `for` and `do`. The value of `for` is a list of
162objects, and the value of `do` is either an object or a list of
163objects. For each object in the `for` property, a set of tasks is
164created by taking a copy of that object for each task in the `do`
165property, updating the object with the properties from the
166corresponding `do` object, using the same rules as for components
167above, and then processing as for a normal task. `$map` rules can also
168be nested.
169
170Note: Although `$map` shares a name with the `$map` used in json-e
171(used. in `.taskcluster.yml`), the semantics are different.
172
173For example
174
175```
176components: {}
177tasks:
178  $map:
179    for:
180      - vars:
181          example: value1
182      - vars:
183          example: value2
184    do:
185      example-${vars.example}
186        prop: ${vars.example}
187```
188
189Results in the tasks
190
191```
192example-value1:
193  prop: value1
194example-value2:
195  prop: value2
196```
197
198Note that in combination with `$map`, variable substitutions are
199applied *twice*; once after the `$map` is evaluated and once after the
200`use` statements are evaluated.
201
202## Chunks
203
204A common requirements for tasks is that they are "chunked" into N
205partial tasks. This is handled specially in the syntax. A top level
206property `chunks` can be used to define the number of individual
207chunks to create for a specific task. Each chunked task is created
208with a `chunks` property set to an object containing an `id` property
209containing the one-based index of the chunk an a `total` property
210containing the total number of chunks. These can be substituted into
211the task definition using the same syntax as for `vars` above
212e.g. `${chunks.id}`. Note that because task names must be unique, it's
213common to specify a `name` property on the task that will override the
214property name e.g.
215
216```
217components: {}
218tasks:
219  - chunked-task:
220      chunks:2
221      command: "task-run --chunk=${chunks.id} --totalChunks=${chunks.total}"
222      name: task-chunk-${chunks.id}
223```
224
225creates tasks:
226
227```
228task-chunk-1:
229  command: "task-run --chunk=1 --totalChunks=2"
230task-chunk-2:
231  command: "task-run --chunk=2 --totalChunks=2"
232```
233
234# Overall processing model
235
236The overall processing model for tasks is as follows:
237 * Evaluate maps
238 * Perform subsitutions
239 * Evaluate use statements
240 * Expand chunks
241 * Perform subsitutions
242
243At each point after maps are evaluated tasks must have a unique name.
244