README.md
1```
2 ___ _ __ (_)/ _|/ _|
3 / __| '_ \| | |_| |_
4 \__ \ |_) | | _| _|
5 |___/ .__/|_|_| |_|
6 |_|
7
8```
9
10---
11
12**NOTE**: *Active development on spiff is currently paused, including Pull Requests. Very severe issues will be addressed, and we will still be actively responding to requests for help via Issues.*
13
14---
15
16spiff is a command line tool and declarative YAML templating system, specially designed for generating BOSH deployment manifests.
17
18Contents:
19- [Installation](#installation)
20- [Usage](#usage)
21- [dynaml Templating Tanguage](#dynaml-templating-language)
22 - [(( foo ))](#-foo-)
23 - [(( foo.bar.[1].baz ))](#-foobar1baz-)
24 - [(( "foo" ))](#-foo--1)
25 - [(( "foo" bar ))](#-foo-bar-)
26 - [(( auto ))](#-auto-)
27 - [(( merge ))](#-merge-)
28 - [<<: (( merge ))](#--merge-)
29 - [merging maps](#merging-maps)
30 - [merging lists](#merging-lists)
31 - [(( a || b ))](#-a--b-)
32 - [(( !foo ))](#-foo--2)
33 - [(( static_ips(0, 1, 3) ))](#-static_ips0-1-3-)
34
35
36# Installation
37
38Official release executable binaries can be downloaded via [Github releases](https://github.com/cloudfoundry-incubator/spiff/releases) for Darwin and Linux machines (and virtual machines).
39
40Some of spiff's dependencies have changed since the last official release, and spiff will not be updated to keep up with these dependencies. Working dependencies are vendored in the `Godeps` directory (more information on the `godep` tool is available [here](https://github.com/tools/godep)). As such, trying to `go get` spiff will likely fail; the only supported way to use spiff is to use an official binary release.
41
42# Usage
43
44### `spiff merge template.yml [template2.ymll ...]`
45
46Merge a bunch of template files into one manifest, printing it out.
47
48See 'dynaml templating language' for details of the template file, or examples/ subdir for more complicated examples.
49
50Example:
51
52```
53spiff merge cf-release/templates/cf-deployment.yml my-cloud-stub.yml
54```
55
56### `spiff diff manifest.yml other-manifest.yml`
57
58Show structural differences between two deployment manifests.
59
60Unlike basic diffing tools and even `bosh diff`, this command has semantic
61knowledge of a deployment manifest, and is not just text-based. For example,
62if two manifests are the same except they have some jobs listed in different
63orders, `spiff diff` will detect this, since job order matters in a manifest.
64On the other hand, if two manifests differ only in the order of their
65resource pools, for instance, then it will yield and empty diff since
66resource pool order doesn't actually matter for a deployment.
67
68Also unlike `bosh diff`, this command doesn't modify either file.
69
70It's tailored for checking differences between one deployment and the next.
71
72Typical flow:
73
74```sh
75$ spiff merge template.yml [templates...] > upgrade.yml
76$ bosh download manifest [deployment] current.yml
77$ spiff diff upgrade.yml current.yml
78$ bosh deployment upgrade.yml
79$ bosh deploy
80```
81
82
83# dynaml Templating Language
84
85Spiff uses a declarative, logic-free templating language called 'dynaml'
86(dynamic yaml).
87
88Every dynaml node is guaranteed to resolve to a YAML node. It is *not*
89string interpolation. This keeps developers from having to think about how
90a value will render in the resulting template.
91
92A dynaml node appears in the .yml file as an expression surrounded by two
93parentheses. They can be used as the value of a map or an entry in a list.
94
95The following is a complete list of dynaml expressions:
96
97
98## `(( foo ))`
99
100Look for the nearest 'foo' key (i.e. lexical scoping) in the current
101template and bring it in.
102
103e.g.:
104
105```yaml
106fizz:
107 buzz:
108 foo: 1
109 bar: (( foo ))
110 bar: (( foo ))
111foo: 3
112bar: (( foo ))
113```
114
115This example will resolve to:
116
117```yaml
118fizz:
119 buzz:
120 foo: 1
121 bar: 1
122 bar: 3
123foo: 3
124bar: 3
125```
126
127The following will not resolve because the key name is the same as the value to be merged in:
128```yaml
129foo: 1
130
131hi:
132 foo: (( foo ))
133```
134
135## `(( foo.bar.[1].baz ))`
136
137Look for the nearest 'foo' key, and from there follow through to .bar.baz.
138
139A path is a sequence of steps separated by dots. A step is either a word for
140maps, or digits surrounded by brackets for list indexing.
141
142If the path cannot be resolved, this evaluates to nil. A reference node at the
143top level cannot evaluate to nil; the template will be considered not fully
144resolved. If a reference is expected to sometimes not be provided, it should be
145used in combination with '||' (see below) to guarantee resolution.
146
147Note that references are always within the template, and order does not matter.
148You can refer to another dynamic node and presume it's resolved, and the
149reference node will just eventually resolve once the dependent node resolves.
150
151e.g.:
152
153```yaml
154properties:
155 foo: (( something.from.the.stub ))
156 something: (( merge ))
157```
158
159This will resolve as long as 'something' is resolveable, and as long as it
160brings in something like this:
161
162```yaml
163from:
164 the:
165 stub: foo
166```
167
168## `(( "foo" ))`
169
170String literal. The only escape character handled currently is '"'.
171
172## `(( "foo" bar ))`
173
174Concatenation (where bar is another dynaml expr).
175
176e.g.
177
178```yaml
179domain: example.com
180uri: (( "https://" domain ))
181```
182
183In this example `uri` will resolve to the value `"https://example.com"`.
184
185## `(( auto ))`
186
187Context-sensitive automatic value calculation.
188
189In a resource pool's 'size' attribute, this means calculate based on the total
190instances of all jobs that declare themselves to be in the current resource
191pool.
192
193e.g.:
194
195```yaml
196resource_pools:
197 - name: mypool
198 size: (( auto ))
199
200jobs:
201 - name: myjob
202 resource_pool: mypool
203 instances: 2
204 - name: myotherjob
205 resource_pool: mypool
206 instances: 3
207 - name: yetanotherjob
208 resource_pool: otherpool
209 instances: 3
210```
211
212In this case the resource pool size will resolve to '5'.
213
214## `(( merge ))`
215
216Bring the current path in from the stub files that are being merged in.
217
218e.g.:
219
220```yaml
221foo:
222 bar:
223 baz: (( merge ))
224```
225
226Will try to bring in `foo.bar.baz` from the first stub, or the second, etc.,
227returning the value from the first stub that provides it.
228
229If the corresponding value is not defined, it will return nil. This then has the
230same semantics as reference expressions; a nil merge is an unresolved template.
231See `||`.
232
233### `<<: (( merge ))`
234
235#### Merging maps
236
237**values.yml**
238```yaml
239foo:
240 a: 1
241 b: 2
242```
243
244**template.yml**
245```yaml
246foo:
247 <<: (( merge ))
248 b: 3
249 c: 4
250```
251
252`spiff merge template.yml values.yml` yields:
253
254```yaml
255foo:
256 a: 1
257 b: 2
258 c: 4
259```
260
261#### Merging lists
262
263**values.yml**
264```yaml
265foo:
266 - 1
267 - 2
268```
269
270**template.yml**
271```yaml
272foo:
273 - 3
274 - <<: (( merge ))
275 - 4
276```
277
278`spiff merge template.yml values.yml` yields:
279
280```yaml
281foo:
282 - 3
283 - 1
284 - 2
285 - 4
286```
287
288## `(( a || b ))`
289
290Uses a, or b if a cannot be resolved.
291
292e.g.:
293
294```yaml
295foo:
296 bar:
297 - name: some
298 - name: complicated
299 - name: structure
300
301mything:
302 complicated_structure: (( merge || foo.bar ))
303```
304
305This will try to merge in `mything.complicated_structure`, or, if it cannot be
306merged in, use the default specified in `foo.bar`.
307
308## `(( !foo ))`
309
310Leaves the string `(( !foo ))` in the resulting merge.
311
312e.g.: calling `spiff merge` on a file with the contents below:
313
314```yaml
315bar: (( !foo ))
316foo: 33
317```
318
319will simply result in:
320
321```yaml
322bar: (( !foo ))
323foo: 33
324```
325
326## `(( static_ips(0, 1, 3) ))`
327
328Generate a list of static IPs for a job.
329
330e.g.:
331
332```yaml
333jobs:
334 - name: myjob
335 instances: 2
336 networks:
337 - name: mynetwork
338 static_ips: (( static_ips(0, 3, 4) ))
339```
340
341This will create 3 IPs from `mynetwork`s subnet, and return two entries, as
342there are only two instances. The two entries will be the 0th and 3rd offsets
343from the static IP ranges defined by the network.
344
345For example, given the file bye.yml:
346
347```yaml
348networks: (( merge ))
349
350jobs:
351 - name: myjob
352 instances: 3
353 networks:
354 - name: cf1
355 static_ips: (( static_ips(0,3,60) ))
356```
357
358and file hi.yml:
359
360```yaml
361networks:
362- name: cf1
363 subnets:
364 - cloud_properties:
365 security_groups:
366 - cf-0-vpc-c461c7a1
367 subnet: subnet-e845bab1
368 dns:
369 - 10.60.3.2
370 gateway: 10.60.3.1
371 name: default_unused
372 range: 10.60.3.0/24
373 reserved:
374 - 10.60.3.2 - 10.60.3.9
375 static:
376 - 10.60.3.10 - 10.60.3.70
377 type: manual
378```
379
380```
381spiff merge bye.yml hi.yml
382```
383
384returns
385
386
387```yaml
388jobs:
389- instances: 3
390 name: myjob
391 networks:
392 - name: cf1
393 static_ips:
394 - 10.60.3.10
395 - 10.60.3.13
396 - 10.60.3.70
397networks:
398- name: cf1
399 subnets:
400 - cloud_properties:
401 security_groups:
402 - cf-0-vpc-c461c7a1
403 subnet: subnet-e845bab1
404 dns:
405 - 10.60.3.2
406 gateway: 10.60.3.1
407 name: default_unused
408 range: 10.60.3.0/24
409 reserved:
410 - 10.60.3.2 - 10.60.3.9
411 static:
412 - 10.60.3.10 - 10.60.3.70
413 type: manual
414```
415.
416
417If bye.yml was instead
418
419```yaml
420networks: (( merge ))
421
422jobs:
423 - name: myjob
424 instances: 2
425 networks:
426 - name: cf1
427 static_ips: (( static_ips(0,3,60) ))
428```
429
430```
431spiff merge bye.yml hi.yml
432```
433
434instead returns
435
436```yaml
437jobs:
438- instances: 2
439 name: myjob
440 networks:
441 - name: cf1
442 static_ips:
443 - 10.60.3.10
444 - 10.60.3.13
445networks:
446- name: cf1
447 subnets:
448 - cloud_properties:
449 security_groups:
450 - cf-0-vpc-c461c7a1
451 subnet: subnet-e845bab1
452 dns:
453 - 10.60.3.2
454 gateway: 10.60.3.1
455 name: default_unused
456 range: 10.60.3.0/24
457 reserved:
458 - 10.60.3.2 - 10.60.3.9
459 static:
460 - 10.60.3.10 - 10.60.3.70
461 type: manual
462```
463