1
2[[filter_elements]]
3= Filter Elements
4
5[partintro]
6--
7* <<fe_aliases,aliases>>
8* <<fe_allocation_type,allocation_type>>
9* <<fe_count,count>>
10* <<fe_date_from,date_from>>
11* <<fe_date_from_format,date_from_format>>
12* <<fe_date_to,date_to>>
13* <<fe_date_to_format,date_to_format>>
14* <<fe_direction,direction>>
15* <<fe_disk_space,disk_space>>
16* <<fe_epoch,epoch>>
17* <<fe_exclude,exclude>>
18* <<fe_field,field>>
19* <<fe_intersect,intersect>>
20* <<fe_key,key>>
21* <<fe_kind,kind>>
22* <<fe_max_num_segments,max_num_segments>>
23* <<fe_pattern,pattern>>
24* <<fe_period_type,period_type>>
25* <<fe_range_from,range_from>>
26* <<fe_range_to,range_to>>
27* <<fe_reverse,reverse>>
28* <<fe_source,source>>
29* <<fe_state,state>>
30* <<fe_stats_result,stats_result>>
31* <<fe_timestring,timestring>>
32* <<fe_threshold_behavior,threshold_behavior>>
33* <<fe_unit,unit>>
34* <<fe_unit_count,unit_count>>
35* <<fe_unit_count_pattern,unit_count_pattern>>
36* <<fe_use_age,use_age>>
37* <<fe_value,value>>
38* <<fe_week_starts_on,week_starts_on>>
39
40You can use <<envvars,environment variables>> in your configuration files.
41--
42
43
44
45[[fe_aliases]]
46== aliases
47
48include::inc_filter_by_aliases.asciidoc[]
49
50NOTE: This setting is used only when using the <<filtertype_alias,alias>>
51    filter.
52
53The value of this setting must be a single alias name, or a list of alias names.
54This can be done in any of the ways YAML allows for lists or arrays.  Here are
55a few examples.
56
57**Single**
58[source,txt]
59-------
60filters:
61- filtertype: alias
62  aliases: my_alias
63  exclude: False
64-------
65
66**List**
67
68- Flow style:
69+
70[source,txt]
71-------
72filters:
73- filtertype: alias
74  aliases: [ my_alias, another_alias ]
75  exclude: False
76-------
77
78- Block style:
79+
80[source,txt]
81-------
82filters:
83- filtertype: alias
84  aliases:
85    - my_alias
86    - another_alias
87  exclude: False
88-------
89
90There is no default value. This setting must be set by the user or an
91exception will be raised, and execution will halt.
92
93
94[[fe_allocation_type]]
95== allocation_type
96
97NOTE: This setting is used only when using the
98    <<filtertype_allocated,allocated>> filter.
99
100[source,yaml]
101-------------
102- filtertype: allocated
103  key: ...
104  value: ...
105  allocation_type: require
106  exclude: True
107-------------
108
109The value of this setting must be one of `require`, `include`, or `exclude`.
110
111Read more about these settings in the
112{ref}/shard-allocation-filtering.html[Elasticsearch documentation].
113
114The default value for this setting is `require`.
115
116
117
118[[fe_count]]
119== count
120
121NOTE: This setting is only used with the <<filtertype_count,count>> filtertype +
122    and is a required setting.
123
124[source,yaml]
125-------------
126- filtertype: count
127  count: 10
128-------------
129
130The value for this setting is a number of indices or snapshots to match.
131
132Items will remain in the actionable list depending on the value of
133<<fe_exclude,exclude>>, and <<fe_reverse,reverse>>.
134
135There is no default value. This setting must be set by the user or an exception
136will be raised, and execution will halt.
137
138
139[[fe_date_from]]
140== date_from
141
142NOTE: This setting is only used with the <<filtertype_period,period>> filtertype +
143    when <<fe_period_type,period_type>> is `absolute`.
144
145[source,yaml]
146-------------
147 - filtertype: period
148   period_type: absolute
149   source: name
150   timestring: '%Y.%m.%d'
151   unit: months
152   date_from: 2017.01
153   date_from_format: '%Y.%m'
154   date_to: 2017.01
155   date_to_format: '%Y.%m'
156-------------
157
158The value for this setting should be a date which can be easily parsed
159using the strftime string in <<fe_date_from_format,`date_from_format`>>:
160
161include::inc_strftime_table.asciidoc[]
162
163[[fe_date_from_format]]
164== date_from_format
165
166NOTE: This setting is only used with the <<filtertype_period,period>> filtertype +
167    when <<fe_period_type,period_type>> is `absolute`.
168
169[source,yaml]
170-------------
171 - filtertype: period
172   period_type: absolute
173   source: name
174   timestring: '%Y.%m.%d'
175   unit: months
176   date_from: 2017.01
177   date_from_format: '%Y.%m'
178   date_to: 2017.01
179   date_to_format: '%Y.%m'
180-------------
181
182The value for this setting should be an strftime string which corresponds to
183the date in <<fe_date_from,`date_from`>>:
184
185include::inc_strftime_table.asciidoc[]
186
187[[fe_date_to]]
188== date_to
189
190NOTE: This setting is only used with the <<filtertype_period,period>> filtertype +
191    when <<fe_period_type,period_type>> is `absolute`.
192
193[source,yaml]
194-------------
195 - filtertype: period
196   period_type: absolute
197   source: name
198   timestring: '%Y.%m.%d'
199   unit: months
200   date_from: 2017.01
201   date_from_format: '%Y.%m'
202   date_to: 2017.01
203   date_to_format: '%Y.%m'
204-------------
205
206The value for this setting should be a date which can be easily parsed
207using the strftime string in <<fe_date_to_format,`date_to_format`>>:
208
209include::inc_strftime_table.asciidoc[]
210
211[[fe_date_to_format]]
212== date_to_format
213
214NOTE: This setting is only used with the <<filtertype_period,period>> filtertype +
215    when <<fe_period_type,period_type>> is `absolute`.
216
217[source,yaml]
218-------------
219 - filtertype: period
220   period_type: absolute
221   source: name
222   timestring: '%Y.%m.%d'
223   unit: months
224   date_from: 2017.01
225   date_from_format: '%Y.%m'
226   date_to: 2017.01
227   date_to_format: '%Y.%m'
228-------------
229
230The value for this setting should be an strftime string which corresponds to
231the date in <<fe_date_to,`date_to`>>:
232
233include::inc_strftime_table.asciidoc[]
234
235
236[[fe_direction]]
237== direction
238
239NOTE: This setting is only used with the <<filtertype_age,age>> filtertype.
240
241[source,yaml]
242-------------
243 - filtertype: age
244   source: creation_date
245   direction: older
246   unit: days
247   unit_count: 3
248-------------
249
250This setting must be either `older` or `younger`.  This setting is used to
251determine whether indices or snapshots are `older` or `younger` than the
252reference point in time determined by <<fe_unit,unit>>,
253<<fe_unit_count,unit_count>>, and optionally, <<fe_epoch,epoch>>.  If
254`direction` is `older`, then indices (or snapshots) which are _older_ than the
255reference point in time will be matched.  Likewise, if `direction` is
256`younger`, then indices (or snapshots) which are _younger_ than the reference
257point in time will be matched.
258
259There is no default value. This setting must be set by the user or an
260exception will be raised, and execution will halt.
261
262
263
264[[fe_disk_space]]
265== disk_space
266
267NOTE: This setting is only used with the <<filtertype_space,space>> filtertype +
268    and is a required setting.
269
270[source,yaml]
271-------------
272- filtertype: space
273  disk_space: 100
274-------------
275
276The value for this setting is a number of gigabytes.
277
278Indices in excess of this number of gigabytes will be matched.
279
280There is no default value. This setting must be set by the user or an exception
281will be raised, and execution will halt.
282
283
284
285[[fe_epoch]]
286== epoch
287
288NOTE: This setting is available in the <<filtertype_age,age>> filtertype, and
289  any filter which has the <<fe_use_age,`use_age`>> setting. This setting is
290  strictly optional.
291
292TIP: This setting is not common.  It is most frequently used for testing.
293
294<<fe_unit,unit>>, <<fe_unit_count,unit_count>>, and optionally,
295<<fe_epoch,epoch>>, are used by Curator to establish the moment in time point of
296reference with this formula:
297
298[source,sh]
299-----------
300point_of_reference = epoch - ((number of seconds in unit) * unit_count)
301-----------
302
303If <<fe_epoch,epoch>> is unset, the current time is used. It is possible to set
304a point of reference in the future by using a negative value for
305<<fe_unit_count,unit_count>>.
306
307=== Example
308
309[source,yaml]
310-------------
311 - filtertype: age
312   source: creation_date
313   direction: older
314   unit: days
315   unit_count: 3
316   epoch: 1491577200
317-------------
318
319The value for this setting must be an epoch timestamp. In this example, the
320given epoch time of `1491577200` is 2017-04-04T15:00:00Z (UTC).  This will use
3213 days older than that timestamp as the point of reference for age comparisons.
322
323
324
325[[fe_exclude]]
326== exclude
327
328NOTE: This setting is available in _all_ filter types.
329
330If `exclude` is `True`, the filter will remove matches from the actionable list.
331If `exclude` is `False`, then only matches will be kept in the actionable list.
332
333The default value for this setting is different for each filter type.
334
335=== Examples
336
337[source,yaml]
338-------------
339- filtertype: opened
340  exclude: True
341-------------
342
343This filter will result in only `closed` indices being in the actionable list.
344
345[source,yaml]
346-------------
347- filtertype: opened
348  exclude: False
349-------------
350
351This filter will result in only `open` indices being in the actionable list.
352
353
354
355[[fe_field]]
356== field
357
358NOTE: This setting is available in the <<filtertype_age,age>> filtertype, and
359  any filter which has the <<fe_use_age,`use_age`>> setting. This setting is
360  strictly optional.
361
362[source,yaml]
363-------------
364 - filtertype: age
365   source: field_stats
366   direction: older
367   unit: days
368   unit_count: 3
369   field: '@timestamp'
370   stats_result: min_value
371-------------
372
373The value of this setting must be a timestamp field name.  This field must be
374present in the indices being filtered or an exception will be raised, and
375execution will halt.
376
377In Curator 5.3 and older, source `field_stats` uses the
378http://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-field-stats.html[Field Stats API]
379to calculate either the `min_value` or the `max_value` of the <<fe_field,`field`>>
380as the <<fe_stats_result,`stats_result`>>, and then use that value for age
381comparisons.  In 5.4 and above, even though it is still called `field_stats`, it
382uses an aggregation to calculate the same values, as the `field_stats` API is
383no longer used in Elasticsearch 6.x and up.
384
385This setting is only used when <<fe_source,source>> is `field_stats`.
386
387The default value for this setting is `@timestamp`.
388
389
390
391[[fe_intersect]]
392== intersect
393
394NOTE: This setting is only available in the <<filtertype_age,period>> filtertype.
395   This setting is strictly optional.
396
397[source,yaml]
398-------------
399 - filtertype: period
400   source: field_stats
401   direction: older
402   intersect: true
403   unit: weeks
404   range_from: -1
405   range_to: -1
406   field: '@timestamp'
407   stats_result: min_value
408-------------
409
410The value of this setting must be `True` or `False`.
411
412`field_stats` uses an aggregation query to
413calculate either the `min_value` and the `max_value` of the <<fe_field,`field`>>
414as the <<fe_stats_result,`stats_result`>>.  If `intersect` is `True`, then
415only indices where the `min_value` _and_ the `max_value` are within the `range_from`
416and `range_to` (relative to `unit`) will match.  This means that either `min_value`
417or `max_value` can be used for <<fe_stats_result,`stats_result`>> when `intersect`
418is `True` with identical results.
419
420This setting is only used when <<fe_source,source>> is `field_stats`.
421
422The default value for this setting is `False`.
423
424
425
426[[fe_key]]
427== key
428
429NOTE: This setting is required when using the
430    <<filtertype_allocated,allocated filtertype>>.
431
432[source,yaml]
433-------------
434- filtertype: allocated
435  key: ...
436  value: ...
437  allocation_type:
438  exclude: True
439-------------
440
441The value of this setting should correspond to a node setting on one or more
442nodes in your cluster.
443
444For example, you might have set
445
446[source,sh]
447-----------
448node.tag: myvalue
449-----------
450
451in your `elasticsearch.yml` file for one or more of your nodes.  To match
452allocation in this case, set key to `tag`.
453
454These special attributes are also supported:
455
456[cols="2*", options="header"]
457|===
458|attribute
459|description
460
461|`_name`
462|Match nodes by node name
463
464|`_host_ip`
465|Match nodes by host IP address (IP associated with hostname)
466
467|`_publish_ip`
468|Match nodes by publish IP address
469
470|`_ip`
471|Match either `_host_ip` or `_publish_ip`
472
473|`_host`
474|Match nodes by hostname
475|===
476
477There is no default value. This setting must be set by the user or an exception
478will be raised, and execution will halt.
479
480
481
482[[fe_kind]]
483== kind
484
485NOTE: This setting is only used with the <<filtertype_pattern,pattern>> +
486    filtertype and is a required setting.
487
488This setting tells the <<filtertype_pattern,pattern>> what pattern type to
489match. Acceptable values for this setting are `prefix`, `suffix`, `timestring`,
490and `regex`.
491
492include::inc_filter_chaining.asciidoc[]
493
494There is no default value. This setting must be set by the user or an exception
495will be raised, and execution will halt.
496
497include::inc_kinds.asciidoc[]
498
499
500
501[[fe_max_num_segments]]
502== max_num_segments
503
504NOTE: This setting is only used with the <<filtertype_forcemerged,forcemerged>>
505  filtertype.
506
507[source,yaml]
508-------------
509- filtertype: forcemerged
510  max_num_segments: 2
511  exclude: True
512-------------
513
514The value for this setting is the cutoff number of segments per shard.  Indices
515which have this number of segments per shard, or fewer, will be actionable
516depending on the value of <<fe_exclude,exclude>>, which is `True` by default for
517the <<filtertype_forcemerged,forcemerged>> filter type.
518
519There is no default value. This setting must be set by the user or an exception
520will be raised, and execution will halt.
521
522
523
524[[fe_pattern]]
525== pattern
526
527NOTE: This setting is only used with the <<filtertype_count,count>> filtertype
528
529[source,yaml]
530-------------
531- filtertype: count
532  count: 1
533  pattern: '^(.*)-\d{6}$'
534  reverse: true
535-------------
536
537This particular example will match indices following the basic rollover pattern
538of `indexname-######`, and keep the highest numbered index for each group.
539
540For example, given indices `a-000001`, `a-000002`, `a-000003` and `b-000006`,
541and `b-000007`, the indices will would be matched are `a-000003` and `b-000007`.
542Indices that do not match the regular expression in `pattern` will be
543automatically excluded.
544
545This is particularly useful with indices created and managed using the
546{ref}/indices-rollover-index.html[Rollover API], as you can select only the
547active indices with the above example (<<fe_exclude,`exclude`>> defaults to `False`).
548Setting <<fe_exclude,`exclude`>> to `True` with the above example will _remove_
549the active rollover indices, leaving only those which have been rolled-over.
550
551While this is perhaps most useful for the aforementioned scenario, it can
552also be used with age-based indices as well.
553
554Items will remain in the actionable list depending on the value of
555<<fe_exclude,exclude>>, and <<fe_reverse,reverse>>.
556
557There is no default value. The value must include a capture group, defined by
558parenthesis, or left empty.  If a value is provided, and there is no capture
559group, and exception will be raised and execution will halt.
560
561
562[[fe_period_type]]
563== period_type
564
565NOTE: This setting is only used with the <<filtertype_period,period>> filtertype
566
567[source,yaml]
568-------------
569 - filtertype: period
570   period_type: absolute
571   source: name
572   timestring: '%Y.%m.%d'
573   unit: months
574   date_from: 2017.01
575   date_from_format: '%Y.%m'
576   date_to: 2017.01
577   date_to_format: '%Y.%m'
578-------------
579
580The value for this setting must be either `relative` or `absolute`. The default
581value is `relative`.
582
583
584[[fe_range_from]]
585== range_from
586
587NOTE: This setting is only used with the <<filtertype_period,period>> filtertype
588
589[source,yaml]
590-------------
591 - filtertype: period
592   source: name
593   range_from: -1
594   range_to: -1
595   timestring: '%Y.%m.%d'
596   unit: days
597-------------
598
599<<fe_range_from,range_from>> and <<fe_range_to,range_to>> are counters of whole
600<<fe_unit,units>>. A negative number indicates a whole unit in the past, while
601a positive number indicates a whole unit in the future. A `0` indicates the
602present unit.
603
604Read more about this setting in context in the
605<<filtertype_period,period filtertype documentation>>, including examples.
606
607
608
609[[fe_range_to]]
610== range_to
611
612NOTE: This setting is only used with the <<filtertype_period,period>> filtertype
613
614[source,yaml]
615-------------
616 - filtertype: period
617   source: name
618   range_from: -1
619   range_to: -1
620   timestring: '%Y.%m.%d'
621   unit: days
622-------------
623
624<<fe_range_from,range_from>> and <<fe_range_to,range_to>> are counters of whole
625<<fe_unit,units>>. A negative number indicates a whole unit in the past, while
626a positive number indicates a whole unit in the future. A `0` indicates the
627present unit.
628
629Read more about this setting in context in the
630<<filtertype_period,period filtertype documentation>>, including examples.
631
632
633
634[[fe_reverse]]
635== reverse
636
637NOTE: This setting is used in the <<filtertype_count,count>> and
638 <<filtertype_space,space>> filtertypes
639
640This setting affects the sort order of the indices.  `True` means
641reverse-alphabetical.  This means that if all index names share the same pattern
642with a date--e.g. index-2016.03.01--older indices will be selected first.
643
644The default value of this setting is `True`.
645
646This setting is ignored if <<fe_use_age,use_age>> is `True`.
647
648TIP: There are context-specific examples of how `reverse` works in the
649  <<filtertype_count,count>> and <<filtertype_space,space>> documentation.
650
651
652
653[[fe_source]]
654== source
655The _source_ from which to derive the index or snapshot age. Can be one of
656`name`, `creation_date`, or `field_stats`.
657
658NOTE: This setting is only used with the <<filtertype_age,age>> filtertype, or +
659with the <<filtertype_space,space>> filtertype when <<fe_use_age,use_age>> is
660set to `True`.
661
662NOTE: When using the <<filtertype_age,age>> filtertype, source requires +
663<<fe_direction,direction>>, <<fe_unit,unit>>, <<fe_unit_count,unit_count>>, +
664and additionally, the optional setting, <<fe_epoch,epoch>>.
665
666include::inc_sources.asciidoc[]
667
668
669
670[[fe_state]]
671== state
672
673NOTE: This setting is only used with the <<filtertype_state,state>> filtertype.
674
675[source,yaml]
676-------------
677- filtertype: state
678  state: SUCCESS
679-------------
680
681The value for this setting must be one of `SUCCESS`, `PARTIAL`, `FAILED`,
682or `IN_PROGRESS`.  This setting determines what kind of snapshots will be
683passed.
684
685The default value for this setting is `SUCCESS`.
686
687
688
689[[fe_stats_result]]
690== stats_result
691
692NOTE: This setting is only used with the <<filtertype_age,age>> filtertype.
693
694[source,yaml]
695-------------
696 - filtertype: age
697   source: field_stats
698   direction: older
699   unit: days
700   unit_count: 3
701   field: '@timestamp'
702   stats_result: min_value
703-------------
704
705The value for this setting can be either `min_value` or `max_value`.  This
706setting is only used when <<fe_source,source>> is `field_stats`, and determines
707whether Curator will use the minimum or maximum value of <<fe_field,field>> for
708time calculations.
709
710The default value for this setting is `min_value`.
711
712
713
714[[fe_timestring]]
715== timestring
716
717NOTE: This setting is only used with the <<filtertype_age,age>> filtertype, or +
718with the <<filtertype_space,space>> filtertype if <<fe_use_age,use_age>> is
719set to `True`.
720
721
722=== strftime
723
724This setting must be a valid Python strftime string.  It is used to match and
725extract the timestamp in an index or snapshot name.
726
727include::inc_strftime_table.asciidoc[]
728
729These identifiers may be combined with each other, and/or separated from each
730other with hyphens `-`, periods `.`, underscores `_`, or other characters valid
731in an index name.
732
733Each identifier must be preceded by a `%` character in the timestring.  For
734example, an index like `index-2016.04.01` would use a timestring of
735`'%Y.%m.%d'`.
736
737When <<fe_source,source>> is `name`, this setting must be set by the user or an
738exception will be raised, and execution will halt. There is no default value.
739
740include::inc_timestring_regex.asciidoc[]
741
742
743
744[[fe_threshold_behavior]]
745== threshold_behavior
746
747NOTE: This setting is only available in the <<filtertype_space,space>> filtertype.
748  This setting is optional, and defaults to `greater_than` to preserve backwards compatability.
749
750[source,yaml]
751-------------
752- filtertype: space
753  disk_space: 5
754  threshold_behavior: less_than
755-------------
756
757The value for this setting is `greater_than` (default) or `less_than`.
758
759When set to `less_than`, indices in less than `disk_space` gigabytes will be matched.
760When set to `greater_than` (default), indices larger than `disk_space` will be matched.
761
762
763
764[[fe_unit]]
765== unit
766
767NOTE: This setting is used with the <<filtertype_age,age>> filtertype, with the
768<<filtertype_period,period>> filtertype, or with the <<filtertype_space,space>>
769filtertype if <<fe_use_age,use_age>> is set to `True`.
770
771[source,yaml]
772-------------
773 - filtertype: age
774   source: creation_date
775   direction: older
776   unit: days
777   unit_count: 3
778-------------
779
780This setting must be one of `seconds`, `minutes`, `hours`, `days`, `weeks`,
781`months`, or `years`. The values `seconds` and `minutes` are not allowed with
782the <<filtertype_period,period>> filtertype and will result in an error
783condition if used there.
784
785For the <<filtertype_age,age>> filtertype, or when <<fe_use_age,use_age>> is set
786to `True`, <<fe_unit,unit>>, <<fe_unit_count,unit_count>>, and optionally,
787<<fe_epoch,epoch>>, are used by Curator to establish the moment in time point of
788reference with this formula:
789
790[source,sh]
791-----------
792point_of_reference = epoch - ((number of seconds in unit) * unit_count)
793-----------
794
795include::inc_unit_table.asciidoc[]
796
797
798If <<fe_epoch,epoch>> is unset, the current time is used. It is possible to set
799a point of reference in the future by using a negative value for
800<<fe_unit_count,unit_count>>.
801
802This setting must be set by the user or an exception will be raised, and
803execution will halt.
804
805TIP: See the <<filtertype_age,age filter documentation>> for more information
806  about time calculation.
807
808
809
810[[fe_unit_count]]
811== unit_count
812
813NOTE: This setting is only used with the <<filtertype_age,age>> filtertype, or +
814with the <<filtertype_space,space>> filtertype if <<fe_use_age,use_age>> is
815set to `True`.
816
817[source,yaml]
818-------------
819 - filtertype: age
820   source: creation_date
821   direction: older
822   unit: days
823   unit_count: 3
824-------------
825
826The value of this setting will be used as a multiplier for <<fe_unit,unit>>.
827
828<<fe_unit,unit>>, <<fe_unit_count,unit_count>>, and optionally,
829<<fe_epoch,epoch>>, are used by Curator to establish the moment in time point of
830reference with this formula:
831
832[source,sh]
833-----------
834point_of_reference = epoch - ((number of seconds in unit) * unit_count)
835-----------
836
837include::inc_unit_table.asciidoc[]
838
839If <<fe_epoch,epoch>> is unset, the current time is used. It is possible to set
840a point of reference in the future by using a negative value for
841<<fe_unit_count,unit_count>>.
842
843This setting must be set by the user or an exception will be raised, and
844execution will halt.
845
846If this setting is used in conjunction with <<fe_unit_count_pattern,unit_count_pattern>>, the configured
847value will only be used as a fallback value in case the pattern could not be matched. The value _-1_ has
848a special meaning in this case and causes the index to be ignored when pattern matching fails.
849
850TIP: See the <<filtertype_age,age filter documentation>> for more information
851  about time calculation.
852
853
854
855[[fe_unit_count_pattern]]
856== unit_count_pattern
857
858NOTE: This setting is only used with the age filtertype to define, whether the
859<<fe_unit_count,unit_count>> value is taken from the configuration or read from
860the index name via a regular expression.
861
862[source,yaml]
863-------------
864 - filtertype: age
865   source: creation_date
866   direction: older
867   unit: days
868   unit_count: 3
869   unit_count_pattern: -([0-9]+)-
870-------------
871
872This setting can be used in cases where the value against which index age should be assessed is not a static
873value but can be different for every index. For this case, there is the option of extracting the index
874specific value from the index names via a regular expression defined in this parameter.
875
876Consider for example the following index name patterns that contain the retention time in their name:
877_logstash-30-yyyy.mm.dd_, _logstash-12-yyyy.mm_, __3_logstash-yyyy.mm.dd_.
878
879To extract a value from the index names, this setting will be compiled as a regular expression and matched
880against index names, for a successful match, the value of the first capture group from the regular expression
881is used as the value for <<fe_unit_count,unit_count>>.
882
883If there is any error during compiling or matching the expression, or the expression does not
884contain a capture group, the value configured in <<fe_unit_count,unit_count>> is used as a fallback value,
885unless it is set to _-1_, in which case the index will be skipped.
886
887TIP: Regular expressions and match groups are not explained here as they are a fairly large and complex topic,
888but there are numerous resources online that will help. Using an online tool for testing regular expressions
889like https://regex101.com/[regex101.com] will help a lot when developing patterns.
890
891*Examples*
892
893* _logstash-30-yyyy.mm.dd_: Daily index that should be deleted after 30 days, indices that
894don't match the pattern will be deleted after 365 days
895[source,yaml]
896-------------
897 - filtertype: age
898   source: creation_date
899   direction: older
900   unit: days
901   unit_count: 365
902   unit_count_pattern: -([0-9]+)-
903-------------
904
905
906* _logstash-12-yyyy.mm_: Monthly index that should be deleted after 12 months, indices that
907don't match the pattern will be deleted after 3 months
908
909[source,yaml]
910-------------
911 - filtertype: age
912   source: creation_date
913   direction: older
914   unit: months
915   unit_count: 3
916   unit_count_pattern: -([0-9]+)-
917-------------
918
919
920* __3_logstash-yyyy.mm.dd_: Daily index that should be deleted after 3 years, indices that
921don't match the pattern will be ignored
922
923[source,yaml]
924-------------
925 - filtertype: age
926   source: creation_date
927   direction: older
928   unit: years
929   unit_count: -1
930   unit_count_pattern: ^_([0-9]+)_
931-------------
932
933IMPORTANT: Be sure to pay attention to the interaction of this parameter and <<fe_unit_count,unit_count>>!
934
935
936[[fe_use_age]]
937== use_age
938
939[source,yaml]
940-------------
941- filtertype: count
942  count: 10
943  use_age: True
944  source: creation_date
945-------------
946
947This setting allows filtering of indices by their age _after_ other
948considerations.
949
950The default value of this setting is `False`
951
952NOTE: Use of this setting requires the additional setting, <<fe_source,source>>.
953
954TIP: There are context-specific examples using `use_age` in the
955  <<filtertype_count,count>> and <<filtertype_space,space>> documentation.
956
957
958
959[[fe_value]]
960== value
961
962NOTE: This setting is only used with the <<filtertype_pattern,pattern>>
963filtertype and is a required setting.  There is a separate
964  <<option_value,value option>> associated with the
965  <<allocation,allocation action>>, and the
966  <<filtertype_allocated,allocated filtertype>>.
967
968The value of this setting is used by <<fe_kind,kind>> as follows:
969
970* `prefix`: Search the first part of an index name for the provided value
971* `suffix`: Search the last part of an index name for the provided value
972* `regex`: Provide your own regular expression, and Curator will find the matches.
973* `timestring`: An strftime string to extrapolate and find indices that match.
974    For example, given a `timestring` of `'%Y.%m.%d'`, matching indices would
975    include `logstash-2016.04.01` and `.marvel-2016.04.01`, but not
976    `myindex-2016-04-01`, as the pattern is different.
977
978IMPORTANT: Whatever you provide for `value` is always going to be a part of a +
979    regular expression.  The safest practice is to always encapsulate within
980    single quotes.  For example: `value: '-suffix'`, or `value: 'prefix-'`
981
982There is no default value. This setting must be set by the user or an exception
983will be raised, and execution will halt.
984
985TIP: There are context-specific examples using `value` in the <<fe_kind,kind>>
986  documentation.
987
988
989
990[[fe_week_starts_on]]
991== week_starts_on
992
993NOTE: This setting is only used with the <<filtertype_period,period>> filtertype +
994    when <<fe_period_type,period_type>> is `relative`.
995
996[source,yaml]
997-------------
998 - filtertype: period
999   source: name
1000   range_from: -1
1001   range_to: -1
1002   timestring: '%Y.%m.%d'
1003   unit: weeks
1004   week_starts_on: sunday
1005-------------
1006
1007The value of this setting indicates whether weeks should be measured starting on
1008`sunday` or `monday`.  Though Monday is the ISO standard, Sunday is frequently
1009preferred.
1010
1011This setting is only used when <<fe_unit,unit>> is set to `weeks`.
1012
1013The default value for this setting is `sunday`.
1014