1 2[[filter_elements]] 3= Filter Elements 4 5[partintro] 6-- 7* <<fe_aliases,aliases>> 8* <<fe_allocation_type,allocation_type>> 9* <<fe_count,count>> 10* <<fe_date_from,date_from>> 11* <<fe_date_from_format,date_from_format>> 12* <<fe_date_to,date_to>> 13* <<fe_date_to_format,date_to_format>> 14* <<fe_direction,direction>> 15* <<fe_disk_space,disk_space>> 16* <<fe_epoch,epoch>> 17* <<fe_exclude,exclude>> 18* <<fe_field,field>> 19* <<fe_intersect,intersect>> 20* <<fe_key,key>> 21* <<fe_kind,kind>> 22* <<fe_max_num_segments,max_num_segments>> 23* <<fe_pattern,pattern>> 24* <<fe_period_type,period_type>> 25* <<fe_range_from,range_from>> 26* <<fe_range_to,range_to>> 27* <<fe_reverse,reverse>> 28* <<fe_source,source>> 29* <<fe_state,state>> 30* <<fe_stats_result,stats_result>> 31* <<fe_timestring,timestring>> 32* <<fe_threshold_behavior,threshold_behavior>> 33* <<fe_unit,unit>> 34* <<fe_unit_count,unit_count>> 35* <<fe_unit_count_pattern,unit_count_pattern>> 36* <<fe_use_age,use_age>> 37* <<fe_value,value>> 38* <<fe_week_starts_on,week_starts_on>> 39 40You can use <<envvars,environment variables>> in your configuration files. 41-- 42 43 44 45[[fe_aliases]] 46== aliases 47 48include::inc_filter_by_aliases.asciidoc[] 49 50NOTE: This setting is used only when using the <<filtertype_alias,alias>> 51 filter. 52 53The value of this setting must be a single alias name, or a list of alias names. 54This can be done in any of the ways YAML allows for lists or arrays. Here are 55a few examples. 56 57**Single** 58[source,txt] 59------- 60filters: 61- filtertype: alias 62 aliases: my_alias 63 exclude: False 64------- 65 66**List** 67 68- Flow style: 69+ 70[source,txt] 71------- 72filters: 73- filtertype: alias 74 aliases: [ my_alias, another_alias ] 75 exclude: False 76------- 77 78- Block style: 79+ 80[source,txt] 81------- 82filters: 83- filtertype: alias 84 aliases: 85 - my_alias 86 - another_alias 87 exclude: False 88------- 89 90There is no default value. This setting must be set by the user or an 91exception will be raised, and execution will halt. 92 93 94[[fe_allocation_type]] 95== allocation_type 96 97NOTE: This setting is used only when using the 98 <<filtertype_allocated,allocated>> filter. 99 100[source,yaml] 101------------- 102- filtertype: allocated 103 key: ... 104 value: ... 105 allocation_type: require 106 exclude: True 107------------- 108 109The value of this setting must be one of `require`, `include`, or `exclude`. 110 111Read more about these settings in the 112{ref}/shard-allocation-filtering.html[Elasticsearch documentation]. 113 114The default value for this setting is `require`. 115 116 117 118[[fe_count]] 119== count 120 121NOTE: This setting is only used with the <<filtertype_count,count>> filtertype + 122 and is a required setting. 123 124[source,yaml] 125------------- 126- filtertype: count 127 count: 10 128------------- 129 130The value for this setting is a number of indices or snapshots to match. 131 132Items will remain in the actionable list depending on the value of 133<<fe_exclude,exclude>>, and <<fe_reverse,reverse>>. 134 135There is no default value. This setting must be set by the user or an exception 136will be raised, and execution will halt. 137 138 139[[fe_date_from]] 140== date_from 141 142NOTE: This setting is only used with the <<filtertype_period,period>> filtertype + 143 when <<fe_period_type,period_type>> is `absolute`. 144 145[source,yaml] 146------------- 147 - filtertype: period 148 period_type: absolute 149 source: name 150 timestring: '%Y.%m.%d' 151 unit: months 152 date_from: 2017.01 153 date_from_format: '%Y.%m' 154 date_to: 2017.01 155 date_to_format: '%Y.%m' 156------------- 157 158The value for this setting should be a date which can be easily parsed 159using the strftime string in <<fe_date_from_format,`date_from_format`>>: 160 161include::inc_strftime_table.asciidoc[] 162 163[[fe_date_from_format]] 164== date_from_format 165 166NOTE: This setting is only used with the <<filtertype_period,period>> filtertype + 167 when <<fe_period_type,period_type>> is `absolute`. 168 169[source,yaml] 170------------- 171 - filtertype: period 172 period_type: absolute 173 source: name 174 timestring: '%Y.%m.%d' 175 unit: months 176 date_from: 2017.01 177 date_from_format: '%Y.%m' 178 date_to: 2017.01 179 date_to_format: '%Y.%m' 180------------- 181 182The value for this setting should be an strftime string which corresponds to 183the date in <<fe_date_from,`date_from`>>: 184 185include::inc_strftime_table.asciidoc[] 186 187[[fe_date_to]] 188== date_to 189 190NOTE: This setting is only used with the <<filtertype_period,period>> filtertype + 191 when <<fe_period_type,period_type>> is `absolute`. 192 193[source,yaml] 194------------- 195 - filtertype: period 196 period_type: absolute 197 source: name 198 timestring: '%Y.%m.%d' 199 unit: months 200 date_from: 2017.01 201 date_from_format: '%Y.%m' 202 date_to: 2017.01 203 date_to_format: '%Y.%m' 204------------- 205 206The value for this setting should be a date which can be easily parsed 207using the strftime string in <<fe_date_to_format,`date_to_format`>>: 208 209include::inc_strftime_table.asciidoc[] 210 211[[fe_date_to_format]] 212== date_to_format 213 214NOTE: This setting is only used with the <<filtertype_period,period>> filtertype + 215 when <<fe_period_type,period_type>> is `absolute`. 216 217[source,yaml] 218------------- 219 - filtertype: period 220 period_type: absolute 221 source: name 222 timestring: '%Y.%m.%d' 223 unit: months 224 date_from: 2017.01 225 date_from_format: '%Y.%m' 226 date_to: 2017.01 227 date_to_format: '%Y.%m' 228------------- 229 230The value for this setting should be an strftime string which corresponds to 231the date in <<fe_date_to,`date_to`>>: 232 233include::inc_strftime_table.asciidoc[] 234 235 236[[fe_direction]] 237== direction 238 239NOTE: This setting is only used with the <<filtertype_age,age>> filtertype. 240 241[source,yaml] 242------------- 243 - filtertype: age 244 source: creation_date 245 direction: older 246 unit: days 247 unit_count: 3 248------------- 249 250This setting must be either `older` or `younger`. This setting is used to 251determine whether indices or snapshots are `older` or `younger` than the 252reference point in time determined by <<fe_unit,unit>>, 253<<fe_unit_count,unit_count>>, and optionally, <<fe_epoch,epoch>>. If 254`direction` is `older`, then indices (or snapshots) which are _older_ than the 255reference point in time will be matched. Likewise, if `direction` is 256`younger`, then indices (or snapshots) which are _younger_ than the reference 257point in time will be matched. 258 259There is no default value. This setting must be set by the user or an 260exception will be raised, and execution will halt. 261 262 263 264[[fe_disk_space]] 265== disk_space 266 267NOTE: This setting is only used with the <<filtertype_space,space>> filtertype + 268 and is a required setting. 269 270[source,yaml] 271------------- 272- filtertype: space 273 disk_space: 100 274------------- 275 276The value for this setting is a number of gigabytes. 277 278Indices in excess of this number of gigabytes will be matched. 279 280There is no default value. This setting must be set by the user or an exception 281will be raised, and execution will halt. 282 283 284 285[[fe_epoch]] 286== epoch 287 288NOTE: This setting is available in the <<filtertype_age,age>> filtertype, and 289 any filter which has the <<fe_use_age,`use_age`>> setting. This setting is 290 strictly optional. 291 292TIP: This setting is not common. It is most frequently used for testing. 293 294<<fe_unit,unit>>, <<fe_unit_count,unit_count>>, and optionally, 295<<fe_epoch,epoch>>, are used by Curator to establish the moment in time point of 296reference with this formula: 297 298[source,sh] 299----------- 300point_of_reference = epoch - ((number of seconds in unit) * unit_count) 301----------- 302 303If <<fe_epoch,epoch>> is unset, the current time is used. It is possible to set 304a point of reference in the future by using a negative value for 305<<fe_unit_count,unit_count>>. 306 307=== Example 308 309[source,yaml] 310------------- 311 - filtertype: age 312 source: creation_date 313 direction: older 314 unit: days 315 unit_count: 3 316 epoch: 1491577200 317------------- 318 319The value for this setting must be an epoch timestamp. In this example, the 320given epoch time of `1491577200` is 2017-04-04T15:00:00Z (UTC). This will use 3213 days older than that timestamp as the point of reference for age comparisons. 322 323 324 325[[fe_exclude]] 326== exclude 327 328NOTE: This setting is available in _all_ filter types. 329 330If `exclude` is `True`, the filter will remove matches from the actionable list. 331If `exclude` is `False`, then only matches will be kept in the actionable list. 332 333The default value for this setting is different for each filter type. 334 335=== Examples 336 337[source,yaml] 338------------- 339- filtertype: opened 340 exclude: True 341------------- 342 343This filter will result in only `closed` indices being in the actionable list. 344 345[source,yaml] 346------------- 347- filtertype: opened 348 exclude: False 349------------- 350 351This filter will result in only `open` indices being in the actionable list. 352 353 354 355[[fe_field]] 356== field 357 358NOTE: This setting is available in the <<filtertype_age,age>> filtertype, and 359 any filter which has the <<fe_use_age,`use_age`>> setting. This setting is 360 strictly optional. 361 362[source,yaml] 363------------- 364 - filtertype: age 365 source: field_stats 366 direction: older 367 unit: days 368 unit_count: 3 369 field: '@timestamp' 370 stats_result: min_value 371------------- 372 373The value of this setting must be a timestamp field name. This field must be 374present in the indices being filtered or an exception will be raised, and 375execution will halt. 376 377In Curator 5.3 and older, source `field_stats` uses the 378http://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-field-stats.html[Field Stats API] 379to calculate either the `min_value` or the `max_value` of the <<fe_field,`field`>> 380as the <<fe_stats_result,`stats_result`>>, and then use that value for age 381comparisons. In 5.4 and above, even though it is still called `field_stats`, it 382uses an aggregation to calculate the same values, as the `field_stats` API is 383no longer used in Elasticsearch 6.x and up. 384 385This setting is only used when <<fe_source,source>> is `field_stats`. 386 387The default value for this setting is `@timestamp`. 388 389 390 391[[fe_intersect]] 392== intersect 393 394NOTE: This setting is only available in the <<filtertype_age,period>> filtertype. 395 This setting is strictly optional. 396 397[source,yaml] 398------------- 399 - filtertype: period 400 source: field_stats 401 direction: older 402 intersect: true 403 unit: weeks 404 range_from: -1 405 range_to: -1 406 field: '@timestamp' 407 stats_result: min_value 408------------- 409 410The value of this setting must be `True` or `False`. 411 412`field_stats` uses an aggregation query to 413calculate either the `min_value` and the `max_value` of the <<fe_field,`field`>> 414as the <<fe_stats_result,`stats_result`>>. If `intersect` is `True`, then 415only indices where the `min_value` _and_ the `max_value` are within the `range_from` 416and `range_to` (relative to `unit`) will match. This means that either `min_value` 417or `max_value` can be used for <<fe_stats_result,`stats_result`>> when `intersect` 418is `True` with identical results. 419 420This setting is only used when <<fe_source,source>> is `field_stats`. 421 422The default value for this setting is `False`. 423 424 425 426[[fe_key]] 427== key 428 429NOTE: This setting is required when using the 430 <<filtertype_allocated,allocated filtertype>>. 431 432[source,yaml] 433------------- 434- filtertype: allocated 435 key: ... 436 value: ... 437 allocation_type: 438 exclude: True 439------------- 440 441The value of this setting should correspond to a node setting on one or more 442nodes in your cluster. 443 444For example, you might have set 445 446[source,sh] 447----------- 448node.tag: myvalue 449----------- 450 451in your `elasticsearch.yml` file for one or more of your nodes. To match 452allocation in this case, set key to `tag`. 453 454These special attributes are also supported: 455 456[cols="2*", options="header"] 457|=== 458|attribute 459|description 460 461|`_name` 462|Match nodes by node name 463 464|`_host_ip` 465|Match nodes by host IP address (IP associated with hostname) 466 467|`_publish_ip` 468|Match nodes by publish IP address 469 470|`_ip` 471|Match either `_host_ip` or `_publish_ip` 472 473|`_host` 474|Match nodes by hostname 475|=== 476 477There is no default value. This setting must be set by the user or an exception 478will be raised, and execution will halt. 479 480 481 482[[fe_kind]] 483== kind 484 485NOTE: This setting is only used with the <<filtertype_pattern,pattern>> + 486 filtertype and is a required setting. 487 488This setting tells the <<filtertype_pattern,pattern>> what pattern type to 489match. Acceptable values for this setting are `prefix`, `suffix`, `timestring`, 490and `regex`. 491 492include::inc_filter_chaining.asciidoc[] 493 494There is no default value. This setting must be set by the user or an exception 495will be raised, and execution will halt. 496 497include::inc_kinds.asciidoc[] 498 499 500 501[[fe_max_num_segments]] 502== max_num_segments 503 504NOTE: This setting is only used with the <<filtertype_forcemerged,forcemerged>> 505 filtertype. 506 507[source,yaml] 508------------- 509- filtertype: forcemerged 510 max_num_segments: 2 511 exclude: True 512------------- 513 514The value for this setting is the cutoff number of segments per shard. Indices 515which have this number of segments per shard, or fewer, will be actionable 516depending on the value of <<fe_exclude,exclude>>, which is `True` by default for 517the <<filtertype_forcemerged,forcemerged>> filter type. 518 519There is no default value. This setting must be set by the user or an exception 520will be raised, and execution will halt. 521 522 523 524[[fe_pattern]] 525== pattern 526 527NOTE: This setting is only used with the <<filtertype_count,count>> filtertype 528 529[source,yaml] 530------------- 531- filtertype: count 532 count: 1 533 pattern: '^(.*)-\d{6}$' 534 reverse: true 535------------- 536 537This particular example will match indices following the basic rollover pattern 538of `indexname-######`, and keep the highest numbered index for each group. 539 540For example, given indices `a-000001`, `a-000002`, `a-000003` and `b-000006`, 541and `b-000007`, the indices will would be matched are `a-000003` and `b-000007`. 542Indices that do not match the regular expression in `pattern` will be 543automatically excluded. 544 545This is particularly useful with indices created and managed using the 546{ref}/indices-rollover-index.html[Rollover API], as you can select only the 547active indices with the above example (<<fe_exclude,`exclude`>> defaults to `False`). 548Setting <<fe_exclude,`exclude`>> to `True` with the above example will _remove_ 549the active rollover indices, leaving only those which have been rolled-over. 550 551While this is perhaps most useful for the aforementioned scenario, it can 552also be used with age-based indices as well. 553 554Items will remain in the actionable list depending on the value of 555<<fe_exclude,exclude>>, and <<fe_reverse,reverse>>. 556 557There is no default value. The value must include a capture group, defined by 558parenthesis, or left empty. If a value is provided, and there is no capture 559group, and exception will be raised and execution will halt. 560 561 562[[fe_period_type]] 563== period_type 564 565NOTE: This setting is only used with the <<filtertype_period,period>> filtertype 566 567[source,yaml] 568------------- 569 - filtertype: period 570 period_type: absolute 571 source: name 572 timestring: '%Y.%m.%d' 573 unit: months 574 date_from: 2017.01 575 date_from_format: '%Y.%m' 576 date_to: 2017.01 577 date_to_format: '%Y.%m' 578------------- 579 580The value for this setting must be either `relative` or `absolute`. The default 581value is `relative`. 582 583 584[[fe_range_from]] 585== range_from 586 587NOTE: This setting is only used with the <<filtertype_period,period>> filtertype 588 589[source,yaml] 590------------- 591 - filtertype: period 592 source: name 593 range_from: -1 594 range_to: -1 595 timestring: '%Y.%m.%d' 596 unit: days 597------------- 598 599<<fe_range_from,range_from>> and <<fe_range_to,range_to>> are counters of whole 600<<fe_unit,units>>. A negative number indicates a whole unit in the past, while 601a positive number indicates a whole unit in the future. A `0` indicates the 602present unit. 603 604Read more about this setting in context in the 605<<filtertype_period,period filtertype documentation>>, including examples. 606 607 608 609[[fe_range_to]] 610== range_to 611 612NOTE: This setting is only used with the <<filtertype_period,period>> filtertype 613 614[source,yaml] 615------------- 616 - filtertype: period 617 source: name 618 range_from: -1 619 range_to: -1 620 timestring: '%Y.%m.%d' 621 unit: days 622------------- 623 624<<fe_range_from,range_from>> and <<fe_range_to,range_to>> are counters of whole 625<<fe_unit,units>>. A negative number indicates a whole unit in the past, while 626a positive number indicates a whole unit in the future. A `0` indicates the 627present unit. 628 629Read more about this setting in context in the 630<<filtertype_period,period filtertype documentation>>, including examples. 631 632 633 634[[fe_reverse]] 635== reverse 636 637NOTE: This setting is used in the <<filtertype_count,count>> and 638 <<filtertype_space,space>> filtertypes 639 640This setting affects the sort order of the indices. `True` means 641reverse-alphabetical. This means that if all index names share the same pattern 642with a date--e.g. index-2016.03.01--older indices will be selected first. 643 644The default value of this setting is `True`. 645 646This setting is ignored if <<fe_use_age,use_age>> is `True`. 647 648TIP: There are context-specific examples of how `reverse` works in the 649 <<filtertype_count,count>> and <<filtertype_space,space>> documentation. 650 651 652 653[[fe_source]] 654== source 655The _source_ from which to derive the index or snapshot age. Can be one of 656`name`, `creation_date`, or `field_stats`. 657 658NOTE: This setting is only used with the <<filtertype_age,age>> filtertype, or + 659with the <<filtertype_space,space>> filtertype when <<fe_use_age,use_age>> is 660set to `True`. 661 662NOTE: When using the <<filtertype_age,age>> filtertype, source requires + 663<<fe_direction,direction>>, <<fe_unit,unit>>, <<fe_unit_count,unit_count>>, + 664and additionally, the optional setting, <<fe_epoch,epoch>>. 665 666include::inc_sources.asciidoc[] 667 668 669 670[[fe_state]] 671== state 672 673NOTE: This setting is only used with the <<filtertype_state,state>> filtertype. 674 675[source,yaml] 676------------- 677- filtertype: state 678 state: SUCCESS 679------------- 680 681The value for this setting must be one of `SUCCESS`, `PARTIAL`, `FAILED`, 682or `IN_PROGRESS`. This setting determines what kind of snapshots will be 683passed. 684 685The default value for this setting is `SUCCESS`. 686 687 688 689[[fe_stats_result]] 690== stats_result 691 692NOTE: This setting is only used with the <<filtertype_age,age>> filtertype. 693 694[source,yaml] 695------------- 696 - filtertype: age 697 source: field_stats 698 direction: older 699 unit: days 700 unit_count: 3 701 field: '@timestamp' 702 stats_result: min_value 703------------- 704 705The value for this setting can be either `min_value` or `max_value`. This 706setting is only used when <<fe_source,source>> is `field_stats`, and determines 707whether Curator will use the minimum or maximum value of <<fe_field,field>> for 708time calculations. 709 710The default value for this setting is `min_value`. 711 712 713 714[[fe_timestring]] 715== timestring 716 717NOTE: This setting is only used with the <<filtertype_age,age>> filtertype, or + 718with the <<filtertype_space,space>> filtertype if <<fe_use_age,use_age>> is 719set to `True`. 720 721 722=== strftime 723 724This setting must be a valid Python strftime string. It is used to match and 725extract the timestamp in an index or snapshot name. 726 727include::inc_strftime_table.asciidoc[] 728 729These identifiers may be combined with each other, and/or separated from each 730other with hyphens `-`, periods `.`, underscores `_`, or other characters valid 731in an index name. 732 733Each identifier must be preceded by a `%` character in the timestring. For 734example, an index like `index-2016.04.01` would use a timestring of 735`'%Y.%m.%d'`. 736 737When <<fe_source,source>> is `name`, this setting must be set by the user or an 738exception will be raised, and execution will halt. There is no default value. 739 740include::inc_timestring_regex.asciidoc[] 741 742 743 744[[fe_threshold_behavior]] 745== threshold_behavior 746 747NOTE: This setting is only available in the <<filtertype_space,space>> filtertype. 748 This setting is optional, and defaults to `greater_than` to preserve backwards compatability. 749 750[source,yaml] 751------------- 752- filtertype: space 753 disk_space: 5 754 threshold_behavior: less_than 755------------- 756 757The value for this setting is `greater_than` (default) or `less_than`. 758 759When set to `less_than`, indices in less than `disk_space` gigabytes will be matched. 760When set to `greater_than` (default), indices larger than `disk_space` will be matched. 761 762 763 764[[fe_unit]] 765== unit 766 767NOTE: This setting is used with the <<filtertype_age,age>> filtertype, with the 768<<filtertype_period,period>> filtertype, or with the <<filtertype_space,space>> 769filtertype if <<fe_use_age,use_age>> is set to `True`. 770 771[source,yaml] 772------------- 773 - filtertype: age 774 source: creation_date 775 direction: older 776 unit: days 777 unit_count: 3 778------------- 779 780This setting must be one of `seconds`, `minutes`, `hours`, `days`, `weeks`, 781`months`, or `years`. The values `seconds` and `minutes` are not allowed with 782the <<filtertype_period,period>> filtertype and will result in an error 783condition if used there. 784 785For the <<filtertype_age,age>> filtertype, or when <<fe_use_age,use_age>> is set 786to `True`, <<fe_unit,unit>>, <<fe_unit_count,unit_count>>, and optionally, 787<<fe_epoch,epoch>>, are used by Curator to establish the moment in time point of 788reference with this formula: 789 790[source,sh] 791----------- 792point_of_reference = epoch - ((number of seconds in unit) * unit_count) 793----------- 794 795include::inc_unit_table.asciidoc[] 796 797 798If <<fe_epoch,epoch>> is unset, the current time is used. It is possible to set 799a point of reference in the future by using a negative value for 800<<fe_unit_count,unit_count>>. 801 802This setting must be set by the user or an exception will be raised, and 803execution will halt. 804 805TIP: See the <<filtertype_age,age filter documentation>> for more information 806 about time calculation. 807 808 809 810[[fe_unit_count]] 811== unit_count 812 813NOTE: This setting is only used with the <<filtertype_age,age>> filtertype, or + 814with the <<filtertype_space,space>> filtertype if <<fe_use_age,use_age>> is 815set to `True`. 816 817[source,yaml] 818------------- 819 - filtertype: age 820 source: creation_date 821 direction: older 822 unit: days 823 unit_count: 3 824------------- 825 826The value of this setting will be used as a multiplier for <<fe_unit,unit>>. 827 828<<fe_unit,unit>>, <<fe_unit_count,unit_count>>, and optionally, 829<<fe_epoch,epoch>>, are used by Curator to establish the moment in time point of 830reference with this formula: 831 832[source,sh] 833----------- 834point_of_reference = epoch - ((number of seconds in unit) * unit_count) 835----------- 836 837include::inc_unit_table.asciidoc[] 838 839If <<fe_epoch,epoch>> is unset, the current time is used. It is possible to set 840a point of reference in the future by using a negative value for 841<<fe_unit_count,unit_count>>. 842 843This setting must be set by the user or an exception will be raised, and 844execution will halt. 845 846If this setting is used in conjunction with <<fe_unit_count_pattern,unit_count_pattern>>, the configured 847value will only be used as a fallback value in case the pattern could not be matched. The value _-1_ has 848a special meaning in this case and causes the index to be ignored when pattern matching fails. 849 850TIP: See the <<filtertype_age,age filter documentation>> for more information 851 about time calculation. 852 853 854 855[[fe_unit_count_pattern]] 856== unit_count_pattern 857 858NOTE: This setting is only used with the age filtertype to define, whether the 859<<fe_unit_count,unit_count>> value is taken from the configuration or read from 860the index name via a regular expression. 861 862[source,yaml] 863------------- 864 - filtertype: age 865 source: creation_date 866 direction: older 867 unit: days 868 unit_count: 3 869 unit_count_pattern: -([0-9]+)- 870------------- 871 872This setting can be used in cases where the value against which index age should be assessed is not a static 873value but can be different for every index. For this case, there is the option of extracting the index 874specific value from the index names via a regular expression defined in this parameter. 875 876Consider for example the following index name patterns that contain the retention time in their name: 877_logstash-30-yyyy.mm.dd_, _logstash-12-yyyy.mm_, __3_logstash-yyyy.mm.dd_. 878 879To extract a value from the index names, this setting will be compiled as a regular expression and matched 880against index names, for a successful match, the value of the first capture group from the regular expression 881is used as the value for <<fe_unit_count,unit_count>>. 882 883If there is any error during compiling or matching the expression, or the expression does not 884contain a capture group, the value configured in <<fe_unit_count,unit_count>> is used as a fallback value, 885unless it is set to _-1_, in which case the index will be skipped. 886 887TIP: Regular expressions and match groups are not explained here as they are a fairly large and complex topic, 888but there are numerous resources online that will help. Using an online tool for testing regular expressions 889like https://regex101.com/[regex101.com] will help a lot when developing patterns. 890 891*Examples* 892 893* _logstash-30-yyyy.mm.dd_: Daily index that should be deleted after 30 days, indices that 894don't match the pattern will be deleted after 365 days 895[source,yaml] 896------------- 897 - filtertype: age 898 source: creation_date 899 direction: older 900 unit: days 901 unit_count: 365 902 unit_count_pattern: -([0-9]+)- 903------------- 904 905 906* _logstash-12-yyyy.mm_: Monthly index that should be deleted after 12 months, indices that 907don't match the pattern will be deleted after 3 months 908 909[source,yaml] 910------------- 911 - filtertype: age 912 source: creation_date 913 direction: older 914 unit: months 915 unit_count: 3 916 unit_count_pattern: -([0-9]+)- 917------------- 918 919 920* __3_logstash-yyyy.mm.dd_: Daily index that should be deleted after 3 years, indices that 921don't match the pattern will be ignored 922 923[source,yaml] 924------------- 925 - filtertype: age 926 source: creation_date 927 direction: older 928 unit: years 929 unit_count: -1 930 unit_count_pattern: ^_([0-9]+)_ 931------------- 932 933IMPORTANT: Be sure to pay attention to the interaction of this parameter and <<fe_unit_count,unit_count>>! 934 935 936[[fe_use_age]] 937== use_age 938 939[source,yaml] 940------------- 941- filtertype: count 942 count: 10 943 use_age: True 944 source: creation_date 945------------- 946 947This setting allows filtering of indices by their age _after_ other 948considerations. 949 950The default value of this setting is `False` 951 952NOTE: Use of this setting requires the additional setting, <<fe_source,source>>. 953 954TIP: There are context-specific examples using `use_age` in the 955 <<filtertype_count,count>> and <<filtertype_space,space>> documentation. 956 957 958 959[[fe_value]] 960== value 961 962NOTE: This setting is only used with the <<filtertype_pattern,pattern>> 963filtertype and is a required setting. There is a separate 964 <<option_value,value option>> associated with the 965 <<allocation,allocation action>>, and the 966 <<filtertype_allocated,allocated filtertype>>. 967 968The value of this setting is used by <<fe_kind,kind>> as follows: 969 970* `prefix`: Search the first part of an index name for the provided value 971* `suffix`: Search the last part of an index name for the provided value 972* `regex`: Provide your own regular expression, and Curator will find the matches. 973* `timestring`: An strftime string to extrapolate and find indices that match. 974 For example, given a `timestring` of `'%Y.%m.%d'`, matching indices would 975 include `logstash-2016.04.01` and `.marvel-2016.04.01`, but not 976 `myindex-2016-04-01`, as the pattern is different. 977 978IMPORTANT: Whatever you provide for `value` is always going to be a part of a + 979 regular expression. The safest practice is to always encapsulate within 980 single quotes. For example: `value: '-suffix'`, or `value: 'prefix-'` 981 982There is no default value. This setting must be set by the user or an exception 983will be raised, and execution will halt. 984 985TIP: There are context-specific examples using `value` in the <<fe_kind,kind>> 986 documentation. 987 988 989 990[[fe_week_starts_on]] 991== week_starts_on 992 993NOTE: This setting is only used with the <<filtertype_period,period>> filtertype + 994 when <<fe_period_type,period_type>> is `relative`. 995 996[source,yaml] 997------------- 998 - filtertype: period 999 source: name 1000 range_from: -1 1001 range_to: -1 1002 timestring: '%Y.%m.%d' 1003 unit: weeks 1004 week_starts_on: sunday 1005------------- 1006 1007The value of this setting indicates whether weeks should be measured starting on 1008`sunday` or `monday`. Though Monday is the ISO standard, Sunday is frequently 1009preferred. 1010 1011This setting is only used when <<fe_unit,unit>> is set to `weeks`. 1012 1013The default value for this setting is `sunday`. 1014