1# RediSearch Aggregations 2 3Aggregations are a way to process the results of a search query, group, sort and transform them - and extract analytic insights from them. Much like aggregation queries in other databases and search engines, they can be used to create analytics reports, or perform [Faceted Search](https://en.wikipedia.org/wiki/Faceted_search) style queries. 4 5For example, indexing a web-server's logs, we can create a report for unique users by hour, country or any other breakdown; or create different reports for errors, warnings, etc. 6 7## Core concepts 8 9The basic idea of an aggregate query is this: 10 11* Perform a search query, filtering for records you wish to process. 12* Build a pipeline of operations that transform the results by zero or more steps of: 13 * **Group and Reduce**: grouping by fields in the results, and applying reducer functions on each group. 14 * **Sort**: sort the results based on one or more fields. 15 * **Apply Transformations**: Apply mathematical and string functions on fields in the pipeline, optionally creating new fields or replacing existing ones 16 * **Limit**: Limit the result, regardless of sorting the result. 17 * **Filter**: Filter the results (post-query) based on predicates relating to its values. 18 19The pipeline is dynamic and reentrant, and every operation can be repeated. For example, you can group by property X, sort the top 100 results by group size, then group by property Y and sort the results by some other property, then apply a transformation on the output. 20 21Figure 1: Aggregation Pipeline Example 22![Aggregation Pipeline](https://docs.google.com/drawings/d/e/2PACX-1vRFyP17ingsG86OYNaienojHHA8DwnlVVv67-WlKxv7a7xTJCluWvs3SzXYQSS6QqwB9QZ1vqDuoJ-0/pub?w=518&h=163) 23 24## Aggregate request format 25 26The aggregate request's syntax is defined as follows: 27 28```sql 29FT.AGGREGATE 30 {index_name:string} 31 {query_string:string} 32 [VERBATIM] 33 [LOAD {nargs:integer} {property:string} ...] 34 [GROUPBY 35 {nargs:integer} {property:string} ... 36 REDUCE 37 {FUNC:string} 38 {nargs:integer} {arg:string} ... 39 [AS {name:string}] 40 ... 41 ] ... 42 [SORTBY 43 {nargs:integer} {string} ... 44 [MAX {num:integer}] ... 45 ] ... 46 [APPLY 47 {EXPR:string} 48 AS {name:string} 49 ] ... 50 [FILTER {EXPR:string}] ... 51 [LIMIT {offset:integer} {num:integer} ] ... 52``` 53 54#### Parameters in detail 55 56Parameters which may take a variable number of arguments are expressed in the 57form of `param {nargs} {property_1... property_N}`. The first argument to the 58parameter is the number of arguments following the parameter. This allows 59RediSearch to avoid a parsing ambiguity in case one of your arguments has the 60name of another parameter. For example, to sort by first name, last name, and 61country, one would specify `SORTBY 6 firstName ASC lastName DESC country ASC`. 62 63* **index_name**: The index the query is executed against. 64 65* **query_string**: The base filtering query that retrieves the documents. It follows **the exact same syntax** as the search query, including filters, unions, not, optional, etc. 66 67* **LOAD {nargs} {property} …**: Load document fields from the document HASH objects. This should be avoided as a general rule of thumb. Fields needed for aggregations should be stored as **SORTABLE**, where they are available to the aggregation pipeline with very low latency. LOAD hurts the performance of aggregate queries considerably since every processed record needs to execute the equivalent of HMGET against a redis key, which when executed over millions of keys, amounts to very high processing times. 68 69* **GROUPBY {nargs} {property}**: Group the results in the pipeline based on one or more properties. Each group should have at least one reducer (See below), a function that handles the group entries, either counting them or performing multiple aggregate operations (see below). 70 71* **REDUCE {func} {nargs} {arg} … [AS {name}]**: Reduce the matching results in each group into a single record, using a reduction function. For example, COUNT will count the number of records in the group. See the Reducers section below for more details on available reducers. 72 73 The reducers can have their own property names using the `AS {name}` optional argument. If a name is not given, the resulting name will be the name of the reduce function and the group properties. For example, if a name is not given to COUNT_DISTINCT by property `@foo`, the resulting name will be `count_distinct(@foo)`. 74 75* **SORTBY {nargs} {property} {ASC|DESC} [MAX {num}]**: Sort the pipeline up until the point of SORTBY, using a list of properties. By default, sorting is ascending, but `ASC` or `DESC ` can be added for each property. `nargs` is the number of sorting parameters, including ASC and DESC. for example: `SORTBY 4 @foo ASC @bar DESC`. 76 77 `MAX` is used to optimized sorting, by sorting only for the n-largest elements. Although it is not connected to `LIMIT`, you usually need just `SORTBY … MAX` for common queries. 78 79* **APPLY {expr} AS {name}**: Apply a 1-to-1 transformation on one or more properties, and either store the result as a new property down the pipeline, or replace any property using this transformation. `expr` is an expression that can be used to perform arithmetic operations on numeric properties, or functions that can be applied on properties depending on their types (see below), or any combination thereof. For example: `APPLY "sqrt(@foo)/log(@bar) + 5" AS baz` will evaluate this expression dynamically for each record in the pipeline and store the result as a new property called baz, that can be referenced by further APPLY / SORTBY / GROUPBY / REDUCE operations down the pipeline. 80 81* **LIMIT {offset} {num}**. Limit the number of results to return just `num` results starting at index `offset` (zero based). AS mentioned above, it is much more efficient to use `SORTBY … MAX` if you are interested in just limiting the optput of a sort operation. 82 83 However, limit can be used to limit results without sorting, or for paging the n-largest results as determined by `SORTBY MAX`. For example, getting results 50-100 of the top 100 results is most efficiently expressed as `SORTBY 1 @foo MAX 100 LIMIT 50 50`. Removing the MAX from SORTBY will result in the pipeline sorting _all_ the records and then paging over results 50-100. 84 85* **FILTER {expr}**. Filter the results using predicate expressions relating to values in each result. They are is applied post-query and relate to the current state of the pipeline. See FILTER Expressions below for full details. 86 87## Quick example 88 89Let's assume we have log of visits to our website, each record containing the following fields/properties: 90 91* **url** (text, sortable) 92* **timestamp** (numeric, sortable) - unix timestamp of visit entry. 93* **country** (tag, sortable) 94* **user_id** (text, sortable, not indexed) 95 96### Example 1: unique users by hour, ordered chronologically. 97 98First of all, we want _all_ records in the index, because why not. The first step is to determine the index name and the filtering query. A filter query of `*` means "get all records": 99 100``` 101FT.AGGREGATE myIndex "*" 102``` 103 104Now we want to group the results by hour. Since we have the visit times as unix timestamps in second resolution, we need to extract the hour component of the timestamp. So we first add an APPLY step, that strips the sub-hour information from the timestamp and stores is as a new property, `hour`: 105 106``` 107FT.AGGREGATE myIndex "*" 108 APPLY "@timestamp - (@timestamp % 3600)" AS hour 109``` 110 111Now we want to group the results by hour, and count the distinct user ids in each hour. This is done by a GROUPBY/REDUCE step: 112 113``` 114FT.AGGREGATE myIndex "*" 115 APPLY "@timestamp - (@timestamp % 3600)" AS hour 116 117 GROUPBY 1 @hour 118 REDUCE COUNT_DISTINCT 1 @user_id AS num_users 119``` 120 121Now we'd like to sort the results by hour, ascending: 122 123``` 124FT.AGGREGATE myIndex "*" 125 APPLY "@timestamp - (@timestamp % 3600)" AS hour 126 127 GROUPBY 1 @hour 128 REDUCE COUNT_DISTINCT 1 @user_id AS num_users 129 130 SORTBY 2 @hour ASC 131``` 132 133And as a final step, we can format the hour as a human readable timestamp. This is done by calling the transformation function `timefmt` that formats unix timestamps. You can specify a format to be passed to the system's `strftime` function ([see documentation](http://strftime.org/)), but not specifying one is equivalent to specifying `%FT%TZ` to `strftime`. 134 135``` 136FT.AGGREGATE myIndex "*" 137 APPLY "@timestamp - (@timestamp % 3600)" AS hour 138 139 GROUPBY 1 @hour 140 REDUCE COUNT_DISTINCT 1 @user_id AS num_users 141 142 SORTBY 2 @hour ASC 143 144 APPLY timefmt(@hour) AS hour 145``` 146 147### Example 2: Sort visits to a specific URL by day and country: 148 149In this example we filter by the url, transform the timestamp to its day part, and group by the day and country, simply counting the number of visits per group. sorting by day ascending and country descending. 150 151``` 152FT.AGGREGATE myIndex "@url:\"about.html\"" 153 APPLY "@timestamp - (@timestamp % 86400)" AS day 154 GROUPBY 2 @day @country 155 REDUCE count 0 AS num_visits 156 SORTBY 4 @day ASC @country DESC 157``` 158 159## GROUPBY reducers 160 161`GROUPBY` step work similarly to SQL `GROUP BY` clauses, and create groups of results based on one or more properties in each record. For each group, we return the "group keys", or the values common to all records in the group, by which they were grouped together - along with the results of zero or more `REDUCE` clauses. 162 163Each `GROUPBY` step in the pipeline may be accompanied by zero or more `REDUCE` clauses. Reducers apply some accumulation function to each record in the group and reduce them into a single record representing the group. When we are finished processing all the records upstream of the `GROUPBY` step, each group emits its reduced record. 164 165For example, the simplest reducer is COUNT, which simply counts the number of records in each group. 166 167If multiple `REDUCE` clauses exist for a single `GROUPBY` step, each reducer works independently on each result and writes its final output once. Each reducer may have its own alias determined using the `AS` optional parameter. If `AS` is not specified, the alias is the reduce function and its parameters, e.g. `count_distinct(foo,bar)`. 168 169### Supported GROUPBY reducers 170 171#### COUNT 172 173**Format** 174 175``` 176REDUCE COUNT 0 177``` 178 179**Description** 180 181Count the number of records in each group 182 183#### COUNT_DISTINCT 184 185**Format** 186 187```` 188REDUCE COUNT_DISTINCT 1 {property} 189```` 190 191**Description** 192 193Count the number of distinct values for `property`. 194 195!!! note 196 The reducer creates a hash-set per group, and hashes each record. This can be memory heavy if the groups are big. 197 198#### COUNT_DISTINCTISH 199 200**Format** 201 202``` 203REDUCE COUNT_DISTINCTISH 1 {property} 204``` 205 206**Description** 207 208Same as COUNT_DISTINCT - but provide an approximation instead of an exact count, at the expense of less memory and CPU in big groups. 209 210!!! note 211 The reducer uses [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) counters per group, at ~3% error rate, and 1024 Bytes of constant space allocation per group. This means it is ideal for few huge groups and not ideal for many small groups. In the former case, it can be an order of magnitude faster and consume much less memory than COUNT_DISTINCT, but again, it does not fit every user case. 212 213#### SUM 214 215**Format** 216 217``` 218REDUCE SUM 1 {property} 219``` 220 221**Description** 222 223Return the sum of all numeric values of a given property in a group. Non numeric values if the group are counted as 0. 224 225#### MIN 226 227**Format** 228 229``` 230REDUCE MIN 1 {property} 231``` 232 233**Description** 234 235Return the minimal value of a property, whether it is a string, number or NULL. 236 237#### MAX 238 239**Format** 240 241``` 242REDUCE MAX 1 {property} 243``` 244 245**Description** 246 247Return the maximal value of a property, whether it is a string, number or NULL. 248 249#### AVG 250 251**Format** 252 253``` 254REDUCE AVG 1 {property} 255``` 256 257**Description** 258 259Return the average value of a numeric property. This is equivalent to reducing by sum and count, and later on applying the ratio of them as an APPLY step. 260 261#### STDDEV 262 263**Format** 264 265``` 266REDUCE STDDEV 1 {property} 267``` 268 269**Description** 270 271Return the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of a numeric property in the group. 272 273#### QUANTILE 274 275**Format** 276 277``` 278REDUCE QUANTILE 2 {property} {quantile} 279``` 280 281**Description** 282 283Return the value of a numeric property at a given quantile of the results. Quantile is expressed as a number between 0 and 1. For example, the median can be expressed as the quantile at 0.5, e.g. `REDUCE QUANTILE 2 @foo 0.5 AS median` . 284 285If multiple quantiles are required, just repeat the QUANTILE reducer for each quantile. e.g. `REDUCE QUANTILE 2 @foo 0.5 AS median REDUCE QUANTILE 2 @foo 0.99 AS p99` 286 287#### TOLIST 288 289**Format** 290 291``` 292REDUCE TOLIST 1 {property} 293``` 294 295**Description** 296 297Merge all **distinct** values of a given property into a single array. 298 299#### FIRST_VALUE 300 301**Format** 302 303``` 304REDUCE FIRST_VALUE {nargs} {property} [BY {property} [ASC|DESC]] 305``` 306 307**Description** 308 309Return the first or top value of a given property in the group, optionally by comparing that or another property. For example, you can extract the name of the oldest user in the group: 310 311``` 312REDUCE FIRST_VALUE 4 @name BY @age DESC 313``` 314 315If no `BY` is specified, we return the first value we encounter in the group. 316 317If you with to get the top or bottom value in the group sorted by the same value, you are better off using the `MIN/MAX` reducers, but the same effect will be achieved by doing `REDUCE FIRST_VALUE 4 @foo BY @foo DESC`. 318 319#### RANDOM_SAMPLE 320 321**Format** 322 323``` 324REDUCE RANDOM_SAMPLE {nargs} {property} {sample_size} 325``` 326 327**Description** 328 329Perform a reservoir sampling of the group elements with a given size, and return an array of the sampled items with an even distribution. 330 331## APPLY expressions 332 333`APPLY` performs a 1-to-1 transformation on one or more properties in each record. It either stores the result as a new property down the pipeline, or replaces any property using this transformation. 334 335The transformations are expressed as a combination of arithmetic expressions and built in functions. Evaluating functions and expressions is recursively nested and can be composed without limit. For example: `sqrt(log(foo) * floor(@bar/baz)) + (3^@qaz % 6)` or simply `@foo/@bar`. 336 337If an expression or a function is applied to values that do not match the expected types, no error is emitted but a NULL value is set as the result. 338 339APPLY steps must have an explicit alias determined by the `AS` parameter. 340 341### Literals inside expressions 342 343* Numbers are expressed as integers or floating point numbers, i.e. `2`, `3.141`, `-34`, etc. `inf` and `-inf` are acceptable as well. 344* Strings are quoted with either single or double quotes. Single quotes are acceptable inside strings quoted with double quotes and vice versa. Punctuation marks can be escaped with backslashes. e.g. `"foo's bar"` ,`'foo\'s bar'`, `"foo \"bar\""` . 345* Any literal or sub expression can be wrapped in parentheses to resolve ambiguities of operator precedence. 346 347### Arithmetic operations 348 349For numeric expressions and properties, we support addition (`+`), subtraction (`-`), multiplication (`*`), division (`/`), modulo (`%`) and power (`^`). We currently do not support bitwise logical operators. 350 351Note that these operators apply only to numeric values and numeric sub expressions. Any attempt to multiply a string by a number, for instance, will result in a NULL output. 352 353### List of numeric APPLY functions 354 355| Function | Description | Example | 356| -------- | ------------------------------------------------------------ | ------------------ | 357| log(x) | Return the logarithm of a number, property or sub-expression | `log(@foo)` | 358| abs(x) | Return the absolute number of a numeric expression | `abs(@foo-@bar)` | 359| ceil(x) | Round to the smallest value not less than x | `ceil(@foo/3.14)` | 360| floor(x) | Round to largest value not greater than x | `floor(@foo/3.14)` | 361| log2(x) | Return the logarithm of x to base 2 | `log2(2^@foo)` | 362| exp(x) | Return the exponent of x, i.e. `e^x` | `exp(@foo)` | 363| sqrt(x) | Return the square root of x | `sqrt(@foo)` | 364 365### List of string APPLY functions 366 367| Function | | | 368| -------------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- | 369| upper(s) | Return the uppercase conversion of s | `upper('hello world')` | 370| lower(s) | Return the lowercase conversion of 2 | `lower("HELLO WORLD")` | 371| substr(s, offset, count) | Return the substring of s, starting at _offset_ and having _count_ characters. <br />If offset is negative, it represents the distance from the end of the string. <br />If count is -1, it means "the rest of the string starting at offset". | `substr("hello", 0, 3)` <br> `substr("hello", -2, -1)` | 372| format( fmt, ...) | Use the arguments following `fmt` to format a string. <br />Currently the only format argument supported is `%s` and it applies to all types of arguments. | `format("Hello, %s, you are %s years old", @name, @age)` | 373| matched_terms([max_terms=100]) | Return the query terms that matched for each record (up to 100), as a list. If a limit is specified, we will return the first N matches we find - based on query order. | `matched_terms()` | 374| split(s, [sep=","], [strip=" "]) | Split a string by any character in the string sep, and strip any characters in strip. If only s is specified, we split by commas and strip spaces. The output is an array. | split("foo,bar") | 375| exists(s) | Checks whether a field exists in a document. | `exists(@field)` | 376 377### List of date/time APPLY functions 378 379| Function | Description | 380| ------------------- | ------------------------------------------------------------ | 381| timefmt(x, [fmt]) | Return a formatted time string based on a numeric timestamp value x. <br /> See [strftime](http://strftime.org/) for formatting options. <br />Not specifying `fmt` is equivalent to `%FT%TZ`. | 382| parsetime(timesharing, [fmt]) | The opposite of timefmt() - parse a time format using a given format string | 383| day(timestamp) | Round a Unix timestamp to midnight (00:00) start of the current day. | 384| hour(timestamp) | Round a Unix timestamp to the beginning of the current hour. | 385| minute(timestamp) | Round a Unix timestamp to the beginning of the current minute. | 386| month(timestamp) | Round a unix timestamp to the beginning of the current month. | 387| dayofweek(timestamp) | Convert a Unix timestamp to the day number (Sunday = 0). | 388| dayofmonth(timestamp) | Convert a Unix timestamp to the day of month number (1 .. 31). | 389| dayofyear(timestamp) | Convert a Unix timestamp to the day of year number (0 .. 365). | 390| year(timestamp) | Convert a Unix timestamp to the current year (e.g. 2018). | 391| monthofyear(timestamp) | Convert a Unix timestamp to the current month (0 .. 11). | 392 393## FILTER expressions 394 395FILTER expressions filter the results using predicates relating to values in the result set. 396 397The FILTER expressions are evaluated post-query and relate to the current state of the pipeline. Thus they can be useful to prune the results based on group calculations. Note that the filters are not indexed and will not speed the processing per se. 398 399Filter expressions follow the syntax of APPLY expressions, with the addition of the conditions `==`, `!=`, `<`, `<=`, `>`, `>=`. Two or more predicates can be combined with logical AND (`&&`) and OR (`||`). A single predicate can be negated with a NOT prefix (`!`). 400 401For example, filtering all results where the user name is 'foo' and the age is less than 20 is expressed as: 402 403``` 404FT.AGGREGATE 405 ... 406 FILTER "@name=='foo' && @age < 20" 407 ... 408``` 409 410Several filter steps can be added, although at the same stage in the pipeline, it is more efficient to combine several predicates into a single filter step. 411 412## Cursor API 413 414``` 415FT.AGGREGATE ... WITHCURSOR [COUNT {read size} MAXIDLE {idle timeout}] 416FT.CURSOR READ {idx} {cid} [COUNT {read size}] 417FT.CURSOR DEL {idx} {cid} 418``` 419 420You can use cursors with `FT.AGGREGATE`, with the `WITHCURSOR` keyword. Cursors allow you to 421consume only part of the response, allowing you to fetch additional results as needed. 422This is much quicker than using `LIMIT` with offset, since the query is executed only 423once, and its state is stored on the server. 424 425To use cursors, specify the `WITHCURSOR` keyword in `FT.AGGREGATE`, e.g. 426 427``` 428FT.AGGREGATE idx * WITHCURSOR 429``` 430 431This will return a response of an array with two elements. The first element is 432the actual (partial) results, and the second is the cursor ID. The cursor ID 433can then be fed to `FT.CURSOR READ` repeatedly, until the cursor ID is 0, in 434which case all results have been returned. 435 436To read from an existing cursor, use `FT.CURSOR READ`, e.g. 437 438``` 439FT.CURSOR READ idx 342459320 440``` 441 442Assuming `342459320` is the cursor ID returned from the `FT.AGGREGATE` request. 443 444Here is an example in pseudo-code: 445 446``` 447response, cursor = FT.AGGREGATE "idx" "redis" "WITHCURSOR"; 448while (1) { 449 processResponse(response) 450 if (!cursor) { 451 break; 452 } 453 response, cursor = FT.CURSOR read "idx" cursor 454} 455``` 456 457Note that even if the cursor is 0, a partial result may still be returned. 458 459### Cursor settings 460 461#### Read size 462 463You can control how many rows are read per each cursor fetch by using the 464`COUNT` parameter. This parameter can be specified both in `FT.AGGREGATE` 465(immediately after `WITHCURSOR`) or in `FT.CURSOR READ`. 466 467``` 468FT.AGGREGATE idx query WITHCURSOR COUNT 10 469``` 470 471Will read 10 rows at a time. 472 473You can override this setting by also specifying `COUNT` in `CURSOR READ`, e.g. 474 475``` 476FT.CURSOR READ idx 342459320 COUNT 50 477``` 478 479Will return at most 50 results. 480 481The default read size is 1000 482 483 484#### Timeouts and limits 485 486Because cursors are stateful resources which occupy memory on the server, they 487have a limited lifetime. In order to safeguard against orphaned/stale cursors, 488cursors have an idle timeout value. If no activity occurs on the cursor before 489the idle timeout, the cursor is deleted. The idle timer resets to 0 whenever 490the cursor is read from using `CURSOR READ`. 491 492The default idle timeout is 300000 milliseconds (or 300 seconds). You can modify 493the idle timeout using the `MAXIDLE` keyword when creating the cursor. Note that 494the value cannot exceed the default 300s. 495 496``` 497FT.AGGREGATE idx query WITHCURSOR MAXIDLE 10000 498``` 499 500Will set the limit for 10 seconds. 501 502### Other cursor commands 503 504Cursors can be explicity deleted using the `CURSOR DEL` command, e.g. 505 506``` 507FT.CURSOR DEL idx 342459320 508``` 509 510Note that cursors are automatically deleted if all their results have been 511returned, or if they have been timed out. 512 513All idle cursors can be forcefully purged at once using `FT.CURSOR GC idx 0` command. 514By default, RediSearch uses a lazy throttled approach to garbage collection, which 515collects idle cursors every 500 operations, or every second - whichever is later. 516