1.. Licensed to the Apache Software Foundation (ASF) under one 2.. or more contributor license agreements. See the NOTICE file 3.. distributed with this work for additional information 4.. regarding copyright ownership. The ASF licenses this file 5.. to you under the Apache License, Version 2.0 (the 6.. "License"); you may not use this file except in compliance 7.. with the License. You may obtain a copy of the License at 8 9.. http://www.apache.org/licenses/LICENSE-2.0 10 11.. Unless required by applicable law or agreed to in writing, 12.. software distributed under the License is distributed on an 13.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 14.. KIND, either express or implied. See the License for the 15.. specific language governing permissions and limitations 16.. under the License. 17 18.. default-domain:: cpp 19.. highlight:: cpp 20.. cpp:namespace:: arrow::compute 21 22================= 23Compute Functions 24================= 25 26.. sidebar:: Contents 27 28 .. contents:: :local: 29 30The generic Compute API 31======================= 32 33.. TODO: describe API and how to invoke compute functions 34 35Functions and function registry 36------------------------------- 37 38Functions represent compute operations over inputs of possibly varying 39types. Internally, a function is implemented by one or several 40"kernels", depending on the concrete input types (for example, a function 41adding values from two inputs can have different kernels depending on 42whether the inputs are integral or floating-point). 43 44Functions are stored in a global :class:`FunctionRegistry` where 45they can be looked up by name. 46 47Input shapes 48------------ 49 50Computation inputs are represented as a general :class:`Datum` class, 51which is a tagged union of several shapes of data such as :class:`Scalar`, 52:class:`Array` and :class:`ChunkedArray`. Many compute functions support 53both array (chunked or not) and scalar inputs, however some will mandate 54either. For example, the ``fill_null`` function requires its second input 55to be a scalar, while ``sort_indices`` requires its first and only input to 56be an array. 57 58Invoking functions 59------------------ 60 61Compute functions can be invoked by name using 62:func:`arrow::compute::CallFunction`:: 63 64 std::shared_ptr<arrow::Array> numbers_array = ...; 65 std::shared_ptr<arrow::Scalar> increment = ...; 66 arrow::Datum incremented_datum; 67 68 ARROW_ASSIGN_OR_RAISE(incremented_datum, 69 arrow::compute::CallFunction("add", {numbers_array, increment})); 70 std::shared_ptr<Array> incremented_array = std::move(incremented_datum).array(); 71 72(note this example uses implicit conversion from ``std::shared_ptr<Array>`` 73to ``Datum``) 74 75Many compute functions are also available directly as concrete APIs, here 76:func:`arrow::compute::Add`:: 77 78 std::shared_ptr<arrow::Array> numbers_array = ...; 79 std::shared_ptr<arrow::Scalar> increment = ...; 80 arrow::Datum incremented_datum; 81 82 ARROW_ASSIGN_OR_RAISE(incremented_datum, 83 arrow::compute::Add(numbers_array, increment)); 84 std::shared_ptr<Array> incremented_array = std::move(incremented_datum).array(); 85 86Some functions accept or require an options structure that determines the 87exact semantics of the function:: 88 89 MinMaxOptions options; 90 options.null_handling = MinMaxOptions::EMIT_NULL; 91 92 std::shared_ptr<arrow::Array> array = ...; 93 arrow::Datum min_max_datum; 94 95 ARROW_ASSIGN_OR_RAISE(min_max_datum, 96 arrow::compute::CallFunction("min_max", {array}, &options)); 97 98 // Unpack struct scalar result (a two-field {"min", "max"} scalar) 99 const auto& min_max_scalar = \ 100 static_cast<const arrow::StructScalar&>(*min_max_datum.scalar()); 101 const auto min_value = min_max_scalar.value[0]; 102 const auto max_value = min_max_scalar.value[1]; 103 104.. seealso:: 105 :doc:`Compute API reference <api/compute>` 106 107Implicit casts 108============== 109 110Functions may require conversion of their arguments before execution if a 111kernel does not match the argument types precisely. For example comparison 112of dictionary encoded arrays is not directly supported by any kernel, but an 113implicit cast can be made allowing comparison against the decoded array. 114 115Each function may define implicit cast behaviour as appropriate. For example 116comparison and arithmetic kernels require identically typed arguments, and 117support execution against differing numeric types by promoting their arguments 118to numeric type which can accommodate any value from either input. 119 120.. _common-numeric-type: 121 122Common numeric type 123------------------- 124 125The common numeric type of a set of input numeric types is the smallest numeric 126type which can accommodate any value of any input. If any input is a floating 127point type the common numeric type is the widest floating point type among the 128inputs. Otherwise the common numeric type is integral and is signed if any input 129is signed. For example: 130 131+-------------------+----------------------+------------------------------------------------+ 132| Input types | Common numeric type | Notes | 133+===================+======================+================================================+ 134| int32, int32 | int32 | | 135+-------------------+----------------------+------------------------------------------------+ 136| int16, int32 | int32 | Max width is 32, promote LHS to int32 | 137+-------------------+----------------------+------------------------------------------------+ 138| uint16, int32 | int32 | One input signed, override unsigned | 139+-------------------+----------------------+------------------------------------------------+ 140| uint32, int32 | int64 | Widen to accommodate range of uint32 | 141+-------------------+----------------------+------------------------------------------------+ 142| uint16, uint32 | uint32 | All inputs unsigned, maintain unsigned | 143+-------------------+----------------------+------------------------------------------------+ 144| int16, uint32 | int64 | | 145+-------------------+----------------------+------------------------------------------------+ 146| uint64, int16 | int64 | int64 cannot accommodate all uint64 values | 147+-------------------+----------------------+------------------------------------------------+ 148| float32, int32 | float32 | Promote RHS to float32 | 149+-------------------+----------------------+------------------------------------------------+ 150| float32, float64 | float64 | | 151+-------------------+----------------------+------------------------------------------------+ 152| float32, int64 | float32 | int64 is wider, still promotes to float32 | 153+-------------------+----------------------+------------------------------------------------+ 154 155In particulary, note that comparing a ``uint64`` column to an ``int16`` column 156may emit an error if one of the ``uint64`` values cannot be expressed as the 157common type ``int64`` (for example, ``2 ** 63``). 158 159.. _compute-function-list: 160 161Available functions 162=================== 163 164Type categories 165--------------- 166 167To avoid exhaustively listing supported types, the tables below use a number 168of general type categories: 169 170* "Numeric": Integer types (Int8, etc.) and Floating-point types (Float32, 171 Float64, sometimes Float16). Some functions also accept Decimal128 input. 172 173* "Temporal": Date types (Date32, Date64), Time types (Time32, Time64), 174 Timestamp, Duration, Interval. 175 176* "Binary-like": Binary, LargeBinary, sometimes also FixedSizeBinary. 177 178* "String-like": String, LargeString. 179 180* "List-like": List, LargeList, sometimes also FixedSizeList. 181 182If you are unsure whether a function supports a concrete input type, we 183recommend you try it out. Unsupported input types return a ``TypeError`` 184:class:`Status`. 185 186Aggregations 187------------ 188 189+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 190| Function name | Arity | Input types | Output type | Options class | 191+==========================+============+====================+=======================+============================================+ 192| all | Unary | Boolean | Scalar Boolean | | 193+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 194| any | Unary | Boolean | Scalar Boolean | | 195+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 196| count | Unary | Any | Scalar Int64 | :struct:`CountOptions` | 197+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 198| mean | Unary | Numeric | Scalar Float64 | | 199+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 200| min_max | Unary | Numeric | Scalar Struct (1) | :struct:`MinMaxOptions` | 201+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 202| mode | Unary | Numeric | Struct (2) | :struct:`ModeOptions` | 203+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 204| quantile | Unary | Numeric | Scalar Numeric (3) | :struct:`QuantileOptions` | 205+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 206| stddev | Unary | Numeric | Scalar Float64 | :struct:`VarianceOptions` | 207+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 208| sum | Unary | Numeric | Scalar Numeric (4) | | 209+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 210| tdigest | Unary | Numeric | Scalar Float64 | :struct:`TDigestOptions` | 211+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 212| variance | Unary | Numeric | Scalar Float64 | :struct:`VarianceOptions` | 213+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 214 215Notes: 216 217* \(1) Output is a ``{"min": input type, "max": input type}`` Struct. 218 219* \(2) Output is an array of ``{"mode": input type, "count": Int64}`` Struct. 220 It contains the *N* most common elements in the input, in descending 221 order, where *N* is given in :member:`ModeOptions::n`. 222 If two values have the same count, the smallest one comes first. 223 Note that the output can have less than *N* elements if the input has 224 less than *N* distinct values. 225 226* \(3) Output is Float64 or input type, depending on QuantileOptions. 227 228* \(4) Output is Int64, UInt64 or Float64, depending on the input type. 229 230Element-wise ("scalar") functions 231--------------------------------- 232 233All element-wise functions accept both arrays and scalars as input. The 234semantics for unary functions are as follow: 235 236* scalar inputs produce a scalar output 237* array inputs produce an array output 238 239Binary functions have the following semantics (which is sometimes called 240"broadcasting" in other systems such as NumPy): 241 242* ``(scalar, scalar)`` inputs produce a scalar output 243* ``(array, array)`` inputs produce an array output (and both inputs must 244 be of the same length) 245* ``(scalar, array)`` and ``(array, scalar)`` produce an array output. 246 The scalar input is handled as if it were an array of the same length N 247 as the other input, with the same value repeated N times. 248 249Arithmetic functions 250~~~~~~~~~~~~~~~~~~~~ 251 252These functions expect two inputs of numeric type and apply a given binary 253operation to each pair of elements gathered from the inputs. If any of the 254input elements in a pair is null, the corresponding output element is null. 255Inputs will be cast to the :ref:`common numeric type <common-numeric-type>` 256(and dictionary decoded, if applicable) before the operation is applied. 257 258The default variant of these functions does not detect overflow (the result 259then typically wraps around). Each function is also available in an 260overflow-checking variant, suffixed ``_checked``, which returns 261an ``Invalid`` :class:`Status` when overflow is detected. 262 263+--------------------------+------------+--------------------+---------------------+ 264| Function name | Arity | Input types | Output type | 265+==========================+============+====================+=====================+ 266| add | Binary | Numeric | Numeric | 267+--------------------------+------------+--------------------+---------------------+ 268| add_checked | Binary | Numeric | Numeric | 269+--------------------------+------------+--------------------+---------------------+ 270| divide | Binary | Numeric | Numeric | 271+--------------------------+------------+--------------------+---------------------+ 272| divide_checked | Binary | Numeric | Numeric | 273+--------------------------+------------+--------------------+---------------------+ 274| multiply | Binary | Numeric | Numeric | 275+--------------------------+------------+--------------------+---------------------+ 276| multiply_checked | Binary | Numeric | Numeric | 277+--------------------------+------------+--------------------+---------------------+ 278| subtract | Binary | Numeric | Numeric | 279+--------------------------+------------+--------------------+---------------------+ 280| subtract_checked | Binary | Numeric | Numeric | 281+--------------------------+------------+--------------------+---------------------+ 282 283Comparisons 284~~~~~~~~~~~ 285 286These functions expect two inputs of numeric type (in which case they will be 287cast to the :ref:`common numeric type <common-numeric-type>` before comparison), 288or two inputs of Binary- or String-like types, or two inputs of Temporal types. 289If any input is dictionary encoded it will be expanded for the purposes of 290comparison. If any of the input elements in a pair is null, the corresponding 291output element is null. 292 293+--------------------------+------------+---------------------------------------------+---------------------+ 294| Function names | Arity | Input types | Output type | 295+==========================+============+=============================================+=====================+ 296| equal, not_equal | Binary | Numeric, Temporal, Binary- and String-like | Boolean | 297+--------------------------+------------+---------------------------------------------+---------------------+ 298| greater, greater_equal, | Binary | Numeric, Temporal, Binary- and String-like | Boolean | 299| less, less_equal | | | | 300+--------------------------+------------+---------------------------------------------+---------------------+ 301 302Logical functions 303~~~~~~~~~~~~~~~~~~ 304 305The normal behaviour for these functions is to emit a null if any of the 306inputs is null (similar to the semantics of ``NaN`` in floating-point 307computations). 308 309Some of them are also available in a `Kleene logic`_ variant (suffixed 310``_kleene``) where null is taken to mean "undefined". This is the 311interpretation of null used in SQL systems as well as R and Julia, 312for example. 313 314For the Kleene logic variants, therefore: 315 316* "true AND null", "null AND true" give "null" (the result is undefined) 317* "true OR null", "null OR true" give "true" 318* "false AND null", "null AND false" give "false" 319* "false OR null", "null OR false" give "null" (the result is undefined) 320 321+--------------------------+------------+--------------------+---------------------+ 322| Function name | Arity | Input types | Output type | 323+==========================+============+====================+=====================+ 324| and | Binary | Boolean | Boolean | 325+--------------------------+------------+--------------------+---------------------+ 326| and_not | Binary | Boolean | Boolean | 327+--------------------------+------------+--------------------+---------------------+ 328| and_kleene | Binary | Boolean | Boolean | 329+--------------------------+------------+--------------------+---------------------+ 330| and_not_kleene | Binary | Boolean | Boolean | 331+--------------------------+------------+--------------------+---------------------+ 332| invert | Unary | Boolean | Boolean | 333+--------------------------+------------+--------------------+---------------------+ 334| or | Binary | Boolean | Boolean | 335+--------------------------+------------+--------------------+---------------------+ 336| or_kleene | Binary | Boolean | Boolean | 337+--------------------------+------------+--------------------+---------------------+ 338| xor | Binary | Boolean | Boolean | 339+--------------------------+------------+--------------------+---------------------+ 340 341.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic#Kleene_and_Priest_logics 342 343String predicates 344~~~~~~~~~~~~~~~~~ 345 346These functions classify the input string elements according to their character 347contents. An empty string element emits false in the output. For ASCII 348variants of the functions (prefixed ``ascii_``), a string element with non-ASCII 349characters emits false in the output. 350 351The first set of functions operates on a character-per-character basis, 352and emit true in the output if the input contains only characters of a 353given class: 354 355+--------------------------+------------+--------------------+----------------+----------------------------------+ 356| Function name | Arity | Input types | Output type | Matched character class | 357+==========================+============+====================+================+==================================+ 358| ascii_is_alnum | Unary | String-like | Boolean | Alphanumeric ASCII | 359+--------------------------+------------+--------------------+----------------+----------------------------------+ 360| ascii_is_alpha | Unary | String-like | Boolean | Alphabetic ASCII | 361+--------------------------+------------+--------------------+----------------+----------------------------------+ 362| ascii_is_decimal | Unary | String-like | Boolean | Decimal ASCII \(1) | 363+--------------------------+------------+--------------------+----------------+----------------------------------+ 364| ascii_is_lower | Unary | String-like | Boolean | Lowercase ASCII \(2) | 365+--------------------------+------------+--------------------+----------------+----------------------------------+ 366| ascii_is_printable | Unary | String-like | Boolean | Printable ASCII | 367+--------------------------+------------+--------------------+----------------+----------------------------------+ 368| ascii_is_space | Unary | String-like | Boolean | Whitespace ASCII | 369+--------------------------+------------+--------------------+----------------+----------------------------------+ 370| ascii_is_upper | Unary | String-like | Boolean | Uppercase ASCII \(2) | 371+--------------------------+------------+--------------------+----------------+----------------------------------+ 372| utf8_is_alnum | Unary | String-like | Boolean | Alphanumeric Unicode | 373+--------------------------+------------+--------------------+----------------+----------------------------------+ 374| utf8_is_alpha | Unary | String-like | Boolean | Alphabetic Unicode | 375+--------------------------+------------+--------------------+----------------+----------------------------------+ 376| utf8_is_decimal | Unary | String-like | Boolean | Decimal Unicode | 377+--------------------------+------------+--------------------+----------------+----------------------------------+ 378| utf8_is_digit | Unary | String-like | Boolean | Unicode digit \(3) | 379+--------------------------+------------+--------------------+----------------+----------------------------------+ 380| utf8_is_lower | Unary | String-like | Boolean | Lowercase Unicode \(2) | 381+--------------------------+------------+--------------------+----------------+----------------------------------+ 382| utf8_is_numeric | Unary | String-like | Boolean | Numeric Unicode \(4) | 383+--------------------------+------------+--------------------+----------------+----------------------------------+ 384| utf8_is_printable | Unary | String-like | Boolean | Printable Unicode | 385+--------------------------+------------+--------------------+----------------+----------------------------------+ 386| utf8_is_space | Unary | String-like | Boolean | Whitespace Unicode | 387+--------------------------+------------+--------------------+----------------+----------------------------------+ 388| utf8_is_upper | Unary | String-like | Boolean | Uppercase Unicode \(2) | 389+--------------------------+------------+--------------------+----------------+----------------------------------+ 390 391* \(1) Also matches all numeric ASCII characters and all ASCII digits. 392 393* \(2) Non-cased characters, such as punctuation, do not match. 394 395* \(3) This is currently the same as ``utf8_is_decimal``. 396 397* \(4) Unlike ``utf8_is_decimal``, non-decimal numeric characters also match. 398 399The second set of functions also consider the character order in a string 400element: 401 402+--------------------------+------------+--------------------+---------------------+---------+ 403| Function name | Arity | Input types | Output type | Notes | 404+==========================+============+====================+=====================+=========+ 405| ascii_is_title | Unary | String-like | Boolean | \(1) | 406+--------------------------+------------+--------------------+---------------------+---------+ 407| utf8_is_title | Unary | String-like | Boolean | \(1) | 408+--------------------------+------------+--------------------+---------------------+---------+ 409 410* \(1) Output is true iff the input string element is title-cased, i.e. any 411 word starts with an uppercase character, followed by lowercase characters. 412 Word boundaries are defined by non-cased characters. 413 414The third set of functions examines string elements on a byte-per-byte basis: 415 416+--------------------------+------------+--------------------+---------------------+---------+ 417| Function name | Arity | Input types | Output type | Notes | 418+==========================+============+====================+=====================+=========+ 419| string_is_ascii | Unary | String-like | Boolean | \(1) | 420+--------------------------+------------+--------------------+---------------------+---------+ 421 422* \(1) Output is true iff the input string element contains only ASCII characters, 423 i.e. only bytes in [0, 127]. 424 425String transforms 426~~~~~~~~~~~~~~~~~ 427 428+--------------------------+------------+-------------------------+---------------------+---------+ 429| Function name | Arity | Input types | Output type | Notes | 430+==========================+============+=========================+=====================+=========+ 431| ascii_lower | Unary | String-like | String-like | \(1) | 432+--------------------------+------------+-------------------------+---------------------+---------+ 433| ascii_upper | Unary | String-like | String-like | \(1) | 434+--------------------------+------------+-------------------------+---------------------+---------+ 435| binary_length | Unary | Binary- or String-like | Int32 or Int64 | \(2) | 436+--------------------------+------------+-------------------------+---------------------+---------+ 437| utf8_lower | Unary | String-like | String-like | \(3) | 438+--------------------------+------------+-------------------------+---------------------+---------+ 439| utf8_upper | Unary | String-like | String-like | \(3) | 440+--------------------------+------------+-------------------------+---------------------+---------+ 441 442 443* \(1) Each ASCII character in the input is converted to lowercase or 444 uppercase. Non-ASCII characters are left untouched. 445 446* \(2) Output is the physical length in bytes of each input element. Output 447 type is Int32 for Binary / String, Int64 for LargeBinary / LargeString. 448 449* \(3) Each UTF8-encoded character in the input is converted to lowercase or 450 uppercase. 451 452 453String trimming 454~~~~~~~~~~~~~~~ 455 456These functions trim off characters on both sides (trim), or the left (ltrim) or right side (rtrim). 457 458+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 459| Function name | Arity | Input types | Output type | Options class | Notes | 460+==========================+============+=========================+=====================+========================================+=========+ 461| ascii_ltrim | Unary | String-like | String-like | :struct:`TrimOptions` | \(1) | 462+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 463| ascii_ltrim_whitespace | Unary | String-like | String-like | | \(2) | 464+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 465| ascii_rtrim | Unary | String-like | String-like | :struct:`TrimOptions` | \(1) | 466+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 467| ascii_rtrim_whitespace | Unary | String-like | String-like | | \(2) | 468+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 469| ascii_trim | Unary | String-like | String-like | :struct:`TrimOptions` | \(1) | 470+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 471| ascii_trim_whitespace | Unary | String-like | String-like | | \(2) | 472+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 473| utf8_ltrim | Unary | String-like | String-like | :struct:`TrimOptions` | \(3) | 474+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 475| utf8_ltrim_whitespace | Unary | String-like | String-like | | \(4) | 476+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 477| utf8_rtrim | Unary | String-like | String-like | :struct:`TrimOptions` | \(3) | 478+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 479| utf8_rtrim_whitespace | Unary | String-like | String-like | | \(4) | 480+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 481| utf8_trim | Unary | String-like | String-like | :struct:`TrimOptions` | \(3) | 482+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 483| utf8_trim_whitespace | Unary | String-like | String-like | | \(4) | 484+--------------------------+------------+-------------------------+---------------------+----------------------------------------+---------+ 485 486* \(1) Only characters specified in :member:`TrimOptions::characters` will be 487 trimmed off. Both the input string and the `characters` argument are 488 interpreted as ASCII characters. 489 490* \(2) Only trim off ASCII whitespace characters (``'\t'``, ``'\n'``, ``'\v'``, 491 ``'\f'``, ``'\r'`` and ``' '``). 492 493* \(3) Only characters specified in :member:`TrimOptions::characters` will be 494 trimmed off. 495 496* \(4) Only trim off Unicode whitespace characters. 497 498 499Containment tests 500~~~~~~~~~~~~~~~~~ 501 502+--------------------+------------+------------------------------------+---------------+----------------------------------------+ 503| Function name | Arity | Input types | Output type | Options class | 504+====================+============+====================================+===============+========================================+ 505| match_substring | Unary | String-like | Boolean (1) | :struct:`MatchSubstringOptions` | 506+--------------------+------------+------------------------------------+---------------+----------------------------------------+ 507| index_in | Unary | Boolean, Null, Numeric, Temporal, | Int32 (2) | :struct:`SetLookupOptions` | 508| | | Binary- and String-like | | | 509+--------------------+------------+------------------------------------+---------------+----------------------------------------+ 510| is_in | Unary | Boolean, Null, Numeric, Temporal, | Boolean (3) | :struct:`SetLookupOptions` | 511| | | Binary- and String-like | | | 512+--------------------+------------+------------------------------------+---------------+----------------------------------------+ 513 514* \(1) Output is true iff :member:`MatchSubstringOptions::pattern` 515 is a substring of the corresponding input element. 516 517* \(2) Output is the index of the corresponding input element in 518 :member:`SetLookupOptions::value_set`, if found there. Otherwise, 519 output is null. 520 521* \(3) Output is true iff the corresponding input element is equal to one 522 of the elements in :member:`SetLookupOptions::value_set`. 523 524 525String splitting 526~~~~~~~~~~~~~~~~ 527 528These functions split strings into lists of strings. All kernels can optionally 529be configured with a ``max_splits`` and a ``reverse`` parameter, where 530``max_splits == -1`` means no limit (the default). When ``reverse`` is true, 531the splitting is done starting from the end of the string; this is only relevant 532when a positive ``max_splits`` is given. 533 534+--------------------------+------------+-------------------------+-------------------+----------------------------------+---------+ 535| Function name | Arity | Input types | Output type | Options class | Notes | 536+==========================+============+=========================+===================+==================================+=========+ 537| split_pattern | Unary | String-like | List-like | :struct:`SplitPatternOptions` | \(1) | 538+--------------------------+------------+-------------------------+-------------------+----------------------------------+---------+ 539| utf8_split_whitespace | Unary | String-like | List-like | :struct:`SplitOptions` | \(2) | 540+--------------------------+------------+-------------------------+-------------------+----------------------------------+---------+ 541| ascii_split_whitespace | Unary | String-like | List-like | :struct:`SplitOptions` | \(3) | 542+--------------------------+------------+-------------------------+-------------------+----------------------------------+---------+ 543 544* \(1) The string is split when an exact pattern is found (the pattern itself 545 is not included in the output). 546 547* \(2) A non-zero length sequence of Unicode defined whitespace codepoints 548 is seen as separator. 549 550* \(3) A non-zero length sequence of ASCII defined whitespace bytes 551 (``'\t'``, ``'\n'``, ``'\v'``, ``'\f'``, ``'\r'`` and ``' '``) is seen 552 as separator. 553 554 555Structural transforms 556~~~~~~~~~~~~~~~~~~~~~ 557 558.. XXX (this category is a bit of a hodgepodge) 559 560+--------------------------+------------+------------------------------------------------+---------------------+---------+ 561| Function name | Arity | Input types | Output type | Notes | 562+==========================+============+================================================+=====================+=========+ 563| fill_null | Binary | Boolean, Null, Numeric, Temporal, String-like | Input type | \(1) | 564+--------------------------+------------+------------------------------------------------+---------------------+---------+ 565| is_nan | Unary | Float, Double | Boolean | \(2) | 566+--------------------------+------------+------------------------------------------------+---------------------+---------+ 567| is_null | Unary | Any | Boolean | \(3) | 568+--------------------------+------------+------------------------------------------------+---------------------+---------+ 569| is_valid | Unary | Any | Boolean | \(4) | 570+--------------------------+------------+------------------------------------------------+---------------------+---------+ 571| list_value_length | Unary | List-like | Int32 or Int64 | \(5) | 572+--------------------------+------------+------------------------------------------------+---------------------+---------+ 573| project | Varargs | Any | Struct | \(6) | 574+--------------------------+------------+------------------------------------------------+---------------------+---------+ 575 576* \(1) First input must be an array, second input a scalar of the same type. 577 Output is an array of the same type as the inputs, and with the same values 578 as the first input, except for nulls replaced with the second input value. 579 580* \(2) Output is true iff the corresponding input element is NaN. 581 582* \(3) Output is true iff the corresponding input element is null. 583 584* \(4) Output is true iff the corresponding input element is non-null. 585 586* \(5) Each output element is the length of the corresponding input element 587 (null if input is null). Output type is Int32 for List, Int64 for LargeList. 588 589* \(6) The output struct's field types are the types of its arguments. The 590 field names are specified using an instance of :struct:`ProjectOptions`. 591 The output shape will be scalar if all inputs are scalar, otherwise any 592 scalars will be broadcast to arrays. 593 594Conversions 595~~~~~~~~~~~ 596 597A general conversion function named ``cast`` is provided which accepts a large 598number of input and output types. The type to cast to can be passed in a 599:struct:`CastOptions` instance. As an alternative, the same service is 600provided by a concrete function :func:`~arrow::compute::Cast`. 601 602+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 603| Function name | Arity | Input types | Output type | Options class | 604+==========================+============+====================+=======================+============================================+ 605| cast | Unary | Many | Variable | :struct:`CastOptions` | 606+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 607| strptime | Unary | String-like | Timestamp | :struct:`StrptimeOptions` | 608+--------------------------+------------+--------------------+-----------------------+--------------------------------------------+ 609 610The conversions available with ``cast`` are listed below. In all cases, a 611null input value is converted into a null output value. 612 613**Truth value extraction** 614 615+-----------------------------+------------------------------------+--------------+ 616| Input type | Output type | Notes | 617+=============================+====================================+==============+ 618| Binary- and String-like | Boolean | \(1) | 619+-----------------------------+------------------------------------+--------------+ 620| Numeric | Boolean | \(2) | 621+-----------------------------+------------------------------------+--------------+ 622 623* \(1) Output is true iff the corresponding input value has non-zero length. 624 625* \(2) Output is true iff the corresponding input value is non-zero. 626 627**Same-kind conversion** 628 629+-----------------------------+------------------------------------+--------------+ 630| Input type | Output type | Notes | 631+=============================+====================================+==============+ 632| Int32 | 32-bit Temporal | \(1) | 633+-----------------------------+------------------------------------+--------------+ 634| Int64 | 64-bit Temporal | \(1) | 635+-----------------------------+------------------------------------+--------------+ 636| (Large)Binary | (Large)String | \(2) | 637+-----------------------------+------------------------------------+--------------+ 638| (Large)String | (Large)Binary | \(3) | 639+-----------------------------+------------------------------------+--------------+ 640| Numeric | Numeric | \(4) \(5) | 641+-----------------------------+------------------------------------+--------------+ 642| 32-bit Temporal | Int32 | \(1) | 643+-----------------------------+------------------------------------+--------------+ 644| 64-bit Temporal | Int64 | \(1) | 645+-----------------------------+------------------------------------+--------------+ 646| Temporal | Temporal | \(4) \(5) | 647+-----------------------------+------------------------------------+--------------+ 648 649* \(1) No-operation cast: the raw values are kept identical, only 650 the type is changed. 651 652* \(2) Validates the contents if :member:`CastOptions::allow_invalid_utf8` 653 is false. 654 655* \(3) No-operation cast: only the type is changed. 656 657* \(4) Overflow and truncation checks are enabled depending on 658 the given :struct:`CastOptions`. 659 660* \(5) Not all such casts have been implemented. 661 662**String representations** 663 664+-----------------------------+------------------------------------+---------+ 665| Input type | Output type | Notes | 666+=============================+====================================+=========+ 667| Boolean | String-like | | 668+-----------------------------+------------------------------------+---------+ 669| Numeric | String-like | | 670+-----------------------------+------------------------------------+---------+ 671 672**Generic conversions** 673 674+-----------------------------+------------------------------------+---------+ 675| Input type | Output type | Notes | 676+=============================+====================================+=========+ 677| Dictionary | Dictionary value type | \(1) | 678+-----------------------------+------------------------------------+---------+ 679| Extension | Extension storage type | | 680+-----------------------------+------------------------------------+---------+ 681| List-like | List-like | \(2) | 682+-----------------------------+------------------------------------+---------+ 683| Null | Any | | 684+-----------------------------+------------------------------------+---------+ 685 686* \(1) The dictionary indices are unchanged, the dictionary values are 687 cast from the input value type to the output value type (if a conversion 688 is available). 689 690* \(2) The list offsets are unchanged, the list values are cast from the 691 input value type to the output value type (if a conversion is 692 available). 693 694 695Array-wise ("vector") functions 696------------------------------- 697 698Associative transforms 699~~~~~~~~~~~~~~~~~~~~~~ 700 701+--------------------------+------------+------------------------------------+----------------------------+ 702| Function name | Arity | Input types | Output type | 703+==========================+============+====================================+============================+ 704| dictionary_encode | Unary | Boolean, Null, Numeric, | Dictionary (1) | 705| | | Temporal, Binary- and String-like | | 706+--------------------------+------------+------------------------------------+----------------------------+ 707| unique | Unary | Boolean, Null, Numeric, | Input type (2) | 708| | | Temporal, Binary- and String-like | | 709+--------------------------+------------+------------------------------------+----------------------------+ 710| value_counts | Unary | Boolean, Null, Numeric, | Input type (3) | 711| | | Temporal, Binary- and String-like | | 712+--------------------------+------------+------------------------------------+----------------------------+ 713 714* \(1) Output is ``Dictionary(Int32, input type)``. 715 716* \(2) Duplicates are removed from the output while the original order is 717 maintained. 718 719* \(3) Output is a ``{"values": input type, "counts": Int64}`` Struct. 720 Each output element corresponds to a unique value in the input, along 721 with the number of times this value has appeared. 722 723Selections 724~~~~~~~~~~ 725 726These functions select a subset of the first input defined by the second input. 727 728+-----------------+------------+---------------+--------------+------------------+-------------------------+-------------+ 729| Function name | Arity | Input type 1 | Input type 2 | Output type | Options class | Notes | 730+=================+============+===============+==============+==================+=========================+=============+ 731| filter | Binary | Any (1) | Boolean | Input type 1 | :struct:`FilterOptions` | \(2) | 732+-----------------+------------+---------------+--------------+------------------+-------------------------+-------------+ 733| take | Binary | Any (1) | Integer | Input type 1 | :struct:`TakeOptions` | \(3) | 734+-----------------+------------+---------------+--------------+------------------+-------------------------+-------------+ 735 736* \(1) Unions are unsupported. 737 738* \(2) Each element in input 1 is appended to the output iff the corresponding 739 element in input 2 is true. 740 741* \(3) For each element *i* in input 2, the *i*'th element in input 1 is 742 appended to the output. 743 744Sorts and partitions 745~~~~~~~~~~~~~~~~~~~~ 746 747In these functions, nulls are considered greater than any other value 748(they will be sorted or partitioned at the end of the array). 749Floating-point NaN values are considered greater than any other non-null 750value, but smaller than nulls. 751 752+-----------------------+------------+-------------------------+-------------------+--------------------------------+----------------+ 753| Function name | Arity | Input types | Output type | Options class | Notes | 754+=======================+============+=========================+===================+================================+================+ 755| partition_nth_indices | Unary | Binary- and String-like | UInt64 | :struct:`PartitionNthOptions` | \(1) \(3) | 756+-----------------------+------------+-------------------------+-------------------+--------------------------------+----------------+ 757| partition_nth_indices | Unary | Numeric | UInt64 | :struct:`PartitionNthOptions` | \(1) | 758+-----------------------+------------+-------------------------+-------------------+--------------------------------+----------------+ 759| array_sort_indices | Unary | Binary- and String-like | UInt64 | :struct:`ArraySortOptions` | \(2) \(3) \(4) | 760+-----------------------+------------+-------------------------+-------------------+--------------------------------+----------------+ 761| array_sort_indices | Unary | Numeric | UInt64 | :struct:`ArraySortOptions` | \(2) \(4) | 762+-----------------------+------------+-------------------------+-------------------+--------------------------------+----------------+ 763| sort_indices | Unary | Binary- and String-like | UInt64 | :struct:`SortOptions` | \(2) \(3) \(5) | 764+-----------------------+------------+-------------------------+-------------------+--------------------------------+----------------+ 765| sort_indices | Unary | Numeric | UInt64 | :struct:`SortOptions` | \(2) \(5) | 766+-----------------------+------------+-------------------------+-------------------+--------------------------------+----------------+ 767 768* \(1) The output is an array of indices into the input array, that define 769 a partial non-stable sort such that the *N*'th index points to the *N*'th 770 element in sorted order, and all indices before the *N*'th point to 771 elements less or equal to elements at or after the *N*'th (similar to 772 :func:`std::nth_element`). *N* is given in 773 :member:`PartitionNthOptions::pivot`. 774 775* \(2) The output is an array of indices into the input, that define a 776 stable sort of the input. 777 778* \(3) Input values are ordered lexicographically as bytestrings (even 779 for String arrays). 780 781* \(4) The input must be an array. The default order is ascending. 782 783* \(5) The input can be an array, chunked array, record batch or 784 table. If the input is a record batch or table, one or more sort 785 keys must be specified. 786 787Structural transforms 788~~~~~~~~~~~~~~~~~~~~~ 789 790+--------------------------+------------+--------------------+---------------------+---------+ 791| Function name | Arity | Input types | Output type | Notes | 792+==========================+============+====================+=====================+=========+ 793| list_flatten | Unary | List-like | List value type | \(1) | 794+--------------------------+------------+--------------------+---------------------+---------+ 795| list_parent_indices | Unary | List-like | Int32 or Int64 | \(2) | 796+--------------------------+------------+--------------------+---------------------+---------+ 797 798* \(1) The top level of nesting is removed: all values in the list child array, 799 including nulls, are appended to the output. However, nulls in the parent 800 list array are discarded. 801 802* \(2) For each value in the list child array, the index at which it is found 803 in the list array is appended to the output. Nulls in the parent list array 804 are discarded. 805 806