1.. _NEP31: 2 3============================================================ 4NEP 31 — Context-local and global overrides of the NumPy API 5============================================================ 6 7:Author: Hameer Abbasi <habbasi@quansight.com> 8:Author: Ralf Gommers <rgommers@quansight.com> 9:Author: Peter Bell <pbell@quansight.com> 10:Status: Draft 11:Type: Standards Track 12:Created: 2019-08-22 13 14 15Abstract 16-------- 17 18This NEP proposes to make all of NumPy's public API overridable via an 19extensible backend mechanism. 20 21Acceptance of this NEP means NumPy would provide global and context-local 22overrides in a separate namespace, as well as a dispatch mechanism similar 23to NEP-18 [2]_. First experiences with ``__array_function__`` show that it 24is necessary to be able to override NumPy functions that *do not take an 25array-like argument*, and hence aren't overridable via 26``__array_function__``. The most pressing need is array creation and coercion 27functions, such as ``numpy.zeros`` or ``numpy.asarray``; see e.g. NEP-30 [9]_. 28 29This NEP proposes to allow, in an opt-in fashion, overriding any part of the 30NumPy API. It is intended as a comprehensive resolution to NEP-22 [3]_, and 31obviates the need to add an ever-growing list of new protocols for each new 32type of function or object that needs to become overridable. 33 34Motivation and Scope 35-------------------- 36 37The primary end-goal of this NEP is to make the following possible: 38 39.. code:: python 40 41 # On the library side 42 import numpy.overridable as unp 43 44 def library_function(array): 45 array = unp.asarray(array) 46 # Code using unumpy as usual 47 return array 48 49 # On the user side: 50 import numpy.overridable as unp 51 import uarray as ua 52 import dask.array as da 53 54 ua.register_backend(da) # Can be done within Dask itself 55 56 library_function(dask_array) # works and returns dask_array 57 58 with unp.set_backend(da): 59 library_function([1, 2, 3, 4]) # actually returns a Dask array. 60 61Here, ``backend`` can be any compatible object defined either by NumPy or an 62external library, such as Dask or CuPy. Ideally, it should be the module 63``dask.array`` or ``cupy`` itself. 64 65These kinds of overrides are useful for both the end-user as well as library 66authors. End-users may have written or wish to write code that they then later 67speed up or move to a different implementation, say PyData/Sparse. They can do 68this simply by setting a backend. Library authors may also wish to write code 69that is portable across array implementations, for example ``sklearn`` may wish 70to write code for a machine learning algorithm that is portable across array 71implementations while also using array creation functions. 72 73This NEP takes a holistic approach: It assumes that there are parts of 74the API that need to be overridable, and that these will grow over time. It 75provides a general framework and a mechanism to avoid a design of a new 76protocol each time this is required. This was the goal of ``uarray``: to 77allow for overrides in an API without needing the design of a new protocol. 78 79This NEP proposes the following: That ``unumpy`` [8]_ becomes the 80recommended override mechanism for the parts of the NumPy API not yet covered 81by ``__array_function__`` or ``__array_ufunc__``, and that ``uarray`` is 82vendored into a new namespace within NumPy to give users and downstream 83dependencies access to these overrides. This vendoring mechanism is similar 84to what SciPy decided to do for making ``scipy.fft`` overridable (see [10]_). 85 86The motivation behind ``uarray`` is manyfold: First, there have been several 87attempts to allow dispatch of parts of the NumPy API, including (most 88prominently), the ``__array_ufunc__`` protocol in NEP-13 [4]_, and the 89``__array_function__`` protocol in NEP-18 [2]_, but this has shown the need 90for further protocols to be developed, including a protocol for coercion (see 91[5]_, [9]_). The reasons these overrides are needed have been extensively 92discussed in the references, and this NEP will not attempt to go into the 93details of why these are needed; but in short: It is necessary for library 94authors to be able to coerce arbitrary objects into arrays of their own types, 95such as CuPy needing to coerce to a CuPy array, for example, instead of 96a NumPy array. In simpler words, one needs things like ``np.asarray(...)`` or 97an alternative to "just work" and return duck-arrays. 98 99Usage and Impact 100---------------- 101 102This NEP allows for global and context-local overrides, as well as 103automatic overrides a-la ``__array_function__``. 104 105Here are some use-cases this NEP would enable, besides the 106first one stated in the motivation section: 107 108The first is allowing alternate dtypes to return their 109respective arrays. 110 111.. code:: python 112 113 # Returns an XND array 114 x = unp.ones((5, 5), dtype=xnd_dtype) # Or torch dtype 115 116The second is allowing overrides for parts of the API. 117This is to allow alternate and/or optimised implementations 118for ``np.linalg``, BLAS, and ``np.random``. 119 120.. code:: python 121 122 import numpy as np 123 import pyfftw # Or mkl_fft 124 125 # Makes pyfftw the default for FFT 126 np.set_global_backend(pyfftw) 127 128 # Uses pyfftw without monkeypatching 129 np.fft.fft(numpy_array) 130 131 with np.set_backend(pyfftw) # Or mkl_fft, or numpy 132 # Uses the backend you specified 133 np.fft.fft(numpy_array) 134 135This will allow an official way for overrides to work with NumPy without 136monkeypatching or distributing a modified version of NumPy. 137 138Here are a few other use-cases, implied but not already 139stated: 140 141.. code:: python 142 143 data = da.from_zarr('myfile.zarr') 144 # result should still be dask, all things being equal 145 result = library_function(data) 146 result.to_zarr('output.zarr') 147 148This second one would work if ``magic_library`` was built 149on top of ``unumpy``. 150 151.. code:: python 152 153 from dask import array as da 154 from magic_library import pytorch_predict 155 156 data = da.from_zarr('myfile.zarr') 157 # normally here one would use e.g. data.map_overlap 158 result = pytorch_predict(data) 159 result.to_zarr('output.zarr') 160 161There are some backends which may depend on other backends, for example xarray 162depending on `numpy.fft`, and transforming a time axis into a frequency axis, 163or Dask/xarray holding an array other than a NumPy array inside it. This would 164be handled in the following manner inside code:: 165 166 with ua.set_backend(cupy), ua.set_backend(dask.array): 167 # Code that has distributed GPU arrays here 168 169Backward compatibility 170---------------------- 171 172There are no backward incompatible changes proposed in this NEP. 173 174Detailed description 175-------------------- 176 177Proposals 178~~~~~~~~~ 179 180The only change this NEP proposes at its acceptance, is to make ``unumpy`` the 181officially recommended way to override NumPy, along with making some submodules 182overridable by default via ``uarray``. ``unumpy`` will remain a separate 183repository/package (which we propose to vendor to avoid a hard dependency, and 184use the separate ``unumpy`` package only if it is installed, rather than depend 185on for the time being). In concrete terms, ``numpy.overridable`` becomes an 186alias for ``unumpy``, if available with a fallback to the a vendored version if 187not. ``uarray`` and ``unumpy`` and will be developed primarily with the input 188of duck-array authors and secondarily, custom dtype authors, via the usual 189GitHub workflow. There are a few reasons for this: 190 191* Faster iteration in the case of bugs or issues. 192* Faster design changes, in the case of needed functionality. 193* ``unumpy`` will work with older versions of NumPy as well. 194* The user and library author opt-in to the override process, 195 rather than breakages happening when it is least expected. 196 In simple terms, bugs in ``unumpy`` mean that ``numpy`` remains 197 unaffected. 198* For ``numpy.fft``, ``numpy.linalg`` and ``numpy.random``, the functions in 199 the main namespace will mirror those in the ``numpy.overridable`` namespace. 200 The reason for this is that there may exist functions in the in these 201 submodules that need backends, even for ``numpy.ndarray`` inputs. 202 203Advantanges of ``unumpy`` over other solutions 204^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 205 206``unumpy`` offers a number of advantanges over the approach of defining a new 207protocol for every problem encountered: Whenever there is something requiring 208an override, ``unumpy`` will be able to offer a unified API with very minor 209changes. For example: 210 211* ``ufunc`` objects can be overridden via their ``__call__``, ``reduce`` and 212 other methods. 213* Other functions can be overridden in a similar fashion. 214* ``np.asduckarray`` goes away, and becomes ``np.overridable.asarray`` with a 215 backend set. 216* The same holds for array creation functions such as ``np.zeros``, 217 ``np.empty`` and so on. 218 219This also holds for the future: Making something overridable would require only 220minor changes to ``unumpy``. 221 222Another promise ``unumpy`` holds is one of default implementations. Default 223implementations can be provided for any multimethod, in terms of others. This 224allows one to override a large part of the NumPy API by defining only a small 225part of it. This is to ease the creation of new duck-arrays, by providing 226default implementations of many functions that can be easily expressed in 227terms of others, as well as a repository of utility functions that help in the 228implementation of duck-arrays that most duck-arrays would require. This would 229allow us to avoid designing entire protocols, e.g., a protocol for stacking 230and concatenating would be replaced by simply implementing ``stack`` and/or 231``concatenate`` and then providing default implementations for everything else 232in that class. The same applies for transposing, and many other functions for 233which protocols haven't been proposed, such as ``isin`` in terms of ``in1d``, 234``setdiff1d`` in terms of ``unique``, and so on. 235 236It also allows one to override functions in a manner which 237``__array_function__`` simply cannot, such as overriding ``np.einsum`` with the 238version from the ``opt_einsum`` package, or Intel MKL overriding FFT, BLAS 239or ``ufunc`` objects. They would define a backend with the appropriate 240multimethods, and the user would select them via a ``with`` statement, or 241registering them as a backend. 242 243The last benefit is a clear way to coerce to a given backend (via the 244``coerce`` keyword in ``ua.set_backend``), and a protocol 245for coercing not only arrays, but also ``dtype`` objects and ``ufunc`` objects 246with similar ones from other libraries. This is due to the existence of actual, 247third party dtype packages, and their desire to blend into the NumPy ecosystem 248(see [6]_). This is a separate issue compared to the C-level dtype redesign 249proposed in [7]_, it's about allowing third-party dtype implementations to 250work with NumPy, much like third-party array implementations. These can provide 251features such as, for example, units, jagged arrays or other such features that 252are outside the scope of NumPy. 253 254Mixing NumPy and ``unumpy`` in the same file 255^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 256 257Normally, one would only want to import only one of ``unumpy`` or ``numpy``, 258you would import it as ``np`` for familiarity. However, there may be situations 259where one wishes to mix NumPy and the overrides, and there are a few ways to do 260this, depending on the user's style:: 261 262 from numpy import overridable as unp 263 import numpy as np 264 265or:: 266 267 import numpy as np 268 269 # Use unumpy via np.overridable 270 271Duck-array coercion 272~~~~~~~~~~~~~~~~~~~ 273 274There are inherent problems about returning objects that are not NumPy arrays 275from ``numpy.array`` or ``numpy.asarray``, particularly in the context of C/C++ 276or Cython code that may get an object with a different memory layout than the 277one it expects. However, we believe this problem may apply not only to these 278two functions but all functions that return NumPy arrays. For this reason, 279overrides are opt-in for the user, by using the submodule ``numpy.overridable`` 280rather than ``numpy``. NumPy will continue to work unaffected by anything in 281``numpy.overridable``. 282 283If the user wishes to obtain a NumPy array, there are two ways of doing it: 284 2851. Use ``numpy.asarray`` (the non-overridable version). 2862. Use ``numpy.overridable.asarray`` with the NumPy backend set and coercion 287 enabled 288 289Aliases outside of the ``numpy.overridable`` namespace 290~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 291 292All functionality in ``numpy.random``, ``numpy.linalg`` and ``numpy.fft`` 293will be aliased to their respective overridable versions inside 294``numpy.overridable``. The reason for this is that there are alternative 295implementations of RNGs (``mkl-random``), linear algebra routines (``eigen``, 296``blis``) and FFT routines (``mkl-fft``, ``pyFFTW``) that need to operate on 297``numpy.ndarray`` inputs, but still need the ability to switch behaviour. 298 299This is different from monkeypatching in a few different ways: 300 301* The caller-facing signature of the function is always the same, 302 so there is at least the loose sense of an API contract. Monkeypatching 303 does not provide this ability. 304* There is the ability of locally switching the backend. 305* It has been `suggested <http://numpy-discussion.10968.n7.nabble.com/NEP-31-Context-local-and-global-overrides-of-the-NumPy-API-tp47452p47472.html>`_ 306 that the reason that 1.17 hasn't landed in the Anaconda defaults channel is 307 due to the incompatibility between monkeypatching and ``__array_function__``, 308 as monkeypatching would bypass the protocol completely. 309* Statements of the form ``from numpy import x; x`` and ``np.x`` would have 310 different results depending on whether the import was made before or 311 after monkeypatching happened. 312 313All this isn't possible at all with ``__array_function__`` or 314``__array_ufunc__``. 315 316It has been formally realised (at least in part) that a backend system is 317needed for this, in the `NumPy roadmap <https://numpy.org/neps/roadmap.html#other-functionality>`_. 318 319For ``numpy.random``, it's still necessary to make the C-API fit the one 320proposed in `NEP-19 <https://numpy.org/neps/nep-0019-rng-policy.html>`_. 321This is impossible for `mkl-random`, because then it would need to be 322rewritten to fit that framework. The guarantees on stream 323compatibility will be the same as before, but if there's a backend that affects 324``numpy.random`` set, we make no guarantees about stream compatibility, and it 325is up to the backend author to provide their own guarantees. 326 327Providing a way for implicit dispatch 328~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 329 330It has been suggested that the ability to dispatch methods which do not take 331a dispatchable is needed, while guessing that backend from another dispatchable. 332 333As a concrete example, consider the following: 334 335.. code:: python 336 337 with unumpy.determine_backend(array_like, np.ndarray): 338 unumpy.arange(len(array_like)) 339 340While this does not exist yet in ``uarray``, it is trivial to add it. The need for 341this kind of code exists because one might want to have an alternative for the 342proposed ``*_like`` functions, or the ``like=`` keyword argument. The need for these 343exists because there are functions in the NumPy API that do not take a dispatchable 344argument, but there is still the need to select a backend based on a different 345dispatchable. 346 347The need for an opt-in module 348~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 349 350The need for an opt-in module is realised because of a few reasons: 351 352* There are parts of the API (like `numpy.asarray`) that simply cannot be 353 overridden due to incompatibility concerns with C/Cython extensions, however, 354 one may want to coerce to a duck-array using ``asarray`` with a backend set. 355* There are possible issues around an implicit option and monkeypatching, such 356 as those mentioned above. 357 358NEP 18 notes that this may require maintenance of two separate APIs. However, 359this burden may be lessened by, for example, parametrizing all tests over 360``numpy.overridable`` separately via a fixture. This also has the side-effect 361of thoroughly testing it, unlike ``__array_function__``. We also feel that it 362provides an oppurtunity to separate the NumPy API contract properly from the 363implementation. 364 365Benefits to end-users and mixing backends 366~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 367 368Mixing backends is easy in ``uarray``, one only has to do: 369 370.. code:: python 371 372 # Explicitly say which backends you want to mix 373 ua.register_backend(backend1) 374 ua.register_backend(backend2) 375 ua.register_backend(backend3) 376 377 # Freely use code that mixes backends here. 378 379The benefits to end-users extend beyond just writing new code. Old code 380(usually in the form of scripts) can be easily ported to different backends 381by a simple import switch and a line adding the preferred backend. This way, 382users may find it easier to port existing code to GPU or distributed computing. 383 384Related Work 385------------ 386 387Other override mechanisms 388~~~~~~~~~~~~~~~~~~~~~~~~~ 389 390* NEP-18, the ``__array_function__`` protocol. [2]_ 391* NEP-13, the ``__array_ufunc__`` protocol. [3]_ 392* NEP-30, the ``__duck_array__`` protocol. [9]_ 393 394Existing NumPy-like array implementations 395~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 396 397* Dask: https://dask.org/ 398* CuPy: https://cupy.chainer.org/ 399* PyData/Sparse: https://sparse.pydata.org/ 400* Xnd: https://xnd.readthedocs.io/ 401* Astropy's Quantity: https://docs.astropy.org/en/stable/units/ 402 403Existing and potential consumers of alternative arrays 404~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 405 406* Dask: https://dask.org/ 407* scikit-learn: https://scikit-learn.org/ 408* xarray: https://xarray.pydata.org/ 409* TensorLy: http://tensorly.org/ 410 411Existing alternate dtype implementations 412~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 413 414* ``ndtypes``: https://ndtypes.readthedocs.io/en/latest/ 415* Datashape: https://datashape.readthedocs.io 416* Plum: https://plum-py.readthedocs.io/ 417 418Alternate implementations of parts of the NumPy API 419~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 420 421* ``mkl_random``: https://github.com/IntelPython/mkl_random 422* ``mkl_fft``: https://github.com/IntelPython/mkl_fft 423* ``bottleneck``: https://github.com/pydata/bottleneck 424* ``opt_einsum``: https://github.com/dgasmith/opt_einsum 425 426Implementation 427-------------- 428 429The implementation of this NEP will require the following steps: 430 431* Implementation of ``uarray`` multimethods corresponding to the 432 NumPy API, including classes for overriding ``dtype``, ``ufunc`` 433 and ``array`` objects, in the ``unumpy`` repository, which are usually 434 very easy to create. 435* Moving backends from ``unumpy`` into the respective array libraries. 436 437Maintenance can be eased by testing over ``{numpy, unumpy}`` via parameterized 438tests. If a new argument is added to a method, the corresponding argument 439extractor and replacer will need to be updated within ``unumpy``. 440 441A lot of argument extractors can be re-used from the existing implementation 442of the ``__array_function__`` protocol, and the replacers can be usually 443re-used across many methods. 444 445For the parts of the namespace which are going to be overridable by default, 446the main method will need to be renamed and hidden behind a ``uarray`` multimethod. 447 448Default implementations are usually seen in the documentation using the words 449"equivalent to", and thus, are easily available. 450 451``uarray`` Primer 452~~~~~~~~~~~~~~~~~ 453 454**Note:** *This section will not attempt to go into too much detail about 455uarray, that is the purpose of the uarray documentation.* [1]_ 456*However, the NumPy community will have input into the design of 457uarray, via the issue tracker.* 458 459``unumpy`` is the interface that defines a set of overridable functions 460(multimethods) compatible with the numpy API. To do this, it uses the 461``uarray`` library. ``uarray`` is a general purpose tool for creating 462multimethods that dispatch to one of multiple different possible backend 463implementations. In this sense, it is similar to the ``__array_function__`` 464protocol but with the key difference that the backend is explicitly installed 465by the end-user and not coupled into the array type. 466 467Decoupling the backend from the array type gives much more flexibility to 468end-users and backend authors. For example, it is possible to: 469 470* override functions not taking arrays as arguments 471* create backends out of source from the array type 472* install multiple backends for the same array type 473 474This decoupling also means that ``uarray`` is not constrained to dispatching 475over array-like types. The backend is free to inspect the entire set of 476function arguments to determine if it can implement the function e.g. ``dtype`` 477parameter dispatching. 478 479Defining backends 480^^^^^^^^^^^^^^^^^ 481 482``uarray`` consists of two main protocols: ``__ua_convert__`` and 483``__ua_function__``, called in that order, along with ``__ua_domain__``. 484``__ua_convert__`` is for conversion and coercion. It has the signature 485``(dispatchables, coerce)``, where ``dispatchables`` is an iterable of 486``ua.Dispatchable`` objects and ``coerce`` is a boolean indicating whether or 487not to force the conversion. ``ua.Dispatchable`` is a simple class consisting 488of three simple values: ``type``, ``value``, and ``coercible``. 489``__ua_convert__`` returns an iterable of the converted values, or 490``NotImplemented`` in the case of failure. 491 492``__ua_function__`` has the signature ``(func, args, kwargs)`` and defines 493the actual implementation of the function. It receives the function and its 494arguments. Returning ``NotImplemented`` will cause a move to the default 495implementation of the function if one exists, and failing that, the next 496backend. 497 498Here is what will happen assuming a ``uarray`` multimethod is called: 499 5001. We canonicalise the arguments so any arguments without a default 501 are placed in ``*args`` and those with one are placed in ``**kwargs``. 5022. We check the list of backends. 503 504 a. If it is empty, we try the default implementation. 505 5063. We check if the backend's ``__ua_convert__`` method exists. If it exists: 507 508 a. We pass it the output of the dispatcher, 509 which is an iterable of ``ua.Dispatchable`` objects. 510 b. We feed this output, along with the arguments, 511 to the argument replacer. ``NotImplemented`` means we move to 3 512 with the next backend. 513 c. We store the replaced arguments as the new arguments. 514 5154. We feed the arguments into ``__ua_function__``, and return the output, and 516 exit if it isn't ``NotImplemented``. 5175. If the default implementation exists, we try it with the current backend. 5186. On failure, we move to 3 with the next backend. If there are no more 519 backends, we move to 7. 5207. We raise a ``ua.BackendNotImplementedError``. 521 522Defining overridable multimethods 523^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 524 525To define an overridable function (a multimethod), one needs a few things: 526 5271. A dispatcher that returns an iterable of ``ua.Dispatchable`` objects. 5282. A reverse dispatcher that replaces dispatchable values with the supplied 529 ones. 5303. A domain. 5314. Optionally, a default implementation, which can be provided in terms of 532 other multimethods. 533 534As an example, consider the following:: 535 536 import uarray as ua 537 538 def full_argreplacer(args, kwargs, dispatchables): 539 def full(shape, fill_value, dtype=None, order='C'): 540 return (shape, fill_value), dict( 541 dtype=dispatchables[0], 542 order=order 543 ) 544 545 return full(*args, **kwargs) 546 547 @ua.create_multimethod(full_argreplacer, domain="numpy") 548 def full(shape, fill_value, dtype=None, order='C'): 549 return (ua.Dispatchable(dtype, np.dtype),) 550 551A large set of examples can be found in the ``unumpy`` repository, [8]_. 552This simple act of overriding callables allows us to override: 553 554* Methods 555* Properties, via ``fget`` and ``fset`` 556* Entire objects, via ``__get__``. 557 558Examples for NumPy 559^^^^^^^^^^^^^^^^^^ 560 561A library that implements a NumPy-like API will use it in the following 562manner (as an example):: 563 564 import numpy.overridable as unp 565 _ua_implementations = {} 566 567 __ua_domain__ = "numpy" 568 569 def __ua_function__(func, args, kwargs): 570 fn = _ua_implementations.get(func, None) 571 return fn(*args, **kwargs) if fn is not None else NotImplemented 572 573 def implements(ua_func): 574 def inner(func): 575 _ua_implementations[ua_func] = func 576 return func 577 578 return inner 579 580 @implements(unp.asarray) 581 def asarray(a, dtype=None, order=None): 582 # Code here 583 # Either this method or __ua_convert__ must 584 # return NotImplemented for unsupported types, 585 # Or they shouldn't be marked as dispatchable. 586 587 # Provides a default implementation for ones and zeros. 588 @implements(unp.full) 589 def full(shape, fill_value, dtype=None, order='C'): 590 # Code here 591 592Alternatives 593------------ 594 595The current alternative to this problem is a combination of NEP-18 [2]_, 596NEP-13 [4]_ and NEP-30 [9]_ plus adding more protocols (not yet specified) 597in addition to it. Even then, some parts of the NumPy API will remain 598non-overridable, so it's a partial alternative. 599 600The main alternative to vendoring ``unumpy`` is to simply move it into NumPy 601completely and not distribute it as a separate package. This would also achieve 602the proposed goals, however we prefer to keep it a separate package for now, 603for reasons already stated above. 604 605The third alternative is to move ``unumpy`` into the NumPy organisation and 606develop it as a NumPy project. This will also achieve the said goals, and is 607also a possibility that can be considered by this NEP. However, the act of 608doing an extra ``pip install`` or ``conda install`` may discourage some users 609from adopting this method. 610 611An alternative to requiring opt-in is mainly to *not* override ``np.asarray`` 612and ``np.array``, and making the rest of the NumPy API surface overridable, 613instead providing ``np.duckarray`` and ``np.asduckarray`` 614as duck-array friendly alternatives that used the respective overrides. However, 615this has the downside of adding a minor overhead to NumPy calls. 616 617Discussion 618---------- 619 620* ``uarray`` blogpost: https://labs.quansight.org/blog/2019/07/uarray-update-api-changes-overhead-and-comparison-to-__array_function__/ 621* The discussion section of NEP-18: https://numpy.org/neps/nep-0018-array-function-protocol.html#discussion 622* NEP-22: https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html 623* Dask issue #4462: https://github.com/dask/dask/issues/4462 624* PR #13046: https://github.com/numpy/numpy/pull/13046 625* Dask issue #4883: https://github.com/dask/dask/issues/4883 626* Issue #13831: https://github.com/numpy/numpy/issues/13831 627* Discussion PR 1: https://github.com/hameerabbasi/numpy/pull/3 628* Discussion PR 2: https://github.com/hameerabbasi/numpy/pull/4 629* Discussion PR 3: https://github.com/numpy/numpy/pull/14389 630 631 632References and Footnotes 633------------------------ 634 635.. [1] uarray, A general dispatch mechanism for Python: https://uarray.readthedocs.io 636 637.. [2] NEP 18 — A dispatch mechanism for NumPy’s high level array functions: https://numpy.org/neps/nep-0018-array-function-protocol.html 638 639.. [3] NEP 22 — Duck typing for NumPy arrays – high level overview: https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html 640 641.. [4] NEP 13 — A Mechanism for Overriding Ufuncs: https://numpy.org/neps/nep-0013-ufunc-overrides.html 642 643.. [5] Reply to Adding to the non-dispatched implementation of NumPy methods: http://numpy-discussion.10968.n7.nabble.com/Adding-to-the-non-dispatched-implementation-of-NumPy-methods-tp46816p46874.html 644 645.. [6] Custom Dtype/Units discussion: http://numpy-discussion.10968.n7.nabble.com/Custom-Dtype-Units-discussion-td43262.html 646 647.. [7] The epic dtype cleanup plan: https://github.com/numpy/numpy/issues/2899 648 649.. [8] unumpy: NumPy, but implementation-independent: https://unumpy.readthedocs.io 650 651.. [9] NEP 30 — Duck Typing for NumPy Arrays - Implementation: https://www.numpy.org/neps/nep-0030-duck-array-protocol.html 652 653.. [10] http://scipy.github.io/devdocs/fft.html#backend-control 654 655 656Copyright 657--------- 658 659This document has been placed in the public domain. 660