1.. highlight:: cython 2 3.. _external-C-code: 4 5********************************** 6Interfacing with External C Code 7********************************** 8 9One of the main uses of Cython is wrapping existing libraries of C code. This 10is achieved by using external declarations to declare the C functions and 11variables from the library that you want to use. 12 13You can also use public declarations to make C functions and variables defined 14in a Cython module available to external C code. The need for this is expected 15to be less frequent, but you might want to do it, for example, if you are 16`embedding Python`_ in another application as a scripting language. Just as a 17Cython module can be used as a bridge to allow Python code to call C code, it 18can also be used to allow C code to call Python code. 19 20.. _embedding Python: https://web.archive.org/web/20120225082358/http://www.freenet.org.nz:80/python/embeddingpyrex/ 21 22External declarations 23======================= 24 25By default, C functions and variables declared at the module level are local 26to the module (i.e. they have the C static storage class). They can also be 27declared extern to specify that they are defined elsewhere, for example,:: 28 29 cdef extern int spam_counter 30 31 cdef extern void order_spam(int tons) 32 33Referencing C header files 34--------------------------- 35 36When you use an extern definition on its own as in the examples above, Cython 37includes a declaration for it in the generated C file. This can cause problems 38if the declaration doesn't exactly match the declaration that will be seen by 39other C code. If you're wrapping an existing C library, for example, it's 40important that the generated C code is compiled with exactly the same 41declarations as the rest of the library. 42 43To achieve this, you can tell Cython that the declarations are to be found in a 44C header file, like this:: 45 46 cdef extern from "spam.h": 47 48 int spam_counter 49 50 void order_spam(int tons) 51 52The ``cdef extern`` from clause does three things: 53 541. It directs Cython to place a ``#include`` statement for the named header file in 55 the generated C code. 562. It prevents Cython from generating any C code 57 for the declarations found in the associated block. 583. It treats all declarations within the block as though they started with 59 ``cdef extern``. 60 61It's important to understand that Cython does not itself read the C header 62file, so you still need to provide Cython versions of any declarations from it 63that you use. However, the Cython declarations don't always have to exactly 64match the C ones, and in some cases they shouldn't or can't. In particular: 65 66#. Leave out any platform-specific extensions to C declarations such as 67 ``__declspec()``. 68 69#. If the header file declares a big struct and you only want to use a few 70 members, you only need to declare the members you're interested in. Leaving 71 the rest out doesn't do any harm, because the C compiler will use the full 72 definition from the header file. 73 74 In some cases, you might not need any of the struct's members, in which 75 case you can just put pass in the body of the struct declaration, e.g.:: 76 77 cdef extern from "foo.h": 78 struct spam: 79 pass 80 81 .. note:: 82 83 you can only do this inside a ``cdef extern from`` block; struct 84 declarations anywhere else must be non-empty. 85 86#. If the header file uses ``typedef`` names such as :c:type:`word` to refer 87 to platform-dependent flavours of numeric types, you will need a 88 corresponding :keyword:`ctypedef` statement, but you don't need to match 89 the type exactly, just use something of the right general kind (int, float, 90 etc). For example,:: 91 92 ctypedef int word 93 94 will work okay whatever the actual size of a :c:type:`word` is (provided the header 95 file defines it correctly). Conversion to and from Python types, if any, will also 96 be used for this new type. 97 98#. If the header file uses macros to define constants, translate them into a 99 normal external variable declaration. You can also declare them as an 100 :keyword:`enum` if they contain normal :c:type:`int` values. Note that 101 Cython considers :keyword:`enum` to be equivalent to :c:type:`int`, so do 102 not do this for non-int values. 103 104#. If the header file defines a function using a macro, declare it as though 105 it were an ordinary function, with appropriate argument and result types. 106 107#. For archaic reasons C uses the keyword ``void`` to declare a function 108 taking no parameters. In Cython as in Python, simply declare such functions 109 as :meth:`foo()`. 110 111A few more tricks and tips: 112 113* If you want to include a C header because it's needed by another header, but 114 don't want to use any declarations from it, put pass in the extern-from 115 block:: 116 117 cdef extern from "spam.h": 118 pass 119 120* If you want to include a system header, put angle brackets inside the quotes:: 121 122 cdef extern from "<sysheader.h>": 123 ... 124 125* If you want to include some external declarations, but don't want to specify 126 a header file (because it's included by some other header that you've 127 already included) you can put ``*`` in place of the header file name:: 128 129 cdef extern from *: 130 ... 131 132* If a ``cdef extern from "inc.h"`` block is not empty and contains only 133 function or variable declarations (and no type declarations of any kind), 134 Cython will put the ``#include "inc.h"`` statement after all 135 declarations generated by Cython. This means that the included file 136 has access to the variables, functions, structures, ... which are 137 declared by Cython. 138 139Implementing functions in C 140--------------------------- 141 142When you want to call C code from a Cython module, usually that code 143will be in some external library that you link your extension against. 144However, you can also directly compile C (or C++) code as part of your 145Cython module. In the ``.pyx`` file, you can put something like:: 146 147 cdef extern from "spam.c": 148 void order_spam(int tons) 149 150Cython will assume that the function ``order_spam()`` is defined in the 151file ``spam.c``. If you also want to cimport this function from another 152module, it must be declared (not extern!) in the ``.pxd`` file:: 153 154 cdef void order_spam(int tons) 155 156For this to work, the signature of ``order_spam()`` in ``spam.c`` must 157match the signature that Cython uses, in particular the function must 158be static: 159 160.. code-block:: c 161 162 static void order_spam(int tons) 163 { 164 printf("Ordered %i tons of spam!\n", tons); 165 } 166 167 168.. _struct-union-enum-styles: 169 170Styles of struct, union and enum declaration 171---------------------------------------------- 172 173There are two main ways that structs, unions and enums can be declared in C 174header files: using a tag name, or using a typedef. There are also some 175variations based on various combinations of these. 176 177It's important to make the Cython declarations match the style used in the 178header file, so that Cython can emit the right sort of references to the type 179in the code it generates. To make this possible, Cython provides two different 180syntaxes for declaring a struct, union or enum type. The style introduced 181above corresponds to the use of a tag name. To get the other style, you prefix 182the declaration with :keyword:`ctypedef`, as illustrated below. 183 184The following table shows the various possible styles that can be found in a 185header file, and the corresponding Cython declaration that you should put in 186the ``cdef extern`` from block. Struct declarations are used as an example; the 187same applies equally to union and enum declarations. 188 189+-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ 190| C code | Possibilities for corresponding Cython Code | Comments | 191+=========================+=============================================+=======================================================================+ 192| .. code-block:: c | :: | Cython will refer to the as ``struct Foo`` in the generated C code. | 193| | | | 194| struct Foo { | cdef struct Foo: | | 195| ... | ... | | 196| }; | | | 197+-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ 198| .. code-block:: c | :: | Cython will refer to the type simply as ``Foo`` in | 199| | | the generated C code. | 200| typedef struct { | ctypedef struct Foo: | | 201| ... | ... | | 202| } Foo; | | | 203+-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ 204| .. code-block:: c | :: | If the C header uses both a tag and a typedef with *different* | 205| | | names, you can use either form of declaration in Cython | 206| typedef struct foo { | cdef struct foo: | (although if you need to forward reference the type, | 207| ... | ... | you'll have to use the first form). | 208| } Foo; | ctypedef foo Foo #optional | | 209| | | | 210| | or:: | | 211| | | | 212| | ctypedef struct Foo: | | 213| | ... | | 214+-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ 215| .. code-block:: c | :: | If the header uses the *same* name for the tag and typedef, you | 216| | | won't be able to include a :keyword:`ctypedef` for it -- but then, | 217| typedef struct Foo { | cdef struct Foo: | it's not necessary. | 218| ... | ... | | 219| } Foo; | | | 220+-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ 221 222See also use of :ref:`external_extension_types`. 223Note that in all the cases below, you refer to the type in Cython code simply 224as :c:type:`Foo`, not ``struct Foo``. 225 226Pointers 227-------- 228When interacting with a C-api there may be functions that require pointers as arguments. 229Pointers are variables that contain a memory address to another variable. 230 231For example:: 232 233 cdef extern from "<my_lib.h>": 234 cdef void increase_by_one(int *my_var) 235 236This function takes a pointer to an integer as argument. Knowing the address of the 237integer allows the function to modify the value in place, so that the caller can see 238the changes afterwards. In order to get the address from an existing variable, 239use the ``&`` operator:: 240 241 cdef int some_int = 42 242 cdef int *some_int_pointer = &some_int 243 increase_by_one(some_int_pointer) 244 # Or without creating the extra variable 245 increase_by_one(&some_int) 246 print(some_int) # prints 44 (== 42+1+1) 247 248If you want to manipulate the variable the pointer points to, you can access it by 249referencing its first element like you would in python ``my_pointer[0]``. For example:: 250 251 cdef void increase_by_one(int *my_var): 252 my_var[0] += 1 253 254For a deeper introduction to pointers, you can read `this tutorial at tutorialspoint 255<https://www.tutorialspoint.com/cprogramming/c_pointers.htm>`_. For differences between 256Cython and C syntax for manipulating pointers, see :ref:`statements_and_expressions`. 257 258Accessing Python/C API routines 259--------------------------------- 260 261One particular use of the ``cdef extern from`` statement is for gaining access to 262routines in the Python/C API. For example,:: 263 264 cdef extern from "Python.h": 265 266 object PyString_FromStringAndSize(char *s, Py_ssize_t len) 267 268will allow you to create Python strings containing null bytes. 269 270Note that Cython comes with ready-to-use declarations of (almost) all C-API functions 271in the cimportable ``cpython.*`` modules. See the list in 272https://github.com/cython/cython/tree/master/Cython/Includes/cpython 273 274Special Types 275-------------- 276 277Cython predefines the name ``Py_ssize_t`` for use with Python/C API routines. To 278make your extensions compatible with 64-bit systems, you should always use 279this type where it is specified in the documentation of Python/C API routines. 280 281Windows Calling Conventions 282---------------------------- 283 284The ``__stdcall`` and ``__cdecl`` calling convention specifiers can be used in 285Cython, with the same syntax as used by C compilers on Windows, for example,:: 286 287 cdef extern int __stdcall FrobnicateWindow(long handle) 288 289 cdef void (__stdcall *callback)(void *) 290 291If ``__stdcall`` is used, the function is only considered compatible with 292other ``__stdcall`` functions of the same signature. 293 294 295.. _resolve-conflicts: 296 297Resolving naming conflicts - C name specifications 298-------------------------------------------------- 299 300Each Cython module has a single module-level namespace for both Python and C 301names. This can be inconvenient if you want to wrap some external C functions 302and provide the Python user with Python functions of the same names. 303 304Cython provides a couple of different ways of solving this problem. The best 305way, especially if you have many C functions to wrap, is to put the extern 306C function declarations into a ``.pxd`` file and thus a different namespace, 307using the facilities described in :ref:`sharing declarations between Cython 308modules <sharing-declarations>`. Writing them into a ``.pxd`` file allows 309their reuse across modules, avoids naming collisions in the normal Python way 310and even makes it easy to rename them on cimport. For example, if your 311``decl.pxd`` file declared a C function ``eject_tomato``:: 312 313 cdef extern from "myheader.h": 314 void eject_tomato(float speed) 315 316then you can cimport and wrap it in a ``.pyx`` file as follows:: 317 318 from decl cimport eject_tomato as c_eject_tomato 319 320 def eject_tomato(speed): 321 c_eject_tomato(speed) 322 323or simply cimport the ``.pxd`` file and use it as prefix:: 324 325 cimport decl 326 327 def eject_tomato(speed): 328 decl.eject_tomato(speed) 329 330Note that this has no runtime lookup overhead, as it would in Python. 331Cython resolves the names in the ``.pxd`` file at compile time. 332 333For special cases where namespacing or renaming on import is not enough, 334e.g. when a name in C conflicts with a Python keyword, you can use a C name 335specification to give different Cython and C names to the C function at 336declaration time. Suppose, for example, that you want to wrap an external 337C function called :func:`yield`. If you declare it as:: 338 339 cdef extern from "myheader.h": 340 void c_yield "yield" (float speed) 341 342then its Cython visible name will be ``c_yield``, whereas its name in C 343will be ``yield``. You can then wrap it with:: 344 345 def call_yield(speed): 346 c_yield(speed) 347 348As for functions, C names can be specified for variables, structs, unions, 349enums, struct and union members, and enum values. For example:: 350 351 cdef extern int one "eins", two "zwei" 352 cdef extern float three "drei" 353 354 cdef struct spam "SPAM": 355 int i "eye" 356 357 cdef enum surprise "inquisition": 358 first "alpha" 359 second "beta" = 3 360 361Note that Cython will not do any validation or name mangling on the string 362you provide. It will inject the bare text into the C code unmodified, so you 363are entirely on your own with this feature. If you want to declare a name 364``xyz`` and have Cython inject the text "make the C compiler fail here" into 365the C file for it, you can do this using a C name declaration. Consider this 366an advanced feature, only for the rare cases where everything else fails. 367 368 369.. _verbatim_c: 370 371Including verbatim C code 372------------------------- 373 374For advanced use cases, Cython allows you to directly write C code 375as "docstring" of a ``cdef extern from`` block: 376 377.. literalinclude:: ../../examples/userguide/external_C_code/verbatim_c_code.pyx 378 379The above is essentially equivalent to having the C code in a file 380``header.h`` and writing :: 381 382 cdef extern from "header.h": 383 long square(long x) 384 void assign(long& x, long y) 385 386This feature is commonly used for platform specific adaptations at 387compile time, for example: 388 389.. literalinclude:: ../../examples/userguide/external_C_code/platform_adaptation.pyx 390 391It is also possible to combine a header file and verbatim C code:: 392 393 cdef extern from "badheader.h": 394 """ 395 /* This macro breaks stuff */ 396 #undef int 397 """ 398 # Stuff from badheader.h 399 400In this case, the C code ``#undef int`` is put right after 401``#include "badheader.h"`` in the C code generated by Cython. 402 403Note that the string is parsed like any other docstring in Python. 404If you require character escapes to be passed into the C code file, 405use a raw docstring, i.e. ``r""" ... """``. 406 407 408Using Cython Declarations from C 409================================ 410 411Cython provides two methods for making C declarations from a Cython module 412available for use by external C code---public declarations and C API 413declarations. 414 415.. note:: 416 417 You do not need to use either of these to make declarations from one 418 Cython module available to another Cython module – you should use the 419 :keyword:`cimport` statement for that. Sharing Declarations Between Cython Modules. 420 421Public Declarations 422--------------------- 423 424You can make C types, variables and functions defined in a Cython module 425accessible to C code that is linked together with the Cython-generated C file, 426by declaring them with the public keyword:: 427 428 cdef public struct Bunny: # a public type declaration 429 int vorpalness 430 431 cdef public int spam # a public variable declaration 432 433 cdef public void grail(Bunny *) # a public function declaration 434 435If there are any public declarations in a Cython module, a header file called 436:file:`modulename.h` file is generated containing equivalent C declarations for 437inclusion in other C code. 438 439A typical use case for this is building an extension module from multiple 440C sources, one of them being Cython generated (i.e. with something like 441``Extension("grail", sources=["grail.pyx", "grail_helper.c"])`` in ``setup.py``. 442In this case, the file ``grail_helper.c`` just needs to add 443``#include "grail.h"`` in order to access the public Cython variables. 444 445A more advanced use case is embedding Python in C using Cython. 446In this case, make sure to call Py_Initialize() and Py_Finalize(). 447For example, in the following snippet that includes :file:`grail.h`: 448 449.. code-block:: c 450 451 #include <Python.h> 452 #include "grail.h" 453 454 int main() { 455 Py_Initialize(); 456 initgrail(); /* Python 2.x only ! */ 457 Bunny b; 458 grail(b); 459 Py_Finalize(); 460 } 461 462This C code can then be built together with the Cython-generated C code 463in a single program (or library). 464 465In Python 3.x, calling the module init function directly should be avoided. Instead, 466use the `inittab mechanism <https://docs.python.org/3/c-api/import.html#c._inittab>`_ 467to link Cython modules into a single shared library or program. 468 469.. code-block:: c 470 471 err = PyImport_AppendInittab("grail", PyInit_grail); 472 Py_Initialize(); 473 grail_module = PyImport_ImportModule("grail"); 474 475If the Cython module resides within a package, then the name of the ``.h`` 476file consists of the full dotted name of the module, e.g. a module called 477:mod:`foo.spam` would have a header file called :file:`foo.spam.h`. 478 479.. NOTE:: 480 481 On some operating systems like Linux, it is also possible to first 482 build the Cython extension in the usual way and then link against 483 the resulting ``.so`` file like a dynamic library. 484 Beware that this is not portable, so it should be avoided. 485 486.. _api: 487 488C API Declarations 489------------------- 490 491The other way of making declarations available to C code is to declare them 492with the :keyword:`api` keyword. You can use this keyword with C functions and 493extension types. A header file called :file:`modulename_api.h` is produced 494containing declarations of the functions and extension types, and a function 495called :func:`import_modulename`. 496 497C code wanting to use these functions or extension types needs to include the 498header and call the :func:`import_modulename` function. The other functions 499can then be called and the extension types used as usual. 500 501If the C code wanting to use these functions is part of more than one shared 502library or executable, then :func:`import_modulename` function needs to be 503called in each of the shared libraries which use these functions. If you 504crash with a segmentation fault (SIGSEGV on linux) when calling into one of 505these api calls, this is likely an indication that the shared library which 506contains the api call which is generating the segmentation fault does not call 507the :func:`import_modulename` function before the api call which crashes. 508 509Any public C type or extension type declarations in the Cython module are also 510made available when you include :file:`modulename_api.h`.: 511 512.. literalinclude:: ../../examples/userguide/external_C_code/delorean.pyx 513 514.. literalinclude:: ../../examples/userguide/external_C_code/marty.c 515 :language: C 516 517.. note:: 518 519 Any types defined in the Cython module that are used as argument or 520 return types of the exported functions will need to be declared public, 521 otherwise they won't be included in the generated header file, and you will 522 get errors when you try to compile a C file that uses the header. 523 524Using the :keyword:`api` method does not require the C code using the 525declarations to be linked with the extension module in any way, as the Python 526import machinery is used to make the connection dynamically. However, only 527functions can be accessed this way, not variables. Note also that for the 528module import mechanism to be set up correctly, the user must call 529Py_Initialize() and Py_Finalize(); if you experience a segmentation fault in 530the call to :func:`import_modulename`, it is likely that this wasn't done. 531 532You can use both :keyword:`public` and :keyword:`api` on the same function to 533make it available by both methods, e.g.:: 534 535 cdef public api void belt_and_braces() except *: 536 ... 537 538However, note that you should include either :file:`modulename.h` or 539:file:`modulename_api.h` in a given C file, not both, otherwise you may get 540conflicting dual definitions. 541 542If the Cython module resides within a package, then: 543 544* The name of the header file contains of the full dotted name of the module. 545* The name of the importing function contains the full name with dots replaced 546 by double underscores. 547 548E.g. a module called :mod:`foo.spam` would have an API header file called 549:file:`foo.spam_api.h` and an importing function called 550:func:`import_foo__spam`. 551 552Multiple public and API declarations 553-------------------------------------- 554 555You can declare a whole group of items as :keyword:`public` and/or 556:keyword:`api` all at once by enclosing them in a :keyword:`cdef` block, for 557example,:: 558 559 cdef public api: 560 void order_spam(int tons) except * 561 char *get_lunch(float tomato_size) except NULL 562 563This can be a useful thing to do in a ``.pxd`` file (see 564:ref:`sharing-declarations`) to make the module's public interface 565available by all three methods. 566 567Acquiring and Releasing the GIL 568--------------------------------- 569 570Cython provides facilities for acquiring and releasing the 571`Global Interpreter Lock (GIL) <https://docs.python.org/dev/glossary.html#term-global-interpreter-lock>`_. 572This may be useful when calling from multi-threaded code into 573(external C) code that may block, or when wanting to use Python 574from a (native) C thread callback. Releasing the GIL should 575obviously only be done for thread-safe code or for code that 576uses other means of protection against race conditions and 577concurrency issues. 578 579Note that acquiring the GIL is a blocking thread-synchronising 580operation, and therefore potentially costly. It might not be 581worth releasing the GIL for minor calculations. Usually, I/O 582operations and substantial computations in parallel code will 583benefit from it. 584 585.. _nogil: 586 587Releasing the GIL 588^^^^^^^^^^^^^^^^^ 589 590You can release the GIL around a section of code using the 591``with nogil`` statement:: 592 593 with nogil: 594 <code to be executed with the GIL released> 595 596Code in the body of the with-statement must not manipulate Python objects 597in any way, and must not call anything that manipulates Python objects without 598first re-acquiring the GIL. Cython validates these operations at compile time, 599but cannot look into external C functions, for example. They must be correctly 600declared as requiring or not requiring the GIL (see below) in order to make 601Cython's checks effective. 602 603Since Cython 3.0, some simple Python statements can be used inside of ``nogil`` 604sections: ``raise``, ``assert`` and ``print`` (the Py2 statement, not the function). 605Since they tend to be lone Python statements, Cython will automatically acquire 606and release the GIL around them for convenience. 607 608.. _gil: 609 610Acquiring the GIL 611^^^^^^^^^^^^^^^^^ 612 613A C function that is to be used as a callback from C code that is executed 614without the GIL needs to acquire the GIL before it can manipulate Python 615objects. This can be done by specifying ``with gil`` in the function 616header:: 617 618 cdef void my_callback(void *data) with gil: 619 ... 620 621If the callback may be called from another non-Python thread, 622care must be taken to initialize the GIL first, through a call to 623`PyEval_InitThreads() <https://docs.python.org/dev/c-api/init.html#c.PyEval_InitThreads>`_. 624If you're already using :ref:`cython.parallel <parallel>` in your module, this will already have been taken care of. 625 626The GIL may also be acquired through the ``with gil`` statement:: 627 628 with gil: 629 <execute this block with the GIL acquired> 630 631.. _gil_conditional: 632 633Conditional Acquiring / Releasing the GIL 634^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 635Sometimes it is helpful to use a condition to decide whether to run a 636certain piece of code with or without the GIL. This code would run anyway, 637the difference is whether the GIL will be held or released. 638The condition must be constant (at compile time). 639 640This could be useful for profiling, debugging, performance testing, and 641for fused types (see :ref:`fused_gil_conditional`).:: 642 643 DEF FREE_GIL = True 644 645 with nogil(FREE_GIL): 646 <code to be executed with the GIL released> 647 648 with gil(False): 649 <GIL is still released> 650 651Declaring a function as callable without the GIL 652-------------------------------------------------- 653 654You can specify :keyword:`nogil` in a C function header or function type to 655declare that it is safe to call without the GIL.:: 656 657 cdef void my_gil_free_func(int spam) nogil: 658 ... 659 660When you implement such a function in Cython, it cannot have any Python 661arguments or Python object return type. Furthermore, any operation 662that involves Python objects (including calling Python functions) must 663explicitly acquire the GIL first, e.g. by using a ``with gil`` block 664or by calling a function that has been defined ``with gil``. These 665restrictions are checked by Cython and you will get a compile error 666if it finds any Python interaction inside of a ``nogil`` code section. 667 668.. NOTE:: The ``nogil`` function annotation declares that it is safe 669 to call the function without the GIL. It is perfectly allowed 670 to execute it while holding the GIL. The function does not in 671 itself release the GIL if it is held by the caller. 672 673Declaring a function ``with gil`` (i.e. as acquiring the GIL on entry) also 674implicitly makes its signature :keyword:`nogil`. 675