1User Manual 2########### 3 4So you want to write algorithms that are easy to maintain as in Python and 5you want performance as in FORTRAN or C++? Lets give a try to Pythran! 6Pythran is a Python-to-c++ translator that turns Python modules into native 7c++11 modules. From a user point of view, you still ``import`` your module, but 8under the hood... There is much more happening! 9 10Disclaimer 11---------- 12 13Pythran is *not* a full Python-to-c++ converter, as is *shedskin*. Instead it 14takes a subset of the Python language and turns it into heavily templatized c++ 15code instantiated for your particular types. 16 17Say hello to: 18 19- polymorphic functions (!) 20- lambdas 21- list comprehension 22- map, reduce and the like 23- dictionary, set, list 24- exceptions 25- file handling 26- (partial) `numpy` support 27 28Say bye bye to: 29 30- classes 31- polymorphic variables [ but not all of them :-) ] 32 33In a nutshell, Pythran makes it possible to write numerical algorithms in 34Python and to have them run faster. Nuff said. 35 36 37Prerequisite 38------------ 39 40Pythran depends on the following packages: 41 42.. include:: ../requirements.txt 43 :literal: 44 45You also need a modern C++11 enabled compiler (e.g. g++>=5, clang>=3.5), that supports 46atomic operations (N3290) and variadic template (N2555). 47 48 49Installation from Sources 50------------------------- 51 52The prefered way to install Pythran is using ``pip install pythran`` or 53``conda install pythran``. Yet if you want to install from sources, 54here is the procedure. 55 56First get the sources:: 57 58 $> git clone https://github.com/serge-sans-paille/pythran 59 60From the source directory, run:: 61 62 $> pip install . 63 64``pythran`` should now be on your ``PATH``. If not, it's possible ``pip`` 65installed to ``.local`` (this happens if the default ``site-packages`` 66location requires elevated permissions) - fix this by setting your path to:: 67 68 $> export PATH=$PATH:$HOME/.local/bin 69 70It makes the ``pythran`` command available to you. 71 72Making Sure Everything is working 73--------------------------------- 74 75The ``setup.py`` scripts automates this. The ``test`` target, as in:: 76 77 $> python setup.py test 78 79runs a whole (and long) validation suite (you will need to install the 80``pytest`` module first to use it). 81 82If these tests fail, you are likely missing some of the requirements. You can 83set site specific flags in your ``~/.pythranrc``, read the doc a bit further! 84 85First Steps 86----------- 87 88To begin with, you need... a Python function in a module. Something like:: 89 90 <<dprod.py>> 91 def dprod(arr0, arr1): 92 return sum([x*y for x,y in zip(arr0, arr1)]) 93 94is perfect. But due to ``\_o<`` typing, ``arr0`` and ``arr1`` can be of any type, 95so Pythran needs a small hint there. Add the following line somewhere in your 96file, say at the top head, or right before the function definition:: 97 98 #pythran export dprod(int list, int list) 99 100This basically tells Pythran the type of the forthcoming arguments. 101 102 103Afterwards, frenetically type:: 104 105 $> pythran dprod.py 106 107``\o/`` a ``dprod.so`` native module has been created and you can play with it 108right *now*, as if it where a normal module:: 109 110 >>> import dprod # this imports the native version if available 111 >>> dprod.dprod([1,2], [3,4]) 112 11 113 114The speedup will not be terrific because of the conversion cost 115from Python to C++. 116 117So let's try again with a well-known example. Let me 118introduce the almighty *matrix multiply*!:: 119 120 <<mm.py>> 121 def zero(n,m): return [[0]*n for col in range(m)] 122 def matrix_multiply(m0, m1): 123 new_matrix = zero(len(m0),len(m1[0])) 124 for i in range(len(m0)): 125 for j in range(len(m1[0])): 126 for k in range(len(m1)): 127 new_matrix[i][j] += m0[i][k]*m1[k][j] 128 return new_matrix 129 130This is a slightly more complex example, as a few intrinsics such as ``range`` and 131``len`` are used, with a function call and even nested list comprehension. But 132Pythran can make its way through this. As you only want to export the 133``matrix_multiply`` function, you can safely ignore the ``zero`` function and 134just add:: 135 136 #pythran export matrix_multiply(float list list, float list list) 137 138to the source file. Note how Pythran can combine different types and infer the 139resulting type. It also respects the nested list structure of Python, so you 140are not limited to matrices... 141 142Enough talk, run:: 143 144 $> pythran mm.py 145 146One touch of magic wand and you have your native binary. Be amazed by the 147generation of a ``mm.so`` native module that runs around 20x faster than the 148original one. ``timeit`` approved! 149 150But scientific computing in Python usually means Numpy. Here is a well-known Numpy snippet:: 151 152 <<arc_distance.py>> 153 import numpy as np 154 def arc_distance(theta_1, phi_1, theta_2, phi_2): 155 """ 156 Calculates the pairwise arc distance 157 between all points in vector a and b. 158 """ 159 temp = (np.sin((theta_2-theta_1)/2)**2 160 + np.cos(theta_1)*np.cos(theta_2) * np.sin((phi_2-phi_1)/2)**2) 161 distance_matrix = 2 * np.arctan2(np.sqrt(temp), np.sqrt(1-temp)) 162 return distance_matrix 163 164This example uses a lot of Numpy `ufunc`. Pythran is reasonably good at 165handling such expressions. As you already know, you need to **export** it, giving its 166argument types by adding:: 167 168 #pythran export arc_distance(float[], float[], float[], float[]) 169 170To the input file. You can compile it as the previous code:: 171 172 $> pythran arc_distance.py 173 174and you'll get a decent binary. But what you really want to do is:: 175 176 $> pythran -DUSE_XSIMD -fopenmp -march=native arc_distance.py 177 178which basically tells the compiler to parallelize and vectorize loops using 179whatever hardware available on your machine. Then you'll get **really** fast 180code! 181 182 183Concerning Pythran specifications 184--------------------------------- 185 186The ``#pythran export`` commands are critical to Pythran. In fact if they are 187missing, Pythran will complain loudly (and fail miserably). So let us dive into 188this complex language! 189 190There is currently only one Pythran command, the ``export`` command. Its syntax is:: 191 192 #pythran export function_name(argument_type* [, argument_type ? *]) 193 194where ``function_name`` is the name of a function defined in the module, and 195``argument_type*`` is a comma separated list of argument types, composed of any 196combination of basic types and constructed types. ``argument_type ? *`` is a 197comma separated list of optional argument types, similar to ``argument_type`` 198but followed by a ``?``. 199 200What is an ``argument_type``? Anything that looks like a Python basic type! 201Constructed types are either tuples, introduced by parenthesis, like ``(int, 202(float, str))`` or lists (resp. set), introduced by the ``list`` (resp. 203``set``) keyword:: 204 205 argument_type = basic_type 206 | (argument_type+) # this is a tuple 207 | argument_type list # this is a list 208 | argument_type set # this is a set 209 | argument_type []+ # this is a ndarray, C-style 210 | argument_type [::]+ # this is a strided ndarray 211 | argument_type [:,...,:]+ # this is a ndarray, Cython style 212 | argument_type [:,...,3]+ # this is a ndarray, some dimension fixed 213 | argument_type:argument_type dict # this is a dictionary 214 215 basic_type = bool | byte | int | float | str | None | slice 216 | uint8 | uint16 | uint32 | uint64 | uintp 217 | int8 | int16 | int32 | int64 | intp 218 | float32 | float64 | float128 219 | complex64 | complex128 | complex256 220 221.. note:: 222 223 When using a 2D array, overloads of the function involved are created to accept both C-style and Fortran-style arrays. 224 To avoid generating too many functions, one can force the memory layout using ``order(C)`` or ``order(F)`` after the 225 array decalaration, as in ``int[:,:] order(C)``. 226 227The same syntax can be used to export global variable (in read only mode):: 228 229 #pythran export var_name 230 231 232In a similar manner to the Python import statement, it's possible to chain the export, as in:: 233 234 #pythran export var_name0, var_name1, function_name(argument_type0) 235 236Multiple overloads can be specified for the same Python function:: 237 238 #pythran export function_name(argument_type0) 239 #pythran export function_name(argument_type1) 240 241In the case of function with default parameters, you can either omit the 242parameter, and in that case it uses the default one, or explicitly state it's 243argument type:: 244 245 #pythran export function_name() 246 #pythran export function_name(argument_type0) 247 #pythran export function_name(argument_type0, argument_type1) 248 def function_name(a0=1, a1=True): 249 pass 250 251When specifying multiple overloads, instead of listing them, you can use the ``or`` operator to list the alternatives, as in:: 252 253 #pythran export function_name(type0 or type1, type2, type3 or type4) 254 255which is exactly equivalent to:: 256 257 #pythran export function_name(type0, type2, type3) 258 #pythran export function_name(type0, type2, type4) 259 #pythran export function_name(type1, type2, type3) 260 #pythran export function_name(type1, type2, type4) 261 262Easy enough, isn't it? 263 264 265 266.. note:: 267 268 Multiline exports are supported, just use comments to protect them, as in:: 269 270 #pythran export river_boa(int, 271 # float, 272 # bool) 273 274.. note:: 275 276 It is in fact possible to analyse a code without specifications, but you 277 cannot go further that generic (a.k.a. heavily templated) c++ code. Use the 278 ``-e`` switch! 279 280 281.pythran files 282************** 283 284Instead of writing the export lines in the ``.py`` file, it is possible to 285write them, **without the #pythran** prefix, inside a file that has the same 286path has the ``.py`` file, but with the ``.pythran`` extension. For instance, 287file ``I_love.py`` can have its export lines in the ``I_love.pythran`` file, using the syntax:: 288 289 export function_name(argument_type*) 290 291Limitations 292*********** 293 294Pythran tries hard to produce code that has the same observable behavior as the original Python code. 295Unfortunately it's not always possible: 296 297- Pythran does not support heterogeneous containers (except tuples). 298 299- There is no BigInt support. All integer operations are performed on 300 ``np.int_``, which maps to C ``long`` type. Beware that as a consequence, the 301 size of this type is system-dependent. 302 303- In most cases (with the notable exception of ``numpy.ndarray``), Pythran is 304 working on a deep copy of the original Python arguments. This copy shares no memory 305 relationship with the original object, which means that modifying the 306 argument content in Pythran won't modify the original argument content. 307 Likewise, objects generated by Pythran cannot share reference (in the sense 308 of ``is``) with one of the input argument. 309 Of curse, this limitation doesn't apply to non exported functions. 310 311 312GIL Interaction 313--------------- 314 315As Pythran translates the Python code in native code that only depends on 316``libpython`` for data translation, it can release the GIL during the actual 317function run. And that's what it does :-) Put an another way, you can rip some 318speedup at the Python level just by spawning multiple ``threading.Thread``. 319 320 321IPython Integration 322------------------- 323 324The magic function ``%%pythran`` is made available to ``ipython`` users through an 325extension. The extension is located in the ``extensions/`` directory 326and can be loaded using IPython's magic function:: 327 328 %load_ext pythran.magic 329 330Once done, you can pythranize your code from the IPython shell:: 331 332 %%pythran 333 #pythran export foo() 334 def foo(): print 'hello' 335 336You can pass arguments to this magic, as in:: 337 338 %%pythran(-O2 -fopenmp) 339 #pythran export foo() 340 def foo(): print 'hello' 341 342 343Distutils Integration 344--------------------- 345 346When distributing a Python application with Pythran modules, you can either: 347 348* declare the module as a regular Python module. After all, they are 100% Python compatible. 349 350* declare them as a ``PythranExtension`` and Pythran will compile them:: 351 352 from distutils.core import setup 353 354 # These two lines are required to be able to use pythran in the setup.py 355 import setuptools 356 setuptools.dist.Distribution(dict(setup_requires='pythran')) 357 358 from pythran.dist import PythranExtension, PythranBuildExt 359 setup(..., 360 ext_modules=[PythranExtension("mymodule", ["mymodule.py"])], 361 cmdclass={"build_ext": PythranBuildExt}) 362 363``PythranBuildExt`` is optional, but necessary to build extensions with 364different C++ compilers. It derives from distuil's ``build_ext`` by default, but 365you can change its base class by using ``PythranBuildExt[base_cls]`` instead. 366 367* all configuration options supported in ``.pythranrc`` can also be passed 368 through the optional ``config`` argument, in the form of a list, e.g. 369 ``config=['compiler.blas=openblas']`` 370 371.. note:: 372 373 There's no strong compatibility guarantee between Pythran version at C++ level. As a 374 consequence, a code distrubuted under pythran version 0.x should depend on that exact 375 version, as version 0.y may introduce some changes. 376 377 This behavior is likely to change with revisions >= 1. 378 379 380Capsule Corp 381------------ 382 383Instead of creating functions that can be used from Python code, Pythran can 384produce native functions to be used by other Python extension, using the 385``Capsule`` mechanism. To do so, just add the ``capsule`` keyword to the export 386line:: 387 388 #pythran export capsule foo(double*, double) 389 390Note that pointer types are only supported within the context of a capsule, as 391they don't match any real Python type. **Any** Pythran type is valid as capsule 392parameter, but beware that non scalar or pointer types only make sense withing 393the Pythran context. 394 395Debug Mode 396---------- 397 398Pythran honors the ``NDEBUG`` macro. If set through ``-DNDEBUG`` (which should 399be the default, check ``python-config --cflags``), it disables all ``assert 400statement`` and doesn't perform any runtime check for indexing bounds etc. 401However, if unset through ``-UNDEBUG``, all ``assert`` are executed and 402eventually raise an ``AssertionError``. Additionnaly, many internal checks are 403done and may fail with a C-ish assertion. 404 405Thread safety 406------------- 407 408By default Pythran does not generate thread-safe code for non-OpenMP code: reference 409counting for automatic deletion of objects is not done atomically by default. It's 410still possible to force pythran to generate thread-safe reference counting by defining 411the flag ``THREAD_SAFE_REF_COUNT`` via ``-DTHREAD_SAFE_REF_COUNT``. There is a small 412performance penalty associated with this. 413 414Advanced Usage 415-------------- 416 417One can use ``-o <filename>`` or ``--output=<filename>`` to control the name of 418the generated file. If ``<filename>`` contains the ``%{ext}`` pattern, it is 419replaced by the extension that matches your current platform. 420 421A failing compilation? A lust for c++ tangled code? Give a try to the ``-E`` 422switch that stops the compilation process right after c++ code generation, so 423that you can inspect it. 424 425Want more performance? Big fan of ``-Ofast -march=native``? Pythran 426_automagically_ forwards these switches to the underlying compiler! 427 428Tired of typing the same compiler switches again and again? Store them in 429``$XDG_CONFIG_HOME/.pythranrc``! 430 431Wants to try your own compiler? Update the ``CC`` and ``CXX`` fields from your 432``pythranrc``, or set the same environment variables to the right compilers. Environment 433variables have greater precedence than configuration file. 434 435Pythran also honors the ``CXXFLAGS`` and ``LDFLAGS`` environment variables. 436 437The careful reader might have noticed the ``-p`` flag from the command line. It 438makes it possible to define your own optimization sequence:: 439 440 pythran -pConstantFolding -pmy_package.MyOptimization 441 442runs the ``ConstantFolding`` optimization from ``pythran.optimizations`` 443followed by a custom optimization found in the ``my_package`` package, loaded 444from ``PYTHONPATH``. 445 446When importing a Python module, one can check for the presence of the 447``__pythran__`` variable at the module scope to see if the module has been 448pythranized:: 449 450 import foo 451 if hasattr(foo, '__pythran__'): 452 print(r'\_o<') 453 454This variable is a tuple that holds three fields: 455 4561. pythran's version 4572. compilation date 4583. sha256 value of the input code 459 460 461Adding OpenMP directives 462------------------------ 463 464OpenMP is a standard set of directives for C, C++ and FORTRAN that makes it 465easier to turn a sequential program into a multi-threaded one. Pythran 466translates OpenMP-like code annotation into OpenMP directives:: 467 468 r=0 469 #omp parallel for reduction(+:r) 470 for x,y in zip(l1,l2): 471 r+=x*y 472 473OpenMP directive parsing is enabled by ``-fopenmp`` when using ``g++`` as the 474back-end compiler. Be careful with the indentation. It has to be correct! 475 476Alternatively, one can run the great:: 477 478 $> pythran -ppythran.analyses.ParallelMaps -e as.py 479 480which runs a code analyzer that displays extra information concerning parallel ``map`` found in the code. 481 482 483Getting Pure C++ 484---------------- 485 486Pythran can be used to generate raw templated C++ code, without any Python 487glue. To do so use the ``-e`` switch. It will turn the Python code into C++ 488code you can call from a C++ program. In that case there is **no** need for a 489particular Pythran specification. 490 491 492Read the optimized Python code 493------------------------------ 494 495Curious Python developers might want to study how Pythran transforms their 496codes. With the ``-P`` switch, Pythran optimizes the Python code, prints the 497result and stops there. Pythran does not care about PEP 8, so a Python 498formatter is often useful:: 499 500 $> pythran -P arc_distance.py | yapf 501 502 503Customizing Your ``.pythranrc`` 504------------------------------- 505 506Pythran checks for a file named ``.pythranrc`` and use it to *replace* the site 507configuration. Here are a few tricks! 508 509You can change the default location of the pythran configuration file using the 510environment variable ``PYTHRANRC``:: 511 512 PYTHRANRC=/opt/company/pythran/config.pythranrc pythran arc_distance.py 513 514All the options in the ``.pythranrc`` file can be specified when running pythran by using the command line argument --config= . 515For example:: 516 517 pythran --config compiler.blas=pythran-openblas this_file.py 518 519would specify that pythran-openblas is the blas library to use. 520 521Options specified using command-line arguments override the options found in the ``.pythranrc`` file 522 523 524``[compiler]`` 525************** 526 527This section contains compiler flags configuration. For education purpose, the default linux configuration is 528 529.. literalinclude:: ../pythran/pythran-linux2.cfg 530 531:``CC``: 532 533 Path to the C compiler to use 534 535:``CXX``: 536 537 Path to the C++ compiler to use 538 539:``defines``: 540 541 Preprocessor definitions. Pythran is sensible to ``USE_XSIMD`` and 542 ``PYTHRAN_OPENMP_MIN_ITERATION_COUNT``. The former turns on `xsimd <https://github.com/QuantStack/xsimd>`_ 543 vectorization and the latter controls the minimal loop trip count to turn a 544 sequential loop into a parallel loop. 545 546:``undefs``: 547 548 Some preprocessor definitions to remove. 549 550:``include_dirs``: 551 552 Additional include directories to search for headers. 553 554:``cflags``: 555 556 Additional random compiler flags (``-f``, ``-O``). Optimization flags generally 557 go there. The default is to set ``-std=c++11`` for C++11 support. 558 559:``libs``: 560 561 Libraries to use during the link process. A typical extension 562 is to add ``tcmalloc_minimal`` to use the allocator from 563 https://code.google.com/p/gperftools/. 564 565:``library_dirs``: 566 567 Extra directories to search for required libraries. 568 569:``ldflags``: 570 571 Additional random linker flags. 572 573:``blas``: 574 575 BLAS library to use. ``none``, ``pythran-openblas``, ``blas``, 576 ``openblas``, ``atlas`` or ``mkl`` are viable choices. 577 ``none`` prevents from linking with blas. 578 ``pythran-openblas`` requires the `pythran-openblas 579 <https://pypi.org/project/pythran-openblas/>`_ package, which provides a 580 statically linked version of `OpenBLAS <https://www.openblas.net/>`_. Other 581 options are system dependant. Depending on your setup, you *may* need to 582 update ``include_dirs`` to point to the location of the BLAS headers, e.g. 583 ``/usr/include/openblas``. 584 585:``ignoreflags``: 586 587 Space-separated list of compiler flags that should not be forwarded to the 588 pythran backend compiler when inherited, for instance, from 589 ``python-config``. For instance ``-Wstrict-prototypes`` is a C-only option 590 that should be pruned. 591 592 593``[pythran]`` 594************* 595 596This one contains internal configuration settings. Play with it at your own risk! 597 598:``optimizations``: 599 600 A list of import paths pointing to transformation classes. This contains the 601 optimization pipeline of Pythran! If you design your own optimizations, 602 register them here! 603 604:``complex_hook``: 605 606 Set this to ``True`` for faster and still Numpy-compliant complex 607 multiplications. Not very portable, but generally works on Linux. 608 609``[typing]`` 610************ 611 612Another internal setting stuff. This controls the accuracy of the typing phase. An extract from the default setting file should convince you not to touch it:: 613 614 [typing] 615 616 # maximum number of container access taken into account during type inference 617 # increasing this value inreases typing accuracy 618 # but slows down compilation time, to the point of making g++ crash 619 max_container_type = 2 620 621 # maximum number of combiner per user function 622 # increasing this value inreases typing accuracy 623 # but slows down compilation time, to the point of making g++ crash 624 max_combiner = 2 625 626 # set this to true to enable a costly yet more accurate type inference algorithm 627 # This algorithms generates code difficult to compile for g++, but not clang++ 628 enable_two_steps_typing = False 629 630 631F.A.Q. 632------ 633 6341. Supported compiler versions: 635 636 - `g++` version 4.9 and above 637 638 - `clang++` version 3.5 and above 639 640Troubleshooting 641--------------- 642 643Plenty of them! Seriously, Pythran is software, so it will crash. You 644may make it abort in unusual ways! And more importantly, please provide 645feedback to serge_sans_paille using its email ``serge.guelton@telecom-bretagne.eu``, 646the IRC channel ``#pythran`` on OFTC, or the mailing list ``pythran@freelists.org`` 647 648**glhf!** 649