1Developer Guide
2###############
3
4Do not hang around in Pythran code base without your developer guide! It is
5the compass that will guide you in the code jungle!
6
7Disclaimer
8----------
9
10This document is a never ending work-in-progress draft. Please contribute!
11
12Configuration
13-------------
14
15Pythran can be configured with a rc file. An example is found in `pythran/pythran.cfg`.
16Look at it! To customize it::
17
18    $> cp pythran/pythran.cfg ~/.pythranrc
19
20In particular, you may want to add ``-g -O0`` to the ``cxxflags``.
21
22Coding Style
23------------
24
25All Python code must be conform to the PEP 8, and the ``flake8`` command must not
26yield any message when run on our database. Additionally, avoid backslashes,
27and try to make your code as concise as possible.
28
29    $> flake8 pythran/*.py pythran/*/*.py --exclude="pythran/tests/test*.py,__init__.py"
30
31C++ code use spaces (no tabs) and a tab width of 4.
32
33File Hierarchy
34--------------
35
36Listing the top level directory yields the following entries:
37
38setup.py
39    The files that describes what gets installed, that holds ``PyPI`` entries
40    and such.
41
42docs/
43    If you're reading this document, you know what it's all about!  ``MANUAL``
44    is the user documentation and ``DEVGUIDE`` is the developer documentation.
45
46    Use ``make`` from this directory to produce the static website.
47
48LICENSE
49    Boring but important stuff.
50
51MANIFEST.in
52    Describe additional stuff to package there.
53
54README.rst
55    Quick introduction and description of _pythran_.
56
57pythran/
58    The source of all things.
59
60pythran/tests/
61    The source of all issues.
62
63pythran/pythonic/
64    Where C++ back-end lies.
65
66
67Validation
68----------
69
70``pythran`` uses the ``unittest`` module and the `pytest
71<http://pytest.org/latest/>`_ package to manage test cases. The whole
72validation suite is run through the command::
73
74    $> python -m pytest pythran/tests
75
76To run it faster we use the ``pytest`` extension `xdist
77<https://pypi.org/project/pytest-xdist/>`_, the test suite will run using all
78available cores. Otherwise it might run **very** slowly, something like four
79hours on a decent laptop :'(.
80
81Note that it is possible to use the ``pytest`` module to pass a subset of the
82test suite::
83
84    $> pytest -n 8 pythran/tests/test_list.py
85
86runs all the tests found in ``pythran/tests/test_list.py``.
87
88Only compiler tests can be check using test filtering::
89
90   $> pytest -n 8 pythran/tests -m "not module"
91
92There are two kinds of tests in ``pythran``:
93
941. unit tests that test a specific feature of the implementation. Such tests
95   are listed as method of a class deriving from ``test_env.TestEnv`` and must
96   call the ``run_test(function_to_translate, *effective_parameters,
97   **name_to_signature)`` method [1]_.  It translates ``function_to_translate``
98   into a native function using the type annotations given in the
99   ``name_to_signature`` dictionary, runs both the python and the native
100   version with ``effective_parameters`` as arguments and asserts the results
101   are the same.
102
103   .. [1] See examples in ``pythran/tests/test_base.py`` for more details.
104
1052. test cases that are just plain python modules to be converted in native
106   module by ``pythran``. It is used to test complex situations, codes or
107   benchmarks found on the web etc. They are just translated, not run. These
108   test cases lie in ``pythran/tests/cases/`` and are listed in
109   ``pythran/tests/test_cases.py``.
110
111
112C++ runtime
113-----------
114
115The C++ code generated by ``pythran`` relies on a specific back-end,
116``pythonic``. It is a set of headers that mimics Python's intrinsics and
117collections behavior in C++. It lies in ``pythran/pythonic/``. There is one
118directory per module, e.g. ``pythran/pythonic/numpy`` for the ``numpy`` module,
119and one file per function, e.g. ``pythran/pythonic/numpy/ones.hpp`` for the
120``numpy.ones`` function. Type definitions are stored in the seperate
121``pythran/pythonic/types`` directory, one header per type. Each function header
122must be ``#includ``-able independently, i.e. it itself includes all the type
123and function definition it needs. This helps keeping compilation time low.
124
125All Pythran functions and types live in the ``pythonic`` namespace. Each extra
126module defines a new namespace, like ``pythonic::math`` or
127``pythonic::random``, and each type is defined in the ``pythonic::types``
128namespace. The ``DECLARE_FUNCTOR`` and ``DEFINE_FUNCTOR`` macros from
129``pythonic/utils/functor.hpp`` is commonly used to convert functions into
130functors and put them into the mandatory ``functor`` namespace.
131
132The pythonic runtime can be used without Python support, so it is important to
133protect all Python-specific stuff inside ``ENABLE_PYTHON_MODULE`` guard.
134
135All methods are represented by functions in Pythran. The associated
136pseudo-modules are prefixed and suffixed by a double underscore ``__``, as in
137``pythran/pythonic/__list__``.
138
139
140Benchmarking and Testing
141------------------------
142
143Stand-alone algorithms are put into ``pythran/tests/cases``. They must be valid
144Pythran input (including spec annotations). To be taken into account by the
145validation suite, they must be listed in ``pythran/tests/test_cases.py``. To be
146taken into account by the benchmarking suite, they must have a line starting
147with the ``#runas`` directive. Check ``pythran/tests/matmul.py`` for a complete
148example.
149
150To run the benchmark suite, one can rely on::
151
152    $> python setup.py bench --mode=<mode>
153
154where *<mode>* is one among:
155
156python
157    Uses the interpreter used to run ``setup.py``.
158
159pythran
160    Uses the Pythran compiler.
161
162pythran+omp
163    Uses the Pythran compiler in OpenMP mode.
164
165All measurements are made using the ``timeit`` module. The number of iterations
166is customizable through the ``--nb-iter`` switch.
167
168How to
169------
170
171:Add support for a new module:
172    1. Provide its C++ implementation in ``pythran/pythonic++/<mymodule>``.
173           ``pythran/pythonic++/math/*.hpp`` and
174           ``pythran/pythonic++/__list__/*.hpp`` are good example to referer to.
175    2. Provide its description in ``pythran/tables.py``. Each function, method
176           or variable must be listed there with the appropriate description.
177    3. Provide its test suite in ``pythran/tests/`` under the name
178           ``test_my_module.py``. One test case per function, method or variable
179           is great.
180
181:Add a new analysis:
182    1. Subclass one of ``ModuleAnalysis``, ``FunctionAnalysis`` or ``NodeAnalysis``.
183    2. List analysis required by yours in the parent constructor, they will be built automatically and stored in the attribute with the corresponding uncameled name.
184    3. Write your analysis as a regular ``ast.NodeVisitor``. The analysis result must be stored in ``self.result``.
185    4. Use it either from another pass's constructor, or through the ``passmanager.gather`` function.
186
187
188:Push changes into the holy trunk:
189    1. Use the ``github`` interface and the pull/push requests features
190    2. Make your dev available on the web and asks for a merge on the IRC
191           channel ``#pythran`` on ``irc.oftc.net``
192           (via your browser: https://webchat.oftc.net)
193