1:mod:`compileall` --- Byte-compile Python libraries
2===================================================
3
4.. module:: compileall
5   :synopsis: Tools for byte-compiling all Python source files in a directory tree.
6
7**Source code:** :source:`Lib/compileall.py`
8
9--------------
10
11This module provides some utility functions to support installing Python
12libraries.  These functions compile Python source files in a directory tree.
13This module can be used to create the cached byte-code files at library
14installation time, which makes them available for use even by users who don't
15have write permission to the library directories.
16
17
18Command-line use
19----------------
20
21This module can work as a script (using :program:`python -m compileall`) to
22compile Python sources.
23
24.. program:: compileall
25
26.. cmdoption:: directory ...
27               file ...
28
29   Positional arguments are files to compile or directories that contain
30   source files, traversed recursively.  If no argument is given, behave as if
31   the command line was ``-l <directories from sys.path>``.
32
33.. cmdoption:: -l
34
35   Do not recurse into subdirectories, only compile source code files directly
36   contained in the named or implied directories.
37
38.. cmdoption:: -f
39
40   Force rebuild even if timestamps are up-to-date.
41
42.. cmdoption:: -q
43
44   Do not print the list of files compiled. If passed once, error messages will
45   still be printed. If passed twice (``-qq``), all output is suppressed.
46
47.. cmdoption:: -d destdir
48
49   Directory prepended to the path to each file being compiled.  This will
50   appear in compilation time tracebacks, and is also compiled in to the
51   byte-code file, where it will be used in tracebacks and other messages in
52   cases where the source file does not exist at the time the byte-code file is
53   executed.
54
55.. cmdoption:: -s strip_prefix
56.. cmdoption:: -p prepend_prefix
57
58   Remove (``-s``) or append (``-p``) the given prefix of paths
59   recorded in the ``.pyc`` files.
60   Cannot be combined with ``-d``.
61
62.. cmdoption:: -x regex
63
64   regex is used to search the full path to each file considered for
65   compilation, and if the regex produces a match, the file is skipped.
66
67.. cmdoption:: -i list
68
69   Read the file ``list`` and add each line that it contains to the list of
70   files and directories to compile.  If ``list`` is ``-``, read lines from
71   ``stdin``.
72
73.. cmdoption:: -b
74
75   Write the byte-code files to their legacy locations and names, which may
76   overwrite byte-code files created by another version of Python.  The default
77   is to write files to their :pep:`3147` locations and names, which allows
78   byte-code files from multiple versions of Python to coexist.
79
80.. cmdoption:: -r
81
82   Control the maximum recursion level for subdirectories.
83   If this is given, then ``-l`` option will not be taken into account.
84   :program:`python -m compileall <directory> -r 0` is equivalent to
85   :program:`python -m compileall <directory> -l`.
86
87.. cmdoption:: -j N
88
89   Use *N* workers to compile the files within the given directory.
90   If ``0`` is used, then the result of :func:`os.cpu_count()`
91   will be used.
92
93.. cmdoption:: --invalidation-mode [timestamp|checked-hash|unchecked-hash]
94
95   Control how the generated byte-code files are invalidated at runtime.
96   The ``timestamp`` value, means that ``.pyc`` files with the source timestamp
97   and size embedded will be generated. The ``checked-hash`` and
98   ``unchecked-hash`` values cause hash-based pycs to be generated. Hash-based
99   pycs embed a hash of the source file contents rather than a timestamp. See
100   :ref:`pyc-invalidation` for more information on how Python validates
101   bytecode cache files at runtime.
102   The default is ``timestamp`` if the :envvar:`SOURCE_DATE_EPOCH` environment
103   variable is not set, and ``checked-hash`` if the ``SOURCE_DATE_EPOCH``
104   environment variable is set.
105
106.. cmdoption:: -o level
107
108   Compile with the given optimization level. May be used multiple times
109   to compile for multiple levels at a time (for example,
110   ``compileall -o 1 -o 2``).
111
112.. cmdoption:: -e dir
113
114   Ignore symlinks pointing outside the given directory.
115
116.. cmdoption:: --hardlink-dupes
117
118   If two ``.pyc`` files with different optimization level have
119   the same content, use hard links to consolidate duplicate files.
120
121.. versionchanged:: 3.2
122   Added the ``-i``, ``-b`` and ``-h`` options.
123
124.. versionchanged:: 3.5
125   Added the  ``-j``, ``-r``, and ``-qq`` options.  ``-q`` option
126   was changed to a multilevel value.  ``-b`` will always produce a
127   byte-code file ending in ``.pyc``, never ``.pyo``.
128
129.. versionchanged:: 3.7
130   Added the ``--invalidation-mode`` option.
131
132.. versionchanged:: 3.9
133   Added the ``-s``, ``-p``, ``-e`` and ``--hardlink-dupes`` options.
134   Raised the default recursion limit from 10 to
135   :py:func:`sys.getrecursionlimit()`.
136   Added the possibility to specify the ``-o`` option multiple times.
137
138
139There is no command-line option to control the optimization level used by the
140:func:`compile` function, because the Python interpreter itself already
141provides the option: :program:`python -O -m compileall`.
142
143Similarly, the :func:`compile` function respects the :attr:`sys.pycache_prefix`
144setting. The generated bytecode cache will only be useful if :func:`compile` is
145run with the same :attr:`sys.pycache_prefix` (if any) that will be used at
146runtime.
147
148Public functions
149----------------
150
151.. function:: compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)
152
153   Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
154   files along the way. Return a true value if all the files compiled successfully,
155   and a false value otherwise.
156
157   The *maxlevels* parameter is used to limit the depth of the recursion; it
158   defaults to ``sys.getrecursionlimit()``.
159
160   If *ddir* is given, it is prepended to the path to each file being compiled
161   for use in compilation time tracebacks, and is also compiled in to the
162   byte-code file, where it will be used in tracebacks and other messages in
163   cases where the source file does not exist at the time the byte-code file is
164   executed.
165
166   If *force* is true, modules are re-compiled even if the timestamps are up to
167   date.
168
169   If *rx* is given, its ``search`` method is called on the complete path to each
170   file considered for compilation, and if it returns a true value, the file
171   is skipped. This can be used to exclude files matching a regular expression,
172   given as a :ref:`re.Pattern <re-objects>` object.
173
174   If *quiet* is ``False`` or ``0`` (the default), the filenames and other
175   information are printed to standard out. Set to ``1``, only errors are
176   printed. Set to ``2``, all output is suppressed.
177
178   If *legacy* is true, byte-code files are written to their legacy locations
179   and names, which may overwrite byte-code files created by another version of
180   Python.  The default is to write files to their :pep:`3147` locations and
181   names, which allows byte-code files from multiple versions of Python to
182   coexist.
183
184   *optimize* specifies the optimization level for the compiler.  It is passed to
185   the built-in :func:`compile` function. Accepts also a sequence of optimization
186   levels which lead to multiple compilations of one :file:`.py` file in one call.
187
188   The argument *workers* specifies how many workers are used to
189   compile files in parallel. The default is to not use multiple workers.
190   If the platform can't use multiple workers and *workers* argument is given,
191   then sequential compilation will be used as a fallback.  If *workers*
192   is 0, the number of cores in the system is used.  If *workers* is
193   lower than ``0``, a :exc:`ValueError` will be raised.
194
195   *invalidation_mode* should be a member of the
196   :class:`py_compile.PycInvalidationMode` enum and controls how the generated
197   pycs are invalidated at runtime.
198
199   The *stripdir*, *prependdir* and *limit_sl_dest* arguments correspond to
200   the ``-s``, ``-p`` and ``-e`` options described above.
201   They may be specified as ``str``, ``bytes`` or :py:class:`os.PathLike`.
202
203   If *hardlink_dupes* is true and two ``.pyc`` files with different optimization
204   level have the same content, use hard links to consolidate duplicate files.
205
206   .. versionchanged:: 3.2
207      Added the *legacy* and *optimize* parameter.
208
209   .. versionchanged:: 3.5
210      Added the *workers* parameter.
211
212   .. versionchanged:: 3.5
213      *quiet* parameter was changed to a multilevel value.
214
215   .. versionchanged:: 3.5
216      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
217      no matter what the value of *optimize* is.
218
219   .. versionchanged:: 3.6
220      Accepts a :term:`path-like object`.
221
222   .. versionchanged:: 3.7
223      The *invalidation_mode* parameter was added.
224
225   .. versionchanged:: 3.7.2
226      The *invalidation_mode* parameter's default value is updated to None.
227
228   .. versionchanged:: 3.8
229      Setting *workers* to 0 now chooses the optimal number of cores.
230
231   .. versionchanged:: 3.9
232      Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments.
233      Default value of *maxlevels* was changed from ``10`` to ``sys.getrecursionlimit()``
234
235.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)
236
237   Compile the file with path *fullname*. Return a true value if the file
238   compiled successfully, and a false value otherwise.
239
240   If *ddir* is given, it is prepended to the path to the file being compiled
241   for use in compilation time tracebacks, and is also compiled in to the
242   byte-code file, where it will be used in tracebacks and other messages in
243   cases where the source file does not exist at the time the byte-code file is
244   executed.
245
246   If *rx* is given, its ``search`` method is passed the full path name to the
247   file being compiled, and if it returns a true value, the file is not
248   compiled and ``True`` is returned. This can be used to exclude files matching
249   a regular expression, given as a :ref:`re.Pattern <re-objects>` object.
250
251   If *quiet* is ``False`` or ``0`` (the default), the filenames and other
252   information are printed to standard out. Set to ``1``, only errors are
253   printed. Set to ``2``, all output is suppressed.
254
255   If *legacy* is true, byte-code files are written to their legacy locations
256   and names, which may overwrite byte-code files created by another version of
257   Python.  The default is to write files to their :pep:`3147` locations and
258   names, which allows byte-code files from multiple versions of Python to
259   coexist.
260
261   *optimize* specifies the optimization level for the compiler.  It is passed to
262   the built-in :func:`compile` function. Accepts also a sequence of optimization
263   levels which lead to multiple compilations of one :file:`.py` file in one call.
264
265   *invalidation_mode* should be a member of the
266   :class:`py_compile.PycInvalidationMode` enum and controls how the generated
267   pycs are invalidated at runtime.
268
269   The *stripdir*, *prependdir* and *limit_sl_dest* arguments correspond to
270   the ``-s``, ``-p`` and ``-e`` options described above.
271   They may be specified as ``str``, ``bytes`` or :py:class:`os.PathLike`.
272
273   If *hardlink_dupes* is true and two ``.pyc`` files with different optimization
274   level have the same content, use hard links to consolidate duplicate files.
275
276   .. versionadded:: 3.2
277
278   .. versionchanged:: 3.5
279      *quiet* parameter was changed to a multilevel value.
280
281   .. versionchanged:: 3.5
282      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
283      no matter what the value of *optimize* is.
284
285   .. versionchanged:: 3.7
286      The *invalidation_mode* parameter was added.
287
288   .. versionchanged:: 3.7.2
289      The *invalidation_mode* parameter's default value is updated to None.
290
291   .. versionchanged:: 3.9
292      Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments.
293
294.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=None)
295
296   Byte-compile all the :file:`.py` files found along ``sys.path``. Return a
297   true value if all the files compiled successfully, and a false value otherwise.
298
299   If *skip_curdir* is true (the default), the current directory is not included
300   in the search.  All other parameters are passed to the :func:`compile_dir`
301   function.  Note that unlike the other compile functions, ``maxlevels``
302   defaults to ``0``.
303
304   .. versionchanged:: 3.2
305      Added the *legacy* and *optimize* parameter.
306
307   .. versionchanged:: 3.5
308      *quiet* parameter was changed to a multilevel value.
309
310   .. versionchanged:: 3.5
311      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
312      no matter what the value of *optimize* is.
313
314   .. versionchanged:: 3.7
315      The *invalidation_mode* parameter was added.
316
317   .. versionchanged:: 3.7.2
318      The *invalidation_mode* parameter's default value is updated to None.
319
320To force a recompile of all the :file:`.py` files in the :file:`Lib/`
321subdirectory and all its subdirectories::
322
323   import compileall
324
325   compileall.compile_dir('Lib/', force=True)
326
327   # Perform same compilation, excluding files in .svn directories.
328   import re
329   compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)
330
331   # pathlib.Path objects can also be used.
332   import pathlib
333   compileall.compile_dir(pathlib.Path('Lib/'), force=True)
334
335.. seealso::
336
337   Module :mod:`py_compile`
338      Byte-compile a single source file.
339