1:mod:`venv` --- Creation of virtual environments
2================================================
3
4.. module:: venv
5   :synopsis: Creation of virtual environments.
6
7.. moduleauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
8.. sectionauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
9
10.. versionadded:: 3.3
11
12**Source code:** :source:`Lib/venv/`
13
14.. index:: pair: Environments; virtual
15
16--------------
17
18The :mod:`venv` module provides support for creating lightweight "virtual
19environments" with their own site directories, optionally isolated from system
20site directories.  Each virtual environment has its own Python binary (which
21matches the version of the binary that was used to create this environment) and
22can have its own independent set of installed Python packages in its site
23directories.
24
25See :pep:`405` for more information about Python virtual environments.
26
27.. seealso::
28
29   `Python Packaging User Guide: Creating and using virtual environments
30   <https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment>`__
31
32
33Creating virtual environments
34-----------------------------
35
36.. include:: /using/venv-create.inc
37
38
39.. _venv-def:
40
41.. note:: A virtual environment is a Python environment such that the Python
42   interpreter, libraries and scripts installed into it are isolated from those
43   installed in other virtual environments, and (by default) any libraries
44   installed in a "system" Python, i.e., one which is installed as part of your
45   operating system.
46
47   A virtual environment is a directory tree which contains Python executable
48   files and other files which indicate that it is a virtual environment.
49
50   Common installation tools such as setuptools_ and pip_ work as
51   expected with virtual environments. In other words, when a virtual
52   environment is active, they install Python packages into the virtual
53   environment without needing to be told to do so explicitly.
54
55   When a virtual environment is active (i.e., the virtual environment's Python
56   interpreter is running), the attributes :attr:`sys.prefix` and
57   :attr:`sys.exec_prefix` point to the base directory of the virtual
58   environment, whereas :attr:`sys.base_prefix` and
59   :attr:`sys.base_exec_prefix` point to the non-virtual environment Python
60   installation which was used to create the virtual environment. If a virtual
61   environment is not active, then :attr:`sys.prefix` is the same as
62   :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as
63   :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment
64   Python installation).
65
66   When a virtual environment is active, any options that change the
67   installation path will be ignored from all :mod:`distutils` configuration
68   files to prevent projects being inadvertently installed outside of the
69   virtual environment.
70
71   When working in a command shell, users can make a virtual environment active
72   by running an ``activate`` script in the virtual environment's executables
73   directory (the precise filename and command to use the file is
74   shell-dependent), which prepends the virtual environment's directory for
75   executables to the ``PATH`` environment variable for the running shell. There
76   should be no need in other circumstances to activate a virtual
77   environment; scripts installed into virtual environments have a "shebang"
78   line which points to the virtual environment's Python interpreter. This means
79   that the script will run with that interpreter regardless of the value of
80   ``PATH``. On Windows, "shebang" line processing is supported if you have the
81   Python Launcher for Windows installed (this was added to Python in 3.3 - see
82   :pep:`397` for more details). Thus, double-clicking an installed script in a
83   Windows Explorer window should run the script with the correct interpreter
84   without there needing to be any reference to its virtual environment in
85   ``PATH``.
86
87
88.. _venv-api:
89
90API
91---
92
93.. highlight:: python
94
95The high-level method described above makes use of a simple API which provides
96mechanisms for third-party virtual environment creators to customize environment
97creation according to their needs, the :class:`EnvBuilder` class.
98
99.. class:: EnvBuilder(system_site_packages=False, clear=False, \
100                      symlinks=False, upgrade=False, with_pip=False, \
101                      prompt=None, upgrade_deps=False)
102
103    The :class:`EnvBuilder` class accepts the following keyword arguments on
104    instantiation:
105
106    * ``system_site_packages`` -- a Boolean value indicating that the system Python
107      site-packages should be available to the environment (defaults to ``False``).
108
109    * ``clear`` -- a Boolean value which, if true, will delete the contents of
110      any existing target directory, before creating the environment.
111
112    * ``symlinks`` -- a Boolean value indicating whether to attempt to symlink the
113      Python binary rather than copying.
114
115    * ``upgrade`` -- a Boolean value which, if true, will upgrade an existing
116      environment with the running Python - for use when that Python has been
117      upgraded in-place (defaults to ``False``).
118
119    * ``with_pip`` -- a Boolean value which, if true, ensures pip is
120      installed in the virtual environment. This uses :mod:`ensurepip` with
121      the ``--default-pip`` option.
122
123    * ``prompt`` -- a String to be used after virtual environment is activated
124      (defaults to ``None`` which means directory name of the environment would
125      be used). If the special string ``"."`` is provided, the basename of the
126      current directory is used as the prompt.
127
128    * ``upgrade_deps`` -- Update the base venv modules to the latest on PyPI
129
130    .. versionchanged:: 3.4
131       Added the ``with_pip`` parameter
132
133    .. versionadded:: 3.6
134       Added the ``prompt`` parameter
135
136    .. versionadded:: 3.9
137       Added the ``upgrade_deps`` parameter
138
139    Creators of third-party virtual environment tools will be free to use the
140    provided :class:`EnvBuilder` class as a base class.
141
142    The returned env-builder is an object which has a method, ``create``:
143
144    .. method:: create(env_dir)
145
146        Create a virtual environment by specifying the target directory
147        (absolute or relative to the current directory) which is to contain the
148        virtual environment.  The ``create`` method will either create the
149        environment in the specified directory, or raise an appropriate
150        exception.
151
152        The ``create`` method of the :class:`EnvBuilder` class illustrates the
153        hooks available for subclass customization::
154
155            def create(self, env_dir):
156                """
157                Create a virtualized Python environment in a directory.
158                env_dir is the target directory to create an environment in.
159                """
160                env_dir = os.path.abspath(env_dir)
161                context = self.ensure_directories(env_dir)
162                self.create_configuration(context)
163                self.setup_python(context)
164                self.setup_scripts(context)
165                self.post_setup(context)
166
167        Each of the methods :meth:`ensure_directories`,
168        :meth:`create_configuration`, :meth:`setup_python`,
169        :meth:`setup_scripts` and :meth:`post_setup` can be overridden.
170
171    .. method:: ensure_directories(env_dir)
172
173        Creates the environment directory and all necessary directories, and
174        returns a context object.  This is just a holder for attributes (such as
175        paths), for use by the other methods. The directories are allowed to
176        exist already, as long as either ``clear`` or ``upgrade`` were
177        specified to allow operating on an existing environment directory.
178
179    .. method:: create_configuration(context)
180
181        Creates the ``pyvenv.cfg`` configuration file in the environment.
182
183    .. method:: setup_python(context)
184
185        Creates a copy or symlink to the Python executable in the environment.
186        On POSIX systems, if a specific executable ``python3.x`` was used,
187        symlinks to ``python`` and ``python3`` will be created pointing to that
188        executable, unless files with those names already exist.
189
190    .. method:: setup_scripts(context)
191
192        Installs activation scripts appropriate to the platform into the virtual
193        environment.
194
195    .. method:: upgrade_dependencies(context)
196
197       Upgrades the core venv dependency packages (currently ``pip`` and
198       ``setuptools``) in the environment. This is done by shelling out to the
199       ``pip`` executable in the environment.
200
201       .. versionadded:: 3.9
202
203    .. method:: post_setup(context)
204
205        A placeholder method which can be overridden in third party
206        implementations to pre-install packages in the virtual environment or
207        perform other post-creation steps.
208
209    .. versionchanged:: 3.7.2
210       Windows now uses redirector scripts for ``python[w].exe`` instead of
211       copying the actual binaries. In 3.7.2 only :meth:`setup_python` does
212       nothing unless running from a build in the source tree.
213
214    .. versionchanged:: 3.7.3
215       Windows copies the redirector scripts as part of :meth:`setup_python`
216       instead of :meth:`setup_scripts`. This was not the case in 3.7.2.
217       When using symlinks, the original executables will be linked.
218
219    In addition, :class:`EnvBuilder` provides this utility method that can be
220    called from :meth:`setup_scripts` or :meth:`post_setup` in subclasses to
221    assist in installing custom scripts into the virtual environment.
222
223    .. method:: install_scripts(context, path)
224
225        *path* is the path to a directory that should contain subdirectories
226        "common", "posix", "nt", each containing scripts destined for the bin
227        directory in the environment.  The contents of "common" and the
228        directory corresponding to :data:`os.name` are copied after some text
229        replacement of placeholders:
230
231        * ``__VENV_DIR__`` is replaced with the absolute path of the environment
232          directory.
233
234        * ``__VENV_NAME__`` is replaced with the environment name (final path
235          segment of environment directory).
236
237        * ``__VENV_PROMPT__`` is replaced with the prompt (the environment
238          name surrounded by parentheses and with a following space)
239
240        * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
241          (either ``bin`` or ``Scripts``).
242
243        * ``__VENV_PYTHON__`` is replaced with the absolute path of the
244          environment's executable.
245
246        The directories are allowed to exist (for when an existing environment
247        is being upgraded).
248
249There is also a module-level convenience function:
250
251.. function:: create(env_dir, system_site_packages=False, clear=False, \
252                     symlinks=False, with_pip=False, prompt=None, \
253                     upgrade_deps=False)
254
255    Create an :class:`EnvBuilder` with the given keyword arguments, and call its
256    :meth:`~EnvBuilder.create` method with the *env_dir* argument.
257
258    .. versionadded:: 3.3
259
260    .. versionchanged:: 3.4
261       Added the ``with_pip`` parameter
262
263    .. versionchanged:: 3.6
264       Added the ``prompt`` parameter
265
266    .. versionchanged:: 3.9
267       Added the ``upgrade_deps`` parameter
268
269An example of extending ``EnvBuilder``
270--------------------------------------
271
272The following script shows how to extend :class:`EnvBuilder` by implementing a
273subclass which installs setuptools and pip into a created virtual environment::
274
275    import os
276    import os.path
277    from subprocess import Popen, PIPE
278    import sys
279    from threading import Thread
280    from urllib.parse import urlparse
281    from urllib.request import urlretrieve
282    import venv
283
284    class ExtendedEnvBuilder(venv.EnvBuilder):
285        """
286        This builder installs setuptools and pip so that you can pip or
287        easy_install other packages into the created virtual environment.
288
289        :param nodist: If true, setuptools and pip are not installed into the
290                       created virtual environment.
291        :param nopip: If true, pip is not installed into the created
292                      virtual environment.
293        :param progress: If setuptools or pip are installed, the progress of the
294                         installation can be monitored by passing a progress
295                         callable. If specified, it is called with two
296                         arguments: a string indicating some progress, and a
297                         context indicating where the string is coming from.
298                         The context argument can have one of three values:
299                         'main', indicating that it is called from virtualize()
300                         itself, and 'stdout' and 'stderr', which are obtained
301                         by reading lines from the output streams of a subprocess
302                         which is used to install the app.
303
304                         If a callable is not specified, default progress
305                         information is output to sys.stderr.
306        """
307
308        def __init__(self, *args, **kwargs):
309            self.nodist = kwargs.pop('nodist', False)
310            self.nopip = kwargs.pop('nopip', False)
311            self.progress = kwargs.pop('progress', None)
312            self.verbose = kwargs.pop('verbose', False)
313            super().__init__(*args, **kwargs)
314
315        def post_setup(self, context):
316            """
317            Set up any packages which need to be pre-installed into the
318            virtual environment being created.
319
320            :param context: The information for the virtual environment
321                            creation request being processed.
322            """
323            os.environ['VIRTUAL_ENV'] = context.env_dir
324            if not self.nodist:
325                self.install_setuptools(context)
326            # Can't install pip without setuptools
327            if not self.nopip and not self.nodist:
328                self.install_pip(context)
329
330        def reader(self, stream, context):
331            """
332            Read lines from a subprocess' output stream and either pass to a progress
333            callable (if specified) or write progress information to sys.stderr.
334            """
335            progress = self.progress
336            while True:
337                s = stream.readline()
338                if not s:
339                    break
340                if progress is not None:
341                    progress(s, context)
342                else:
343                    if not self.verbose:
344                        sys.stderr.write('.')
345                    else:
346                        sys.stderr.write(s.decode('utf-8'))
347                    sys.stderr.flush()
348            stream.close()
349
350        def install_script(self, context, name, url):
351            _, _, path, _, _, _ = urlparse(url)
352            fn = os.path.split(path)[-1]
353            binpath = context.bin_path
354            distpath = os.path.join(binpath, fn)
355            # Download script into the virtual environment's binaries folder
356            urlretrieve(url, distpath)
357            progress = self.progress
358            if self.verbose:
359                term = '\n'
360            else:
361                term = ''
362            if progress is not None:
363                progress('Installing %s ...%s' % (name, term), 'main')
364            else:
365                sys.stderr.write('Installing %s ...%s' % (name, term))
366                sys.stderr.flush()
367            # Install in the virtual environment
368            args = [context.env_exe, fn]
369            p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath)
370            t1 = Thread(target=self.reader, args=(p.stdout, 'stdout'))
371            t1.start()
372            t2 = Thread(target=self.reader, args=(p.stderr, 'stderr'))
373            t2.start()
374            p.wait()
375            t1.join()
376            t2.join()
377            if progress is not None:
378                progress('done.', 'main')
379            else:
380                sys.stderr.write('done.\n')
381            # Clean up - no longer needed
382            os.unlink(distpath)
383
384        def install_setuptools(self, context):
385            """
386            Install setuptools in the virtual environment.
387
388            :param context: The information for the virtual environment
389                            creation request being processed.
390            """
391            url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py'
392            self.install_script(context, 'setuptools', url)
393            # clear up the setuptools archive which gets downloaded
394            pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz')
395            files = filter(pred, os.listdir(context.bin_path))
396            for f in files:
397                f = os.path.join(context.bin_path, f)
398                os.unlink(f)
399
400        def install_pip(self, context):
401            """
402            Install pip in the virtual environment.
403
404            :param context: The information for the virtual environment
405                            creation request being processed.
406            """
407            url = 'https://bootstrap.pypa.io/get-pip.py'
408            self.install_script(context, 'pip', url)
409
410    def main(args=None):
411        compatible = True
412        if sys.version_info < (3, 3):
413            compatible = False
414        elif not hasattr(sys, 'base_prefix'):
415            compatible = False
416        if not compatible:
417            raise ValueError('This script is only for use with '
418                             'Python 3.3 or later')
419        else:
420            import argparse
421
422            parser = argparse.ArgumentParser(prog=__name__,
423                                             description='Creates virtual Python '
424                                                         'environments in one or '
425                                                         'more target '
426                                                         'directories.')
427            parser.add_argument('dirs', metavar='ENV_DIR', nargs='+',
428                                help='A directory in which to create the '
429                                     'virtual environment.')
430            parser.add_argument('--no-setuptools', default=False,
431                                action='store_true', dest='nodist',
432                                help="Don't install setuptools or pip in the "
433                                     "virtual environment.")
434            parser.add_argument('--no-pip', default=False,
435                                action='store_true', dest='nopip',
436                                help="Don't install pip in the virtual "
437                                     "environment.")
438            parser.add_argument('--system-site-packages', default=False,
439                                action='store_true', dest='system_site',
440                                help='Give the virtual environment access to the '
441                                     'system site-packages dir.')
442            if os.name == 'nt':
443                use_symlinks = False
444            else:
445                use_symlinks = True
446            parser.add_argument('--symlinks', default=use_symlinks,
447                                action='store_true', dest='symlinks',
448                                help='Try to use symlinks rather than copies, '
449                                     'when symlinks are not the default for '
450                                     'the platform.')
451            parser.add_argument('--clear', default=False, action='store_true',
452                                dest='clear', help='Delete the contents of the '
453                                                   'virtual environment '
454                                                   'directory if it already '
455                                                   'exists, before virtual '
456                                                   'environment creation.')
457            parser.add_argument('--upgrade', default=False, action='store_true',
458                                dest='upgrade', help='Upgrade the virtual '
459                                                     'environment directory to '
460                                                     'use this version of '
461                                                     'Python, assuming Python '
462                                                     'has been upgraded '
463                                                     'in-place.')
464            parser.add_argument('--verbose', default=False, action='store_true',
465                                dest='verbose', help='Display the output '
466                                                   'from the scripts which '
467                                                   'install setuptools and pip.')
468            options = parser.parse_args(args)
469            if options.upgrade and options.clear:
470                raise ValueError('you cannot supply --upgrade and --clear together.')
471            builder = ExtendedEnvBuilder(system_site_packages=options.system_site,
472                                           clear=options.clear,
473                                           symlinks=options.symlinks,
474                                           upgrade=options.upgrade,
475                                           nodist=options.nodist,
476                                           nopip=options.nopip,
477                                           verbose=options.verbose)
478            for d in options.dirs:
479                builder.create(d)
480
481    if __name__ == '__main__':
482        rc = 1
483        try:
484            main()
485            rc = 0
486        except Exception as e:
487            print('Error: %s' % e, file=sys.stderr)
488        sys.exit(rc)
489
490
491This script is also available for download `online
492<https://gist.github.com/vsajip/4673395>`_.
493
494
495.. _setuptools: https://pypi.org/project/setuptools/
496.. _pip: https://pypi.org/project/pip/
497