1:mod:`venv` --- Creation of virtual environments
2================================================
3
4.. module:: venv
5   :synopsis: Creation of virtual environments.
6
7.. moduleauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
8.. sectionauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
9
10.. versionadded:: 3.3
11
12**Source code:** :source:`Lib/venv/`
13
14.. index:: pair: Environments; virtual
15
16--------------
17
18The :mod:`venv` module provides support for creating lightweight "virtual
19environments" with their own site directories, optionally isolated from system
20site directories.  Each virtual environment has its own Python binary (which
21matches the version of the binary that was used to create this environment) and
22can have its own independent set of installed Python packages in its site
23directories.
24
25See :pep:`405` for more information about Python virtual environments.
26
27.. seealso::
28
29   `Python Packaging User Guide: Creating and using virtual environments
30   <https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment>`__
31
32
33Creating virtual environments
34-----------------------------
35
36.. include:: /using/venv-create.inc
37
38
39.. _venv-def:
40
41.. note:: A virtual environment is a Python environment such that the Python
42   interpreter, libraries and scripts installed into it are isolated from those
43   installed in other virtual environments, and (by default) any libraries
44   installed in a "system" Python, i.e., one which is installed as part of your
45   operating system.
46
47   A virtual environment is a directory tree which contains Python executable
48   files and other files which indicate that it is a virtual environment.
49
50   Common installation tools such as setuptools_ and pip_ work as
51   expected with virtual environments. In other words, when a virtual
52   environment is active, they install Python packages into the virtual
53   environment without needing to be told to do so explicitly.
54
55   When a virtual environment is active (i.e., the virtual environment's Python
56   interpreter is running), the attributes :attr:`sys.prefix` and
57   :attr:`sys.exec_prefix` point to the base directory of the virtual
58   environment, whereas :attr:`sys.base_prefix` and
59   :attr:`sys.base_exec_prefix` point to the non-virtual environment Python
60   installation which was used to create the virtual environment. If a virtual
61   environment is not active, then :attr:`sys.prefix` is the same as
62   :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as
63   :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment
64   Python installation).
65
66   When a virtual environment is active, any options that change the
67   installation path will be ignored from all :mod:`distutils` configuration
68   files to prevent projects being inadvertently installed outside of the
69   virtual environment.
70
71   When working in a command shell, users can make a virtual environment active
72   by running an ``activate`` script in the virtual environment's executables
73   directory (the precise filename and command to use the file is
74   shell-dependent), which prepends the virtual environment's directory for
75   executables to the ``PATH`` environment variable for the running shell. There
76   should be no need in other circumstances to activate a virtual
77   environment; scripts installed into virtual environments have a "shebang"
78   line which points to the virtual environment's Python interpreter. This means
79   that the script will run with that interpreter regardless of the value of
80   ``PATH``. On Windows, "shebang" line processing is supported if you have the
81   Python Launcher for Windows installed (this was added to Python in 3.3 - see
82   :pep:`397` for more details). Thus, double-clicking an installed script in a
83   Windows Explorer window should run the script with the correct interpreter
84   without there needing to be any reference to its virtual environment in
85   ``PATH``.
86
87
88.. _venv-api:
89
90API
91---
92
93.. highlight:: python
94
95The high-level method described above makes use of a simple API which provides
96mechanisms for third-party virtual environment creators to customize environment
97creation according to their needs, the :class:`EnvBuilder` class.
98
99.. class:: EnvBuilder(system_site_packages=False, clear=False, \
100                      symlinks=False, upgrade=False, with_pip=False, \
101                      prompt=None)
102
103    The :class:`EnvBuilder` class accepts the following keyword arguments on
104    instantiation:
105
106    * ``system_site_packages`` -- a Boolean value indicating that the system Python
107      site-packages should be available to the environment (defaults to ``False``).
108
109    * ``clear`` -- a Boolean value which, if true, will delete the contents of
110      any existing target directory, before creating the environment.
111
112    * ``symlinks`` -- a Boolean value indicating whether to attempt to symlink the
113      Python binary rather than copying.
114
115    * ``upgrade`` -- a Boolean value which, if true, will upgrade an existing
116      environment with the running Python - for use when that Python has been
117      upgraded in-place (defaults to ``False``).
118
119    * ``with_pip`` -- a Boolean value which, if true, ensures pip is
120      installed in the virtual environment. This uses :mod:`ensurepip` with
121      the ``--default-pip`` option.
122
123    * ``prompt`` -- a String to be used after virtual environment is activated
124      (defaults to ``None`` which means directory name of the environment would
125      be used).
126
127    .. versionchanged:: 3.4
128       Added the ``with_pip`` parameter
129
130    .. versionadded:: 3.6
131       Added the ``prompt`` parameter
132
133    Creators of third-party virtual environment tools will be free to use the
134    provided :class:`EnvBuilder` class as a base class.
135
136    The returned env-builder is an object which has a method, ``create``:
137
138    .. method:: create(env_dir)
139
140        Create a virtual environment by specifying the target directory
141        (absolute or relative to the current directory) which is to contain the
142        virtual environment.  The ``create`` method will either create the
143        environment in the specified directory, or raise an appropriate
144        exception.
145
146        The ``create`` method of the :class:`EnvBuilder` class illustrates the
147        hooks available for subclass customization::
148
149            def create(self, env_dir):
150                """
151                Create a virtualized Python environment in a directory.
152                env_dir is the target directory to create an environment in.
153                """
154                env_dir = os.path.abspath(env_dir)
155                context = self.ensure_directories(env_dir)
156                self.create_configuration(context)
157                self.setup_python(context)
158                self.setup_scripts(context)
159                self.post_setup(context)
160
161        Each of the methods :meth:`ensure_directories`,
162        :meth:`create_configuration`, :meth:`setup_python`,
163        :meth:`setup_scripts` and :meth:`post_setup` can be overridden.
164
165    .. method:: ensure_directories(env_dir)
166
167        Creates the environment directory and all necessary directories, and
168        returns a context object.  This is just a holder for attributes (such as
169        paths), for use by the other methods. The directories are allowed to
170        exist already, as long as either ``clear`` or ``upgrade`` were
171        specified to allow operating on an existing environment directory.
172
173    .. method:: create_configuration(context)
174
175        Creates the ``pyvenv.cfg`` configuration file in the environment.
176
177    .. method:: setup_python(context)
178
179        Creates a copy or symlink to the Python executable in the environment.
180        On POSIX systems, if a specific executable ``python3.x`` was used,
181        symlinks to ``python`` and ``python3`` will be created pointing to that
182        executable, unless files with those names already exist.
183
184    .. method:: setup_scripts(context)
185
186        Installs activation scripts appropriate to the platform into the virtual
187        environment.
188
189    .. method:: post_setup(context)
190
191        A placeholder method which can be overridden in third party
192        implementations to pre-install packages in the virtual environment or
193        perform other post-creation steps.
194
195    .. versionchanged:: 3.7.2
196       Windows now uses redirector scripts for ``python[w].exe`` instead of
197       copying the actual binaries. In 3.7.2 only :meth:`setup_python` does
198       nothing unless running from a build in the source tree.
199
200    .. versionchanged:: 3.7.3
201       Windows copies the redirector scripts as part of :meth:`setup_python`
202       instead of :meth:`setup_scripts`. This was not the case in 3.7.2.
203       When using symlinks, the original executables will be linked.
204
205    In addition, :class:`EnvBuilder` provides this utility method that can be
206    called from :meth:`setup_scripts` or :meth:`post_setup` in subclasses to
207    assist in installing custom scripts into the virtual environment.
208
209    .. method:: install_scripts(context, path)
210
211        *path* is the path to a directory that should contain subdirectories
212        "common", "posix", "nt", each containing scripts destined for the bin
213        directory in the environment.  The contents of "common" and the
214        directory corresponding to :data:`os.name` are copied after some text
215        replacement of placeholders:
216
217        * ``__VENV_DIR__`` is replaced with the absolute path of the environment
218          directory.
219
220        * ``__VENV_NAME__`` is replaced with the environment name (final path
221          segment of environment directory).
222
223        * ``__VENV_PROMPT__`` is replaced with the prompt (the environment
224          name surrounded by parentheses and with a following space)
225
226        * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
227          (either ``bin`` or ``Scripts``).
228
229        * ``__VENV_PYTHON__`` is replaced with the absolute path of the
230          environment's executable.
231
232        The directories are allowed to exist (for when an existing environment
233        is being upgraded).
234
235There is also a module-level convenience function:
236
237.. function:: create(env_dir, system_site_packages=False, clear=False, \
238                     symlinks=False, with_pip=False, prompt=None)
239
240    Create an :class:`EnvBuilder` with the given keyword arguments, and call its
241    :meth:`~EnvBuilder.create` method with the *env_dir* argument.
242
243    .. versionadded:: 3.3
244
245    .. versionchanged:: 3.4
246       Added the ``with_pip`` parameter
247
248    .. versionchanged:: 3.6
249       Added the ``prompt`` parameter
250
251An example of extending ``EnvBuilder``
252--------------------------------------
253
254The following script shows how to extend :class:`EnvBuilder` by implementing a
255subclass which installs setuptools and pip into a created virtual environment::
256
257    import os
258    import os.path
259    from subprocess import Popen, PIPE
260    import sys
261    from threading import Thread
262    from urllib.parse import urlparse
263    from urllib.request import urlretrieve
264    import venv
265
266    class ExtendedEnvBuilder(venv.EnvBuilder):
267        """
268        This builder installs setuptools and pip so that you can pip or
269        easy_install other packages into the created virtual environment.
270
271        :param nodist: If true, setuptools and pip are not installed into the
272                       created virtual environment.
273        :param nopip: If true, pip is not installed into the created
274                      virtual environment.
275        :param progress: If setuptools or pip are installed, the progress of the
276                         installation can be monitored by passing a progress
277                         callable. If specified, it is called with two
278                         arguments: a string indicating some progress, and a
279                         context indicating where the string is coming from.
280                         The context argument can have one of three values:
281                         'main', indicating that it is called from virtualize()
282                         itself, and 'stdout' and 'stderr', which are obtained
283                         by reading lines from the output streams of a subprocess
284                         which is used to install the app.
285
286                         If a callable is not specified, default progress
287                         information is output to sys.stderr.
288        """
289
290        def __init__(self, *args, **kwargs):
291            self.nodist = kwargs.pop('nodist', False)
292            self.nopip = kwargs.pop('nopip', False)
293            self.progress = kwargs.pop('progress', None)
294            self.verbose = kwargs.pop('verbose', False)
295            super().__init__(*args, **kwargs)
296
297        def post_setup(self, context):
298            """
299            Set up any packages which need to be pre-installed into the
300            virtual environment being created.
301
302            :param context: The information for the virtual environment
303                            creation request being processed.
304            """
305            os.environ['VIRTUAL_ENV'] = context.env_dir
306            if not self.nodist:
307                self.install_setuptools(context)
308            # Can't install pip without setuptools
309            if not self.nopip and not self.nodist:
310                self.install_pip(context)
311
312        def reader(self, stream, context):
313            """
314            Read lines from a subprocess' output stream and either pass to a progress
315            callable (if specified) or write progress information to sys.stderr.
316            """
317            progress = self.progress
318            while True:
319                s = stream.readline()
320                if not s:
321                    break
322                if progress is not None:
323                    progress(s, context)
324                else:
325                    if not self.verbose:
326                        sys.stderr.write('.')
327                    else:
328                        sys.stderr.write(s.decode('utf-8'))
329                    sys.stderr.flush()
330            stream.close()
331
332        def install_script(self, context, name, url):
333            _, _, path, _, _, _ = urlparse(url)
334            fn = os.path.split(path)[-1]
335            binpath = context.bin_path
336            distpath = os.path.join(binpath, fn)
337            # Download script into the virtual environment's binaries folder
338            urlretrieve(url, distpath)
339            progress = self.progress
340            if self.verbose:
341                term = '\n'
342            else:
343                term = ''
344            if progress is not None:
345                progress('Installing %s ...%s' % (name, term), 'main')
346            else:
347                sys.stderr.write('Installing %s ...%s' % (name, term))
348                sys.stderr.flush()
349            # Install in the virtual environment
350            args = [context.env_exe, fn]
351            p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath)
352            t1 = Thread(target=self.reader, args=(p.stdout, 'stdout'))
353            t1.start()
354            t2 = Thread(target=self.reader, args=(p.stderr, 'stderr'))
355            t2.start()
356            p.wait()
357            t1.join()
358            t2.join()
359            if progress is not None:
360                progress('done.', 'main')
361            else:
362                sys.stderr.write('done.\n')
363            # Clean up - no longer needed
364            os.unlink(distpath)
365
366        def install_setuptools(self, context):
367            """
368            Install setuptools in the virtual environment.
369
370            :param context: The information for the virtual environment
371                            creation request being processed.
372            """
373            url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py'
374            self.install_script(context, 'setuptools', url)
375            # clear up the setuptools archive which gets downloaded
376            pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz')
377            files = filter(pred, os.listdir(context.bin_path))
378            for f in files:
379                f = os.path.join(context.bin_path, f)
380                os.unlink(f)
381
382        def install_pip(self, context):
383            """
384            Install pip in the virtual environment.
385
386            :param context: The information for the virtual environment
387                            creation request being processed.
388            """
389            url = 'https://bootstrap.pypa.io/get-pip.py'
390            self.install_script(context, 'pip', url)
391
392    def main(args=None):
393        compatible = True
394        if sys.version_info < (3, 3):
395            compatible = False
396        elif not hasattr(sys, 'base_prefix'):
397            compatible = False
398        if not compatible:
399            raise ValueError('This script is only for use with '
400                             'Python 3.3 or later')
401        else:
402            import argparse
403
404            parser = argparse.ArgumentParser(prog=__name__,
405                                             description='Creates virtual Python '
406                                                         'environments in one or '
407                                                         'more target '
408                                                         'directories.')
409            parser.add_argument('dirs', metavar='ENV_DIR', nargs='+',
410                                help='A directory in which to create the
411                                     'virtual environment.')
412            parser.add_argument('--no-setuptools', default=False,
413                                action='store_true', dest='nodist',
414                                help="Don't install setuptools or pip in the "
415                                     "virtual environment.")
416            parser.add_argument('--no-pip', default=False,
417                                action='store_true', dest='nopip',
418                                help="Don't install pip in the virtual "
419                                     "environment.")
420            parser.add_argument('--system-site-packages', default=False,
421                                action='store_true', dest='system_site',
422                                help='Give the virtual environment access to the '
423                                     'system site-packages dir.')
424            if os.name == 'nt':
425                use_symlinks = False
426            else:
427                use_symlinks = True
428            parser.add_argument('--symlinks', default=use_symlinks,
429                                action='store_true', dest='symlinks',
430                                help='Try to use symlinks rather than copies, '
431                                     'when symlinks are not the default for '
432                                     'the platform.')
433            parser.add_argument('--clear', default=False, action='store_true',
434                                dest='clear', help='Delete the contents of the '
435                                                   'virtual environment '
436                                                   'directory if it already '
437                                                   'exists, before virtual '
438                                                   'environment creation.')
439            parser.add_argument('--upgrade', default=False, action='store_true',
440                                dest='upgrade', help='Upgrade the virtual '
441                                                     'environment directory to '
442                                                     'use this version of '
443                                                     'Python, assuming Python '
444                                                     'has been upgraded '
445                                                     'in-place.')
446            parser.add_argument('--verbose', default=False, action='store_true',
447                                dest='verbose', help='Display the output '
448                                                   'from the scripts which '
449                                                   'install setuptools and pip.')
450            options = parser.parse_args(args)
451            if options.upgrade and options.clear:
452                raise ValueError('you cannot supply --upgrade and --clear together.')
453            builder = ExtendedEnvBuilder(system_site_packages=options.system_site,
454                                           clear=options.clear,
455                                           symlinks=options.symlinks,
456                                           upgrade=options.upgrade,
457                                           nodist=options.nodist,
458                                           nopip=options.nopip,
459                                           verbose=options.verbose)
460            for d in options.dirs:
461                builder.create(d)
462
463    if __name__ == '__main__':
464        rc = 1
465        try:
466            main()
467            rc = 0
468        except Exception as e:
469            print('Error: %s' % e, file=sys.stderr)
470        sys.exit(rc)
471
472
473This script is also available for download `online
474<https://gist.github.com/vsajip/4673395>`_.
475
476
477.. _setuptools: https://pypi.org/project/setuptools/
478.. _pip: https://pypi.org/project/pip/
479