1.. currentmodule:: asyncio
2
3.. _asyncio-subprocess:
4
5============
6Subprocesses
7============
8
9**Source code:** :source:`Lib/asyncio/subprocess.py`,
10:source:`Lib/asyncio/base_subprocess.py`
11
12----------------------------------------
13
14This section describes high-level async/await asyncio APIs to
15create and manage subprocesses.
16
17.. _asyncio_example_subprocess_shell:
18
19Here's an example of how asyncio can run a shell command and
20obtain its result::
21
22    import asyncio
23
24    async def run(cmd):
25        proc = await asyncio.create_subprocess_shell(
26            cmd,
27            stdout=asyncio.subprocess.PIPE,
28            stderr=asyncio.subprocess.PIPE)
29
30        stdout, stderr = await proc.communicate()
31
32        print(f'[{cmd!r} exited with {proc.returncode}]')
33        if stdout:
34            print(f'[stdout]\n{stdout.decode()}')
35        if stderr:
36            print(f'[stderr]\n{stderr.decode()}')
37
38    asyncio.run(run('ls /zzz'))
39
40will print::
41
42    ['ls /zzz' exited with 1]
43    [stderr]
44    ls: /zzz: No such file or directory
45
46Because all asyncio subprocess functions are asynchronous and asyncio
47provides many tools to work with such functions, it is easy to execute
48and monitor multiple subprocesses in parallel.  It is indeed trivial
49to modify the above example to run several commands simultaneously::
50
51    async def main():
52        await asyncio.gather(
53            run('ls /zzz'),
54            run('sleep 1; echo "hello"'))
55
56    asyncio.run(main())
57
58See also the `Examples`_ subsection.
59
60
61Creating Subprocesses
62=====================
63
64.. coroutinefunction:: create_subprocess_exec(program, *args, stdin=None, \
65                          stdout=None, stderr=None, loop=None, \
66                          limit=None, **kwds)
67
68   Create a subprocess.
69
70   The *limit* argument sets the buffer limit for :class:`StreamReader`
71   wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
72   (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
73
74   Return a :class:`~asyncio.subprocess.Process` instance.
75
76   See the documentation of :meth:`loop.subprocess_exec` for other
77   parameters.
78
79   .. deprecated-removed:: 3.8 3.10
80
81      The *loop* parameter.
82
83.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \
84                          stdout=None, stderr=None, loop=None, \
85                          limit=None, **kwds)
86
87   Run the *cmd* shell command.
88
89   The *limit* argument sets the buffer limit for :class:`StreamReader`
90   wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
91   (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
92
93   Return a :class:`~asyncio.subprocess.Process` instance.
94
95   See the documentation of :meth:`loop.subprocess_shell` for other
96   parameters.
97
98   .. important::
99
100      It is the application's responsibility to ensure that all whitespace and
101      special characters are quoted appropriately to avoid `shell injection
102      <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
103      vulnerabilities. The :func:`shlex.quote` function can be used to properly
104      escape whitespace and special shell characters in strings that are going
105      to be used to construct shell commands.
106
107   .. deprecated-removed:: 3.8 3.10
108
109      The *loop* parameter.
110
111.. note::
112
113   Subprocesses are available for Windows if a :class:`ProactorEventLoop` is
114   used. See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>`
115   for details.
116
117.. seealso::
118
119   asyncio also has the following *low-level* APIs to work with subprocesses:
120   :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`,
121   :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`,
122   as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>`
123   and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`.
124
125
126Constants
127=========
128
129.. data:: asyncio.subprocess.PIPE
130
131   Can be passed to the *stdin*, *stdout* or *stderr* parameters.
132
133   If *PIPE* is passed to *stdin* argument, the
134   :attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute
135   will point to a :class:`StreamWriter` instance.
136
137   If *PIPE* is passed to *stdout* or *stderr* arguments, the
138   :attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and
139   :attr:`Process.stderr <asyncio.subprocess.Process.stderr>`
140   attributes will point to :class:`StreamReader` instances.
141
142.. data:: asyncio.subprocess.STDOUT
143
144   Special value that can be used as the *stderr* argument and indicates
145   that standard error should be redirected into standard output.
146
147.. data:: asyncio.subprocess.DEVNULL
148
149   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
150   to process creation functions.  It indicates that the special file
151   :data:`os.devnull` will be used for the corresponding subprocess stream.
152
153
154Interacting with Subprocesses
155=============================
156
157Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
158functions return instances of the *Process* class.  *Process* is a high-level
159wrapper that allows communicating with subprocesses and watching for
160their completion.
161
162.. class:: asyncio.subprocess.Process
163
164   An object that wraps OS processes created by the
165   :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
166   functions.
167
168   This class is designed to have a similar API to the
169   :class:`subprocess.Popen` class, but there are some
170   notable differences:
171
172   * unlike Popen, Process instances do not have an equivalent to
173     the :meth:`~subprocess.Popen.poll` method;
174
175   * the :meth:`~asyncio.subprocess.Process.communicate` and
176     :meth:`~asyncio.subprocess.Process.wait` methods don't have a
177     *timeout* parameter: use the :func:`wait_for` function;
178
179   * the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method
180     is asynchronous, whereas :meth:`subprocess.Popen.wait` method
181     is implemented as a blocking busy loop;
182
183   * the *universal_newlines* parameter is not supported.
184
185   This class is :ref:`not thread safe <asyncio-multithreading>`.
186
187   See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
188   section.
189
190   .. coroutinemethod:: wait()
191
192      Wait for the child process to terminate.
193
194      Set and return the :attr:`returncode` attribute.
195
196      .. note::
197
198         This method can deadlock when using ``stdout=PIPE`` or
199         ``stderr=PIPE`` and the child process generates so much output
200         that it blocks waiting for the OS pipe buffer to accept
201         more data. Use the :meth:`communicate` method when using pipes
202         to avoid this condition.
203
204   .. coroutinemethod:: communicate(input=None)
205
206      Interact with process:
207
208      1. send data to *stdin* (if *input* is not ``None``);
209      2. read data from *stdout* and *stderr*, until EOF is reached;
210      3. wait for process to terminate.
211
212      The optional *input* argument is the data (:class:`bytes` object)
213      that will be sent to the child process.
214
215      Return a tuple ``(stdout_data, stderr_data)``.
216
217      If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError`
218      exception is raised when writing *input* into *stdin*, the
219      exception is ignored.  This condition occurs when the process
220      exits before all data are written into *stdin*.
221
222      If it is desired to send data to the process' *stdin*,
223      the process needs to be created with ``stdin=PIPE``.  Similarly,
224      to get anything other than ``None`` in the result tuple, the
225      process has to be created with ``stdout=PIPE`` and/or
226      ``stderr=PIPE`` arguments.
227
228      Note, that the data read is buffered in memory, so do not use
229      this method if the data size is large or unlimited.
230
231   .. method:: send_signal(signal)
232
233      Sends the signal *signal* to the child process.
234
235      .. note::
236
237         On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`.
238         ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes
239         started with a *creationflags* parameter which includes
240         ``CREATE_NEW_PROCESS_GROUP``.
241
242   .. method:: terminate()
243
244      Stop the child process.
245
246      On POSIX systems this method sends :py:data:`signal.SIGTERM` to the
247      child process.
248
249      On Windows the Win32 API function :c:func:`TerminateProcess` is
250      called to stop the child process.
251
252   .. method:: kill()
253
254      Kill the child process.
255
256      On POSIX systems this method sends :py:data:`SIGKILL` to the child
257      process.
258
259      On Windows this method is an alias for :meth:`terminate`.
260
261   .. attribute:: stdin
262
263      Standard input stream (:class:`StreamWriter`) or ``None``
264      if the process was created with ``stdin=None``.
265
266   .. attribute:: stdout
267
268      Standard output stream (:class:`StreamReader`) or ``None``
269      if the process was created with ``stdout=None``.
270
271   .. attribute:: stderr
272
273      Standard error stream (:class:`StreamReader`) or ``None``
274      if the process was created with ``stderr=None``.
275
276   .. warning::
277
278      Use the :meth:`communicate` method rather than
279      :attr:`process.stdin.write() <stdin>`,
280      :attr:`await process.stdout.read() <stdout>` or
281      :attr:`await process.stderr.read <stderr>`.
282      This avoids deadlocks due to streams pausing reading or writing
283      and blocking the child process.
284
285   .. attribute:: pid
286
287      Process identification number (PID).
288
289      Note that for processes created by the :func:`create_subprocess_shell`
290      function, this attribute is the PID of the spawned shell.
291
292   .. attribute:: returncode
293
294      Return code of the process when it exits.
295
296      A ``None`` value indicates that the process has not terminated yet.
297
298      A negative value ``-N`` indicates that the child was terminated
299      by signal ``N`` (POSIX only).
300
301
302.. _asyncio-subprocess-threads:
303
304Subprocess and Threads
305----------------------
306
307Standard asyncio event loop supports running subprocesses from different threads by
308default.
309
310On Windows subprocesses are provided by :class:`ProactorEventLoop` only (default),
311:class:`SelectorEventLoop` has no subprocess support.
312
313On UNIX *child watchers* are used for subprocess finish waiting, see
314:ref:`asyncio-watchers` for more info.
315
316
317.. versionchanged:: 3.8
318
319   UNIX switched to use :class:`ThreadedChildWatcher` for spawning subprocesses from
320   different threads without any limitation.
321
322   Spawning a subprocess with *inactive* current child watcher raises
323   :exc:`RuntimeError`.
324
325Note that alternative event loop implementations might have own limitations;
326please refer to their documentation.
327
328.. seealso::
329
330   The :ref:`Concurrency and multithreading in asyncio
331   <asyncio-multithreading>` section.
332
333
334Examples
335--------
336
337An example using the :class:`~asyncio.subprocess.Process` class to
338control a subprocess and the :class:`StreamReader` class to read from
339its standard output.
340
341.. _asyncio_example_create_subprocess_exec:
342
343The subprocess is created by the :func:`create_subprocess_exec`
344function::
345
346    import asyncio
347    import sys
348
349    async def get_date():
350        code = 'import datetime; print(datetime.datetime.now())'
351
352        # Create the subprocess; redirect the standard output
353        # into a pipe.
354        proc = await asyncio.create_subprocess_exec(
355            sys.executable, '-c', code,
356            stdout=asyncio.subprocess.PIPE)
357
358        # Read one line of output.
359        data = await proc.stdout.readline()
360        line = data.decode('ascii').rstrip()
361
362        # Wait for the subprocess exit.
363        await proc.wait()
364        return line
365
366    date = asyncio.run(get_date())
367    print(f"Current date: {date}")
368
369
370See also the :ref:`same example <asyncio_example_subprocess_proto>`
371written using low-level APIs.
372