1:mod:`mmap` --- Memory-mapped file support
2==========================================
3
4.. module:: mmap
5   :synopsis: Interface to memory-mapped files for Unix and Windows.
6
7--------------
8
9Memory-mapped file objects behave like both :class:`bytearray` and like
10:term:`file objects <file object>`.  You can use mmap objects in most places
11where :class:`bytearray` are expected; for example, you can use the :mod:`re`
12module to search through a memory-mapped file.  You can also change a single
13byte by doing ``obj[index] = 97``, or change a subsequence by assigning to a
14slice: ``obj[i1:i2] = b'...'``.  You can also read and write data starting at
15the current file position, and :meth:`seek` through the file to different positions.
16
17A memory-mapped file is created by the :class:`~mmap.mmap` constructor, which is
18different on Unix and on Windows.  In either case you must provide a file
19descriptor for a file opened for update. If you wish to map an existing Python
20file object, use its :meth:`fileno` method to obtain the correct value for the
21*fileno* parameter.  Otherwise, you can open the file using the
22:func:`os.open` function, which returns a file descriptor directly (the file
23still needs to be closed when done).
24
25.. note::
26   If you want to create a memory-mapping for a writable, buffered file, you
27   should :func:`~io.IOBase.flush` the file first.  This is necessary to ensure
28   that local modifications to the buffers are actually available to the
29   mapping.
30
31For both the Unix and Windows versions of the constructor, *access* may be
32specified as an optional keyword parameter. *access* accepts one of four
33values: :const:`ACCESS_READ`, :const:`ACCESS_WRITE`, or :const:`ACCESS_COPY` to
34specify read-only, write-through or copy-on-write memory respectively, or
35:const:`ACCESS_DEFAULT` to defer to *prot*.  *access* can be used on both Unix
36and Windows.  If *access* is not specified, Windows mmap returns a
37write-through mapping.  The initial memory values for all three access types
38are taken from the specified file.  Assignment to an :const:`ACCESS_READ`
39memory map raises a :exc:`TypeError` exception.  Assignment to an
40:const:`ACCESS_WRITE` memory map affects both memory and the underlying file.
41Assignment to an :const:`ACCESS_COPY` memory map affects memory but does not
42update the underlying file.
43
44.. versionchanged:: 3.7
45   Added :const:`ACCESS_DEFAULT` constant.
46
47To map anonymous memory, -1 should be passed as the fileno along with the length.
48
49.. class:: mmap(fileno, length, tagname=None, access=ACCESS_DEFAULT[, offset])
50
51   **(Windows version)** Maps *length* bytes from the file specified by the
52   file handle *fileno*, and creates a mmap object.  If *length* is larger
53   than the current size of the file, the file is extended to contain *length*
54   bytes.  If *length* is ``0``, the maximum length of the map is the current
55   size of the file, except that if the file is empty Windows raises an
56   exception (you cannot create an empty mapping on Windows).
57
58   *tagname*, if specified and not ``None``, is a string giving a tag name for
59   the mapping.  Windows allows you to have many different mappings against
60   the same file.  If you specify the name of an existing tag, that tag is
61   opened, otherwise a new tag of this name is created.  If this parameter is
62   omitted or ``None``, the mapping is created without a name.  Avoiding the
63   use of the tag parameter will assist in keeping your code portable between
64   Unix and Windows.
65
66   *offset* may be specified as a non-negative integer offset. mmap references
67   will be relative to the offset from the beginning of the file. *offset*
68   defaults to 0.  *offset* must be a multiple of the :const:`ALLOCATIONGRANULARITY`.
69
70   .. audit-event:: mmap.__new__ fileno,length,access,offset mmap.mmap
71
72.. class:: mmap(fileno, length, flags=MAP_SHARED, prot=PROT_WRITE|PROT_READ, access=ACCESS_DEFAULT[, offset])
73   :noindex:
74
75   **(Unix version)** Maps *length* bytes from the file specified by the file
76   descriptor *fileno*, and returns a mmap object.  If *length* is ``0``, the
77   maximum length of the map will be the current size of the file when
78   :class:`~mmap.mmap` is called.
79
80   *flags* specifies the nature of the mapping. :const:`MAP_PRIVATE` creates a
81   private copy-on-write mapping, so changes to the contents of the mmap
82   object will be private to this process, and :const:`MAP_SHARED` creates a
83   mapping that's shared with all other processes mapping the same areas of
84   the file.  The default value is :const:`MAP_SHARED`.
85
86   *prot*, if specified, gives the desired memory protection; the two most
87   useful values are :const:`PROT_READ` and :const:`PROT_WRITE`, to specify
88   that the pages may be read or written.  *prot* defaults to
89   :const:`PROT_READ \| PROT_WRITE`.
90
91   *access* may be specified in lieu of *flags* and *prot* as an optional
92   keyword parameter.  It is an error to specify both *flags*, *prot* and
93   *access*.  See the description of *access* above for information on how to
94   use this parameter.
95
96   *offset* may be specified as a non-negative integer offset. mmap references
97   will be relative to the offset from the beginning of the file. *offset*
98   defaults to 0. *offset* must be a multiple of :const:`ALLOCATIONGRANULARITY`
99   which is equal to :const:`PAGESIZE` on Unix systems.
100
101   To ensure validity of the created memory mapping the file specified
102   by the descriptor *fileno* is internally automatically synchronized
103   with physical backing store on Mac OS X and OpenVMS.
104
105   This example shows a simple way of using :class:`~mmap.mmap`::
106
107      import mmap
108
109      # write a simple example file
110      with open("hello.txt", "wb") as f:
111          f.write(b"Hello Python!\n")
112
113      with open("hello.txt", "r+b") as f:
114          # memory-map the file, size 0 means whole file
115          mm = mmap.mmap(f.fileno(), 0)
116          # read content via standard file methods
117          print(mm.readline())  # prints b"Hello Python!\n"
118          # read content via slice notation
119          print(mm[:5])  # prints b"Hello"
120          # update content using slice notation;
121          # note that new content must have same size
122          mm[6:] = b" world!\n"
123          # ... and read again using standard file methods
124          mm.seek(0)
125          print(mm.readline())  # prints b"Hello  world!\n"
126          # close the map
127          mm.close()
128
129
130   :class:`~mmap.mmap` can also be used as a context manager in a :keyword:`with`
131   statement::
132
133      import mmap
134
135      with mmap.mmap(-1, 13) as mm:
136          mm.write(b"Hello world!")
137
138   .. versionadded:: 3.2
139      Context manager support.
140
141
142   The next example demonstrates how to create an anonymous map and exchange
143   data between the parent and child processes::
144
145      import mmap
146      import os
147
148      mm = mmap.mmap(-1, 13)
149      mm.write(b"Hello world!")
150
151      pid = os.fork()
152
153      if pid == 0:  # In a child process
154          mm.seek(0)
155          print(mm.readline())
156
157          mm.close()
158
159   .. audit-event:: mmap.__new__ fileno,length,access,offset mmap.mmap
160
161   Memory-mapped file objects support the following methods:
162
163   .. method:: close()
164
165      Closes the mmap. Subsequent calls to other methods of the object will
166      result in a ValueError exception being raised. This will not close
167      the open file.
168
169
170   .. attribute:: closed
171
172      ``True`` if the file is closed.
173
174      .. versionadded:: 3.2
175
176
177   .. method:: find(sub[, start[, end]])
178
179      Returns the lowest index in the object where the subsequence *sub* is
180      found, such that *sub* is contained in the range [*start*, *end*].
181      Optional arguments *start* and *end* are interpreted as in slice notation.
182      Returns ``-1`` on failure.
183
184      .. versionchanged:: 3.5
185         Writable :term:`bytes-like object` is now accepted.
186
187
188   .. method:: flush([offset[, size]])
189
190      Flushes changes made to the in-memory copy of a file back to disk. Without
191      use of this call there is no guarantee that changes are written back before
192      the object is destroyed.  If *offset* and *size* are specified, only
193      changes to the given range of bytes will be flushed to disk; otherwise, the
194      whole extent of the mapping is flushed.  *offset* must be a multiple of the
195      :const:`PAGESIZE` or :const:`ALLOCATIONGRANULARITY`.
196
197      ``None`` is returned to indicate success.  An exception is raised when the
198      call failed.
199
200      .. versionchanged:: 3.8
201         Previously, a nonzero value was returned on success; zero was returned
202         on error under Windows.  A zero value was returned on success; an
203         exception was raised on error under Unix.
204
205
206   .. method:: madvise(option[, start[, length]])
207
208      Send advice *option* to the kernel about the memory region beginning at
209      *start* and extending *length* bytes.  *option* must be one of the
210      :ref:`MADV_* constants <madvise-constants>` available on the system.  If
211      *start* and *length* are omitted, the entire mapping is spanned.  On
212      some systems (including Linux), *start* must be a multiple of the
213      :const:`PAGESIZE`.
214
215      Availability: Systems with the ``madvise()`` system call.
216
217      .. versionadded:: 3.8
218
219
220   .. method:: move(dest, src, count)
221
222      Copy the *count* bytes starting at offset *src* to the destination index
223      *dest*.  If the mmap was created with :const:`ACCESS_READ`, then calls to
224      move will raise a :exc:`TypeError` exception.
225
226
227   .. method:: read([n])
228
229      Return a :class:`bytes` containing up to *n* bytes starting from the
230      current file position. If the argument is omitted, ``None`` or negative,
231      return all bytes from the current file position to the end of the
232      mapping. The file position is updated to point after the bytes that were
233      returned.
234
235      .. versionchanged:: 3.3
236         Argument can be omitted or ``None``.
237
238   .. method:: read_byte()
239
240      Returns a byte at the current file position as an integer, and advances
241      the file position by 1.
242
243
244   .. method:: readline()
245
246      Returns a single line, starting at the current file position and up to the
247      next newline.
248
249
250   .. method:: resize(newsize)
251
252      Resizes the map and the underlying file, if any. If the mmap was created
253      with :const:`ACCESS_READ` or :const:`ACCESS_COPY`, resizing the map will
254      raise a :exc:`TypeError` exception.
255
256
257   .. method:: rfind(sub[, start[, end]])
258
259      Returns the highest index in the object where the subsequence *sub* is
260      found, such that *sub* is contained in the range [*start*, *end*].
261      Optional arguments *start* and *end* are interpreted as in slice notation.
262      Returns ``-1`` on failure.
263
264      .. versionchanged:: 3.5
265         Writable :term:`bytes-like object` is now accepted.
266
267
268   .. method:: seek(pos[, whence])
269
270      Set the file's current position.  *whence* argument is optional and
271      defaults to ``os.SEEK_SET`` or ``0`` (absolute file positioning); other
272      values are ``os.SEEK_CUR`` or ``1`` (seek relative to the current
273      position) and ``os.SEEK_END`` or ``2`` (seek relative to the file's end).
274
275
276   .. method:: size()
277
278      Return the length of the file, which can be larger than the size of the
279      memory-mapped area.
280
281
282   .. method:: tell()
283
284      Returns the current position of the file pointer.
285
286
287   .. method:: write(bytes)
288
289      Write the bytes in *bytes* into memory at the current position of the
290      file pointer and return the number of bytes written (never less than
291      ``len(bytes)``, since if the write fails, a :exc:`ValueError` will be
292      raised).  The file position is updated to point after the bytes that
293      were written.  If the mmap was created with :const:`ACCESS_READ`, then
294      writing to it will raise a :exc:`TypeError` exception.
295
296      .. versionchanged:: 3.5
297         Writable :term:`bytes-like object` is now accepted.
298
299      .. versionchanged:: 3.6
300         The number of bytes written is now returned.
301
302
303   .. method:: write_byte(byte)
304
305      Write the integer *byte* into memory at the current
306      position of the file pointer; the file position is advanced by ``1``. If
307      the mmap was created with :const:`ACCESS_READ`, then writing to it will
308      raise a :exc:`TypeError` exception.
309
310.. _madvise-constants:
311
312MADV_* Constants
313++++++++++++++++
314
315.. data:: MADV_NORMAL
316          MADV_RANDOM
317          MADV_SEQUENTIAL
318          MADV_WILLNEED
319          MADV_DONTNEED
320          MADV_REMOVE
321          MADV_DONTFORK
322          MADV_DOFORK
323          MADV_HWPOISON
324          MADV_MERGEABLE
325          MADV_UNMERGEABLE
326          MADV_SOFT_OFFLINE
327          MADV_HUGEPAGE
328          MADV_NOHUGEPAGE
329          MADV_DONTDUMP
330          MADV_DODUMP
331          MADV_FREE
332          MADV_NOSYNC
333          MADV_AUTOSYNC
334          MADV_NOCORE
335          MADV_CORE
336          MADV_PROTECT
337
338   These options can be passed to :meth:`mmap.madvise`.  Not every option will
339   be present on every system.
340
341   Availability: Systems with the madvise() system call.
342
343   .. versionadded:: 3.8
344