1Metadata-Version: 2.1 2Name: xopen 3Version: 1.1.0 4Summary: Open compressed files transparently 5Home-page: https://github.com/marcelm/xopen/ 6Author: Marcel Martin 7Author-email: mail@marcelm.net 8License: MIT 9Description: .. image:: https://travis-ci.org/marcelm/xopen.svg?branch=master 10 :target: https://travis-ci.org/marcelm/xopen 11 :alt: 12 13 .. image:: https://img.shields.io/pypi/v/xopen.svg?branch=master 14 :target: https://pypi.python.org/pypi/xopen 15 16 .. image:: https://img.shields.io/conda/v/conda-forge/xopen.svg 17 :target: https://anaconda.org/conda-forge/xopen 18 :alt: 19 20 .. image:: https://codecov.io/gh/marcelm/xopen/branch/master/graph/badge.svg 21 :target: https://codecov.io/gh/marcelm/xopen 22 :alt: 23 24 ===== 25 xopen 26 ===== 27 28 This small Python module provides an ``xopen`` function that works like the 29 built-in ``open`` function, but can also deal with compressed files. 30 Supported compression formats are gzip, bzip2 and xz. They are automatically 31 recognized by their file extensions `.gz`, `.bz2` or `.xz`. 32 33 The focus is on being as efficient as possible on all supported Python versions. 34 For example, ``xopen`` uses ``pigz``, which is a parallel version of ``gzip``, 35 to open ``.gz`` files, which is faster than using the built-in ``gzip.open`` 36 function. ``pigz`` can use multiple threads when compressing, but is also faster 37 when reading ``.gz`` files, so it is used both for reading and writing if it is 38 available. For gzip compression levels 1 to 3, 39 `igzip <https://github.com/intel/isa-l/>`_ is used for an even greater speedup. 40 41 For use cases where using only the main thread is desired xopen can be used 42 with ``threads=0``. This will use `python-isal 43 <https://github.com/pycompression/python-isal>`_ (which binds isa-l) if 44 python-isal is installed (automatic on Linux systems, as it is a requirement). 45 For installation instructions for python-isal please 46 checkout the `python-isal homepage 47 <https://github.com/pycompression/python-isal>`_. If python-isal is not 48 available ``gzip.open`` is used. 49 50 This module has originally been developed as part of the `Cutadapt 51 tool <https://cutadapt.readthedocs.io/>`_ that is used in bioinformatics to 52 manipulate sequencing data. It has been in successful use within that software 53 for a few years. 54 55 ``xopen`` is compatible with Python versions 3.6 and later. 56 57 58 Usage 59 ----- 60 61 Open a file for reading:: 62 63 from xopen import xopen 64 65 with xopen('file.txt.xz') as f: 66 content = f.read() 67 68 Or without context manager:: 69 70 from xopen import xopen 71 72 f = xopen('file.txt.xz') 73 content = f.read() 74 f.close() 75 76 Open a file in binary mode for writing:: 77 78 from xopen import xopen 79 80 with xopen('file.txt.gz', mode='wb') as f: 81 f.write(b'Hello') 82 83 84 Credits 85 ------- 86 87 The name ``xopen`` was taken from the C function of the same name in the 88 `utils.h file which is part of 89 BWA <https://github.com/lh3/bwa/blob/83662032a2192d5712996f36069ab02db82acf67/utils.h>`_. 90 91 Kyle Beauchamp <https://github.com/kyleabeauchamp/> has contributed support for 92 appending to files. 93 94 Ruben Vorderman <https://github.com/rhpvorderman/> contributed improvements to 95 make reading and writing gzipped files faster. 96 97 Benjamin Vaisvil <https://github.com/bvaisvil> contributed support for 98 format detection from content. 99 100 Some ideas were taken from the `canopener project <https://github.com/selassid/canopener>`_. 101 If you also want to open S3 files, you may want to use that module instead. 102 103 104 Changes 105 ------- 106 v1.1.0 107 ~~~~~~ 108 * Python 3.5 support is dropped. 109 * On Linux systems, `python-isal <https://github.com/pycompression/python-isal>`_ 110 is now added as a requirement. This will speed up the reading of gzip files 111 significantly when no external processes are used. 112 113 v1.0.0 114 ~~~~~~ 115 * If installed, the ``igzip`` program (part of 116 `Intel ISA-L <https://github.com/intel/isa-l/>`_) is now used for reading 117 and writing gzip-compressed files at compression levels 1-3, which results 118 in a significant speedup. 119 120 v0.9.0 121 ~~~~~~ 122 * When the file name extension of a file to be opened for reading is not 123 available, the content is inspected (if possible) and used to determine 124 which compression format applies. 125 * This release drops Python 2.7 and 3.4 support. Python 3.5 or later is 126 now required. 127 128 v0.8.4 129 ~~~~~~ 130 * When reading gzipped files, force ``pigz`` to use only a single process. 131 ``pigz`` cannot use multiple cores anyway when decompressing. By default, 132 it would use extra I/O processes, which slightly reduces wall-clock time, 133 but increases CPU time. Single-core decompression with ``pigz`` is still 134 about twice as fast as regular ``gzip``. 135 * Allow ``threads=0`` for specifying that no external ``pigz``/``gzip`` 136 process should be used (then regular ``gzip.open()`` is used instead). 137 138 v0.8.3 139 ~~~~~~ 140 * When reading gzipped files, let ``pigz`` use at most four threads by default. 141 This limit previously only applied when writing to a file. 142 * Support Python 3.8 143 144 v0.8.0 145 ~~~~~~ 146 * Speed improvements when iterating over gzipped files. 147 148 v0.6.0 149 ~~~~~~ 150 * For reading from gzipped files, xopen will now use a ``pigz`` subprocess. 151 This is faster than using ``gzip.open``. 152 * Python 2 support will be dropped in one of the next releases. 153 154 v0.5.0 155 ~~~~~~ 156 * By default, pigz is now only allowed to use at most four threads. This hopefully reduces 157 problems some users had with too many threads when opening many files at the same time. 158 * xopen now accepts pathlib.Path objects. 159 160 161 Contributors 162 ------------ 163 164 * Marcel Martin 165 * Ruben Vorderman 166 * For more contributors, see <https://github.com/marcelm/xopen/graphs/contributors> 167 168 169 Links 170 ----- 171 172 * `Source code <https://github.com/marcelm/xopen/>`_ 173 * `Report an issue <https://github.com/marcelm/xopen/issues>`_ 174 * `Project page on PyPI (Python package index) <https://pypi.python.org/pypi/xopen/>`_ 175 176Platform: UNKNOWN 177Classifier: Development Status :: 5 - Production/Stable 178Classifier: License :: OSI Approved :: MIT License 179Classifier: Programming Language :: Python :: 3 180Requires-Python: >=3.6 181Provides-Extra: dev 182