1Metadata-Version: 2.1 2Name: diskcache 3Version: 4.1.0 4Summary: Disk Cache -- Disk and file backed persistent cache. 5Home-page: http://www.grantjenks.com/docs/diskcache/ 6Author: Grant Jenks 7Author-email: contact@grantjenks.com 8License: Apache 2.0 9Platform: UNKNOWN 10Classifier: Development Status :: 5 - Production/Stable 11Classifier: Intended Audience :: Developers 12Classifier: License :: OSI Approved :: Apache Software License 13Classifier: Natural Language :: English 14Classifier: Programming Language :: Python 15Classifier: Programming Language :: Python :: 2 16Classifier: Programming Language :: Python :: 2.7 17Classifier: Programming Language :: Python :: 3 18Classifier: Programming Language :: Python :: 3.4 19Classifier: Programming Language :: Python :: 3.5 20Classifier: Programming Language :: Python :: 3.6 21Classifier: Programming Language :: Python :: 3.7 22Classifier: Programming Language :: Python :: Implementation :: CPython 23Classifier: Programming Language :: Python :: Implementation :: PyPy 24 25DiskCache: Disk Backed Cache 26============================ 27 28`DiskCache`_ is an Apache2 licensed disk and file backed cache library, written 29in pure-Python, and compatible with Django. 30 31The cloud-based computing of 2019 puts a premium on memory. Gigabytes of empty 32space is left on disks as processes vie for memory. Among these processes is 33Memcached (and sometimes Redis) which is used as a cache. Wouldn't it be nice 34to leverage empty disk space for caching? 35 36Django is Python's most popular web framework and ships with several caching 37backends. Unfortunately the file-based cache in Django is essentially 38broken. The culling method is random and large caches repeatedly scan a cache 39directory which slows linearly with growth. Can you really allow it to take 40sixty milliseconds to store a key in a cache with a thousand items? 41 42In Python, we can do better. And we can do it in pure-Python! 43 44:: 45 46 In [1]: import pylibmc 47 In [2]: client = pylibmc.Client(['127.0.0.1'], binary=True) 48 In [3]: client[b'key'] = b'value' 49 In [4]: %timeit client[b'key'] 50 51 10000 loops, best of 3: 25.4 µs per loop 52 53 In [5]: import diskcache as dc 54 In [6]: cache = dc.Cache('tmp') 55 In [7]: cache[b'key'] = b'value' 56 In [8]: %timeit cache[b'key'] 57 58 100000 loops, best of 3: 11.8 µs per loop 59 60**Note:** Micro-benchmarks have their place but are not a substitute for real 61measurements. DiskCache offers cache benchmarks to defend its performance 62claims. Micro-optimizations are avoided but your mileage may vary. 63 64DiskCache efficiently makes gigabytes of storage space available for 65caching. By leveraging rock-solid database libraries and memory-mapped files, 66cache performance can match and exceed industry-standard solutions. There's no 67need for a C compiler or running another process. Performance is a feature and 68testing has 100% coverage with unit tests and hours of stress. 69 70Testimonials 71------------ 72 73`Daren Hasenkamp`_, Founder -- 74 75 "It's a useful, simple API, just like I love about Redis. It has reduced 76 the amount of queries hitting my Elasticsearch cluster by over 25% for a 77 website that gets over a million users/day (100+ hits/second)." 78 79`Mathias Petermann`_, Senior Linux System Engineer -- 80 81 "I implemented it into a wrapper for our Ansible lookup modules and we were 82 able to speed up some Ansible runs by almost 3 times. DiskCache is saving 83 us a ton of time." 84 85Does your company or website use `DiskCache`_? Send us a `message 86<contact@grantjenks.com>`_ and let us know. 87 88.. _`Daren Hasenkamp`: https://www.linkedin.com/in/daren-hasenkamp-93006438/ 89.. _`Mathias Petermann`: https://www.linkedin.com/in/mathias-petermann-a8aa273b/ 90 91Features 92-------- 93 94- Pure-Python 95- Fully Documented 96- Benchmark comparisons (alternatives, Django cache backends) 97- 100% test coverage 98- Hours of stress testing 99- Performance matters 100- Django compatible API 101- Thread-safe and process-safe 102- Supports multiple eviction policies (LRU and LFU included) 103- Keys support "tag" metadata and eviction 104- Developed on Python 3.7 105- Tested on CPython 2.7, 3.4, 3.5, 3.6, 3.7 and PyPy 106- Tested on Linux, Mac OS X, and Windows 107- Tested using Travis CI and AppVeyor CI 108 109.. image:: https://api.travis-ci.org/grantjenks/python-diskcache.svg?branch=master 110 :target: http://www.grantjenks.com/docs/diskcache/ 111 112.. image:: https://ci.appveyor.com/api/projects/status/github/grantjenks/python-diskcache?branch=master&svg=true 113 :target: http://www.grantjenks.com/docs/diskcache/ 114 115Quickstart 116---------- 117 118Installing `DiskCache`_ is simple with `pip <http://www.pip-installer.org/>`_:: 119 120 $ pip install diskcache 121 122You can access documentation in the interpreter with Python's built-in help 123function:: 124 125 >>> import diskcache 126 >>> help(diskcache) 127 128The core of `DiskCache`_ is three data types intended for caching. `Cache`_ 129objects manage a SQLite database and filesystem directory to store key and 130value pairs. `FanoutCache`_ provides a sharding layer to utilize multiple 131caches and `DjangoCache`_ integrates that with `Django`_:: 132 133 >>> from diskcache import Cache, FanoutCache, DjangoCache 134 >>> help(Cache) 135 >>> help(FanoutCache) 136 >>> help(DjangoCache) 137 138Built atop the caching data types, are `Deque`_ and `Index`_ which work as a 139cross-process, persistent replacements for Python's ``collections.deque`` and 140``dict``. These implement the sequence and mapping container base classes:: 141 142 >>> from diskcache import Deque, Index 143 >>> help(Deque) 144 >>> help(Index) 145 146Finally, a number of `recipes`_ for cross-process synchronization are provided 147using an underlying cache. Features like memoization with cache stampede 148prevention, cross-process locking, and cross-process throttling are available:: 149 150 >>> from diskcache import memoize_stampede, Lock, throttle 151 >>> help(memoize_stampede) 152 >>> help(Lock) 153 >>> help(throttle) 154 155Python's docstrings are a quick way to get started but not intended as a 156replacement for the `DiskCache Tutorial`_ and `DiskCache API Reference`_. 157 158.. _`Cache`: http://www.grantjenks.com/docs/diskcache/tutorial.html#cache 159.. _`FanoutCache`: http://www.grantjenks.com/docs/diskcache/tutorial.html#fanoutcache 160.. _`DjangoCache`: http://www.grantjenks.com/docs/diskcache/tutorial.html#djangocache 161.. _`Django`: https://www.djangoproject.com/ 162.. _`Deque`: http://www.grantjenks.com/docs/diskcache/tutorial.html#deque 163.. _`Index`: http://www.grantjenks.com/docs/diskcache/tutorial.html#index 164.. _`recipes`: http://www.grantjenks.com/docs/diskcache/tutorial.html#recipes 165 166User Guide 167---------- 168 169For those wanting more details, this part of the documentation describes 170tutorial, benchmarks, API, and development. 171 172* `DiskCache Tutorial`_ 173* `DiskCache Cache Benchmarks`_ 174* `DiskCache DjangoCache Benchmarks`_ 175* `Case Study: Web Crawler`_ 176* `Case Study: Landing Page Caching`_ 177* `Talk: All Things Cached - SF Python 2017 Meetup`_ 178* `DiskCache API Reference`_ 179* `DiskCache Development`_ 180 181.. _`DiskCache Tutorial`: http://www.grantjenks.com/docs/diskcache/tutorial.html 182.. _`DiskCache Cache Benchmarks`: http://www.grantjenks.com/docs/diskcache/cache-benchmarks.html 183.. _`DiskCache DjangoCache Benchmarks`: http://www.grantjenks.com/docs/diskcache/djangocache-benchmarks.html 184.. _`Talk: All Things Cached - SF Python 2017 Meetup`: http://www.grantjenks.com/docs/diskcache/sf-python-2017-meetup-talk.html 185.. _`Case Study: Web Crawler`: http://www.grantjenks.com/docs/diskcache/case-study-web-crawler.html 186.. _`Case Study: Landing Page Caching`: http://www.grantjenks.com/docs/diskcache/case-study-landing-page-caching.html 187.. _`DiskCache API Reference`: http://www.grantjenks.com/docs/diskcache/api.html 188.. _`DiskCache Development`: http://www.grantjenks.com/docs/diskcache/development.html 189 190Comparisons 191----------- 192 193Comparisons to popular projects related to `DiskCache`_. 194 195Key-Value Stores 196................ 197 198`DiskCache`_ is mostly a simple key-value store. Feature comparisons with four 199other projects are shown in the tables below. 200 201* `dbm`_ is part of Python's standard library and implements a generic 202 interface to variants of the DBM database — dbm.gnu or dbm.ndbm. If none of 203 these modules is installed, the slow-but-simple dbm.dumb is used. 204* `shelve`_ is part of Python's standard library and implements a “shelf” as a 205 persistent, dictionary-like object. The difference with “dbm” databases is 206 that the values can be anything that the pickle module can handle. 207* `sqlitedict`_ is a lightweight wrapper around Python's sqlite3 database with 208 a simple, Pythonic dict-like interface and support for multi-thread 209 access. Keys are arbitrary strings, values arbitrary pickle-able objects. 210* `pickleDB`_ is a lightweight and simple key-value store. It is built upon 211 Python's simplejson module and was inspired by Redis. It is licensed with the 212 BSD three-caluse license. 213 214.. _`dbm`: https://docs.python.org/3/library/dbm.html 215.. _`shelve`: https://docs.python.org/3/library/shelve.html 216.. _`sqlitedict`: https://github.com/RaRe-Technologies/sqlitedict 217.. _`pickleDB`: https://pythonhosted.org/pickleDB/ 218 219**Features** 220 221================ ============= ========= ========= ============ ============ 222Feature diskcache dbm shelve sqlitedict pickleDB 223================ ============= ========= ========= ============ ============ 224Atomic? Always Maybe Maybe Maybe No 225Persistent? Yes Yes Yes Yes Yes 226Thread-safe? Yes No No Yes No 227Process-safe? Yes No No Maybe No 228Backend? SQLite DBM DBM SQLite File 229Serialization? Customizable None Pickle Customizable JSON 230Data Types? Mapping/Deque Mapping Mapping Mapping Mapping 231Ordering? Insert/Sorted None None None None 232Eviction? LRU/LFU/more None None None None 233Vacuum? Automatic Maybe Maybe Manual Automatic 234Transactions? Yes No No Maybe No 235Multiprocessing? Yes No No No No 236Forkable? Yes No No No No 237Metadata? Yes No No No No 238================ ============= ========= ========= ============ ============ 239 240**Quality** 241 242================ ============= ========= ========= ============ ============ 243Project diskcache dbm shelve sqlitedict pickleDB 244================ ============= ========= ========= ============ ============ 245Tests? Yes Yes Yes Yes Yes 246Coverage? Yes Yes Yes Yes No 247Stress? Yes No No No No 248CI Tests? Linux/Windows Yes Yes Linux No 249Python? 2/3/PyPy All All 2/3 2/3 250License? Apache2 Python Python Apache2 3-Clause BSD 251Docs? Extensive Summary Summary Readme Summary 252Benchmarks? Yes No No No No 253Sources? GitHub GitHub GitHub GitHub GitHub 254Pure-Python? Yes Yes Yes Yes Yes 255Server? No No No No No 256Integrations? Django None None None None 257================ ============= ========= ========= ============ ============ 258 259**Timings** 260 261These are rough measurements. See `DiskCache Cache Benchmarks`_ for more 262rigorous data. 263 264================ ============= ========= ========= ============ ============ 265Project diskcache dbm shelve sqlitedict pickleDB 266================ ============= ========= ========= ============ ============ 267get 25 µs 36 µs 41 µs 513 µs 92 µs 268set 198 µs 900 µs 928 µs 697 µs 1,020 µs 269delete 248 µs 740 µs 702 µs 1,717 µs 1,020 µs 270================ ============= ========= ========= ============ ============ 271 272Caching Libraries 273................. 274 275* `joblib.Memory`_ provides caching functions and works by explicitly saving 276 the inputs and outputs to files. It is designed to work with non-hashable and 277 potentially large input and output data types such as numpy arrays. 278* `klepto`_ extends Python’s `lru_cache` to utilize different keymaps and 279 alternate caching algorithms, such as `lfu_cache` and `mru_cache`. Klepto 280 uses a simple dictionary-sytle interface for all caches and archives. 281 282.. _`klepto`: https://pypi.org/project/klepto/ 283.. _`joblib.Memory`: https://joblib.readthedocs.io/en/latest/memory.html 284 285Data Structures 286............... 287 288* `dict`_ is a mapping object that maps hashable keys to arbitrary 289 values. Mappings are mutable objects. There is currently only one standard 290 Python mapping type, the dictionary. 291* `pandas`_ is a Python package providing fast, flexible, and expressive data 292 structures designed to make working with “relational” or “labeled” data both 293 easy and intuitive. 294* `Sorted Containers`_ is an Apache2 licensed sorted collections library, 295 written in pure-Python, and fast as C-extensions. Sorted Containers 296 implements sorted list, sorted dictionary, and sorted set data types. 297 298.. _`dict`: https://docs.python.org/3/library/stdtypes.html#typesmapping 299.. _`pandas`: https://pandas.pydata.org/ 300.. _`Sorted Containers`: http://www.grantjenks.com/docs/sortedcontainers/ 301 302Pure-Python Databases 303..................... 304 305* `ZODB`_ supports an isomorphic interface for database operations which means 306 there's little impact on your code to make objects persistent and there's no 307 database mapper that partially hides the datbase. 308* `CodernityDB`_ is an open source, pure-Python, multi-platform, schema-less, 309 NoSQL database and includes an HTTP server version, and a Python client 310 library that aims to be 100% compatible with the embedded version. 311* `TinyDB`_ is a tiny, document oriented database optimized for your 312 happiness. If you need a simple database with a clean API that just works 313 without lots of configuration, TinyDB might be the right choice for you. 314 315.. _`ZODB`: http://www.zodb.org/ 316.. _`CodernityDB`: https://pypi.org/project/CodernityDB/ 317.. _`TinyDB`: https://tinydb.readthedocs.io/ 318 319Object Relational Mappings (ORM) 320................................ 321 322* `Django ORM`_ provides models that are the single, definitive source of 323 information about data and contains the essential fields and behaviors of the 324 stored data. Generally, each model maps to a single SQL database table. 325* `SQLAlchemy`_ is the Python SQL toolkit and Object Relational Mapper that 326 gives application developers the full power and flexibility of SQL. It 327 provides a full suite of well known enterprise-level persistence patterns. 328* `Peewee`_ is a simple and small ORM. It has few (but expressive) concepts, 329 making it easy to learn and intuitive to use. Peewee supports Sqlite, MySQL, 330 and PostgreSQL with tons of extensions. 331* `SQLObject`_ is a popular Object Relational Manager for providing an object 332 interface to your database, with tables as classes, rows as instances, and 333 columns as attributes. 334* `Pony ORM`_ is a Python ORM with beautiful query syntax. Use Python syntax 335 for interacting with the database. Pony translates such queries into SQL and 336 executes them in the database in the most efficient way. 337 338.. _`Django ORM`: https://docs.djangoproject.com/en/dev/topics/db/ 339.. _`SQLAlchemy`: https://www.sqlalchemy.org/ 340.. _`Peewee`: http://docs.peewee-orm.com/ 341.. _`dataset`: https://dataset.readthedocs.io/ 342.. _`SQLObject`: http://sqlobject.org/ 343.. _`Pony ORM`: https://ponyorm.com/ 344 345SQL Databases 346............. 347 348* `SQLite`_ is part of Python's standard library and provides a lightweight 349 disk-based database that doesn’t require a separate server process and allows 350 accessing the database using a nonstandard variant of the SQL query language. 351* `MySQL`_ is one of the world’s most popular open source databases and has 352 become a leading database choice for web-based applications. MySQL includes a 353 standardized database driver for Python platforms and development. 354* `PostgreSQL`_ is a powerful, open source object-relational database system 355 with over 30 years of active development. Psycopg is the most popular 356 PostgreSQL adapter for the Python programming language. 357* `Oracle DB`_ is a relational database management system (RDBMS) from the 358 Oracle Corporation. Originally developed in 1977, Oracle DB is one of the 359 most trusted and widely used enterprise relational database engines. 360* `Microsoft SQL Server`_ is a relational database management system developed 361 by Microsoft. As a database server, it stores and retrieves data as requested 362 by other software applications. 363 364.. _`SQLite`: https://docs.python.org/3/library/sqlite3.html 365.. _`MySQL`: https://dev.mysql.com/downloads/connector/python/ 366.. _`PostgreSQL`: http://initd.org/psycopg/ 367.. _`Oracle DB`: https://pypi.org/project/cx_Oracle/ 368.. _`Microsoft SQL Server`: https://pypi.org/project/pyodbc/ 369 370Other Databases 371............... 372 373* `Memcached`_ is free and open source, high-performance, distributed memory 374 object caching system, generic in nature, but intended for use in speeding up 375 dynamic web applications by alleviating database load. 376* `Redis`_ is an open source, in-memory data structure store, used as a 377 database, cache and message broker. It supports data structures such as 378 strings, hashes, lists, sets, sorted sets with range queries, and more. 379* `MongoDB`_ is a cross-platform document-oriented database program. Classified 380 as a NoSQL database program, MongoDB uses JSON-like documents with 381 schema. PyMongo is the recommended way to work with MongoDB from Python. 382* `LMDB`_ is a lightning-fast, memory-mapped database. With memory-mapped 383 files, it has the read performance of a pure in-memory database while 384 retaining the persistence of standard disk-based databases. 385* `BerkeleyDB`_ is a software library intended to provide a high-performance 386 embedded database for key/value data. Berkeley DB is a programmatic toolkit 387 that provides built-in database support for desktop and server applications. 388* `LevelDB`_ is a fast key-value storage library written at Google that 389 provides an ordered mapping from string keys to string values. Data is stored 390 sorted by key and users can provide a custom comparison function. 391 392.. _`Memcached`: https://pypi.org/project/python-memcached/ 393.. _`MongoDB`: https://api.mongodb.com/python/current/ 394.. _`Redis`: https://redis.io/clients#python 395.. _`LMDB`: https://lmdb.readthedocs.io/ 396.. _`BerkeleyDB`: https://pypi.org/project/bsddb3/ 397.. _`LevelDB`: https://plyvel.readthedocs.io/ 398 399Reference 400--------- 401 402* `DiskCache Documentation`_ 403* `DiskCache at PyPI`_ 404* `DiskCache at GitHub`_ 405* `DiskCache Issue Tracker`_ 406 407.. _`DiskCache Documentation`: http://www.grantjenks.com/docs/diskcache/ 408.. _`DiskCache at PyPI`: https://pypi.python.org/pypi/diskcache/ 409.. _`DiskCache at GitHub`: https://github.com/grantjenks/python-diskcache/ 410.. _`DiskCache Issue Tracker`: https://github.com/grantjenks/python-diskcache/issues/ 411 412License 413------- 414 415Copyright 2016-2019 Grant Jenks 416 417Licensed under the Apache License, Version 2.0 (the "License"); you may not use 418this file except in compliance with the License. You may obtain a copy of the 419License at 420 421 http://www.apache.org/licenses/LICENSE-2.0 422 423Unless required by applicable law or agreed to in writing, software distributed 424under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR 425CONDITIONS OF ANY KIND, either express or implied. See the License for the 426specific language governing permissions and limitations under the License. 427 428.. _`DiskCache`: http://www.grantjenks.com/docs/diskcache/ 429 430 431
