1.. currentmodule:: motor.motor_tornado 2 3Tutorial: Using Motor With Tornado 4================================== 5 6.. These setups are redundant because I can't figure out how to make doctest 7 run a common setup *before* the setup for the two groups. A "testsetup:: *" 8 is the obvious answer, but it's run *after* group-specific setup. 9 10.. testsetup:: before-inserting-2000-docs 11 12 import pymongo 13 import motor 14 import tornado.web 15 from tornado.ioloop import IOLoop 16 from tornado import gen 17 db = motor.motor_tornado.MotorClient().test_database 18 19.. testsetup:: after-inserting-2000-docs 20 21 import pymongo 22 import motor 23 import tornado.web 24 from tornado.ioloop import IOLoop 25 from tornado import gen 26 db = motor.motor_tornado.MotorClient().test_database 27 sync_db = pymongo.MongoClient().test_database 28 sync_db.test_collection.drop() 29 sync_db.test_collection.insert_many( 30 [{'i': i} for i in range(2000)]) 31 32.. testcleanup:: * 33 34 import pymongo 35 pymongo.MongoClient().test_database.test_collection.delete_many({}) 36 37A guide to using MongoDB and Tornado with Motor. 38 39.. contents:: 40 41Tutorial Prerequisites 42---------------------- 43You can learn about MongoDB with the `MongoDB Tutorial`_ before you learn Motor. 44 45Install pip_ and then do:: 46 47 $ pip install tornado motor 48 49Once done, the following should run in the Python shell without raising an 50exception: 51 52.. doctest:: 53 54 >>> import motor.motor_tornado 55 56This tutorial also assumes that a MongoDB instance is running on the 57default host and port. Assuming you have `downloaded and installed 58<http://docs.mongodb.org/manual/installation/>`_ MongoDB, you 59can start it like so: 60 61.. code-block:: bash 62 63 $ mongod 64 65.. _pip: http://www.pip-installer.org/en/latest/installing.html 66 67.. _MongoDB Tutorial: http://docs.mongodb.org/manual/tutorial/getting-started/ 68 69Object Hierarchy 70---------------- 71Motor, like PyMongo, represents data with a 4-level object hierarchy: 72 73* :class:`MotorClient` represents a mongod process, or a cluster of them. You 74 explicitly create one of these client objects, connect it to a running mongod 75 or mongods, and use it for the lifetime of your application. 76* :class:`MotorDatabase`: Each mongod has a set of databases (distinct 77 sets of data files on disk). You can get a reference to a database from a 78 client. 79* :class:`MotorCollection`: A database has a set of collections, which 80 contain documents; you get a reference to a collection from a database. 81* :class:`MotorCursor`: Executing :meth:`~MotorCollection.find` on 82 a :class:`MotorCollection` gets a :class:`MotorCursor`, which 83 represents the set of documents matching a query. 84 85Creating a Client 86----------------- 87You typically create a single instance of :class:`MotorClient` at the time your 88application starts up. 89 90.. doctest:: before-inserting-2000-docs 91 92 >>> client = motor.motor_tornado.MotorClient() 93 94This connects to a ``mongod`` listening on the default host and port. You can 95specify the host and port like: 96 97.. doctest:: before-inserting-2000-docs 98 99 >>> client = motor.motor_tornado.MotorClient('localhost', 27017) 100 101Motor also supports `connection URIs`_: 102 103.. doctest:: before-inserting-2000-docs 104 105 >>> client = motor.motor_tornado.MotorClient('mongodb://localhost:27017') 106 107Connect to a replica set like: 108 109 >>> client = motor.motor_tornado.MotorClient('mongodb://host1,host2/?replicaSet=my-replicaset-name') 110 111.. _connection URIs: http://docs.mongodb.org/manual/reference/connection-string/ 112 113Getting a Database 114------------------ 115A single instance of MongoDB can support multiple independent 116`databases <http://docs.mongodb.org/manual/reference/glossary/#term-database>`_. 117From an open client, you can get a reference to a particular database with 118dot-notation or bracket-notation: 119 120.. doctest:: before-inserting-2000-docs 121 122 >>> db = client.test_database 123 >>> db = client['test_database'] 124 125Creating a reference to a database does no I/O and does not accept a callback 126or return a Future. 127 128Tornado Application Startup Sequence 129------------------------------------ 130Now that we can create a client and get a database, we're ready to start 131a Tornado application that uses Motor:: 132 133 db = motor.motor_tornado.MotorClient().test_database 134 135 application = tornado.web.Application([ 136 (r'/', MainHandler) 137 ], db=db) 138 139 application.listen(8888) 140 tornado.ioloop.IOLoop.current().start() 141 142There are two things to note in this code. First, the ``MotorClient`` 143constructor doesn't actually connect to the server; the client will 144initiate a connection when you attempt the first operation. 145Second, passing the database as the ``db`` keyword argument to ``Application`` 146makes it available to request handlers:: 147 148 class MainHandler(tornado.web.RequestHandler): 149 def get(self): 150 db = self.settings['db'] 151 152.. warning:: It is a common mistake to create a new client object for every 153 request; this comes at a dire performance cost. Create the client 154 when your application starts and reuse that one client for the lifetime 155 of the process, as shown in these examples. 156 157Getting a Collection 158-------------------- 159A `collection <http://docs.mongodb.org/manual/reference/glossary/#term-collection>`_ 160is a group of documents stored in MongoDB, and can be thought of as roughly 161the equivalent of a table in a relational database. Getting a 162collection in Motor works the same as getting a database: 163 164.. doctest:: before-inserting-2000-docs 165 166 >>> collection = db.test_collection 167 >>> collection = db['test_collection'] 168 169Just like getting a reference to a database, getting a reference to a 170collection does no I/O and doesn't accept a callback or return a Future. 171 172Inserting a Document 173-------------------- 174As in PyMongo, Motor represents MongoDB documents with Python dictionaries. To 175store a document in MongoDB, call :meth:`~MotorCollection.insert_one` with a 176document and a callback: 177 178.. doctest:: before-inserting-2000-docs 179 180 >>> from tornado.ioloop import IOLoop 181 >>> def my_callback(result, error): 182 ... print('result %s' % repr(result.inserted_id)) 183 ... IOLoop.current().stop() 184 ... 185 >>> document = {'key': 'value'} 186 >>> db.test_collection.insert_one(document, callback=my_callback) 187 >>> IOLoop.current().start() 188 result ObjectId('...') 189 190There are several differences to note between Motor and PyMongo. One is that, 191unlike PyMongo's :meth:`~pymongo.collection.Collection.insert_one`, Motor's has no 192return value. Another is that ``insert_one`` accepts an optional callback function. 193The function must take two arguments and it must be passed to ``insert_one`` as a 194keyword argument, like:: 195 196 db.test_collection.insert_one(document, callback=some_function) 197 198.. warning:: Passing the callback function using the ``callback=`` syntax is 199 required. (This requirement is a side-effect of the technique Motor uses to 200 wrap PyMongo.) If you pass the callback as a positional argument instead, 201 you may see an exception like ``TypeError: method takes exactly 1 argument (2 202 given)``, or ``TypeError: callable is required``, or some silent misbehavior. 203 204:meth:`insert_one` is *asynchronous*. This means it returns immediately, and 205the actual work of inserting the document into the collection is performed in 206the background. When it completes, the callback is executed. If the insert 207succeeded, the ``result`` parameter is a 208:class:`~pymongo.results.InsertOneResult` with the new document's unique id and 209the ``error`` parameter is ``None``. If there was an error, ``result`` is 210``None`` and ``error`` is an ``Exception`` object. For example, we can trigger 211a duplicate-key error by trying to insert two documents with the same unique 212id: 213 214.. doctest:: before-inserting-2000-docs 215 216 >>> loop = IOLoop.current() 217 >>> def my_callback(result, error): 218 ... print('result %s error %s' % (repr(result), repr(error))) 219 ... IOLoop.current().stop() 220 ... 221 >>> def insert_two_documents(): 222 ... db.test_collection.insert_one({'_id': 1}, callback=my_callback) 223 ... 224 >>> IOLoop.current().add_callback(insert_two_documents) 225 >>> IOLoop.current().start() 226 result <pymongo.results.InsertOneResult ...> error None 227 >>> IOLoop.current().add_callback(insert_two_documents) 228 >>> IOLoop.current().start() 229 result None error DuplicateKeyError(...) 230 231The first insert results in ``my_callback`` being called with result 1 and 232error ``None``. The second insert triggers ``my_callback`` with result None and 233a :class:`~pymongo.errors.DuplicateKeyError`. 234 235A typical beginner's mistake with Motor is to insert documents in a loop, 236not waiting for each insert to complete before beginning the next:: 237 238 >>> for i in range(2000): 239 ... db.test_collection.insert_one({'i': i}) 240 241.. Note that the above is NOT a doctest!! 242 243In PyMongo this would insert each document in turn using a single socket, but 244Motor attempts to run all the :meth:`insert_one` operations at once. This requires 245up to ``max_pool_size`` open sockets connected to MongoDB, 246which taxes the client and server. To ensure instead that all inserts use a 247single connection, wait for acknowledgment of each. This is a bit complex using 248callbacks: 249 250.. doctest:: before-inserting-2000-docs 251 252 >>> i = 0 253 >>> def do_insert(result, error): 254 ... global i 255 ... if error: 256 ... raise error 257 ... i += 1 258 ... if i < 2000: 259 ... db.test_collection.insert_one({'i': i}, callback=do_insert) 260 ... else: 261 ... IOLoop.current().stop() 262 ... 263 >>> # Start 264 >>> db.test_collection.insert_one({'i': i}, callback=do_insert) 265 >>> IOLoop.current().start() 266 267You can simplify this code with ``gen.coroutine``. 268 269Using Motor with `gen.coroutine` 270-------------------------------- 271The :mod:`tornado.gen` module lets you use generators to simplify asynchronous 272code. There are two parts to coding with generators: 273:func:`coroutine <tornado.gen.coroutine>` and 274:class:`~tornado.concurrent.Future`. 275 276First, decorate your generator function with ``@gen.coroutine``: 277 278 >>> @gen.coroutine 279 ... def do_insert(): 280 ... pass 281 282If you pass no callback to one of Motor's asynchronous methods, it returns a 283``Future``. Yield the ``Future`` instance to wait for an operation to complete 284and obtain its result: 285 286.. doctest:: before-inserting-2000-docs 287 288 >>> @gen.coroutine 289 ... def do_insert(): 290 ... for i in range(2000): 291 ... future = db.test_collection.insert_one({'i': i}) 292 ... result = yield future 293 ... 294 >>> IOLoop.current().run_sync(do_insert) 295 296In the code above, ``result`` is the ``_id`` of each inserted document. 297 298.. seealso:: :doc:`examples/bulk`. 299 300.. seealso:: :ref:`Detailed example of Motor and gen.coroutine <coroutine-example>` 301 302.. mongodoc:: insert 303 304.. doctest:: before-inserting-2000-docs 305 :hide: 306 307 >>> # Clean up from previous insert 308 >>> pymongo.MongoClient().test_database.test_collection.delete_many({}) 309 <pymongo.results.DeleteResult ...> 310 311Using native coroutines 312----------------------- 313 314Starting in Python 3.5, you can define a `native coroutine`_ with `async def` 315instead of the `gen.coroutine` decorator. Within a native coroutine, wait 316for an async operation with `await` instead of `yield`: 317 318.. doctest:: before-inserting-2000-docs 319 320 >>> async def do_insert(): 321 ... for i in range(2000): 322 ... result = await db.test_collection.insert_one({'i': i}) 323 ... 324 >>> IOLoop.current().run_sync(do_insert) 325 326Within a native coroutine, the syntax to use Motor with Tornado or asyncio 327is often identical. 328 329.. _native coroutine: https://www.python.org/dev/peps/pep-0492/ 330 331Getting a Single Document With :meth:`~MotorCollection.find_one` 332---------------------------------------------------------------- 333Use :meth:`~MotorCollection.find_one` to get the first document that 334matches a query. For example, to get a document where the value for key "i" is 335less than 1: 336 337.. doctest:: after-inserting-2000-docs 338 339 >>> @gen.coroutine 340 ... def do_find_one(): 341 ... document = yield db.test_collection.find_one({'i': {'$lt': 1}}) 342 ... pprint.pprint(document) 343 ... 344 >>> IOLoop.current().run_sync(do_find_one) 345 {'_id': ObjectId('...'), 'i': 0} 346 347The result is a dictionary matching the one that we inserted previously. 348 349The returned document contains an ``"_id"``, which was 350automatically added on insert. 351 352(We use ``pprint`` here instead of ``print`` to ensure the document's key names 353are sorted the same in your output as ours.) 354 355.. mongodoc:: find 356 357Querying for More Than One Document 358----------------------------------- 359Use :meth:`~MotorCollection.find` to query for a set of documents. 360:meth:`~MotorCollection.find` does no I/O and does not take a callback, 361it merely creates a :class:`MotorCursor` instance. The query is actually 362executed on the server when you call :meth:`~MotorCursor.to_list` or 363:meth:`~MotorCursor.each`, or yield :attr:`~motor.motor_tornado.MotorCursor.fetch_next`. 364 365To find all documents with "i" less than 5: 366 367.. doctest:: after-inserting-2000-docs 368 369 >>> @gen.coroutine 370 ... def do_find(): 371 ... cursor = db.test_collection.find({'i': {'$lt': 5}}).sort('i') 372 ... for document in (yield cursor.to_list(length=100)): 373 ... pprint.pprint(document) 374 ... 375 >>> IOLoop.current().run_sync(do_find) 376 {'_id': ObjectId('...'), 'i': 0} 377 {'_id': ObjectId('...'), 'i': 1} 378 {'_id': ObjectId('...'), 'i': 2} 379 {'_id': ObjectId('...'), 'i': 3} 380 {'_id': ObjectId('...'), 'i': 4} 381 382A ``length`` argument is required when you call to_list to prevent Motor from 383buffering an unlimited number of documents. 384 385To get one document at a time with :attr:`~motor.motor_tornado.MotorCursor.fetch_next` 386and :meth:`~MotorCursor.next_object`: 387 388.. doctest:: after-inserting-2000-docs 389 390 >>> @gen.coroutine 391 ... def do_find(): 392 ... cursor = db.test_collection.find({'i': {'$lt': 5}}) 393 ... while (yield cursor.fetch_next): 394 ... document = cursor.next_object() 395 ... pprint.pprint(document) 396 ... 397 >>> IOLoop.current().run_sync(do_find) 398 {'_id': ObjectId('...'), 'i': 0} 399 {'_id': ObjectId('...'), 'i': 1} 400 {'_id': ObjectId('...'), 'i': 2} 401 {'_id': ObjectId('...'), 'i': 3} 402 {'_id': ObjectId('...'), 'i': 4} 403 404You can apply a sort, limit, or skip to a query before you begin iterating: 405 406.. doctest:: after-inserting-2000-docs 407 408 >>> @gen.coroutine 409 ... def do_find(): 410 ... c = db.test_collection 411 ... cursor = c.find({'i': {'$lt': 5}}) 412 ... # Modify the query before iterating 413 ... cursor.sort('i', -1).limit(2).skip(2) 414 ... while (yield cursor.fetch_next): 415 ... document = cursor.next_object() 416 ... pprint.pprint(document) 417 ... 418 >>> IOLoop.current().run_sync(do_find) 419 {'_id': ObjectId('...'), 'i': 2} 420 {'_id': ObjectId('...'), 'i': 1} 421 422``fetch_next`` does not actually retrieve each document from the server 423individually; it gets documents efficiently in `large batches`_. 424 425.. _`large batches`: https://docs.mongodb.com/manual/tutorial/iterate-a-cursor/#cursor-batches 426 427`async for` 428----------- 429 430In a native coroutine defined with `async def`, replace the while-loop with 431`async for`: 432 433.. doctest:: after-inserting-2000-docs 434 435 >>> async def do_find(): 436 ... c = db.test_collection 437 ... async for document in c.find({'i': {'$lt': 2}}): 438 ... pprint.pprint(document) 439 ... 440 >>> IOLoop.current().run_sync(do_find) 441 {'_id': ObjectId('...'), 'i': 0} 442 {'_id': ObjectId('...'), 'i': 1} 443 444This version of the code is dramatically faster. 445 446Counting Documents 447------------------ 448Use :meth:`~MotorCursor.count` to determine the number of documents in 449a collection, or the number of documents that match a query: 450 451.. doctest:: after-inserting-2000-docs 452 453 >>> @gen.coroutine 454 ... def do_count(): 455 ... n = yield db.test_collection.find().count() 456 ... print('%s documents in collection' % n) 457 ... n = yield db.test_collection.find({'i': {'$gt': 1000}}).count() 458 ... print('%s documents where i > 1000' % n) 459 ... 460 >>> IOLoop.current().run_sync(do_count) 461 2000 documents in collection 462 999 documents where i > 1000 463 464:meth:`~MotorCursor.count` uses the *count command* internally; we'll 465cover commands_ below. 466 467.. seealso:: `Count command <http://docs.mongodb.org/manual/reference/command/count/>`_ 468 469Updating Documents 470------------------ 471 472:meth:`~MotorCollection.replace_one` changes a document. It requires two 473parameters: a *query* that specifies which document to replace, and a 474replacement document. The query follows the same syntax as for :meth:`find` or 475:meth:`find_one`. To replace a document: 476 477.. doctest:: after-inserting-2000-docs 478 479 >>> @gen.coroutine 480 ... def do_replace(): 481 ... coll = db.test_collection 482 ... old_document = yield coll.find_one({'i': 50}) 483 ... print('found document: %s' % pprint.pformat(old_document)) 484 ... _id = old_document['_id'] 485 ... result = yield coll.replace_one({'_id': _id}, {'key': 'value'}) 486 ... print('replaced %s document' % result.modified_count) 487 ... new_document = yield coll.find_one({'_id': _id}) 488 ... print('document is now %s' % pprint.pformat(new_document)) 489 ... 490 >>> IOLoop.current().run_sync(do_replace) 491 found document: {'_id': ObjectId('...'), 'i': 50} 492 replaced 1 document 493 document is now {'_id': ObjectId('...'), 'key': 'value'} 494 495You can see that :meth:`replace_one` replaced everything in the old document 496except its ``_id`` with the new document. 497 498Use :meth:`~MotorCollection.update_one` with MongoDB's modifier operators to 499update part of a document and leave the 500rest intact. We'll find the document whose "i" is 51 and use the ``$set`` 501operator to set "key" to "value": 502 503.. doctest:: after-inserting-2000-docs 504 505 >>> @gen.coroutine 506 ... def do_update(): 507 ... coll = db.test_collection 508 ... result = yield coll.update_one({'i': 51}, {'$set': {'key': 'value'}}) 509 ... print('updated %s document' % result.modified_count) 510 ... new_document = yield coll.find_one({'i': 51}) 511 ... print('document is now %s' % pprint.pformat(new_document)) 512 ... 513 >>> IOLoop.current().run_sync(do_update) 514 updated 1 document 515 document is now {'_id': ObjectId('...'), 'i': 51, 'key': 'value'} 516 517"key" is set to "value" and "i" is still 51. 518 519:meth:`update_one` only affects the first document it finds, you can 520update all of them with :meth:`update_many`:: 521 522 yield coll.update_many({'i': {'$gt': 100}}, 523 {'$set': {'key': 'value'}}) 524 525.. mongodoc:: update 526 527Removing Documents 528------------------ 529 530:meth:`~MotorCollection.delete_many` takes a query with the same syntax as 531:meth:`~MotorCollection.find`. 532:meth:`delete_many` immediately removes all matching documents. 533 534.. doctest:: after-inserting-2000-docs 535 536 >>> @gen.coroutine 537 ... def do_delete_many(): 538 ... coll = db.test_collection 539 ... n = yield coll.count() 540 ... print('%s documents before calling delete_many()' % n) 541 ... result = yield db.test_collection.delete_many({'i': {'$gte': 1000}}) 542 ... print('%s documents after' % (yield coll.count())) 543 ... 544 >>> IOLoop.current().run_sync(do_delete_many) 545 2000 documents before calling delete_many() 546 1000 documents after 547 548.. mongodoc:: remove 549 550Commands 551-------- 552Besides the "CRUD" operations--insert, update, delete, and find--all other 553operations on MongoDB are commands. Run them using 554the :meth:`~MotorDatabase.command` method on :class:`MotorDatabase`: 555 556.. doctest:: after-inserting-2000-docs 557 558 >>> from bson import SON 559 >>> @gen.coroutine 560 ... def use_count_command(): 561 ... response = yield db.command(SON([("count", "test_collection")])) 562 ... print('response: %s' % pprint.pformat(response)) 563 ... 564 >>> IOLoop.current().run_sync(use_count_command) 565 response: {'n': 1000, 'ok': 1.0, ...} 566 567Since the order of command parameters matters, don't use a Python dict to pass 568the command's parameters. Instead, make a habit of using :class:`bson.SON`, 569from the ``bson`` module included with PyMongo:: 570 571 yield db.command(SON([("distinct", "test_collection"), ("key", "my_key")])) 572 573Many commands have special helper methods, such as 574:meth:`~MotorDatabase.create_collection` or 575:meth:`~MotorCollection.aggregate`, but these are just conveniences atop 576the basic :meth:`command` method. 577 578.. mongodoc:: commands 579 580Further Reading 581--------------- 582The handful of classes and methods introduced here are sufficient for daily 583tasks. The API documentation for :class:`MotorClient`, :class:`MotorDatabase`, 584:class:`MotorCollection`, and :class:`MotorCursor` provides a 585reference to Motor's complete feature set. 586 587Learning to use the MongoDB driver is just the beginning, of course. For 588in-depth instruction in MongoDB itself, see `The MongoDB Manual`_. 589 590.. _The MongoDB Manual: http://docs.mongodb.org/manual/ 591