1txredisapi 2========== 3 4[![Build Status](https://secure.travis-ci.org/fiorix/txredisapi.png)](http://travis-ci.org/fiorix/txredisapi) 5 6 7*For the latest source code, see <http://github.com/fiorix/txredisapi>* 8 9 10``txredisapi`` is a non-blocking client driver for the [redis](http://redis.io) 11database, written in Python. It uses [Twisted](http://twistedmatrix.com) for 12the asynchronous communication with redis. 13 14It started as a fork of the original 15[redis protocol for twisted](http://pypi.python.org/pypi/txredis/), and evolved 16into a more robust, reliable, and complete solution for applications like web 17servers. These types of applications often need a fault-tolerant pool of 18connections with multiple redis servers, making it possible to easily develop 19and maintain distributed systems. 20 21Most of the [redis commands](http://redis.io/commands) are supported, as well 22as other features such as silent reconnection, connection pools, and automatic 23sharding. 24 25This driver is distributed as part of the [cyclone](http://cyclone.io) web 26framework. 27 28### Changelog ### 29 30See [CHANGELOG.md](CHANGELOG.md) 31 32### Features ### 33 34- Connection Pools 35- Lazy Connections 36- Automatic Sharding 37- Automatic Reconnection 38- Connection using Redis Sentinel 39- Publish/Subscribe (PubSub) 40- Transactions 41- Unix Socket Connections 42 43 44Install 45------- 46 47Bear in mind that ``txredisapi.py`` is pure-python, in a single file. 48Thus, there's absolutely no need to install it. Instead, just copy it to your 49project directory and start using. 50 51Latest source code is at <https://github.com/fiorix/txredisapi>. 52 53If you have [cyclone](http://cyclone.io), you probably already have it too. 54Try the following: 55 56 $ python 57 >>> import cyclone.redis 58 >>> cyclone.redis.version 59 '1.0' 60 61However, if you really really insist in installing, get it from pypi: 62 63 pip install txredisapi 64 65 66### Unit Tests ### 67 68[Twisted Trial](http://twistedmatrix.com/trac/wiki/TwistedTrial) unit tests 69are available. Just start redis, and run ``trial ./tests``. 70If *unix sockets* are disabled in redis, it will silently skip those tests. 71 72Make sure you run `redis-cli flushall` to clean up redis after the tests. 73 74Usage 75----- 76 77First thing to do is choose what type of connection you want. The driver 78supports single connection, connection pools, sharded connections (with 79automatic distribution based on a built-in consistent hashing algorithm), 80sharded connection pools, and all of these different types can be *lazy*, 81which is explained later (because I'm lazy now). 82 83Basically, you want normal connections for simple batch clients that connect 84to redis, execute a couple of commands and disconnect - like crawlers, etc. 85 86Example: 87 88 #!/usr/bin/env python 89 # coding: utf-8 90 91 import txredisapi as redis 92 93 from twisted.internet import defer 94 from twisted.internet import reactor 95 96 97 @defer.inlineCallbacks 98 def main(): 99 rc = yield redis.Connection() 100 print rc 101 102 yield rc.set("foo", "bar") 103 v = yield rc.get("foo") 104 print "foo:", repr(v) 105 106 yield rc.disconnect() 107 108 109 if __name__ == "__main__": 110 main().addCallback(lambda ign: reactor.stop()) 111 reactor.run() 112 113 114Easily switch between ``redis.Connection()`` and ``redis.ConnectionPool()`` 115with absolutely no changes to the logic of your program. 116 117These are all the supported methods for connecting to Redis:: 118 119 Connection(host, port, dbid, reconnect, charset) 120 lazyConnection(host, port, dbid, reconnect, charset) 121 122 ConnectionPool(host, port, dbid, poolsize, reconnect, charset) 123 lazyConnectionPool(host, port, dbid, poolsize, reconnect, charset) 124 125 ShardedConnection(hosts, dbid, reconnect, charset) 126 lazyShardedConnection(hosts, dbid, reconnect, charset) 127 128 ShardedConnectionPool(hosts, dbid, poolsize, reconnect, charset) 129 lazyShardedConnectionPool(hosts, dbid, poolsize, reconnect, charset) 130 131 UnixConnection(path, dbid, reconnect, charset) 132 lazyUnixConnection(path, dbid, reconnect, charset) 133 134 UnixConnectionPool(unix, dbid, poolsize, reconnect, charset) 135 lazyUnixConnectionPool(unix, dbid, poolsize, reconnect, charset) 136 137 ShardedUnixConnection(paths, dbid, reconnect, charset) 138 lazyShardedUnixConnection(paths, dbid, reconnect, charset) 139 140 ShardedUnixConnectionPool(paths, dbid, poolsize, reconnect, charset) 141 lazyShardedUnixConnectionPool(paths, dbid, poolsize, reconnect, charset) 142 143 144The arguments are: 145 146- host: the IP address or hostname of the redis server. [default: localhost] 147- port: port number of the redis server. [default: 6379] 148- path: path of redis server's socket [default: /tmp/redis.sock] 149- dbid: database id of redis server. [default: 0] 150- poolsize: how many connections to make. [default: 10] 151- reconnect: auto-reconnect if connection is lost. [default: True] 152- charset: string encoding. Do not decode/encode strings if None. 153 [default: utf-8] 154- hosts (for sharded): list of ``host:port`` pairs. [default: None] 155- paths (for sharded): list of ``pathnames``. [default: None] 156- password: password for the redis server. [default: None] 157 158 159### Connection Handlers ### 160 161All connection methods return a connection handler object at some point. 162 163Normal connections (not lazy) return a deferred, which is fired with the 164connection handler after the connection is established. 165 166In case of connection pools, it will only fire the callback after all 167connections are set up, and ready. 168 169Connection handler is the client interface with redis. It accepts all the 170commands supported by redis, such as ``get``, ``set``, etc. It is the ``rc`` 171object in the example below. 172 173Connection handlers will automatically select one of the available connections 174in connection pools, and automatically reconnect to redis when necessary. 175 176If the connection with redis is lost, all commands will raise the 177``ConnectionError`` exception, to indicate that there's no active connection. 178However, if the ``reconnect`` argument was set to ``True`` during the 179initialization, it will continuosly try to reconnect, in background. 180 181Example: 182 183 #!/usr/bin/env python 184 # coding: utf-8 185 186 import txredisapi as redis 187 188 from twisted.internet import defer 189 from twisted.internet import reactor 190 191 192 def sleep(n): 193 d = defer.Deferred() 194 reactor.callLater(5, lambda *ign: d.callback(None)) 195 return d 196 197 198 @defer.inlineCallbacks 199 def main(): 200 rc = yield redis.ConnectionPool() 201 print rc 202 203 # set 204 yield rc.set("foo", "bar") 205 206 # sleep, so you can kill redis 207 print "sleeping for 5s, kill redis now..." 208 yield sleep(5) 209 210 try: 211 v = yield rc.get("foo") 212 print "foo:", v 213 214 yield rc.disconnect() 215 except redis.ConnectionError, e: 216 print str(e) 217 218 219 if __name__ == "__main__": 220 main().addCallback(lambda ign: reactor.stop()) 221 reactor.run() 222 223 224### Lazy Connections ### 225 226This type of connection will immediately return the connection handler object, 227even before the connection is made. 228 229It will start the connection, (or connections, in case of connection pools) in 230background, and automatically reconnect if necessary. 231 232You want lazy connections when you're writing servers, like web servers, or 233any other type of server that should not wait for the redis connection during 234the initialization of the program. 235 236The example below is a web application, which will expose redis set, get and 237delete commands over HTTP. 238 239If the database connection is down (either because redis is not running, or 240whatever reason), the web application will start normally. If connection is 241lost during the operation, nothing will change. 242 243When there's no connection, all commands will fail, therefore the web 244application will respond with HTTP 503 (Service Unavailable). It will resume to 245normal once the connection with redis is re-established. 246 247Try killing redis server after the application is running, and make a couple 248of requests. Then, start redis again and give it another try. 249 250Example: 251 252 #!/usr/bin/env python 253 # coding: utf-8 254 255 import sys 256 257 import cyclone.web 258 import cyclone.redis 259 from twisted.internet import defer 260 from twisted.internet import reactor 261 from twisted.python import log 262 263 264 class Application(cyclone.web.Application): 265 def __init__(self): 266 handlers = [ (r"/text/(.+)", TextHandler) ] 267 268 RedisMixin.setup() 269 cyclone.web.Application.__init__(self, handlers, debug=True) 270 271 272 class RedisMixin(object): 273 redis_conn = None 274 275 @classmethod 276 def setup(self): 277 RedisMixin.redis_conn = cyclone.redis.lazyConnectionPool() 278 279 280 # Provide GET, SET and DELETE redis operations via HTTP 281 class TextHandler(cyclone.web.RequestHandler, RedisMixin): 282 @defer.inlineCallbacks 283 def get(self, key): 284 try: 285 value = yield self.redis_conn.get(key) 286 except Exception, e: 287 log.msg("Redis failed to get('%s'): %s" % (key, str(e))) 288 raise cyclone.web.HTTPError(503) 289 290 self.set_header("Content-Type", "text/plain") 291 self.write("%s=%s\r\n" % (key, value)) 292 293 @defer.inlineCallbacks 294 def post(self, key): 295 value = self.get_argument("value") 296 try: 297 yield self.redis_conn.set(key, value) 298 except Exception, e: 299 log.msg("Redis failed to set('%s', '%s'): %s" % (key, value, str(e))) 300 raise cyclone.web.HTTPError(503) 301 302 self.set_header("Content-Type", "text/plain") 303 self.write("%s=%s\r\n" % (key, value)) 304 305 @defer.inlineCallbacks 306 def delete(self, key): 307 try: 308 n = yield self.redis_conn.delete(key) 309 except Exception, e: 310 log.msg("Redis failed to del('%s'): %s" % (key, str(e))) 311 raise cyclone.web.HTTPError(503) 312 313 self.set_header("Content-Type", "text/plain") 314 self.write("DEL %s=%d\r\n" % (key, n)) 315 316 317 def main(): 318 log.startLogging(sys.stdout) 319 reactor.listenTCP(8888, Application(), interface="127.0.0.1") 320 reactor.run() 321 322 323 if __name__ == "__main__": 324 main() 325 326 327This is the server running in one terminal:: 328 329 $ ./helloworld.py 330 2012-02-17 15:40:25-0500 [-] Log opened. 331 2012-02-17 15:40:25-0500 [-] Starting factory <redis.Factory instance at 0x1012f0560> 332 2012-02-17 15:40:25-0500 [-] __main__.Application starting on 8888 333 2012-02-17 15:40:25-0500 [-] Starting factory <__main__.Application instance at 0x100f42290> 334 2012-02-17 15:40:53-0500 [RedisProtocol,client] 200 POST /text/foo (127.0.0.1) 1.20ms 335 2012-02-17 15:41:01-0500 [RedisProtocol,client] 200 GET /text/foo (127.0.0.1) 0.97ms 336 2012-02-17 15:41:09-0500 [RedisProtocol,client] 200 DELETE /text/foo (127.0.0.1) 0.65ms 337 (killed redis-server) 338 2012-02-17 15:48:48-0500 [HTTPConnection,0,127.0.0.1] Redis failed to get('foo'): Not connected 339 2012-02-17 15:48:48-0500 [HTTPConnection,0,127.0.0.1] 503 GET /text/foo (127.0.0.1) 2.99ms 340 341 342And these are the requests, from ``curl`` in another terminal. 343 344Set: 345 346 $ curl -D - -d "value=bar" http://localhost:8888/text/foo 347 HTTP/1.1 200 OK 348 Content-Length: 9 349 Content-Type: text/plain 350 351 foo=bar 352 353Get: 354 355 $ curl -D - http://localhost:8888/text/foo 356 HTTP/1.1 200 OK 357 Content-Length: 9 358 Etag: "b63729aa7fa0e438eed735880951dcc21d733676" 359 Content-Type: text/plain 360 361 foo=bar 362 363Delete: 364 365 $ curl -D - -X DELETE http://localhost:8888/text/foo 366 HTTP/1.1 200 OK 367 Content-Length: 11 368 Content-Type: text/plain 369 370 DEL foo=1 371 372When redis is not running: 373 374 $ curl -D - http://localhost:8888/text/foo 375 HTTP/1.1 503 Service Unavailable 376 Content-Length: 89 377 Content-Type: text/html; charset=UTF-8 378 379 <html><title>503: Service Unavailable</title> 380 <body>503: Service Unavailable</body></html> 381 382 383### Sharded Connections ### 384 385They can be normal, or lazy connections. They can be sharded connection pools. 386Not all commands are supported on sharded connections. 387 388If the command you're trying to run is not supported on sharded connections, 389the connection handler will raise the ``NotImplementedError`` exception. 390 391Simple example with automatic sharding of keys between two redis servers: 392 393 #!/usr/bin/env python 394 # coding: utf-8 395 396 import txredisapi as redis 397 398 from twisted.internet import defer 399 from twisted.internet import reactor 400 401 402 @defer.inlineCallbacks 403 def main(): 404 rc = yield redis.ShardedConnection(["localhost:6379", "localhost:6380"]) 405 print rc 406 print "Supported methods on sharded connections:", rc.ShardedMethods 407 408 keys = [] 409 for x in xrange(100): 410 key = "foo%02d" % x 411 yield rc.set(key, "bar%02d" % x) 412 keys.append(key) 413 414 # yey! mget is supported! 415 response = yield rc.mget(keys) 416 for val in response: 417 print val 418 419 yield rc.disconnect() 420 421 422 if __name__ == "__main__": 423 main().addCallback(lambda ign: reactor.stop()) 424 reactor.run() 425 426 427### Transactions ### 428 429For obvious reasons, transactions are NOT supported on sharded connections. 430But they work pretty good on normal or lazy connections, and connection pools. 431 432NOTE: redis uses the following methods for transactions: 433 434- WATCH: synchronization 435- MULTI: start the transaction 436- EXEC: commit the transaction 437- DISCARD: you got it. 438 439Because ``exec`` is a reserved word in Python, the command to commit is 440``commit``. 441 442Example: 443 444 #!/usr/bin/env python 445 # coding: utf-8 446 447 import txredisapi as redis 448 449 from twisted.internet import defer 450 from twisted.internet import reactor 451 452 453 @defer.inlineCallbacks 454 def main(): 455 rc = yield redis.ConnectionPool() 456 457 # Remove the keys 458 yield rc.delete(["a1", "a2", "a3"]) 459 460 # Start transaction 461 t = yield rc.multi() 462 463 # These will return "QUEUED" - even t.get(key) 464 yield t.set("a1", "1") 465 yield t.set("a2", "2") 466 yield t.set("a3", "3") 467 yield t.get("a1") 468 469 # Try to call get() while in a transaction. 470 # It will fail if it's not a connection pool, or if all connections 471 # in the pool are in a transaction. 472 # Note that it's rc.get(), not the transaction object t.get(). 473 try: 474 v = yield rc.get("foo") 475 print "foo=", v 476 except Exception, e: 477 print "can't get foo:", e 478 479 # Commit, and get all responses from transaction. 480 r = yield t.commit() 481 print "commit=", repr(r) 482 483 yield rc.disconnect() 484 485 486 if __name__ == "__main__": 487 main().addCallback(lambda ign: reactor.stop()) 488 reactor.run() 489 490A "COUNTER" example, using WATCH/MULTI: 491 492 #!/usr/bin/env python 493 # coding: utf-8 494 495 import txredisapi as redis 496 497 from twisted.internet import defer 498 from twisted.internet import reactor 499 500 501 @defer.inlineCallbacks 502 def main(): 503 rc = yield redis.ConnectionPool() 504 505 # Reset keys 506 yield rc.set("a1", 0) 507 508 # Synchronize and start transaction 509 t = yield rc.watch("a1") 510 511 # Load previous value 512 a1 = yield t.get("a1") 513 514 # start the transactional pipeline 515 yield t.multi() 516 517 # modify and retrieve the new a1 value 518 yield t.set("a1", a1 + 1) 519 yield t.get("a1") 520 521 print "simulating concurrency, this will abort the transaction" 522 yield rc.set("a1", 2) 523 524 try: 525 r = yield t.commit() 526 print "commit=", repr(r) 527 except redis.WatchError, e: 528 a1 = yield rc.get("a1") 529 print "transaction has failed." 530 print "current a1 value: ", a1 531 532 yield rc.disconnect() 533 534 535 if __name__ == "__main__": 536 main().addCallback(lambda ign: reactor.stop()) 537 reactor.run() 538 539 540Calling ``commit`` will cause it to return a list with the return of all 541commands executed in the transaction. ``discard``, on the other hand, will 542normally return just an ``OK``. 543 544### Pipelining ### 545 546txredisapi automatically [pipelines](http://redis.io/topics/pipelining) all commands 547by sending next commands without waiting for the previous one to receive reply from 548server. This works even on single connections and increases performance by reducing 549number of round-trip delays and. There are two exceptions, though: 550 - no commands will be sent after blocking `blpop`, `brpop` or `brpoplpush` until 551 response is received; 552 - transaction by `multi`/`commit` are also blocking connection making all other 553 commands to wait until transaction is executed. 554 555When you need to load tons of data to Redis it might be more effective to sent 556commands in batches grouping them together offline to save on TCP packets and network 557stack overhead. You can do this using `pipeline` method to explicitly accumulate 558commands and send them to server in a single batch. Be careful to not accumulate too 559many commands: unreasonable batch size may eat up unexpected amount of memory on both 560client and server side. Group commands in batches of, for example, 10k commands instead 561of sending all your data at once. The speed will be nearly the same, but the additional 562memory used will be at max the amount needed to queue this 10k commands 563 564To send commands in a batch: 565 566 #!/usr/bin/env python 567 # coding: utf-8 568 569 import txredisapi as redis 570 571 from twisted.internet import defer 572 from twisted.internet import reactor 573 574 @defer.inlineCallbacks 575 def main(): 576 rc = yield redis.ConnectionPool() 577 578 # Start grouping commands 579 pipeline = yield rc.pipeline() 580 581 pipeline.set("foo", 123) 582 pipeline.set("bar", 987) 583 pipeline.get("foo") 584 pipeline.get("bar") 585 586 # Write those 2 sets and 2 gets to redis all at once, and wait 587 # for all replies before continuing. 588 results = yield pipeline.execute_pipeline() 589 590 print "foo:", results[2] # should be 123 591 print "bar:", results[3] # should be 987 592 593 yield rc.disconnect() 594 595 if __name__ == "__main__": 596 main().addCallback(lambda ign: reactor.stop()) 597 reactor.run() 598 599### Authentication ### 600 601This is how to authenticate:: 602 603 #!/usr/bin/env python 604 605 import txredisapi 606 from twisted.internet import defer 607 from twisted.internet import reactor 608 609 610 @defer.inlineCallbacks 611 def main(): 612 redis = yield txredisapi.Connection(password="foobared") 613 yield redis.set("foo", "bar") 614 print (yield redis.get("foo")) 615 reactor.stop() 616 617 618 if __name__ == "__main__": 619 main() 620 reactor.run() 621 622### Connection using Redis Sentinel ### 623 624`txredisapi` can discover Redis master and slaves addresses using 625[Redis Sentinel](http://redis.io/topics/sentinel) and automatically failover 626in case of server failure. 627 628 #!/usr/bin/env python 629 630 from twisted.internet.task import react 631 import txredisapi 632 633 @defer.inlineCallbacks 634 def main(reactor): 635 sentinel = txredisapi.Sentinel([("sentinel-a", 26379), ("sentinel-b", 26379), ("sentinel-c", 26379)]) 636 redis = sentinel.master_for("service_name") 637 yield redis.set("foo", "bar") 638 print (yield redis.get("foo")) 639 yield redis.disconnect() 640 yield sentinel.disconnect() 641 642 react(main) 643 644Usual connection arguments like `dbid=N` or `poolsize=N` can be specified in 645`master_for()` call. Use `sentinel.slave_for()` to connect to one of the slaves 646instead of master. 647 648Add `min_other_sentinels=N` to `Sentinel` constructor call to make it obey information 649only from sentinels that currently connected to specified number of other sentinels 650to minimize a risk of split-brain in case of network partitioning. 651 652 653Credits 654======= 655Thanks to (in no particular order): 656 657- Alexandre Fiori 658 659 - Author of txredisapi 660 661- Gleicon Moraes 662 663 - Bug fixes, testing, and [RestMQ](http://github.com/gleicon/restmq>). 664 - For writing the Consistent Hashing algorithm used for sharding. 665 666- Dorian Raymer and Ludovico Magnocavallo 667 668 - Authors of the original *redis protocol for twisted*. 669 670- Vanderson Mota 671 672 - Initial pypi setup, and patches. 673 674- Jeethu Rao 675 676 - Contributed with test cases, and other ideas like support for travis-ci 677 678- Jeremy Archer 679 680 - Minor bugfixes. 681 682- Christoph Tavan (@ctavan) 683 684 - Idea and test case for nested multi bulk replies, minor command enhancements. 685 686- dgvncsz0f 687 688 - WATCH/UNWATCH commands 689 690- Ilia Glazkov 691 692 - Free connection selection algorithm for pools. 693 - Non-unicode charset fixes. 694 - SCAN commands 695 696- Matt Pizzimenti (mjpizz) 697 698 - pipelining support 699 700- Nickolai Novik (jettify) 701 702 - update of SET command 703 704- Evgeny Tataurov (etataurov) 705 706 - Ability to use hiredis protocol parser 707 708- Ilya Skriblovsky (IlyaSkriblovsky) 709 710 - Sentinel support 711