1========================== 2testtools for test authors 3========================== 4 5If you are writing tests for a Python project and you (rather wisely) want to 6use testtools to do so, this is the manual for you. 7 8We assume that you already know Python and that you know something about 9automated testing already. 10 11If you are a test author of an unusually large or unusually unusual test 12suite, you might be interested in :doc:`for-framework-folk`. 13 14You might also be interested in the `testtools API docs`_. 15 16 17Introduction 18============ 19 20testtools is a set of extensions to Python's standard unittest module. 21Writing tests with testtools is very much like writing tests with standard 22Python, or with Twisted's "trial_", or nose_, except a little bit easier and 23more enjoyable. 24 25Below, we'll try to give some examples of how to use testtools in its most 26basic way, as well as a sort of feature-by-feature breakdown of the cool bits 27that you could easily miss. 28 29 30The basics 31========== 32 33Here's what a basic testtools unit tests look like:: 34 35 from testtools import TestCase 36 from myproject import silly 37 38 class TestSillySquare(TestCase): 39 """Tests for silly square function.""" 40 41 def test_square(self): 42 # 'square' takes a number and multiplies it by itself. 43 result = silly.square(7) 44 self.assertEqual(result, 49) 45 46 def test_square_bad_input(self): 47 # 'square' raises a TypeError if it's given bad input, say a 48 # string. 49 self.assertRaises(TypeError, silly.square, "orange") 50 51 52Here you have a class that inherits from ``testtools.TestCase`` and bundles 53together a bunch of related tests. The tests themselves are methods on that 54class that begin with ``test_``. 55 56Running your tests 57------------------ 58 59You can run these tests in many ways. testtools provides a very basic 60mechanism for doing so:: 61 62 $ python -m testtools.run exampletest 63 Tests running... 64 Ran 2 tests in 0.000s 65 66 OK 67 68where 'exampletest' is a module that contains unit tests. By default, 69``testtools.run`` will *not* recursively search the module or package for unit 70tests. To do this, you will need to either have the discover_ module 71installed or have Python 2.7 or later, and then run:: 72 73 $ python -m testtools.run discover packagecontainingtests 74 75For more information see the Python 2.7 unittest documentation, or:: 76 77 python -m testtools.run --help 78 79As your testing needs grow and evolve, you will probably want to use a more 80sophisticated test runner. There are many of these for Python, and almost all 81of them will happily run testtools tests. In particular: 82 83* testrepository_ 84* Trial_ 85* nose_ 86* unittest2_ 87* `zope.testrunner`_ (aka zope.testing) 88 89From now on, we'll assume that you know how to run your tests. 90 91Running test with Distutils 92~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 93 94If you are using Distutils_ to build your Python project, you can use the testtools 95Distutils_ command to integrate testtools into your Distutils_ workflow:: 96 97 from distutils.core import setup 98 from testtools import TestCommand 99 setup(name='foo', 100 version='1.0', 101 py_modules=['foo'], 102 cmdclass={'test': TestCommand} 103 ) 104 105You can then run:: 106 107 $ python setup.py test -m exampletest 108 Tests running... 109 Ran 2 tests in 0.000s 110 111 OK 112 113For more information about the capabilities of the `TestCommand` command see:: 114 115 $ python setup.py test --help 116 117You can use the `setup configuration`_ to specify the default behavior of the 118`TestCommand` command. 119 120Assertions 121========== 122 123The core of automated testing is making assertions about the way things are, 124and getting a nice, helpful, informative error message when things are not as 125they ought to be. 126 127All of the assertions that you can find in Python standard unittest_ can be 128found in testtools (remember, testtools extends unittest). testtools changes 129the behaviour of some of those assertions slightly and adds some new 130assertions that you will almost certainly find useful. 131 132 133Improved assertRaises 134--------------------- 135 136``TestCase.assertRaises`` returns the caught exception. This is useful for 137asserting more things about the exception than just the type:: 138 139 def test_square_bad_input(self): 140 # 'square' raises a TypeError if it's given bad input, say a 141 # string. 142 e = self.assertRaises(TypeError, silly.square, "orange") 143 self.assertEqual("orange", e.bad_value) 144 self.assertEqual("Cannot square 'orange', not a number.", str(e)) 145 146Note that this is incompatible with the ``assertRaises`` in unittest2 and 147Python2.7. 148 149 150ExpectedException 151----------------- 152 153If you are using a version of Python that supports the ``with`` context 154manager syntax, you might prefer to use that syntax to ensure that code raises 155particular errors. ``ExpectedException`` does just that. For example:: 156 157 def test_square_root_bad_input_2(self): 158 # 'square' raises a TypeError if it's given bad input. 159 with ExpectedException(TypeError, "Cannot square.*"): 160 silly.square('orange') 161 162The first argument to ``ExpectedException`` is the type of exception you 163expect to see raised. The second argument is optional, and can be either a 164regular expression or a matcher. If it is a regular expression, the ``str()`` 165of the raised exception must match the regular expression. If it is a matcher, 166then the raised exception object must match it. The optional third argument 167``msg`` will cause the raised error to be annotated with that message. 168 169 170assertIn, assertNotIn 171--------------------- 172 173These two assertions check whether a value is in a sequence and whether a 174value is not in a sequence. They are "assert" versions of the ``in`` and 175``not in`` operators. For example:: 176 177 def test_assert_in_example(self): 178 self.assertIn('a', 'cat') 179 self.assertNotIn('o', 'cat') 180 self.assertIn(5, list_of_primes_under_ten) 181 self.assertNotIn(12, list_of_primes_under_ten) 182 183 184assertIs, assertIsNot 185--------------------- 186 187These two assertions check whether values are identical to one another. This 188is sometimes useful when you want to test something more strict than mere 189equality. For example:: 190 191 def test_assert_is_example(self): 192 foo = [None] 193 foo_alias = foo 194 bar = [None] 195 self.assertIs(foo, foo_alias) 196 self.assertIsNot(foo, bar) 197 self.assertEqual(foo, bar) # They are equal, but not identical 198 199 200assertIsInstance 201---------------- 202 203As much as we love duck-typing and polymorphism, sometimes you need to check 204whether or not a value is of a given type. This method does that. For 205example:: 206 207 def test_assert_is_instance_example(self): 208 now = datetime.now() 209 self.assertIsInstance(now, datetime) 210 211Note that there is no ``assertIsNotInstance`` in testtools currently. 212 213 214expectFailure 215------------- 216 217Sometimes it's useful to write tests that fail. For example, you might want 218to turn a bug report into a unit test, but you don't know how to fix the bug 219yet. Or perhaps you want to document a known, temporary deficiency in a 220dependency. 221 222testtools gives you the ``TestCase.expectFailure`` to help with this. You use 223it to say that you expect this assertion to fail. When the test runs and the 224assertion fails, testtools will report it as an "expected failure". 225 226Here's an example:: 227 228 def test_expect_failure_example(self): 229 self.expectFailure( 230 "cats should be dogs", self.assertEqual, 'cats', 'dogs') 231 232As long as 'cats' is not equal to 'dogs', the test will be reported as an 233expected failure. 234 235If ever by some miracle 'cats' becomes 'dogs', then testtools will report an 236"unexpected success". Unlike standard unittest, testtools treats this as 237something that fails the test suite, like an error or a failure. 238 239 240Matchers 241======== 242 243The built-in assertion methods are very useful, they are the bread and butter 244of writing tests. However, soon enough you will probably want to write your 245own assertions. Perhaps there are domain specific things that you want to 246check (e.g. assert that two widgets are aligned parallel to the flux grid), or 247perhaps you want to check something that could almost but not quite be found 248in some other standard library (e.g. assert that two paths point to the same 249file). 250 251When you are in such situations, you could either make a base class for your 252project that inherits from ``testtools.TestCase`` and make sure that all of 253your tests derive from that, *or* you could use the testtools ``Matcher`` 254system. 255 256 257Using Matchers 258-------------- 259 260Here's a really basic example using stock matchers found in testtools:: 261 262 import testtools 263 from testtools.matchers import Equals 264 265 class TestSquare(TestCase): 266 def test_square(self): 267 result = square(7) 268 self.assertThat(result, Equals(49)) 269 270The line ``self.assertThat(result, Equals(49))`` is equivalent to 271``self.assertEqual(result, 49)`` and means "assert that ``result`` equals 49". 272The difference is that ``assertThat`` is a more general method that takes some 273kind of observed value (in this case, ``result``) and any matcher object 274(here, ``Equals(49)``). 275 276The matcher object could be absolutely anything that implements the Matcher 277protocol. This means that you can make more complex matchers by combining 278existing ones:: 279 280 def test_square_silly(self): 281 result = square(7) 282 self.assertThat(result, Not(Equals(50))) 283 284Which is roughly equivalent to:: 285 286 def test_square_silly(self): 287 result = square(7) 288 self.assertNotEqual(result, 50) 289 290 291Stock matchers 292-------------- 293 294testtools comes with many matchers built in. They can all be found in and 295imported from the ``testtools.matchers`` module. 296 297Equals 298~~~~~~ 299 300Matches if two items are equal. For example:: 301 302 def test_equals_example(self): 303 self.assertThat([42], Equals([42])) 304 305 306Is 307~~~ 308 309Matches if two items are identical. For example:: 310 311 def test_is_example(self): 312 foo = object() 313 self.assertThat(foo, Is(foo)) 314 315 316IsInstance 317~~~~~~~~~~ 318 319Adapts isinstance() to use as a matcher. For example:: 320 321 def test_isinstance_example(self): 322 class MyClass:pass 323 self.assertThat(MyClass(), IsInstance(MyClass)) 324 self.assertThat(MyClass(), IsInstance(MyClass, str)) 325 326 327The raises helper 328~~~~~~~~~~~~~~~~~ 329 330Matches if a callable raises a particular type of exception. For example:: 331 332 def test_raises_example(self): 333 self.assertThat(lambda: 1/0, raises(ZeroDivisionError)) 334 335This is actually a convenience function that combines two other matchers: 336Raises_ and MatchesException_. 337 338 339DocTestMatches 340~~~~~~~~~~~~~~ 341 342Matches a string as if it were the output of a doctest_ example. Very useful 343for making assertions about large chunks of text. For example:: 344 345 import doctest 346 347 def test_doctest_example(self): 348 output = "Colorless green ideas" 349 self.assertThat( 350 output, 351 DocTestMatches("Colorless ... ideas", doctest.ELLIPSIS)) 352 353We highly recommend using the following flags:: 354 355 doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE | doctest.REPORT_NDIFF 356 357 358GreaterThan 359~~~~~~~~~~~ 360 361Matches if the given thing is greater than the thing in the matcher. For 362example:: 363 364 def test_greater_than_example(self): 365 self.assertThat(3, GreaterThan(2)) 366 367 368LessThan 369~~~~~~~~ 370 371Matches if the given thing is less than the thing in the matcher. For 372example:: 373 374 def test_less_than_example(self): 375 self.assertThat(2, LessThan(3)) 376 377 378StartsWith, EndsWith 379~~~~~~~~~~~~~~~~~~~~ 380 381These matchers check to see if a string starts with or ends with a particular 382substring. For example:: 383 384 def test_starts_and_ends_with_example(self): 385 self.assertThat('underground', StartsWith('und')) 386 self.assertThat('underground', EndsWith('und')) 387 388 389Contains 390~~~~~~~~ 391 392This matcher checks to see if the given thing contains the thing in the 393matcher. For example:: 394 395 def test_contains_example(self): 396 self.assertThat('abc', Contains('b')) 397 398 399MatchesException 400~~~~~~~~~~~~~~~~ 401 402Matches an exc_info tuple if the exception is of the correct type. For 403example:: 404 405 def test_matches_exception_example(self): 406 try: 407 raise RuntimeError('foo') 408 except RuntimeError: 409 exc_info = sys.exc_info() 410 self.assertThat(exc_info, MatchesException(RuntimeError)) 411 self.assertThat(exc_info, MatchesException(RuntimeError('bar')) 412 413Most of the time, you will want to uses `The raises helper`_ instead. 414 415 416NotEquals 417~~~~~~~~~ 418 419Matches if something is not equal to something else. Note that this is subtly 420different to ``Not(Equals(x))``. ``NotEquals(x)`` will match if ``y != x``, 421``Not(Equals(x))`` will match if ``not y == x``. 422 423You only need to worry about this distinction if you are testing code that 424relies on badly written overloaded equality operators. 425 426 427KeysEqual 428~~~~~~~~~ 429 430Matches if the keys of one dict are equal to the keys of another dict. For 431example:: 432 433 def test_keys_equal(self): 434 x = {'a': 1, 'b': 2} 435 y = {'a': 2, 'b': 3} 436 self.assertThat(x, KeysEqual(y)) 437 438 439MatchesRegex 440~~~~~~~~~~~~ 441 442Matches a string against a regular expression, which is a wonderful thing to 443be able to do, if you think about it:: 444 445 def test_matches_regex_example(self): 446 self.assertThat('foo', MatchesRegex('fo+')) 447 448 449HasLength 450~~~~~~~~~ 451 452Check the length of a collection. The following assertion will fail:: 453 454 self.assertThat([1, 2, 3], HasLength(2)) 455 456But this one won't:: 457 458 self.assertThat([1, 2, 3], HasLength(3)) 459 460 461File- and path-related matchers 462------------------------------- 463 464testtools also has a number of matchers to help with asserting things about 465the state of the filesystem. 466 467PathExists 468~~~~~~~~~~ 469 470Matches if a path exists:: 471 472 self.assertThat('/', PathExists()) 473 474 475DirExists 476~~~~~~~~~ 477 478Matches if a path exists and it refers to a directory:: 479 480 # This will pass on most Linux systems. 481 self.assertThat('/home/', DirExists()) 482 # This will not 483 self.assertThat('/home/jml/some-file.txt', DirExists()) 484 485 486FileExists 487~~~~~~~~~~ 488 489Matches if a path exists and it refers to a file (as opposed to a directory):: 490 491 # This will pass on most Linux systems. 492 self.assertThat('/bin/true', FileExists()) 493 # This will not. 494 self.assertThat('/home/', FileExists()) 495 496 497DirContains 498~~~~~~~~~~~ 499 500Matches if the given directory contains the specified files and directories. 501Say we have a directory ``foo`` that has the files ``a``, ``b`` and ``c``, 502then:: 503 504 self.assertThat('foo', DirContains(['a', 'b', 'c'])) 505 506will match, but:: 507 508 self.assertThat('foo', DirContains(['a', 'b'])) 509 510will not. 511 512The matcher sorts both the input and the list of names we get back from the 513filesystem. 514 515You can use this in a more advanced way, and match the sorted directory 516listing against an arbitrary matcher:: 517 518 self.assertThat('foo', DirContains(matcher=Contains('a'))) 519 520 521FileContains 522~~~~~~~~~~~~ 523 524Matches if the given file has the specified contents. Say there's a file 525called ``greetings.txt`` with the contents, ``Hello World!``:: 526 527 self.assertThat('greetings.txt', FileContains("Hello World!")) 528 529will match. 530 531You can also use this in a more advanced way, and match the contents of the 532file against an arbitrary matcher:: 533 534 self.assertThat('greetings.txt', FileContains(matcher=Contains('!'))) 535 536 537HasPermissions 538~~~~~~~~~~~~~~ 539 540Used for asserting that a file or directory has certain permissions. Uses 541octal-mode permissions for both input and matching. For example:: 542 543 self.assertThat('/tmp', HasPermissions('1777')) 544 self.assertThat('id_rsa', HasPermissions('0600')) 545 546This is probably more useful on UNIX systems than on Windows systems. 547 548 549SamePath 550~~~~~~~~ 551 552Matches if two paths actually refer to the same thing. The paths don't have 553to exist, but if they do exist, ``SamePath`` will resolve any symlinks.:: 554 555 self.assertThat('somefile', SamePath('childdir/../somefile')) 556 557 558TarballContains 559~~~~~~~~~~~~~~~ 560 561Matches the contents of a tarball. In many ways, much like ``DirContains``, 562but instead of matching on ``os.listdir`` matches on ``TarFile.getnames``. 563 564 565Combining matchers 566------------------ 567 568One great thing about matchers is that you can readily combine existing 569matchers to get variations on their behaviour or to quickly build more complex 570assertions. 571 572Below are a few of the combining matchers that come with testtools. 573 574 575Not 576~~~ 577 578Negates another matcher. For example:: 579 580 def test_not_example(self): 581 self.assertThat([42], Not(Equals("potato"))) 582 self.assertThat([42], Not(Is([42]))) 583 584If you find yourself using ``Not`` frequently, you may wish to create a custom 585matcher for it. For example:: 586 587 IsNot = lambda x: Not(Is(x)) 588 589 def test_not_example_2(self): 590 self.assertThat([42], IsNot([42])) 591 592 593Annotate 594~~~~~~~~ 595 596Used to add custom notes to a matcher. For example:: 597 598 def test_annotate_example(self): 599 result = 43 600 self.assertThat( 601 result, Annotate("Not the answer to the Question!", Equals(42)) 602 603Since the annotation is only ever displayed when there is a mismatch 604(e.g. when ``result`` does not equal 42), it's a good idea to phrase the note 605negatively, so that it describes what a mismatch actually means. 606 607As with Not_, you may wish to create a custom matcher that describes a 608common operation. For example:: 609 610 PoliticallyEquals = lambda x: Annotate("Death to the aristos!", Equals(x)) 611 612 def test_annotate_example_2(self): 613 self.assertThat("orange", PoliticallyEquals("yellow")) 614 615You can have assertThat perform the annotation for you as a convenience:: 616 617 def test_annotate_example_3(self): 618 self.assertThat("orange", Equals("yellow"), "Death to the aristos!") 619 620 621AfterPreprocessing 622~~~~~~~~~~~~~~~~~~ 623 624Used to make a matcher that applies a function to the matched object before 625matching. This can be used to aid in creating trivial matchers as functions, for 626example:: 627 628 def test_after_preprocessing_example(self): 629 def PathHasFileContent(content): 630 def _read(path): 631 return open(path).read() 632 return AfterPreprocessing(_read, Equals(content)) 633 self.assertThat('/tmp/foo.txt', PathHasFileContent("Hello world!")) 634 635 636MatchesAll 637~~~~~~~~~~ 638 639Combines many matchers to make a new matcher. The new matcher will only match 640things that match every single one of the component matchers. 641 642It's much easier to understand in Python than in English:: 643 644 def test_matches_all_example(self): 645 has_und_at_both_ends = MatchesAll(StartsWith("und"), EndsWith("und")) 646 # This will succeed. 647 self.assertThat("underground", has_und_at_both_ends) 648 # This will fail. 649 self.assertThat("found", has_und_at_both_ends) 650 # So will this. 651 self.assertThat("undead", has_und_at_both_ends) 652 653At this point some people ask themselves, "why bother doing this at all? why 654not just have two separate assertions?". It's a good question. 655 656The first reason is that when a ``MatchesAll`` gets a mismatch, the error will 657include information about all of the bits that mismatched. When you have two 658separate assertions, as below:: 659 660 def test_two_separate_assertions(self): 661 self.assertThat("foo", StartsWith("und")) 662 self.assertThat("foo", EndsWith("und")) 663 664Then you get absolutely no information from the second assertion if the first 665assertion fails. Tests are largely there to help you debug code, so having 666more information in error messages is a big help. 667 668The second reason is that it is sometimes useful to give a name to a set of 669matchers. ``has_und_at_both_ends`` is a bit contrived, of course, but it is 670clear. The ``FileExists`` and ``DirExists`` matchers included in testtools 671are perhaps better real examples. 672 673If you want only the first mismatch to be reported, pass ``first_only=True`` 674as a keyword parameter to ``MatchesAll``. 675 676 677MatchesAny 678~~~~~~~~~~ 679 680Like MatchesAll_, ``MatchesAny`` combines many matchers to make a new 681matcher. The difference is that the new matchers will match a thing if it 682matches *any* of the component matchers. 683 684For example:: 685 686 def test_matches_any_example(self): 687 self.assertThat(42, MatchesAny(Equals(5), Not(Equals(6)))) 688 689 690AllMatch 691~~~~~~~~ 692 693Matches many values against a single matcher. Can be used to make sure that 694many things all meet the same condition:: 695 696 def test_all_match_example(self): 697 self.assertThat([2, 3, 5, 7], AllMatch(LessThan(10))) 698 699If the match fails, then all of the values that fail to match will be included 700in the error message. 701 702In some ways, this is the converse of MatchesAll_. 703 704 705MatchesListwise 706~~~~~~~~~~~~~~~ 707 708Where ``MatchesAny`` and ``MatchesAll`` combine many matchers to match a 709single value, ``MatchesListwise`` combines many matches to match many values. 710 711For example:: 712 713 def test_matches_listwise_example(self): 714 self.assertThat( 715 [1, 2, 3], MatchesListwise(map(Equals, [1, 2, 3]))) 716 717This is useful for writing custom, domain-specific matchers. 718 719If you want only the first mismatch to be reported, pass ``first_only=True`` 720to ``MatchesListwise``. 721 722 723MatchesSetwise 724~~~~~~~~~~~~~~ 725 726Combines many matchers to match many values, without regard to their order. 727 728Here's an example:: 729 730 def test_matches_setwise_example(self): 731 self.assertThat( 732 [1, 2, 3], MatchesSetwise(Equals(2), Equals(3), Equals(1))) 733 734Much like ``MatchesListwise``, best used for writing custom, domain-specific 735matchers. 736 737 738MatchesStructure 739~~~~~~~~~~~~~~~~ 740 741Creates a matcher that matches certain attributes of an object against a 742pre-defined set of matchers. 743 744It's much easier to understand in Python than in English:: 745 746 def test_matches_structure_example(self): 747 foo = Foo() 748 foo.a = 1 749 foo.b = 2 750 matcher = MatchesStructure(a=Equals(1), b=Equals(2)) 751 self.assertThat(foo, matcher) 752 753Since all of the matchers used were ``Equals``, we could also write this using 754the ``byEquality`` helper:: 755 756 def test_matches_structure_example(self): 757 foo = Foo() 758 foo.a = 1 759 foo.b = 2 760 matcher = MatchesStructure.byEquality(a=1, b=2) 761 self.assertThat(foo, matcher) 762 763``MatchesStructure.fromExample`` takes an object and a list of attributes and 764creates a ``MatchesStructure`` matcher where each attribute of the matched 765object must equal each attribute of the example object. For example:: 766 767 matcher = MatchesStructure.fromExample(foo, 'a', 'b') 768 769is exactly equivalent to ``matcher`` in the previous example. 770 771 772MatchesPredicate 773~~~~~~~~~~~~~~~~ 774 775Sometimes, all you want to do is create a matcher that matches if a given 776function returns True, and mismatches if it returns False. 777 778For example, you might have an ``is_prime`` function and want to make a 779matcher based on it:: 780 781 def test_prime_numbers(self): 782 IsPrime = MatchesPredicate(is_prime, '%s is not prime.') 783 self.assertThat(7, IsPrime) 784 self.assertThat(1983, IsPrime) 785 # This will fail. 786 self.assertThat(42, IsPrime) 787 788Which will produce the error message:: 789 790 Traceback (most recent call last): 791 File "...", line ..., in test_prime_numbers 792 self.assertThat(42, IsPrime) 793 MismatchError: 42 is not prime. 794 795 796MatchesPredicateWithParams 797~~~~~~~~~~~~~~~~~~~~~~~~~~ 798 799Sometimes you can't use a trivial predicate and instead need to pass in some 800parameters each time. In that case, MatchesPredicateWithParams is your go-to 801tool for creating ad hoc matchers. MatchesPredicateWithParams takes a predicate 802function and message and returns a factory to produce matchers from that. The 803predicate needs to return a boolean (or any truthy object), and accept the 804object to match + whatever was passed into the factory. 805 806For example, you might have an ``divisible`` function and want to make a 807matcher based on it:: 808 809 def test_divisible_numbers(self): 810 IsDivisibleBy = MatchesPredicateWithParams( 811 divisible, '{0} is not divisible by {1}') 812 self.assertThat(7, IsDivisibleBy(1)) 813 self.assertThat(7, IsDivisibleBy(7)) 814 self.assertThat(7, IsDivisibleBy(2))) 815 # This will fail. 816 817Which will produce the error message:: 818 819 Traceback (most recent call last): 820 File "...", line ..., in test_divisible 821 self.assertThat(7, IsDivisibleBy(2)) 822 MismatchError: 7 is not divisible by 2. 823 824 825Raises 826~~~~~~ 827 828Takes whatever the callable raises as an exc_info tuple and matches it against 829whatever matcher it was given. For example, if you want to assert that a 830callable raises an exception of a given type:: 831 832 def test_raises_example(self): 833 self.assertThat( 834 lambda: 1/0, Raises(MatchesException(ZeroDivisionError))) 835 836Although note that this could also be written as:: 837 838 def test_raises_example_convenient(self): 839 self.assertThat(lambda: 1/0, raises(ZeroDivisionError)) 840 841See also MatchesException_ and `the raises helper`_ 842 843 844Writing your own matchers 845------------------------- 846 847Combining matchers is fun and can get you a very long way indeed, but 848sometimes you will have to write your own. Here's how. 849 850You need to make two closely-linked objects: a ``Matcher`` and a 851``Mismatch``. The ``Matcher`` knows how to actually make the comparison, and 852the ``Mismatch`` knows how to describe a failure to match. 853 854Here's an example matcher:: 855 856 class IsDivisibleBy(object): 857 """Match if a number is divisible by another number.""" 858 def __init__(self, divider): 859 self.divider = divider 860 def __str__(self): 861 return 'IsDivisibleBy(%s)' % (self.divider,) 862 def match(self, actual): 863 remainder = actual % self.divider 864 if remainder != 0: 865 return IsDivisibleByMismatch(actual, self.divider, remainder) 866 else: 867 return None 868 869The matcher has a constructor that takes parameters that describe what you 870actually *expect*, in this case a number that other numbers ought to be 871divisible by. It has a ``__str__`` method, the result of which is displayed 872on failure by ``assertThat`` and a ``match`` method that does the actual 873matching. 874 875``match`` takes something to match against, here ``actual``, and decides 876whether or not it matches. If it does match, then ``match`` must return 877``None``. If it does *not* match, then ``match`` must return a ``Mismatch`` 878object. ``assertThat`` will call ``match`` and then fail the test if it 879returns a non-None value. For example:: 880 881 def test_is_divisible_by_example(self): 882 # This succeeds, since IsDivisibleBy(5).match(10) returns None. 883 self.assertThat(10, IsDivisibleBy(5)) 884 # This fails, since IsDivisibleBy(7).match(10) returns a mismatch. 885 self.assertThat(10, IsDivisibleBy(7)) 886 887The mismatch is responsible for what sort of error message the failing test 888generates. Here's an example mismatch:: 889 890 class IsDivisibleByMismatch(object): 891 def __init__(self, number, divider, remainder): 892 self.number = number 893 self.divider = divider 894 self.remainder = remainder 895 896 def describe(self): 897 return "%r is not divisible by %r, %r remains" % ( 898 self.number, self.divider, self.remainder) 899 900 def get_details(self): 901 return {} 902 903The mismatch takes information about the mismatch, and provides a ``describe`` 904method that assembles all of that into a nice error message for end users. 905You can use the ``get_details`` method to provide extra, arbitrary data with 906the mismatch (e.g. the contents of a log file). Most of the time it's fine to 907just return an empty dict. You can read more about Details_ elsewhere in this 908document. 909 910Sometimes you don't need to create a custom mismatch class. In particular, if 911you don't care *when* the description is calculated, then you can just do that 912in the Matcher itself like this:: 913 914 def match(self, actual): 915 remainder = actual % self.divider 916 if remainder != 0: 917 return Mismatch( 918 "%r is not divisible by %r, %r remains" % ( 919 actual, self.divider, remainder)) 920 else: 921 return None 922 923When writing a ``describe`` method or constructing a ``Mismatch`` object the 924code should ensure it only emits printable unicode. As this output must be 925combined with other text and forwarded for presentation, letting through 926non-ascii bytes of ambiguous encoding or control characters could throw an 927exception or mangle the display. In most cases simply avoiding the ``%s`` 928format specifier and using ``%r`` instead will be enough. For examples of 929more complex formatting see the ``testtools.matchers`` implementatons. 930 931 932Details 933======= 934 935As we may have mentioned once or twice already, one of the great benefits of 936automated tests is that they help find, isolate and debug errors in your 937system. 938 939Frequently however, the information provided by a mere assertion failure is 940not enough. It's often useful to have other information: the contents of log 941files; what queries were run; benchmark timing information; what state certain 942subsystem components are in and so forth. 943 944testtools calls all of these things "details" and provides a single, powerful 945mechanism for including this information in your test run. 946 947Here's an example of how to add them:: 948 949 from testtools import TestCase 950 from testtools.content import text_content 951 952 class TestSomething(TestCase): 953 954 def test_thingy(self): 955 self.addDetail('arbitrary-color-name', text_content("blue")) 956 1 / 0 # Gratuitous error! 957 958A detail an arbitrary piece of content given a name that's unique within the 959test. Here the name is ``arbitrary-color-name`` and the content is 960``text_content("blue")``. The name can be any text string, and the content 961can be any ``testtools.content.Content`` object. 962 963When the test runs, testtools will show you something like this:: 964 965 ====================================================================== 966 ERROR: exampletest.TestSomething.test_thingy 967 ---------------------------------------------------------------------- 968 arbitrary-color-name: {{{blue}}} 969 970 Traceback (most recent call last): 971 File "exampletest.py", line 8, in test_thingy 972 1 / 0 # Gratuitous error! 973 ZeroDivisionError: integer division or modulo by zero 974 ------------ 975 Ran 1 test in 0.030s 976 977As you can see, the detail is included as an attachment, here saying 978that our arbitrary-color-name is "blue". 979 980 981Content 982------- 983 984For the actual content of details, testtools uses its own MIME-based Content 985object. This allows you to attach any information that you could possibly 986conceive of to a test, and allows testtools to use or serialize that 987information. 988 989The basic ``testtools.content.Content`` object is constructed from a 990``testtools.content.ContentType`` and a nullary callable that must return an 991iterator of chunks of bytes that the content is made from. 992 993So, to make a Content object that is just a simple string of text, you can 994do:: 995 996 from testtools.content import Content 997 from testtools.content_type import ContentType 998 999 text = Content(ContentType('text', 'plain'), lambda: ["some text"]) 1000 1001Because adding small bits of text content is very common, there's also a 1002convenience method:: 1003 1004 text = text_content("some text") 1005 1006To make content out of an image stored on disk, you could do something like:: 1007 1008 image = Content(ContentType('image', 'png'), lambda: open('foo.png').read()) 1009 1010Or you could use the convenience function:: 1011 1012 image = content_from_file('foo.png', ContentType('image', 'png')) 1013 1014The ``lambda`` helps make sure that the file is opened and the actual bytes 1015read only when they are needed – by default, when the test is finished. This 1016means that tests can construct and add Content objects freely without worrying 1017too much about how they affect run time. 1018 1019 1020A realistic example 1021------------------- 1022 1023A very common use of details is to add a log file to failing tests. Say your 1024project has a server represented by a class ``SomeServer`` that you can start 1025up and shut down in tests, but runs in another process. You want to test 1026interaction with that server, and whenever the interaction fails, you want to 1027see the client-side error *and* the logs from the server-side. Here's how you 1028might do it:: 1029 1030 from testtools import TestCase 1031 from testtools.content import attach_file, Content 1032 from testtools.content_type import UTF8_TEXT 1033 1034 from myproject import SomeServer 1035 1036 class SomeTestCase(TestCase): 1037 1038 def setUp(self): 1039 super(SomeTestCase, self).setUp() 1040 self.server = SomeServer() 1041 self.server.start_up() 1042 self.addCleanup(self.server.shut_down) 1043 self.addCleanup(attach_file, self.server.logfile, self) 1044 1045 def attach_log_file(self): 1046 self.addDetail( 1047 'log-file', 1048 Content(UTF8_TEXT, 1049 lambda: open(self.server.logfile, 'r').readlines())) 1050 1051 def test_a_thing(self): 1052 self.assertEqual("cool", self.server.temperature) 1053 1054This test will attach the log file of ``SomeServer`` to each test that is 1055run. testtools will only display the log file for failing tests, so it's not 1056such a big deal. 1057 1058If the act of adding at detail is expensive, you might want to use 1059addOnException_ so that you only do it when a test actually raises an 1060exception. 1061 1062 1063Controlling test execution 1064========================== 1065 1066.. _addCleanup: 1067 1068addCleanup 1069---------- 1070 1071``TestCase.addCleanup`` is a robust way to arrange for a clean up function to 1072be called before ``tearDown``. This is a powerful and simple alternative to 1073putting clean up logic in a try/finally block or ``tearDown`` method. For 1074example:: 1075 1076 def test_foo(self): 1077 foo.lock() 1078 self.addCleanup(foo.unlock) 1079 ... 1080 1081This is particularly useful if you have some sort of factory in your test:: 1082 1083 def make_locked_foo(self): 1084 foo = Foo() 1085 foo.lock() 1086 self.addCleanup(foo.unlock) 1087 return foo 1088 1089 def test_frotz_a_foo(self): 1090 foo = self.make_locked_foo() 1091 foo.frotz() 1092 self.assertEqual(foo.frotz_count, 1) 1093 1094Any extra arguments or keyword arguments passed to ``addCleanup`` are passed 1095to the callable at cleanup time. 1096 1097Cleanups can also report multiple errors, if appropriate by wrapping them in 1098a ``testtools.MultipleExceptions`` object:: 1099 1100 raise MultipleExceptions(exc_info1, exc_info2) 1101 1102 1103Fixtures 1104-------- 1105 1106Tests often depend on a system being set up in a certain way, or having 1107certain resources available to them. Perhaps a test needs a connection to the 1108database or access to a running external server. 1109 1110One common way of doing this is to do:: 1111 1112 class SomeTest(TestCase): 1113 def setUp(self): 1114 super(SomeTest, self).setUp() 1115 self.server = Server() 1116 self.server.setUp() 1117 self.addCleanup(self.server.tearDown) 1118 1119testtools provides a more convenient, declarative way to do the same thing:: 1120 1121 class SomeTest(TestCase): 1122 def setUp(self): 1123 super(SomeTest, self).setUp() 1124 self.server = self.useFixture(Server()) 1125 1126``useFixture(fixture)`` calls ``setUp`` on the fixture, schedules a clean up 1127to clean it up, and schedules a clean up to attach all details_ held by the 1128fixture to the test case. The fixture object must meet the 1129``fixtures.Fixture`` protocol (version 0.3.4 or newer, see fixtures_). 1130 1131If you have anything beyond the most simple test set up, we recommend that 1132you put this set up into a ``Fixture`` class. Once there, the fixture can be 1133easily re-used by other tests and can be combined with other fixtures to make 1134more complex resources. 1135 1136 1137Skipping tests 1138-------------- 1139 1140Many reasons exist to skip a test: a dependency might be missing; a test might 1141be too expensive and thus should not berun while on battery power; or perhaps 1142the test is testing an incomplete feature. 1143 1144``TestCase.skipTest`` is a simple way to have a test stop running and be 1145reported as a skipped test, rather than a success, error or failure. For 1146example:: 1147 1148 def test_make_symlink(self): 1149 symlink = getattr(os, 'symlink', None) 1150 if symlink is None: 1151 self.skipTest("No symlink support") 1152 symlink(whatever, something_else) 1153 1154Using ``skipTest`` means that you can make decisions about what tests to run 1155as late as possible, and close to the actual tests. Without it, you might be 1156forced to use convoluted logic during test loading, which is a bit of a mess. 1157 1158 1159Legacy skip support 1160~~~~~~~~~~~~~~~~~~~ 1161 1162If you are using this feature when running your test suite with a legacy 1163``TestResult`` object that is missing the ``addSkip`` method, then the 1164``addError`` method will be invoked instead. If you are using a test result 1165from testtools, you do not have to worry about this. 1166 1167In older versions of testtools, ``skipTest`` was known as ``skip``. Since 1168Python 2.7 added ``skipTest`` support, the ``skip`` name is now deprecated. 1169No warning is emitted yet – some time in the future we may do so. 1170 1171 1172addOnException 1173-------------- 1174 1175Sometimes, you might wish to do something only when a test fails. Perhaps you 1176need to run expensive diagnostic routines or some such. 1177``TestCase.addOnException`` allows you to easily do just this. For example:: 1178 1179 class SomeTest(TestCase): 1180 def setUp(self): 1181 super(SomeTest, self).setUp() 1182 self.server = self.useFixture(SomeServer()) 1183 self.addOnException(self.attach_server_diagnostics) 1184 1185 def attach_server_diagnostics(self, exc_info): 1186 self.server.prep_for_diagnostics() # Expensive! 1187 self.addDetail('server-diagnostics', self.server.get_diagnostics) 1188 1189 def test_a_thing(self): 1190 self.assertEqual('cheese', 'chalk') 1191 1192In this example, ``attach_server_diagnostics`` will only be called when a test 1193fails. It is given the exc_info tuple of the error raised by the test, just 1194in case it is needed. 1195 1196 1197Twisted support 1198--------------- 1199 1200testtools provides *highly experimental* support for running Twisted tests – 1201tests that return a Deferred_ and rely on the Twisted reactor. You should not 1202use this feature right now. We reserve the right to change the API and 1203behaviour without telling you first. 1204 1205However, if you are going to, here's how you do it:: 1206 1207 from testtools import TestCase 1208 from testtools.deferredruntest import AsynchronousDeferredRunTest 1209 1210 class MyTwistedTests(TestCase): 1211 1212 run_tests_with = AsynchronousDeferredRunTest 1213 1214 def test_foo(self): 1215 # ... 1216 return d 1217 1218In particular, note that you do *not* have to use a special base ``TestCase`` 1219in order to run Twisted tests. 1220 1221You can also run individual tests within a test case class using the Twisted 1222test runner:: 1223 1224 class MyTestsSomeOfWhichAreTwisted(TestCase): 1225 1226 def test_normal(self): 1227 pass 1228 1229 @run_test_with(AsynchronousDeferredRunTest) 1230 def test_twisted(self): 1231 # ... 1232 return d 1233 1234Here are some tips for converting your Trial tests into testtools tests. 1235 1236* Use the ``AsynchronousDeferredRunTest`` runner 1237* Make sure to upcall to ``setUp`` and ``tearDown`` 1238* Don't use ``setUpClass`` or ``tearDownClass`` 1239* Don't expect setting .todo, .timeout or .skip attributes to do anything 1240* ``flushLoggedErrors`` is ``testtools.deferredruntest.flush_logged_errors`` 1241* ``assertFailure`` is ``testtools.deferredruntest.assert_fails_with`` 1242* Trial spins the reactor a couple of times before cleaning it up, 1243 ``AsynchronousDeferredRunTest`` does not. If you rely on this behavior, use 1244 ``AsynchronousDeferredRunTestForBrokenTwisted``. 1245 1246force_failure 1247------------- 1248 1249Setting the ``testtools.TestCase.force_failure`` instance variable to ``True`` 1250will cause the test to be marked as a failure, but won't stop the test code 1251from running (see :ref:`force_failure`). 1252 1253 1254Test helpers 1255============ 1256 1257testtools comes with a few little things that make it a little bit easier to 1258write tests. 1259 1260 1261TestCase.patch 1262-------------- 1263 1264``patch`` is a convenient way to monkey-patch a Python object for the duration 1265of your test. It's especially useful for testing legacy code. e.g.:: 1266 1267 def test_foo(self): 1268 my_stream = StringIO() 1269 self.patch(sys, 'stderr', my_stream) 1270 run_some_code_that_prints_to_stderr() 1271 self.assertEqual('', my_stream.getvalue()) 1272 1273The call to ``patch`` above masks ``sys.stderr`` with ``my_stream`` so that 1274anything printed to stderr will be captured in a StringIO variable that can be 1275actually tested. Once the test is done, the real ``sys.stderr`` is restored to 1276its rightful place. 1277 1278 1279Creation methods 1280---------------- 1281 1282Often when writing unit tests, you want to create an object that is a 1283completely normal instance of its type. You don't want there to be anything 1284special about its properties, because you are testing generic behaviour rather 1285than specific conditions. 1286 1287A lot of the time, test authors do this by making up silly strings and numbers 1288and passing them to constructors (e.g. 42, 'foo', "bar" etc), and that's 1289fine. However, sometimes it's useful to be able to create arbitrary objects 1290at will, without having to make up silly sample data. 1291 1292To help with this, ``testtools.TestCase`` implements creation methods called 1293``getUniqueString`` and ``getUniqueInteger``. They return strings and 1294integers that are unique within the context of the test that can be used to 1295assemble more complex objects. Here's a basic example where 1296``getUniqueString`` is used instead of saying "foo" or "bar" or whatever:: 1297 1298 class SomeTest(TestCase): 1299 1300 def test_full_name(self): 1301 first_name = self.getUniqueString() 1302 last_name = self.getUniqueString() 1303 p = Person(first_name, last_name) 1304 self.assertEqual(p.full_name, "%s %s" % (first_name, last_name)) 1305 1306 1307And here's how it could be used to make a complicated test:: 1308 1309 class TestCoupleLogic(TestCase): 1310 1311 def make_arbitrary_person(self): 1312 return Person(self.getUniqueString(), self.getUniqueString()) 1313 1314 def test_get_invitation(self): 1315 a = self.make_arbitrary_person() 1316 b = self.make_arbitrary_person() 1317 couple = Couple(a, b) 1318 event_name = self.getUniqueString() 1319 invitation = couple.get_invitation(event_name) 1320 self.assertEqual( 1321 invitation, 1322 "We invite %s and %s to %s" % ( 1323 a.full_name, b.full_name, event_name)) 1324 1325Essentially, creation methods like these are a way of reducing the number of 1326assumptions in your tests and communicating to test readers that the exact 1327details of certain variables don't actually matter. 1328 1329See pages 419-423 of `xUnit Test Patterns`_ by Gerard Meszaros for a detailed 1330discussion of creation methods. 1331 1332Test attributes 1333--------------- 1334 1335Inspired by the ``nosetests`` ``attr`` plugin, testtools provides support for 1336marking up test methods with attributes, which are then exposed in the test 1337id and can be used when filtering tests by id. (e.g. via ``--load-list``):: 1338 1339 from testtools.testcase import attr, WithAttributes 1340 1341 class AnnotatedTests(WithAttributes, TestCase): 1342 1343 @attr('simple') 1344 def test_one(self): 1345 pass 1346 1347 @attr('more', 'than', 'one) 1348 def test_two(self): 1349 pass 1350 1351 @attr('or') 1352 @attr('stacked') 1353 def test_three(self): 1354 pass 1355 1356General helpers 1357=============== 1358 1359Conditional imports 1360------------------- 1361 1362Lots of the time we would like to conditionally import modules. testtools 1363uses the small library extras to do this. This used to be part of testtools. 1364 1365Instead of:: 1366 1367 try: 1368 from twisted.internet import defer 1369 except ImportError: 1370 defer = None 1371 1372You can do:: 1373 1374 defer = try_import('twisted.internet.defer') 1375 1376 1377Instead of:: 1378 1379 try: 1380 from StringIO import StringIO 1381 except ImportError: 1382 from io import StringIO 1383 1384You can do:: 1385 1386 StringIO = try_imports(['StringIO.StringIO', 'io.StringIO']) 1387 1388 1389Safe attribute testing 1390---------------------- 1391 1392``hasattr`` is broken_ on many versions of Python. The helper ``safe_hasattr`` 1393can be used to safely test whether an object has a particular attribute. Like 1394``try_import`` this used to be in testtools but is now in extras. 1395 1396 1397Nullary callables 1398----------------- 1399 1400Sometimes you want to be able to pass around a function with the arguments 1401already specified. The normal way of doing this in Python is:: 1402 1403 nullary = lambda: f(*args, **kwargs) 1404 nullary() 1405 1406Which is mostly good enough, but loses a bit of debugging information. If you 1407take the ``repr()`` of ``nullary``, you're only told that it's a lambda, and 1408you get none of the juicy meaning that you'd get from the ``repr()`` of ``f``. 1409 1410The solution is to use ``Nullary`` instead:: 1411 1412 nullary = Nullary(f, *args, **kwargs) 1413 nullary() 1414 1415Here, ``repr(nullary)`` will be the same as ``repr(f)``. 1416 1417 1418.. _testrepository: https://launchpad.net/testrepository 1419.. _Trial: http://twistedmatrix.com/documents/current/core/howto/testing.html 1420.. _nose: http://somethingaboutorange.com/mrl/projects/nose/ 1421.. _unittest2: http://pypi.python.org/pypi/unittest2 1422.. _zope.testrunner: http://pypi.python.org/pypi/zope.testrunner/ 1423.. _xUnit test patterns: http://xunitpatterns.com/ 1424.. _fixtures: http://pypi.python.org/pypi/fixtures 1425.. _unittest: http://docs.python.org/library/unittest.html 1426.. _doctest: http://docs.python.org/library/doctest.html 1427.. _Deferred: http://twistedmatrix.com/documents/current/core/howto/defer.html 1428.. _discover: http://pypi.python.org/pypi/discover 1429.. _`testtools API docs`: http://mumak.net/testtools/apidocs/ 1430.. _Distutils: http://docs.python.org/library/distutils.html 1431.. _`setup configuration`: http://docs.python.org/distutils/configfile.html 1432.. _broken: http://chipaca.com/post/3210673069/hasattr-17-less-harmful 1433