1 2:LastChangedDate: $LastChangedDate$ 3:LastChangedRevision: $LastChangedRevision$ 4:LastChangedBy: $LastChangedBy$ 5 6Unit Tests in Twisted 7===================== 8 9 10 11 12 13Each *unit test* tests one bit of functionality in the 14software. Unit tests are entirely automated and complete quickly. 15Unit tests for the entire system are gathered into one test suite, 16and may all be run in a single batch. The result of a unit test 17is simple: either it passes, or it doesn't. All this means you 18can test the entire system at any time without inconvenience, and 19quickly see what passes and what fails. 20 21 22 23 24 25Unit Tests in the Twisted Philosophy 26------------------------------------ 27 28 29 30The Twisted development team adheres to the practice of `Extreme Programming <http://c2.com/cgi/wiki?ExtremeProgramming>`_ (XP), 31and the usage of unit tests is a cornerstone XP practice. Unit tests are a 32tool to give you increased confidence. You changed an algorithm -- did you 33break something? Run the unit tests. If a test fails, you know where to 34look, because each test covers only a small amount of code, and you know it 35has something to do with the changes you just made. If all the tests pass, 36you're good to go, and you don't need to second-guess yourself or worry that 37you just accidentally broke someone else's program. 38 39 40 41 42 43What to Test, What Not to Test 44------------------------------ 45 46 47 48 49 You don't have to write a test for every single 50 method you write, only production methods that could possibly break. 51 52 53 54 55 56 57-- Kent Beck, Extreme Programming Explained 58 59 60 61 62 63Running the Tests 64----------------- 65 66 67 68 69How 70~~~ 71 72 73 74From the root of the Twisted source tree, run 75`Trial <https://twistedmatrix.com/trac/wiki/TwistedTrial>`_ : 76 77 78 79 80 81 82.. code-block:: console 83 84 85 $ bin/trial twisted 86 87 88 89 90You'll find that having something like this in your emacs init 91files is quite handy: 92 93 94 95 96 97:: 98 99 100 (defun runtests () (interactive) 101 (compile "python /somepath/Twisted/bin/trial /somepath/Twisted")) 102 103 (global-set-key [(alt t)] 'runtests) 104 105 106 107 108When 109~~~~ 110 111 112 113Always, always, *always* be sure `all the tests pass <https://ronjeffries.com/xprog/classics/expunittestsat100/>`_ before committing any code. If someone else 114checks out code at the start of a development session and finds 115failing tests, they will not be happy and may decide to *hunt you down* . 116 117 118 119 120Since this is a geographically dispersed team, the person who can help 121you get your code working probably isn't in the room with you. You may want 122to share your work in progress over the network, but you want to leave the 123main Git tree in good working order. 124So `use a branch <https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging>`_ , 125and merge your changes back in only after your problem is solved and all the 126unit tests pass again. 127 128 129 130 131 132Adding a Test 133------------- 134 135 136 137Please don't add new modules to Twisted without adding tests 138for them too. Otherwise we could change something which breaks 139your module and not find out until later, making it hard to know 140exactly what the change that broke it was, or until after a 141release, and nobody wants broken code in a release. 142 143 144 145 146Tests go into dedicated test packages such as 147``twisted/test/`` or ``twisted/conch/test/`` , 148and are named ``test_foo.py`` , where ``foo`` is the name 149of the module or package being tested. Extensive documentation on using 150the PyUnit framework for writing unit tests can be found in the 151:ref:`links section <core-development-policy-test-standard-links>` below. 152 153 154 155 156 157One deviation from the standard PyUnit documentation: To ensure 158that any variations in test results are due to variations in the 159code or environment and not the test process itself, Twisted ships 160with its own, compatible, testing framework. That just 161means that when you import the unittest module, you will ``from twisted.trial import unittest`` instead of the 162standard ``import unittest`` . 163 164 165 166 167As long as you have followed the module naming and placement 168conventions, ``trial`` will be smart 169enough to pick up any new tests you write. 170 171 172 173 174PyUnit provides a large number of assertion methods to be used when 175writing tests. Many of these are redundant. For consistency, Twisted 176unit tests should use the ``assert`` forms rather than the 177``fail`` forms. Also, use ``assertEqual`` , 178``assertNotEqual`` , and ``assertAlmostEqual`` rather 179than ``assertEquals`` , ``assertNotEquals`` , and 180``assertAlmostEquals`` . ``assertTrue`` is also 181preferred over ``assert_`` . You may notice this convention is 182not followed everywhere in the Twisted codebase. If you are changing 183some test code and notice the wrong method being used in nearby code, 184feel free to adjust it. 185 186 187 188 189When you add a unit test, make sure all methods have docstrings 190specifying at a high level the intent of the test. That is, a description 191that users of the method would understand. 192 193 194 195 196 197Test Implementation Guidelines 198------------------------------ 199 200 201 202Here are some guidelines to follow when writing tests for the Twisted 203test suite. Many tests predate these guidelines and so do not follow them. 204When in doubt, follow the guidelines given here, not the example of old unit 205tests. 206 207 208 209 210Naming Test Classes 211~~~~~~~~~~~~~~~~~~~ 212 213 214 215When writing tests for the Twisted test suite, test classes are named 216``FooTests``, where ``Foo`` is the name of the component being tested. 217Here is an example: 218 219 220 221 222 223.. code-block:: python 224 225 226 class SSHClientTests(unittest.TestCase): 227 def test_sshClient(self): 228 foo() # the actual test 229 230 231 232 233 234Real I/O 235~~~~~~~~ 236 237Most unit tests should avoid performing real, platform-implemented I/O operations. 238Real I/O is slow, unreliable, and unwieldy. 239 240When implementing a protocol, :py:class:`twisted.internet.testing.StringTransport` can be used instead of a real TCP transport. 241``StringTransport`` is fast, deterministic, and can easily be used to exercise all possible network behaviors. 242 243If you need pair a client to a server and have them talk to each other, use ``twisted.test.iosim.connect`` with ``twisted.test.iosim.FakeTransport`` transports. 244 245 246Real Time 247~~~~~~~~~ 248 249Most unit tests should also avoid waiting for real time to pass. 250Unit tests which construct and advance a :py:class:`twisted.internet.task.Clock` are fast and deterministic. 251 252When designing your code allow for the reactor to be injected during tests. 253 254.. code-block:: python 255 256 from twisted.internet.task import Clock 257 258 def test_timeBasedFeature(self): 259 """ 260 In this test a Clock scheduler is used. 261 """ 262 clock = Clock() 263 yourThing = SomeThing() 264 yourThing._reactor = clock 265 266 state = yourThing.getState() 267 268 clock.advance(10) 269 270 # Get state after 10 seconds. 271 state = yourThing.getState() 272 273 274Test Data 275~~~~~~~~~ 276 277Keeping test data in the source tree should be avoided where possible. 278 279In some cases it can not be avoided, but where it's obvious how to do so, do it. 280Test data can be generated at run time or stored inside the test modules as constants. 281 282When file system access is required, dumping data into a temporary path during the test run opens up more testing opportunities. 283Inside the temporary path you can control various path properties or permissions. 284 285You should design your code so that data can be read from arbitrary input streams. 286 287Tests should be able to run even if they are run inside an installed copy of Twisted. 288 289.. code-block:: python 290 291 publicRSA_openssh = ("ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAGEArzJx8OYOnJmzf4tf" 292 "vLi8DVPrJ3/c9k2I/Az64fxjHf9imyRJbixtQhlH9lfNjUIx+4LmrJH5QNRsFporcHDKOTwTT" 293 "h5KmRpslkYHRivcJSkbh/C+BR3utDS555mV comment") 294 295 def test_loadOpenSSHRSAPublic(self): 296 """ 297 L{keys.Key.fromStrea} can load RSA keys serialized in OpenSSH format. 298 """ 299 keys.Key.fromStream(StringIO(publicRSA_openssh)) 300 301 302The Global Reactor 303~~~~~~~~~~~~~~~~~~ 304 305Since unit tests are avoiding real I/O and real time, they can usually avoid using a real reactor. 306The only exceptions to this are unit tests for a real reactor implementation. 307Unit tests for protocol implementations or other application code should not use a reactor. 308Unit tests for real reactor implementations should not use the global reactor, but should 309instead use ``twisted.internet.test.reactormixins.ReactorBuilder`` so they can be applied to all of the reactor implementations automatically. 310In no case should new unit tests use the global reactor. 311 312 313Skipping Tests 314-------------- 315 316Trial, the Twisted unit test framework, has some extensions which are 317designed to encourage developers to add new tests. One common situation is 318that a test exercises some optional functionality: maybe it depends upon 319certain external libraries being available, maybe it only works on certain 320operating systems. The important common factor is that nobody considers 321these limitations to be a bug. 322 323 324 325 326To make it easy to test as much as possible, some tests may be skipped in 327certain situations. Individual test cases can raise the ``SkipTest`` exception to indicate that they should be skipped, and 328the remainder of the test is not run. In the summary (the very last thing 329printed, at the bottom of the test output) the test is counted as a"skip" instead of a "success" or "fail" . This should be used 330inside a conditional which looks for the necessary prerequisites: 331 332 333 334 335 336.. code-block:: python 337 338 339 class SSHClientTests(unittest.TestCase): 340 def test_sshClient(self): 341 if not ssh_path: 342 raise unittest.SkipTest("cannot find ssh, nothing to test") 343 foo() # do actual test after the SkipTest 344 345 346 347 348You can also set the ``.skip`` attribute on the method, with a 349string to indicate why the test is being skipped. This is convenient for 350temporarily turning off a test case, but it can also be set conditionally (by 351manipulating the class attributes after they've been defined): 352 353 354 355 356 357.. code-block:: python 358 359 360 class SomeThingTests(unittest.TestCase): 361 def test_thing(self): 362 dotest() 363 test_thing.skip = "disabled locally" 364 365 366 367 368 369.. code-block:: python 370 371 372 class MyTests(unittest.TestCase): 373 def test_one(self): 374 ... 375 def test_thing(self): 376 dotest() 377 378 if not haveThing: 379 MyTests.test_thing.im_func.skip = "cannot test without Thing" 380 # but test_one() will still run 381 382 383 384 385Finally, you can turn off an entire TestCase at once by setting the .skip 386attribute on the class. If you organize your tests by the functionality they 387depend upon, this is a convenient way to disable just the tests which cannot 388be run. 389 390 391 392 393 394.. code-block:: python 395 396 397 class TCPTests(unittest.TestCase): 398 ... 399 class SSLTests(unittest.TestCase): 400 if not haveSSL: 401 skip = "cannot test without SSL support" 402 # but TCPTests will still run 403 ... 404 405 406Testing New Functionality 407------------------------- 408 409Two good practices which arise from the "XP" development process are 410sometimes at odds with each other: 411 412 413 414 415 416 417- Unit tests are a good thing. Good developers recoil in horror when 418 they see a failing unit test. They should drop everything until the test 419 has been fixed. 420- Good developers write the unit tests first. Once tests are done, they 421 write implementation code until the unit tests pass. Then they stop. 422 423 424 425 426 427These two goals will sometimes conflict. The unit tests that are written 428first, before any implementation has been done, are certain to fail. We want 429developers to commit their code frequently, for reliability and to improve 430coordination between multiple people working on the same problem together. 431While the code is being written, other developers (those not involved in the 432new feature) should not have to pay attention to failures in the new code. 433We should not dilute our well-indoctrinated Failing Test Horror Syndrome by 434crying wolf when an incomplete module has not yet started passing its unit 435tests. To do so would either teach the module author to put off writing or 436committing their unit tests until *after* all the functionality is 437working, or it would teach the other developers to ignore failing test 438cases. Both are bad things. 439 440 441 442 443".todo" is intended to solve this problem. When a developer first 444starts writing the unit tests for functionality that has not yet been 445implemented, they can set the ``.todo`` attribute on the test 446methods that are expected to fail. These methods will still be run, but 447their failure will not be counted the same as normal failures: they will go 448into an "expected failures" category. Developers should learn to treat 449this category as a second-priority queue, behind actual test failures. 450 451 452 453 454As the developer implements the feature, the tests will eventually start 455passing. This is surprising: after all those tests are marked as being 456expected to fail. The .todo tests which nevertheless pass are put into a"unexpected success" category. The developer should remove the .todo 457tag from these tests. At that point, they become normal tests, and their 458failure is once again cause for immediate action by the entire development 459team. 460 461 462 463 464The life cycle of a test is thus: 465 466#. Test is created, marked ``.todo`` . Test fails: "expected failure" . 467#. Code is written, test starts to pass. "unexpected success" . 468#. ``.todo`` tag is removed. Test passes. "success" . 469#. Code is broken, test starts to fail. "failure" . Developers spring 470 into action. 471#. Code is fixed, test passes once more. "success" . 472 473``.todo`` may be of use while you are developing a feature, but by the time you are ready to commit anything all the tests you have written should be passing. 474In other words **never** commit to trunk tests marked as ``.todo``. 475For unfinished tests you should create a follow up ticket and add the tests to the ticket's description. 476 477You can also ignore the ``.todo`` marker and just make sure you write test first to see them failing before starting to work on the fix. 478 479 480Line Coverage Information 481------------------------- 482 483Trial provides line coverage information, which is very useful to ensure 484old code has decent coverage. Passing the ``--coverage`` option to Trial will generate the coverage information in a file called ``coverage`` which can be found in the ``_trial_temp`` 485folder. 486 487 488 489 490 491Associating Test Cases With Source Files 492---------------------------------------- 493 494 495 496Please add a ``test-case-name`` tag to the source file that is 497covered by your new test. This is a comment at the beginning of the file 498which looks like one of the following: 499 500 501 502 503 504.. code-block:: python 505 506 507 # -*- test-case-name: twisted.test.test_defer -*- 508 509 510 511 512or 513 514 515 516 517 518.. code-block:: python 519 520 521 #!/usr/bin/env python 522 # -*- test-case-name: twisted.test.test_defer -*- 523 524 525 526 527This format is understood by emacs to mark "File Variables" . The 528intention is to accept ``test-case-name`` anywhere emacs would on 529the first or second line of the file (but not in the ``File Variables:`` block that emacs accepts at the end of the file). If you 530need to define other emacs file variables, you can either put them in the ``File Variables:`` block or use a semicolon-separated list of 531variable definitions: 532 533 534 535 536 537.. code-block:: python 538 539 540 # -*- test-case-name: twisted.test.test_defer; fill-column: 75; -*- 541 542 543 544 545If the code is exercised by multiple test cases, those may be marked by 546using a comma-separated list of tests, as follows: (NOTE: not all tools can 547handle this yet.. ``trial --testmodule`` does, though) 548 549 550 551 552 553.. code-block:: python 554 555 556 # -*- test-case-name: twisted.test.test_defer,twisted.test.test_tcp -*- 557 558 559 560 561The ``test-case-name`` tag will allow ``trial --testmodule twisted/dir/myfile.py`` to determine which test cases need 562to be run to exercise the code in ``myfile.py`` . Several tools (as 563well as https://launchpad.net/twisted-emacs's ``twisted-dev.el`` 's F9 command) use this to automatically 564run the right tests. 565 566 567 568 569 570Links 571----- 572.. _core-development-policy-test-standard-links: 573 574 575 576 577 578 579 580 581 582 583- A chapter on `Unit Testing <https://www.diveintopython3.net/unit-testing.html>`_ 584 in Mark Pilgrim's `Dive Into Python <https://www.diveintopython3.net/>`_ . 585- :mod:`unittest` module documentation, in the `Python Library Reference <https://docs.python.org/3/library>`_ . 586- `UnitTest <http://c2.com/cgi/wiki?UnitTest>`__ on 587 the `PortlandPatternRepository Wiki <http://c2.com/cgi/wiki>`_ , where all the cool `ExtremeProgramming <http://c2.com/cgi/wiki?ExtremeProgramming>`_ kids hang out. 588- `Unit Tests <http://www.extremeprogramming.org/rules/unittests.html>`_ in `Extreme Programming: A Gentle Introduction <http://www.extremeprogramming.org>`_ . 589- Ron Jeffries expounds on the importance of `Unit Tests at 100% <https://ronjeffries.com/xprog/classics/expunittestsat100/>`_ . 590- Ron Jeffries writes about the `Unit Test <https://web.archive.org/web/20140708115244/http://www.xprogramming.com/Practices/PracUnitTest.html>`_ in the `Extreme Programming practices of C3 <https://web.archive.org/web/20140827044941/http://www.xprogramming.com/Practices/xpractices.htm>`_ . 591- `PyUnit's homepage <http://pyunit.sourceforge.net>`_ . 592- The top-level tests directory, `twisted/test <https://github.com/twisted/twisted/tree/trunk/twisted/test>`_. 593 594 595 596 597 598See also :doc:`Tips for writing tests for Twisted code <../../howto/testing>` . 599 600 601 602