1.. index:: 2 single: unit testing 3 single: integration testing 4 single: functional testing 5 6.. _testing_chapter: 7 8Unit, Integration, and Functional Testing 9========================================= 10 11*Unit testing* is, not surprisingly, the act of testing a "unit" in your 12application. In this context, a "unit" is often a function or a method of a 13class instance. The unit is also referred to as a "unit under test". 14 15The goal of a single unit test is to test **only** some permutation of the 16"unit under test". If you write a unit test that aims to verify the result of 17a particular codepath through a Python function, you need only be concerned 18about testing the code that *lives in the function body itself*. If the 19function accepts a parameter that represents a complex application "domain 20object" (such as a resource, a database connection, or an SMTP server), the 21argument provided to this function during a unit test *need not be* and likely 22*should not be* a "real" implementation object. For example, although a 23particular function implementation may accept an argument that represents an 24SMTP server object, and the function may call a method of this object when the 25system is operating normally that would result in an email being sent, a unit 26test of this codepath of the function does *not* need to test that an email is 27actually sent. It just needs to make sure that the function calls the method 28of the object provided as an argument that *would* send an email if the 29argument happened to be the "real" implementation of an SMTP server object. 30 31An *integration test*, on the other hand, is a different form of testing in 32which the interaction between two or more "units" is explicitly tested. 33Integration tests verify that the components of your application work together. 34You *might* make sure that an email was actually sent in an integration test. 35 36A *functional test* is a form of integration test in which the application is 37run "literally". You would *have to* make sure that an email was actually sent 38in a functional test, because it tests your code end to end. 39 40It is often considered best practice to write each type of tests for any given 41codebase. Unit testing often provides the opportunity to obtain better 42"coverage": it's usually possible to supply a unit under test with arguments 43and/or an environment which causes *all* of its potential codepaths to be 44executed. This is usually not as easy to do with a set of integration or 45functional tests, but integration and functional testing provides a measure of 46assurance that your "units" work together, as they will be expected to when 47your application is run in production. 48 49The suggested mechanism for unit and integration testing of a :app:`Pyramid` 50application is the Python :mod:`unittest` module. Although this module is 51named :mod:`unittest`, it is actually capable of driving both unit and 52integration tests. A good :mod:`unittest` tutorial is available within `Dive 53Into Python <http://www.diveintopython.net/unit_testing/index.html>`_ by Mark 54Pilgrim. 55 56:app:`Pyramid` provides a number of facilities that make unit, integration, and 57functional tests easier to write. The facilities become particularly useful 58when your code calls into :app:`Pyramid`-related framework functions. 59 60.. index:: 61 single: test setup 62 single: test tear down 63 single: unittest 64 65.. _test_setup_and_teardown: 66 67Test Set Up and Tear Down 68------------------------- 69 70:app:`Pyramid` uses a "global" (actually :term:`thread local`) data structure 71to hold two items: the current :term:`request` and the current 72:term:`application registry`. These data structures are available via the 73:func:`pyramid.threadlocal.get_current_request` and 74:func:`pyramid.threadlocal.get_current_registry` functions, respectively. See 75:ref:`threadlocals_chapter` for information about these functions and the data 76structures they return. 77 78If your code uses these ``get_current_*`` functions or calls :app:`Pyramid` 79code which uses ``get_current_*`` functions, you will need to call 80:func:`pyramid.testing.setUp` in your test setup and you will need to call 81:func:`pyramid.testing.tearDown` in your test teardown. 82:func:`~pyramid.testing.setUp` pushes a registry onto the :term:`thread local` 83stack, which makes the ``get_current_*`` functions work. It returns a 84:term:`Configurator` object which can be used to perform extra configuration 85required by the code under test. :func:`~pyramid.testing.tearDown` pops the 86thread local stack. 87 88Normally when a Configurator is used directly with the ``main`` block of a 89Pyramid application, it defers performing any "real work" until its ``.commit`` 90method is called (often implicitly by the 91:meth:`pyramid.config.Configurator.make_wsgi_app` method). The Configurator 92returned by :func:`~pyramid.testing.setUp` is an *autocommitting* Configurator, 93however, which performs all actions implied by methods called on it 94immediately. This is more convenient for unit testing purposes than needing to 95call :meth:`pyramid.config.Configurator.commit` in each test after adding extra 96configuration statements. 97 98The use of the :func:`~pyramid.testing.setUp` and 99:func:`~pyramid.testing.tearDown` functions allows you to supply each unit test 100method in a test case with an environment that has an isolated registry and an 101isolated request for the duration of a single test. Here's an example of using 102this feature: 103 104.. code-block:: python 105 :linenos: 106 107 import unittest 108 from pyramid import testing 109 110 class MyTest(unittest.TestCase): 111 def setUp(self): 112 self.config = testing.setUp() 113 114 def tearDown(self): 115 testing.tearDown() 116 117The above will make sure that :func:`~pyramid.threadlocal.get_current_registry` 118called within a test case method of ``MyTest`` will return the 119:term:`application registry` associated with the ``config`` Configurator 120instance. Each test case method attached to ``MyTest`` will use an isolated 121registry. 122 123The :func:`~pyramid.testing.setUp` and :func:`~pyramid.testing.tearDown` 124functions accept various arguments that influence the environment of the test. 125See the :ref:`testing_module` API for information about the extra arguments 126supported by these functions. 127 128If you also want to make :func:`~pyramid.threadlocal.get_current_request` 129return something other than ``None`` during the course of a single test, you 130can pass a :term:`request` object into the :func:`pyramid.testing.setUp` within 131the ``setUp`` method of your test: 132 133.. code-block:: python 134 :linenos: 135 136 import unittest 137 from pyramid import testing 138 139 class MyTest(unittest.TestCase): 140 def setUp(self): 141 request = testing.DummyRequest() 142 self.config = testing.setUp(request=request) 143 144 def tearDown(self): 145 testing.tearDown() 146 147If you pass a :term:`request` object into :func:`pyramid.testing.setUp` within 148your test case's ``setUp``, any test method attached to the ``MyTest`` test 149case that directly or indirectly calls 150:func:`~pyramid.threadlocal.get_current_request` will receive the request 151object. Otherwise, during testing, 152:func:`~pyramid.threadlocal.get_current_request` will return ``None``. We use a 153"dummy" request implementation supplied by 154:class:`pyramid.testing.DummyRequest` because it's easier to construct than a 155"real" :app:`Pyramid` request object. 156 157Test setup using a context manager 158~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 159 160An alternative style of setting up a test configuration is to use the ``with`` 161statement and :func:`pyramid.testing.testConfig` to create a context manager. 162The context manager will call :func:`pyramid.testing.setUp` before the code 163under test and :func:`pyramid.testing.tearDown` afterwards. 164 165This style is useful for small self-contained tests. For example: 166 167.. code-block:: python 168 :linenos: 169 170 import unittest 171 172 class MyTest(unittest.TestCase): 173 174 def test_my_function(self): 175 from pyramid import testing 176 with testing.testConfig() as config: 177 config.add_route('bar', '/bar/{id}') 178 my_function_which_needs_route_bar() 179 180What? 181~~~~~ 182 183Thread local data structures are always a bit confusing, especially when 184they're used by frameworks. Sorry. So here's a rule of thumb: if you don't 185*know* whether you're calling code that uses the 186:func:`~pyramid.threadlocal.get_current_registry` or 187:func:`~pyramid.threadlocal.get_current_request` functions, or you don't care 188about any of this, but you still want to write test code, just always call 189:func:`pyramid.testing.setUp` in your test's ``setUp`` method and 190:func:`pyramid.testing.tearDown` in your tests' ``tearDown`` method. This 191won't really hurt anything if the application you're testing does not call any 192``get_current*`` function. 193 194.. index:: 195 single: pyramid.testing 196 single: Configurator testing API 197 198Using the ``Configurator`` and ``pyramid.testing`` APIs in Unit Tests 199--------------------------------------------------------------------- 200 201The ``Configurator`` API and the :mod:`pyramid.testing` module provide a number 202of functions which can be used during unit testing. These functions make 203:term:`configuration declaration` calls to the current :term:`application 204registry`, but typically register a "stub" or "dummy" feature in place of the 205"real" feature that the code would call if it was being run normally. 206 207For example, let's imagine you want to unit test a :app:`Pyramid` view 208function. 209 210.. code-block:: python 211 :linenos: 212 213 from pyramid.httpexceptions import HTTPForbidden 214 215 def view_fn(request): 216 if request.has_permission('edit'): 217 raise HTTPForbidden 218 return {'greeting':'hello'} 219 220.. note:: 221 222 This code implies that you have defined a renderer imperatively in a 223 relevant :class:`pyramid.config.Configurator` instance, otherwise it would 224 fail when run normally. 225 226Without doing anything special during a unit test, the call to 227:meth:`~pyramid.request.Request.has_permission` in this view function will 228always return a ``True`` value. When a :app:`Pyramid` application starts 229normally, it will populate an :term:`application registry` using 230:term:`configuration declaration` calls made against a :term:`Configurator`. 231But if this application registry is not created and populated (e.g., by 232initializing the configurator with an authorization policy), like when you 233invoke application code via a unit test, :app:`Pyramid` API functions will tend 234to either fail or return default results. So how do you test the branch of the 235code in this view function that raises 236:exc:`~pyramid.httpexceptions.HTTPForbidden`? 237 238The testing API provided by :app:`Pyramid` allows you to simulate various 239application registry registrations for use under a unit testing framework 240without needing to invoke the actual application configuration implied by its 241``main`` function. For example, if you wanted to test the above ``view_fn`` 242(assuming it lived in the package named ``my.package``), you could write a 243:class:`unittest.TestCase` that used the testing API. 244 245.. code-block:: python 246 :linenos: 247 248 import unittest 249 from pyramid import testing 250 251 class MyTest(unittest.TestCase): 252 def setUp(self): 253 self.config = testing.setUp() 254 255 def tearDown(self): 256 testing.tearDown() 257 258 def test_view_fn_forbidden(self): 259 from pyramid.httpexceptions import HTTPForbidden 260 from my.package import view_fn 261 self.config.testing_securitypolicy(userid='hank', 262 permissive=False) 263 request = testing.DummyRequest() 264 request.context = testing.DummyResource() 265 self.assertRaises(HTTPForbidden, view_fn, request) 266 267 def test_view_fn_allowed(self): 268 from my.package import view_fn 269 self.config.testing_securitypolicy(userid='hank', 270 permissive=True) 271 request = testing.DummyRequest() 272 request.context = testing.DummyResource() 273 response = view_fn(request) 274 self.assertEqual(response, {'greeting':'hello'}) 275 276In the above example, we create a ``MyTest`` test case that inherits from 277:class:`unittest.TestCase`. If it's in our :app:`Pyramid` application, it will 278be found when ``py.test`` is run. It has two test methods. 279 280The first test method, ``test_view_fn_forbidden`` tests the ``view_fn`` when 281the authentication policy forbids the current user the ``edit`` permission. Its 282third line registers a "dummy" "non-permissive" authorization policy using the 283:meth:`~pyramid.config.Configurator.testing_securitypolicy` method, which is a 284special helper method for unit testing. 285 286We then create a :class:`pyramid.testing.DummyRequest` object which simulates a 287WebOb request object API. A :class:`pyramid.testing.DummyRequest` is a request 288object that requires less setup than a "real" :app:`Pyramid` request. We call 289the function being tested with the manufactured request. When the function is 290called, :meth:`pyramid.request.Request.has_permission` will call the "dummy" 291authentication policy we've registered through 292:meth:`~pyramid.config.Configurator.testing_securitypolicy`, which denies 293access. We check that the view function raises a 294:exc:`~pyramid.httpexceptions.HTTPForbidden` error. 295 296The second test method, named ``test_view_fn_allowed``, tests the alternate 297case, where the authentication policy allows access. Notice that we pass 298different values to :meth:`~pyramid.config.Configurator.testing_securitypolicy` 299to obtain this result. We assert at the end of this that the view function 300returns a value. 301 302Note that the test calls the :func:`pyramid.testing.setUp` function in its 303``setUp`` method and the :func:`pyramid.testing.tearDown` function in its 304``tearDown`` method. We assign the result of :func:`pyramid.testing.setUp` as 305``config`` on the unittest class. This is a :term:`Configurator` object and 306all methods of the configurator can be called as necessary within tests. If you 307use any of the :class:`~pyramid.config.Configurator` APIs during testing, be 308sure to use this pattern in your test case's ``setUp`` and ``tearDown``; these 309methods make sure you're using a "fresh" :term:`application registry` per test 310run. 311 312See the :ref:`testing_module` chapter for the entire :app:`Pyramid`-specific 313testing API. This chapter describes APIs for registering a security policy, 314registering resources at paths, registering event listeners, registering views 315and view permissions, and classes representing "dummy" implementations of a 316request and a resource. 317 318.. seealso:: 319 320 See also the various methods of the :term:`Configurator` documented in 321 :ref:`configuration_module` that begin with the ``testing_`` prefix. 322 323.. index:: 324 single: integration tests 325 326.. _integration_tests: 327 328Creating Integration Tests 329-------------------------- 330 331In :app:`Pyramid`, a *unit test* typically relies on "mock" or "dummy" 332implementations to give the code under test enough context to run. 333 334"Integration testing" implies another sort of testing. In the context of a 335:app:`Pyramid` integration test, the test logic exercises the functionality of 336the code under test *and* its integration with the rest of the :app:`Pyramid` 337framework. 338 339Creating an integration test for a :app:`Pyramid` application usually means 340invoking the application's ``includeme`` function via 341:meth:`pyramid.config.Configurator.include` within the test's setup code. This 342causes the entire :app:`Pyramid` environment to be set up, simulating what 343happens when your application is run "for real". This is a heavy-hammer way of 344making sure that your tests have enough context to run properly, and tests your 345code's integration with the rest of :app:`Pyramid`. 346 347.. seealso:: 348 349 See also :ref:`including_configuration` 350 351Writing unit tests that use the :class:`~pyramid.config.Configurator` API to 352set up the right "mock" registrations is often preferred to creating 353integration tests. Unit tests will run faster (because they do less for each 354test) and are usually easier to reason about. 355 356.. index:: 357 single: functional tests 358 359.. _functional_tests: 360 361Creating Functional Tests 362------------------------- 363 364Functional tests test your literal application. 365 366In Pyramid, functional tests are typically written using the :term:`WebTest` 367package, which provides APIs for invoking HTTP(S) requests to your application. 368We also like ``py.test`` and ``pytest-cov`` to provide simple testing and 369coverage reports. 370 371Regardless of which testing :term:`package` you use, be sure to add a 372``tests_require`` dependency on that package to your application's ``setup.py`` 373file. Using the project ``MyProject`` generated by the starter scaffold as 374described in :doc:`project`, we would insert the following code immediately 375following the ``requires`` block in the file ``MyProject/setup.py``. 376 377.. literalinclude:: MyProject/setup.py 378 :language: python 379 :linenos: 380 :lines: 11-22 381 :lineno-start: 11 382 :emphasize-lines: 8- 383 384Remember to change the dependency. 385 386.. literalinclude:: MyProject/setup.py 387 :language: python 388 :linenos: 389 :lines: 40-44 390 :lineno-start: 40 391 :emphasize-lines: 2-4 392 393As always, whenever you change your dependencies, make sure to run the correct 394``pip install -e`` command. 395 396.. code-block:: bash 397 398 $VENV/bin/pip install -e ".[testing]" 399 400In your ``MyPackage`` project, your :term:`package` is named ``myproject`` 401which contains a ``views`` module, which in turn contains a :term:`view` 402function ``my_view`` that returns an HTML body when the root URL is invoked: 403 404 .. literalinclude:: MyProject/myproject/views.py 405 :linenos: 406 :language: python 407 408The following example functional test demonstrates invoking the above 409:term:`view`: 410 411 .. literalinclude:: MyProject/myproject/tests.py 412 :linenos: 413 :pyobject: FunctionalTests 414 :language: python 415 416When this test is run, each test method creates a "real" :term:`WSGI` 417application using the ``main`` function in your ``myproject.__init__`` module, 418using :term:`WebTest` to wrap that WSGI application. It assigns the result to 419``self.testapp``. In the test named ``test_root``, the ``TestApp``'s ``GET`` 420method is used to invoke the root URL. Finally, an assertion is made that the 421returned HTML contains the text ``Pyramid``. 422 423See the :term:`WebTest` documentation for further information about the methods 424available to a :class:`webtest.app.TestApp` instance. 425