1 2.. _usage: 3 4Usage and Invocations 5========================================== 6 7 8.. _cmdline: 9 10Calling pytest through ``python -m pytest`` 11----------------------------------------------------- 12 13.. versionadded:: 2.0 14 15You can invoke testing through the Python interpreter from the command line:: 16 17 python -m pytest [...] 18 19This is almost equivalent to invoking the command line script ``pytest [...]`` 20directly, except that calling via ``python`` will also add the current directory to ``sys.path``. 21 22Possible exit codes 23-------------------------------------------------------------- 24 25Running ``pytest`` can result in six different exit codes: 26 27:Exit code 0: All tests were collected and passed successfully 28:Exit code 1: Tests were collected and run but some of the tests failed 29:Exit code 2: Test execution was interrupted by the user 30:Exit code 3: Internal error happened while executing tests 31:Exit code 4: pytest command line usage error 32:Exit code 5: No tests were collected 33 34Getting help on version, option names, environment variables 35-------------------------------------------------------------- 36 37:: 38 39 pytest --version # shows where pytest was imported from 40 pytest --fixtures # show available builtin function arguments 41 pytest -h | --help # show help on command line and config file options 42 43 44.. _maxfail: 45 46Stopping after the first (or N) failures 47--------------------------------------------------- 48 49To stop the testing process after the first (N) failures:: 50 51 pytest -x # stop after first failure 52 pytest --maxfail=2 # stop after two failures 53 54.. _select-tests: 55 56Specifying tests / selecting tests 57--------------------------------------------------- 58 59Pytest supports several ways to run and select tests from the command-line. 60 61**Run tests in a module** 62 63:: 64 65 pytest test_mod.py 66 67**Run tests in a directory** 68 69:: 70 71 pytest testing/ 72 73**Run tests by keyword expressions** 74 75:: 76 77 pytest -k "MyClass and not method" 78 79This will run tests which contain names that match the given *string expression*, which can 80include Python operators that use filenames, class names and function names as variables. 81The example above will run ``TestMyClass.test_something`` but not ``TestMyClass.test_method_simple``. 82 83.. _nodeids: 84 85**Run tests by node ids** 86 87Each collected test is assigned a unique ``nodeid`` which consist of the module filename followed 88by specifiers like class names, function names and parameters from parametrization, separated by ``::`` characters. 89 90To run a specific test within a module:: 91 92 pytest test_mod.py::test_func 93 94 95Another example specifying a test method in the command line:: 96 97 pytest test_mod.py::TestClass::test_method 98 99**Run tests by marker expressions** 100 101:: 102 103 pytest -m slow 104 105Will run all tests which are decorated with the ``@pytest.mark.slow`` decorator. 106 107For more information see :ref:`marks <mark>`. 108 109**Run tests from packages** 110 111:: 112 113 pytest --pyargs pkg.testing 114 115This will import ``pkg.testing`` and use its filesystem location to find and run tests from. 116 117 118Modifying Python traceback printing 119---------------------------------------------- 120 121Examples for modifying traceback printing:: 122 123 pytest --showlocals # show local variables in tracebacks 124 pytest -l # show local variables (shortcut) 125 126 pytest --tb=auto # (default) 'long' tracebacks for the first and last 127 # entry, but 'short' style for the other entries 128 pytest --tb=long # exhaustive, informative traceback formatting 129 pytest --tb=short # shorter traceback format 130 pytest --tb=line # only one line per failure 131 pytest --tb=native # Python standard library formatting 132 pytest --tb=no # no traceback at all 133 134The ``--full-trace`` causes very long traces to be printed on error (longer 135than ``--tb=long``). It also ensures that a stack trace is printed on 136**KeyboardInterrupt** (Ctrl+C). 137This is very useful if the tests are taking too long and you interrupt them 138with Ctrl+C to find out where the tests are *hanging*. By default no output 139will be shown (because KeyboardInterrupt is caught by pytest). By using this 140option you make sure a trace is shown. 141 142 143.. _pdb-option: 144 145Dropping to PDB_ (Python Debugger) on failures 146----------------------------------------------- 147 148.. _PDB: http://docs.python.org/library/pdb.html 149 150Python comes with a builtin Python debugger called PDB_. ``pytest`` 151allows one to drop into the PDB_ prompt via a command line option:: 152 153 pytest --pdb 154 155This will invoke the Python debugger on every failure (or KeyboardInterrupt). 156Often you might only want to do this for the first failing test to understand 157a certain failure situation:: 158 159 pytest -x --pdb # drop to PDB on first failure, then end test session 160 pytest --pdb --maxfail=3 # drop to PDB for first three failures 161 162Note that on any failure the exception information is stored on 163``sys.last_value``, ``sys.last_type`` and ``sys.last_traceback``. In 164interactive use, this allows one to drop into postmortem debugging with 165any debug tool. One can also manually access the exception information, 166for example:: 167 168 >>> import sys 169 >>> sys.last_traceback.tb_lineno 170 42 171 >>> sys.last_value 172 AssertionError('assert result == "ok"',) 173 174.. _breakpoints: 175 176Setting breakpoints 177------------------- 178 179.. versionadded: 2.4.0 180 181To set a breakpoint in your code use the native Python ``import pdb;pdb.set_trace()`` call 182in your code and pytest automatically disables its output capture for that test: 183 184* Output capture in other tests is not affected. 185* Any prior test output that has already been captured and will be processed as 186 such. 187* Any later output produced within the same test will not be captured and will 188 instead get sent directly to ``sys.stdout``. Note that this holds true even 189 for test output occurring after you exit the interactive PDB_ tracing session 190 and continue with the regular test run. 191 192 193.. _`breakpoint-builtin`: 194 195Using the builtin breakpoint function 196------------------------------------- 197 198Python 3.7 introduces a builtin ``breakpoint()`` function. 199Pytest supports the use of ``breakpoint()`` with the following behaviours: 200 201 - When ``breakpoint()`` is called and ``PYTHONBREAKPOINT`` is set to the default value, pytest will use the custom internal PDB trace UI instead of the system default ``Pdb``. 202 - When tests are complete, the system will default back to the system ``Pdb`` trace UI. 203 - If ``--pdb`` is called on execution of pytest, the custom internal Pdb trace UI is used on ``bothbreakpoint()`` and failed tests/unhandled exceptions. 204 - If ``--pdbcls`` is used, the custom class debugger will be executed when a test fails (as expected within existing behaviour), but also when ``breakpoint()`` is called from within a test, the custom class debugger will be instantiated. 205 206.. _durations: 207 208Profiling test execution duration 209------------------------------------- 210 211.. versionadded: 2.2 212 213To get a list of the slowest 10 test durations:: 214 215 pytest --durations=10 216 217 218Creating JUnitXML format files 219---------------------------------------------------- 220 221To create result files which can be read by Jenkins_ or other Continuous 222integration servers, use this invocation:: 223 224 pytest --junitxml=path 225 226to create an XML file at ``path``. 227 228.. versionadded:: 3.1 229 230To set the name of the root test suite xml item, you can configure the ``junit_suite_name`` option in your config file: 231 232.. code-block:: ini 233 234 [pytest] 235 junit_suite_name = my_suite 236 237.. _record_property example: 238 239record_property 240^^^^^^^^^^^^^^^ 241 242.. versionadded:: 2.8 243.. versionchanged:: 3.5 244 245 Fixture renamed from ``record_xml_property`` to ``record_property`` as user 246 properties are now available to all reporters. 247 ``record_xml_property`` is now deprecated. 248 249If you want to log additional information for a test, you can use the 250``record_property`` fixture: 251 252.. code-block:: python 253 254 def test_function(record_property): 255 record_property("example_key", 1) 256 assert True 257 258This will add an extra property ``example_key="1"`` to the generated 259``testcase`` tag: 260 261.. code-block:: xml 262 263 <testcase classname="test_function" file="test_function.py" line="0" name="test_function" time="0.0009"> 264 <properties> 265 <property name="example_key" value="1" /> 266 </properties> 267 </testcase> 268 269Alternatively, you can integrate this functionality with custom markers: 270 271.. code-block:: python 272 273 # content of conftest.py 274 275 276 def pytest_collection_modifyitems(session, config, items): 277 for item in items: 278 for marker in item.iter_markers(name="test_id"): 279 test_id = marker.args[0] 280 item.user_properties.append(("test_id", test_id)) 281 282And in your tests: 283 284.. code-block:: python 285 286 # content of test_function.py 287 import pytest 288 289 290 @pytest.mark.test_id(1501) 291 def test_function(): 292 assert True 293 294Will result in: 295 296.. code-block:: xml 297 298 <testcase classname="test_function" file="test_function.py" line="0" name="test_function" time="0.0009"> 299 <properties> 300 <property name="test_id" value="1501" /> 301 </properties> 302 </testcase> 303 304.. warning:: 305 306 ``record_property`` is an experimental feature and may change in the future. 307 308 Also please note that using this feature will break any schema verification. 309 This might be a problem when used with some CI servers. 310 311record_xml_attribute 312^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 313 314.. versionadded:: 3.4 315 316To add an additional xml attribute to a testcase element, you can use 317``record_xml_attribute`` fixture. This can also be used to override existing values: 318 319.. code-block:: python 320 321 def test_function(record_xml_attribute): 322 record_xml_attribute("assertions", "REQ-1234") 323 record_xml_attribute("classname", "custom_classname") 324 print("hello world") 325 assert True 326 327Unlike ``record_property``, this will not add a new child element. 328Instead, this will add an attribute ``assertions="REQ-1234"`` inside the generated 329``testcase`` tag and override the default ``classname`` with ``"classname=custom_classname"``: 330 331.. code-block:: xml 332 333 <testcase classname="custom_classname" file="test_function.py" line="0" name="test_function" time="0.003" assertions="REQ-1234"> 334 <system-out> 335 hello world 336 </system-out> 337 </testcase> 338 339.. warning:: 340 341 ``record_xml_attribute`` is an experimental feature, and its interface might be replaced 342 by something more powerful and general in future versions. The 343 functionality per-se will be kept, however. 344 345 Using this over ``record_xml_property`` can help when using ci tools to parse the xml report. 346 However, some parsers are quite strict about the elements and attributes that are allowed. 347 Many tools use an xsd schema (like the example below) to validate incoming xml. 348 Make sure you are using attribute names that are allowed by your parser. 349 350 Below is the Scheme used by Jenkins to validate the XML report: 351 352 .. code-block:: xml 353 354 <xs:element name="testcase"> 355 <xs:complexType> 356 <xs:sequence> 357 <xs:element ref="skipped" minOccurs="0" maxOccurs="1"/> 358 <xs:element ref="error" minOccurs="0" maxOccurs="unbounded"/> 359 <xs:element ref="failure" minOccurs="0" maxOccurs="unbounded"/> 360 <xs:element ref="system-out" minOccurs="0" maxOccurs="unbounded"/> 361 <xs:element ref="system-err" minOccurs="0" maxOccurs="unbounded"/> 362 </xs:sequence> 363 <xs:attribute name="name" type="xs:string" use="required"/> 364 <xs:attribute name="assertions" type="xs:string" use="optional"/> 365 <xs:attribute name="time" type="xs:string" use="optional"/> 366 <xs:attribute name="classname" type="xs:string" use="optional"/> 367 <xs:attribute name="status" type="xs:string" use="optional"/> 368 </xs:complexType> 369 </xs:element> 370 371LogXML: add_global_property 372^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 373 374.. versionadded:: 3.0 375 376If you want to add a properties node in the testsuite level, which may contains properties that are relevant 377to all testcases you can use ``LogXML.add_global_properties`` 378 379.. code-block:: python 380 381 import pytest 382 383 384 @pytest.fixture(scope="session") 385 def log_global_env_facts(f): 386 387 if pytest.config.pluginmanager.hasplugin("junitxml"): 388 my_junit = getattr(pytest.config, "_xml", None) 389 390 my_junit.add_global_property("ARCH", "PPC") 391 my_junit.add_global_property("STORAGE_TYPE", "CEPH") 392 393 394 @pytest.mark.usefixtures(log_global_env_facts.__name__) 395 def start_and_prepare_env(): 396 pass 397 398 399 class TestMe(object): 400 def test_foo(self): 401 assert True 402 403This will add a property node below the testsuite node to the generated xml: 404 405.. code-block:: xml 406 407 <testsuite errors="0" failures="0" name="pytest" skips="0" tests="1" time="0.006"> 408 <properties> 409 <property name="ARCH" value="PPC"/> 410 <property name="STORAGE_TYPE" value="CEPH"/> 411 </properties> 412 <testcase classname="test_me.TestMe" file="test_me.py" line="16" name="test_foo" time="0.000243663787842"/> 413 </testsuite> 414 415.. warning:: 416 417 This is an experimental feature, and its interface might be replaced 418 by something more powerful and general in future versions. The 419 functionality per-se will be kept. 420 421Creating resultlog format files 422---------------------------------------------------- 423 424.. deprecated:: 3.0 425 426 This option is rarely used and is scheduled for removal in 4.0. 427 428 An alternative for users which still need similar functionality is to use the 429 `pytest-tap <https://pypi.org/project/pytest-tap/>`_ plugin which provides 430 a stream of test data. 431 432 If you have any concerns, please don't hesitate to 433 `open an issue <https://github.com/pytest-dev/pytest/issues>`_. 434 435To create plain-text machine-readable result files you can issue:: 436 437 pytest --resultlog=path 438 439and look at the content at the ``path`` location. Such files are used e.g. 440by the `PyPy-test`_ web page to show test results over several revisions. 441 442.. _`PyPy-test`: http://buildbot.pypy.org/summary 443 444 445Sending test report to online pastebin service 446----------------------------------------------------- 447 448**Creating a URL for each test failure**:: 449 450 pytest --pastebin=failed 451 452This will submit test run information to a remote Paste service and 453provide a URL for each failure. You may select tests as usual or add 454for example ``-x`` if you only want to send one particular failure. 455 456**Creating a URL for a whole test session log**:: 457 458 pytest --pastebin=all 459 460Currently only pasting to the http://bpaste.net service is implemented. 461 462Disabling plugins 463----------------- 464 465To disable loading specific plugins at invocation time, use the ``-p`` option 466together with the prefix ``no:``. 467 468Example: to disable loading the plugin ``doctest``, which is responsible for 469executing doctest tests from text files, invoke pytest like this:: 470 471 pytest -p no:doctest 472 473.. _`pytest.main-usage`: 474 475Calling pytest from Python code 476---------------------------------------------------- 477 478.. versionadded:: 2.0 479 480You can invoke ``pytest`` from Python code directly:: 481 482 pytest.main() 483 484this acts as if you would call "pytest" from the command line. 485It will not raise ``SystemExit`` but return the exitcode instead. 486You can pass in options and arguments:: 487 488 pytest.main(['-x', 'mytestdir']) 489 490You can specify additional plugins to ``pytest.main``:: 491 492 # content of myinvoke.py 493 import pytest 494 class MyPlugin(object): 495 def pytest_sessionfinish(self): 496 print("*** test run reporting finishing") 497 498 pytest.main(["-qq"], plugins=[MyPlugin()]) 499 500Running it will show that ``MyPlugin`` was added and its 501hook was invoked:: 502 503 $ python myinvoke.py 504 . [100%]*** test run reporting finishing 505 506 507.. note:: 508 509 Calling ``pytest.main()`` will result in importing your tests and any modules 510 that they import. Due to the caching mechanism of python's import system, 511 making subsequent calls to ``pytest.main()`` from the same process will not 512 reflect changes to those files between the calls. For this reason, making 513 multiple calls to ``pytest.main()`` from the same process (in order to re-run 514 tests, for example) is not recommended. 515 516 517.. include:: links.inc 518