Running Tests via the CLI¶
To find and run tests in your project, you can run
ward without any arguments.
This will recursively search through the project for modules with a name starting with
test_ or ending with
and execute any tests contained in the modules it finds.
A test in Ward can finish with one of several different outcomes. The outcome of a test will be clearly indicated during the run, and a summary of those outcomes is displayed after the run completes or is cancelled.
PASS: The test passed. It completed without any exceptions occurring or assertions failing.
FAIL: The test failed. An exception occurred or an assertion failed.
SKIP: The test was skipped. It wasn’t executed at all because it was decorated with
XFAIL: An expected failure. The test is decorated with
@xfail, indicating that we currently expect it to fail… and it did!
XPASS: An unexpected pass. The test is decorated with
@xfail, indicating that we expected it to fail. However, the test passed unexpectedly.
DRYRUN: The status is only used during a dry-run (using the
--dry-runoption). The test nor any injected fixtures were executed.
Specifying test paths with
You can run tests in a specific directory or module using the
--path option. For example, to run all tests inside a directory named
ward --path tests.
To run all the tests in your project, you can just type
ward from anywhere inside your project.
Ward considers your project to be the directory containing your
pyproject.toml config file and all directories within. If you don’t have a
pyproject.toml file, then
Ward will look for a
.hg folder/file and consider that as your project root.
If Ward cannot find a project root, the running
ward without a
--path is equivalent to running
ward --path ..
You can directly specify a test module, for example:
ward --path tests/api/test_get_user.py.
You can supply multiple test directories by providing the
--path option multiple times:
ward --path "unit" --path "integration".
Ward will run all tests it finds across all given paths. If one of the specified paths is contained within another, they’ll only be included once. Ward will only run a test once per session.
Excluding modules or paths with
You can tell Ward to ignore specific modules or directories using the
--exclude command line option. For example:
ward --exclude path/to/dir1 --exclude path/to/dir2
You can also exclude paths using
[tool.ward] exclude = ["tests/resources", "tests/utilities.py"]
Loosely search for tests with
You can choose to limit which tests are collected and ran by Ward using the
--search option. Module names, test descriptions and test function bodies will be searched, and those which contain the argument will be ran.
Here are some examples:
Run all tests that call the
ward --search "fetch_users("
Run all tests that check if a
ward --search "raises(ZeroDivisionError)"
Run all tests decorated with the
ward --search "@xfail"
Run a test described with
"my_function should return False":
ward --search "my_function should return False"
Running tests inside a module: The search takes place on the fully qualified name, so you can run a single module (e.g. my_module) using the following command:
ward --search my_module.
Of course, if a test name or body contains the string
"my_module.", that test will also be selected and will run.
This approach is useful for quickly running tests which match a simple query, making it useful for development.
Customising the output with
As your project grows, it may become impractical to print each test result on its own line. Ward provides alternative test output styles that can be configured using the
ward --test-output-style [test-per-line|dots-module|dots-global|live]
The default test output of Ward looks like this (
If you run Ward with
--test-output-style=dots-module, each module will be printed on its own line, and a single character will be used to represent the outcome of each test in that module:
If that is still too verbose, you may wish to represent every test outcome with a single character, without grouping them by modules (
--test-output-style=live will cause Ward to output results in the format shown below.
Displaying test session progress with
Ward offers two ways of informing you of progress through a test run: inline progress percentage (on by default), and/or a dynamic progress bar.
By default, the percentage progress through a test run will appear at the right hand side of the output, which corresponds to
You can also have Ward display a dynamic progress bar during the test run, using the
--progress-style bar option.
If you wish, can pass supply
--progress-style with multiple times (to display a progress bar and inline progress, for example).
The progress bar is currently only available with the default output mode (
By default, Ward captures everything that is written to stdout and stderr as your tests run. If a test fails, everything that was printed during the time it was running will be printed as part of the failure output.
With output capturing enabled, if you run a debugger such as pdb during test execution, everything it writes to the stdout will be captured by Ward too.
Disabling output capturing with
If you wish to disable output capturing you can do so using the
--no-capture-output flag on the command line.
Anything printed to stdout or stderr will no longer be captured by Ward, and will be printed to the terminal as your tests run,
regardless of outcome.
You can also disable output capturing using the
capture-output config in your
[tool.ward] capture-output = false
Randomise test execution order with
--order "random" when running your tests to have Ward randomise the order they run in:
ward --order "random".
Running tests in a random order can help identify tests that have hidden dependencies on each other. Tests should pass regardless of the order they run in, and they should pass if run in isolation.
To have Ward always run tests in a random order, use the
order config in your
[tool.ward] order = "random"
Cancelling after a number of failures with
If you wish for Ward to cancel a run immediately after a specific number of failing tests, you can use the
--fail-limit option. To have a run end immediately after 5 tests fail:
ward --fail-limit 5
Finding slow running tests with
--show-slowest N to print the N tests with the highest execution time after the test run completes.
Performing a dry run with
--dry-run option to have Ward search for and collect tests without running them (or any fixtures they depend on).
--dry-run, tests will return with an outcome of
This is useful for determining which tests Ward would run if invoked normally.
Format strings in test descriptions may not be resolved during a dry-run, since no fixtures are evaluated and the data may therefore be missing.
Displaying symbols in diffs with
--show-diff-symbols when invoking
ward in order to have the diff output present itself with symbols instead
of the colour-based highlighting. This may be useful in a continuous integration environment that doesn’t support coloured terminal
Debugging your code with
Ward will automatically disable output capturing when you use pdb.set_trace() or breakpoint(), and re-enable it when you exit the debugger.