Crate test_casing

Minimalistic testing framework for generating tests for a given set of test cases and decorating them to add retries, timeouts, sequential test processing etc. In other words, the framework implements:

• Parameterized tests of reasonably low cardinality for the standard Rust test runner
• Fully code-based, composable and extensible test decorators.

Overview

Test cases

test_casing attribute macro wraps a free-standing function with one or more arguments and transforms it into a collection of test cases. The arguments to the function are supplied by an iterator (more precisely, an expression implementing IntoIterator).

For convenience, there is TestCases, a lazy iterator wrapper that allows constructing test cases which cannot be constructed in compile time (e.g., ones requiring access to heap). TestCases can be instantiated using the cases! macro.

Since a separate test wrapper is generated for each case, their number should be reasonably low (roughly speaking, no more than 20). Isolating each test case makes most sense if the cases involve some heavy lifting (spinning up a runtime, logging considerable amount of information, etc.).

Test decorators

decorate attribute macro can be placed on a test function to add generic functionality, such as retries, timeouts or running tests in a sequence.

The decorators module defines some basic decorators and the DecorateTest trait allowing to define custom decorators. Test decorators support async tests, tests returning Results and test cases; see the module docs for more details.

Test cases structure

The generated test cases are placed in a module with the same name as the target function near the function. This allows specifying the (potentially qualified) function name to restrict the test scope.

If the nightly crate feature is not enabled, names of particular test cases are not descriptive; they have the case_NN format, where NN is the 0-based case index. The values of arguments provided to the test are printed to the standard output at the test start. (The standard output is captured and thus may be not visible unless the --nocapture option is specified in the cargo test command.)

If the nightly feature is enabled, the names are more descriptive, containing Debug presentation of all args together with their names. Here’s an excerpt from the integration tests for this crate:

test number_can_be_converted_to_string::case_1 [number = 3, expected = "3"] ... ok
test number_can_be_converted_to_string::case_2 [number = 5, expected = "5"] ... ok
test numbers_are_large::case_0 [number = 2] ... ignored, testing that `#[ignore]` attr works
test numbers_are_large::case_1 [number = 3] ... ignored, testing that `#[ignore]` attr works
test string_conversion_fail::case_0 [bogus_str = "not a number"] - should panic ... ok
test string_conversion_fail::case_1 [bogus_str = "-"] - should panic ... ok
test string_conversion_fail::case_2 [bogus_str = ""] - should panic ... ok

The names are fully considered when filtering tests, meaning that it’s possible to run particular cases using a filter like cargo test 'number = 5'.

Alternatives and similar tools

• The approach to test casing from this crate can be reproduced with some amount of copy-pasting by manually feeding necessary inputs to a common parametric testing function. Optionally, these tests may be collected in a module for better structuring. The main downside of this approach is the amount of copy-pasting.
• Alternatively, multiple test cases may be run in a single #[test] (e.g., in a loop). This is fine for the large amount of small cases (e.g., mini-fuzzing), but may have downsides such as overflowing or overlapping logs and increased test runtimes.
• The test-case crate uses a similar approach to test case structuring, but differs in how test case inputs are specified. Subjectively, the approach used by this crate is more extensible and easier to read.
• Property testing / quickcheck-like frameworks provide much more exhaustive approach to parameterized testing, but they require significantly more setup effort.
• rstest supports test casing and some of the test decorators (e.g., timeouts).
• nextest is an alternative test runner that supports most of the test decorators defined in the decorators module. It does not use code-based decorator config and does not allow for custom decorator.

Crate features

nightly

(Off by default)

Uses custom test frameworks APIs together with a generous spicing of hacks to include arguments in the names of the generated tests (see an excerpt above for an illustration). test_casing actually does not require a custom test runner, but rather hacks into the standard one; thus, the generated test cases can run alongside with ordinary / non-parameterized tests.

Requires a nightly Rust toolchain and specifying #![feature(test, custom_test_frameworks)] in the using crate. Because custom_test_frameworks APIs may change between toolchain releases, the feature may break. See the CI config for the nightly toolchain version the crate is tested against.

Re-exports

• pub use crate::test_casing::case;
• pub use crate::test_casing::ArgNames;

Modules

• decorators
  Test decorator trait and implementations.

Macros

• cases
  Creates TestCases based on the provided expression implementing IntoIterator (e.g., an array, a range or an iterator).

Structs

• Product
  Cartesian product of several test cases.
• ProductIter
  Iterator over test cases in Product.
• TestCases
  Container for test cases based on a lazily evaluated iterator. Should be constructed using the cases! macro.

Attribute Macros

• decorate
  Wraps a tested function to add retries, timeouts etc.
• test_casing
  Flattens a parameterized test into a collection of test cases.