The Little Things: Testing with Catch2

This post will go over testing with Catch2 and will be very example heavy. I want to cover first the basic usage of Catch2 (tests, assertions, sections, test grouping) and then some more advanced and less used features. I will not cover every feature Catch2 has, just those that I think are most likely to be generally helpful.

Note that this post is not about the whys, hows, and whats of testing. I intend to write a post about that too, but this one exists to show off Catch2.

All examples in this post will be written against the v3 branch of Catch2.

Catch2 basics

As with all testing frameworks, the two most fundamental parts of Catch2 are test cases that contain assertions. Assertions exist in the REQUIRE^[1] macro and must be contained within a test case^[2], which in turn is created using the TEST_CASE macro.

The following simple example defines a single test case with 3 assertions. The test case is called "simple test case", which we can use to refer to the test case later. There is also an implementation of factorial with a tiny bug which the tests will run into.

#include <catch2/catch_test_macros.hpp>

static int factorial(int n) {
    if (n <= 1) {
        return n;
    }
    return n * factorial(n - 1);
}

TEST_CASE("Simple test case") {
    REQUIRE(factorial( 1) == 1);
    REQUIRE(factorial(10) == 3'628'800);
    REQUIRE(factorial( 0) == 1);
}

Compiling and running the example gives this (abridged) output:

...............................................................................

/app/example.cpp:13: FAILED:
  REQUIRE( factorial( 0) == 1 )
with expansion:
  0 == 1

===============================================================================
test cases: 1 | 1 failed
assertions: 3 | 2 passed | 1 failed

The interesting part of it is that in the case of a failure^[3], we see both the original expression, REQUIRE(factorial( 0) == 1), and the actual compared values: 0 == 1.

Do you see the bug?^[4]

Sections

Sections are a feature that is not common in the xUnit family of testing frameworks. They allow defining multiple paths through a test case. These paths can (partially) overlap and thus can be used to provide set-up and tear-down functionality. In the simple example below, there will be two paths through the test. The first one will print "1A\n", and the other will print "1B\n".

#include <catch2/catch_test_macros.hpp>

#include <iostream>

TEST_CASE("Section showcase") {
    std::cout << '1';
    SECTION("A") {
        std::cout << 'A';
    }
    SECTION("B") {
        std::cout << 'B';
    }
    std::cout << '\n';
}

(try it on godbolt)

Sections can also be nested. The following example defines 4 paths through the test case, printing "1Aa\n", "1Ab\n", "1Ba\n", and "1Bb\n" respectively.

#include <catch2/catch_test_macros.hpp>

#include <iostream>

TEST_CASE("Section showcase") {
    std::cout << '1';
    SECTION("A") {
        std::cout << 'A';
        SECTION("a") { std::cout << 'a'; }
        SECTION("b") { std::cout << 'b'; }
    }
    SECTION("B") {
        std::cout << 'B';
        SECTION("a") { std::cout << 'a'; }
        SECTION("b") { std::cout << 'b'; }
    }
    std::cout << '\n';
}

(try it on godbolt)

Ultimately, the use of SECTIONs boils down to defining a tree of tests that share some of the code. The tests are then run in a depth-first, top-to-bottom order.

Please note that while the only absolute limit on nesting sections is whatever your compiler can handle before giving out/running out of memory, nesting beyond 2-3 levels is usually unreadable in practice.

Stringifying custom types

In the very first example, when the assertion failed, Catch2 showed us the actual values on both sides of the comparison. To do this, it needs to know how to turn a type into a string it can display; otherwise, it will just show the value as "{ ? }". There are two ways^[5] to have your type properly stringified by Catch2:

1. Provide ADL-findable overload of operator<<(std::ostream&, T const&) for your type.
2. Specialize Catch::StringMaker<T> for your type.

The second option has higher priority, so if a type has both operator<< overload and StringMaker specialization, the specialization will be used.

(try it on godbolt)

Test case tagging and grouping

Test cases can also be associated with strings called tags. Tags have two purposes. One is to allow users of Catch2 to group tests that have something in common, e.g. tests for custom allocators, and the other is to mark a test as having some specific property, e.g. that it is expected to fail.

Test cases are assigned their tags via the second^[6] (optional) argument to TEST_CASE macro, e.g. TEST_CASE("widgets can be constructed from strings", "[widget][input-validation]") creates a test case with two tags, [widget] and [input-validation].

Some tags can also have special meaning. In general, Catch2 reserves tag names starting with "!" for its own purposes, e.g. [!shouldfail] inverts the pass/fail logic of a test. If an assertion fails, the test case succeeds, but if no assertion fails, then the test case fails. Catch2 also ascribes special meaning to tags starting with ".", e.g. [.] or [.widget]. These mark the tagged tests as "hidden" – hidden tests will be run if they are explicitly selected, they will not be run by default.

Let's take a look at an example:

#include <catch2/catch_test_macros.hpp>
#include <iostream>

TEST_CASE("first", "[A][foo]") {
    std::cout << "first\n";
}

TEST_CASE("second", "[B][.foo]") {
    std::cout << "second\n";
}

TEST_CASE("third", "[C][bar]") {
    std::cout << "third\n";
}

TEST_CASE("fourth", "[A][.][bar]") {
    std::cout << "fourth\n";
}

Compiling the tests above into their own binary and running it without further arguments will run tests "first" and "third" because the other two tests are hidden. Specifying the "[foo]" tag will run tests "first" and "second", and so on. You can also ask for all tests that are not tagged with "[foo]" by negating the tag: "~[foo]". This will run only one test, "third".

You can also specify multiple tags as the test filter; "[tag1][tag2]" means run tests that have both tags, "[tag1],[tag2]" means run tests that have either of the two tags.

More advanced features

There are three more advanced features that I want to showcase:

• Matchers
• Generators
• Benchmarking

Matchers

Matchers are helpful for testing more complex properties than can be expressed with a simple comparison operator. For example, if a function returns a set of values but does not promise a specific order, we cannot compare the result to expected values directly.

In Catch2, matchers are usually^[7] used in the REQUIRE_THAT(expression, matcher) macro. This is shown in the example below, where we check that the (shuffled) vector contains the correct elements in an unspecified order:

#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_vector.hpp>

#include <algorithm>
#include <random>

TEST_CASE("vector unordered matcher", "[matchers][vector]") {
    using Catch::Matchers::UnorderedEquals;
    std::vector<int> vec{0, 1, 2, 3, 4};
    
    std::shuffle(vec.begin(), vec.end(), std::random_device{});
    
    REQUIRE_THAT(vec, UnorderedEquals<int>({0, 1, 2, 3, 4}));
}

(try it on godbolt)

Catch2's matchers can also be combined together with logical operators &&, || and !. These do what you expect given their meaning for boolean expression, so that matcher1 && !matcher2 only accepts input if matcher1 accepts it and matcher2 does not. Thus, in the example below, the combined matcher requires the input string to either not contain "MongoDB" or "web scale".

#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_string.hpp>

std::string description() {
    return "MongoDB is web scale!";
}

TEST_CASE("combining matchers") {
    using Catch::Matchers::Contains;
    
    REQUIRE_THAT(description(),
                 !Contains("MongoDB") || !Contains("web scale"));
}

(try it on godbolt)

For more on Catch2's matchers (e.g. which matchers are implemented in Catch2 and how to implement your own matchers), look into the matcher documentation.

Generators

Generators are Catch2's implementation of data-driven testing. The core idea is that you can keep the same test code but feed the test code different inputs to test different cases.

Data generators are declared inside test cases with the GENERATE macro, and a generator expression inside it. The example below shows a test case that will be run for 3 different inputs - 2, 4, and 6:

#include <catch2/catch_test_macros.hpp>
#include <catch2/generators/catch_generators.hpp>

TEST_CASE("Simple generator use") {
    auto number = GENERATE(2, 4, 5);
    CAPTURE(number);
    REQUIRE(number % 2 == 0);
}

(try it on godbolt)

Generators can be mixed with sections. When doing so, you can reason about them as if they defined another section from their GENERATE statement until the end of the scope, and that section will be entered for each generated input. This means that the example below will print 6 lines, "A\n", "B\n", "B\n", "A\n", "B\n", and "B\n".

#include <catch2/catch_test_macros.hpp>
#include <catch2/generators/catch_generators.hpp>

#include <iostream>

TEST_CASE("Simple generator use") {
    auto number = GENERATE(2, 4);
    SECTION("A") {
        std::cout << "A\n";
    }
    SECTION("B") {
        auto number2 = GENERATE(1, 3);
        std::cout << "B\n";
    }
}

(try it on godbolt)

Catch2 also provides some built-in utility generators, like table, which helps with defining sets of inputs and the expected results:

#include <catch2/catch_test_macros.hpp>
#include <catch2/generators/catch_generators.hpp>

#include <string.h>
#include <tuple>

TEST_CASE("tables", "[generators]") {
    auto data = GENERATE(table<char const*, int>({
        {"first", 5},
        {"second", 6},
        {"third", 5},
        {"etc...", 6}
    }));

    REQUIRE(strlen(std::get<0>(data)) == static_cast<size_t>(std::get<1>(data)));
}

(try it on godbolt)

There is also variety of higher-order generators, e.g. filter, or take. These can be used to create complex test data generators, like in the example below where we generate 10 odd random integers in range [-100, 100]:

#include <catch2/catch_test_macros.hpp>
#include <catch2/generators/catch_generators_adapters.hpp>
#include <catch2/generators/catch_generators_random.hpp>

TEST_CASE("Chaining generators") {
    auto i = GENERATE(take(10, filter([](int i) {
                              return i % 2 == 1;
                           }, random(-100, 100))));
    REQUIRE(i > -100);
    REQUIRE(i < 100);
    REQUIRE(i % 2 == 1);
}

(try it on godbolt)

For more on Catch2's generators (e.g. which generators are implemented in Catch2 and how to implement your own), look into the generator documentation.

(Micro)Benchmarking

Catch2 also provides basic microbenchmarking support. You can insert a benchmark into any test case using the BENCHMARK macro followed by a block of code to benchmark. You can also combine benchmarks and assertions^[8], as shown in the example below:

#include <catch2/catch_test_macros.hpp>
#include <catch2/benchmark/catch_benchmark.hpp>

static int factorial(int n) {
    return n <= 1? 1 : n * factorial(n-1);
}

TEST_CASE("Simple benchmark") {
    REQUIRE(factorial(12) == 479'001'600);

    BENCHMARK("factorial 12") {
        return factorial(12); // <-- returned values won't be optimized away
    }; // <--- !! semicolon !!
}

(try it on godbolt)

If you want to run benchmarks for different input sizes, you can combine generators with benchmarks, like in the example below:

#include <catch2/catch_test_macros.hpp>
#include <catch2/benchmark/catch_benchmark.hpp>
#include <catch2/generators/catch_generators.hpp>

static int factorial(int n) {
    return n <= 1? 1 : n * factorial(n-1);
}

TEST_CASE("Validated benchmark") {
    int input, expected_result;
    std::tie(input, expected_result) = GENERATE(table<int, int>( {
        {0, 1},
        {1, 1},
        {5, 120},
        {10, 3'628'800},
        {12, 479'001'600},
    }));

    REQUIRE(factorial(input) == expected_result);

    BENCHMARK("factorial " + std::to_string(input)) {
        return factorial(input);
    };
}

(try it on godbolt)

For more on Catch2's microbenchmarking support (e.g. how to handle constructors and destructors, or how to add a setup step for your benchmark), look into the benchmarking documentation.

Final words

The above is by no means everything that Catch2 provides. I picked three features that I feel are most widely useful while being least widely known, and just on top of my head, I know I have skipped over at least:

• Templated test cases (same test across different types)
• Running specific sections in a test case
• Running test cases in random order
• Facilities for comparing floating-point numbers
• Writing your own reporters
• Logging extra information during a test run

And even I definitely do not remember everything present in Catch2. However, most^[9] of the things provided are documented, and often you can find handy features by reading through the documentation.