How to get code coverage from CI

Recently I wanted to know how well we test Catch2 during CI (Continuous Integration). To this end, I decided to collect code coverage statistics from the tests we run for each commit and integrate it with our GitHub. I knew about two services that provide GitHub coverage integration, coveralls and codecov and the cpplang slack recommended me codecov^[1] and so I spent about three days of work (interspersed over a week) getting things to work^[2] on both TravisCI and AppVeyor.

Because that is an entirely unacceptable amount of time to set up something that should be common enough, I decided to write a post and make it easier for the next person. Do note that parts of this are written assuming you have CMake + CTest setup. If you are using something different, you will have to adapt what is written here somewhat.

Obviously, the first step is to register your project at codecov.io.^[3] Afterwards, you can start adding steps to your build to generate the coverage info from your test suite. I recommend starting with Travis Linux builds because they are much more straightforward to get started.

Travis CI

On Linux, you will want to use gcov to generate test coverage. However, Clang does not talk gcov natively, so I recommend using something to abstract over the differences -- for Catch2 I am using CMake-codecov. After copying its file to <ProjectDir>/CMake, you need to modify your CMakelists.txt like so:

    if (ENABLE_COVERAGE)
        list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}/CMake")
        find_package(codecov)
        add_coverage(SelfTest)
        list(APPEND LCOV_REMOVE_PATTERNS "/usr/")
        coverage_evaluate()
    endif()

Where SelfTest is the name of Catch's test suite target, /usr/ is where the standard library headers live on Linux and coverage_evaluate() is a special functions that needs to be called after coverage is added to all desired targets.

You will also need to modify your .travis.yml, to add new post-build steps. For Catch2 it was

if [[ "${TRAVIS_OS_NAME}" == "linux" ]]; then
    make gcov
    make lcov
    bash <(curl -s https://codecov.io/bash) -X gcov || echo "Codecov did not collect coverage reports"
fi

This runs two special targets generated by CMake-codecov, gcov and lcov to extract and preprocess coverage information gained from running tests using CTest and then uploads the results using codecov.io's bash uploader.

You might also need to install lcov package on your Travis image.

AppVeyor

Generating code coverage for C++ code on Windows seems to be a surprisingly obscure topic. After some googling, I found OpenCppCoverage, which is a stand-alone code coverage tool, which can generate output in a format that codecov.io recognises (cobertura).

Using it on its own is fairly straightforward:
OpenCppCoverage C:\projects\Catch2\Build\Debug\SelfTest.exe
will generate an HTML report in the same directory where it was run. It contains coverage information for all files compiled into the binary, including the standard library. This makes it a good idea to set some source file filtering, like so:
OpenCppCoverage --sources C:\projects\Catch2 -- C:\projects\Catch2\Build\Debug\SelfTest.exe
The --sources <path> argument tells OpenCppCoverage to only include files whose path starts with <path> in the final report.

However, using it in context of an already existing test suite in CTest is not, because CTest does not allow you to extract the test commands easily. Luckily, CTest has some Valgrind integration and even lets you specify your own Valgrind-alike tool and we can use that to have all tests redirected to our own executable^[4].

Note that we have to rewrite paths passed to --sources, because OpenCppCoverage does not normalise them and so if you call it this way OpenCppCoverage --quiet --sources C:/projects/Catch2 -- C:/projects/Catch2/Build/Debug/SelfTest.exe the resulting report will be empty^[5]. However, the tests will still run, making the issue annoyingly hard to diagnose. We also specify a different export type and filename, via --export_type binary:cov-report<num>.bin, to save the coverage information in a binary format that we can work with later on, in a file named cov-report<num>.bin.

You will also need to modify CMakelists.txt to enable CTest's MemCheck integration, but it should be enough to replace enable_testing() with include(CTest).

On the AppVeyor side, if you want to keep coverage collection to Debug builds (doing so in Release builds tends to lead to weird results), integrating with AppVeyor will be harder, because of a known bug in how multiline batch scripts in appveyor.yml are passed to cmd.

The above-mentioned bug means that non-trivial ifs need to be kept in separate script files. In Catch2, we ended up with 2 extra batch scripts, one to configure the build, and the other to run the tests and upload coverage to codecov.io.

In your appveyor.yml, you have to install OpenCppCoverage and codecov uploader utility. This installs the python uploader from pip, but you can also install both from chocolatey^[6]:

install:
  - ps: if (($env:CONFIGURATION) -eq "Debug" ) { python -m pip install codecov }
  - ps: if (($env:CONFIGURATION) -eq "Debug" ) { .\scripts\installOpenCppCoverage.ps1 }

installOpenCppCoverage.ps1 is a separate powershell script^[4:1] to handle downloading and installing an OpenCppCoverage release.

When configuring CMake build, you need to configure the MemoryCheck binary for CTest redirection. Doing so looks like this:

cmake -H. -BBuild -A%PLATFORM% -DMEMORYCHECK_COMMAND=build-misc\Debug\CoverageHelper.exe -DMEMORYCHECK_COMMAND_OPTIONS=--sep-- -DMEMORYCHECK_TYPE=Valgrind

build-misc\Debug\CoverageHelper.exe is our binary that parses arguments given to it by CTest, prepares a call to OpenCppCoverage and then passes output from the tests back to CTest.

To get ctest to send the test commands to the MemCheck binary, you need to call ctest with special configuration argument, like so:

ctest -j 2 -C %CONFIGURATION% -D ExperimentalMemCheck

This will run all of your tests registered with CTest through your redirection binary and thus generate coverage reports for all of their runs.

Afterwards, you will need to merge the resulting files, transform them into a format that codecov.io understands and upload the resulting report. OpenCppCoverage can do this for you, you just need to call it like this:

OpenCppCoverage --quiet --export_type=cobertura:cobertura.xml --input_coverage=<file1> --input_coverage=<file2> ...

Note that the return value of this command will be the highest return value from all the runs. In Catch2, we have a Python script that collects the coverage outputs and calls OpenCppCoverage^[4:2].

Then you will need to upload the final report like this:

codecov --root .. --no-color --disable gcov -f cobertura.xml -t %CODECOV_TOKEN%

Note that we supply %CODECOV_TOKEN% even though the documentation says it is not needed for public repository. The documentation lies and without the token uploads from AppVeyor fail.

The final sequence of commands for Debug build looks then like this:

ctest -j 2 -C %CONFIGURATION% -D ExperimentalMemCheck
python ..\misc\appveyorMergeCoverageScript.py
codecov --root .. --no-color --disable gcov -f cobertura.xml -t %CODECOV_TOKEN%

Customizing codecov

After you are done, you will likely want to customize the codecov.io report further, ie by removing some files from the coverage report, or defining expected scale of your coverage. As an example, in Catch2 we ignore our test files, external dependencies and non-default reporters:

coverage:
  ignore:
    - "projects/SelfTest"
    - "**/catch_reporter_tap.hpp"
    - "**/catch_reporter_automake.hpp"
    - "**/catch_reporter_teamcity.hpp"
    - "**/external/clara.hpp"

When customizing codecov, use codecov.yml, not .codecov.yml or ignoring files will not work. This is another known bug of codecov.io.