One day I was venting on the Cpp Slack about people submitting low-quality issues that need a lot of time to read, thus making maintaining Catch2 take up a lot more time than it needs to. And, because just venting might be satisfying but does not solve anything, I decided to write this post about submitting good bug reports.

Before we start, one thing needs to be made clear: submitting good bug reports takes more time than submitting bad bug reports, sometimes considerably so. However, spending the extra time is in your own interest, as creating good bug reports increases the chances of your bug being fixed quickly. This is because

  • good bug reports are looked-at first, and
  • good bug reports make it easier for the maintainers to fix the problem.

The first is important because there are always more bugs to fix than there is time, new features to be implemented and possible refactoring to made. Making your bug report pleasant to read is an easy way to get in front of the queue. The second is important because the less energy the maintainer has to spend figuring out what needs to be fixed, the more energy he can spend to figure out how it can be fixed.

Now, without further ado...

What should go into the bug report

  • Description of the bug

This means what should happen versus what does happen. Sometimes this is easy ("this code should compile, but it doesn't"), sometimes this is more subtle ("The JUnit reporter should write out all assertions, but it writes out only the failed ones"), but without a description of the bug, there is no bug report.

Also keep in mind that your bug report usually needs a title, and while the title should be kept short, it needs to be accurate. A recent example of a bad title was

Not working hidden test tags

where, after reading the reproduction steps, it turns out that the actual issue was that the ParseAndAddCatchTests.cmake script registered hidden tests in the CTest runner even though it shouldn't have[1].

  • Steps to reproduce the bug

These should be as short (and simple) as possible, and self-contained. Ideally, the steps consist of downloading a single file, compiling it and then running the resulting binary. Cloning a simple repository is still ok, but we've had an issue where the repro steps boiled down to

  1. Clone a repo
  2. Build it using a Ruby-based tool on Windows
  3. Wait for ~20 minutes of compilation
  4. Get inscrutable compilation error

In the end, I fixed it anyway, but it would take me less time (and thus let me respond to more issues), and it would be fixed a lot faster if the person reporting it reduced and simplified the repro steps.

  • Additional information

Every bug report should contain some extra information. The first and foremost should be the version of the software your bug report is written against. A bug filed against a no-longer-supported version is pretty pointless by itself, and you should check if it still reproduces with the current version.

This also applies towards non-released versions of the code, such as git's commit hash or SVN's revision number. If you aren't using a released version, but are living off a specific commit in version control, you should say what commit you are using, and definitely should check that a newer commit doesn't fix your issue.

Another important piece of extra information is the compiler version[2] and platform. Sometimes an issue can only be reproduced with a sufficiently new compiler, or maybe there was a bug in the compiler, and the compiler version is still used widely-enough to warrant implementing a workaround in the project itself.

This means that when reporting a bug, you should check if you can reproduce it using a different version of the same compiler, different compiler, or even different version of the standard library[3].

Miscellaneous tips

While the text above explains the fundamentals of writing a good bug report, this part provides some (less important) miscellaneous tips towards writing a good bug report and addresses things that I've seen people do on Catch's issue tracker.

  • Projects often have a template for reporting[4]

GitHub allows projects to provide a template that prefills the issue (bug report) text. If you are opening an issue against a project that has it, READ IT and use it. However, you should also remember to use your brain. As an example, Catch's issue template contains this part asking for extra information:

### Extra information
<!--
Fill in any extra information that might be important for your issue.

If your issue is a bugreport, definitely fill out at least the following.
-->
* Catch version: **v42.42.42**
* Operating System: **Joe's discount operating system**
* Compiler+version: **Hidden Dragon v1.2.3**

If you are opening up an issue to request a new feature, the usefulness of "Catch version" field is low, so you can probably leave it out. The usefulness of the other two fields is next to zero, so you should leave them out.

  • Spare some thought to how you provide attachments

When the attachment is a short (~20 lines) file, placing it inline with the bug report's text is ok, and possibly even a better idea than separating it out. However, large attachments are better done as a separate file, linked from the text.

  • Use markdown properly

Related to the previous point. If you are opening bug report in a system that supports markdown (or other ways of formatting), take at least a little while to figure it out. As an example, paragraph headings are an excellent way of visually separating different parts of the bug report, and if you want to paste the compilation error to your bug report, it should be enclosed in a code block, or other "verbatim" environment[5].

  • Be careful when filtering errors for the bug report

We've had an issue about compilation error because of an ambiguous operator overload. The code was proprietary, and the submitter did not reduce it into a publicly publishable test case, which wouldn't be too bad if he also didn't cut out the part of the compilation error that says which overloads were ambiguous. But he did, and the issue went unsolved for a while...

  • Keep up with the bug report

Often an issue can be solved just from the initial filing, but sometimes it cannot, e.g. when you are using a niche platform[6]. However, even if the issue was solved just from the original information provided, projects will often request your confirmation that the bug was fixed. Other cases where you should report back are when you found a workaround, or the issue no longer reproduces, or it turns out that the bug was from a 3rd party, or ...

  • Have some manners

This point should be almost painfully obvious and yet I have to write it down. Having manners in this context does not mean niceties like starting the bug report with "Hello"[7], but rather not being this guy[8], who started an issue with "... or is it you who cannot make it work", because he was too lazy read either the documentation or output of -h, and then he started waving around his position as a CTO. Just don't be him, okay?

Examples

Now you know what (and why) goes into a proper bug report, but I wanted to provide some examples of useful bug reports.

This bug report nicely describes the problem, points out that documentation became out of sync with reality and even pinpoints down the commit that changed it.

Again, another bug report with a good explanation of the problem, good reproduction steps and even suggests a reasonable fix.

This is an excellent example of keeping up with the issue. At one point a bug was filed against 3rd party, but as it turned out the original code contained an error, and the person who submitted the bug reported this back to our issue tracker.

That's all folks. Also, if you are not actively opening bug reports when things go wrong, you should. It takes some time and effort, but over time that effort translates into a better software.

  1. As of the time of writing, I was also unable to reproduce the issue locally, and the original author didn't answer further inquiries. ↩︎

  2. This post is implicitly aimed at C++ developers and C++ OSS projects, but you can just substitute interpreter version if you are working in, e.g. Python. ↩︎

  3. As a teacher, I always tell my students that they should look for bugs in their code first, as the compiler and standard library are battle tested and relatively bug-free. However, in practice compilers, interpreters, standard libraries and co. also contain bugs. They are rare, but they do happen, e.g. with Catch, even after no longer providing support for various old compilers, we still had to implement workarounds for various 3rd party bugs. ↩︎

  4. If you have a project on GitHub with an active community, consider providing these. When I added them to Catch, the quality of newly opened issues increased considerably. ↩︎

  5. Thanks to GitHub's automatic linking of issues and people not using markdown correctly, the first couple of issues are linked to from other random issues. ↩︎

  6. We have had a bug that reproduced only with a specific version of compiler targetting IBM AIX. As you might imagine, neither of the maintainers had access to such system, so there was no way for us to test the fix ourselves. ↩︎

  7. However, I will admit that an issue starting along the lines of "Thank you for the great library" is always nice to read. ↩︎

  8. Even though all of the "dos" and "don'ts" are based on things people actually did Catch's issue tracker, I wanted to avoid calling out specific issues/people. In the end, I decided to make one exception. ↩︎