Code Coverage

What is an acceptable level of code coverage?

Choosing an arbitrary percentage seems unsatisfying.

I've read that code coverage in a project should increase over time. This seems like good advice when introducing tests to a legacy codebase.

Uncle Bob suggests 100% as the only reasonable asymptotic goal. I largely agree but think we can refine this a little.

Let's consider the reasons for untested code. Here are two groups:

1. Valid reasons:
   • Calling an external service (web API, database, etc.) that might change or become unavailable.
   • There might be some more.
2. Invalid reasons:
   • Sloppy TDD.
   • Code added by a coworker.
   • Deleting tests to avoid maintaining them.
   • There are many more.

There is a third camp but I think that these can be assigned to one of the above on a case by case basis:

3. Sometimes valid
   • Automatically generated code.
   • etc...

If there are only valid and invalid reasons to test then it follows that there is

• code that is desirable to test and
• code that is not desirable to test

An obvious conclusion is wanting to test all of the first category (desirable to test) and none of the second (undesirable to test).

What if we separate these two categories by putting them in separate components? For example, the web service call can go in its own small component. If we do that then we really can achieve 100% coverage on the code we want to test and can be happy with 0% on the residue.

If a project uses this convention then it follows that there are two acceptable levels of code coverage: 100% and 0%, and the latter can be excluded from the metrics.

I can see some advantages to this approach:

• a well-defined, attainable level of code coverage.
• ability to trivially detect gaps (if it's not 100% then there are gaps).
• visibility over which parts must be covered in an end-to-end test.
• forcing the injection of untestable code.

The last one is the key benefit as far as I am concerned. The untestable code is likely to be calls to external parts of the system like web APIs, databases, third party libraries, etc. The plugin architecture we end up with makes it easier to substitute replacements. By the time the system is complete, there will already be two alternatives: the real one and a test double. Writing a third should be relatively simple.

21 January 2018