Test Coverage: More isn't always better.

It's my very first day as an apprentice for 8th Light, and I'm hitting the ground running. In fact, I might argue that the first thing they tell me is the most important. And what's the very first thing to know as an apprentice?

Keep your tests consistent, simple and easy-to-read.

Here's what Justin Searls at Test Double has to say about it.

what spoke to me though, was to avoid redundant test coverage.

There's a lot of talk (at least in my circles) about having as much test coverage as possible in your code base. 75% coverage? better than 55%, but better still is 95%. People make it a point of pride to have 100% coverage. And why not? that means you'll catch anything that breaks immediately, right?

And if 100% is good 120% is really good. and 256% is really really really good. If enough is good, then more than enough is even better.

Well, that's not actually true for anything in the world. Except for money. But besides money, it isn't good for your ice cream intake, it isn't good for your sun tan, and it isn't good for the health of your test suite either.

Even the money thing is debatable... but anyway, back on track:

The problem with more-than-100% coverage is that it can only mean one thing. At least some of the production code is covered twice. But what's wrong with that? I mean, it's covered, right?

Redundancy isn't DRY, so that's a smell right there. And more lines of code means more code to maintain, and more code to go wrong.

But having tried the "making 150% sure my code was covered" policy myself, I can say it becomes a problem when you need to make a change. You change one line of code, twenty-five tests fail. Why?! Because that one line of code has actually been covered twenty-five different ways.

That has a number of detrimental effects on your productivity.

  1. For every line of code you change, you have to also change 25+ other lines of test cases. That is not even close to an efficient use of your time. And if that's not bad enough, you are a human. Humans have feelings. So there are further negative effects for you.

  2. It's a drag, man. You get bummed-out if every line of code you change creates that much drudgery to wade through. And bummed-out developers are not as good at writing brilliant code as non-bummed-out developers. Developing is a creative process. Your mood matters.

  3. Anticipating that massive breakage also makes you afraid to make improvements. This is the exact opposite of what you want. You want the code to be easy to change. That's what the tests are there to support, to allow you to experiment and find the best way without fear. But if you're afraid to take risks because you're worried about the twenty-five lines of test cases you have to fix (when nothing is actually broken), then you won't try new things. You won't refactor the code to make it easy to maintain, and it will cost everyone time (read: money) as they slog through your poor code and brittle test suite.

So with that in mind, if you've got 75% coverage, sure, shoot for 100%. But if you have a choice between 95% coverage and 105%, it's better to stick with the 95%...

Then nudge it up to 97%.