We love a dubious dichotomy in software engineering: orchestration vs choreography, REST vs GraphQL, good developers vs JavaScript developers and so on. In this article, we’re going to take advantage of that little obsession, and pit mocking against not mocking in unit tests.
If you’re already thinking perfectly reasonable things like “surely, it depends” or “right tool for the job” or “this is stupid”, kindly shush your brain or skip to the end. For one thing, you’re undermining my article’s delicate sense of self-worth. Secondly, you’ll need to keep an open mind as we attempt to grade each opposing strategy.
How will we grade them? The qualities I’ve gone for are:
Ease of Use
How much effort is needed to apply the strategy.
Stability
How that strategy copes with an evolving application.
Tests as Documentation
How well it explains the application to developers.
Regression Prevention
How likely it is to stop bugs from ever going live.
Each of these qualities will then be given a score out of 10.
Yes, my scoring system is a bit more Top Trumps than it is Kent Beck, but it should do the job.
It’s time to meet our first contender…
Nonstop Mocks
In a pro-mock world everything but the unit under test is mocked. Typically, that means all of its dependencies, its collaborators and often external code composed within it, are all mocked. Here are a couple of quick examples to show how that might look:
- React examples
- OO examples (coming soon)
How big or small is a ‘unit’?
The pro-mock stance doesn’t necessitate a certain size, you’ll just end up with a greater or lesser number of mocks accordingly. Typically though, a unit ends up being a class or a function or a component – the responsibility and complexity limits for each, and thus the ‘unit’, are left up to the developer.
Ease of Use: 10/10
The superpower of this approach is that it works with absolutely everything. Whatever the code, whatever it’s doing, you can knock up the tests with little to no mental effort at all.
Try and use TDD here and you’ll often be hard pressed to think of all the mockable entities needed upfront. Instead, simply apply Nonstop Mocks after the unit has already been written and it’ll be an absolute breeze.
Should you want it, 100% code coverage is also within easy reach. Write enough test cases, with your mocks pre-arranged in every required permutation, and you can travel down every logic path of your unit in no time.
As such, Ease of Use gets a perfect score of 10 out of 10. Naughty developers who choose to omit sufficient tests can be cut off from craft beers accordingly.
Stability: 5/10
This one becomes a tale of two very distinct scenarios.
By their nature, mocks keep us loosely coupled to everything but interfaces. So long as the interfaces of other mocked units don’t change – no code changes within them can make our unit’s tests fail. If this was the only scenario, Stability would be another perfect 10 out of 10.
But… it isn’t.
Cracks begin to appear as we consider refactoring scenarios. Refactor without needing to change the interface of your unit, or the interface of anything it mocks, or the sequencing or the invocations of those same mocks… and your existing tests should still pass.
Move outside of those tight restrictions and you’re going to need to change your tests to make them pass again. Want to add a new feature, use a different design pattern or even fix a non-trivial bug? Your highly-implementation-detail-aware tests are going to break.
What’s the big deal? Well, if you need to rewrite your tests to make them pass after a bit of light refactoring, you’ve got to wonder – what exactly are they protecting?
During refactoring, failing tests tell you one thing only: the unit under test is no longer written how it used to be written. You already knew that, you’re the one rewriting it – 0 out of 10.
That brings our final Stability score way down to just 5 out of 10.
Tests as Documentation: 2/10
If you’re someone who wants their tests to give other developers useful insights into the codebase or even confidence to approve your latest PR, then the Nonstop Mocks approach might not be the one for you.
As we saw with stability under refactoring, our tests are protecting the how of execution and not the why. Any developers reading the tests will learn from the assertions that this unit interacts with its mocked counterparts in particular ways at particular moments.
“Invoke this unit with X arguments and we can expect it to call Y method on Z mock with A arguments, causing B side-effects.” The assertions reflect the implementation of the unit perfectly. So perfectly, that we may as well have ignored the test and just read the underlying unit itself instead.
To counter this, you could use a very large helping of poetic licence and title your tests with a description of why we are expecting such behaviours. The disparity between test title and assertions would, however, require a giant leap of faith from your readers.
Of course, other developers could instead acquire this confidence by building an encyclopaedic knowledge of not only this unit test, but every other unit test in the project…
Nope, there’s no way of sugar coating this one. If you want to help other developers understand your code, mock-centric tests aren’t going to help you. Instead you’ll need to invest heavily in wikis, meetings and retaining employees who’ve amassed years and years of domain knowledge – 2 out of 10.
Regression Prevention: 3/10
The scoring here is scoped to what any individual repo can do with unit tests to prevent regression. Yes, in the real world, buggy third parties or infrastructure issues can take us down, but for simplicity and in fairness to our topic, they’re out of scope right now.
(…actually, take that as a given for all our scoring sections, past and present…)
As previously mentioned, pro-mock tests will break to indicate that a refactoring is underway, but will they also break to indicate that a feature – something clients actually care about – has regressed?
The answer to that question is a highly qualified “maybe”.
Given that features are not brought to life by one unit, but by many, any test within the scope of a single unit cannot keep a feature as a whole from regressing. At best, they can claim that their specific granular contribution to the wider codebase, from which a feature might emerge, is still working.
This means that, so long as…
- each of those many individual unit tests pass
- they cover all necessary logic paths
- each unit behaves exactly as all mocks which impersonate them suggest that they do
- and any required states (local and global) are exactly aligned
…then, and only then, can one or more emergent, implicitly defined features be considered protected.
This tall (and highly italicised) order is understandably rarely met. Ever pushed a bug into production while all your mock heavy unit tests were fully passing? Me too. This is the underlying reason why.
3 out of 10 might actually be a little generous.
Overall Score: 3/10
If you’re thinking my grasp of basic maths is terrible right now, you’d be right, but that’s not actually why we’ve landed on a meagre 3. Yes, Ease of Use got a perfect 10 and if I’d taken that into equally weighted consideration we’d be at a more ‘okay’ score of 5.
However, what’s the point of something being easy to do if that something is a bad thing?
Inflexible to the point of being obsolete under basic refactoring, self documenting in the sense that they vaguely summarise actual implementation, happily passing while key features regress in production – that perfect Ease of Use was likely the siren song that lured our ship onto the rocks in the first place.
Terrible at protecting what matters, but great at telling us that our code has changed (hold on, have we just reinvented version control?) – go full-on-mocks and you’ll need to fall back to constant manual verification tests.
If that doesn’t sound like a great solution, you could also make up for their inadequacies by investing in slower running, less cohesive and flakier tests further up the testing pyramid. Eeek.
“Hold on…
…who is this anti-mock, pipe-dream chasing fool? Is he about to tell me that abandoning mocks will lead to better unit tests?”
Spoiler: no I’m not, that’s also a terrible, terrible idea.
Let find out why, when we meet our challenger: