Testing Without Mocks: A Pattern Language (2023)

`makeThingDoerForTest` is a standalone function (or static method) that never gets called in production code. I don't see why it needs to be in an entirely separate module. If you want, you could put it in a separate thing_test module, but then you'd have to expose a more powerful constructor for thing_test to access. I'd rather hide that inside the thing module, and I see no benefit to splitting this functionality across two modules just for the sake of not mixing my peas with my potatoes.

That's the trick - it is not test only code. You use the same objects, the same code.

The code has to be written so it takes in the altered unit via parameters or other DI or so it knows that the current system is set to short circuit some calls.

Ie, you could have MyClient(testHTTPClient).GetResource(foo) used in tests and MyClient(realHTTPClient).GetResource(foo). The testHTTPClient would get to the actual connection part and return a configured response or error.

Your entire logic is the same, your code could "receive" a forced 404 or timeout or 200 OK. It is up to the testHTTPClient that is only changing how the http connection is handled.

I call these unit-integration tests. You are checking all units work together while not actually working with the outside world.

It goes a bit into the same direction rust went by shipping the unit tests usually inside the module. The thought is intriguing to keep the „mock“ setup up to date even when doing refactors. But I wonder if this solves the issue. I tend to not use mocks. And I try to stay away from spies. These two tie the tests to close to the implementation like the article also described. The reason why I don‘t ship test setup in production code is the simple fact that the test / spec should not be based on production code. Hard to put in words for me but the test should be impartial. It sets up its test world and run the subject in it. It should not use the subject to describe the test environment. I think this will lead to false positives.

I have an example where a coworker used constants from production to assert return values. I stated that the test can‘t use them because the test is the specification and needs to be in control what is correct and what is not. Even if that means we write duplicate string values. In this case somebody changed a const value to a wrong value. But for the test this was all fine.

What is the harm in a test only path in production? If you are an embedded systen with limited memory there is obvious harm but most code isn't that. There are other possible harms but they are less obvious and may not imply. There are also many different ways to implement this.

i did something like this where in production I have if function pointer isn't null return the result of it instead of running the production code (and an unlikely compiler directive on the if). The function itself is implemented in a test only shared library so the cost to production is low. If you can break in you can set that function - but if you can inject arbitary code we are already sunk.

> no way, test-only code paths should never reach prod, and code that ships should never even know that testing is a thing that happens.

What's your reasoning for this? As a counter-example, the hardware you are using right now almost certainly has test points throughout the circuitry. Beyond all the uses the test points get during development and manufacturing, they are essential for repair and re-calibration after being sold to the customer and put into production.

I guess overcoming that dogma is exactly the tradeoff.

> test-only code paths should never reach prod

Why? Humanities most reliable engineered objects contain their own test and monitoring systems.

I agree. Arguments against the classical test pyramid are common, but IME they often come from a place of laziness. It essentially boils down to "unit tests are a chore to maintain, hence ...<hand waving>... integration tests are more valuable". Laziness _is_ one of the three virtues of a good programmer, but these arguments completely miss the mark. I would actually go a step further and argue that their popularity is doing tangible harm to software quality.

For a program to be robust it needs to be tested at all layers, and the testing pyramid is still a good rule of thumb to follow. Designing the program with testing in mind is crucial, and as you say, interfaces and their equivalent in any language are the right approach to achieve this. There was a period of time when monkeypatching was popular in dynamically typed languages, and perhaps still is (I've been out of the loop for a few years), but we always knew that it was a dirty trick, and that the software should be architected better to make it testable.

If anyone is curious what this might look like, I wrote a small project in Go[1] last year that is built from the ground up with testing in mind. There's a global `App` struct that is initialized in `main() `, and serves as the main container for all I/O the program does. This means that internally no component ever makes direct OS calls, but does it via interfaces, which can be easily mocked in tests. This makes unit and integration testing a breeze, and allows other niceties like passing down a global logger. I've adopted this pattern in all Go apps I write from scratch, and advocated for it on existing projects I've worked on, and it's worked great so far.

[1]: https://github.com/hackfixme/disco

Interfaces are only useful if there is reason they won't change otherwise they get in the way. Reason is often policical - this is where another group talks to my code and so the less we have to know about the other the better (not that the code is hidden just that you ignore it) reason could also be it is used everywhere so any change would be months of effort to make - longer if anyone else needs to impment anything.

if change is easy then better to make the change - you never really know all future requiresments so getting it right is impossible.

unless getting it right is possible. if you are implementing a new language you should know enough to get the interface to your strings right on the first try. That is not the normal case. (Strings will soon become impossible to change)

I'll give it a try.

The beauty of the nullable approach is that when you're testing the system under test (SUT), you're testing it together with its dependencies. Not just that - you're running the real production code of the dependencies (including transitive dependencies!), just with a off switch toggled.

What the off switch does is really just prevent side effects, like making actual HTTP requests to the outside world or reading bytes from the file system. Using embedded stub makes this a breeze (another revelation for me).

For example, while building the "/v1/add-account" endpoint, you write a test for the request handler function. If you write this test using Shore's approach, you'll exercise the whole tree of dependencies, down to the leaves.

This is what is meant by sociable tests - the SUT isn't isolated from its dependencies. It's the real code all the way down, except that you're calling createNull() instead of create() to instantiate the classes. There are many benefits here, but to me the most important ones are: (1) you don't need to write mocks as separate classes, (2) you can catch errors that arise in the interplay of the various classes or modules and (3) as a result you get a lot more confidence that everything is still working after refactoring.

A sociable microtest is a little like an integration test, but you don't exercise any out-of-process dependencies (perhaps with the exception of the RDBMS) so the tests run in milliseconds rather than seconds.

You commented elsewhere that you're worried about the separation of test code and prod code. Yes, this is a bit of a holy cow that I also had trouble adapting to. It turns out, having test-focused code mixed in with your prod classes is totally fine. I'd perhaps prefer a cleaner separation but honestly it's not a big issue in practice.

Part of testing is risk management. With mocks you run the risk of the behaviour of your mocks being out of sync with the actual systems you’re mocking. By moving the test-time behaviour closer to the runtime behaviour, you reduce that risk somewhat. If you have other ways of managing that risk or you just never see that happening, you’re good.

You can use mocks, however, I found in practice that hand written fake implementations (aka Nullables) are much easier to maintain.

Yes.

When you change behavior you should expect to re-record.

The "unit" in unit test refers to the test, not the code under test. High level tests that are isolated from other tests using mechanisms like fixtures, setup/teardown etc are still unit tests.

I don't really see how production-code mocks differ from library-based mocks. If you add but a single constructor parameter to your class, you already need to change your production-mock/stub too.

How are you imagining that variable gets flipped? If an attacker has the ability to run arbitrary code, you’re already screwed.

To put it another way, somewhere in your code you do authorization: `if (isAuthorized) enterTheCastle()`. The gate to the castle, guarded by a single variable. In both cases, you have to be careful to program it correctly.

In the case of Nullables, it’s easy to program it correctly. Multiple things would have to go wrong for the nulled version to be accidentally used in production. Also, unless you go out of your way to make a security vulnerability, Nullables’ default behavior is to do nothing. They default closed rather than open.

I guess we have used different mocking libraries. The ones I'm most familiar with start with very simple use cases ("here is an object that looks like this Class, and when you call function X on it, return Y") and don't appear to have any noticeable amount of performance degradation.

One could just as easily have each class/file export a mocked object rather than a real one that happens to do nothing.

That angle of it though, I can see being beneficial. Where each class has the responsibility of defining how it should be used in a test case, vs each individual test case having to define the mocking behavior for any class it touches.