ARE INTEGRATION TESTS REALLY THE BEST WAY?

Do you write integration tests? Do you enjoy doing it? No? Thought so, me neither.

Integration tests are inherently more complex than unit tests. They have to be, they test so much more than just that single piece of functionality that a unit test does. Setup is more demanding because there are more execution paths to consider reaching the goal of the test.

And that is also one of the major problems with them. To have confidence that they don’t let bugs through you need coverage. And that takes effort. As soon as you start combining things complexity rise and that means more tests. The thing is, combining objects gives an exponential rise in the number of possible ways the code can execute.

If you have one if-state in object-1 and one in object-2 the number of possible paths are 2 * 2 = 4. And if you combine three objects, you get 2 * 2 * 2 = 16 paths. So, to get coverage, even if you skip the paths that aren’t part of the integration paths, you end up with a lot of tests to get coverage, even if it is just to see that the objects work well together.

But what if there was a way to make this process as simple as writing unit tests?

What if you actually could write unit tests instead to reach the same goal?

Who’s should read this?

Before you move forward, I just want to check if this is written for you. This article is aimed at developers that has experience of writing unit tests, that know what a mock object and a stub is. If you are a beginner or have never heard of these words, I suggest that you start with learning that first.

Background

I was searching the Net for information on Integration Tests when I came over this video by J B Rainsberger, proclaiming that “Integrated Tests are a Scam”. With a heading like that, how could I NOT see what he had to say?

It turns out he has spent some time thinking about how to avoid the exponential growth of integration tests. And has come up with a solution too. It IS possible to only write unit tests and achieve the same thing as integration tests are supposed to do. And by doing so, you enter a linear path in the number of tests you have to write instead of an exponential. The result? Simpler and fewer tests.

Where do I start?

First some definitions.

Unit tests make sure that the functionality within the object does what it is supposed to do. It is not interested in what goes on outside of the object. If you add two numbers, the test checks so the sum is correct.

Integration testing on the other hand is all about collaboration. The tests are a way to make sure that objects work correctly together. We can boil what that means down to four questions:

1. Does the object under test call the collaborator object that it should?
2. Does the object under test handle all results the collaborator can send?
3. Does the collaborator handle all the variations of data that the object under test can send?
4. Does the collaborator actually send a reply for all the variations of data that the object under test expects to receive?

Let’s elaborate a bit.

The first two questions are basically what you have been doing all along if you have been writing unit tests. It’s like the core of unit tests alongside the actual testing of logic.

The first question makes sure that whenever you need to interact with a collaborator object, the correct calls are made, and with the correct parameters.

For instance, if the collaborator has a method to get the user data, that function is called and nothing else. And you also make sure that the correct data is sent into the function. So, if you have a user id and want to get the full user data from the data layer, you make sure that you call the GetUser() function with the user id.

This is realized by using mock objects. If you use FakeItEasy, you might recognize this:

A.CallTo(() => sut.GetUser(userId)).MustHaveHappened();

where userId is the expected user id.

The second question makes sure that you can handle any data that the collaborator object can send. A collaborator can return many things. The expected data, an empty list, an error code or an exception. All these has to be handled in the code or they will be considered undefined behaviour, and they will probably result in weird bugs. It’s important to study the collaborator to find all variations of what it can return and cover these with tests.

You do this by creating a Stub. Once again, if you use FakeItEasy, this is how you do it:

A.CallTo(() => sut.GetUser(A<uint>.Ignore)).Returns(new User{…});

or

A.CallTo(() => sut.GetUser(A<uint>.Ignore)).Throws(new ArgumentException);

So far, so good. This is probably what you have been doing all along with your unit tests. And that is great! But it is only half of the contract. Sure, your function can send and receive data, and it does what it is supposed to with it. But you do not know if the other half of the equation, the collaborators, do the same? So now we will take a look at the part that is missing, that makes it possible for your unit tests to replace the integration tests.

The third question makes sure that for everything that you can send from the object under test, there is a corresponding action in the collaborator object. You have to study the object under test to find all variations of what it can send and cover these with tests.

Technically, this is exactly the same thing that you code in the second question but for the collaboration object instead. If there already are tests written for the collaborator function, your task is to make sure that these include all the tests that is needed to answer this question.

And if no tests exist yet, write down notes in the empty collaborator test-file, so that when you do write the tests you don’t forget anything.

So, for each A.CallTo().MustHaveHappened() in the object under test, there is a test for the collaborator that verifies that that exact data that is sent.

The fourth question. I guess you already have figured out what this is. It’s basically the same thing that you do in the first question, but for the collaborator. You make sure that the collaborator has code that actually do send the all data that the object under test expects. If there is code in the object under test that handles both admin-users and regular in the object under tests, you have to make sure that there is code in the collaborator that can return both kinds of users. Or else you will have written code that only acts on test data, fantasy data; and not on data from the real world. The expectation is probably correct, but it never got implemented.

So, for each A.CallTo().Returns() in the object under test, there is a test for the collaborator that verifies that that exact data that is sent.

Sure, the collaborator might be able to send more data than the object under test receives because there is more code that use this object than the object under test, and that’s ok, you write tests for that too. But that’s another story than integration testing.

Conclusion

The important thing is to really make sure that the tests have a 1:1 relationship between the two objects. If not, you will have a function that does something that nobody expects. Kind of like the Spanish Inquisition, as Monty Python portrayed it. It jumps in under dramatic circumstances from nowhere and starts wreaking havoc on the customers runtime experience.

Integration tests will probably always have a role in testing, but parts of it can be switched out for a different way of doing it. It’s just a matter of trying something new. If we never did that, we would never have left the caves. So why don’t try this way and see if it works? Maybe you will have discovered something that helps you be a better developer. Or not. That’s up to you.

— Cheers!