The Test Scope Creep

In “Should I use multiple asserts” we described the problem when using them in a single test. In the example I gave, while checking the result of a single operation, it still made sense to get information on both ends of the transaction. Debugging can unravel the issue, yet two separate test results shorten the way to the fix.

We are using asserts to lead us to the right problem to solve. But hey, we can do much worse.

We’ll write a different test for checking the money transfer from the source account, but also add a check for the integrity of the account after the transfer.

@Test
public void getBalance_AfterTransfer_SrcAccountDecremented()
{
    bankAccount accountSrc = new bankAccount(5000);
    bankAccount accountTarget = new bankAccount(0);
		
    accountSrc.Transfer(500, accountTarget);

    assertTrue(accountSrc.isValid())
    assertEquals(accountSrc.getBalance(), 4500);;
}

Looks harmless. After a complex operation, the account’s state needs to be validated. Even more so, it’s part of the “unit of work” we’re testing, and the assertions are on the same object. So, there’s no apparent rule against not using two asserts.

What happens, months from now, when the test fails because the first assert fails?

We’ll see in the build report:  “getBalance_AfterTransfer_SrcAccountDecremented failed”. The reported reason would be that the source account wasn’t valid.

We don’t know the balance, because the test failed before that. It may be the correct balance.

It may even be that the account is not valid because of a side affect, something completely irrelevant to our balance check.

Now we are confused. Even misled.

We think the problem is with the balance, although it might not be. It will take us more time to solve the problem, than if there were two separate tests, with specific names.

This is test scope creep. We added a bit more scope to our test, and much like in feature scope creep, we got bitten.

The problem can even grow further. The more conditions we assert, we tend to generalize the test name. After all, if it checks all things in my workflow, why not call it “testWorkflow”?

If we don’t need specific names, and match the assertion, we’re causing our future selves more work in the future.