Why you need to make your tests fail

Test Driven Development (TDD) have many benefits. For start it’s a design methodology that help avoiding “Analysis paralysis” and make sure that you only have the needed code to solve a problem.
Yesterday I found another benefit of writing the tests before the code – you get to see them fail!
A while back I wrote about another shortcoming of MSTest – how it does not have any built in facility to test against expected exception message.

Unknown to me at the time there was a bug hidden in the code I’ve published – see if you can spot it:

public static class MyAssert
{
    public static void BadThrows;<T>(Action action) where T : Exception
    {
        try
        {
            action();
            Assert.Fail("Exception of type {0} should be thrown.", typeof(T));
        }
        catch (T exception)
        {
     // All is well!
        }
    }
}

BTW this example was simplified for your viewing pleasure – I want to make a point here, not stick to the facts. have you found the bug yet?

Consider the following test:

[TestMethod]
public void ThisTestWouldNeverFail()
{
    MyAssert.BadThrows<exception>(() => Dummy.SomeMethod());
}

Have you guessed it? (hint: its in the test name)

That’s right! this test would never have failed. If we take a closer look at BadThrows we’ll see the problem.

MSTest just like any other .NET unit testing framework throws an exception when assertions fails. In this case because we’re expecting an exception of type Exception any other exception thrown including the AssertFailedException would be caught – causing the test to pass.

That’s right – because the user expects an exception of type”Exception” to be thrown plus a minor bug the test is completely useless – if you want to see the working code just head to this post.

I did not find this bug mainly because I don’t like to throw and assert for Exception seems like a bad practice and in retrospective I should have avoided my co-workers from doing so by throwing exception of my own in case they try to do it.

The guy that did find this bug found it because he wrote a test and wanted to see it fail. That’s right it wasn’t enough for him to know that his code made the test pass – he had to make sure that it failed first – some people…
A quick search in our code found multiple tests that did exactly that which means two things:

1. These test were not written using TDD
2. The authors did not even bother to see the tests fail

After fixing the bug we found one test that started failing – which means that not only was the test wrong – so was the code it was testing.

I’ve talked to the team, explained why throwing “Exception” is not a best practice and why tests need to be “tested” by seeing them fail and we’ve fixed the tests and the code.

I think I learnt quite a lot from this experience, I know now who does TDD and who doesn’t and I found a new problem that occurs when developers write their tests after the code.

Happy coding…

Reference: Why you need to make your tests fail from our NCG partner Dror Helper at the Helper Code blog.