WPF View Smoke Testing

I enjoy TDD. I enjoy the way it makes me code and I enjoy the rhythm. That is, I really enjoy TDD’ing the logic in my applications: The business logic, the controller logic, the domain logic; wherever there is logic that I can isolate and unit test cleanly TDD is just all fun and productivity. I enjoy TDD’ind integration points a little less: The database access, the web service calls, and so on are slow to test, so the TDD rhythm is broken. I still prefer TDD’ing these parts too though. I do not enjoy TDD’ing UIs: In my experience the tests are slow, either the rate of false negatives is high or the rate of false positives is high, and UI test are a pain to write (don’t even get me started on the horrors of recording UI tests). Therefore I usually settle for subcut tests and manual UI testing when it comes to views. That is: I want to TDD my presentation logic, but am usually ok with checking that the pixels are in the right spot manually.

Working with WPF this is – at first sight – supported by MVVM: The view models sit right under the views, and are able to cut the .xaml.cs code behinds down to (almost) nothing, so testing the view models is subcut testing right? Right?? Not quite: There is a little bit of stuff going on in the bindings. I want to test that stuff.

WPF has been around for a while so I expected this to be a solved problem. Maybe it is, but my googling didn’t yield an answer. It did however yield a number of candidated solutions. Below is a quick summary of my evaluation I of these candidates. (TLDR: None of the libraries worked, but Window.Show and FrameworkElement.FindName turned out to be my friend, and maybe, just maybe. Approval Tests could finally win me over to UI testing).

Below there is a section for each of the candidates I quickly evaluated with some rough notes on how it went. The evaluations are not terrible thorough nor objective, so YMMV

IcuTest is a library that works by asking you the first time a test is run to accept or reject the result: It shows you a snapshot of the window to assert on, and you accept or reject. If you accept the image is saved and used as the acceptance criteria in subsequent runs.

I have a few issues with this: The asserts come down to bitmap comparisons, which means (1) that they are imprecise – they assert on the whole window/control instead of just the one thing the current test is about and (2) they depend on the machine the test is running on. These are execatly the issues with UI testing that have bitten me in the past. So IcuTest is not what I’m after.

White seems to be the most grown up UI testing framework for Windows applications around. Promising. But: I installed the NuGet in my projects and was immediately sent into the Log4Net strong naming cirkus. Annoying but not Whites fault.I went on to clone the source off Github. It compiled out of the box, but most of the tests failed. After fiddling around for a few hours I got like 60% of the tests running. Not a good sign to be honest. The fiddling included changing things like which properties were used to input text into a textbox in ways that were pure guessing on my part. All in all this seemed like a route that would lead to unstable tests. So White is not what I’m after.

The AvalonTestRunner is an old thing from back when WPF was still Avalon. It doesn’t claim to do a lot either. But none of these are bad things in themselves. If it works it works. One of the things AvalonTestRunner claims to do is sanity check all the bindings in a view: I.e. throw an exception if there is a binding path that is not found in the data context. That is part of what I’m after and actually a quite nice first line of defense to have. Only problem: I couldn’t get it to work, not out of the box and not after debugging through the source for a while. Maybe it’s just me, but again I concluded: AvalonTestRunner is not what I’m after.

Guia seems to try to solve the same thing as white but only for WPF. It also seems defunct: The current version is 0.1.1 and is 2 years ole (at the time of writing) . But again: If it works it works. Sadly it didn’t work out of the box, and because of the lack of activity I didn’t investigate further. Guia is not what I’m after.

With all the libraries I tried out not really working I’m left with hand coding the tests myself. As it turns out this is in fact not nearly as bad as it sounds – not for the subcut type of tests I’m after anyway.

MS UI Automation is a library intended to support building screen readers, remote controls and other types of applications that need to automate the UI of some other – known or unknown – application. It seems to provide everything you’d need to write UI test for WPF. I haven’t tried it out though because it seems like overkill for what I’m looking for.

As it turns out I probably shouldn’t have spent so much time wading through testing libraries because the simple little kinds of tests I set out to write: Tests that drive and check the bindings in my XAML are actually (for the most part at least) easily written by just opening the window under test directly, finding the controls of interest by calling .FindName on the window under test and then manipulating or asserting against properties on those controls. Like this:

  public void Tag_is_bound_to_model_on_construction()
  {
    var model = new MainWindowViewModel(tag: "testing");
    var sut = new MainWindow(model);
    sut.Show();
    var tag = sut.FindName("Tag") as TextBox;
    Assert.Equal(model.Tag, tag.Text);
  }

This is so darn easy, that once I realized, that this is actually what I am after I stopped looking at libraries although there are still a couple on my list. So there you have it: It’s just not complicated enough to need a library.

In spite of the conclusion above the following may be worth checking out. I’m pretty sure at least ApprovalTests work although I haven’t tried it. WhiPFlash seems active, but beyond that I know nothing.  

Reference: WPF View Smoke Testing from our NCG partner Christian Horsdal at the Christian Horsda’s blog blog.