I attended the Canberra .NET User Group meeting this evening (Febuary 16th). This month it was Adam Cogan who was presenting “15 Rules to Successful Projects” (related link).
Adam and I (and probably a few other people) have been having an ongoing discussion about one of the things he does in the various tools and systems that he works on. That is the embedding of unit tests in the application code so that a user can run them – often just by clicking a button!
My first reaction to this was mixed. I kinda like the idea but I saw a whole heap of unresolved issues – especially around licensing. Although upon reading the license agreenment for NUnit, I need not have worried.
Tonight I think Adam finally started convincing me that this was a good idea but I still think his implementation of a good idea sucks. Firstly – NUnit is a developer focused tool, so when you run it it basically traverses the code base looking for unit tests and then lays them out in a namespace/class hierarchy as it executes each and every test.
Thats great – but what have you told the user other than something is “all green” not “not all green”?
Adam was also lamenting the fact that Visual Studio Team System’s unit testing framework isn’t really something that you can bundle up and ship with your application like he does with NUnit. The reality is that most people aren’t going to ship unit tests with their product and the level of integration between the testing frameworks and MSBuild and Visual Studio 2005 would make that dependency pretty large.
It occurs to me that what Adam really needs is a system test framework, not a unit test framework. If you look at Adam’s Rules for Better Unit Tests you can see that a lot of them are for checking dependencies like databases, DLLs and web-services. Personally I like my unit tests to be as free standing as possible and take as few external dependencies as possible.
A system test framework on the other hand would be designed to run tests after an application had been initially installed where the test code can rightly expect a fully functioning environment to be present.
The epiphany about this came to me when I saw another recommendation of Adam’s which was to limit the intelligence that you put into setup executables. I like this idea (although I don’t always follow it) and its similar to what BizTalk does with its relatively simple (heh) routine with the post installation ConfigureFramework.exe step.
System tests and post installation steps are actually related – when a system test fails it is usually because some installation step has been missed. So by doing a bunch of system tests when the application starts and then on demand (and whenever the application is crashing) you can associate the specific test failures with remediation steps (general term).
The diagram above summarises how a system test would be related to a remediation step. The benefit of doing it this way rather than just using NUnit is that you not only get the system to tell you how its feeling but you can actually get the system to suggest how to fix itself if possible – from a user experience point of view this is slick.
I’d love to take a stab at implementing something like this when I get some bandwidth:
TaskQueue.Enqueue(Idea);