A good definition of unit tests is a test which 1) runs fast <5ms and 2) when it fails you can determine what is wrong without resorting to a debugger. This implies that a unit-tests must execute very limited amount of code so that the failures are well isolated. The ability to be able to execute any piece of application code in isolation is a style of programming which is not immediately obvious.

Unit-testing is not the only way to test. The other kind is scenario testing. In scenario testing it is not a requirement that each piece of functionality can be executed in an isolation, because in scenario testing we are pretending that we are a user. Testability Explorer measures how unit-testable the product is. This implies that one could have a lot of scenario tests and stills score high on unit-tests cost.

Let’s see if we could validate the theory. Here is a list of projects which cater to unit-testing, hence I would expect that the authors understand the nuances of unit-testing and as a result these projects should score very low on testability cost. On the other hand authors of projects which focus on scenario testing would be less likely to use unit-tests in their products. Bellow is a list of products in the order in which they help unit-testing and how they score on testability score.

• JUnit [cost=40]: the unit-testing framework from the fathers of unit testing.
• PicoContainer [cost=15] / GUICE [cost=15] / Spring [cost=60]: dependency injection frameworks focus directly on unit-testing hence one would expect that they themselves are very unit-testable.
• Hudson [cost=30]: A continuous build system which is used to run your unit tests on every check in.
• Waffle [cost=6]: Frameworks whose goal is to make web-apps easy to test. A very impressive cost of 6.
• HtmlUnit [cost=80]/ HttpUnit [cost=200]: Scenario based testing frameworks. Notice the higher costs of HttpUnit.

[kml_flashembed movie="http://www.testabilityexplorer.org/flex/TestabilityCharts.swf" height="450" width="450" fvars="data=http://www.testabilityexplorer.org/flex/projectInfo.csv&filter=junit/;hudson/;picocontainer/picocontainer/;guice/guice/;waffle/waffle/;spring/spring/;dbunit/;httpunit/;htmlunit/.swf" /]

First notice that with the exception of HttpUnit all projects are bellow 100 on testability cost. This is a great number since overall average for all open source projects is somewhere in the vicinity of 3000. This implies that people who write tools which aid in testing themselves take testing seriously. However notice the difference between HttpUnit / HtmlUnit and and everything else. HttpUnit is much higher then all other testing related projects. My theory is that since HttpUnit caters to scenario testing the authors of this tool test it in scenario based way. This would make sense since the tool is meant for scenario based testing.

– Miško Hevery