Presented at 2008 OOPSLA by Miško Hevery a Best Practices Coach @ Google

Abstract

Testability Explorer is an open-source tool that identifies hard-to-test Java code. Testability Explorer provides a repeatable objective metric of “testability.” This metric becomes a key component of engineering a social change within an organization of developers. The Testability Explorer report provides actionable information to developers which can be used as (1) measure of progress towards a goal and (2) a guide to refactoring towards a more testable code-base.
Keywords: unit-testing; testability; refactoring; byte-code analysis; social engineering.

1. Testability Explorer Overview

In order to unit-test a class, it is important that the class can be instantiated in isolation as part of a unit-test. The most common pitfalls of testing are (1) mixing object-graph instantiation with application-logic and (2) relying on global state. The Testability Explorer can point out both of these pitfalls.

1.1 Non-Mockable Cyclomatic Complexity

Cyclomatic complexity is a count of all possible paths through code-base. For example: a main method will have a large cyclomatic complexity since it is a sum of all of the conditionals in the application. To limit the size of the cyclomatic complexity in a test, a common practice is to replace the collaborators of class-under-test with mocks, stubs, or other test doubles.

Let’s define “non-mockable cyclomatic complexity” as what is left when the class-under-test has all of its accessible collaborators replaced with mocks. A code-base where the responsibility of object-creation and application-logic is separated (using Dependency Injection) will have high degree of accessible collaborators; as a result most of its collaborators will easily be replaceable with mocks, leaving only the cyclomatic complexity of the class-under-test behind.

In applications, where the class-under-test is responsible for instantiating its own collaborators, these collaborators will not be accessible to the test and as a result will not be replaceable for mocks. (There is no place to inject test doubles.) In such classes the cyclomatic complexity will be the sum of the class-under-test and its non-mockable collaborators.

The higher the non-mockable cyclomatic complexity the harder it will be to write a unit-test. Each non-mockable conditional translates to a single unit of cost on the Testability Explorer report. The cost of static class initialization and class construction is automatically included for each method, since a class needs to be instantiated before it can be exercised in a test.

1.2 Transitive Global-State

Good unit-tests can be run in parallel and in any order. To achieve this, the tests need to be well isolated. This implies that only the stimulus from the test has an effect on the code execution, in other words, there is no global-state.

Global-state has a transitive property. If a global variable refers to a class than all of the references of that class (and all of its references) are globally accessible as well. Each globally accessible variable, that is not final, results in a cost of ten units on the Testability Explorer.

2. Testability Explorer Report

A chain is only as strong as its weakest link. Therefore the cost of testing a class is equal to the cost of the class’ costliest method. In the same spirit the application’s overall testability is de-fined in terms of a few un-testable classes rather than a large number of testable ones. For this reason when computing the overall score of a project the un-testable classes are weighted heavier than the testable ones.

3. How to Interpret the Report

By default the classes are categorized into three categories: “Excellent” (green) for classes whose cost is below 50; “Good” (yellow) for classes whose cost is below 100; and “Needs work” (red) for all other classes. For convenience the data is presented as both a pie chart and histogram distribution and overall (weighted average) cost shown on a dial.

[-]ClassRepository [ 323 ]
  [-]ClassInfo getClass(String) [ 323 ]
    line 51:
      ClassInfo parseClass(InputStream) [318]
      InputStream inputStreamForClass(String) [2]
  [-]ClassInfo parseClass(InputStream) [318]
    line 77: void accept(ClassVisitor, int) [302]
    line 75: ClassReader(InputStream) [15]

Clicking on the class ClassRepository allows one to drill down into the classes to get more information. For example the above report shows that ClassRepository has a high cost of 318 due to the parseClass(InputStream) method. Looking in closer we see that the cost comes from line 77 and an invocation of the accept() method.

73:ClassInfo parseClass(InputStream is) {
74:  try {
75:    ClassReader reader = new ClassReader(is);
76:    ClassBuilder v = new ClassBuilder (this);
77:    reader.accept(v, 0);
78:    return visitor.getClassInfo();
79:  } catch (IOException e) {
80:    throw new RuntimeException(e);
81:  }
82:}

As you can see from the code the ClassReader can never be replaced for a mock and as a result the cyclomatic complexity of the accept method can not be avoided in a test — resulting in a high testability cost. (Injecting the ClassReader would solve this problem and make the class more test-able.)

4. Social Engineering

In order to produce a lasting change in the behavior of developers it helps to have a measurable number to answer where the project is and where it should be. Such information can provide in-sight into whether or not the project is getting closer or farther from its testability goal.

People respond to what is measured. Integrating the Testability Explorer with the project’s continuous build and publishing the report together with build artifacts communicate the importance of testability to the team. Publishing a graph of overall score over time allows the team to see changes on per check-in basis.

If Testability Explorer is used to identify the areas of code that need to be refactored, than compute the rate of improvement and project expected date of refactoring completion and create a sense of competition among the team members.

It is even possible to set up a unit test for Testability Explorer that will only allow the class whose testability cost is better than some predetermined cost.

So you’ve figured out a better way of doing things, but how do you get everyone to change the way they work and start writing code in this better way? This is something I face daily in my line of work as a best practices coach. Here are my battle tested tricks which work even after I leave the project.

What usually happens

Most change initiatives fail because people don’t appreciate the reasons for making a change, or because they get demotivated by the progress of change.

Think about what usually happens.  Suppose that you have recently discovered the benefits of dependency injection. The dependency injection principle requires that object creation code be kept separate from application logic.  Dependencies should be found through the constructor, not by creating them or looking them up in global state. This makes code more maintainable and easier to test, which results in a higher quality codebase. Armed with all of these benefits you decide to implement this on your project. As the lead of the project you give a tech talk to the team on the benefits of dependency injection. Developers come to your tech talk, you all take a vote and everyone agrees that this is what needs to be done.  Everybody goes back to their workstations full of enthusiasm.  People attempt dependency injection in the first few changes they make and you are excited, but a month later everyone is back to their old routine and nothing has changed. How can we break this cycle and achieve lasting change?

Step 1: Get Buy-In

The first step is to make sure that everyone is willing to try something new. Although buy-in meetings don’t achieve anything tangible, they help people understand the reasons why a particular goal is important, and it gives the entire team the opportunity to collectively decide to go after it.

But buy-in isn’t enough.  We must have some way of knowing that we are implementing the change. To do that, we can use a metric as a proxy for the goal. This makes progress measurable, keeps the goal front-and-center for developers, and lets us know when we’re done and can declare victory.  Without a metric, we risk losing direction (how do we know our steps are making things better?), losing motivation (individual steps are too small to be seen against the big goal) and losing any sense of accomplishment (because there is always something that can be improved, we don’t know when we are done with this round of improvements).

Step 2: Define Your Goal in Terms of an Objective Metric

The next step is to have a measurable number that gives us an indication of where we are and where we want to be. Although it doesn’t matter whether this number is an exact measure or just a proxy, it must be repeatable and calculable in an objective way at each stage of adoption.

In Java, byte-code analysis tools are a great way to get to a metric. These can measure very specific things, such as the total number of calls to deprecated APIs or classes/methods which we want to remove.  They can also be used to apply heuristic rules to code, such as determining whether it breaks the Law of Demeter, contains excessively deep inheritance, or has too many lines of code in methods or classes.  There are a lot of open source tools available that can measure attributes of code, including ASM to write your own metrics, JDepend / Japan for dependency enforcement, Panopticode for code-quality metrics, and Testability Explorer for identifying code that is hard to test.

There are some interesting side-effects to using metrics.  One is that people may initially debate the value of making a change, but the moment there’s a number – even if it is a home-grown formula – the arguments stop.  Another is that management is more likely to support the change: anything that can be measured conveys a sense of visibility and tangible benefit that an unmeasurable change cannot.  Again, these are side-effects, not risks:  having already established team buy-in in the previous step, we’re not “managing to a number” and missing the point of making the change. We’re simply bringing attention to the progress we’re making toward that change.

In the example above, we want to get developers to do dependency injection. I developed a metric and corresponding open-source tool called Testability Explorer which measures the smallest number of conditions which cannot be mocked out in a test. The idea is that code where branches can be isolated will be easier to test than code where the logic cannot be isolated.  In the latter case, one test will exercise a lot of code and will most likely require complex setup, assertions and tear down. The best metrics are those where gaming the metric produces the desirable outcome in the code. In Testability Explorer, using dependency injection is a good way to improve the score, which is exactly what we are trying to achieve.

Checkpoint: Socially Acceptable, but Not Yet Sustainable

Buy-in reinforced with metrics make our proposed change scientific as well as socially acceptable.  But this isn’t enough: it must be constantly reinforced.  Part II of this series will present ways to make progress toward change highly visible yet subtly encouraged with each and every commit.

Unit-testing is not the only way to test. The other kind is scenario testing. In scenario testing it is not a requirement that each piece of functionality can be executed in an isolation, because in scenario testing we are pretending that we are a user. Testability Explorer measures how unit-testable the product is. This implies that one could have a lot of scenario tests and stills score high on unit-tests cost.

Let’s see if we could validate the theory. Here is a list of projects which cater to unit-testing, hence I would expect that the authors understand the nuances of unit-testing and as a result these projects should score very low on testability cost. On the other hand authors of projects which focus on scenario testing would be less likely to use unit-tests in their products. Bellow is a list of products in the order in which they help unit-testing and how they score on testability score.

• JUnit [cost=40]: the unit-testing framework from the fathers of unit testing.
• PicoContainer [cost=15] / GUICE [cost=15] / Spring [cost=60]: dependency injection frameworks focus directly on unit-testing hence one would expect that they themselves are very unit-testable.
• Hudson [cost=30]: A continuous build system which is used to run your unit tests on every check in.
• Waffle [cost=6]: Frameworks whose goal is to make web-apps easy to test. A very impressive cost of 6.
• HtmlUnit [cost=80]/ HttpUnit [cost=200]: Scenario based testing frameworks. Notice the higher costs of HttpUnit.

[kml_flashembed movie="http://www.testabilityexplorer.org/flex/TestabilityCharts.swf" height="450" width="450" fvars="data=http://www.testabilityexplorer.org/flex/projectInfo.csv&filter=junit/;hudson/;picocontainer/picocontainer/;guice/guice/;waffle/waffle/;spring/spring/;dbunit/;httpunit/;htmlunit/.swf" /]

First notice that with the exception of HttpUnit all projects are bellow 100 on testability cost. This is a great number since overall average for all open source projects is somewhere in the vicinity of 3000. This implies that people who write tools which aid in testing themselves take testing seriously. However notice the difference between HttpUnit / HtmlUnit and and everything else. HttpUnit is much higher then all other testing related projects. My theory is that since HttpUnit caters to scenario testing the authors of this tool test it in scenario based way. This would make sense since the tool is meant for scenario based testing.

– Miško Hevery

My hypothesis is: AspectJ was written with tests and hence is written in a more testable manner. Over time the advantage of test was that the code base was more malleable and hence AspectJ ended up with more features and so has won the AOP war.

Why am I picking on AspectJ vs AspectWerkz? Well because they are two projects with identical goals but very different outcomes. As such I think we can compare them and learn from their differences.

AspectJ is more testable then AspectWerkz. How hard would it be to write a unit-test for any one of the classes in AspectJ or AspectWerkz? Look at the graph below and you will see that AspecttWerkz has all of the dots above the score of 2000 in testability-cost. While in AspectJ case all dots are below 2000 (and to the right) in testability-score. The simples way to think about the scores is that to write a unit-tests for AspectWerkz I would encounter an average of 5000 IFs per unit-test. While I would encounter about 1800 IFs per unit test in AspectJ. The AspectJ test would be a lot more focused (closer to unit-tests) then AspectWerkz test would be. It also means that a failure in AspectJ will be easier to diagnose since the tests are a lot smaller.

Is it not true that any large application becomes hard to test? Well look at the final size of the JAR or the number of classes in the JAR. As you can see AspectJ has a lot more of both. Hence I would assume that it has a lot more features as well. If all large applications would be hard to test we would expect a linear relationship between size and testability-cost. Something which we do not see in the graph bellow.

[kml_flashembed movie="http://www.testabilityexplorer.org/flex/TestabilityCharts.swf" height="450" width="450" fvars="data=http://www.testabilityexplorer.org/flex/projectInfo.csv&filter=aspectj/;aspectwerkz/.swf" /]

How do you know your hypothesis is right? I don’t know since I have never used either product hence I don’t know how hard would it be to use it or how many bugs I would encounter. But simple search on Google reveals that AspectJ is a lot more popular.

– Miško Hevery

So I had an idea to write a bytecode analysis tool which would identify the red-flags for me. The motivation was that it is hard to teach people how to write quality tests if the code base is not testable. One only learns how to write quality tests after one has written lots of tests. And it is hard to write a lot of tests if your code is hard to test. So I had an idea:

“Perhaps I can’t make developers to write tests, but perhaps I could prevent developer from getting into untestable mess by avoiding the red flags. This way even if they don’t write tests today, they could write tests tomorrow.”

The idea was to build the Testability-Explorer and then to integrate it into project’s continuous build so that whenever a developer tried to check in some hard to tests code the Testability-Explorer would flag the pieces of the code as hard to test and then fail the build. However, I wanted to make sure that the Testability-Explorer would not only fail the build but also give useful suggestions as to which lines of code are a problem and how to refactor them. Without this feedback the developers would only get frustrated, with the feedback the developers would learn why a particular way of coding is a problem, from a testing point of view. This way the code-base would always remain in easy to test state. And even if developers were not writing tests today, nothing would prevent them from writing the tests tomorrow.

Once we have built the Testability-Explorer we realized that we had a chance to change not just our developers but a whole open source community and the idea of http://TestabilityExplorer.org the web site was born. Here we publish the testability reports for most major Java open-source projects. We hope that this tool will be a flashlight into which open-source projects are doing a great job and should get recognized for their hard work and also we hope that we can offer suggestions on how to refactor those projects which are less then ideal from unit-testing point of view.

– by Miško Hevery