A great question from the reader…

The only thing that does not fully convince me in your articles is usage of Guice. I’m currently unable to see clearly its advantages over plain factories, crafted by hand. Do you recommend using of Guice in every single case? I strongly suspect, there are cases, where hand-crafted factories make a better fit than Guice. Could you comment on that (possibly at your website)?

I think this is multi-part question:

1. Should I be using dependency-injection?
2. Should I be using manual dependency-injection or automatic dependency-injection framework?
3. Which automatic dependency-injection framework should I use?

Should I be using dependency-injection?

The answer to this question should be a resounding yes! We covered this many times how to think about the new-operator, singletons are liars, and of course the talk on dependency-injection.

Dependency injection is simply a good idea and it helps with: testability; maintenance; and bringing new people up to speed on new code-base. Dependency-injection helps you with writing good software whether it is a small project of one or large project with a team of collaborators.

Should I be using manual dependency-injection or automatic dependency-injection framework?

Whether or not to use a framework for dependency injection depends a lot on your preferences and the size of your project. You don’t get any additional magical powers by using a framework. I personally like to use frameworks on medium to large projects but stick to manual DI with small projects. Here are some arguments both ways to help you make a decision.

In favor of manual DI:

• Simple: Nothing to learn, no dependencies.
• No  reflection magic: In IDE it is easy to find out who calls the constructors.
• Even developers who do not understand DI can follow and contribute to projects.

In favor of automatic DI framework:

• Consistency: On a large team a lot can be said in doing things in consistent manner. Frameworks help a lot here.
• Declarative: The wiring, scopes and rules of instantiation are declarative. This makes it easier to understand how the application is wired together and easier to change.
• Less typing: No need to create the factory classes by hand.
• Helps with end-to-end tests: For end-to-end tests we often need to replace key components of the application with fake implementations, an automated framework can be of great help.

Which automatic dependency-injection framework should I use?

There are three main DI frameworks which I am aware off: GUICE, Pico Container and Spring.

I work for Google, I have used GUICE extensively therefor my default recommendation will be GUICE. However I am going to attempt to be objective about the differences. Keep in mind that I have not actually used the other ones on real projects.

Spring was first. As a result it goes far beyond DI and has everything and kitchen sink integrated into it which is very impressive. The DI part of Spring has some differences worth pointing out. Unlike GUICE or Pico, Spring uses XML files for configuration. Both are declarative but GUICE is compiled and as a result GUICE can take advantage of compiler type safety and generics, which I think is a great plus for GUICE.

Historically, Spring started with setter injection. Pico introduced constructor injection. Today, all frameworks can do both setter and constructor injection, but the developers using these frameworks still have their preferences. GUICE and Pico strongly prefer constructor  injection while Spring is in the setter injection camp. I prefer constructor injection but the reasons are better left for another post.

Personally, I think all of the three have been around for a while and have proven themselves extensively, so no matter which one you chose you will benefit greatly from your decision. All three frameworks have been heavily influenced by each other and on a macro level are very similar.

(I personally would not chose Spring because XML and setter injection puts me off. However I am looking forward to using Pico on my next open-source project so that i can become more objective about the differences.)

Your milage may very.

Shahar asks an excellent question about how to deal with frameworks which we use in our projects, but which were not written with testability in mind.

Hi Misko, First I would like to thank you for the “Guide to Writing Testable Code”, which really helped me to think about better ways to organize my code and architecture. Trying to apply the guide to the code I’m working on, I came up with some difficulties. Our code is based on external frameworks and libraries. Being dependent on external frameworks makes it harder to write tests, since test setup is much more complex. It’s not just a single class we’re using, but rather a whole bunch of classes, base classes, definitions and configuration files. Can you provide some tips about using external libraries or frameworks, in a manner that will allow easy testing of the code?

– Thanks, Shahar

There are two different kind of situations you can get yourself into:

1. Either your code calls a third-party library (such as you calling into LDAP authentication, or JDBC driver)
2. Or a third party library calls you and forces you to implement an interface or extend a base class (such as when using servlets).

Unless these APIs are written with testability in mind, they will hamper your ability to write tests.

Calling Third-Party Libraries

I always try to separate myself from third party library with a Facade and an Adapter. Facade is an interface which has a simplified view of the third-party API. Let me give you an example. Have a look at javax.naming.ldap. It is a collection of several interfaces and classes, with a complex way in which you have to call them. If your code depends on this interface you will drown in mocking hell. Now I don’t know why the API is so complex, but I do know that my application only needs a fraction of these calls. I also know that many of these calls are configuration specific and outside of bootstrapping code these APIs are cluttering what I have to mock out.

I start from the other end. I ask myself this question. ‘What would an ideal API look like for my application?’ The key here is ‘my application’ An application which only needs to authenticate will have a very different ‘ideal API’ than an application which needs to manage the LDAP. Because we are focusing on our application the resulting API is significantly simplified. It is very possible that for most applications the ideal interface may be something along these lines.

interface Authenticator {
  boolean authenticate(String username,
                       String password);
}

As you can see this interface is a lot simpler to mock and work with than the original one as a result it is a lot more testable. In essence the ideal interfaces are what separates the testable world from the legacy world.

Once we have an ideal interface all we have to do is implement the adapter which bridges our ideal interface with the actual one. This adapter may be a pain to test, but at least the pain is in a single location.

The benefit of this is that:

• We can easily implement an InMemoryAuthenticator for running our application in the QA environment.
• If the third-party APIs change than those changes only affect our adapter code.
• If we now have to authenticate against a Kerberos or Windows registry the implementation is straight forward.
• We are less likely to introduce a usage bug since calling the ideal API is simpler than calling the original API.

Plugging into an Existing Framework

Let’s take servlets as an example of hard to test framework. Why are servlets hard to test?

• Servlets require a no argument constructor which prevents us from using dependency injection. See how to think about the new operator.
• Servlets pass around HttpServletRequest and HttpServletResponse which are very hard to instantiate or mock.

At a high level I use the same strategy of separating myself from the servlet APIs. I implement my actions in a separate class

class LoginPage {
  Authenticator authenticator;
  boolean success;
  String errorMessage;
  LoginPage(Authenticator authenticator) {
    this.authenticator = authenticator;
  }

  String execute(Map<String, String> parameters,
                 String cookie) {
    // do some work
    success = ...;
    errorMessage = ...;
  }

  String render(Writer writer) {
    if (success)
      return "redirect URL";
    else
      writer.write(...);
  }
}

The code above is easy to test because:

• It does not inherit from any base class.
• Dependency injection allows us to inject mock authenticator (Unlike the no argument constructor in servlets).
• The work phase is separated from the rendering phase. It is really hard to assert anything useful on the Writer but we can assert on the state of the LoginPage, such as success and errorMessage.
• The input parameters to the LoginPage are very easy to instantiate. (Map<String, String>, String for cookie, or a StringWriter for the writer).

What we have achieved is that all of our application logic is in the LoginPage and all of the untestable mess is in the LoginServlet which acts like an adapter. We can than test the LoginPage in depth. The LoginSevlet is not so simple, and in most cases I just don’t bother testing it since there can only be wiring bug in that code. There should be no application logic in the LoginServlet since we have moved all of the application logic to LoginPage.

Let’s look at the adapter class:

class LoginServlet extends HttpServlet {
  Provider<LoginPage> loginPageProvider;

  // no arg constructor required by
  // Servlet Framework
  LoginServlet() {
    this(Global.injector
           .getProvider(LoginPage.class));
  }

  // Dependency injected constructor used for testing
  LoginServlet(Provider<LoginPage> loginPageProvider) {
    this.loginPageProvider = loginPageProvider;
  }

  service(HttpServletRequest req,
          HttpServletResponse resp) {
    LoginPage page = loginPageProvider.get();
    page.execute(req.getParameterMap(),
         req.getCookies());
    String redirect = page.render(resp.getWriter())
    if (redirect != null)
      resp.sendRedirect(redirect);
  }
}

Notice the use of two constructors. One fully dependency injected and the other no argument. If I write a test I will use the dependency injected constructor which will than allow me to mock out all of my dependencies.

Also notice that the no argument constructor is forcing me to use global state, which is very bad, but in the case of servlets I have no choice.  However, I make sure that only servlets access the global state and the rest of my application is unaware of this global variable and uses proper dependency injection techniques.

BTW there are many frameworks out there which sit on top of servlets and which provide you a very testable APIs. They all achieve this by separating you from the servlet implementation and from HttpServletRequest and HttpServletResponse. For example Waffle and WebWork

I think of bugs as being classified into three fundamental kinds of bugs.

• Logical: Logical bug is the most common and classical “bug.” This is your “if”s, “loop”s, and other logic in your code. It is by far the most common kind of bug in an application. (Think: it does the wrong thing)
• Wiring: Wiring bug is when two different objects are miswired. For example wiring the first-name to the last-name field. It could also mean that the output of one object is not what the input of the next object expects. (Think: Data gets clobbered in process to where it is needed.)
• Rendering: Rendering bug is when the output (typical some UI or a report) does not look right. The key here is that it takes a human to determine what “right” is. (Think: it “looks” wrong)

NOTE: A word of caution. Some developers think that since they are building UI everything is a rendering bug! A rendering bug would be that the button text overlaps with the button border. If you click the button and the wrong thing happens than it is either because you wired it wrong (wiring problem) or your logic is wrong (a logical bug). Rendering bugs are rare.

Typical Application Distribution (without Testability in Mind)

The first thing to notice about these three bug types is that the probability is not evenly distributed. Not only is the probability not even, but the cost of finding and fixing them is different. (I am sure you know this from experience). My experience from building web-apps tells me that the Logical bugs are by far the most common, followed by wiring and finally rendering bugs.

Cost of Finding the Bug

Logical bugs are notoriously hard to find. This is because they only show up when the right set of input conditions are present and finding that magical set of inputs or reproducing it tends to be hard. On the other hand wiring bugs are much easier to spot since the wiring of the application is mostly fixed. So if you made a wiring error, it will show up every time you execute that code, for the most part independent of input conditions. Finally, the rendering bugs are the easiest. You simply look at the page and quickly spot that something “looks” off.

Cost of Fixing the Bug

Our experience also tells us how hard it is to fix things. A logical bug is hard to fix, since you need to understand all of the code paths before you know what is wrong and can create a solution. Once the solution is created, it is really hard to be sure that we did not break the existing functionality. Wiring problems are much simpler, since they either manifest themselves with an exception or data in wrong location. Finally rendering bugs are easy since you “look” at the page and immediately know what went wrong and how to fix it. The reason it is easy to fix is that we design our application knowing that rendering will be something which will be constantly changing.

	Logical	Wiring	Rendering
Probability of Occurrence	High	Medium	Low
Difficulty of Discovering	Difficult	Easy	Trivial
Cost of Fixing	High Cost	Medium	Low

How does testability change the distribution?

It turns out that testable code has effect on the distribution of the bugs. Testable code needs:

• Clear separation between classes (Testable Seems) –> clear separation between classes makes it less likely that a wiring problem is introduced. Also, less code per class lowers the probability of logical bug.
• Dependency Injection –> makes wiring explicit (unlike singletons, globals or service locators).
• Clear separation of Logic from Wiring –> by having wiring in a single place it is easier to verify.

The result of all of this is that the number of wiring bugs are significantly reduced. (So as a percentage we gain Logical Bugs. However total number of bugs is decreased.)

The interesting thing to notice is that you can get benefit from testable code without writing any tests. Testable code is better code! (When I hear people say that they sacrificed “good” code for testability, I know that they don’t really understand testable-code.)

We Like writing Unit-Tests

Unit-tests give you greatest bang for the buck. A unit test focuses on the most common bugs, hardest to track down and hardest to fix. And a unit-test forces you to write testable code which indirectly helps with wiring bugs. As a result when writing automated tests for your application we want to overwhelmingly focus on unit test. Unit-tests are tests which focus on the logic and focus on one class/method at a time.

• Unit-tests focus on the logical bugs. Unit tests focus on your “if”s and “loop”s, a Focused unit-test does not directly check the wiring. (and certainly not rendering)
• Unit-test are focused on a single CUT (class-under-test). This is important, since you want to make sure that unit-tests will not get in the way of future refactoring. Unit-tests should HELP refactoring not PREVENT refactorings. (Again, when I hear people say that tests prevent refactorings, I know that they have not understood what unit-tests are)
• Unit-tests do not directly prove that wiring is OK. They do so only indirectly by forcing you to write more testable code.
• Functional tests verify wiring, however there is a trade-off. You “may” have hard time refactoring if you have too many functional test OR, if you mix functional and logical tests.

Managing Your Bugs

I like to think of tests as bug management. (with the goal of bug free) Not all types of errors are equally likley, therefore I pick my battles of which tests I focus on. I find that I love unit-tests. But they need to be focused! Once a test starts testing a lot of classes in a single pass I may enjoy high coverage, but it is really hard to figure out what is going on when the test is red. It also may hinder refactorings. I tend to go very easy on Functional tests. A single test to prove that things are wired together is good enough to me.

I find that a lot of people claim that they write unit-tests, but upon closer inspection it is a mix of functional (wiring) and unit (logic) test.  This happens becuase people wirte tests after code, and therefore the code is not testable. Hard to test code tends to create mockeries. (A mockery is a test which has lots of mocks, and mocks returning other mocks in order to execute the desired code) The result of a mockery is that you prove little. Your test is too high level to assert anything of interest on method level. These tests are too intimate with implementation ( the intimace comes from too many mocked interactions) making any refactorings very painful.

Presented at 2008 OOPSLA by Miško Hevery a Best Practices Coach @ Google

Abstract

Testability Explorer is an open-source tool that identifies hard-to-test Java code. Testability Explorer provides a repeatable objective metric of “testability.” This metric becomes a key component of engineering a social change within an organization of developers. The Testability Explorer report provides actionable information to developers which can be used as (1) measure of progress towards a goal and (2) a guide to refactoring towards a more testable code-base.
Keywords: unit-testing; testability; refactoring; byte-code analysis; social engineering.

1. Testability Explorer Overview

In order to unit-test a class, it is important that the class can be instantiated in isolation as part of a unit-test. The most common pitfalls of testing are (1) mixing object-graph instantiation with application-logic and (2) relying on global state. The Testability Explorer can point out both of these pitfalls.

1.1 Non-Mockable Cyclomatic Complexity

Cyclomatic complexity is a count of all possible paths through code-base. For example: a main method will have a large cyclomatic complexity since it is a sum of all of the conditionals in the application. To limit the size of the cyclomatic complexity in a test, a common practice is to replace the collaborators of class-under-test with mocks, stubs, or other test doubles.

Let’s define “non-mockable cyclomatic complexity” as what is left when the class-under-test has all of its accessible collaborators replaced with mocks. A code-base where the responsibility of object-creation and application-logic is separated (using Dependency Injection) will have high degree of accessible collaborators; as a result most of its collaborators will easily be replaceable with mocks, leaving only the cyclomatic complexity of the class-under-test behind.

In applications, where the class-under-test is responsible for instantiating its own collaborators, these collaborators will not be accessible to the test and as a result will not be replaceable for mocks. (There is no place to inject test doubles.) In such classes the cyclomatic complexity will be the sum of the class-under-test and its non-mockable collaborators.

The higher the non-mockable cyclomatic complexity the harder it will be to write a unit-test. Each non-mockable conditional translates to a single unit of cost on the Testability Explorer report. The cost of static class initialization and class construction is automatically included for each method, since a class needs to be instantiated before it can be exercised in a test.

1.2 Transitive Global-State

Good unit-tests can be run in parallel and in any order. To achieve this, the tests need to be well isolated. This implies that only the stimulus from the test has an effect on the code execution, in other words, there is no global-state.

Global-state has a transitive property. If a global variable refers to a class than all of the references of that class (and all of its references) are globally accessible as well. Each globally accessible variable, that is not final, results in a cost of ten units on the Testability Explorer.

2. Testability Explorer Report

A chain is only as strong as its weakest link. Therefore the cost of testing a class is equal to the cost of the class’ costliest method. In the same spirit the application’s overall testability is de-fined in terms of a few un-testable classes rather than a large number of testable ones. For this reason when computing the overall score of a project the un-testable classes are weighted heavier than the testable ones.

3. How to Interpret the Report

By default the classes are categorized into three categories: “Excellent” (green) for classes whose cost is below 50; “Good” (yellow) for classes whose cost is below 100; and “Needs work” (red) for all other classes. For convenience the data is presented as both a pie chart and histogram distribution and overall (weighted average) cost shown on a dial.

[-]ClassRepository [ 323 ]
  [-]ClassInfo getClass(String) [ 323 ]
    line 51:
      ClassInfo parseClass(InputStream) [318]
      InputStream inputStreamForClass(String) [2]
  [-]ClassInfo parseClass(InputStream) [318]
    line 77: void accept(ClassVisitor, int) [302]
    line 75: ClassReader(InputStream) [15]

Clicking on the class ClassRepository allows one to drill down into the classes to get more information. For example the above report shows that ClassRepository has a high cost of 318 due to the parseClass(InputStream) method. Looking in closer we see that the cost comes from line 77 and an invocation of the accept() method.

73:ClassInfo parseClass(InputStream is) {
74:  try {
75:    ClassReader reader = new ClassReader(is);
76:    ClassBuilder v = new ClassBuilder (this);
77:    reader.accept(v, 0);
78:    return visitor.getClassInfo();
79:  } catch (IOException e) {
80:    throw new RuntimeException(e);
81:  }
82:}

As you can see from the code the ClassReader can never be replaced for a mock and as a result the cyclomatic complexity of the accept method can not be avoided in a test — resulting in a high testability cost. (Injecting the ClassReader would solve this problem and make the class more test-able.)

4. Social Engineering

In order to produce a lasting change in the behavior of developers it helps to have a measurable number to answer where the project is and where it should be. Such information can provide in-sight into whether or not the project is getting closer or farther from its testability goal.

People respond to what is measured. Integrating the Testability Explorer with the project’s continuous build and publishing the report together with build artifacts communicate the importance of testability to the team. Publishing a graph of overall score over time allows the team to see changes on per check-in basis.

If Testability Explorer is used to identify the areas of code that need to be refactored, than compute the rate of improvement and project expected date of refactoring completion and create a sense of competition among the team members.

It is even possible to set up a unit test for Testability Explorer that will only allow the class whose testability cost is better than some predetermined cost.

After reading the article on Singletons (the design anti-pattern) and how they are really global variables and dependency injection suggestion to simply pass in the reference to the singleton in a constructor (instead of looking them up in global state), many people incorrectly concluded that now they will have to pass the singleton all over the place. Let me demonstrate the myth with the following example.

Let’s say that you have a LoginPage which uses the UserRepository for authentication. The UserRepository in turn uses Database Singleton to get a hold of the global reference to the database connection, like this:

class UserRepository {
  private static final BY_USERNAME_SQL = "Select ...";

  User loadUser(String user) {
    Database db = Database.getInstance();
    return db.query(BY_USERNAME_SQL, user);
  }
}

class LoginPage {
  UserRepository repo = new UserRepository();

  login(String user, String pwd) {
    User user = repo.loadUser(user);
    if (user == null || user.checkPassword(pwd)) {
      throw new IllegalLoginException();
    }
  }
}

The first thought is that if you follow the advice of dependency injection you will have to pass in the Database into the LoginPage just so you can pass it to the UserRepository. The argument goes that this kind of coding will make the code hard to maintain, and understand. Let’s see what it would look like after we get rid of the global variable Singleton Database look up.

First, lets have a look at the UserRepository.

class UserRepository {
  private static final BY_USERNAME_SQL = "Select ...";
  private final Database db;

  UserRepository(Database db) {
    this.db = db;
  }

  User loadUser(String user) {
    return db.query(BY_USERNAME_SQL, user);
  }
}

Notice how the removal of Singleton global look up has cleaned up the code. This code is now easy to test since in a test we can instantiate a new UserRepository and pass in a fake database connection into the constructor. This improves testability. Before, we had no way to intercept the calls to the Database and hence could never test against a Database fake. Not only did we have no way of intercepting the calls to Database, we did not even know by looking at the API that Database is involved. (see Singletons are Pathological Lairs) I hope everyone would agree that this change of explicitly passing in a Database reference greatly improves the code.

Now lets look what happens to the LoginPage…

class LoginPage {
  UserRepository repo;

  LoginPage(Database db) {
    repo = new UserRepository(db);
  }

  login(String user, String pwd) {
    User user = repo.loadUser(user);
    if (user == null || user.checkPassword(pwd)) {
      throw new IllegalLoginException();
    }
  }
}

Since UserRepository can no longer do a global look-up to get a hold of the Database it musk ask for it in the constructor. Since LoginPage is doing the construction it now needs to ask for the Databse so that it can pass it to the constructor of the UserRepository. The myth we are describing here says that this makes code hard to understand and maintain. Guess what?! The myth is correct! The code as it stands now is hard to maintain and understand. Does that mean that dependency injection is wrong? NO! it means that you only did half of the work! In how to think about the new operator we go into the details why it is important to separate your business logic from the new operators. Notice how the LoginPage violates this. It calls a new on User Repository. The issue here is that LoginPage is breaking a the Law of Demeter. LoginPage is asking for the Database even though it itself has no need for the Database (This greatly hinders testability as explained here). You can tell since LoginPage does not invoke any method on the Database. This code, like the myth suggest, is bad! So how do we fix that?

We fix it by doing more Dependency Injection.

class LoginPage {
  UserRepository repo;

  LoginPage(UserRepository repo) {
    this.repo = repo;
  }

  login(String user, String pwd) {
    User user = repo.loadUser(user);
    if (user == null || user.checkPassword(pwd)) {
      throw new IllegalLoginException();
    }
  }
}

LoginPage needs UserRepository. So instead of trying to construct the UserRepository itself, it should simply ask for the UserRepository in the constructor. The fact that UserRepository needs a reference to Database is not a concern of the LoginPage. Neither is it a concern of LoginPage how to construct a UserRepository. Notice how this LoginPage is now cleaner and easier to test. To test we can simply instantiate a LoginPage and pass in a fake UserRepository with which we can emulate what happens on successful login as well as on unsuccessful login and or exceptions. It also nicely takes care of the concern of this myth. Notice that every object simply knows about the objects it directly interacts with. There is no passing of objects reference just to get them into the right location where they are needed. If you get yourself into this myth then all it means is that you have not fully applied dependency injection.

So here are the two rules of Dependency Injection:

• Always ask for a reference! (don’t create, or look-up a reference in global space aka Singleton design anti-pattern)
• If you are asking for something which you are not directly using, than you are violating a Law of Demeter. Which really means that you are either trying to create the object yourself or the parameter should be passed in through the constructor instead of through a method call. (We can go more into this in another blog post)

So where have all the new operators gone you ask? Well we have already answered that question here. And with that I hope we have put the myth to rest!

BTW, for those of you which are wondering why this is a common misconception the reason is that people incorrectly assume that the constructor dependency graph and the call graph are inherently identical (see this post). If you construct your objects in-line (as most developers do, see thinking about the new operator) then yes the two graphs are very similar. However, if you separate the object graph instantiation from the its execution, than the two graphs are independent. This independence is what allows us to inject the dependencies directly where they are needed without passing the reference through the intermediary collaborators.

Dependency injection asks us to separate the new operators from the application logic. This separation forces your code to have factories which are responsible for wiring your application together. However, better than writing factories,  we want to use automatic dependency injection such as GUICE to do the wiring for us. But can DI really save us from all of the new operators?

Lets look at two extremes. Say you have a class MusicPlayer which needs to get a hold of AudioDevice. Here we want to use DI and ask for the AudioDevice in the constructor of the MusicPlayer. This will allow us to inject a test friendly AudioDevice which we can use to assert that correct sound is coming out of our MusicPlayer. If we were to use the new operator to instantiate the BuiltInSpeakerAudioDevice we would have hard time testing. So lets call objects such as AudioDevice or MusicPlayer “Injectables.” Injectables are objects which you will ask for in the constructors and expect the DI framework to supply.

Now, lets look at the other extreme. Suppose you have primitive “int” but you want to auto-box it into an “Integer” the simplest thing is to call new Integer(5) and we are done. But if DI is the new “new” why are we calling the new in-line? Will this hurt our testing? Turns out that DI frameworks can’t really give you the Integer you are looking for since it does not know which Integer you are referring to. This is a bit of a toy example so lets look at something more complex.

Lets say the user entered the email address into the log-in box and you need to call new Email(”a@b.com”). Is that OK, or should we ask for the Email in our constructor. Again, the DI framework has no way of supplying you with the Email since it first needs to get a hold of a String where the email is. And there are a lot of Strings to chose from. As you can see there are a lot of objects out there which DI framework will never be able to supply. Lets call these “Newables” since you will be forced to call new on them manually.

First, lets lay down some ground rules. An Injectable class can ask for other Injectables in its constructor. (Sometimes I refer to Injectables as Service Objects, but that term is overloaded.) Injectables tend to have interfaces since chances are we may have to replace them with an implementation friendly to testing. However, Injectable can never ask for a non-Injectable (Newable) in its constructor. This is because DI framework does not know how to produce a Newable. Here are some examples of classes I would expect to get from my DI framework: CreditCardProcessor, MusicPlayer, MailSender, OfflineQueue. Similarly Newables can ask for other Newables in their constructor, but not for Injectables (Sometimes I refer to Newables as Value Object, but again, the term is overloaded). Some examples of Newables are: Email, MailMessage, User, CreditCard, Song. If you keep this distinctions your code will be easy to test and work with. If you break this rule your code will be hard to test.

Lets look at an example of a MusicPlayer and a Song

class Song {
  Song(String name, byte[] content);
}
class MusicPlayer {
  @Injectable
  MusicPlayer(AudioDevice device);
  play(Song song);
}

Notice that Song only asks for objects which are Newables. This makes it very easy to construct a Song in a test. Music player is fully Injectable, and so is its argument the AudioDevice, therefore, it can be gotten from DI framework.

Now lets see what happens if the MusicPlayer breaks the rule and asks for Newable in its constructor.

class Song {
  String name;
  byte[] content;
  Song(String name, byte[] content);
}
class MusicPlayer {
  AudioDevice device;
  Song song;
  @Injectable
  MusicPlayer(AudioDevice device, Song song);
  play();
}

Here the Song is still Newable and it is easy to construct in your test or in your code. The MusicPlayer is the problem. If you ask DI framework for MusicPlayer it will fail, since the DI framework will not know which Song you are referring to. Most people new to DI frameworks rarely make this mistake since it is so easy to see: your code will not run.

Now lets see what happens if the Song breaks the rule and ask for Injectable in its constructor.

class MusicPlayer {
  AudioDevice device;
  @Injectable
  MusicPlayer(AudioDevice device);
}
class Song {
  String name;
  byte[] content;
  MusicPlayer palyer;
  Song(String name, byte[] content, MusicPlayer player);
  play();
}
class SongReader {
  MusicPlayer player
  @Injectable
  SongReader(MusicPlayer player) {
    this.player = player;
  }
  Song read(File file) {
    return new Song(file.getName(),
                    readBytes(file),
                    player);
  }
}

At first the world looks OK. But think about how the Songs will get created. Presumably the songs are stored on a disk and so we will need a SongReader. The SongReader will have to ask for MusicPlayer so that when it calls the new on a Song it can satisfy the dependencies of Song on MusicPlayer. See anything wrong here? Why in the world does SongReader need to know about the MusicPlayer. This is a violation of Law of Demeter. The SongReader does not need to know about MusicPlayer. You can tell since SongReader does not call any method on the MusicPlayer. It only knows about the MusicPlayer because the Song has violated the Newable/Injectable separation. The SongReader pays the price for a mistake in Song. Since the place where the mistake is made and where the pain is felt are not the same this mistake is very subtle and hard to diagnose. It also means that a lot of people make this mistake.

Now from the testing point of view this is a real pain. Suppose you have a SongWriter and you want to verify that it correctly serializes the Song to disk. Why do you have to create a MockMusicPlayer so that you can pass it into a Song so that you can pass it into the SongWritter. Why is MusicPlayer in the picture? Lets look at it from a different angle. Song is something you may want to serialize, and simplest way to do that is to use Java serialization. This will serialize not only the Song but also the MusicPlayer and the AudioDevice. Neither MusicPlayer nor the AudioDevice need to be serialized. As you can see a subtle change makes a whole lot of difference in the easy of testability.

As you can see the code is easiest to work with if we keep these two kinds objects distinct. If you mix them your code will be hard to test.  Newables are objects which are at the end of your application object graph. Newables may depend on other Newables as in CreditCard may depend on Address which may depend on a City but these things are leafs of the application graph. Since they are leafs, and they don’t talk to any external services (external services are Injectables) there is no need to mock them. Nothing behaves more like a String like than a String. Why would I mock User if I can just new User, Why mock any of these: Email, MailMessage, User, CreditCard, Song? Just call new and be done with it.

Now here is something very subtle. It is OK for Newable to know about Injectable. What is not OK is for the Newable to have a field reference to Injectable. In other words it is OK for Song to know about MusicPlayer. For example it is OK for an Injectable MusicPlayer to be passed in through the stack to a Newable Song. This is because the stack passing is independent of DI framework. As in this example:

class Song {
  Song(String name, byte[] content);
  boolean isPlayable(MusicPlayer player);
}

The problem becomes when the Song has a field reference to MusicPlayer. Field references are set through the constructor which will force a Law of Demeter violation for the caller and we will have hard time to test.

In My main() Method Is Better Than Yours we looked into what a main() method should look like. There we introduced a clear separation between (1) the responsibility of constructing the object graph and (2) the responsibility of running the application. The reason that this separation is important was outlined in How to Think About the “new” Operator. So let us look at where have all of the new operators gone…

Before we go further I want you to visualize your application in your mind. Think of the components of your application as physical boxes which need to be wired together to work. The wires are the references one component has to another. In an ideal application you can change the behavior of the application just by wiring the components differently. For example instead of instantiating LDAPAuthenticator you instantiate KerberosAuthenticator and you wire the KerberosAuthenticator to appropriate components which need to know about Authenticator. That is the basic idea. By removing the new operators from the application logic you have separated the responsibility of wiring the components from the application logic, and this is highly desirable. So now the problem becomes, where have all the new operators gone?

First lets look at a manual wiring process. In the main() method we asked the ServerFactory to build us a Server (in our case a Jetty Web Server) Now, server needs to be wired together with servlets. The servlets, in turn, need to be wired with their services and so on. Notice that the factory bellow is full of “new” operators. We are new-ing the components and we are passing the references of one component to another to create the wiring. This is the instantiation and wiring activity I asked you to visualize above. (Full source):

  public Server buildServer() {
    Server server = new Server();

    SocketConnector socketConnector
         = new SocketConnector();
    socketConnector.setPort(8080);
    server.addConnector(socketConnector);

    new ServletBuilder(server)
      .addServlet("/calc", new CalculatorServlet(
                           new Calculator()))
      .addServlet("/time", new TimeServlet(
                           new Provider() {
        public Date get() {
          return new Date();
        }
      }));

    return server;
  }

When I first suggest to people that application logic should not instantiate its own dependencies, I get two common objections which are myths:

1. “So now each class needs a factory, therefore I have twice as many classes!” Heavens No! Notice how our ServerFactory acted as a factory for many different classes. Looking at it I counted 7 or so classes which we instantiated in order to wire up our application. So it is not true that we have one to one correspondence. In theory you only need one Factory per object lifetime. You need one factory for all long-lived objects (your singletons) and one for all request-lifetime objects and so on. Now in practice we further split those by related concepts. (But that is a discussion for a separate blog article.) The important thing to realize is that: yes, you will have few more classes, but it will be no where close to doubling your load.
2. “If each object asks for its dependencies, than I will have to pass those dependencies through all of the callers. This will make it really hard to add new dependencies to the classes.” The myth here is that call-graph and instantiation-graph are one and the same. We looked into this myth in Where have all the Singletons Gone. Notice that the Jetty server calls the TimeServlet which calls the Date. If the constructor of Date or TimeServlet all of a sudden needed a new argument it would not effect any of the callers. The only code which would have to change is factory class above. This is because we have isolated the instantiation/wiring problem into this factory class. So in reality this makes it easier to add dependencies not harder.

Now there are few important things to remember. Factories should have no logic! Just instantiation/wiring (so you will probably not have any conditionals or loops). I should be able to call the factory to create a server in a unit test without any access to the file-system, threads or any other expensive CPU or I/O operations. Factory creates the server, but does not run it. The other thing you want to keep in mind is that the wiring process is often controlled by the command line arguments. This makes is so that your application can behave differently depending what you pass in on a command line. The difference in behavior is not conditionals sprinkled throughout your code-base but rather a different way of wiring your application up.

Finally, here are few thoughts on my love/hate of Singletons (mentioned here and here) First a little review of singletons. A singleton with a lower case ’s’ is a good singleton and simply means a single instance of some class. A Singleton with an upper case ‘S’ is a design pattern which is a singleton (one instance of some class) with a global “instance” variable which makes it accessible from anywhere. It is the global instance variable which makes it globally accessible , which turns a singleton into a Singleton. So singleton is acceptable, and sometimes very helpful for a design, but Singleton relies on mutable global state, which inhibits testability and makes a brittle, hard to test design. Now notice that our factory created a whole bunch of singletons as in a single instance of something . Also notice how those singletons got explicitly passed into the services that needed them. So if you need a singleton you simply create a single instance of it in the factory and than pass that instance into all of the components which need them. There is no need for the global variable.

For example a common use of Singleton is for a DB connection pool. In our example you would simply instantiate a new DBConnectionPool class in the top-most factory (above) which is responsible for creating the long-lived objects. Now lets say that both CalculatorServlet and TimeServlet would need a connection pool. In that case we would simply pass the same instance of the DBConnectionPool into each of the places where it is needed. Notice we have a singleton (DBConnectionPool) but we don’t have any global variables associated with that singleton.

  public Server buildServer() {
    Server server = new Server();

    SocketConnector socketConnector
         = new SocketConnector();
    socketConnector.setPort(8080);
    server.addConnector(socketConnector);

    DBConnectionPool pool = new DBConnectionPool();
    new ServletBuilder(server)
      .addServlet("/calc", new CalculatorServlet(
                           pool,
                           new Calculator()))
      .addServlet("/time", new TimeServlet(
                           pool,
                           new Provider() {
        public Date get() {
          return new Date();
        }
      }));

    return server;
  }

In Part I of this series, we took a realistic look at what usually happens when we initiate change.  We also took a look at the initial steps of effective change: defining a metric and getting people to accept it as a goal. In this second and final part, we’ll introduce two additional steps and also highlight the point at which it becomes clear that change has taken effect.

Step 3: Make Progress Visible

When progress toward achieving a goal is highly visible, it can’t be ignored:

• It keeps the goal fresh in everyone’s mind and helps to prevent regressive behaviors.
• It communicates progress to all stakeholders, especially those who may not understand the details of the work.
• It provides an opportunity for many small celebrations for a team on the way to achieving its goal.
• There’s a direct relationship between what developers do and changes to the visible “progress meter,” providing a source of pride and “bragging rights” for individuals.
• Regression can be easily identified, right down to the work done (the commit) and the “guilty” (the committer).

We can make progress visible by publishing both the raw metric and a burn down chart which computes the estimated date of completion based on the rate of progress.

This makes it easier to answer the obvious question, “When is it going to be done?” It also provides a fact-based response should there be a need to push back on demands, e.g., “you need to get this done by X date.”

The continuous build status page is very well suited to this. With every build, the change metric can be computed and published automatically.  The continuous build status page will then show a current chart of progress over time.

Going back to our example, we can use Testability Explorer to compute an overall cost number. Let’s say that our project scores a cost of 500, and we make it our goal to lower the cost to 50 or below.  As part of our continuous build we can produce a graph of the testability cost over time. We can project this graph in a highly visible location (on the wall, on the ceiling) so that it is on everyone’s mind all the time.  We can also easily identify people who are improving the situation with the code they commit.  By calling out their contribution to the team, we get additional buy-in from the team.

Before long, there will be an anxious manager running around asking, “why is the graph is not falling fast enough?” When that happens, the change process is self-running and will complete itself.  By keeping the graph current and visible even after the goal is reached, the change will be durable.

Step 4: Make it Required

By and large, developers care only about checking-in code. They are content to continue bad habits if those habits enable them to check in work sooner. When the priority is to get code checked-in, all those fancy graphs will simply show the team getting further away from its stated goal.

If this happens, no amount of meetings and discussion will make a difference, but one thing will: create a unit test that makes sure that any code change can only move the metric in a positive direction. This way any changes to the source code that take the team further from the goal fail the build. Once the build goes into a “red” or broken state, it needs to be fixed. This means that every developer will have to deal with the test to get the build to pass. Fixing the code to pass the build brings the team closer to its goal. A specific code change made to resolve a broken build makes the “what has to change” very clear to every person on the team.

Expect a lot of uproar, and that each developer will require individual attention to help them re-factor their code so that it passes the unit test. The screaming will stop as soon as each person has received individual attention to learn how to code correctly.  This won’t move the needle on the chart, but the numbers won’t get any worse.

This suggests that early on, it will need to be someone’s job to re-factor code. Only then will the chart move in the desired direction. Very likely, this is the only way the team will make any progress initially. This is because developers will know how to write new code, but they won’t know how (or won’t be motivated) to re-factor the old code to conform to new standard. Additionally, they will not yet have experienced any benefits from this new way of doing things. This typically takes a few months.  While this is taking root, you will need to be patient to make slow but steady progress refactoring the existing code until developers catch up.

The Tipping Point

At some point, developers themselves will start refactoring old code and the rate of progress will accelerate. The interesting thing is that even if you - the agent of change - leave a project at this point, the goal will eventually be reached. The visibility of the charts clearly show where the team is and how much remains to achieve the goal, and this becomes ingrained into the everyday life of both developers and management. People will see the estimated date of completion and start counting the days to it. Each person will also keep the team focused on the goal. Because the estimated time of completion is based on past progress, the projected date of completion will slip if the rate of refactoring slows down. When this happens, people will ask why the date slipped. This brings attention back to the effort and reignites it. This feedback loop will exist even without a full-time change agent.

Every change has its opponents. At some point the most vociferous opponents become the most vocal champions because they experience the benefits of it first-hand. At the same time, all “passengers” in a project team find they’re doing something for so long that they can’t imagine working any other way. The team achieves its goal either by strong desire or because it has developed new “muscle memory.”

Behaviour Changes when Goals are Visible and Reinforced

To successfully make change, it isn’t enough to get consensus to try something new. You need a means by which to quantify a goal, objectively measure progress, and constantly remind people of how well they’re doing toward achieving that goal.  If these things aren’t done, developers will find reasons and excuses not to change. This creates a downward spiral: by carrying on with business as usual, skills stagnate and technical debt accumulates. The irony is that precicely when a team most neads to make change, its own behaviors work against it. A high degree of visibility, and immediate, constant feedback, can overcome even the most difficult team situations.

People are good at turning concrete examples into generalization. The other way around, it does not work so well. So when I write about general concepts it is hard for people to know how to translate the general concept into concrete code. To remedy this I will try to show few examples of how to build a web application from ground up. But I can’t fit all of that into a single blog post … So lets get started at the beginning…

Here is what your main method should look like (no matter how complex your application) if you are using GUICE: (src)

public static void main(String[] args)
  throws Exception {
    // Creation Phase
    Injector injector = Guice.createInjector(
             new CalculatorServerModule(args));
    Server server = injector.getInstance(Server.class);
    // Run Phase
    server.start();
}

Or if you want to do manual dependency injection: (src)

public static void main(String[] args)
  throws Exception {
    // Creation Phase
    Server server = new ServerFactory(args)
                               .createServer();
    // Run Phase
    server.start();
}

The truth is I don’t know how to test the main method. The main method is static and as a result there are no places where we can inject test-doubles. (I know we can fight static with static, but we already said that global state is bad here, here and here). The reason we can’t test this is that the moment you execute the main method the whole application runs, and that is not what we want and there is nothing we can do to prevent that.

But the method is so short that I don’t bother testing it since it has some really cool properties:

1. Notice how the creation-phase contains the code which builds the object graph of the application. The last line runs the application. The separation is very important. We can test the ServerFactory in isolation. Passing it different arguments and than asserting that the correct object graph got built. But, in order to do that the Factory class should do nothing but object graph construction. The object constructors better do nothing but field assignments. No reading of files, starting of threads, or any other work which would cause problems in unit-test. All we do is simply instantiate some graph of objects. The graph construction is controlled by the command line arguments which we passed into the constructor. So we can test creation-phase in isolation with unit-test. (Same applies for GUICE example)
2. The last line gets the application running. Here is where you can do all of your fun threads, file IO etc code. However, because the application is build from lots of objects collaborating together it is easy to test each object in isolation. In test I just instantiate the Server and pass in some test doubles in the constructor to mock out the not so interesting/hard to test code.

As you can see we have a clear separation of the object graph construction responsibility from the application logic code. If you were to examine the code in more detail you would find that all of the new operators have migrated from the run-phase  to creation-phase (See How to Think About the “new” Operator) And that is very important. New operator in application code is enemy of testing, but new in tests and factories is your friend. (The reason is that in tests we want to use test-doubles which are usually a subclass or an implementation of the parent class. If application code calls new than you can never replace that new with a subclass or different implementation.) The key is that the object creation responsibility and the the application code are two different responsibilities and they should not be mixed. Especially in the main method!

A good way to think about this is that you want to design your application such that you can control the application behavior by controlling the way you wire the objects together (Object collaborator graph). Whether you wire in a InMemory, File or Database repository, PopServer or IMAPServer, LDAP or file based authentication. All these different behaviors should manifest themselves as different object graphs. The knowledge of how to wire the objects together should be stored in your factory class. If you want to prevent something from running in a test, you don’t place an if statement in front of it. Instead you wire up a different graph of objects. You wire NullAthenticator in place of LDAPAuthenticator. Wiring your objects differently is how the tests determines what gets run and what gets mocked out. This is why it is important for the tests to have control of the new operators (or putting it differently the application code does not have the new operators). This is why we don’t know how to test the main method. Main method is static and hence procedural. I don’t know how to test procedural code since there is nothing to wire differently. I can’t wire the call graph different in procedural world to prevent things from executing, the call graph is determined at compile time.

In my experience that main method usually is some of the scariest code I have seen. Full of singleton initialization and threads. Completely untestable. What you want is that each object simply declares its dependencies in its constructor. (Here is the list of things I need to know about) Then when you start to write the Factory it will practically write itself. You simply try to new the object you need to return, which declares its dependencies, you in turn try to new those dependencies, etc… If there are some singletons you just have to make sure that you call the new operator only once. But more on factories in our next blog post…

Lets get the definition right. There is Singleton the design pattern (Notice the capital “S” as in name of something) and there is a singleton as in one of something (notice the lower case “s”). There is nothing wrong with having a single instance of a class, lots of reasons why you may want to do that. However, when I complain about the Singletons, I complain about the design pattern. Specifically: (1) private constructor and (2) global instance variable which refers to the singleton. So from now on when I say Singleton, I mean the design (anti)pattern.

I would say that at this point most developers recognize that global state is harmful to your application design. Singletons have global instance variable which points to the singleton. The instance is global. The trouble with global variables is that they are transitive. It is not just the global variable marked with static which is global but any other variable/object which is reachable by traversing the object graph. All of it is global! Singletons, usually are complex objects which contain a lot of state. As a result all of the state of Singleton is global as well. I like to say that “Singletons are global state in sheep’s clothing.” Most developers agree that global state is bad, but they love their Singletons.

The moment you traverse a global variable your API lies about its true dependencies (see: Singletons are Pathological Liars) The root problem is not the Singleton design pattern, the root problem here is the global reference to singleton. But the moment you get rid of the global variable you get rid of the Singleton design pattern. So from my point of view blaming Singletons or blaming global state is one and the same. You can’t have a Singleton design pattern and at the same time not have the global state.

Someone pointed out that any design pattern can be abused. I agree, but with Singleton design pattern, I don’t know how I can possibly use it in a good way. The global reference and hence the global state is ever so present. Now, in my line of work I don’t see too much global state in classical sense of the word, but I see a lot of global state masquerading as Singletons. Hence, I complain about Singletons. If I would complain about global state no one would care, as that is old news.

Now, there is one kind of Singleton which is OK. That is a singleton where all of the reachable objects are immutable. If all objects are immutable than Singleton has no global state, as everything is constant. But it is so easy to turn this kind of singleton into mutable one, it is very slippery slope. Therefore, I am against these Singletons too, not because they are bad, but because it is very easy for them to go bad. (As a side note Java enumeration are just these kind of singletons. As long as you don’t put state into your enumeration you are OK, so please don’t.)

The other kind of Singletons, which are semi-acceptable are those which don’t effect the execution of your code. Logging is perfect example. It is loaded with Singletons and global state. It is acceptable (as in it will not hurt you) because your application does not behave any different whether or not a given logger is enabled. The information here flows one way: From your application into the logger. Even thought loggers are global state since no information flows from loggers into your application, loggers are acceptable. You should still inject your logger if you want your test to assert that something is getting logged, but in general Loggers are not harmful despite being full of state.

So the root cause is “GLOBAL STATE!” Keep in mind that global state is transitive, so any object which is reachable from a global variable is global as well. It is not possible to have a Singleton and not have a global state. Therefore, Singleton design patter can not be used in “the right way.” Now you could have a immutable singleton, but outside of limited use as enumerations, they have little value. Most applications are full of Singletons which have lots of global state, and where the information flows both directions.