...and the Java community is collecting hard data to prove it.

UPDATE 5/22/08: Please read this first/instead. It is a much clearer statement of my ideas. I am leaving the original unchanged below, of course.

The Testability Explorer is an open source project that lets you measure the testability of Java code. This is an interesting idea: a metric not of direct quality, or of testing, or of test coverage, but of ease of testing. Presumably code that is easy to test will get tested, and will therefore be of higher quality, other things being equal.

How does it work? By calculating two proxy measures for the difficulty of isolating code in a test:

• Non-mockable total recursive cyclomatic complexity. (Rough translation: difficulty caused by code that cannot be stubbed out for test purposes.)
• Global mutable state.

These measures are then aggregated at the class level, and converted into one of three simple assessments: "excellent", "good", or "needs work". These class-level assessments then roll up to a color code score for an entire project. You can see the color codes for a bunch of Java projects on the Testability Explorer home page.

The overall approach may not be perfect, but for the moment, I'll take the approach as given and ask this question: "How would the metric apply to code written in a lower-ceremony language like Ruby?" I recompiled the Testability Explorer, adding a new CEREMONY flag. HIGH ceremony assumes Java as it is, and LOW ceremony assumes a hypothetical Java that is more like Groovy or Ruby.

I then ran Testability Explorer in HIGH CEREMONY mode on ant.jar. Here are the results, elided for brevity.

       Analyzed classes:   751
   Excellent classes (.):   497  66.2%
        Good classes (=):    28   3.7%
  Needs work classes (@):   226  30.1%

Having 30.1% of classes in the "needs work" category earns the Ant project an ugly orange splotch on the Testability Explorer home page. I then ran the same test again, in LOW CEREMONY mode:

        Analyzed classes:   751
   Excellent classes (.):   732  97.5%
        Good classes (=):    15   2.0%
  Needs work classes (@):     4   0.5%

Wow. If ant.jar were written in LOW CEREMONY Java, it would earn a happy green spot on the Testability Explorer home page. And in fact, ant.jar is already written in LOW CEREMONY Java. I didn't change ant.jar, I changed the language.

What does this mean? Maybe nothing. But here is one story you might tell:

• It is difficult to write testable code in Java. Even some solid, successful open source projects score poorly.
• Testability problems are baked into the Java language.
• Making code testable (at least in the terms defined by the TE project) does not have to be difficult for developers. With some languages, testability happens naturally.

Congratulations to the Java projects that are in the green on Testability Explorer. It isn't impossible, but it sure is hard work.