PCL -> Clojure, Chapter 9

This article is part of a series describing a port of the samples from Practical Common Lisp (PCL) to Clojure. You will probably want to read the intro first.

This article covers Chapter 9, Practical: Building a Unit Test Framework.

Tests and reports

To build a minimal testing library, I need nothing more than tests and results. To keep reporting as simple as possible, I will start with console output. The report-result function tests a result, and prints pass or FAIL, plus a form with supporting detail:

  (defn report-result [result form]
    (println (format "%s: %s" (if result "pass" "FAIL") (pr-str form))))

Now any function can be a test. The detail message can often be the same form that caused the error, so I will pass the same form twice: once for evaluation, and again (quoted!) for use in the detail message:

  (defn test-+ []
    (report-result (= (+ 1 2) 3) '(= (+ 1 2) 3))
    (report-result (= (+ 1 2 3) 6) '(= (+ 1 2 3) 6))
    (report-result (= (+ -1 -3) -4) '(= (+ -1 -3) -4)))

The console output for test-+ looks like this:

  user=> (test-+)
  pass: (= (+ 1 2) 3)
  FAIL: (= (+ 1 2 3) 7)
  pass: (= (+ -1 -3) -4)

Inferring the detail message

The fact that I want to pass the same form twice, but with different evaluation semantics, just screams macro. Sure enough, I can clean up the code with a macro:

  (defmacro check [form]
    `(report-result ~form '~form))

The macro expands the form twice, once for evaluation and once quoted for the detail message. Now I can replace calls to report-result with simpler calls to check:

  (defn test-* []
    (check (= (* 1 2) 3))
    (check (= (* 1 2 3) 6))
    (check (= (* -1 -3) -4)))

Hmm. The calls to check are cleaner than the calls to report-result in the earlier example, but the check itself still looks repetitive. Solution: a better check macro that can handle multiple forms:

  (defmacro check [& forms]
    `(do
       ~@(map (fn [f] `(report-result ~f '~f))  forms)))

The quoting and unquoting is a little more complex–play around with macroexpand-1 to see how it works.

With the better check in place, test functions are quite simple:

  (defn test-rem []
    (check (= (rem 10 3) 1)
  	 (= (rem 6 2) 0)
  	 (= (rem 7 4) 3)))

Aggregating results

So far I have tests and console output. Next, I need some way to aggregate a set of checks into a single, top-level "checks passed" or "checks failed".

I would like to simply and together all the individual checks, but that does not quite work. As in many languages, Clojure's and short-circuits and stops evaluating when it encounters a logical false. That's no good here: Even if one test fails, I still want all the tests to run.

Since it is a question of optional evaluation, a macro is appropriate. The combine-results macro works like and, but it always evaluates all the forms:

  (defmacro combine-results [& forms]
    `(every? identity (list ~@forms)))

Now check can use combine-results instead of do.

  (defmacro check [& forms]
    `(combine-results
      ~@(map (fn [f] `(report-result ~f '~f)) forms)))

All existing functionality still works, and now I can see a useful return value from a test.

  user=> (test-*)
  pass: (= (* 2 4) 8)
  pass: (= (* 3 3) 9)
  true

Capturing test names

Tests ought to have names. In fact, tests ought to support multiple names. You can imagine a test detail report saying:

Check math->addition->associative passed: ...

Where associative is the name of a check, addition is the name of a function, and math is the name of another function that called addition.

First, I need a variable to store a sequence of names:

  (def *test-name* [])

Printing the variable as part of a result is easy:

  (defn report-result [result form]
    (println (format "%s: %s %s" 
  		   (if result "pass" "fail") 
  		   (pr-str *test-name*) 
  		   (pr-str form)))
    result)

Now for the hard part: populating the collection of names. For this, I will introduce a deftest macro:

  (defmacro deftest [name & forms]
    `(defn ~name []
       (binding [*test-name* (conj *test-name* (str '~name))]
         ~@forms)))

The macro expansion perfomed by deftest is nothing new: deftest turns around and defns a new function named name. The interesting part is the call to binding, which rebinds *test-name* to a new collection built from the old *test-name* plus the name of the current test.

The new binding of *test-name* is visible anywhere inside the dynamic scope of the binding form. The dynamic scope includes any function calls made inside the binding, and their function calls, and so on ad infinitum … or until another binding performs the same trick again. This gives exactly the semantics we want:

  • The dynamic scope allows callers to influence callees without having to pass test-name an an argument all over the place. Nested functions "remember" a stack of their caller's names through *test-name*.
  • The unwinding of the dynamic scope protects readers of *test-name* outside a binding. Code after the binding will never see the values *test-name* takes during the binding.
  • Dynamic bindings are thread-local (and therefore thread-safe).

With deftest in place, I can defined a hierarchy of nested tests:

  (deftest test-*
    (check (= (* 2 4) 8)
  	 (= (* 3 3) 9)))

  (deftest test-math
    ; TODO: test rest of math
    (test-*))

  (deftest test-all-of-nature
    ; TODO: test rest of nature
    (test-math))

Calling test-all-of-nature will demonstrate multiple levels of nested name in a test report:

  user=> (test-all-of-nature)
  pass: ["test-all-of-nature" "test-math" "test-*"] (= (* 2 4) 8)
  pass: ["test-all-of-nature" "test-math" "test-*"] (= (* 3 3) 9)
  true                         

From here, better formatting of the console message is just mopping up.

Wrapping up

When I first read Practical Common Lisp, this was my favorite chapter. The testing library evolves quickly and naturally to a substantial feature set. (In case you didn't keep count, the entire "framework" is less than twenty lines of code.)

Try implementing the unit-testing example in your language of choice. Don't just implement the finished design. Work through each of the iterations described above:

  1. tests and results
  2. inferring the detail message
  3. aggregating results
  4. capturing test names

I would love to hear about your results, and I will link to them here.

Notes

Get In Touch