There’s quite a bit of exten­si­bility in clojure.test. A variety of test runners and report format­ters, such as James Reeves’ Eftest, Metosin’s boot-alt-test, and Paul Stadig’s Humane Test Output hook into the clojure.test reporting framework.

There’s also assert-expr, which allows you to extend the is macro to encap­su­late domain-spe­cific test­ing, as well as add domain-spe­cific report­ing. Many of clojure.test’s own asser­tions, such as instance? and thrown?, are imple­mented as assert-expr meth­ods. Daniel Solano Gómez has written a library with helpers for Clojure spec called tacular which includes a number of custom asser­tions. The test.check library uses a custom asser­tion to improve reporting in its defspec testing macro.

Writing these custom asser­tions in a portable way can be a bit of a trick, so let’s work through an exam­ple: an asser­tion that tests the data returned from a thrown instance of ExceptionInfo.

It’s often useful to start off with an idea of how the library is to be used, even prior to writing a single line of implementation.

;; file: test/src/com/grzm/ex/assertions_test.cljc
(ns com.grzm.ex.assertions-test
  (:require
   [clojure.test :refer [deftest is]]
   [com.grzm.ex.assertions]))

(deftest test-thrown-with-data
  (is (com.grzm.ex.assertions/thrown-with-data?
        #(= ::self-inflicted (:cause %)) ;; function to match ex-data
        ;; and something that throws ExceptionInfo
        (throw (ex-info "An error was thrown" {:cause ::self-inflicted})))))

Requiring the com.grzm.ex.assertions namespace will make the asser­tion avail­able for the test. And we want it to be avail­able in both Clojure and Clojure­Script, so we have the test in a cljc file and condi­tion­ally require both clojure.test and cljs.test.

The namespaced symbol com.grzm.ex.assertions/thrown-with-data? will dispatch to the custom asser­tion imple­men­ta­tion. Extending the assert-expr multi­method is global for the running instance, not just the file we require it in. Namespacing the symbol prevents us from stomping on (or being stomped on by) another asser­tion dispatched by the same unnamespaced symbol. Let’s play nice with others.

Clojure implementation

Let’s start out with the Clojure imple­men­ta­tion in a clj file.

;; file: src/com/grzm/ex/assertions.clj
(ns com.grzm.ex.assertions
  (:require
   [clojure.test :as test]))

(defmethod test/assert-expr 'com.grzm.ex.assertions/thrown-with-data?
  [msg [_ data-fn & body :as form]]
  ~(test/do-report
     (try
       ~@body
       ;; We expect body to throw. If we get here, it's a failure
       {:type     :fail
        :message  (str (when ~msg (str ~msg ": "))
                       "expected exception")
        :expected '~form
        :actual   nil}
       (catch clojure.lang.ExceptionInfo e#
         (let [d# (ex-data e#)]
           (if (~data-fn d#)
             {:type     :pass
              :message  ~msg
              :expected '~form
              :actual   e#}
             {:type     :fail
              :message   (str (when ~msg (str ~msg ": "))
                              "ex-data doesn't match")
              :expected '~data-fn
              :actual   d#}))))))

So, let’s see if it works. In the Cider repl, I run cider-test-run-ns-tests.

Test Summary
com.grzm.ex.assertions-test

Tested 1 namespaces
Ran 1 assertions, in 1 test functions
1 passed

Grammar aside, the result looks good. Let’s see how failure reporting works. We have three cases to consider:

  • body doesn’t throw
  • body throws ExceptionInfo, but the data doesn’t match
  • body throws something other than ExceptionInfo

We handle the first two explic­itly in the asser­tion. For the last, we won’t catch the excep­tion, so we rely on clojure.test’s default error handling. Let’s add tests for these cases.

(deftest test-thrown-with-data-no-throw
  (is (com.grzm.ex.assertions/thrown-with-data?
        #(= ::nope (:cause %))
        (constantly true))))

(deftest test-thrown-with-data-fail
  (is (com.grzm.ex.assertions/thrown-with-data?
        #(= ::nope (:cause %))
        (throw (ex-info "An error was thrown" {:cause ::self-inflicted})))))

(deftest test-thrown-with-data-non-ex-info
  (is (com.grzm.ex.assertions/thrown-with-data?
        #(= ::nope (:cause %))
        (throw (Exception. "I'm just a plain Exception")))))

And the test results:

Test Summary
com.grzm.ex.assertions-test

Tested 1 namespaces
Ran 4 assertions, in 4 test functions
2 failures
1 errors


Results

com.grzm.ex.assertions-test
1 non-passing tests:

Fail in test-thrown-with-data-fail
ex-data doesn't match
expected: (fn*
 [p1__14287#]
 (= :com.grzm.ex.assertions-test/nope (:cause p1__14287#)))
  actual: {:cause :com.grzm.ex.assertions-test/self-inflicted}

com.grzm.ex.assertions-test
1 non-passing tests:

Fail in test-thrown-with-data-no-throw
expected exception
expected: (com.grzm.ex.assertions/thrown-with-data?
 (fn*
  [p1__14278#]
  (= :com.grzm.ex.assertions-test/nope (:cause p1__14278#)))
 (constantly true))
  actual: nil

com.grzm.ex.assertions-test
1 non-passing tests:

Error in test-thrown-with-data-non-ex-info
expected: (com.grzm.ex.assertions/thrown-with-data?
 (fn*
  [p1__14296#]
  (= :com.grzm.ex.assertions-test/nope (:cause p1__14296#)))
 (throw (Exception. "I'm just a plain Exception")))
   error: java.lang.Exception: I'm just a plain Exception

Looks pretty good. We have different messages depending on the failure mode. We can see the anony­mous func­tions used to match the ExceptionInfo data. In test-thrown-with-data-fail we have the actual data. When we throw with some­thing other than ExceptionInfo, we get an error, not a failure.

ClojureScript implementation

This asser­tion won’t work for Clojure­Script, however. We need to extend cljs.test/assert-expr (which has three argu­ments) instead of clojure.test/assert-expr (which has two), as well as catch cljs.core/ExceptionInfo instead of clojure.java.ExceptionInfo. Another wrinkle is that is is a macro: the asser­tion needs to be avail­able to is, so we need to define the asser­tion in a clj file.

So, we need to compile a clj file that will be used only in the cljs envi­ron­ment. That can’t be the assertions.clj file, as we need that one for the Clojure target. So, we’ll make a sepa­rate clj file for the Clojure­Script target: assertions/cljs.clj.

;; file: src/com/grzm/ex/assertions/cljs.clj
(ns com.grzm.ex.assertions.cljs
  (:require
   [cljs.test :as test])) ;; *Not* clojure.test

(defmethod test/assert-expr 'com.grzm.ex.assertions/thrown-with-data?
  [_env msg [_ data-fn & body :as form]] ;; 3 arguments instead of 2
  `(test/do-report
     (try
       ~@body
       ;; We expect body to throw. If we get here, it's a failure
       {:type     :fail
        :message  (str (when ~msg (str ~msg ": "))
                       "expected exception")
        :expected '~form
        :actual   nil}
       (catch cljs.core/ExceptionInfo e# ;; not clojure.lang.ExceptionInfo
         (let [d# (ex-data e#)]
           (if (~data-fn d#)
             {:type     :pass
              :message  ~msg
              :expected '~form
              :actual   e#}
             {:type     :fail
              :message  (str (when ~msg (str ~msg ": "))
                             "ex-data doesn't match")
              :expected '~data-fn
              :actual   d#})))))

Note we’re requiring cljs.test, not clojure.test. Else­where, we’re relying on Clojure­Scrip­t/­Clo­jure namespace aliasing. Compi­la­tion here is in Clojure, and clojure.test/assert-expr is not the same as cljs.test/assert-expr. We’re extending cljs.test/assert-expr, so we need to refer to it explicitly.

And, as we want to use the same com.grzm.ex.assertions namespace to include the asser­tion, we’ll create a sepa­rate cljs file to require the Clojure­Script macros.

;; file: src/com/grzm/ex/assertions.cljs
(ns com.grzm.ex.assertions
  (:require-macros [com.grzm.ex.assertions.cljs]))

We also need to update the test where we throw an excep­tion: we’ll throw js/Error instead of java.lang.Exception.

(deftest test-thrown-with-data-non-ex-info
  (is (com.grzm.ex.assertions/thrown-with-data?
        #(= ::nope (:cause %))
        (throw #?(:clj (Exception. "I'm just a plain Exception")
                  :cljs (js/Error. "I'm just a plain Error"))))))

After setting up the a lein cljsbuild envi­ron­ment for testing with a Node.js runner, here are the test results:

λ lein clean && lein cljsbuild once
Compiling ClojureScript...
Compiling "target/cljs/node_dev/tests.js" from ["src" "test/src" "test/node/src"]...

Testing com.grzm.ex.assertions-test

FAIL in (test-thrown-with-data-no-throw) (cljs/test.js:443:85)
expected exception
expected: (com.grzm.ex.assertions/thrown-with-data? (fn* [p1__11833#] (= :com.grzm.ex.assertions-test/nope (:cause p1__11833#))) (constantly true))
  actual: nil

FAIL in (test-thrown-with-data-fail) (cljs/test.js:443:85)
ex-data doesn't match
expected: (fn* [p1__11837#] (= :com.grzm.ex.assertions-test/nope (:cause p1__11837#)))
  actual: {:cause :com.grzm.ex.assertions-test/self-inflicted}

ERROR in (test-thrown-with-data-non-ex-info) (Error:NaN:NaN)
expected: (com.grzm.ex.assertions/thrown-with-data? (fn* [p1__11841#] (= :com.grzm.ex.assertions-test/nope (:cause p1__11841#))) (throw (js/Error. "I'm just a plain Error")))
  actual: #object[Error Error: I'm just a plain Error]

Ran 4 tests containing 4 assertions.
2 failures, 1 errors.
Successfully compiled "target/cljs/node_dev/tests.js" in 3.412 seconds.

Looks very similar to the Clojure results.

We still have quite a bit of code dupli­ca­tion. Let’s DRY that up. We can combine assertions.clj and assertions.cljs into a single assertions.cljc file with reader conditionals.

;; file: src/com/grzm/ex/assertions.cljc
(ns com.grzm.ex.assertions
  #?(:cljs (:require-macros [com.grzm.ex.assertions.cljs]))
  #?(:clj (:require [clojure.test :as test])))

#?(:clj
   (defmethod test/assert-expr 'com.grzm.ex.assertions/thrown-with-data?
     [msg [_ data-fn & body :as form]]
     `(test/do-report
        (try
          ~@body
          ;; We expect body to throw. If we get here, it's a failure
          {:type     :fail
           :message  (str (when ~msg (str ~msg ": "))
                          "expected exception")
           :expected '~form
           :actual   nil}
          (catch clojure.lang.ExceptionInfo e#
            (let [d# (ex-data e#)]
              (if (~data-fn d#)
                {:type     :pass
                 :message  ~msg
                 :expected '~form
                 :actual   e#}
                {:type     :fail
                 :message  (str (when ~msg (str ~msg ": "))
                                "ex-data doesn't match")
                 :expected '~data-fn
                 :actual   d#})))))))

And we can extract the common logic from the assert-expr method imple­men­ta­tions. Note we need to pass in the class of excep­tion we’re catching as it’s different depending on the target.

;; file src/com/grzm/ex/assertions/impl.clj
(ns com.grzm.ex.assertions.impl)

(defn thrown-with-data?
  [msg [_ data-fn & body :as form] klass]
  `(test/report
     (try
       ~@body
       ;; We expect body to throw. If we get here, it's a failure
       {:type     :fail
        :message  (str (when ~msg (str ~msg ": "))
                       "expected exception")
        :expected '~form
        :actual   nil}
       (catch ~klass e#
         (let [d# (ex-data e#)]
           (if (~data-fn d#)
             {:type     :pass
              :message  ~msg
              :expected '~form
              :actual   e#}
             {:type     :fail
              :message  (str (when ~msg (str ~msg ": "))
                             "ex-data doesn't match")
              :expected '~data-fn
              :actual   d#}))))))

And that allows us to clean up the assert-expr method implementations.

;; file: src/com/grzm/ex/assertions.cljc
(ns com.grzm.ex.assertions
  #?(:cljs (:require-macros [com.grzm.ex.assertions.cljs]))
  #?(:clj (:require
           [clojure.test :as test]
           [com.grzm.ex.assertions.impl :as impl])))

#?(:clj
   (defmethod test/assert-expr 'com.grzm.ex.assertions/thrown-with-data?
     [msg [_ data-fn & body :as form]]
     `(test/do-report
        ~(impl/thrown-with-data? msg form 'clojure.lang.ExceptionInfo))))
;; file src/com/grzm/ex/assertions/cljs.clj
(ns com.grzm.ex.assertions.cljs
  (:require
   [cljs.test :as test]
   [com.grzm.ex.assertions.impl :as impl]))

(defmethod test/assert-expr 'com.grzm.ex.assertions/thrown-with-data?
  [_env msg form] ;; has 3 arguments instead of 2
  `(test/do-report
     ~(impl/thrown-with-data? msg form 'cljs.core/ExceptionInfo)))

So, we have three files (excluding tests):

  • src/com/grzm/ex.assertions.cljc - namespace which loads the assertion
  • src/com/grzm/ex/assertions/cljs.clj - ClojureScript-target macros
  • src/com/grzm/ex/assertions/impl.clj - common Clojure/ClojureScript code

We have custom reporting depending on the failure case which improves on the clojure.test default behav­ior, and we can use the asser­tion in both Clojure and ClojureScript.

Self-hosted ClojureScript

But what about self­-hosted Clojure­Script? We need all three of these files to load in a Clojure­Scrip­t-only envi­ron­ment. We need cljs.clj and impl.clj to be cljc files so they’re loaded during self­-hosted (i.e., Clojure­Script) compi­la­tion. For impl.clj, it’s simply a name change. But the asser­tion imple­men­ta­tion is a bit more involved.

Compiling the Clojure­Script target with the JVM requires a clj file. This won’t be loaded when self­-hosted. We need to imple­ment the asser­tion in Clojure­Script during Clojure­Script compi­la­tion but only when self-hosted.

How can we deter­mine the compi­la­tion envi­ron­ment? One way is to detect the exis­tence of objects that exist only in the self­-hosted envi­ron­ment. One of those is js/cljs.test$macros. And we do this in the com.grzm.ex.assertions namespace. This is not support for the Clojure­Script target in Clojure: it’s Clojure­Script we’re loading during self­-hosted Clojure­Script compilation.

;; src/com/grzm/ex/assertions.cljc
(ns com.grzm.ex.assertions
  #?(:cljs (:require-macros [com.grzm.ex.assertions.cljs]))
  (:require
   [clojure.test :as test]
   [com.grzm.ex.assertions.impl :as impl]))

#?(:clj
   (defmethod test/assert-expr 'com.grzm.ex.assertions/thrown-with-data?
     [msg [_ data-fn & body :as form]]
     `(test/do-report
        ~(impl/thrown-with-data? msg form 'clojure.lang.ExceptionInfo)))
   :cljs
   (when (exists? js/cljs.test$macros)
     (defmethod js/cljs.test$macros.assert_expr 'com.grzm.ex.assertions/thrown-with-data?
       [_env msg form]
       `(test/do-report
          ~(impl/thrown-with-data? msg form 'cljs.core/ExceptionInfo)))))

The com.grzm.ex.assertions.cljs namespace is still required during Clojure­Script compi­la­tion, so we need to make that avail­able as a cljc file. We don’t want the asser­tion imple­men­ta­tion however, so we wrap that in a reader conditional.

;; src/com/grzm/ex/assertions/cljs.cljc
(ns com.grzm.ex.assertions.cljs
  (:require
   [cljs.test :as test]
   [com.grzm.ex.assertions.impl :as impl]))

#?(:clj
   (defmethod test/assert-expr 'com.grzm.ex.assertions/thrown-with-data?
     [_env msg form] ;; has 3 arguments instead of 2
     `(test/do-report
        ~(impl/thrown-with-data? msg form 'cljs.core/ExceptionInfo))))

After setting up the appro­priate self­-hosted test envi­ron­ment, let’s see how it works.

λ scripts/test-self-host 

Testing com.grzm.ex.assertions-test

FAIL in (test-thrown-with-data-no-throw) (at .js:431:14)
expected exception
expected: (com.grzm.ex.assertions/thrown-with-data? (fn* [p1__17#] (= :com.grzm.ex.assertions-test/nope (:cause p1__17#))) (constantly true))
  actual: nil

FAIL in (test-thrown-with-data-fail) (at .js:431:14)
ex-data doesn't match
expected: (fn* [p1__18#] (= :com.grzm.ex.assertions-test/nope (:cause p1__18#)))
  actual: {:cause :com.grzm.ex.assertions-test/self-inflicted}

ERROR in (test-thrown-with-data-non-ex-info) (Error:NaN:NaN)
expected: (com.grzm.ex.assertions/thrown-with-data? (fn* [p1__19#] (= :com.grzm.ex.assertions-test/nope (:cause p1__19#))) (throw (js/Error. "I'm just a plain Error")))
  actual: #object[Error Error: I'm just a plain Error]

Ran 4 tests containing 4 assertions.
2 failures, 1 errors.

Success! (It’s not often you say that when the output ends with “2 fail­ures, 1 errors”.) And a quick check of our Clojure and JVM-­Clo­jure­Script tests confirms they still work as expected.

Wrapping up

I’m a fan of testing and am always looking for ways to write more expres­sive tests and improve test report­ing. This lead me to test.check and writ­ing, with Gary Fred­er­icks’ guid­ance, a custom assertion to improve reporting in both Clojure and Clojure­Script. Mike Fikes added support for self­-hosted ClojureScript.

Here I’ve applied that same pattern to write another asser­tion. You can find the completed code along with the supporting test envi­ron­ments on Github. (Thanks to Mike Fikes again for providing an example of an envi­ron­ment for self­-hosted Clojure­Script.) I’ve also collected a small set of custom asser­tions I use in the Tespresso library (which I’m in the process of porting to Clojure­Script). I hope this provides an self­-­con­tained example that encour­ages others to write asser­tions they find useful.

Thanks

Thank you to Mike Fikes for reviewing this post and reminding me about Clojure­Scrip­t/­Clo­jure namespace alias­ing. You can read more about that on his blog.