TL;DR: threading all the things

Update: 1Sep15

Whilst writing I discovered this post has not been updated after I’d received some feedback from Shaun Parker. No idea what happened. Sorry Shaun! You might want to jump straight to the “Update 1Sep15” section, and have a look at my some-as→ macro.

Introduction

The other day I was writing a Clojure let block to transform a map. It was a pretty usual Clojure pipeline of functions, a use case Clojure excels at. The pipeline included a cond, a couple of maps, some of my own functions, and finally an assoc and dissoc to “update” the input map with the result of the pipeline and delete some redundant keys.

Even though Clojure syntax is quite spare there was quite a bit of inevitable clutter in the code and it struck me the code would be cleaner and clearer if I could use the thread first (→) macro.

If you grok macros you can probably guess the rest of this post (likely you will have seen the title and probably said to yourself “Oh yeah, that’s obvious, nothing to see here” and moved along ☺ )

Threading Macros

Threading Macros - “thread first” and “thread last”

Clojure core has a family of threading macros including the “thread first” ones of →, some→, as→, and cond→, and their equivalent thread last (-») ones.

I’m not going to explain the threading macros in depth as these have been well covered already - see for example this very nice post by Debasish Ghosh (btw Debasish’s book DSLs in Action is worth your money).

Simply put: the threading macros allow a pipeline of functions to be written in visually clean and clear way — pre-empting the need to write a perhaps deep inside-to-outside functional form — by weaving the result of the previous function into the current function as either the first (“thread first”) or last (“thread last”) argument.

The example below of using “thread last” to sum the balances of a bank’s savings account has been taken from Debasish’s post . I have reworked his example slightly and added some narrative comments to make it completely clear what is going on:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31 ;; Debasish's example slighty reworked
;; For original see http://debasishg.blogspot.co.uk/2010/04/thrush-in-clojure.html

(def all-accounts
  [{:no 101 :name "debasish" :type 'savings :balance 100}
   {:no 102 :name "john p." :type 'checking :balance 200}
   {:no 103 :name "me" :type 'checking :balance -500}
   {:no 104 :name "you" :type 'savings :balance 750}])

(def savings-accounts-balance-sum-using-thread-last
  (
   ;; use the thread-last macro
   ->>

   ;; ... and start from the collection of all accounts
   all-accounts

   ;; ... select only the savings accounts
   (filter #(= (:type %) 'savings))

   ;; ... get the balances from all the saving accounts
   (map :balance)

   ;; ... and add up all their balances
   (apply +)))

(doall (println "savings-accounts-balance-sum-using-thread-last"
                savings-accounts-balance-sum-using-thread-last))

;; check the answer
(assert (= 850 savings-accounts-balance-sum-using-thread-last))

Threading Macros - what’s not to like?

Nothing really although there are limitations of course. For example to thread a map needs “thread last” and assoc requires “thread first”, but you can’t mix first and last together directly.

Although “thread first” and “thread last” cover a wide range of use cases, there are times where you have to go through hoops to incorporate code that requires the current value of the pipeline as other than the first or last argument, or maybe need to use the value multiple times in multiple subforms.

Threading Macros - using a partial

There are ways around the limitations of course and one way is to use partial with “thread first” to supply the argument as the last argument to the function.

This horrid example using “thread first” with lots of partials is bonkers but does demonstrate the point:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24 ;; Using partials with thread-first

(def savings-accounts-balance-sum-using-thread-first
  (
   ;; use the thread-first macro
   ->

   ;; ... and start from the collection of all accounts
   all-accounts

   ;; ... select only the savings accounts
   ((partial filter #(= (:type %) 'savings)))

   ;; ... get the balances from all the saving accounts
   ((partial map :balance))

   ;; ... and add up all their balances
   ((partial apply +))))

(doall (println "savings-accounts-balance-sum-using-thread-first" 
                savings-accounts-balance-sum-using-thread-first))

;; check the answer
(assert (= 850 savings-accounts-balance-sum-using-thread-first))

Note each partial call is the first (and only) form inside another form; the “thread first” macro will weave the input to the partial as the second value in the outer form. (Else the macro would weave the previous result into the partial declaration itself.)

Threading Macros - using an in-line function

More generally, you can always escape the confines of the first or last constraint by using an in-line function.

The following example sums the balances of all the checking accounts in deficit, applying 10% interest, to find the total owed to the bank. It uses an in-line function to apply the interest.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41 ;; Calculate the total balance of all the checking accounts in deficit

;; applies interest to any in deficit

(def deficit-accounts-balance-sum-using-interest-function
  (
   ;; use the thread-last macro
   ->>

   ;; ... and start from the collection of all accounts
   all-accounts

   ;; ... select only the checking accounts
   (filter #(= (:type %) 'checking))

   ;; ... select the accounts in deficit
   (filter #(> 0 (:balance %)))

   ;; add 10% interest to any in deficit
   ;; interest rate is first argument; second (last) is the deficit accounts
   ((fn [interest-rate deficit-accounts]
      (map
       (fn [deficit-account]
         (let [balance (:balance deficit-account)
               interest (* interest-rate balance)]
           (assoc deficit-account :balance (+ balance interest))))
       deficit-accounts))
       ;; interest rate is 10%
       0.1)

   ;; ... get the balances from all the deficit accounts
   (map :balance)

   ;; ... and add up all their balances to get net balance
   (apply +)))

(doall (println "deficit-accounts-balance-sum-using-interest-function" 
                deficit-accounts-balance-sum-using-interest-function))

;; check the answer
(assert (= -550.0 deficit-accounts-balance-sum-using-interest-function))

Note the in-line function declaration is the first form inside another form (for the same reason as the partials above were).

Threading Macros - capturing the result of the previous step

In the above examples the steps were calls to core functions: filter, map and apply.

But the step can be a call to a macro and the macro will be passed the current value of the form being evaluated by the threading macro.

In the example below, a simple macro show-the-argument will print the current evaluated form and return it to continue the evaluation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39 ;; using a simple macro to show what's passed to each step in the pipeline

(defmacro show-the-argument
  [argument]
  (doall (println argument))
  `(do
     ~argument))

(def savings-accounts-balance-sum-and-show-the-argument
  (
   ;; use the thread-last macro
   ->>

   ;; ... and start from the collection of all accounts
   all-accounts

   ;; show the argument
   (show-the-argument)

   ;; ... select only the savings accounts
   (filter #(= (:type %) 'savings))

   ;; show the argument
   (show-the-argument)

   ;; ... get the balances from all the saving accounts
   (map :balance)

   ;; show the argument
   (show-the-argument)

   ;; ... and add up all their balances
   (apply +)))

(doall (println "savings-accounts-balance-sum-and-show-the-argument" 
                savings-accounts-balance-sum-and-show-the-argument))

;; check the answer
(assert (= 850 savings-accounts-balance-sum-and-show-the-argument))

If you look at the prints, you’ll see something like the below for the output for the post filter call to show-the-argument (I’ve reformatted to aid clarity):

1
2
3 (filter
 (fn* [p1__1419#] (= (:type p1__1419#) (quote savings)))
 (show-the-argument all-accounts))

You can see how “thread last” has woven the previous call to show-the-argument (after all-accounts) into the filter form. The calls to show-the-argument will be evaluated after “thread last” has presented its evaluated form for compilation.

Threading Macros - “thread-first-let”

show-the-argument demonstrates how simple it is in a macro to grab hold of the current form being evaluated and do something with it.

Let’s do that then. The macro “thread-first-let” below takes the argument together with some body forms, and returns a new form that assigns the argument to a let gensym local called x# and evaluates the body forms in the context of the let so the body forms can use x# anywhere needed.

The macro includes a println to shows the form returned by “thread-first-let” to the compiler:

1
2
3
4
5
6
7
8
9
10 ;; Using the "thread-first-let" macro create a let and assigns the current form to x#
;; and evaluates the body in the let context, with x# available in the body

(defmacro thread-first-let
  [argument & body]
  (let [let-form# `(let [~'x# ~argument]
                     (~@body))]
    (doall (println let-form#))
    `(do
       ~let-form#)))

Let’s reprise to the horrid example above where I used partials with “thread first” and use “there-first-let” instead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24 ;; Using "thread-first-let" inside a "thread-first"

(def savings-accounts-balance-sum-using-thread-first-let
  (
   ;; use the thread-first macro
   ->

   ;; ... and start from the collection of all accounts
   all-accounts

   ;; ... select only the savings accounts
   (thread-first-let filter #(= (:type %) 'savings) x#)

   ;; ... get the balances from all the saving accounts
   (thread-first-let map :balance x#)

   ;; ... and add up all their balances
   (thread-first-let apply + x#)))

(doall (println "savings-accounts-balance-sum-using-thread-first-let"
                savings-accounts-balance-sum-using-thread-first-let))

;; check the answer
(assert (= 850 savings-accounts-balance-sum-using-thread-first-let))

One of the prints from “thread-first-let” shows the final form of the filter:

1
2 (clojure.core/let [x# all-accounts]
  (filter (fn* [p1__1431#] (= (:type p1__1431#) (quote savings))) x#))

The takeaway here is that after the thread-first-let the code is exactly the same as you would write outside of a core threading macro, and you have the freedom to use the argument (x#) wherever and whenever you need, not just once and / or “last”.

Final Words

“thread-first-let” is a very simple seven line macro allowing arbitrary code to participate in the core threading macros making the whole pipeline as clean as clear as possible.

It not hard to see how this simple idea could taken forward to define macros to support “thread-last”, or even “packaged” macros such as thread-first-map.

The more general point is how even a trivial use of macros makes for a welcome improvement in keeping the code clean and clear, and really does bring home how tractable and malleable macros make Clojure.

Update 1Sep15

Feedback from Shawn Parker on the as→ macro

Shaun Parker was kind enough to email me and recommend I take a closer look at the as→ macro. I must admit I was not too familiar with that member of the threading family; its true to say that “thread-first-let” is a derivative form of as→.

Shaun provided a couple of more idiomatic reworkings of the accounts example using as→ where the name used is $.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29 ;; Shaun's examples of using the as-> macro

(def all-accounts
  [{:no 101 :name "debasish" :type :savings :balance 100}
   {:no 102 :name "john p." :type :checking :balance 200}
   {:no 103 :name "me" :type :checking :balance -500}
   {:no 104 :name "you" :type :savings :balance 750}])

(def shaun_as1 (as-> all-accounts $
       (filter #(= (:type %) :savings) $)
       (map :balance $)
       (reduce + $)))

(doall (println "shaun_as1" shaun_as1))

;; check the answer
(assert (= 850 shaun_as1))

;; example with adding another account, showing mixed threading
(def shaun_as2 (as-> all-accounts $
                 (conj $ {:no 105 :name "shaun" :type :savings :balance 100000.42})
                 (filter #(= (:type %) :savings) $)
                 (map :balance $)
                 (reduce + $)))

(doall (println "shaun_as2" shaun_as2))

;; check the answer
(assert (= 100850.42 shaun_as2))

The Core some→ macro

On of the reasons I hadn’t used as→ that much was because my “go to” threading macro has been usually some→.

The latter is very similar to the normal “thread-first” (→) macro but additionally the pipeline is aborted if the result of the previous step is nil; early termination. (some→ compiles each step in the pipeline to an if statement)

If you look at the code (1.7.0) for some→ is a awesome tribute to recursion, conciseness and clarity, I keep thinking “where’s all the code?”

1
2
3
4
5
6
7
8
9
10 (defmacro some->
  "When expr is not nil, threads it into the first form (via ->),
  and when that result is not nil, through the next etc"
  {:added "1.5"}
  [expr & forms]
  (let [g (gensym)
        pstep (fn [step] `(if (nil? ~g) nil (-> ~g ~step)))]
    `(let [~g ~expr
           ~@(interleave (repeat g) (map pstep forms))]
       ~g)))

The Core as→ macro

The as→ macro is similarly elegant:

1
2
3
4
5
6
7
8
9 (defmacro as->
  "Binds name to expr, evaluates the first form in the lexical context
  of that binding, then binds name to that result, repeating for each
  successive form, returning the result of the last form."
  {:added "1.5"}
  [expr name & forms]
  `(let [~name ~expr
         ~@(interleave (repeat name) forms)]
     ~name))

My attempt at a some-as→ macro

I got to thinking if would be good to have the flexibility of as→ as to how / where the current value will be used in the current step, together with the early termination behaviour of some→.

Here is what I came up with:

1
2
3
4
5
6
7
8
9 (defmacro some-as->
  "Conflation of some-> and as->"
  [expr name & forms]
  (let [pstep (fn [step]
                `(let [step-value# (~'if (~'nil? ~name) nil ~step)]
                   step-value#))]
    `(let [~name ~expr
           ~@(interleave (repeat name) (map pstep forms))]
       ~name)))

Let try Shaun’s examples with some-as→ just to check nothing has gone wrong; early termination wont be invoked here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23 ;; Shaauns' example using some-as->

(def shaun_as1a (some-as-> all-accounts $
       (filter #(= (:type %) :savings) $)
       (map :balance $)
       (reduce + $)))

(doall (println "shaun_as1a" shaun_as1a))

;; check the answer
(assert (= 850 shaun_as1a))

;; example with adding another account, showing mixed threading
(def shaun_as2a (some-as-> all-accounts $
                 (conj $ {:no 105 :name "shaun" :type :savings :balance 100000.42})
                 (filter #(= (:type %) :savings) $)
                 (map :balance $)
                 (reduce + $)))

(doall (println "shaun_as2a" shaun_as2a))

;; check the answer
(assert (= 100850.42 shaun_as2a))

Ok, an (contrived) example to demonstrate early termination being invoked:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25 ;; Example of using some-as->

;; Without the early termination feature of some-> this code would stack
;; trace as symbol fails when given a nil value

(def some_as_ex1 (some-as-> all-accounts $

                   ;; find accounts with my first name - there are none!
                   (filter #(= (:name %) "ian") $)

                   ;; taking the first of empty collection will return nil
                   (first $)

                   ;; extract my name
                   ;; but this step and subsequent ones will not be executed
                   (:name $) 

                   ;; create a symbol of my name
                   ;; symbol will stack trace if given a nil value
                   (symbol $)))

(doall (println "some_as_ex1" some_as_ex1))

;; check the answer
(assert (= nil some_as_ex1))

An Ostler in IT

Horses for Courses

Some syntactic sugar for Clojure's threading macros