When I first started learning Elixir, I didn’t begin to appreciate the power of pattern matching until I’d read Sasa’s book and Dave’s book. Chapter 5 in Dave’s book had some really good prose explaining pattern matching the arguments of anonymous and (Chapter 6) named functions.

But it took me a while to assimilate what the Elixir (and Erlang) books were telling me, so I started writing my Elixir functions rather ignorant of how I could better structure them.

But as ignorance was displaced by experience with Elixir, I started to use pattern matching for functions all over the place, once I’d “seen” that many of my functions were variations (variants) on the same theme, really families of functions with the same purpose.

After a while of using bread’n’butter function matching, I found I’d developed a style of using pattern matching to create a DSL with its own set of verbs.

Then I’d write dsl programs using the verbs. Coffee anyone?

My very own Drink Maker

I’m using the example a (virtual) Drink Maker. Its not the real application that took me on my “unexpected journey”, but it demonstrates what I want to show in a (I hope) simple way.

Have a look at the CoffeeMaker drink dsl module below which has a number of variants of the same function make_drink.

make_drink is passed a state (a map), a dsl verb (an atom e.g. boil_water) and an optional keyword list of arguments (opts). All the make_drink variants have the same signature and return the (usually changed) state. A familiar enough (recursive) pattern and pretty simple stuff, but highly effective.

make_drink pattern matches on the verb (but could also use the contents of the state).

The variant make_instant_coffee runs an `Enum.reduce/3` with another set of verbs specific to that recipe. The last thing it does is set the :ready_drink key in the state to :instant_coffee – I’ll use this value in the testing later on.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27 defmodule CoffeeMaker do
  # header
  def make_drink(state, verb, opts \\ [])
  def make_drink(state, :boil_water, _opts) do
    # blah
    state
  end
  def make_drink(state, :wash_cup, _opts) do
    # blah
    state
  end
  def make_drink(state, :add_instant_coffee_to_cup, _opts) do
    # blah
    state
  end
  # more steps (verbs) in the recipe 

  # this variant recursively calls make_drink
  def make_drink(state, :make_instant_coffee, opts) do
    [:boil_water, :wash_cup, :add_instant_coffee_to_cup,
       # more verbs here
    ]
    |> Enum.reduce(state, fn verb, state -> make_drink(state, verb, opts) end)
    # mark the drink ready
    |> Map.put(:ready_drink, :instant_coffee)
  end
end

Lets have coffee!

1
2 # make an instant coffee
%{} |> CoffeeMaker.make_drink(:make_instant_coffee)

Yes I know real developers supposedly don’t drink instant coffee but, hey, we’re a broad church :)

Diaster Strikes!

The dsl style worked really well and I created other dsl programs to make different types of coffee (hot, iced, espresso, latte, etc).

Then disaster struck. Already, at the back of my mind, I’d been a bit worried at the growing line count of the CoffeeMaker module; it was becoming unmanageable.

But the clincher came when I realised I wanted to be able to use my drink dsl in other applications (e.g. a TeaMaker) and enable those applications to make new drinks, adding their own recipes (make_drink) for tea, juice, beers, cocktails, whatever. And also maybe selectively modify existing verbs in the base recipes (e.g there lots of different ways to boil_water).

I wanted to allow any drink dsl module to be able to use normal pattern matching in its own code, but also call the verbs in other drink dsl modules, something like this where TeaMaker needs to use some verbs from CoffeeMaker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 defmodule TeaMaker do
  # header
  def make_drink(state, verb, opts \\ [])
  def make_drink(state, :add_earl_grey_tea_bag, _opts) do
    # blah
    state
  end
  def make_drink(state, :brew_for_3_minutes, _opts) do
    # blah
    state
  end

  def make_drink(state, :make_earl_grey_tea, opts) do
    [:boil_water, :wash_cup, :add_earl_grey_tea_bag, :brew_for_3_minutes,
       # more verbs here
    ]
    |> Enum.reduce(state, fn verb, state -> make_drink(state, verb, opts) end)
    # mark the drink ready
    |> Map.put(:ready_drink, :earl_grey_tea)
  end
end

Tea time!

1
2 # make a cup of earl grey tea
%{} |> TeaMaker.make_drink(:make_earl_grey_tea)

Fail! This wont work of course because pattern matching on functions is scoped to the module and TeaMaker has no access to CoffeeMaker’s make_drink so boil_water and wash_cup will fail to pattern match.

Thinking how to get around this, I considered making the list of steps in the recipe (partial) functions rather than verbs, but it starts tightly coupling verbs to modules, and visually looks a bit of a mess and difficult to parse by eye. In short: yuck!

1
2
3
4
5
6
7
8
9 def make_drink(state, :make_earl_grey_tea, opts) do
  [
    &(CoffeeMaker.make_drink(&1, :boil_water)),
    &(CoffeeMaker.make_drink(&1, :wash_cup)),
    &(TeaMaker.make_drink(&1, :add_earl_grey_tea_bag)),
    # more funs here
  ]
  |> Enum.reduce(%{}, fn fun, state -> fun.(state) end)
end

Or I could go back to having unique, explicit functions I could call from any where but that felt horrible too.

1
2
3
4 def make_drink_boil_water(state, opts) do
  # blah
  state
end

I also thought about structuring the dsl into a hierarchical form such as a module for the different ways of boiling water, one for cleaning the cups, and so on.

But everthing I thought of introduced constraints, confusion and complexity into what was currently quite a simple, easy to understand and elegant solution.

Creating a Drinks Menu

To maintain the simplicity of an apparently “global” make_drink function, I needed to collect all the make_drink variants with different verbs from CoffeeMaker, TeaMaker, and any other drink dsl module, into a combined drinks_menu, and use the combined drinks_menu when I need to find the make_drink variant for a particular verb (e.g. boil_water).

Somehow I needed to

identify the various make_drink variants from each drink dsl module
save their details in their own drinks_menu
combine the modules’ drinks_menus
make the combined drinks_menu accessible by all the modules

But I didn’t want to have to have write the make_drink variants in radically different ways to be usable either within the module or from another module. (“Pro cake and pro eating it”)

Building a module’s drinks_menu

By “save the details” I mean: the module the make_drink variant was defined in, its verb and any other arguments.

I decided to save the details using the MFA (module, function, arguments list) format that you can use with `Kernel.apply/3` to call the function.

For example, the MFA 3tuple for the boil_water variant in CofffeeMaker would be

1 {CoffeeMaker, :make_drink, [state, :boil_water, opts]}

The state and opts are variable so I’ve abbreviated the stored MFA to hold only the “constant” elements like the verb, and leave the variable arguments to be supplied at the time of use (see later).

1 {CoffeeMaker, :make_drink, [:boil_water]}

All of the make_drink variants for a module can then be held in a module-specific drinks_menu (a map), keyed on the verb e.g. for CofffeeMaker

1
2
3
4
5 drinks_menu =
  %{boil_water: {CoffeeMaker, :make_drink, [:boil_water]},
    wash_cup: {CoffeeMaker, :make_drink, [:wash_cup]},
    add_instant_coffee_to_cup: {CoffeeMaker, :make_drink, [:add_instant_coffee_to_cup]},
    make_instant_coffee: {CoffeeMaker, :make_drink, [:make_instant_coffee]}}

Its worth stressing the drinks menu is just a plain map you can get, put, etc the verbs, and the values can be whatever you like; interpreting the values is a job for some other code (I’ll come on to that).

Also the value is not necessarily dependent (coupled) in any way on the verb itself, and one could have other non-MFA values. One obvious non-MFA value would be an anonymous function:

1
2
3
4
5
6
7
8 add_two_sugars:  fn state ->
                      # do something with state
                      state

                    state, opts ->
                      # do something with state & opts
                      state
                 end

Saving the module’s drinks_menu in a persistent module attribute

The module’s drinks_menu is built at compile time but needs to be available at run time.

I bumped into how to do this while reading Chapter 2 of Chris’s metaprogramming book but the :persist option of `Module.register_attribute/3` is in the reference documentation.

Simply, if you register a module attribute, e.g. @drinks_menu, with the persist: true option,

1
2
3
4
5
6
7
8
9
10 defmodule CoffeeMaker do

  # register the drinks_menu attribute with the persistent option
  Module.register_attribute(__MODULE__, :drinks_menu, persist: true)

  # initialize the drinks_menu
  Module.put_attribute(__MODULE__, :drinks_menu, %{})

  # all the /make_drink/ variants, etc here
end

… you can get the value of drinks_menu at run time using the module’s __info__ function called with :attributes

1
2
3
4
5
6
7
8 # get module's e.g. CoffeeMaker drinks_menu at run time
def get_drinks_menu(module) do
  :attributes
  |> module.__info__
  |> Keyword.fetch!(:drinks_menu)
  # persist always seems to make the attribute a List of one item
  |> List.first
end

Collecting the details of the make_drink variants

I’ve talked about how I’d save the details of the make_drink variants in the drinks_menu but not how I’d find out what they are in the first place.

I needed to be able to run some of my code when a new make_drink variant was defined so I could save its details in the drinks_menu.

I’d already done this sort of thing in Ruby (at run-time of course) using Module’s #method_added.

Well it turns out that Elixir has the same (compile-time) functionality: Module’s compiler callback @on_definition registers the callback function (e.g. make_drink_on_def_callback in the example below) the compiler will call when a variant is defined.

The @on_definition callback function

The callback is passed all the information you could wish for in a comprehensive list of six arguments:

ArgName	Contents
env	the module environment - think __ENV__
kind	one of :def, :defp, :defmacro, or :defmacrop
name	the function's name
args	quoted arguments
guards	quoted guard clauses
body	quoted body

The callback will be called for all functions, not just make_drink, but simple pattern matching on the name (i.e. make_drink) allows the selection of wanted variants.

The args value will be a list:

1 [{:state, [line: 12], nil}, :boil_water, {:opts, [line: 12], nil}]

Remember these are quoted so the state and opts arguments are in `Macro.var/2` format. The body is similarly quoted.

Here is the callback’s code (note the return value is ignored).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19 # add new make_drink variants to the drinks_menu
def make_drink_on_def_callback(env, :def, :make_drink, args, _guards, _body) do

  # the calling module e.g. CoffeeMaker is the env's module key
  env_module = env.module

  # the verb is the second item in the args
  verb = args |> Enum.at(1)

  # update the current drinks_menu with new entry
  drinks_menu = env_module
  |> Module.get_attribute(:drinks_menu)
  # add the new verb + MFA 3tuple
  |> Map.put(verb, {env_module, :make_drink, [verb]})

  # save the update drinks_menu in the persistent attribute
  env_module |> Module.put_attribute(:drinks_menu, drinks_menu)

end

Registering the @on_definition callback

To register the callback for the module is quite simple:

1 @on_definition {DrinkMaker.Menu, :make_drink_on_def_callback}

Pulling all the drinks_menu code together

Since all the drink dsl modules will need to register the @on_definition callback, create the drinks_menu, etc, its more useful to put all drinks_menu code, including the callback, in a utility module: DrinkMaker.Menu. (I’ve not shown the complete module here as much has already been shown above.)

Generating a module’s drinks_menu

Generating the drinks_menu in any drink dsl module is then just a matter of use-ing DrinkMaker.Menu before the make_drink variants are defined e.g.

1
2
3
4
5
6
7
8 defmodule CoffeeMaker do

  # build the drinks_menu
  use DrinkMaker.Menu

  # CoffeeMaker's make_drink variants, etc go here
  # (not shown for brevity)
end

Creating a combined drinks_menu

Creating a combined drinks_menu is just a matter of unifying the individual drinks_menus into one.

The drinks_menu is really a dictionaries of transformations that can be run in an e.g. `Enum.reduce/3` pipeline (i.e one or more verbs) against a supplied state

Since the drinks menus are maps, they could be merged using `Map.merge/2`. Alternatively, one could e.g. take or drop verbs as needed and combine them that way.

How the combination is done depends on the application’s need and there is nothing to prevent the creation of different menus for different purposes e.g. hot_drinks_menu, cold_drinks_menu, free_drinks_menu, paid_drinks_menu, whatever.

Here the drinks_menus are just merged:

1
2
3
4
5
6
7
8
9
10 # need to require all the drinks dsl module to ensure the individual
# modules' drinks_menus have been created
require CoffeeMaker
require TeaMaker
# require any other drink dsl modules

# create drinks_menu as a module attribute
@drinks_menu [CoffeeMaker, TeaMaker]
|> Stream.map(fn module -> module |> DrinkMaker.Menu.get_drinks_menu end)
|> Enum.reduce(%{}, fn m, s -> s |> Map.merge(m) end)

Using a combined drinks_menu

To make drinks from a combined drinks_menu needs a helper function — defined for convenience in the DrinkMaker.Menu and also, but not necessarily, called make_drink — to find the verbs and call (usually) the right drink dsl module’s make_drink variant.

Note, the helper expects to find the drinks_menu in the state.

Its worth stressing the helper doesn’t “know” about the make_drink function and its variants; it only uses the values in the drinks_menu i.e the MFA 3tuple (usually) where the function to call in the 2nd element.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37 def make_drink(state, verbs, opts \\ []) do

  # the drinks_menu must be in the state
  drinks_menu = state |> Map.fetch!(:drinks_menu)

  # reduce the state using the values of the verbs
  # in the drinks_menu
  verbs
  |> List.wrap
  |> Enum.reduce(state,
  fn verb, state ->

    # must find the verb in the drinks_menu
    case drinks_menu |> Map.fetch!(verb) do

      # is the verb's value a MFA 3tuple?
      # note: the fun_name will always be make_drink in this example
      # but the code works for any fun_name
      {module, fun_name, mfa_args} ->

        # create the complete arguments list
        all_args = [state] ++ mfa_args ++ [opts]

        # run the function
        apply(module, fun_name, all_args)

      # is the verb's value a fun?
      fun when is_function(fun, 1) -> fun.(state)
      fun when is_function(fun, 2) -> fun.(state, opts)

      # no default => error

    end

  end)

end

To see how the helper would be used, let briefly revisit TeaMaker’s make_earl_grey_tea variant, the one above that failed because it needed to call verbs in CoffeeMaker. Assuming the state passed to TeaMaker’s make_earl_grey_tea now holds the drinks_menu, the variant would be rewritten to use `DrinkMaker.Menu.make_drink/3` (as would any other variant that needed to call make_drink variants in other drink dsl modules).

1
2
3
4
5
6
7
8
9
10
11 # this is the TeaMaker variant rewritten to use the DrinkMaker.Menu helper
def make_drink(state, :make_earl_grey_tea, opts) do
  verbs = [:boil_water, :wash_cup, 
           :add_earl_grey_tea_bag, :brew_for_3_minutes,
           # more verbs here
          ]
  state 
  |> DrinkMaker.Menu.make_drink(verbs, opts)
  # mark the drink ready
  |> Map.put(:ready_drink, :earl_grey_tea)
end

The state could hold many drinks menus and the decision on which one to use taken by the variant, the helper, whatever and based on any criteria.

The DrinkMaker GenServer

To show a testable example, I’ve created a GenServer to make drinks, holding the drinks_menu in the GenServer’s state.

(I want to have a nice easy Map interface to the drinks_menu so I’ll use my Hex package Amlapio)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64 defmodule DrinkMaker do

  use GenServer

  # need to require all the drinks dsl module to ensure the individual
  # modules' drinks_menus have been created
  require CoffeeMaker
  require TeaMaker
  # require any other drink dsl modules

  # create drinks_menu as a module attribute
  @drinks_menu [CoffeeMaker, TeaMaker]
  |> Stream.map(fn module -> module |> DrinkMaker.Menu.get_drinks_menu end)
  |> Enum.reduce(%{}, fn m, s -> s |> Map.merge(m) end)

  # start the GenServer
  def start_link(state \\ %{}) do
    GenServer.start_link(__MODULE__, state)
  end

  # use Amlapio to generate the accessors and mutators for the drinks_menu and state
  use Amlapio, funs: [:get, :put], genserver_api: [:drinks_menu]
  use Amlapio, funs: [:get, :put], genserver_api: nil,
    namer: fn _map_name, fun_name -> 
    ["drink_maker_state_", to_string(fun_name)] |> Enum.join |> String.to_atom
  end

  # << more API calls >>

  def make_drink(pid, verbs, opts \\ []) do
    GenServer.call(pid, {:make_drink, verbs, opts})
  end

  # GenServer Callbacks

  def init(state \\ %{}) when is_map(state) do
    # merge the drinks_menu to the state
    drinks_menu = state
    |> Map.get(:drinks_menu, %{})
    # any drinks_menu in the state overrides the default drinks_menu
    |> Map.merge(@drinks_menu, fn _k, v1, _v2 -> v1 end)

    {:ok, state |> Map.put(:drinks_menu, drinks_menu)}
  end

  # make_drink's handle_call
  def handle_call({:make_drink, verbs, opts}, _fromref, state) do

    state = state
    |> DrinkMaker.Menu.make_drink(verbs, opts)

    # reply with result self and (updated) state
    {:reply, self, state}    

  end

  # use Amlapio to generate the accessors and mutators for the drinks_menu
  use Amlapio, funs: [:get, :put], genserver_handle_call: [:drinks_menu]
  use Amlapio, funs: [:get, :put], genserver_handle_call: nil,
    namer: fn _map_name, fun_name -> 
    ["drink_maker_state_", to_string(fun_name)] |> Enum.join |> String.to_atom
  end

end

Testing the DrinkMaker GenServer

In this test Jane and Lucy set their favourite drink and then make it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35 test "drink_maker" do

  # create a drink maker for Jane and Lucy
  {:ok, janes_drink_maker} = DrinkMaker.start_link
  {:ok, lucys_drink_maker} = DrinkMaker.start_link

  # add Jane's favourite drink (instant coffee) to her drinks menu
  instant_cofffe_recipe =
    DrinkMaker.drinks_menu_get(janes_drink_maker, :make_instant_coffee)

  assert janes_drink_maker == janes_drink_maker
  |> DrinkMaker.drinks_menu_put(:make_my_favourite_drink, instant_cofffe_recipe) 

  # Lucy prefers Earl Grey tea
  earl_grey_tea_recipe =
    DrinkMaker.drinks_menu_get(janes_drink_maker, :make_earl_grey_tea)

  assert lucys_drink_maker == lucys_drink_maker 
  |> DrinkMaker.drinks_menu_put(:make_my_favourite_drink, earl_grey_tea_recipe)

  # now can make their favourite drink in a polymorphic way
  assert janes_drink_maker == janes_drink_maker
  |> DrinkMaker.make_drink(:make_my_favourite_drink)

  assert lucys_drink_maker == lucys_drink_maker
  |> DrinkMaker.make_drink(:make_my_favourite_drink)

  # check the drinks are the right ones
  assert :instant_coffee == janes_drink_maker
  |> DrinkMaker.drink_maker_state_get(:ready_drink)

  assert :earl_grey_tea == lucys_drink_maker
  |> DrinkMaker.drink_maker_state_get(:ready_drink)

end

How is this Polymorphism?

The GenServer example above shows that Jane’s and Lucy’s preferred drink depends on the value of the make_my_favourite_drink verb in their respective drinks_menu.

This is a example of runtime polymorphism where the decision on which function to dispatch is determined from the value of the verb (in the call to `DrinkMaker.make_drink/3`).

Rich’s Clojure documentation says:

The basic idea behind runtime polymorphism is that a single function designator dispatches to multiple independently-defined function definitions based upon some value of the call.

(btw the single function designator in my example is `DrinkMaker.Menu.make_drink/3`)

Rich goes on to say:

… go further still to allow the dispatch value to be the result of an arbitrary function of the arguments

It would be trivial to add an arbitrary function to the state, taking the state and verb as arguments, and use it to decide what make_drink variant to call, allowing arbitrarily complex multiple dispatch decisions (albeit maybe with a performance penalty).

Final Words

I’ve focused on just one dispatch function (make_drink) but it is straightforward to extend the code to other functions as well – I’ve already done so in the code for the actual project that gave rise to the idea for this blog post.

I’ve found this really very simple technique to be both flexible and powerful, especially in providing a simple, open way for multiple modules and applications to participate fully in the dsl, adding new, or modifying existing, verbs.

Code on Github

The code is on Github:

1
2
3
4 cd /tmp
git clone git@github.com:ianrumford/drink_maker.git
cd drink_maker
mix test

An Ostler in IT

Horses for Courses

Pattern Matching to Polymorphism - an Unexpected Journey