TL;DR: A macro to generate a Map API for a GenServer’s state or a Module with Agent-held state

The Trouble with State

Modules often need to keep state and share it but, as getting started says

Elixir is an immutable language where nothing is shared by default.

In Elixir there are two basic ways of sharing state: processes and ETS (which I wont consider further).

Arguably the most common way of implementing process-held state is GenServer. But for a module that does little more than hold state, GenServer can be overkill. As an alternative, an Agent provides a simpler, alternative way for a module to hold and manage its state.

But neither GenServer nor Agent provide any built-in API calls to manage the content of the state, not surprisingly as the state’s value can be anything.

More often than not though, I want the state to be a map, and sometimes with keys that are (sub)maps. So I usually want to make Map-like API calls — get, put, take, update etc — both for the state itself and any submaps.

Previously, I’ve written the wrappers manually. This is not hard but is just a bit tedious to do given the possible combinations of state wrappers, submap wrappers, arities, etc (lots and lots of cut’n’paste’edit!). So recently I’ve bitten the bullet and written a first cut of a macro to do the heavy lifting.

The interface I wanted for my applications’ modules was to simply use the module (that I’ve called Amlapio) that makes the API wrapper functions.

As many will already know use calls the __using__ macro in the module being used, and the code generated by __using__ is injected into the caller module.

Here’s an example (from the repo’s tests) of a module using an Agent to hold its state.

The wanted submap wrappers are defined by the agent: [:buttons, :menus, :checkboxes] option.

To generate state wrappers the submaps have been set to nil: agent: nil

Generating wrappers for only a subset of Map functions can be given using the funs options e.g, for the state wrappers, funs: [:get, :put, :pop]

Finally, a namer function has been given to name the state wrappers. It is passed two arguments: the map name (e.g. buttons for a submap, but nil for a state wrapper) and the Map function name (e.g. pop).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
defmodule ExampleAgent1 do

  # generate wrappers for three submaps
  use Amlapio, agent: [:buttons, :menus, :checkboxes]

  # generate *only* get, put and pop wrappers for the state itself and
  # use a namer function to name the wrappers "agent_state_get",
  # "agent_state_put" and "agent_state_pop"
  use Amlapio, agent: nil, funs: [:get, :put, :pop],
    namer: fn _map_name, fun_name ->
    ["agent_state_", to_string(fun_name)] |> Enum.join |> String.to_atom
  end

  # create the agent; note the default state is an empty map
  def start_link(state \\ %{}) do
    Agent.start_link(fn -> state end)
  end

end

These tests from the repo show how the submap wrappers would be used, pretty much how you’d expect. (The repo has tests for the state wrappers as well.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
test "agent_submap1" do

  buttons_state = %{1 => :button_back, 2 => :button_next, 3 => :button_exit}
  menus_state = %{menu_a: 1, menu_b: :two, menu_c: "tre"}
  checkboxes_state = %{checkbox_yesno: [:yes, :no], checkbox_bool: [true, false]}
  agent_state = %{buttons: buttons_state, 
                  menus: menus_state, checkboxes: checkboxes_state}

  # create the agent
  {:ok, agent} = ExampleAgent1.start_link(agent_state)

  # some usage examples

  assert :button_back == agent |> ExampleAgent1.buttons_get(1)
  assert :button_default == 
    agent |> ExampleAgent1.buttons_get(99, :button_default)

  assert agent == agent |> ExampleAgent1.menus_put(:menu_d, 42)
  assert menus_state |> Map.put(:menu_d, 42) == agent |> ExampleAgent1.agent_state_get(:menus)

  assert {[:yes, :no], agent} == 
    agent |> ExampleAgent1.checkboxes_pop(:checkbox_yesno)

end

Creating wrappers for a GenServer’s state is very similar. However, each wrapper has two “parts”: an api function and a handle_call function.

The api wrapper for e.g. `buttons_get/3` looks like this:

1
2
3
4
# api wrapper for buttons_get
def buttons_get(pid, button_name, button_default \\ nil) do
 GenServer.call(pid, {:buttons_get, button_name, button_default})
end

… while the matching handle_call looks like:

1
2
3
4
def handle_call({:buttons_get, button_name, button_default}, _fromref, state) do
  value = state |> Map.get(:buttons, %{}) |> Map.get(button_name, button_default)
  {:reply, value, state}
end

Here’s example of generating wrappers for a GenServer.

Remember: all handle_call functions must be kept together in the source else the compiler will complain.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
defmodule ExampleGenServer1 do

  # its a genserver
  use GenServer

  # generate API wrappers for three submaps
  use Amlapio, genserver_api: [:buttons, :menus, :checkboxes]

  # generate *only* get, put, pop and take wrappers for the state itself and
  # use a namer function to name the wrappers "state_get",
  # "state_put", "state_pop", and "state_take"
  use Amlapio, genserver_api: nil, funs: [:get, :put, :pop, :take],
    namer: fn _map_name, fun_name ->
    ["state_", to_string(fun_name)] |> Enum.join |> String.to_atom
  end

  # create the genserver; note the default state is an empty map
  def start_link(state \\ %{}) do
    GenServer.start_link(__MODULE__, state)
  end

  # << more functions>>

  # handle_calls start here

  # generate the handle_call functions for three submaps' wrappers
  use Amlapio, genserver_handle_call: [:buttons, :menus, :checkboxes]

  # generate the handle_call functions for the state wrappers.
  use Amlapio, genserver_handle_call: nil, funs: [:get, :put, :pop, :take],
    namer: fn _map_name, fun_name ->
    ["state_", to_string(fun_name)] |> Enum.join |> String.to_atom
  end

end

There are tests got the GenServer example in the repo but they look almost identical to the Agent example.

If you just want to give it a whirl, that pretty much it. Its available on Hex and just needs to be added to the dependencies in mix.exs in the usual way.

1
{:amlapio, "~> 0.1.0"}

But if you are interested in a high-level explanation of how I implemented Amlapio, read on.

Code on Github

You might want to have a look at the code if you are following along.

1
2
3
4
cd /tmp
git clone git@github.com:ianrumford/amlapio.git
cd amlapio
mix test

Wrapper Patterns

Map has three types of functions: accessors (e.g. get), mutators (e.g. put), and the (four) ‘combo’ functions that are both accessors and mutators (e.g. pop) (called hereafter poppers).

Wrapper Pattern for a Mutator

Here’s an example of put for the buttons submap of an agent showing the steps in the wrapper, illustrating that all mutators have a common pattern.

BTW I will be using this wrapper as my example when I show code, etc below unless I say otherwise.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# the buttons_put submap wrapper for an agent
def buttons_put(agent, button_name, button_value) do

  # start with the agent and get its state
  state = agent |> Agent.get(fn s -> s end)

  new_buttons = state 
  # get buttons from the state (default to an empty map)
  |> Map.get(:buttons, %{})
  # put the button_name & button_value key-value pair into buttons
  |> Map.put(button_name, button_value)

  # update the state with the new buttons submap
  new_state = state |> Map.put(:buttons, new_buttons)

  # update the agent with the new state
  :ok = agent |> Agent.update(fn s -> new_state end)

  # return the agent
  agent

end

The mutator pattern for both an agent and GenServer wrapper has a maximum of six logical steps:

1. get the state (line 5)

2. (optional) get the submap of the state (line 9)

   This step is only needed when creating wrappers for a submap.

3. run the Map API call (line 11)

   In this case its `Map.put/3`.

4. (optional) put the (new) submap into the state (line 14)

   Again, this step is only needed for submap wrappers.

5. update the state (line 17)

6. return the result (line 20)

Wrapper Pattern for an Accessor

The steps for an accessor (e.g. buttons_get) are even simpler:

1. get the state

2. (optional) get the submap

3. run the Map API call

4. return the result

Wrapper Pattern for a Popper

The pattern for a popper (e.g. buttons_pop) is very similar to a mutator, the main difference being the wrapper’s result is a tuple e.g.

1
{button_value, agent}

where the state of the agent no longer has the button_name key in the buttons submap.

Wrapper Generation

The wrapper patterns are implicitly recipes for the generation of each wrapper’s code: each step gives rise to its own fragment of code.

Sometimes the code fragment will be nil e.g state wrappers have no submaps so steps 2 & 4 (of the mutator) will be nil.

Wrapper Generation Spec (map_wrap)

Amlapio’s __using__ macro doesn’t do all that much, it just hands off its arguments to the `Amlapio.process_opts/1` function.

process_opts creates a list of maps (called a map_wrap in the code), one for every wrapper name & arity combo e.g.

1
2
3
# the map_wrap for an agent's buttons_put wrapper
%{fun_type: :mutator, fun_name: :buttons_put, fun_arity: 3,
  map_type: :agent, map_name: :buttons, map_fun: :put} = map_wrap

btw map_name is always nil for a state wrapper

The values of the map_wrap’s keys enable normal pattern matching to be used to determine the wrapper being generated and acts accordingly.

(During generation, the code fragment is usually stored under the :fun_ast key.)

Wrapper Generation DSL

To define a wrapper’s recipe in a programmable way, I’ve invented a very simple DSL (implemented by `Amlapio.DSL`).

The dsl essentially defines a pipeline of transforms that takes the initial map_wrap and incrementally builds the full code for an specific wrapper.

Here is an example of the “top level” make_fun dsl:

1
2
3
4
5
6
# make_fun dsl for an agent's submap mutator e.g. buttons_put
[push: [make: :state_get, make: :assign_state],
 push: [make: :make_body],
 push: [make: :state_put],
 push: [make: :result_value],
 make: {&AMLUtils.map_wrap_push_wraps_asts_reduce_recursive/1, :fun_def}] 

The make_body is itself another dsl:

1
2
3
4
5
6
# make_body dsl for an agent's submap mutator e.g. buttons_put
[push:
 [pipe: [make: :state,
         pipe: [pipe: [make: :state, make: :submap_get, make: :fun_apply],
                make: {&AMLUtils.map_wrap_ast_index_set_1/1, :submap_put}]],
  make: :assign_state]]

Each individual dsl is a Keyword where the values may themselves be another dsl, an atom (called a snippet - see make) or a tuple (explained in custom logic).

The keys in the dsl are the verbs and there are three of them: push, pipe and make.

Wrapper Generation DSL verb: make

make normally creates a quoted fragment of code.

The value of the make verb (the snippet e.g. state_get) is the 1st argument in the call to `Amlapio.Snippets.map_wrap_make_snippet/2`. The 2nd argument is the map_wrap. For example:

1
2
3
4
5
6
def map_wrap_make_snippet(:state_get, %{map_type: :agent} = _map_wrap) do
  quote do
    # start with the agent (pid) and get its state
    pid |> Agent.get(fn state -> state end)
  end
end

The ast returned is stored under the :fun_ast key.

Wrapper Generation DSL verb: push

After running the push’s dsl, the returned map_wrap is pushed onto the tail of list held under the :push_wraps key in the map_wrap

The push verb provides a way of caching a list of map_wraps, each of which usually has one statement of the wrapper stored under its :fun_ast key.

Wrapper Generation DSL verb: pipe

Similar to push, pipe runs its dsl, but then takes all of the fun_asts from the map_wraps returned from running each step in the dsl, and pipes them together.

For example, this “clause” of the dsl above:

1
2
pipe: [pipe: [make: :state, make: :submap_get, make: :fun_apply],
       make: {&AMLUtils.map_wrap_ast_index_set_1/1, :submap_put},

generates this code fragment:

1
Map.put(:buttons, state |> Map.get(:buttons, %{}) |> Map.put(button_name, button_value))

Wrapper Generation DSL: custom logic

In the dsl clause above the value of the make was a tuple.

1
make: {&AMLUtils.map_wrap_ast_index_set_1/1, :submap_put}

The submap_put is the snippet as usual.

But the first element is a function that’s called with the map_wrap before the snippet is made.

More generally, the make value can be a 3tuple:

1
{funs_ante, snippet, funs_post}

where funs_ante and funs_post can be either nil, a function or list of functions.

The tuple elements are used to create a list of functions that take one argument – map_wrap – which are then used with `Enum.reduce/3`:

1
2
3
4
5
6
[funs_ante,
 fn map_wrap -> map_wrap_dsl_verb(verb, snippet, map_wrap) end,
 funs_post]
|> List.flatten
|> Stream.reduce(&is_nil/1)
|> Enum.reduce(map_wrap, fn fun, map_wrap -> fun.(map_wrap) end)

(btw I currently use only the 2tuple form)

Just to complete the explanation of this example, `AmlUtils.map_wrap_ast_index_set_1/1` sets the index value to be used in the call to `Macro.pipe/3` to 1. This makes the code fragment produced by the prior pipe

1
state |> Map.get(:buttons, %{}) |> Map.put(button_name, button_value)

… the 2nd argument of the submap_put

1
Map.put(:buttons, state |> Map.get(:buttons, %{}) |> Map.put(button_name, button_value))

Final Words

The use of regular pattern matching together with a very simple DSL made it quite straightforward to generate the code for the various wrapper patterns.

The technique is also quite extensible: by adding e.g. more calls to `Amlapio.Snippets.map_wrap_make_snippet/2`, or maybe even new dsl verbs, one could address other code building needs.

Again though, it highlights that manipulating asts, mostly using the functions you use day in and day out, is really no more difficult than any other challenge in Elixir.

+1 macros