Metaprogramming Without Macros
TL;DR: Code is Data!
Backstory
Originally this post was part of another post. But the whole post was getting too big, so I’ve pulled it out here.
Although really just part one, its really stand-alone and may be of more general interest.
The repo has the examples.
Metaprogramming with Macros
I guess anybody who knows anything, has read anything or used Elixir at all will be aware the core features of the language are implemented as a collection of macros – the special forms.
Elixir’s macro support is incredibly useful and I’d recommend anybody interested in learning some of the nitty-gritty to read Chris’s excellent book. Alternatively, there are also many excellent blog articles out there (like Sasa’s), and I’ve even had a go myself.
At the heart of macros are quoted forms, Elixir’s name for its abstract syntax tree (AST) representation.
When you run a macro, Elixir passes the arguments as quoted form(s) and expects quoted form(s) as the result (which will then be compiled). Inside the macro though, its just regular Elixir code, albeit manipulating ASTs.
I tend to think of the relationship (contract) between the Elixir compiler and a macro as simply this:
I’ll maybe pass you some asts, do what you want with them but give me back some asts and I’ll compile them for you.
But macros are not the only or best answer to some metaprogramming needs.
Metaprogramming without Macros
You can create your own ast simply by calling quote:
1
2
3
4
5
combine_plus_ast = quote do
def combine_plus(x, y \\ 42) when is_number(x) and is_number(y) do
x + y
end
end
You can also ask the Elixir compiler to compile an ast using Code.eval_quoted/3 function.
1
Code.eval_quoted(combine_plus_ast, [], __ENV__)
The tests show the function works as expected:
1
2
3
4
5
assert 7 == combine_plus(5,2)
assert 47 == combine_plus(5)
assert_raise FunctionClauseError, fn ->
combine_plus(1, :two)
end
Transforming an Ast with Quote and Unquote
The usual way of building (composing) asts is just a combination of quote and unquote.
Here the function’s body is a separate ast and is unquoted into the function’s ast.
1
2
3
4
5
6
7
8
9
10
# define the body code as a separate ast
combine_list_body_ast = quote(do: x ++ y)
# use unquote to "insert" the body ast into the function definition
quote do
def combine_list(x, y \\ [4,5,6]) when is_list(x) and is_list(y) do
unquote(combine_list_body_ast)
end
end
# and compile the function
|> Code.eval_quoted([], __ENV__)
Works as expected:
1
2
3
4
5
assert [1,2,3] == combine_list([1], [2,3])
assert [1,2,4,5,6] == combine_list([1,2])
assert_raise FunctionClauseError, fn ->
combine_list([1,2], 42)
end
Transforming an AST with Postwalk
Elixir provides a much more fine-grained way of transforming (editing) an ast, namely Macro.postwalk/2 and related functions.
Consider this function, the third in our series, to combine (merge) two maps.
1
2
3
4
5
6
combine_map_body_ast = quote(do: Map.merge(a, b))
combine_map_ast = quote do
def combine_map(x, y \\ %{d: 4}) when is_map(x) and is_map(y) do
unquote(combine_map_body_ast)
end
end
Note the arguments in the body are called a and b, not x and y as in the function’s arguments. This code wont compile since the variables and arguments are inconsistent.
But you can use Macro.postwalk/2 to edit the ast for the complete function, with inserted body, and change the a to x and b to y.
To understand how this works, you need to appreciate that the quoted forms for the x, y, a and b variables are e.g. {:x, [], definingmodule} where definingmodule is the name (atom) of the module where the quoted form was created (or nil).
Quoted variables can be created using Macro.var/2. For example:
1
2
:x |> Macro.var(__MODULE__)
:a |> Macro.var(nil)
Macro.postwalk/2 takes a function that is passed each snippet (a “subast”) from the complete ast and replaces the original snippet with the result of the function, rebuilding the complete ast (a bit like mapping each element in an enumerable). The “trick” is identifying which snippets to replace using e.g. pattern matching.
1
2
3
4
5
6
7
8
9
10
11
# edit the a & b vars to x & y
combine_map_edit_ast = combine_map_ast
|> Macro.postwalk(fn
{:a, [], module} when is_atom(module) -> Macro.var(:x, __MODULE__)
{:b, [], module} when is_atom(module) -> Macro.var(:y, __MODULE__)
# passthru
v -> v
end)
# and compile the function
combine_map_edit_ast |> Code.eval_quoted([], __ENV__)
A couple of tests:
1
2
assert Map.equal?(%{a: 1, b: 2, c: 3}, combine_map(%{a: 1}, %{b: 2, c: 3}))
assert Map.equal?(%{a: 1, d: 4}, combine_map(%{a: 1}))
Ok, after the postwalk, you can’t “see” directly the changes but the tests demonstrate the function works. However, using Macro.to_string/1, you can create the “textualised” ast and compared it to what’s expected i.e the a and b have been replaced by x and y.
1
2
3
4
5
6
# create stringifed code
combine_map_stringified_ast = combine_map_edit_ast |> Macro.to_string
# confirm stringifed code is as expected i.e. no a or b vars
assert combine_map_stringified_ast ==
"def(combine_map(x, y \\\\ %{d: 4}) when is_map(x) and is_map(y)) do\n Map.merge(x, y)\nend"
Transforming an AST Template with Postwalk
All three of the functions above have the same form and could be generalised into a template
1
2
3
4
5
combine_template_ast = quote do
def fun_name(arg0, arg1 \\ arg1_default) when arg0_guard and arg1_guard do
fun_body
end
end
The template has a number of placeholders (proxies) such as fun_name, fun_body, arg1_guard for the concrete code.
Here is a dictionary of the replacements for the combine_plus function above. The keys are the proxy names and the values are valid asts.
1
2
3
4
5
6
7
8
combine_plus_proxies = %{
fun_name: :combine_plus,
fun_body: quote(do: x + y),
arg0: Macro.var(:x, __MODULE__),
arg1: Macro.var(:y, __MODULE__),
arg1_default: 42,
arg0_guard: quote(do: is_number(:arg0)),
arg1_guard: quote(do: is_number(:arg1))}
Notice how e.g the :arg1_with_default proxy uses the :arg1 proxy in the former’s defintion; the transformation will be recursive.
Matching all the proxies in the template is a bit more involved than the above example but the regularity of an ast makes it a straightforward data transformation.
First we need to create a helper function (in another module) to use (apply) a proxy dictionary during the transformation of the template. The helper uses standard pattern matching to decide which clause to use for each snippet from the template.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# helper to apply a proxy dictionary in the
# transformation of a template snippet
defp helper_transform_template_snippet(template, proxy_dict)
# simple atom
defp helper_transform_template_snippet(n, proxy_dict) when is_atom(n) do
case proxy_dict |> Map.has_key?(n) do
true -> proxy_dict |> Map.get(n)
_ -> n
end
end
# simple var
defp helper_transform_template_snippet({n, ctx, m}, proxy_dict)
when is_atom(n) and is_atom(m) do
case proxy_dict |> Map.has_key?(n) do
true -> proxy_dict |> Map.get(n)
_ -> {n, ctx, m}
end
end
# fun call with args
defp helper_transform_template_snippet({n, ctx, args}, proxy_dict)
when (is_atom(n) or (is_tuple(n) and tuple_size(n) == 3))
and is_list(args) do
# fun_name may need transforming
n = n |> helper_transform_template(proxy_dict)
# apply proxies to the args
args = args |> Enum.map(fn arg -> arg |> helper_transform_template(proxy_dict) end)
# return the template with the transformed args
{n, ctx, args}
end
# passthru
defp helper_transform_template_snippet(n, _proxy_dict) do
n
end
# main transform function
def helper_transform_template(template, proxy_dict) when is_map(proxy_dict) do
template
|> helper_transform_template_snippet(proxy_dict)
|> case do
# no changes? => completely transformed
^template -> template
# need to recurse until no further chnages
new_template -> new_template |> helper_transform_template(proxy_dict)
end
end
Next we need to create simple (anonymous) function that generates another anonymous function that calls the helper with the right proxy dictionary for the postwalk:
1
2
3
4
5
6
# edit the template with the dictionary
postwalk_fun_generator = fn proxy_dict ->
# return a function that embeds the proxy dictionary in the call
# to helper_transform_template
fn template -> template |> helper_transform_template(proxy_dict) end
end
Now we can postwalk the template with the generated fun embedding the combine_plus_proxies dictionary to create and compile the combine_plus function:
1
2
3
4
5
# edit the template with the dictionary
combine_template_ast
|> Macro.postwalk(postwalk_fun_generator.(combine_plus_proxies))
# and compile the function
|> Code.eval_quoted([], __ENV__)
The same tests as above work no differently
1
2
3
4
5
assert 7 == combine_plus(5,2)
assert 47 == combine_plus(5)
assert_raise FunctionClauseError, fn ->
combine_plus(1, :two)
end
The combine_list_proxies dictionary looks as you’d expect:
1
2
3
4
5
6
7
8
combine_list_proxies = %{
fun_name: :combine_list,
fun_body: quote(do: x ++ y),
arg0: Macro.var(:x, __MODULE__),
arg1: Macro.var(:y, __MODULE__),
arg1_default: [4,5,6],
arg0_guard: quote(do: is_list(:arg0)),
arg1_guard: quote(do: is_list(:arg1))}
I’ve not shown the rest of the code here for brevity but combine_list and combine_map can be created from the same template using the correct proxy dictionary. (The rest of the code and tests are in the repo)
Final Words
The only “hard part” above was writing the transformation function used by postwalk.
A bit of effort but the technique “has legs”: postwalk can apply any transformation.
Using a proxy dictionary to hold the transformations that can be resolved and replaced recursively is very powerful. Since the dictionary is just a map, it can be updated to change selectively any proxy (e.g. updating a default such as :arg1_default but leaving the argument name :arg1 the same)
Most of all the technique highlights to me that in Elixir code is data!.
Examples on Github
1
2
3
4
5
cd /tmp
git clone git@github.com:ianrumford/metaprogrammingwithoutmacros_examples.git
cd metaprogramming_with_macros_examples
mix deps.get
mix test
