Testing Individual Elixir Doctests
Doctests: FTW!
I’ve been working on new releases of a couple of my published Hex packages recently, as well as some new packages that should be soon published fairly soon.
All of them have doctests, sometimes a lot of them.
I think doctests are great; I’m a fan. “But there’s a problem”
Advocacy Spot: Writing Doctests is a Good Habit
My ritual when writing a new package for publication seems to have settled on writing some “bottom up” prototype code, writing some rough’n’ready tests to flesh the code out a bit, and then moving to document the package using “top down” doctests.
I’ve found the discipline of writing “top down” explain-through-doctests to convey what the package and its functions do a good discipline for ensuring the package makes sense, hangs together and is both coherent and consistent.
Whilst writing doctests, I’ve had more than a few “no, not like that, it needs to be like this” moments when I’ve realised the ideas and design that drove my prototype were wrong and need to be reworked: I’ve come to accept that the explain-through-doctests style sometimes will lead to a not insignificant amount of rework, hopefully though leading to a better package.
In the same vein, after writing the “obvious” tests, I’ve frequently thought “hmm, that would be a useful feature” and would write a quick doctest to try it out and see how the code behaves and adding the feature if it turns out useful. So I’ve found doctests also make it easy to do a bit of test driven development.
Doctests: what’s not to like?
Nothing really. Once you’ve gotten enough markdown under your belt, and the knack of how to embed code and tests, you can crack out the documentation pretty quickly: I find I can write a doctest generally just as quick, and maybe even quicker, than tests in a standalone test file.
The main problem I have with them is also their strength: they can pack a large number of tests under the control of the test file that calls the doctest macro. Using individual tests in a test file, I can “package” the tests exactly how I want, even down to just having one test per file if that’s sensible. But doctests are always enabled and the only way to disable selectively one or more is to comment them out. Alternatively you use the :except or :only options to doctest but they work at the function level, not the individual tests.
This is usually not a problem until I have a good number of existing doctests and am, say, adding new functions with doctests, or doing some other rework, and one or more doctests fail requiring me to track the causes down.
On a development branch I run with print diagnostics enabled because its still the easiest and most effective way of tracing through the code (especially complicated and/or recursive code). But when I’m running lots of doctests, the volume of prints become hard to pick apart and I can’t “see” just the ones for the failing test(s).
I could just write a standalone test in a test file for the failing ones but that’s a diversion: I want to end up with a full set of working doctests.
Previously I just commented stuff out leaving just the new and/or failing ones. But that was both tedious and a bit silly, especially as when I uncommented the commented tests something else could (and often did) break.
Then I realised I already knew how to test specific, individual doctests while leaving the others enabled.
Writing The Real Module
To explain how I test individual tests, I’m going to have to set the scene with a quick overview of how I write doctests “normally” for a real module.
Lets assume I’ve prototyped my new module in ./lib/real_module.ex, and have added the first doctest.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
defmodule RealModule do
@moduledoc ~S"""
Real Module To Demonstrate Testing Individual Doctests
"""
@doc ~S"""
`add_xy/2` adds two integers together.
## Examples
iex> add_xy(39, 3)
42
"""
def add_xy(x, y) when is_integer(x) and is_integer(y) do
x + y
end
end
Running The Real Doctests
I now need a test file (./test/real_doctest_test.exs) to call the doctest macro with the module being tested (i.e. RealModule).
1
2
3
4
5
defmodule RealDoctest do
use ExUnit.Case, async: true
import RealModule
doctest RealModule
end
I can now run the doctests using mix
1
mix test ./test/real_doctest_test.exs
But having to rerun this command manually everytime I make a change to the module gets old pretty quickly so I use inotify-tools and have the following running in a terminal (bash) console:
1
while inotifywait -e close_write ./lib/real_module.ex; do mix docs; mix test ./test/real_doctest_test.exs; done
Now every time I save the module file, the docs are rebuilt and the doctests run automatically in the console running next to my editor window. A quick refresh of the browser tab to see the updated docs and I have all the feedback I need.
Writing The Fake Module
To test an individual doctest, I need a fake module that “impersonates” the real one.
The fake module will need access to all the real module’s functions: the easiest way to do that is to delegate from fake to real.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
defmodule FakeModule do
@moduledoc ~S"""
Fake Module To Demonstrate Testing Individual Doctests
"""
# these vars will be used for args in the defdelegates below
@args_vars 5 |> Macro.generate_arguments(__MODULE__)
# get the names and arities of the real module's function
@funs :functions |> RealModule.__info__
# use unquote fragments to delegate to the real's module functions
for {name,arity} <- @funs do
defdelegate unquote(name)(unquote_splicing(Enum.take(@args_vars, arity))), to: RealModule
end
@doc ~S"""
I can add tests here one by one and see how the code behaves
iex> add_xy(-3, 45)
42
iex> add_xy(-7, -35)
42
"""
def here_to_satisfy_mix do
nil
end
end
Note the strange function here_to_satisfy_mix; mix will warn if there isn’t a function definition following the @doc.
There is nothing to stop you writing and testing completely new functions in the fake module and then copy’n’paste into the real module.
Compiling the Fake Module
fake_module.ex is a regular Elixir file and needs to be compiled but shouldn’t live in ./lib as its only used for test.
I could put the code in ./test_helper.exs but I don’t like to clutter the helper and prefer generally to put test-only .ex files in their own folder e.g. ./test/lib. So the fake module file becomes ./test/lib/fake_module.ex.
To tell mix to compile the test-only .ex files requires a couple of changes to mix.exs.
First this line needs to be added to the project function. Its tells mix where to find the .ex files and calls a function (defined in mix.exs) called elixirc_paths, passing the mix environment.
1
elixirc_paths: elixirc_paths(Mix.env),
Next the elixirc_paths function needs to be defined:
1
2
defp elixirc_paths(:test), do: ["lib", "test/lib"]
defp elixirc_paths(_), do: ["lib"]
So when running mix test, the mix env will be :test and mix will compile not only ./lib but also the .ex files in ./test/lib
Running The Fake Doctests
Finally, I’ll need a test file to run the fake module’s doctests.
I will call this ./test/fake_doctest i.e without the “_test.exs” suffix to prevent it running automatically when a mix test is run.
The fake doctests can be run manually or automatically, just like the real ones:
1
mix test ./test/fake_doctest
1
while inotifywait -e close_write ./test/lib/fake_module.ex; do mix docs; mix test ./test/fake_doctest; done
Final Words
Nothing special or original about any of this, its all standard, legal Elixir, just need to ensure you understand what files need to live where (e.g. ./test_lib)
Eating my own dog food: I now have editor frames and inotifywaits running for both my real and fake modules at the same time and find it really flexible and pain-free.
Code on Github
BTW: There are a couple of scripts in ./bin to run the inotifywaits.
1
2
3
4
5
6
cd /tmp
git clone git@github.com:ianrumford/testing_individual_elixir_doctests.git
cd testing_individual_elixir_doctests
mix deps.get
mix test
mix test ./test/fake_doctest
