Etudes in Macro Programming

The way ModuleMixins is implemented, is that we start out with something relatively simple, and build out from that. This means there will be some redudant code. Macros are hard to engineer, this takes you through the entire process.

Before we explain the full @compose macro, we can do some finger exercises to understand the mechanisms behind its implementation.

@spec

The @spec macro creates a new module, and stores its own AST inside that module.

#| file: src/Spec.jl
module Spec

using MacroTools: @capture
export @spec

<<spec>>

end

We may test that this works using a small example.

#| file: test/SpecSpec.jl
<<test-spec-toplevel>>

@testset "ModuleMixins.Spec" begin
    using MacroTools: prewalk, rmlines
    clean(expr) = prewalk(rmlines, expr)

    <<test-spec>>
end

#| id: test-spec-toplevel
using ModuleMixins.Spec: @spec, @spec_mixin, @mixin

@spec module MySpec
const msg = "hello"
end

#| id: test-spec
@testset "@spec" begin
    @test clean.(MySpec.AST) == clean.([:(const msg = "hello")])
    @test MySpec.msg == "hello"
end

This may seem like a silly example, but storing the AST of a module inside itself is very powerful. It means that inside macros we can always return to original expressions of other modules and devise ways of combining, composing and compiling new modules from them.

#| id: spec
"""
    @spec module *name*
        *body*...
    end

Create a spec. The `@spec` macro itself doesn't perform any operations other than creating a module and storing its own AST as `const *name*.AST`.

This macro is only here for teaching purposes.
"""
macro spec(mod)
    @assert @capture(mod, module name_
    body__
    end)

    esc(Expr(:toplevel, :(module $name
    $(body...)
    const AST = $body
    end)))
end

Inside ModuleMixins we make extensive use of MacroTools.@capture. Preceding @capture with @assert is a quickfire way of making sure that our macro was called with the correct syntax.

When defining a new module we have to create a top-level expression, and call esc on the entire expression to make sure no symbols are mangled.

@spec_mixin

We now add the @mixin syntax. This still doesn't do anything, other than storing the names of parent modules.

#| id: test-spec-toplevel
@spec_mixin module MyMixinSpecOne
@mixin A
end
@spec_mixin module MyMixinSpecMany
@mixin A, B, C
end

#| id: test-spec
@testset "@spec_mixin" begin
    @test MyMixinSpecOne.PARENTS == [:A]
    @test MyMixinSpecMany.PARENTS == [:A, :B, :C]
end

Here's the @mixin macro:

#| id: spec
macro mixin(deps)
    if @capture(deps, (multiple_deps__,))
        esc(:(const PARENTS = [$(QuoteNode.(multiple_deps)...)]))
    else
        esc(:(const PARENTS = [$(QuoteNode(deps))]))
    end
end

The QuoteNode calls prevent the symbols from being evaluated at macro expansion time. We need to make sure that the @mixin syntax is also available from within the module.

#| id: spec

macro spec_mixin(mod)
    @assert @capture(mod, module name_
    body__
    end)

    esc(Expr(:toplevel, :(module $name
    import ..@mixin

    $(body...)

    const AST = $body
    end)))
end

Module Templates

How cool it would be, if we can parametrise a module. Julia rules that modules can only be defined at top-level. So we have to go by the macro route once more. First, we try to implement the desired effect in isolation.

We define a module with arguments using the @lambda macro.

#| file: src/Lambda.jl
module Lambda
    using MacroTools: @capture
    import ..Passes: Pass, pass, no_match, walk
    export @lambda, @instantiate

    struct Parameter
        name::Symbol
        has_default::Bool
        default::Any
    end

    name(par) = par.name
    has_default(parameter) = parameter.has_default

    function make_parameter(expr)
        if @capture(expr, name_ = default_)
            return Parameter(name, true, default)
        elseif expr isa Symbol
            return Parameter(expr, false, nothing)
        else
            return nothing
        end
    end

    struct Argument
        name::Union{Symbol, Nothing}
        value::Any
    end

    positional(argument) = argument.name === nothing

    make_argument(mod) = function (expr)
        value = if @capture(expr, name_ = value_)
            value
        else
            expr
        end
        return Argument(name, @eval(mod, $(value)))
    end

    struct BoundVariable
        name::Symbol
        value::Any
    end

    function bind(pars, args)
        positional_args = Iterators.takewhile(positional, args) |> collect
        keyword_args = Dict(arg.name => arg for arg in args[length(positional_args)+1:end])
        keyword_pars = pars[length(positional_args)+1:end]
        @assert(!any(positional, values(keyword_args)))
        positional_bindings = (BoundVariable(par.name, arg.value)
            for (par, arg) in Iterators.zip(pars, positional_args))
        keyword_bindings = (BoundVariable(par.name, par.name in keys(keyword_args) ?
            keyword_args[par.name].value :
            par.default) for par in keyword_pars)
        bindings = Iterators.flatten((positional_bindings, keyword_bindings)) |> collect
        @assert(name.(pars) == name.(bindings))
        return bindings
    end

    function as_expression(bv::BoundVariable)
        return :(const $(bv.name) = $(bv.value))
    end

    struct ModuleTemplate
        name::Symbol
        parameters::Vector{Parameter}
        body::Vector{Any}
    end

    macro lambda(expr)
        @assert @capture(expr, module name_ body__ end)
        if isempty(body) | !@capture(body[1], {raw_parameters__})
            return esc(Expr(:toplevel, :(module $name
                $(body...)
                const AST = $body
            end)))
        end
        parameters = raw_parameters .|> make_parameter |> filter(!isnothing) |> collect
        println("module $(name) with parameters $(parameters)")
        template = ModuleTemplate(name, parameters, body[2:end])
        return esc(:(const $name = $template))
    end

    macro instantiate(expr)
        @assert @capture(expr, instance_name_ = template_name_{raw_arguments__})
        arguments = raw_arguments .|> make_argument(__module__)
        template = getfield(__module__, template_name)
        bound_vars = bind(template.parameters, arguments)
        return esc(Expr(:toplevel, :(module $(instance_name)
            const AST = $(template.body)
            $(as_expression.(bound_vars)...)
            $(template.body...)
        end)))
    end
end