Imagine you’re working on a recipe app. Users can type in what ingredients they have to hand and the app will suggest some recipe ideas from its library.

As usual, each service in the app gets an interface:

interface ILogger
{
    void Log(string message);
}
interface IRecipeRepository
{
    IEnumerable<Recipe> FindRecipesWithIngredient(string ingredientName);
}
interface IRecipeSuggester
{
    IEnumerable<Recipe> SuggestRecipes(IEnumerable<string> ingredients);
}

When a service needs to depend on another service (via its interface), the traditional way is to use a class constructor: each dependency becomes a constructor parameter. You can change out implementations of a service by passing different arguments, and compose multiple dependencies by using multiple parameters. While it’s possible to hand-write your app’s composition root with new expressions, it can be tedious, so people use an IOC container which calls all of your services’ constructors by magic.

The design I want to explore instead involves declaring dependencies by using constraints on a type parameter. Each service in the system is going to take a TServices type parameter,

class TopTenRecipeSuggester<TServices> : IRecipeSuggester

and each interface will have a corresponding “provider”.

interface ILoggerProvider
{
    static abstract ILogger Logger { get; }
}
interface IRecipeRepositoryProvider
{
    static abstract IRecipeRepository RecipeRepository { get; }
}
interface IRecipeSuggesterProvider
{
    static abstract IRecipeSuggester RecipeSuggester { get; }
}

(Why make these abstract properties static? It’ll become clear later.) I think of these provider interfaces as defining composable fragments of a namespace. Each provider claims one name (or more, why not) for its corresponding interface(s). Generic type constraints let us put the building blocks together.

class TopTenRecipeSuggester<TServices> : IRecipeSuggester
    where TServices : IRecipeRepositoryProvider, ILoggerProvider

That’s why the providers need to be interfaces — so we can use more than one in a constraint.

This is nice and composable. Each service specifies exactly the dependencies it needs, as a set of constraints on the shared TServices type parameter. The type system takes care of union-ing these constraints; eventually we’ll specify a single concrete type to stand in for all the instances of TServices and we’ll be required to implement exactly the providers that were requested by the services.

Now when the code needs to call a dependency, it just looks it up in the TServices namespace.

class TopTenRecipeSuggester<TServices> : IRecipeSuggester
    where TServices : IRecipeRepositoryProvider, ILoggerProvider
{
    public IEnumerable<Recipe> SuggestRecipes(IEnumerable<string> ingredients)
    {
        TServices.Logger.Log("finding recipes...");

        var results = (
            from i in ingredients
            from r in TServices.RecipeRepository.FindRecipesWithIngredient(i)
            group i by r into g
            orderby g.Count() descending
            select g.Key
        ).Take(10).ToList();

        TServices.Logger.Log("found some recipes");

        return results;
    }
}

The dependencies are injected through a single (type) parameter. Is this the dreaded “Service Locator” anti-pattern? No! The issue with Service Locator is that it makes dependencies invisible: you can’t tell how to set up a component’s dependencies without reading its source code. The constraints on the TServices type parameter are explicit and visible. If you want to use a service, you’re forced by the type checker to satisfy its dependencies. And the service class can’t access any dependencies other than the ones declared in the where clause.

How do you satisfy a set of dependencies? At the app’s composition root, we need to provide a type that can stand in for the TServices type parameter. That amounts to defining a class that implements all of the Provider interfaces by supplying a concrete implementation of each service.

class CompositionRoot : ILoggerProvider, IRecipeRepositoryProvider, IRecipeSuggesterProvider
{
    // CompositionRoot supplies itself as the TServices type parameter
    public static ILogger Logger { get; }
        = new ConsoleLogger<CompositionRoot>();

    public static IRecipeRepository RecipeRepository { get; }
        = new DbRecipeRepository<CompositionRoot>();

    public static IRecipeSuggester RecipeSuggester { get; }
        = new TopTenRecipeSuggester<CompositionRoot>();

    // ...
}

Finally, at the app’s entry point, you can pick out services directly from the CompositionRoot. (Typically you want to keep this layer thin: just call out to a controller object.)

// Program.cs
CompositionRoot.RecipeSuggestionController.Run();

Composable Composition Root

One problem with the CompositionRoot is that it’s difficult to swap out an implementation of a service. CompositionRoot passes itself as the TServices type parameter — not “the concrete type of the instance” — so you can’t override a property by subclassing it.

class TestCompositionRoot : CompositionRoot
{
    // This doesn't work - the TopTenRecipeSuggester is still going to
    // look in CompositionRoot to find its dependencies.
    public static override IRecipeRepository RecipeRepository { get; }
        = new MockRecipeRepository<TestCompositionRoot>();
}

You’d have to define an entirely separate composition root for testing instead.

Note

I actually don’t think this is too bad, or at least, it’s also a weak spot for classic constructor injection. With classic constructor injection, tests typically new up the SUT and specify all of its dependencies by hand anyway. Type-Level DI has somewhat more boilerplate but not, like, asymptotically more.

Anyway, we can patch this up using one of my favourite cursed tricks: F-bounds (aka the curiously recurring template pattern). CompositionRoot will (become an interface and) take a TServices type parameter itself. TServices stands in for the eventual concrete type of the composition root.

interface ICompositionRoot<TServices> : ILoggerProvider, IRecipeRepositoryProvider, IRecipeSuggesterProvider
    where TServices : ICompositionRoot<TServices>
{
    static ILogger ILoggerProvider.Logger { get; }
        = new ConsoleLogger<TServices>();

    static IRecipeRepository IRecipeRepositoryProvider.RecipeRepository { get; }
        = new DbRecipeRepository<TServices>();

    static IRecipeSuggester IRecipeSuggesterProvider.RecipeSuggester { get; }
        = new TopTenRecipeSuggester<TServices>();
}

Note

Now you can see why I made the providers’ properties static. Interfaces can’t have (non-static) fields, and accordingly non-static interface properties can’t have initialisers because there’s no corresponding storage location.

This arrangement allows you to derive another interface from CompositionRoot, override whatever you need, and forward the type parameter.

interface IMockCompositionRoot<TServices> : ICompositionRoot<TServices>
    where TServices : IMockCompositionRoot<TServices>
{
    static IRecipeRepository IRecipeRepositoryProvider.RecipeRepository { get; }
        = new MockRecipeRepository<TServices>();
}

Eventually, when it’s time to run code, someone will derive an empty concrete class which plugs itself in as the TServices type parameter.

class CompositionRoot : ICompositionRoot<CompositionRoot> {}
class MockCompositionRoot : IMockCompositionRoot<MockCompositionRoot> {}

Why stop here? Let’s split up CompositionRoot’s fields into individual interfaces. Each concrete service implementation now has its own provider…

interface IConsoleLoggerProvider<TServices> : ILoggerProvider
{
    static ILogger ILoggerProvider.Logger { get; }
        = new ConsoleLogger<TServices>();
}
interface IDbRecipeRepositoryProvider<TServices> : IRecipeRepositoryProvider
{
    static IRecipeRepository IRecipeRepositoryProvider.RecipeRepository { get; }
        = new DbRecipeRepository<TServices>();
}
interface ITopTenRecipeSuggesterProvider<TServices> : IRecipeSuggesterProvider
    where TServices : ILoggerProvider, IRecipeRepositoryProvider
{
    static IRecipeSuggester IRecipeSuggesterProvider.RecipeSuggester { get; }
        = new TopTenRecipeSuggester<TServices>();
}

… and the CompositionRoot just composes the concrete providers by inheritance, once again passing itself as the TServices type parameter. You don’t need to write out the concrete service implementations any more.

class CompositionRoot :
    IConsoleLoggerProvider<CompositionRoot>,
    IDbRecipeRepositoryProvider<CompositionRoot>,
    ITopTenRecipeSuggesterProvider<CompositionRoot>
{
}

Beautiful! (I rather like nesting the concrete providers inside their corresponding concrete service, so, ConsoleLogger<TServices>.Provider instead of IConsoleLoggerProvider<TServices>. You can even nest the abstract providers in the service interfaces: ILogger.Provider.)

You can mix and match if you like: bundle some or all of your services into a single interface, and override them one at a time with inheritance.

interface ICompositionRoot<TServices> :
    IConsoleLoggerProvider<TServices>,
    IDbRecipeRepositoryProvider<TServices>,
    ITopTenRecipeSuggesterProvider<TServices>
    where TServices : ICompositionRoot<TServices>
{
}
class CompositionRoot : ICompositionRoot<CompositionRoot>
{
}
class MockCompositionRoot : ICompositionRoot<MockCompositionRoot>
{
    static IRecipeRepository IRecipeRepositoryProvider.RecipeRepository { get; }
        = new MockRecipeRepository<TServices>();
}

There’s one minor inconvenience with this arrangement: at your app’s entry point, it’s now a little more difficult to use CompositionRoot directly. You have to go via a (constrained) type parameter to access the interface’s static properties.

// Program.cs
GetController<CompositionRoot>().Run();

static IRecipeSuggestionController GetController<T>()
    where T : IRecipeSuggestionControllerProvider
    => T.RecipeSuggestionController;

Writing an additional provider interface for each component in your system (a concrete provider for each concrete service, and an abstract provider for each of their interfaces) is tedious boilerplate, but at least it’s the kind of thing you can automate with a source generator.

Some Thoughts on Modules

Why do we do dependency injection with constructors and interfaces? Object oriented languages expect you to do all your modelling, at every level of your system, using the tools of objects. But in practice I feel there are (at least) two separate types of objects. Honest-to-goodness objects manage data, have a lifetime, may have many instances. But services are really just organisational conveniences — they simply bundle up a set of functions and dependencies, they’re stateless, and there’s typically only one of each service in a given application.

It’s weird that we model both of these concepts with classes! We want to do modular programming: make components loosely coupled, program to well-designed interfaces and hide implementations. But mainstream object oriented languages don’t have tools for programming modules, so we’re stuck reusing constructs that are better suited to managing units of data. Shut up and write classes!

I freely admit my design for TLDI is grotesque. But let’s admit that IOC containers like MEDI are grotesque too. They exist to paper over the deficiency of modular programming in OO languages. It was tedious to write new statements to wire up concrete dependencies for every service in your system, and frameworks like ASP.NET needed a generic way to create instances of service classes, manage their lifetimes, and provide their dependencies. I don’t think IOC registration code is very much more tasteful than new statements, but in any case we should recognise that “instances of service classes” is a thoroughly object-oriented framing of the problem: IOC containers are caused by constructor injection.

I don’t quite understand why today’s industrial strength languages don’t have built-in facilities for managing components and dependencies. I would love to see a language (other than OCaml) take modular programming and dependency injection seriously. Maybe next time I’ll jot down some ideas of how to design such a system.

I learned a lot about Wadler’s algorithm by translating it to an imperative language. I think Wadler’s explanation skims over a couple of interesting behavioural details of his algorithm, and those details are actually a little easier to see in an imperative setting.

You can find (a productionised version of) the code I’m going to present today in my library Gutenberg. As you’ll see, the code fuses together lots of my favourite programming ideas: it’s a combinator library, with a purely functional front end, backed by a logic programming engine, written as a stack machine! Let’s dive in.

Wadler’s Code

Wadler’s library is designed around a datatype of documents. A document represents a block of text with some optional line breaks and indentation. To use the library, you construct a document using the provided combinators, and then invoke the layout algorithm to render the document as a string.

For example, you might construct a document containing a code listing for a function call. If the function’s arguments can all fit on one line then the expression should be flattened,

printf("My %s is named %s.", "cat", "Prune");

but a longer expression might need to be broken up with each argument on its own line.

printf(
    "My %s is named %s. She eats %s",
    "cat",
    "Prune",
    "Tender Chicken Chunks"
);

Typically you’ll annotate your document with several different sets of line break hints. So a given document can be laid out in a number of ways. The layout algorithm’s job is to choose the best layout — meaning one which makes good use of the available horizontal space and doesn’t introduce more line breaks than necessary.

Internally, the notion of a choice of layouts is represented by the :<|> constructor: doc1 :<|> doc2 means “doc1 is a preferable layout, but doc2 can be used as a fallback if doc1 causes the text to overflow the page width”. The layout algorithm takes a document containing :<|>s and produces one without.

The work is done by the function best. Here it is, with some cosmetic tweaks:

best
    :: Int  -- ^ The page width
    -> Int  -- ^ The number of characters already consumed on the current line
    -> DOC  -- ^ The document to lay out
    -> Doc  -- ^ The laid-out document
best w k x = be k [(0, x)]
    where
        be k [] = Nil
        be k ((i, NIL):z) = be k z
        be k ((i, x :<> y):z) = be k ((i, x):(i, y):z)
        be k ((i, NEST j x):z) = be k ((i+j, x):z)
        be k ((i, TEXT s):z) = Text s (be (k + length s) z)
        be k ((i, LINE):z) = Line i (be i z)
        be k ((i, x :<|> y):z)
            = better k
                (be k ((i, x):z))
                (be k ((i, y):z))

        better k x y =
            if fits (w-k) x
            then x
            else y

fits w x | w < 0 = False
fits w Nil = True
fits w (Text s x) = fits (w - length s) x
fits w (Line i x) = True

Let’s try to reconstruct Wadler’s code in an imperative style.

Wadler’s Two Documents

Section 3 of the paper is all about designing an efficient representation for documents. Wadler calls it DOC. It took me a while to grasp the reason for the new name: he’s keeping the old Doc around (in modified form — it lacks the Union constructor).

• DOC is the API. It’s the core representation of documents that users of the library interact with. It’s an abstract type — you create DOCs using the provided combinators, not by calling its constructors directly.
• Doc is an intermediate representation. It’s the result of laying out a document (it’s returned by best). Wadler gives a function (awkwardly named layout) to display a Doc as a string but you can imagine other backends (for example, writing directly to the console).

I think Wadler’s choice of names could use some improvement — the modern prettyprinter library uses SimpleDocStream for the latter. Structurally, Doc(/SimpleDocStream) is a list of literal text chunks interspersed with line-breaks-with-indentation:

data Doc
    = Nil
    | Text String Doc
    | Line Int Doc
-- equivalently,
type Doc = [Either String Int]

But in an imperative language it might be better to conceive of Doc as a sequence of instructions for a rendering backend. In C#, it’s more natural to abstract over backends using an interface, rather than an intermediate data structure:

interface IDocumentRenderer
{
    void WriteText(string text);
    void WriteLineBreak(int indentation);
}

best is a Stack Machine

In Section 3, Wadler somewhat opaquely describes “generalising each operation to work on a list of indentation-document pairs”. He’s talking specifically about be (best’s inner loop). Let’s take a look at a few of its clauses to figure out what he means:

best w k x = be k [(0, x)]
    where
        be k [] = Nil
        be k ((i, NIL):z) = be k z
        be k ((i, x :<> y):z) = be k ((i, x):(i, y):z)

The list is initialised to a single item and thereafter the code only manipulates the front few items. The list is being treated as a stack. This stack represents a work stream — each item in the stack is a fragment of the document which hasn’t been laid out yet. The top item is the next fragment to be laid out.

So let’s write a loop that mutates a stack. (I’ll come back to the :<|> case — it requires special consideration!)

void Layout(Document doc, IDocumentRenderer renderer)
{
    var stack = new Stack<(int, Document)>();
    stack.Push((0, doc));

    while (stack.Count > 0)
    {
        var (indentation, current) = stack.Pop();
        switch (current)
        {
            case Empty:  // NIL
                break;
            case LineBreak:  // LINE
                renderer.WriteLineBreak(indentation);
                break;
            case Text(var s):  // TEXT
                renderer.WriteText(s);
                break;
            case Concat(var l, var r):  // l :<> r
                stack.Push((indentation, r));
                stack.Push((indentation, l));
                break;
            case Nested(var i, var d):  // NEST i d
                stack.Push((indentation + i, d));
                break;
        }
    }
}

For example, here’s a diagram of what happens to the stack when processing a document consisting of two literal strings.

                             +-----------+
                             |   "foo"   |
+---------------------+      +-----------+      +-----------+
| Concat("foo","bar") |      |   "bar"   |      |   "bar"   |
+---------------------+  =>  +-----------+  =>  +-----------+  =>  +-------+
|          ...        |      |    ...    |      |    ...    |      |  ...  |
                                         Output:    "foo"          "foobar"

Backtracking

Now let’s look at the :<|> case.

be k ((i, x :<|> y):z)
    = better k
        (be k ((i, x):z))
        (be k ((i, y):z))

better k x y =
    if fits (w-k) x
    then x
    else y

fits w x | w < 0 = False
fits w Nil = True
fits w (Text s x) = fits (w - length s) x
fits w (Line i x) = True

This is the key part of the layout algorithm: deciding which of two alternative layouts to use, based on whether it causes the document to overflow the available horizontal space.

The stack z represents the rest of the document to be laid out. The code concatenates the first alternative x onto the front of the document z and lays out the document to see if it’ll fit in the allocated page width. If it doesn’t fit, we’ll push the fallback layout y onto the stack and lay that out instead.

It’s important that we attempt to lay out the whole document — x and z, not just x — when seeing if it’ll fit. Choosing the x alternative might cause an overflow later (when processing a document that’s right now buried somewhere below the current top of the stack), even if x fits on its own. In other words this is a backtracking operation: if a layout with x overflows later in the line, we backtrack to the point at which we chose x, forgetting any layout decisions we made while rendering x:z.

This code is tricky to port to an imperative style because it makes central use of immutability and laziness:

• The rest of the document is duplicated. Using z twice like that requires z to be immutable (well, persistent). If we mutate z while laying out the first alternative, by pushing and popping things from it, it won’t be in the same state by the time we decide to fall back on y if it doesn’t fit!
• fits only reads as far as the end of the line, so backtracking is bounded to lineWidth characters. If the current line fits, then there’s nothing to be gained by trying a less-flat alternative. This requires lazy evaluation: we don’t want to lay out the entire document just to see whether the first line fits!
• We need some way of undoing any calls to the IDocumentRenderer that were made in the course of laying out x.

Aside: A Thought On Laziness

Wadler wrote a function that looks like it lays out the entire document, but because the language is lazy it never actually processes more than one line at a time. Cool! But, I dunno, understanding this took me some head-scratching because this important aspect of the control flow is not visible in the text of the code. Laziness can be a mixed blessing — it lets us separate producers and consumers in novel ways, but it makes control flow implicit. Perhaps I’m still just not good enough at thinking lazily.

(I tried implementing the backtracking behaviour using a failure continuation. The code is certainly more complex and uglier than Wadler’s — there are two mutually recursive continuation types! — but it has the benefit of being very explicit about when backtracking can happen and what to do when it does.)

Anyway, we can handle the latter two issues imperatively by buffering the output text one line at a time. If we reach the end of the line without overflowing, we can flush the buffer’s contents out to the IDocumentRenderer.

void Layout(Document doc, IDocumentRenderer renderer)
{
    var buffer = new List<char>();
    var stack = new Stack<(int, Document)>();
    stack.Push((0, doc));

    while (stack.Count > 0)
    {
        var (indentation, current) = stack.Pop();
        switch (current)
        {
            case LineBreak:
                // Flush the buffer
                renderer.WriteText(new string(buffer.ToArray()));
                renderer.WriteLineBreak(indentation);
                buffer.Clear();
                break;
            case Text(var s):
                buffer.Add(s);
                break;
            // ... other cases unchanged
        }
    }
}

What to do in the case of an overflow? We need to reset the state of the layout algorithm to how it was before we tried to lay out x. This involves putting the stack back how it was at that time and also dropping any chunks that were written to the buffer since then.

Let’s record every alternative we’ve tried in a separate stack of choice points. A choice point is a data structure containing a fallback alternative and the current state of the layout algorithm — consisting of the length of the buffer and the state of the stack. We have to make a copy of the stack because it’s mutable.

var choicePoints = new Stack<ChoicePoint>();

// ... in the switch block:
case ChoiceDoc(var x, var y):  // x :<|> y
    choicePoints.Push(new ChoicePoint(
        indentation,
        y,
        buffer.Count,
        Copy(stack)
    ));
    stack.Push((indentation, x));
    break;

Now when an overflow happens we can backtrack to the last choice point. And if we reach the end of the line without overflowing, we can discard any remaining choice points.

case Text(var s):
    buffer.AddRange(s);
    if (buffer.Count > LineWidth && choicePoints.Count > 0)
    {
        Backtrack();
    }
    break;
case LineBreak:
    renderer.WriteText(new string(buffer.ToArray()));
    renderer.WriteLineBreak(indentation);
    buffer.Clear();
    Commit();
    break;


void Backtrack()
{
    var (i, y, bufLen, s) = choicePoints.Pop();
    stack = s;
    buffer.RemoveRange(bufLen, buffer.Count - bufLen);
    stack.Push((i, y));
}

void Commit()
{
    choicePoints.Clear();
}

A Cue From Prolog

Copying the stack when we create a choice point is ugly and inefficient: each copy operation takes linear time and space, and we make a linear number of copies (one for every choice point), resulting in quadratic performance.

I thought about trying to amortise the copying operation with some sort of copy-on-write mechanism. But around the time I was working on this library I’d been reading about Warren’s Abstract Machine — a backend for Prolog implementations — and that gave me a better idea.

In the WAM, choice points are stored on the execution stack interspersed with the predicates’ stack frames. Things are arranged such that choice points are only popped from the stack after backtracking, not during normal execution. This prevents any preceding stack frames from being discarded, so that the state of the call stack can be restored upon backtracking. (The relevant section of the book is Section 4.1, Environment Protection.)

We can apply this idea to the layout engine. The items in the work stream will now be either a Document to be rendered or a choice point.

case ChoiceDoc(var x, var y):
    stack.Push((
        indentation,
        new ChoicePoint(
            y,
            buffer.Count,
            canBacktrack,
            continuation: stack.Count - 1  // see below
        )
    ));
    stack.Push((indentation, x));
    canBacktrack = true;
    break;

The stack transition looks like this (supposing that the Choice document is encountered when the stack has two items):

                          +------------------+
3                         |         x        |
    +--------------+      +------------------+
    |              |      |   ChoicePoint    |
2   | Choice(x, y) |      | Continuation = 1 |
    |              |      |   Fallback = y   |
    +--------------+  =>  +------------------+
1   |     ...      |      |        ...       |
    +--------------+      +------------------+
0   |              |      |                  |

Upon encountering a choice point in the work stream, we’ll leave the choice point where it is and continue rendering the document that’s underneath it. This may result in new document fragments being piled on top of the choice point (and then being processed themselves). That means we’ll likely encounter each choice point multiple times. Each choice point will maintain a continuation pointer to the next document fragment to render in the non-backtracking code path.

case ChoicePoint cp:
    var continuation = GetContinuation(cp.Continuation);
    if (continuation < 0)
    {
        // We’ve processed the whole document!
        renderer.WriteText(new string(buffer.ToArray()));
        return;
    }
    // put the choice point back where it was
    // and process the continuation
    cp.Continuation = continuation - 1;
    stack.Push((indentation, cp));
    stack.Push(stack[continuation]);
    break;

int GetContinuation(int i)
{
    while (i >= 0 && stack[i] is ChoicePoint cp)
    {
        i = cp.Continuation;
    }
    return i;
}

The stack transition looks like this. The document fragment that was at the Continuation index is duplicated to the top of the stack.

                              +------------------+
3                             |        z         |
    +------------------+      +------------------+
    |   ChoicePoint    |      |   ChoicePoint    |
2   | Continuation = 1 |      | Continuation = 0 |
    |   Fallback = y   |      |   Fallback = y   |
    +------------------+  =>  +------------------+
1   |        z         |      |        z         |
    +------------------+      +------------------+
0   |       ...        |      |       ...        |

When backtracking, we’ll discard any stack items on top of the latest choice point.

void Backtrack()
{
    while (stack.Count > 0)
    {
        var (indentation, item) = stack.Pop();
        if (item is ChoicePoint(var y, var bufLen, var b, _))
        {
            stack.Push((indentation, y));
            buffer.RemoveRange(bufLen, buffer.Count - bufLen);
            canBacktrack = b;
            return;
        }
    }
}

Finally, when we reach the end of a line, any choice points (and the document fragments they were protecting) can be discarded. The Empty document does double duty as a tombstone.

void Commit()
{
    // why iterate backwards?
    // https://github.com/benjamin-hodgson/Gutenberg/blob/9ce96d354806ccb7d9c33d89514108045951b3b8/Gutenberg/LayoutEngine.cs#L395-L400
    for (var i = stack.Count - 1; i >= 0; i--)
    {
        var (_, item) = stack[i];
        if (stack[i] is ChoicePoint cp)
        {
            // Everything between a ChoicePoint and
            // its continuation has been written to
            // the renderer.
            for (var j = cp.Continuation + 1; j <= i; j++)
            {
                stack[j] = (0, new Empty());
            }
        }
    }
}

Anyway, here’s one of the incremental build issues I fixed. It’s a bit outside my usual wheelhouse but I wanted to write it down because I couldn’t find anything online when I was first investigating this issue.

Debugging the Up-To-Date Check

I noticed that my team’s project wasn’t building incrementally: whenever I tried to re-run a unit test I had to wait for the whole project to recompile. The first thing to do, when you notice yourself feeling annoyed by recompilation, is to enable Up-To-Date Check Logging in Visual Studio. (I recommend just leaving the logging on “minimal” mode at all times.) The “Up-To-Date Check” is the component of Visual Studio which is responsible for incremental builds. It works by looking at the timestamps of the project’s input and output files; if all of the output files are newer than all of the input files then the project doesn’t need to be rebuilt.

After turning on the logging and rebuilding, I saw this message in the Output window:

FastUpToDate: Input Compile item 'CodeGen\Templates\AdapterTextTemplate.cs' is newer than earliest output 'bin\net472\ScopeCompiler.dll', not up-to-date.

That AdapterTextTemplate.cs file is generated using T4 Text Templating. We’d set T4 up to run as part of the build, so we didn’t need to check in the generated files and you didn’t have to manually run T4 before building. We did this by setting TransformOnBuild in the csproj file, and adding a Target to include any newly generated cs files:

<PropertyGroup>
    <!--
        In MSBuild parlance, a "property" is a scalar (string) value.
        Anything appearing inside a `PropertyGroup` is a property.
        The `TransformOnBuild` property tells T4 to run at build time.
    -->
    <TransformOnBuild>true</TransformOnBuild>
</PropertyGroup>

<ItemGroup>
    <!--
        MSBuild manages lists of "items"; an item roughly corresponds
        to a file. Inside an `ItemGroup` you can add items to a list
        using `Include`. So this line of code adds all of the `tt`
        files in the `CodeGen\Templates` folder to the list of
        `T4Transform` items.
    -->
    <T4Transform Include="CodeGen\Templates\*.tt" />

    <!-- Make the .tt files visible in Visual Studio -->
    <None Include="CodeGen\Templates\*.tt" />
</ItemGroup>

<!-- This `targets` file contains code to read the `T4Transform` item list and invoke T4. -->
<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\TextTemplating\Microsoft.TextTemplating.targets" />

<!--
    A "target" is a coarse-grained unit of work —
    a stage in MSBuild's pluggable pipeline.

    This target runs in between the `ExecuteTransformations`
    target (from `Microsoft.TextTemplating.targets`) and the
    standard `Compile` target. It adds any output files that
    were generated by T4 (the `GeneratedFiles` item list)
    to the list of `Compile` items.
-->
<Target Name="IncludeT4GeneratedFiles"
        AfterTargets="ExecuteTransformations"
        BeforeTargets="Compile">

    <ItemGroup>
        <!-- Don't include already included files -->
        <Compile Include="@(GeneratedFiles)" Exclude="@(Compile)" />
    </ItemGroup>
</Target>

I realised that the incremental build failure was due to an interaction between build-time text templating and MSBuild’s default includes. The Up-To-Date Check scans the source tree for any cs files; if any of them have changed (or been created) since the last time a build ran, then the project needs rebuilding. Then, during the build (before compilation), T4 runs and generates some cs files; by default these cs files are placed alongside the tt file — ie, in the source tree. The next time you run a build, Visual Studio sees that the generated cs files have been touched, meaning the project needs to be rebuilt, meaning that T4 needs to run, which touches the generated cs files…

The Fix

For the fix I made a few changes:

1. I excluded the generated cs files from the default list of files to track in the Up-To-Date Check.
2. Instead, I included them dynamically, after T4 has run. (Visual Studio doesn’t look at dynamically added items when determining whether a project needs rebuilding.)
3. I configured MSBuild to run T4 only if the input tt files have changed.

Excluding the Generated Files

One way to do this would be to tell T4 to put the generated files in the obj directory by default:

<!-- An `ItemDefinitionGroup` contains default metadata definitions for items. -->
<ItemDefinitionGroup>
    <T4Transform>
        <OutputFilePath>$(MSBuildProjectDirectory)$(IntermediateOutputPath)</OutputFilePath>
    </T4Transform>
    <T4Preprocess>
        <OutputFilePath>$(MSBuildProjectDirectory)$(IntermediateOutputPath)</OutputFilePath>
    </T4Preprocess>
</ItemDefinitionGroup>

(As far as I can tell the OutputFilePath metadata is not documented anywhere; I found it by decompiling the TransformTemplatesBase task in Microsoft.TextTemplating.Build.Tasks.dll.)

This probably would’ve been the cleaner way to do it, as it doesn’t pollute the source tree with generated files, but I didn’t want to break my teammates’ workflow by making it harder to find the generated code. Instead, I simply removed the generated files’ Compile items:

<ItemGroup>
    <!-- by convention, an underscore denotes private implementation details -->
    <_T4OutputFile Include="@(T4Preprocess -> '%(RelativeDir)%(Filename).cs')" />
    <_T4OutputFile Include="@(T4Transform -> '%(RelativeDir)%(Filename).cs')" />
    <Compile Remove="@(_T4OutputFile)" />
</ItemGroup>

T4 can generate any text file, not just cs files, so you’d need to tweak this code if your project happens to have T4 templates which generate other files.

Including the Generated Files Dynamically

Top-level properties and items are evaluated up-front, at the start of a build, but you can also manipulate properties and item lists dynamically by nesting them inside a Target. So all I had to do was plug in to the TransformDuringBuild target (implemented in Microsoft.TextTemplating.targets) and include the _T4OutputFiles after they’d been generated — much like the IncludeT4GeneratedFiles target I mentioned earlier.

I could’ve used AfterTargets="TransformDuringBuild" but instead I decided to override the TransformDuringBuild target. (To override a target, you just define a new target with the same name — last one wins. Overriding targets is usually not a good idea, but I had a good reason which will shortly become apparent.) The standard TransformDuringBuild target is defined like this:

<Target
    Name="TransformDuringBuild"
    Condition="'$(TransformOnBuild)' == 'true'"
    BeforeTargets="CoreCompile"
    DependsOnTargets="TransformAll">
</Target>

It’s a no-op target which simply causes another target (TransformAll) to be run when the TransformOnBuild property is set. So, for my override, I pasted that code and added an ItemGroup to the target’s body:

<Target
    Name="TransformDuringBuild"
    Condition="'$(TransformOnBuild)' == 'true'"
    BeforeTargets="CoreCompile"
    DependsOnTargets="TransformAll">

    <ItemGroup>
        <Compile Include="@(_T4OutputFile)">
            <AutoGen>True</AutoGen>
            <DesignTime>True</DesignTime>
        </Compile>
        <!--
            Add `FileWrites` items in order to make the generated
            files get deleted by Visual Studio's "clean" operation
        -->
        <FileWrites Include="@(_T4OutputFile)" />
    </ItemGroup>
</Target>

Running T4 Incrementally

MSBuild supports incremental building at the target level (so, finer-grained than a whole project). You can tell MSBuild to skip a target if its input files haven’t changed, using the Inputs and Outputs attributes. If all of the files listed in the Outputs are newer than the Inputs, then MSBuild will skip the target to save time. (Any items and properties defined in the target will still be included — it caches them from the last time the target was run.) This is why I decided to override the TransformDuringBuild target above — I wanted to give it Inputs and Outputs.

My final TransformDuringBuild target looked like this:

<Target
    Name="TransformDuringBuild"
    Condition="'$(TransformOnBuild)' == 'true'"
    BeforeTargets="CoreCompile"
    DependsOnTargets="TransformAll"
    Inputs="@(T4Preprocess);@(T4Transform)"
    Outputs="@(_T4OutputFile)">

    <ItemGroup>
        <Compile Include="@(_T4OutputFile)">
            <AutoGen>True</AutoGen>
            <DesignTime>True</DesignTime>
        </Compile>
        <FileWrites Include="@(_T4OutputFile)" />
    </ItemGroup>
</Target>

The Code

I put all of this code in a file called TextTemplating.CSharp.targets and imported it into the projects which used T4.

Here is the final file in handy pasteable form:

<Project>

    <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\TextTemplating\Microsoft.TextTemplating.targets" />

    <ItemGroup>
        <Service Include="{508349b6-6b84-4df5-91f0-309beebad82d}" />
        <AvailableItem Include="T4Preprocess" />
        <AvailableItem Include="T4Transform" />
    </ItemGroup>

    <ItemGroup>
        <_T4OutputFile Include="@(T4Preprocess -> '%(RelativeDir)%(Filename).cs')" />
        <_T4OutputFile Include="@(T4Transform -> '%(RelativeDir)%(Filename).cs')" />

        <!--
            Don't include the generated files before they've been generated
            as it causes incremental build failure. Instead, include them after
            running T4, in the (overridden) TransformDuringBuild target
        -->
        <Compile Remove="@(_T4OutputFile)" />

        <!-- make visible in visual studio -->
        <None Include="@(T4Preprocess);@(T4Transform)" />
        <None Include="@(_T4OutputFile)" />
    </ItemGroup>

    <!-- Override the target from MS.TextTemplating.targets in order to set the Inputs/Outputs -->
    <Target
        Name="TransformDuringBuild"
        Condition="'$(TransformOnBuild)' == 'true'"
        BeforeTargets="CoreCompile"
        DependsOnTargets="TransformAll"
        Inputs="@(T4Preprocess);@(T4Transform)"
        Outputs="@(_T4OutputFile)">

        <ItemGroup>
            <Compile Include="@(_T4OutputFile)">
                <AutoGen>True</AutoGen>
                <DesignTime>True</DesignTime>
            </Compile>
            <FileWrites Include="@(_T4OutputFile)" />
        </ItemGroup>
    </Target>

</Project>

That’s the general recipe when you need to generate something at build time: write a Target with Inputs and Outputs. Put an ItemGroup inside the target to inform the rest of the build pipeline about the newly generated bits.

data SourcePos = SourcePos {
    line :: Natural,
    col :: Natural
} deriving (Eq, Ord)

startOfFile :: SourcePos
startOfFile = SourcePos 1 1

The parser keeps track of the current SourcePos by looping over the characters in the input file and calling a method called calculateSourcePos. calculateSourcePos’s job is to update the current SourcePos for a single character.

newSourcePos = foldl' calculateSourcePos oldSourcePos inputText

You can supply your own calculateSourcePos implementation, but the default one looks roughly like this:

calculateSourcePos :: SourcePos -> Char -> SourcePos
calculateSourcePos sp '\n' = SourcePos {
    line = line sp + 1,  -- increment the line count
    col = 1  -- and reset the column count
}
calculateSourcePos sp _ = sp { col = col sp + 1 }

When the parser encounters a new line, the column counter is reset back to 1.

The signature of calculateSourcePos poses a performance problem: it takes the previous SourcePos as an argument. Each iteration of the loop depends on the result of the previous iteration. This data dependency means the loop can’t be parallelised — you have to wait for each iteration of the loop to finish before you can start the next one.

This article is about redesigning SourcePos to be more parallelisable.

Monoids are Embarrassingly Parallel

Monoid is the high-falutin’ mathsy name for a certain variety of composable objects. Composability is very important in programming, and consequently monoids show up all over the place. There are plenty of introductions to monoids out there (mostly by better teachers than me), so I’ll keep it brief.

For an object to be a monoid you need two things:

1. An operator <> which combines two values of your type into a bigger value.

   • It shouldn’t matter how you nest a sequence of <> operations: (x <> y) <> z == x <> (y <> z).

2. A mempty value, representing some notion of “zero”.

   • Combining mempty with another value should leave that value unchanged: mempty <> x == x == x <> mempty.

class Monoid m where
    mempty :: m
    (<>) :: m -> m -> m

The rule about nesting <> is what makes monoids good for parallelism. Suppose you have a big array of monoidal values, and you want to combine them all into a single value using <>. It doesn’t matter what order you perform the additions in, so you can safely divide your array into chunks, sum up each chunk in parallel, and then combine the results. (This is the basic idea behind MapReduce.)

mconcat :: Monoid m => Vector m -> m
mconcat = Vector.foldr (<>) mempty

-- performs the same computation but in parallel
pmconcat :: Monoid m => Vector m -> m
pmconcat = List.foldr (<>) mempty
    . parMap mconcat
    . chunksOf 1024

To put it another way, monoids don’t suffer from the data dependency which made calculateSourcePos hard to parallelise. The challenge, then, is to come up with a way to make SourcePos into a monoid.

Deltas

If you go hiking, you might bring with you a list of directions on a piece of paper. Directions are monoidal: if you have directions from your house to the beach, and from the beach to the pub, you can follow those directions sequentially to get from your house to the pub. (I’ll tolerate hiking as long as the destination is a pub.)

Directions are relative, not absolute. The directions on your paper might tell you how to get from your house to the pub, but if you start somewhere other than your house you can still follow the directions — you just have to hope you end up in a different pub.

So by analogy, let’s stop worrying about absolute locations in a source file and instead think about offsets relative to an arbitrary location.

data SourcePosDelta = SourcePosDelta {
    lines :: Natural,
    cols :: Natural
}

You can add a relative SourcePosDelta to an absolute SourcePos to get a new absolute SourcePos. This is analogous to setting off on a hike from a given starting location.

add :: SourcePos -> SourcePosDelta -> SourcePos
add sp delta = SourcePos {
    line = line sp + lines delta,
    col = (if lines delta == 0 then col sp else 0) + cols delta
}

When the delta spans multiple lines, we discard col sp and only take cols delta. This reflects the behaviour of computeSourcePos, which resets the column counter when a new line is encountered. The asymmetry is interesting, though; the line calculation depends only on the two lines fields, but the col field has some interference from the lines.

Likewise, you can find the difference between two SourcePoses to get the path that would take you from one to the other:

subtract :: SourcePos -> SourcePos -> SourcePosDelta
subtract start end = SourcePosDelta {
    lines = line end - line start,
    cols = col end - (if line end == line start then col start else 0)
}

From here we can see how to add a pair of SourcePosDeltas (and write the Monoid instance). In fact, the code is more or less identical to add:

instance Monoid SourcePosDelta where
    mempty = SourcePosDelta 0 0
    
    delta1 <> delta2 = SourcePosDelta {
        lines = lines delta1 + lines delta2,
        cols = (if lines delta2 == 0 then cols delta1 else 0) + cols delta2
    }

I’ll leave it up to you to convince yourself that this definition satisfies the monoid laws.

Each character in the input file corresponds to a small SourcePosDelta: '\n' corresponds to SourcePosDelta 1 0 and each other character corresponds to SourcePosDelta 0 1. The parser can map each character to a SourcePosDelta, add up the monoidal deltas (possibly in parallel), and then add the result to the SourcePos corresponding to the start of the inputText.

calculateSourcePosDelta inputText = pmconcat $ parMap toDelta inputText
    where
        toDelta '\n' = SourcePosDelta 1 0
        toDelta _ = SourcePosDelta 0 1

Hardware Parallelism

I tested some implementations of <> (in C#) which made use of modern CPUs’ support for hardware parallelism (also known as SIMD, for Single Instruction Multiple Data). Recent versions of .NET come bundled with opt-in SIMD support, to be found in the System.Numerics namespace.

CPUs with SIMD have extra-wide registers (like, 256 bits) divided into lanes. You can stuff four 64-bit integers (or eight 32-bit ones) into a single register, and the CPU supports special instructions to perform the same operation on each of those four integers at once — so (for example) you can add four sets of two integers with a single instruction. Each SIMD operation is a little bit slower than its corresponding single-target instruction, but since they operate on four times the amount of data you can often get reasonable speedups when you’re processing lots of data.

My fastest attempt involved packing SourcePosDelta’s two integers into a single 64-bit integer and doing some bit manipulation to implement <>’s “annihilation” semantics. Then I divided the input array into four chunks — one per lane — and ran the bit manipulation algorithm on each lane. Since it’s best to load contiguous data into a SIMD register’s lanes, I had to rearrange the input array to line the chunks up with their SIMD lanes.

I decided against putting the SIMD code into the library, because in practice the number of SourcePosDeltas which need summing is often not very large. In any case, the SIMD code offered comparable performance to a much simpler algorithm: loop backwards until you find the last newline in a chunk of text, then ignore any non-newline characters to the left of it.

Spans and Shifts

Here’s another cool thing you can do with SourcePosDelta.

In general, each node in a parse tree corresponds to a certain span of the input document: the section of the input text containing a given syntactic construct.

data Span = Span {
    start :: SourcePos,
    width :: SourcePosDelta
}

end :: Span -> SourcePos
end span = add (start span) (width span)

containsPos :: Span -> SourcePos -> Bool
span `containsPos` pos = pos >= start span && pos <= end span

It’s common in compilers to keep track of these spans in order to report error messages. For example, you might want to colour a piece of code red in an IDE if it contains an error.

data Node = Node {
    span :: Span,
    data :: Expr
}
data Expr = Lit Int | Add Node Node

In an interactive IDE, you have a programmer typing code; the IDE’s parser needs to respond in real time to each keystroke. So, for performance’s sake, you only want to re-parse the fragment of code that the user is currently typing, and reuse the rest of the document’s parse tree from the last time you parsed it.

But when the user types characters into the middle of a document, all of the constructs after it get moved right and down. So you need to shift the parse tree over by the amount that the user typed.

shift :: SourcePosDelta -> Node -> Node
shift d (Node span data) = Node (shiftSpan d span) (shiftData data)
    where
        shiftData l@(Lit _) = l
        shiftData (Add l r) = Add (shift d l) (shift d r)

shiftSpan :: SourcePosDelta -> Span -> Span
shiftSpan d (Span start width) = Span
    -- we need to add `d` on the _left_ of `start`,
    -- so we can't use `add start d`. Instead,
    -- round-trip via `SourcePosDelta`.
    (add startOfFile (d <> subtract startOfFile start))
    width

shift rebuilds the entire tree! That’s asymptotically as bad as re-parsing the entire file — clearly not something we want to do on every keystroke. Instead, let’s annotate the tree with the amount by which it’s been shifted. This lets us shift a whole subtree at once without mutating it.

data Node = Node {
    shiftedBy :: SourcePosDelta,
    originalSpan :: Span,
    expr :: Expr
}

span :: Node -> Span
span (Node shift span _) = shiftSpan shift span

shift :: SourcePosDelta -> Node -> Node
shift delta node = node { shiftedBy = delta <> shiftedBy node }

shift is very fast now; it only rebuilds a single node. When the compiler traverses the parse tree, it simply needs to keep track of the current shift amount and apply that shift to any locations reported in error messages.

For completeness, here is some code to apply an edit to a parse tree. edit searches the parse tree to find the node where the edit occurred, replaces that node, and shifts everything that was textually to the right of that node.

editSpan :: Span  -- ^ The location and amount the user typed
         -> Span  -- ^ The original span
         -> Maybe Span  -- ^ The span with the edit applied
editSpan edit span
    | not (span `containsPos` start edit) = Nothing
    | otherwise = Just $ Span
        (start span)
        -- insert `edit`'s `SourcePosDelta` into `span`'s
        (subtract (start span) (start edit)
            <> width edit
            <> subtract (end span) (end edit))

edit :: Span  -- ^ The location and amount the user typed
     -> Expr  -- ^ The replacement parse tree
     -> Node  -- ^ The original parse tree
     -> Node  -- ^ The edited parse tree
edit s replacement node = Node
    mempty
    -- this `fromJust` is ok assuming that the edit was inside `span node`
    (fromJust $ editSpan s (span node))
    (editExpr (expr node))
    where
        editExpr (Lit _) = replacement
        editExpr (Add l r)
            | span l `containsPos` start s =
                Add (edit s replacement l) (shift (width s) r)
            | span r `containsPos` start s =
                Add l (edit s replacement r)
            | otherwise = replacement

This code is linear in the depth of the tree, which is usually shallow (typically dozens of nodes deep, rather than hundreds or thousands). You could do even better asymptotically by storing the parse tree in a zipper focused on the node the user is currently editing, and lazily apply shifts as the zipper moves around.

Monoid Actions

Let’s look a little closer at the formalism underlying SourcePosDelta.

This idea of building up a monoid and then using it to transform some other value is formalised using the notion of actions (also known as modules). A monoidal object is said to be a monoid action on some other object a if it can be used to transform a:

-- "Action a m" means "m acts on a"
class Monoid m => Action a m where
    act :: m -> a -> a

I always thought action monoid would make a better name for this concept than monoid action. The monoidal value m represents an action to be carried out on some other object a.

To be a well-behaved monoid action, the act method should respect the monoidal nature of the action:

1. mempty should be a no-op action: act mempty == id.

2. Building a monoidal value with <> and then acting with it should be the same as acting with the individual pieces of the monoid: (m <> n) `act` x == m `act` (n `act` x).

   • This rule defines a “left” monoid action. It turns out that SourcePosDelta is actually a “right” monoid action, for which this rule is flipped: (m <> n) `act` x == n `act` (m `act` x).

   • In fact, let’s separate the ideas of “left” and “right” actions. I find that the composition law for right actions makes more sense when the arguments are the other way around.

     class Monoid m => LAction a m where
         lact :: m -> a -> a
     
     class Monoid m => RAction a m where
         ract :: a -> m -> a

     So now RAction’s composition law reads x `ract` (m <> n) == (x `ract` m) `ract` n.

Basically, these laws assert that you can build actions with <> and it’ll behave in the way you expect.

Here are a couple of examples to help you think about monoid actions.

1. Yes, SourcePosDelta is a monoid action on SourcePos. Adding a SourcePosDelta to a SourcePos gives you a new SourcePos representing somewhere later in the file — a SourcePosDelta represents the action of moving to a later location. You can convince yourself that the definition of ract does indeed satisfy the laws I outlined above:

   instance RAction SourcePos SourcePosDelta where
       ract sp delta = SourcePos {
           line = line sp + lines delta,
           col = (if lines delta == 0 then col sp else 0) + cols delta
       }

2. I first learned about monoid actions (well, group actions) during a crystallography course at university. Crystallographers are very concerned with symmetry, because crystals are repeating structures which look the same in every direction. Crystals can be categorised by the collection of rotations, reflections and translations (the space group) under which the crystal looks the same. You monoidally build up a sequence of spatial transformations, and then use those transformations to act on a crystal.

   A more familiar way of phrasing the same example: When you go hiking with written directions, you can think of the directions as acting on your current location. When you follow a direction such as “walk 200 metres”, you update your location accordingly.

3. Any monoid can always be thought of as acting upon itself, simply by defining act = (<>).

Actions All the Way Down

It turns out monoid actions can also explain the strange asymmetry in SourcePosDelta’s (<>) method. When computing delta1 <> delta2, we can think of delta2’s lines as acting on delta1’s cols.

newtype Lines = Lines Natural deriving Monoid via (Sum Natural)
newtype Cols = Cols Natural deriving Monoid via (Sum Natural)

instance RAction Cols Lines where
    ract c (Lines 0) = c
    ract _ _ = Cols 0

Lines’s action on Cols is to erase the Cols altogether when the number of Lines is not zero. It’s a weirdly degenerate monoid action, but it’s an action nonetheless. (It seems like Lines basically represents an annihilator for Cols.)

With Lines’s action on Cols in hand, SourcePosDelta’s Monoid instance can be explained as an instance of a (right) semi-direct product. A semi-direct product is like a regular product type (a, b), except its Monoid instance allows for one of the fields to interfere with the other by acting on it:

data RSemiDirect a b = RSD a b

instance (Monoid a, Monoid b, RAction a b) => Monoid (RSemiDirect a b) where
    mempty = RSD mempty mempty

    (RSD x1 y1) <> (RSD x2 y2) = RSD
        (ract x1 y2 <> x2)
        (y1 <> y2)

RSemiDirect a b is a monoid when b’s action on a respects a’s monoidal structure:

1. Acting on an empty monoid produces another empty monoid: ract mempty x == mempty.

2. b’s monoidal structure distributes over a’s: ract (x <> y) z == ract x z <> ract y z.

I won’t prove it here, but these do hold for Lines’s action on Cols. So we have a clean explanation for SourcePosDelta’s asymmetric monoidal structure in terms of the semi-direct product:

type SourcePosDelta = RSemiDirect Cols Lines

I first read about semi-direct products a few years ago in the “twisted functors” paper. They use semi-direct products to manage pointer arithmetic while writing to buffers — monoidally building up an offset and using it to act on a base pointer. They also give a couple of other examples.

I’m a way off having a final design, but I thought I’d publish my unfinished notes, hastily written in an afternoon. You’ll see a few Thinking Emojis throughout this article, indicating things I haven’t figured out yet. Please do get in touch if any of this gives you ideas.

A quick look at some prior art

unbound/unbound-generics

• Name is an abstract data type with fixed representation.
  • Bound is a depth and an index. Requires patterns to have a canonical ordering, and bind each name only once
  • Bound discards the original name, but library API expects patterns to contain Names
• Statically encoded patterns with built in type combinators.
  • I like:
    • Statically declare structure of binders
    • Embed your own datatypes as required by language syntax — method header can be [(Name, Type)]
  • I don’t like:
    • Generics magic to find/traverse patterns
    • Type constructor soup, eg | LetRec (Bind (Rec [(Name, Embed Expr)]) Expr). Would be even noisier in C#.
• Quite closely integrated with generics libs
  • Most users will be using RepLib/GHC.Generics
  • Lib uses RepLib/GHC.Generics internally to traverse patterns

bound

• Choose your own name (data Expr a = Var a | ...).
• Choose your own pattern/index.
  • Library explicitly doesn’t manage patterns or names. It just does substitution using your monad. Scope doesn’t contain a “pattern” object.
  • Me gusta
• Nested datatype manages de Bruijn depth statically.
  • Nested datatype is quite awkward in practice — doesn’t play nicely with mutual recursion or Plated.
  • Clever trick in Scope to speed up shifting at cost of canonicity. Prob a bit of a gimmick unless you’re dealing with huge trees — but also, not super costly complexity-wise.
• Works really nicely with standard classes: Foldable/Traversable to look at FVs, Functor for renaming, Monad for substitution. Eq1 for alpha equivalence, etc. Very pleasing.
  • OTOH, doesn’t work with a non-monadic AST.
• Overall, probably not such a good fit for C# since we don’t have HKTs, Monad etc. Not very library-able.

moniker

• More or less a port of unbound.
• Choose your own name (Free<N>/Bound<N>; trait BoundTerm<N>).
• Library defines a couple of traits (BoundTerm/BoundPattern) to locate variables, do substitution, etc. unbound uses generics for this.
• Traverse your own terms, but derive magic assists with implementing the library’s traits.

Representing Names

I definitely want to use de Bruijn indexes (managed by the library) for bound variables.

readonly struct Bound
{
    public int Depth { get; }  // how many binding sites up should I look?
    public int Index { get; }  // where inside the binding site should I look?
}

For now, I’m not storing names in Bound. If you want to know the original name, go and find it in the pattern. Use of int for Index requires patterns to have a canonical ordering (and bind each variable only once).

Free variables: need an easy way to freshen them.

readonly struct Free
{
    public string OriginalName { get; }
    public int Freshness { get; }
}

Now just union them to represent names.

readonly struct Name
{
    public Bound Bound { get; }
    public Free Free { get; }
    public Name(Bound bound)
    {
        Bound = bound;
        Free = default;
    }
    public Name(Free free)
    {
        Bound = default;
        Free = free;
    }
    public bool IsBound => free.OriginalName == null;
    public bool IsFree => free.OriginalName != null;
}

How much of this should be public? Should these be structs or classes? (Name is like 4 words.) 🤔

Representing Scopes

I definitely want an explicit type to represent a scope. I personally think BindingSite is a slightly better name, although perhaps that should be reserved for names inside patterns? 🤔

The idea is to treat BindingSite as an abstract data type. To create a BindingSite (such as when parsing a lambda expression), you call Binder.Bind(variables, body). The binder traverses the body AST, looking for variables which are bound by the lambda and converting them to Bound de Bruijn indexes. Likewise, when you need to go inside a BindingSite, you have to explicitly unbind it.

Don’t want to discard the original names, of course.

Idea 1: Store the original name with the Bound variable (just as an annotation, ignore in alpha-equivalence etc).

readonly struct Bound
{
    public int Depth { get; }
    public int Index { get; }
    public Free OriginalName { get; }
}

Potential consistency issues — different mentions of the same variable could end up with different tags.

Idea 2: Store a flat list of original names in the BindingSite.

readonly struct BindingSite<T>  // aka Scope (bound/moniker) or Bind (unbound)
{
    // How much of this should be public? 🤔
    public ImmutableArray<Name> OriginalNames { get; }
    public T Body { get; }
}

It feels kinda ugly. Also potential consistency issues — if your binding sites have more syntactic structure than that (patterns, type signatures, etc), you’re gonna end up duplicating information. Also, it doesn’t seem like this would easily support recursive patterns such as letrec or Agda’s dot notation.

Idea 3: Store the actual pattern in the BindingSite. This is more in line with how unbound/moniker do it.

readonly struct BindingSite<T>
{
    public Pattern Pattern { get; }
    public T Body { get; }
}
abstract class Pattern {}
class Variable : Pattern  // Do I need to support "anonymous" variables? 🤔
{
    public Name Name { get; }
}
class Embed<T> : Pattern
{
    public T Embedded { get; }
}
class Telescope : Pattern  // known as Rebind in other libs
{
    public BindingSite<Pattern> BindingSite { get; }
}
class Rec : Pattern
{
    // technically this is a binding site too. 🤔
    public Pattern Body { get; }
}

Some things about this work really well. We can make Pattern extensible by having it implement IRewritable abstractly. I define a few base patterns which my library recognises, and you can add your own. To (eg) find the variables in a pattern I just need to be able to traverse your custom pattern types with GetChildren.

abstract class Pattern : IRewritable<Pattern>
{
    // ...
    public IEnumerable<Free> GetNames()
        => this
            .SelfAndDescendants()
            .OfType<Variable>()
            // names which aren't free in a pattern are recursive
            // references to other names bound in this pattern
            .Where(v => v.Name.IsFree)
            .Select(x => x.Name.Free);
}

Ways this doesn’t scale:

• If your language has multiple types of patterns (eg, type variables and value variables) you can’t statically distinguish them.
• The structure of the pattern lives only at runtime — it all gets stuffed into a Pattern-typed variable. You can’t statically encode the syntactic structure of the binding site (at which point it’s not a whole lot better than Idea 2).
• “Rewrite all the embedded terms” is an ad-hoc affair. Long-standing shortcoming of IRewritable, for which I haven’t come up with a satisfactorily simple solution.

What should be the advice for implementing IRewritable on objects containing a binding site? Should you traverse to the body? (Problematic because de Bruijn indexes from different scopes are not comparable.) Are you meant to unbind the body? That would mean you can’t use IRewritable as is, because IRewritable doesn’t have a parameter for a source of fresh names. Perhaps we should be using IRewriter and not IRewritable, although that’s a messier API. 🤔

This ☝🏻 is a pressing concern because Telescope features a BindingSite!

Idea 3.1: Statically type the Pattern using generics.

readonly struct BindingSite<TPattern, T> where TPattern : Pattern
{
    public TPattern Pattern { get; }
    public T Body { get; }
}
class Telescope<L, R> : Pattern
    where L : Pattern
    where R : Pattern
{
    // open question on how to treat BindingSite with IRewritable
    public BindingSite<L, R> BindingSite { get; }
}
class Rec<T> : Pattern
    where T : Pattern
{
    public T Body { get; }
}

Some things about this work really well too, as evidenced by its use in unbound/moniker. I find the “type soup” usability issue a bit concerning, but more pressingly it doesn’t play nicely with IRewritable.

class Telescope<L, R>
{
    // ...
    public int CountChildren() => 2;
    public void GetChildren(Span<Pattern> children)
    {
        // debatably we should only be descending to the Pattern,
        // because the Body is in a different scope
        children[0] = BindingSite.Pattern;
        children[1] = BindingSite.Body;
    }
    public Pattern SetChildren(ReadOnlySpan<Pattern> children)
        => new Telescope<L, R>((L)children[0], (R)children[1]);
        //                      ^ damn!
}

That unsafe cast really is unsafe, because a Rewrite operation is liable to change the type of the pattern. Maybe patterns should be read-only? Do I need to split the read/write parts of IRewritable? 🤔

Idea 4: A reflection-based API, perhaps with attributes? Seems like it definitely has legs, but I find reflection-based APIs distasteful — they’re hard to reason about and hard to program with. (It’s much easier to manipulate values than code.) I’d prefer to come up with a model, which you can manipulate directly, and then layer a reflection API on top.

ASTs with Binding Structure

This part is fairly worked out I think. I’m assuming that your syntax tree has a case for variables, containing a Name (and no children) and a case for binding sites, containing a BindingSite (and no children). So all we need to add to IRewritable is a way to identify those nodes:

interface IBindable<T> : IRewritable<T> where T : IBindable<T>
{
    bool TryGetName(out Name name);
    T SetName(Name newName);

    bool TryGetBindingSite(out BindingSite<T> bindingSite);
    T SetBindingSite(BindingSite<T> newBindingSite);
}

Not so different than IUnifiable.

Freshening

When you go under a binder you (usually) want to replace the de Bruijn indexes in there with named variables. Those variables need to be fresh, that is, they need to not clash with (or capture) any other variables in scope. As a UX concern, you probably want the fresh name to be somewhat similar to the name the programmer typed.

So we have IFreshener:

interface IFreshener
{
    Free Freshen(Free name);
}

And a straightforward implementation with a global counter.

class Freshener : IFreshener
{
    private int _counter = 0;
    public Free Freshen(Free name)
    {
        _counter++;  // Interlocked.Increment if you need thread safety
        return new Free(name.OriginalName, _counter);
    }
}

As an optimisation (and as a UX improvement), you can avoid freshening variables if you’re sure they don’t clash with any other names which are in scope. That means keeping track of the set of names we want to avoid:

interface IFreshener
{
    Free Freshen(Free name);
    void EnterScope(IEnumerable<Free> names);
    void ExitScope();  // free up the names from the last EnterScope call
}
class Freshener : IFreshener
{
    public Free Freshen(Free name) { /* ... */ }
    public void EnterScope(IEnumerable<Free> names) {}
    public void ExitScope() {}
}
class LocalFreshener : IFreshener
{
    // use a better data structure in practice, obv
    private readonly Stack<IEnumerable<Free>> _names = new();
    public Free Freshen(Free name)
    {
        while (_names.Any(ns => ns.Contains(name)))
        {
            name = name.IncrementFreshness();
        }
        return name;
    }
    public void EnterScope(IEnumerable<Free> names)
    {
        _names.Push(names);
    }
    public void ExitScope()
    {
        _names.Pop();
    }
}

Traversing the tree

OK, finally we have everything we need to bind and unbind variables in ASTs. Let’s suppose we went with Idea 2 from above — just storing a flat list of names at the binding site.

static class Binder
{
    public static BindingSite<T> Bind<T>(ImmutableArray<Free> variables, T body) where T : IBindable<T>
    {
        var variablesLookup = variables
            .Select((x, i) => new KeyValuePair<Free, int>(x, i))
            .ToImmutableDictionary();

        var level = 0;
        T Go(T x)
        {
            if (x.TryGetName(out var name))  // x is a variable
            {
                if (name.IsFree && variablesLookup.TryGetValue(name.Free, out var ix))
                {
                    return x.SetName(new Name(new Bound(level, ix)));
                }
                return x;
            }
            else if (x.TryGetBindingSite(out var bs))  // x is a binding site
            {
                level++;
                var newBody = Go(bs.Body);
                level--;
                return x.SetBindingSite(bs.WithBody(newBody));
            }
            else
            {
                return x.RewriteChildren(Go);
            }
        }
        return new BindingSite<T>(variables, Go(body));
    }

    public static (ImmutableArray<Free>, T) Unbind<T>(BindingSite<T> bindingSite, IFreshener freshener) where T : IBindable<T>
    {
        var variables = bindingSite
            .Variables
            .Select(freshener.Freshen)
            .ToImmutableArray();

        var level = 0;
        T Go(T x)
        {
            if (x.TryGetName(out var name))
            {
                if (name.IsBound && name.Bound.Level == level)
                {
                    return x.SetName(new Name(variables[name.Bound.Index]));
                }
                return x;
            }
            else if (x.TryGetBindingSite(out var bs))
            {
                level++;
                var newBody = Go(bs.Body);
                level--;
                if (!ReferenceEquals(bs.Body, newBody))
                {
                    return x.SetBindingSite(bs.WithBody(newBody));
                }
                return x;
            }
            else
            {
                return x.RewriteChildren(Go);
            }
        }
        return (variables, Go(bindingSite.Body));
    }
}

You can develop versions of Rewrite, SelfAndDescendants, and so on, which appropriately Bind and Unbind variables inside the tree.

There are a couple of design issues regarding the performance of Bind/Unbind. Each call traverses and rewrites the whole tree and potentially chews through a bunch of fresh names, so if you bind and unbind a lot then you’re gonna have performance problems.

There might be room for improvement here — off the top of my head, you could rewrite everything up to the next binding site and just store a “substitution to be applied” there. You can defer applying the changes until you’re asked to open that binding site. I suspect that’d give you asymptotic speedups (quadratic -> linear?) for certain algorithms which do lots of renaming.

Also, some tree operations don’t require you to Unbind and re-Bind every single binding site. If you’re doing some sort of operation where you’re not generating new names or binding sites, or moving nodes across binding sites, you can just leave Bound variables where they are. That’s an argument for making (eg) bindingSite.Body public, and going under it in GetChildren/SetChildren. Although in practice that’s the sort of thing which is hard to get right and can cause subtle (or not-so-subtle) bugs in your language implementation.

Today’s the day! We’re going to turn last week’s unification algorithm into an actual programming language by filling in the bottom-right part of the above diagram. Prolog’s rules engine is the system which processes the predicates and facts in your program to answer queries.

The Database

When you load a program into your Prolog interpreter, it reads the collection of rules into an internal data structure. This data structure is known as Prolog’s database. For simplicity we can represent the database as an array of rules (ie, the same structure which comes out of the parser), but a real Prolog interpreter would use a more specialised data structure to make querying more efficient.

class Engine
{
    private readonly ImmutableArray<Rule> _database;

    public Engine(ImmutableArray<Rule> database)
    {
        _database = database;
    }

    // ...
}

A First Attempt

Prolog’s interactive terminal accepts queries (predicates) and outputs a substitution which satisfies the query, according to the rules in the database. So the signature for Query is

public ImmutableDictionary<string, Term> Query(Term goal)

If a goal can’t be satisfied, Query will return null.

When you type a query, Prolog looks for the first rule in its database which matches the goal. In code, that means looping over _database and ignoring rules whose Head doesn’t unify with the goal.

public ImmutableDictionary<string, Term> Query(Term goal)
{
    foreach (var rule in _database)
    {
        // call the overload below
        var result = Query(rule, goal);
        if (result != null)
        {
            return result;
        }
    }
    return null;
}
private ImmutableDictionary<string, Term> Query(Rule rule, Term goal)
{
    ImmutableDictionary<string, Term> subst;
    try
    {
        subst = rule.Head.Unify(goal);
    }
    catch (UnificationError)
    {
        return null;
    }
    // ...
}

If the rule’s head does match the goal, Prolog needs to check the conditions on the right hand side of the rule. Each predicate on the right-hand side becomes a sub-goal; we’ll recursively call Query for each of these sub-goals. These recursive Query calls will return a substitution, and the information from those substitutions needs to be propagated bi-directionally. It’s just like when we were solving a system of equations in the last post: we’ll propagate information forwards by Applying the current substitution to later sub-goals, and propagate it backwards by Compose-ing the substitutions from each recursive Query call into the current substitution.

public ImmutableDictionary<string, Term> Query(Rule rule, Term goal)
{
    // ... 
    foreach (var predicate in rule.Body)
    {
        var subst2 = Query(subst.Apply(predicate));
        if (subst2 == null)
        {
            return null;
        }
        subst = subst.Compose(subst2);
    }
    return subst;
}

If any of the recursive Query calls failed, that means one of the conditions on the right hand side was not satisfiable, so the whole rule fails (this function returns null) and we have to try the next one. Eventually we hope to find a rule whose head matches the goal and whose conditions are satisfied. This means the query was successful, so we return the subst corresponding to the rule.

That’s a decent first stab at the code for Query. Now we have to fix the bugs.

Backtracking

Here’s a trimmed-down version of the example database from the first post.

likes(benjamin, asparagus).
likes(benjamin, pizza).
likes(clio, pizza).

dinner(Food) :- likes(benjamin, Food), likes(clio, Food).

When we feed this database into our Engine and ask it what’s for dinner,

var program = @"
likes(benjamin, asparagus).
likes(benjamin, pizza).
likes(clio, pizza).

dinner(Food) :- likes(benjamin, Food), likes(clio, Food).
";
var ast = PrologParser.ParseProgram(program);
var engine = new Engine(ast);

var query = PrologParser.ParseQuery("dinner(Food)");
var result = engine.Query(query);

if (result == null)
{
    Console.WriteLine("no solution");
}
else
{
    Console.WriteLine(Write(result));
}

it replies no solution, even though there is a clear solution (Food := pizza). What did we do wrong?

Let’s think about how our engine is going to execute this query.

1. Find a rule whose head unifies with the goal. There’s only one option: the dinner(Food) line.
2. Query for the first predicate in the rule’s body, namely likes(benjamin, Food).
3. Find a rule whose head unifies with the predicate. There are two options, likes(benjamin, asparagus) and likes(benjamin, pizza). The engine chooses the first one, so Food := asparagus.
4. Now query for the second predicate in the rule’s body, which is likes(clio, Food). Since Food := asparagus, we’re looking for likes(clio, asparagus)
5. Since there are no rules matching likes(clio, asparagus) the query fails.

The bug was in step 3, when the engine committed to the first rule which matched. It later realised that Food couldn’t be asparagus, so it needs to be able to backtrack and undo that decision. (This is very similar to the backtracking which parsers do, as detailed in an earlier post.)

The reality is that each goal may have multiple substitutions which satisfy it. likes(benjamin, Food) has two solutions, Food := asparagus and Food := pizza, but we threw away the second one. So our Query signature really should’ve looked like this:

public IEnumerable<ImmutableDictionary<string, Term>> Query(Term goal)

When there’s no solution, Query will return an empty list (rather than null as before). I’ll update the implementation to work with a collection of substitutions.

public IEnumerable<ImmutableDictionary<string, Term>> Query(Term goal)
    => _database.SelectMany(rule => Query(rule, goal));

private IEnumerable<ImmutableDictionary<string, Term>> Query(Rule rule, Term goal)
{
    IEnumerable<ImmutableDictionary<string, Term>> substs;
    try
    {
        substs = new[] { rule.Head.Unify(goal) };
    }
    catch (UnificationError)
    {
        return new ImmutableDictionary<string, Term>[] { };
    }
        
    foreach (var predicate in rule.Body)
    {
        substs =
            from subst in substs
            from subst2 in Query(subst.Apply(predicate))
            select subst.Compose(subst2);
    }

    return substs;
}

SelectMany (and from...select, which is the same thing under the hood) coming in clutch here. Each predicate in the body of a rule has potentially many solutions, so each recursive call to Query introduces potentially many possible branches to choose from. SelectMany lets us take all of them — but because IEnumerable is lazy, at runtime it doesn’t actually explore more branches than necessary to find a single solution. It also handles the “no solutions” case gracefully: if substs is empty then the body of the foreach loop does nothing. (SelectMany always returns an empty collection when its input collection is empty.)

Working with immutable data is of utmost importance here. Combining lazy IEnumerables with mutable data can have catastrophic consequences because it’s hard to predict when your code will be run. The body of my loop computes a new substs to be used for the next iteration; it doesn’t update a single collection of solutions. I don’t know what would happen if you did write it like that but I know it would be wrong!

Feeding the dinner(Food) query into this version of the code correctly outputs Food := pizza.

Scope Management

Discharging

Here’s another example program which reveals a bug in Query.

foo(X).
bar(baz).

test() :- foo(wibble), bar(X).

test() is straightforwardly true, but our interpreter again prints out no solution. Once again, let’s walk through the steps the engine will take.

1. Query foo(wibble). The only rule matching this goal is foo(X)., which matches when X := wibble. So the recursive call to Query returns the substitution X := wibble.
2. Apply that substitution to the rest of the test rule.
3. Query the second predicate in test, which is bar(wibble). This goal fails because the only bar-related fact in the database is bar(baz).

The bug is in step 1, when we returned the X := wibble substitution. foo’s X escaped its scope and collided with the X in test’s body. We need to adjust our code so that variables can’t escape their scope by being returned in a substitution.

The fix is to discharge some of the equalities in a substitution at the end of running a rule (that is, on the last line of Query). We want to throw out equalities related to variables which were local to the rule, keeping only the ones which tell you something about the variables in the goal.

private IEnumerable<ImmutableDictionary<string, Term>> Query(Rule rule, Term goal)
{
    // ...
    return substs.Select(s => Discharge(s, goal));
}

private static ImmutableDictionary<string, Term> Discharge(ImmutableDictionary<string, Term> subst, Term goal)
    => subst
        .Where(kvp => goal.Variables().Contains(kvp.Key))
        .ToImmutableDictionary();

This fixes the bug. test() now correctly returns an empty substitution.

Freshening

There’s one final bug to fix in the engine.

foo(a, X).

Running the query foo(X, b) should return X := a, but the interpreter prints no solution. This is of course another name collision issue. When unifying the goal with the left hand side, the interpreter has mistaken the two Xs for being the same X. The X in foo’s left hand side should be local to foo.

To avoid unintentional collisions we need to freshen local variables. Whenever the interpreter enters a rule, it should rename the rule’s local variables to new variables which aren’t being used anywhere in the goal. An easy way to guarantee that a name isn’t being used anywhere in the goal is to stick a number on the end of it, and increase that number by 1 each time. (I’m also going to put an ? on the front of the auto-generated variable names, because variables beginning with ? is not valid Prolog so it’s guaranteed not to collide with a name typed by a programmer.)

private int _freshenCounter = 0;
private Rule Freshen(Rule rule)
{
    // find all of the rule's local variables.
    // NB: local variables may appear in the rule's body but not the head.
    var variables = rule.Body
        .Select(x => ((Term)x).Variables())
        .Aggregate(((Term)rule.Head).Variables(), (xs, ys) => xs.Union(ys));
    
    // build a substitution which maps each variable to a fresh version of that variable
    var subst = ImmutableDictionary<string, Term>.Empty;
    foreach (var variable in variables)
    {
        subst = subst.Add(variable, new Variable("?" + variable + _freshenCounter));
        _freshenCounter++;
    }

    // apply the substitution everywhere in the rule
    return new Rule(
        (Predicate)subst.Apply(rule.Head),
        rule.Body.Select(subst.Apply).Cast<Predicate>().ToImmutableArray()
    );
}

Then at the start of Query we need to Freshen the rule we’re about to enter.

private IEnumerable<ImmutableDictionary<string, Term>> Query(Rule rule, Term goal)
{
    rule = Freshen(rule);
    // ...
}

The foo(X, b) query now correctly returns X := a.

Here’s the final implementation of Query in full.

public IEnumerable<ImmutableDictionary<string, Term>> Query(Term goal)
    => _database.SelectMany(rule => Query(rule, goal));

private IEnumerable<ImmutableDictionary<string, Term>> Query(Rule rule, Term goal)
{
    rule = Freshen(rule);

    IEnumerable<ImmutableDictionary<string, Term>> substs;
    try
    {
        substs = new[] { rule.Head.Unify(goal) };
    }
    catch (UnificationError)
    {
        return new ImmutableDictionary<string, Term>[] { };
    }
    
    foreach (var predicate in rule.Body)
    {
        substs =
            from subst in substs
            from subst2 in Query(subst.Apply(predicate))
            select subst.Compose(subst2);
    }

    return substs.Select(s => Discharge(s, goal));
}

Here are some exercises you could try:

• Because we implemented _database as an array, finding a matching rule is linear in the size of the program. Try optimising the _database data structure.
• Try optimising Freshen so that it doesn’t rename more variables than it needs to.
  • It’s actually possible to avoid freshening altogether by changing the way variables are represented.
• Backtracking is enabled by default in Prolog (unlike in Pidgin where it’s disabled by default). The full Prolog language includes the ! operator (pronounced cut) to disable backtracking when necessary. When Prolog encounters ! in the right-hand side of a rule, it commits to all of the choices it’s made since (and including) selecting that rule from the database. Try implementing cut as an exercise. (You’ll need to change the parser, the AST, and the rules engine.)

Today’s code is available in the example repo, in the Engine.cs file.

We’re getting on to the fun part of writing an interpreter: actually interpreting the input code. In this post we’re going to implement a unification algorithm using my generic programming library Sawmill. Unification slots somewhere in the middle or bottom-right of this diagram:

Unification

Unification is a process for solving equations, much like the equations you’d solve in algebra at school. It’s all about plugging holes — given a pair of terms with variables inside them, what subterms can I plug in for the variables to make the terms equal? Unifying two terms produces a substitution — a mapping from variables to terms. If you plug the terms in for the variables (on both sides of the equation), then the equation should be satisfied.

The algorithm works by examining the syntactic structure of the two terms. It tries to match up each node in one term’s syntax tree with a corresponding node in the other. When you find yourself matching a term against a variable, you know that the variable must equal the term. Unification is a purely syntactic process — you can’t apply domain-specific rules such as “add a number to both sides” like you can in algebra.

Let’s work through some examples to get a feel for it.

• Unify f(a, X, Y) with f(a, b, g(x)).

  • Starting at the outermost part of the two terms’ syntax, we see f applied to three arguments in each of them — a match. Now we can match up the arguments and try to unify each of them.
  • The first argument in each is the atom a — so they match.
  • Unifying the second arguments gives us X ~ b. X is a variable, and we now know it must be equal to b, so that gets added to the output substitution.
  • Unifying the third arguments, we find Y ~ g(x), so that also goes in the output. It’s OK to match a variable to a composite term.
  • The resulting substitution is X := b; Y := g(x). You can try plugging this substitution back in to the two terms to verify that it does indeed make them equal.
    

• Unify f(X, g(X)) with f(m(b), g(m(b))). In this example, the X variable appears twice in the left hand argument; these are the same X and at the end of unification they have to agree.

  • The outermost fs match, so we descend to the arguments.
  • The first argument gives us X ~ m(b).
  • Looking at the second argument, you see g applied to one argument in both cases. This is a match, so we can once more continue to compare the arguments.
  • Comparing the arguments gives us X ~ m(b). This is the same constraint as the one we got from the first argument.
  • The resulting substitution is X := m(b).
    

• Unify f(g(X), a) with f(g(Y), X). In this example we see variables appearing on both sides. We have to come up with a value for all of the variables in both terms; once again the X appearing on both sides is the same X.

  • Once more, the outermost fs match.
  • Matching up the first arguments, we get g(X) ~ g(Y), so X ~ Y. It’s OK to match variables to each other.
  • Matching up the first arguments gives us a ~ X.
  • The resulting substitution is X := a; Y := X (or, equivalently, X := a; Y := a).
    

• A couple of exercises:

  • Does f(g(X), a) unify with f(g(b), X)?
  • Does f(X, Y) unify with f(g(Y), Z)?
  • Does f(X, Y) unify with f(Y, g(X))?
  • Try coming up with some interesting examples of terms which don’t unify.

Let’s sketch out an algorithm for unification.

1. Look at the outermost node in the two terms’ ASTs.
2. If either one of them is a variable, bind the variable to the term in the output substitution. If it’s already bound in the output substitution, make sure that this binding doesn’t conflict.
3. If both are leaf nodes, check that they match.
4. If both are composite terms, check that they match and that they have the same number of children. If so, repeat steps 1-3 for each pair of children.
5. Otherwise, the terms don’t match and we should abort unification.

Hopefully you can see how I’ve applied these steps in the examples above. I’ve glossed over some important details here (particularly regarding what happens in step 2 when a variable is already in the substitution) but I’ll be sure to fill them in when we get to writing code. (In fact we’re going to design our code so that a variable will never already be in the substitution.)

IUnifiable

Unification is a central part of Prolog’s programming model, but it has applications outside of Prolog as well (type inference, for example). Any time you have a syntactic structure with placeholder variables, the question “what can I plug in for these variables?” makes sense.

Let’s quantify that. Reading back over the description of the algorithm, it seems to build on a few core concepts:

• Terms
• Terms which are variables
• Composite terms with children
• Leaf nodes with no children
• Matching a pair of terms

Abstracting over these concepts should allow us to implement unification once and for all, without depending on the specifics of Prolog’s terms. Don’t solve one problem. Solve them all!

Sawmill’s core IRewritable interface is a generic way of looking at tree-shaped terms with children. The only concepts we need to add on top of IRewritable are variables and matching.

interface IUnifiable<T> : IRewritable<T> where T : IUnifiable<T>
{
    bool Match(T other);
    string AsVariable();
}

Match tells you whether the current node matches another node. In Prolog’s case, it tests whether the two nodes have the same name and number of children. AsVariable checks whether the current node is a variable, returning its name as a string (or null if it’s not). Here’s what it looks like implemented on the Term syntax tree:

abstract class Term : IUnifiable<Term>
{
    // ...
    public abstract bool Match(Term other);
    public abstract string AsVariable();
}
class Predicate : Term
{
    // ...
    public override bool Match(Term other)
        => other is Predicate p
        && p.Name == Name
        && p.Args.Length == Args.Length;
    public override string AsVariable() => null;
}
class Variable : Term
{
    // ...
    public override bool Match(Term other)
        => other is Variable v
        && v.Name == Name;
    public override string AsVariable() => Name;
}
class Atom : Term
{
    // ...
    public override bool Match(Term other)
        => other is Atom a
        && a.Value == Value;
    public override string AsVariable() => null;
}

For example, here’s a LINQ query which finds all of the variables mentioned anywhere in a term. It uses SelfAndDescendants together with AsVariable to find the nodes which were in fact variables.

public static ImmutableHashSet<string> Variables<T>(this T value) where T : IUnifiable<T>
    => value
        .SelfAndDescendants()
        .Select(x => x.AsVariable())
        .Where(name => name != null)
        .Distinct()
        .ToImmutableHashSet();

Substitutions

I mentioned that the output of unification is a substitution — a mapping from variable names to terms. We’ll model that using an ordinary dictionary of strings and Ts. Our signature for Unify is going to look like this:

public static ImmutableDictionary<string, T> Unify<T>(this T left, T right) where T : IUnifiable<T>

The main thing you can do with a substitution is apply it to a term. That means replacing all the variables in the term with the sub-terms they’re mapped to by the substitution. This should eliminate any variables in the term which are bound by the substitution.

public static T Apply<T>(this ImmutableDictionary<string, T> subst, T value) where T : IUnifiable<T>
    => value.Rewrite(
        x =>
        {
            var name = x.AsVariable();
            if (name != null)
            {
                return subst.GetValueOrDefault(name, x);
            }
            return x;
        }
    );

We’re going to maintain the invariant that the terms in a substitution are never going to contain any variables which are bound by the substitution. (It’s possible, and indeed desirable for efficiency, to relax this invariant, but today we’re going to focus on simplicity.)

Binding Single Variables & the Occurs Check

Time to fill in the body of Unify. Let’s start with step 2 of the algorithm: what to do when either of the input terms is a variable.

public static ImmutableDictionary<string, T> Unify<T>(this T left, T right) where T : IUnifiable<T>
{
    var leftName = left.AsVariable();
    if (leftName != null)  // then the left term was a variable
    {
        return Bind(leftName, right);
    }
    var rightName = right.AsVariable();
    if (rightName != null)  // then the right term was a variable
    {
        return Bind(rightName, left);
    }
    // ...
}

Bind(name, value) returns a substitution with the variable name bound to the term value. However! There are some important edge cases to watch out for which could result in breaking the invariant I mentioned above.

1. The term value could be the same variable as name. This would produce a substitution like X := X (that is, a do-nothing substitution), so we’ll just return an empty substitution in that case.
2. The term value could mention the variable name somewhere inside it. This would produce a substitution like X := foo(X). If you try and find a concrete value for X you’d end up with an infinitely long chain of foos! Some systems allow infinite terms like this, but others disallow it by checking whether a variable occurs in the term it’s being bound to. This check is suitably called the occurs check.

private static ImmutableDictionary<string, T> Bind<T>(string name, T value) where T : IUnifiable<T>
{
    var valueName = value.AsVariable();
    if (valueName != null && name == valueName)
    {
        return ImmutableDictionary<string, T>.Empty;
    }
    if (value.Variables().Contains(name))
    {
        throw new UnificationError("occurs check");
    }
    return ImmutableDictionary<string, T>.Empty.Add(name, value);
}

Handling Non-Variables

Let’s continue implementing Unify. At this point we know that neither left nor right are variables, so they must be either composite terms or leaf nodes. In either case, we need to check whether they match.

public static ImmutableDictionary<string, T> Unify<T>(this T left, T right) where T : IUnifiable<T>
{
    // ...
    if (!left.Match(right))
    {
        throw new UnificationError("values didn't match");
    }
    // ...
}

If they did match, it’s time to try matching up their children. (If the two nodes were leaf nodes, their list of children is empty, so the following code does nothing.)

public static ImmutableDictionary<string, T> Unify<T>(this T left, T right) where T : IUnifiable<T>
{
    // ...
    var leftChildren = left.GetChildren();
    var rightChildren = right.GetChildren();
    return Solve(leftChildren, rightChildren);
}

Unifying Children — Systems of Equations

Unifying the children means solving a system of equations. If more than one equation mentions the same variable, they have to agree on what that variable should be.

The plain is to pair up the children in the two lists and try to unify each with their partner, building up a substitution as we go.

public static ImmutableDictionary<string, T> Solve<T>(T[] left, T[] right) where T : IUnifiable<T>
{
    if (left.Length != right.Length)
    {
        throw new UnificationError("values didn't match");
    }
    var subst = ImmutableDictionary<string, T>.Empty;
    foreach (var (l, r) in left.Zip(right))
    {
        // ...
    }
    return subst;
}

This loop requires some thinking. As you unify each pair of children, you learn something about the variables in the system of equations. These learnings need to be propagated to the other equations. So as we build our substitution we need to make sure to apply it to the remaining equations in the system. This propagates information from left to right, ensuring that you don’t encounter variables which have already been bound by earlier equations.

public static ImmutableDictionary<string, T> Solve<T>(T[] left, T[] right) where T : IUnifiable<T>
{
    // ...
    foreach (var (l, r) in left.Zip(right))
    {
        var newL = subst.Apply(l);
        var newR = subst.Apply(r);
        var newSubst = newL.Unify(newR);
        // ...
    }
    // ...
}

But we also need to propagate information from right to left. When you unify f(X, Y) with f(Y, a), the first argument sets X := Y, but the second argument sets Y := a. Simply combining the two substitutions results in X := Y; Y := a which breaks our invariant that bound variables don’t appear inside a substitution. When you combine two substitutions, you need to Apply the second substitution to the terms inside the first one.

public static ImmutableDictionary<string, T> Solve<T>(T[] left, T[] right) where T : IUnifiable<T>
{
    // ...
    foreach (var (l, r) in left.Zip(right))
    {
        // ...
        subst = subst.Compose(newSubst);
    }
    // ...
}
public static ImmutableDictionary<string, T> Compose<T>(
    this ImmutableDictionary<string, T> left,
    ImmutableDictionary<string, T> right
) where T : IUnifiable<T>
    => left
        .Select(kvp => new KeyValuePair<string, T>(kvp.Key, right.Apply(kvp.Value)))
        .Where(kvp => !(kvp.Value is Variable v && v.Name == kvp.Key))
        .Concat(right)
        .ToImmutableDictionary();

Note right.Apply(kvp.Value) inside the Select. This can sometimes result in substitution entries of the form X := X (for example when unifying f(X, Y) with f(Y, X)), so I’m just filtering them out with Where.

That’s the whole algorithm! I’ll repeat Unify and Solve in full:

public static ImmutableDictionary<string, T> Unify<T>(this T left, T right) where T : IUnifiable<T>
{
    var leftName = left.AsVariable();
    if (leftName != null)
    {
        return Bind(leftName, right);
    }
    var rightName = right.AsVariable();
    if (rightName != null)
    {
        return Bind(rightName, left);
    }
    if (!left.Match(right))
    {
        throw new UnificationError("values didn't match");
    }
    var leftChildren = left.GetChildren();
    var rightChildren = right.GetChildren();
    return Solve(leftChildren, rightChildren);
}
public static ImmutableDictionary<string, T> Solve<T>(T[] left, T[] right) where T : IUnifiable<T>
{
    if (left.Length != right.Length)
    {
        throw new UnificationError("values didn't match");
    }
    var subst = ImmutableDictionary<string, T>.Empty;
    foreach (var (l, r) in left.Zip(right))
    {
        var newL = subst.Apply(l);
        var newR = subst.Apply(r);
        var newSubst = newL.Unify(newR);
        subst = subst.Compose(newSubst);
    }
    return subst;
}

A quick test of one of our examples:

static void Main(string[] args)
{
    Term left = PrologParser.ParseQuery("f(g(X), a)");
    Term right = PrologParser.ParseQuery("f(g(Y), X)");

    var subst = left.Unify(right);

    Console.WriteLine(string.Join("\n", subst.Select(kvp => kvp.Key + " := " + kvp.Value)));
    // prints out:
    // X := a
    // Y := a

    var newLeft = subst.Apply(left);
    var newRight = subst.Apply(right);

    Console.WriteLine(newLeft);
    Console.WriteLine(newRight);
    // both print out f(g(a), a)
}

Some exercises you could try:

• This implementation does a lot of Applying substitutions to terms, specifically in Solve and Compose.
  • Since Rewrite is linear in the size of the tree (it visits every node), what’s the complexity of Unify?
  • Can you make it more efficient?
    • Hint 1: you need to relax the invariant that substitutions can’t contain bound variables.
    • Hint 2: you’ll need to tweak Bind to avoid a common bug.
    • Hint 3: Applying a substitution to a term will no longer eliminate all the bound variables inside the term. Try using RewriteIter to reestablish that invariant.
• IUnifiable assumes that variables can always be represented by strings. Many practical programming language implementations use richer data structures than strings (such as variables tagged with their scope) to represent variables. Can you change this design to break the dependency on strings? What API usability issues do you encounter?

All of this code can be found in the example repo, in the Unification.cs and Syntax.Unification.cs files. In the final part of this series, we’ll apply unification to build Prolog’s rules engine.

In this post I’m going to focus on the left-hand part of that diagram. We’ll use my parsing library Pidgin to convert Prolog source code into the abstract syntax classes I outlined in the previous post.

About Pidgin

Pidgin is a parser combinator library, meaning it consists of three things:

1. A type Parser<TToken, T>, representing a process which consumes a sequence of TTokens and produces a T. (In our case, TToken will be char because we’re parsing textual data from a string.)
2. Methods to create simple Parsers, such as “consume a single character”.
3. Methods to combine Parsers into more complex and interesting ones, such as “run two parsers sequentially”.

The power of parser combinators comes from the fact that parsers are treated as ordinary values. You can pass them around as parameters, return them, store them in fields… Programming with parser combinators is just programming. Contrast this with “language workbench” tools like Antlr, which are often more powerful than parser combinators but have a separate syntax, toolchain, etc. Parser combinators are great for medium-sized parsing tasks like this one.

It’s worth elaborating a bit on what I mean by “consume a sequence”. A parser walks left to right along an input sequence, keeping track of its location in the input. Consuming a token means moving the “current location” pointer along to the next token. This is much like how the Stream classes work in .NET — calling a stream’s Read method moves the stream’s current position, so that successive calls to Read return different results. (Pidgin indeed lets you feed a Stream to a Parser.)

When a parser makes a decision about what to do next, that decision is based only on the current character and not on what comes next. If the parser moves further along the string and then realises that decision was wrong, it has to backtrack to the position where it made the decision before it can proceed with a different choice. Backtracking can be catastrophically costly (as described in a short talk by my esteemed ex-colleague Balpha), so you want to code your parser to minimise the amount of backtracking it might have to do. With Pidgin, backtracking is disabled by default; it’s enabled by the Try function.

Handling Tokens and Whitespace

Let’s start with the basics and build upwards. Prolog is a whitespace-insensitive language, so we need a way to skip over the whitespace in the source code. We’ll use a convention where each component Parser will consume any whitespace after parsing any relevant non-whitespace characters.

Here’s a method which creates a Parser which consumes a specific character, skips over any whitespace, and then returns the original character.

static Parser<char, char> Tok(char value)  // for "token"
    => Char(value).Before(SkipWhitespaces);

(Char comes from the Pidgin.Parser static class; all of today’s code is in a file with using static Pidgin.Parser and using static Pidgin.Parser<char> at the top.) Char is a parser which matches a specific character; SkipWhitespaces greedily consumes and discards a sequence of whitespace characters; and Before glues two parsers together sequentially, keeping only the result of the first one. So Tok('(') will match the input strings "(" and "( " but not " (" or ")".

Here’s an equivalent parser for a specific string:

static Parser<char, string> Tok(string value)
    => Try(String(value)).Before(SkipWhitespaces);

Why am I using the backtracking function Try here? Remember, a parser consumes its input one character at a time, and makes decisions based only on the current character. If the first character of the input looks like it matches value, Pidgin will commit to running the String(value) parser; if matching the string fails after the first character then the parser needs to return to where it was so that it can try any alternative ways to parse from that location.

For an example of why we need Try, consider the parser String("prolog").Or(String("programming")). Or is Pidgin’s choice function — it attempts to run one parser, and falls back on the other one if the first parser failed without consuming any input. Here’s our parser represented pictorally:

             +--Or--+
             |      |
String("prolog")  String("programming")

Let’s think about how this parser behaves when you feed it the string "programming". Naïvely we might expect this to succeed, because String("programming") is one of Or’s options. But in fact this parser will fail to parse the string "programming". Looking at it step by step reveals why: