If you are an Emacs user with a keen eye, you will have noticed that in Emacs Lisp, code is data. After all, 'Lisp' is shorthand for 'List Processing'. One of Elisp's most beautiful features is the fortuitous blur between the thing that is processing the list (the program) and the list itself (the data). The macro in Elisp is a utility that exploits this blur and allows you to leverage this dualism between program and data in many useful and fascinating ways.

In this post, I want to swoon about macros, explain what "homoiconic" actually means, demonstrate their ubiquity in Elisp, depict their beauty on a detour through Hofstadter's strange loops and Escher's lithographs, and finally show off some tooling (macroexpand, emacs-lisp-macroexpand, macrostep) that enhances both comprehension and appreciation of macros.

Here is the Escher imagery we'll be leaning on along the way:

The kernel of Lisp has a crystalline purity that not only appeals to the esthetic sense, but also makes Lisp a far more flexible language than most others.

— Douglas Hofstadter

An important word for this post is homoiconic. A language is homoiconic when its programs are written in the language's own data structures.

Many languages are homoiconic, but perhaps none more obviously so than in Emacs Lisp (Elisp). In Elisp, source code is lists, symbols, strings, and numbers. Code looks exactly the same as lists you build with cons and take apart with car and cdr.

The distinction between program and data is exhibited by a specific, special character, the glorious ':

;; a program: evaluates to 3
(+ 1 2)

;; data: a list of three elements — a symbol and two numbers
'(+ 1 2)

The quote turns the contents of the following parentheses into a list of data elements.

To emphasize that program and data are equivalent in Elisp, running eval on the quoted list (as in (eval '(+ 1 2))) will turn it into a program, where the function is addition, and its arguments are the numbers 1 and 2.

That dualism lies at the heart of the language. Any piece of code is one character away from being a value you can inspect, transform, and rebuild; and any suitably-shaped value is one function call away from being a program.

So in Elisp, we say Program = Data, even though that's a little too simplistic, because we saw how correctly the Lisp interpreter deciphers when a list is being represented as a program versus when it is being represented as data…. The point is the list: that's the unifying form. Maybe more appropriately, we can say the program and data take the same form, or as previously mentioned, Elisp's programs are written in Elisp's own data structures.

This post was motivated by a simultaneous obsession with Douglas Hofstadter's writing and Elisp macros, so be prepared for many depictive metaphors from one of Hofstadter's favourite artists, M.C. Escher. Here's the first:

Escher drew this kind of dualism as a woodcut. The ants of Möbius Strip II appear to march on both sides of a strip. The image is provocative enough at first glance, but I invite you to follow any one of them around and discover that the two sides are one continuous surface. Program and data are the two sides of Elisp's homoiconic Möbius.

Most languages shoehorn metaprogramming into other features that are difficult to understand and difficult to use, but in Elisp there was never a wall between code and data to tunnel through. It's the same language, and the same data structures, all the way down.

Consider a regular function in Elisp. A function receives values and computes a value at runtime.

On the other hand, a macro receives code (the raw, unevaluated forms typed at its call site) and returns new code, which is then evaluated in its place. Macros run at expansion time, before your program does. I like to think of macros as little programs that write other programs given the arbitrary forms they can accept. The complexity of that form -> program projection is essentially infinite, or at least bounded by what you can express in Elisp, which is very likely bounded by your imagination.

The macro's form -> program toolkit is quasiquotation: backquote ` builds a code template, comma , splices a computed piece in, and ,@ splices in a whole list. Here's the smallest real macro I can write, a reimplementation of unless:

(defmacro my-unless (condition &rest body)
  "Run BODY unless CONDITION is non-nil."
  (declare (indent 1))
  `(if ,condition nil ,@body))

We can actually ask Emacs what this macro will get expanded to. The first code block is the macro-expansion, the second is the expansion of the macro:

(macroexpand-1
 '(my-unless (file-exists-p "~/notes")
    (make-directory "~/notes")
    (message "created it")))

(if (file-exists-p "~/notes") nil
  (make-directory "~/notes") (message "created it"))

To anticipate a common question: why couldn't my-unless be a function? Function arguments are evaluated eagerly, before the function ever sees them. A function version would have already created the directory and printed the message while its arguments were being prepared. In other words, the function receives the results of the body, but the point of using a macro here is to decide whether the body runs at all. A macro receives the body as inert data, so control flow itself is up for grabs. In this way, you are extending what the language can express.

Macros may seem specialist or eccentric…. I hope this surprises the Elispiens who are reading this! It certainly surprised me!

• when and unless are macros over if.
• dolist and dotimes are macros over while.
• push, pop, and setf are macros that rewrite themselves into the right mutation for the place you provide them with.
• with-current-buffer, with-temp-buffer, and ignore-errors are macros that wrap your code in the correct save-and-restore ceremony so you never have to type it.
• Even defun is a macro!

The most justifiably famous macro in any Emacs config is use-package:

(use-package magit
  :bind ("C-c g" . magit-status)
  :hook (git-commit-mode . flyspell-mode))

:bind and :hook aren't Elisp, but rather keywords in a small configuration language, and the use-package macro is its 'compiler', expanding the declaration into the require calls, keymap bindings, hooks, and autoload deferrals that would otherwise need to be written out by hand in their full, verbose form.

The define-minor-mode macro is similar in this way. One declaration expands into a variable, an interactive toggle command, keymap wiring, and documentation. This is what is meant by macros letting you grow a language toward the problem. With macro use, your config can read more like a declarative description of what you want, because someone built a macro for that (in use-package's case, shout out to John Wiegley).

The most shocking instance in my deep dive was defun. Evaluate (macrop 'defun) and Emacs says t. Yes, defun is a macro (again, the first code block is the macro expand, the second is the expanded macro):

(macroexpand-1 '(defun greet (name) "Say hi." (message "Hi, %s" name)))

(defalias 'greet #'(lambda (name) "Say hi." (message "Hi, %s" name)))

Defining a function turns out to mean this:

• build an anonymous function
• alias a symbol to it

More surprises….

Did you know that lambda itself is also a macro (albeit a delightfully small one that expands into a quoted version of itself)? Surely not my beloved defcustom? Yes, my fellow Emacsapien, that is also a macro.

(macroexpand-1
 '(defcustom chiply/favorite-lithograph "Drawing Hands"
    "Which Escher lithograph to contemplate while macroexpanding."
    :type 'string
    :group 'chiply))

(custom-declare-variable
 'chiply/favorite-lithograph '"Drawing Hands"
 "Which Escher lithograph to contemplate while macroexpanding."
 :type 'string :group 'chiply)

cl-loop, that entire iteration mini-language, much overused by yours truly? That's also a macro!

If you want to see all the macros, just run this.

(let (names)
  (mapatoms (lambda (s) (when (macrop s) (push (symbol-name s) names))))
  (with-temp-buffer
    (setq fill-column 72)
    (insert (mapconcat #'identity (sort names #'string<) " "))
    (fill-region (point-min) (point-max))
    (buffer-string)))

It seems like all the code you are writing is somehow being compiled in place to other code, so where does this end? It ends at the special forms — if, let, setq, while, quote, save-excursion, condition-case, and their friends implemented in C. When you macroexpand any Elisp program all the way down, what remains is composed of exactly three things: special forms, plain function calls, and constants. The functions do the work (about fifteen hundred are C primitives like car and cons; the rest are written in Elisp). The special forms decide how evaluation flows. And every scrap of syntax above that floor (when, dolist, setf, use-package, even defun) is macros written in Elisp. The foundation is C, but the architecture is built out of the tower's own bricks.

The distinction between special form and macro deserves closer inspection, because at the call site a macro and a special form are indistinguishable. Neither special forms nor macros evaluate their arguments the normal way, which is why when and if feel similar. The difference is clear when you introspect. when is a macro, so it is obliged to explain itself: macroexpand turns it into if. In contrast to when, if explains nothing about itself — it is an evaluation rule, wired into the interpreter's C. A macro must always expand away, whereas a special form is where expanding stops. Put another way, a macro is a special form you're allowed to write yourself, on the condition that it has to compile down to the real forms. The real ones number exactly twenty-two in the Emacs I'm writing this in (swap special-form-p for macrop into the census above to meet them), and cond, and, and or are among them.

Escher cut his Tower of Babel in 1928, and its subject is a construction project failing, because the builders stopped sharing a language. Elisp's tower stands for precisely the opposite reason: from use-package at the summit down to the special forms at the footing, every floor is written in the same tongue.

I think this is a common use case. Let's say you keep writing a command that sets a variable and reports what happened. Maybe you have one for debug-on-error, one for truncate-lines, etc…. The pattern of thought is: "give me a command that toggles this variable." Capturing the pattern in a macro is useful in this case:

(defmacro deftoggle (var)
  "Define a command `chiply/toggle-VAR' that toggles the variable VAR."
  `(defun ,(intern (format "chiply/toggle-%s" var)) ()
     ,(format "Toggle the variable `%s'." var)
     (interactive)
     (setq ,var (not ,var))
     (message "%s is now %s" ',var (if ,var "on" "off"))))

One line per toggle, forever after:

(deftoggle debug-on-error)
(deftoggle truncate-lines)

Expanding the first one shows what you actually wrote:

(macroexpand-1 '(deftoggle debug-on-error))

(defun chiply/toggle-debug-on-error nil
  "Toggle the variable `debug-on-error'." (interactive)
  (setq debug-on-error (not debug-on-error))
  (message "%s is now %s" 'debug-on-error
           (if debug-on-error "on" "off")))

It's important to consider where a function would've fallen short, and our new macro invocation didn't.

• It interned a new symbol — M-x chiply/toggle-debug-on-error now exists as a command.
• It wrote a docstring, computed at expansion time, that shows up properly in C-h f.
• It emitted an interactive declaration.

You haven't written a helper, but more importantly, you've added a new defining form to the language, a small sibling of defun and defvar that communicates in the diction of your problem domain. That's another beauty of Elisp macros: the abstraction is expressed and interpreted at the same level as the primitives it imitates.

I've been recently obsessed with the writing of Douglas Hofstadter, and he spent three of his Scientific American columns in 1983 teaching Lisp (the first, "Lisp: Atoms and Lists", survives online), later collected in Metamagical Themas. He opens with a mission statement:

Why is most AI work done in Lisp? There are many reasons, most of which are somewhat technical, but one of the best is quite simple: Lisp is crisp. Or as Marilyn Monroe said in The Seven-Year Itch, "I think it's just elegant!"

— Douglas Hofstadter, "Lisp: Atoms and Lists" (1983)

It's no coincidence that the author of the great book about self-reference (Gödel, Escher, Bach) fell for this language: Lisp is probably the most GEB-shaped artifact in computing.

The engine of GEB is Gödel numbering: encoding statements about arithmetic as arithmetic, painstakingly numbering every symbol until number theory could be made to talk about itself. (For a gentle tour of how the proof uses it, see Quanta's explainer; for this post, the gist is enough.)

It took a stroke of genius to build that bridge, because sentences and numbers live in different worlds. In Lisp, it seems, the bridge comes built-in, as the sentence already is the data structure. Hofstadter saw the temptation, and near the end of GEB he stages this exact argument, letting the Crab assume the burden of proof, in the book's most enchanting 'fugue':

Well, in the programming language LISP, you can talk about your own programs directly, instead of indirectly, because programs and data have exactly the same form. Gödel should have just thought up LISP, and then—

— the Crab, in Gödel, Escher, Bach (20th-anniversary ed.), p. 738

The Crab is making this post's argument: programs and data have exactly the same form, and quote is precisely the formalized quotation he goes on to wish Gödel had invented. But!

But the Author (Hofstadter) interrupts him:

Author: …no reference is truly direct — every reference depends on SOME kind of coding scheme. It's just a question of how implicit it is. Therefore, no self-reference is direct, not even in LISP.

Hofstadter is right, of course. Look under a quoted form and there is still a code: reader syntax, interned symbols, and cons cells laid out in memory. Lisp didn't abolish Gödel's bridge, but arguably simplified it for the programming use case. It built the bridge so well, and sank it so deep beneath the syntax, that you can cross it naively.

A strange loop is Hofstadter's coinage, and GEB defines it in its opening pages:

"The 'Strange Loop' phenomenon occurs whenever, by moving upwards (or downwards) through the levels of some hierarchical system, we unexpectedly find ourselves right back where we started."

Escher's Drawing Hands is his canonical image of this phenomenon. In this hand-drawn drawing of drawing hands, we see a right hand drawing the left hand that is drawing it, each one both sketcher and sketch. Elisp hides the same lithograph in its bootstrap. defmacro, the form you use to create macros, is itself a macro. The hand that draws hands is drawn; the macro that defines macros is a macro.

If the Drawing Hands image has you thinking about macros, know that the loops nest. A macro can expand into code that contains more macro calls — remember that defun hiding inside deftoggle? Take it one story higher:

(defmacro deftoggles (&rest vars)
  "Define a toggle command for each variable in VARS."
  `(progn ,@(mapcar (lambda (v) `(deftoggle ,v)) vars)))

(deftoggles debug-on-error truncate-lines)
;; ⇒ (progn (deftoggle debug-on-error) (deftoggle truncate-lines))
;; ⇒ ... (defun chiply/toggle-debug-on-error () ...)
;; ⇒ ... (defalias 'chiply/toggle-debug-on-error #'(lambda () ...))

A program writing a program writing a program writing a program. Hofstadter's running metaphor for the Lisp interpreter is a genie granting wishes, and even while introducing the language's basics, having just shown the reader that Lisp statements are themselves lists, he spots exactly this loop:

…the Lisp genie, by manipulating lists and atoms, can actually construct new wishes by itself. Thus the object of a wish can be the construction — and subsequent evaluation — of a new wish!

— Douglas Hofstadter, "Lisp: Atoms and Lists" (1983)

A macro is precisely that: a wish whose object is a new wish.

This kind of macro expansion makes me think of Escher's Print Gallery, where a young man stands in a gallery looking at a print of a seaport, and the print swells outward until it contains the gallery, and the young man, inside it. Each level of a macro expansion is a picture that turns out to contain the room you were standing in.

Escher famously couldn't finish this paradox. At the center of the lithograph, where the loop closes on itself, he left a blank patch and signed his name. The Elisp tower has its blank patch too. When you expand all the way down, you bottom out at the special forms, where the language stops being written in itself and things move over to C.

The program–data dualism has its own lithograph: Reptiles, where a lizard crawls out of a flat sketchbook drawing, climbs up over a book and a dodecahedron as a living, three-dimensional creature, and then climbs back into the page to become a drawing again. That is quote and eval exactly. A quoted form is the lizard on paper (inert, flat, safe to handle), whereas eval is when it climbs off the page and comes to life. Macros do their work on the paper lizards, rearranging drawings that will shortly be alive, their hearts beating in the Lisp interpreter.

There's one more Hofstadter obsession that Lisp exhibits. GEB's deepest question is how meaning condenses out of meaningless symbol-shuffling, layer by layer. Lisp is unabashed about the importance of symbols here because its atoms are literally called symbols. Symbols are Elisp's first-class objects you can pass around, compare, and define (deftoggle interned one for you). And each floor of the expansion tower speaks its own language: the use-package form speaks configuration, its expansion speaks hooks and keymaps, and the floors below speak control flow (special forms again), until meaning has condensed all the way into machine operations. No floor is the "real" one. Instead, the whole thing is a tangled hierarchy that you can inhabit. With Emacs, you can ride up and down at will. Here's how.

If macros were opaque, all of this would be unsettling, because you could create arbitrarily abstracted code-expanding-code that defies introspection. What keeps macros honest is that Emacs will show you the expansion at every level, as long as you know the utilities needed to make it do that. There's a passage late in GEB, where Hofstadter is explaining why introspection can't reach our own machinery, that makes the stakes of that vivid:

We feel self-programmed. Indeed, we couldn't feel any other way, for we are shielded from the lower levels, the neural tangle. Our thoughts seem to run about in their own space, creating new thoughts and modifying old ones, and we never notice any neurons helping us out! But that is to be expected. We can't.

An analogous double-entendre can happen with LISP programs that are designed to reach in and change their own structure. If you look at them on the LISP level, you will say that they change themselves; but if you shift levels, and think of LISP programs as data to the LISP interpreter (see chapter X), then in fact the sole program that is running is the interpreter, and the changes being made are merely changes in the pieces of data. The LISP interpreter itself is shielded from changes.

— Douglas Hofstadter, Gödel, Escher, Bach (20th-anniversary ed.), p. 692

The second paragraph is again evocative of Print Gallery's blank patch. However wildly your macros rewrite the language, the machinery below them is never touched. The first paragraph highlights a special way in which your editor (Emacs) is better off than your brain. We feel self-programmed but are shielded from our own neural tangle; we cannot watch our thoughts being implemented. In Emacs, the shield is optional.

Escher made a self-portrait of that privilege: Hand with Reflecting Sphere, the artist holding the mirror in which the artist, the room, and the holding hand are all visible at once. Every level of the expansion is there to be seen. In Emacs, these are the tools you can use to introspect macros and detangle the tangled hierarchy:

• The functions. macroexpand-1 performs exactly one step of expansion — when becomes if, and stops. macroexpand keeps expanding the top-level form until it isn't a macro call anymore. macroexpand-all recurses into subforms too, grinding everything down to special forms and function calls. Evaluate them in *scratch* or ielm, wrapped in pp for readable output.
• In place, in your buffer. M-x pp-macroexpand-last-sexp with point after a form pops the pretty-printed expansion into a separate buffer — the low-commitment option. M-x emacs-lisp-macroexpand with point before a form is the committed one: it replaces the form in your buffer with its expansion, properly indented. It's unbound by default and undo restores the original, so it's a safe and weirdly satisfying way to peel a layer off right where you're working.
• Interactively: macrostep. macrostep (on MELPA) makes macro debugging feel like using a debugger. M-x macrostep-expand on a macro call shows the expansion inline, as an overlay. Press e to expand the next macro call inside the expansion, c to collapse a level, q to collapse everything and leave. Macro-generated symbols are highlighted, so you can see exactly which code came from your template and which came from the call site. When a macro you're writing misbehaves, stepping through its expansion layer by layer, in place, is usually all the debugging you need.

And if you want to know what running macrostep feels like, Escher printed that too. Metamorphosis II is a single strip, four metres long, that begins with the word metamorphose, dissolves it into a checkerboard, the checkerboard into lizards, the lizards into honeycomb, bees, fish, birds, a town on the Mediterranean, a chessboard, and finally, at the far end, the word it began with. Stepping through an expansion is walking that strip: deftoggles to deftoggle to defun to defalias. Meaning gets transformed one panel at a time, and the first and final forms are both Elisp.

Hofstadter, watching definitions build on definitions in that same column, issues a warning: "The whole thing snowballs rather miraculously, and you can quickly become overwhelmed by the power you wield."

He's right, and power over syntax cuts both ways. A macro nobody else can read is a private language. A macro that evaluates its arguments twice, or accidentally captures a variable the caller was using (the classic fix is generating fresh symbols with gensym), fails in ways plain functions don't.

Escher drew this failure mode. A buggy macro is Escher's Belvedere, where every line of the expansion is locally reasonable, but where the whole is impossible. (Notice the boy on the bench in the foreground, calmly studying the impossible cube in his hands. That's you, mid-macroexpand.)

And for the macro that expands into a call to itself with no base case, Escher supplied his classic staircase illusion, where the monks of Ascending and Descending climb a loop that rises forever and never exits.

The Emacs community's rules of thumb are worth keeping in mind. Reach for a function first, and use a macro when you need to do something that a function can't, like controlling evaluation, establishing bindings, or defining new things. And keep expansions boring. The cleverness belongs in the macro's template, not its output.

I hope you notice these are the responsibilities of a language designer, because that's what a macro makes you.

Because programs are data in Elisp, the language can be reshaped in the language. The boundary between writing a program and designing a language dissolves.

Escher's Waterfall is a portrait of such a machine. The water falls, turns the wheel, and sets off along an aqueduct, where, three bends later, it pours over the top of its own fall again. The loop powers itself. That is what a self-hosting language looks like from the outside: Elisp, extended by macros, written in Elisp.