From 43e49fca436712dfbe54a7dc974f48ed9a38eb4e Mon Sep 17 00:00:00 2001 From: Kodi Arfer Date: Sat, 11 Nov 2023 10:47:26 -0500 Subject: [PATCH] Overhaul the manual's chapter on macros --- docs/api.rst | 41 +------- docs/macros.rst | 245 +++++++++++++++++++++++++++++++++++------------- 2 files changed, 182 insertions(+), 104 deletions(-) diff --git a/docs/api.rst b/docs/api.rst index 69344d4d0..c67c69432 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -1,6 +1,7 @@ API === +.. _core-macros: Core Macros ----------- @@ -1308,6 +1309,8 @@ base names, such that ``hy.core.macros.foo`` can be called as just ``foo``. Only one pragma is currently implemented: +.. _warn-on-core-shadow: + - ``:warn-on-core-shadow``: If true (the default), :hy:func:`defmacro` and :hy:func:`require` will raise a warning at compile-time if you define a macro with the same name as a core macro. Shadowing a core macro in this fashion is @@ -1371,47 +1374,9 @@ the following methods .. hy:autoclass:: hy.I -.. _reader-macros: - Reader Macros ------------- -Reader macros allow one to hook directly into Hy's reader to customize how -different forms are parsed. Reader macros can be imported from other libraries -using :hy:func:`require`, and can be defined directly using -:hy:func:`defreader`. - -Like regular macros, reader macros should return a Hy form that will then be -passed to the compiler for execution. Reader macros access the Hy reader using -the ``&reader`` name. It gives access to all of the text- and form-parsing logic -that Hy uses to parse itself. See :py:class:`HyReader ` and -its base class :py:class:`Reader ` for details regarding -the available processing methods. - -Note that Hy reads and parses each top-level form completely before it is executed, -so the following code will throw an exception: - -.. code-block:: hylang - - => (do - ... (defreader up - ... (.slurp-space &reader) - ... (.upper (.read-one-form &reader))) - ... (print #up "hello?")) - ;; !! ERROR reader macro '#up' is not defined - -Since the entire ``do`` block is read at once, the ``defreader`` will not have -yet been evaluated when the parser encounters the call to ``#up``. However, if -the reader macro isn't used until a later top-level form, then it will work: - -.. code-block:: hylang - - => (defreader up - ... (.slurp-space &reader) - ... (.upper (.read-one-form &reader))) - => (print #up "hy there!") - HY THERE! - .. autoclass:: hy.reader.hy_reader.HyReader :members: parse, parse_one_form, parse_forms_until, read_default, fill_pos diff --git a/docs/macros.rst b/docs/macros.rst index 5ba82f46d..182df6bd9 100644 --- a/docs/macros.rst +++ b/docs/macros.rst @@ -2,94 +2,207 @@ Macros ====== -Operations on macros +Macros, and the metaprogramming they enable, are one of the characteristic features of Lisp, and one of the main advantages Hy offers over vanilla Python. Much of the material covered in this chapter will be familiar to veterans of other Lisps, but there are also a lot of Hyly specific details. + +What are macros for? -------------------- -The currently defined global macros can be accessed as a dictionary ``_hy_macros`` in each module. (Use ``bulitins._hy_macros``, attached to Python's usual :py:mod:`builtins` module, to see core macros.) The keys are mangled macro names and the values are the function objects that implement the macros. You can operate on this dictionary to list, add, delete, or get help on macros, but be sure to use :hy:func:`eval-and-compile` or :hy:func:`eval-when-compile` when you need the effect to happen at compile-time. You can call :hy:func:`local-macros ` to list local macros, but adding or deleting elements in this case is ineffective. The core macro :hy:func:`get-macro ` provides some syntactic sugar. :: +The gist of `metaprogramming +`_ is that it allows you to program the programming language itself, hence the term "metaprogramming". You can create new control structures, like :ref:`do-while `, or other kinds of new syntax, like a concise literal notation for your favorite data structure. You can also modify how existing syntax is understood within a region of code, as by making identifiers that start with a capital letter implicitly imported from a certain module. With a Lisp-like macro system, you can metaprogram in a slicker and less error-prone way than generating code as text with conventional string-formatting techniques. - (defmacro m [] - "This is a docstring." - `(print "Hello, world.")) - (print (in "m" _hy_macros)) ; => True - (help (get-macro m)) - (m) ; => "Hello, world." - (eval-and-compile - (del (get _hy_macros "m"))) - (m) ; => NameError +Types of macros +--------------- -``_hy_reader_macros`` is a dictionary like ``_hy_macros`` for reader macros, but here, the keys aren't mangled. +Hy offers two kinds of macros: regular macros and reader macros. -.. _using-gensym: +**Regular macros**, typically defined with :hy:func:`defmacro`, are the kind usually being referred to when Lispers talk about "macros". Regular macros are called like a function, with an :ref:`expression ` whose head is the macro name: for example, ``(foo a b)`` could call a macro named ``foo``. A regular macro is called at compile-time, after the entire top-level form in which it appears is parsed, and receives parsed :ref:`models ` as arguments. Regular macros come in :ref:`three varieties according to scope `. -Using gensym for Safer Macros ------------------------------ +**Reader macros**, typically defined with :hy:func:`defreader`, are lower-level than regular macros. They're called with the hash sign ``#``; for example, ``#foo`` calls a reader macro named ``foo``. A reader macro is called at parse-time. It doesn't receive conventional arguments. Instead, it uses an implicitly available :class:`HyReader ` object named ``&reader`` to parse the subsequent source text. When it returns, the standard Hy parser picks up where it left off. -When writing macros, one must be careful to avoid capturing external variables -or using variable names that might conflict with user code. +Related constructs +~~~~~~~~~~~~~~~~~~ -We will use an example macro ``nif`` (see http://letoverlambda.com/index.cl/guest/chap3.html#sec_5 -for a more complete description.) ``nif`` is an example, something like a numeric ``if``, -where based on the expression, one of the 3 forms is called depending on if the -expression is positive, zero or negative. +There are three other constructs that perform compile-time processing much like macros, and hence are worth mentioning here. -A first pass might be something like:: +- :hy:func:`do-mac` is essentially shorthand for defining and then immediately calling a one-shot regular macro with no arguments. +- :hy:func:`eval-when-compile` evaluates some code at compile-time, but contributes no code to the final program, like a macro that returns ``None`` in a context where the ``None`` doesn't do anything. +- :hy:func:`eval-and-compile` evaluates some code at compile-time, like :hy:func:`eval-when-compile`, but also leaves the same code to be re-evaluated at run-time. - (defmacro nif [expr pos-form zero-form neg-form] - `(do - (setv obscure-name ~expr) - (cond (> obscure-name 0) ~pos-form - (= obscure-name 0) ~zero-form - (< obscure-name 0) ~neg-form))) +When to use what +~~~~~~~~~~~~~~~~ + +The variety of options can be intimidating. In addition to all of Hy's features listed above, Python is a dynamic programming language that allows you to do a lot of things at run-time that other languages would blanch at. For example, you can dynamically define a new class by calling :class:`type`. So, watch out for cases where your first thought is to use a macro, but you don't actually need one. + +When deciding what to use, a good rule of thumb is to use the least powerful option that suffices for the syntax, semantics, and performance that you want. So first, see if Python's dynamic features are enough. If they aren't, try a macro-like construct or a regular macro. If even those aren't enough, try a reader macro. Using the least powerful applicable option will help you avoid the :ref:`macro pitfalls described below `, as well as other headaches such as wanting to use a macro where a Python API needs a function. + +The basics +---------- + +A regular macro can be defined with :hy:func:`defmacro` using a syntax similar to that of :hy:func:`defn`. Here's how you could define and call a trivial macro that takes no arguments and returns a constant:: + + (defmacro seventeen [] + 17) + + (print (seventeen)) + +To see that ``seventeen`` is expanded at compile-time, run ``hy2py`` on this script and notice that it ends with ``print(17)`` rather than ``print(seventeen())``. If you insert a ``print`` call inside the macro definition, you'll also see that the print happens when the file is compiled, but not when it's rerun (provided an up-to-date bytecode file exists). + +A more useful macro returns code. You can construct a model the long way, like this:: + + (defmacro addition [] + (hy.models.Expression [ + (hy.models.Symbol "+") + (hy.models.Integer 1) + (hy.models.Integer 1)])) + +or more concisely with :hy:func:`quote`, like this:: + + (defmacro addition [] + '(+ 1 1)) + +You don't need to always return a model because :hy:func:`hy.as-model` will called to convert your return value into a model. Thus ``17`` above is equivalent to ``(hy.models.Integer 17)``. But returning something from a macro that can't be converted to a model, like a function object, is an error. + +Arguments, too, are generally passed in as models. You can use quasiquotation (see :hy:func:`quasiquote`) to concisely define a model with partly literal and partly evaluated components:: + + (defmacro set-to-2 [variable] + `(setv ~variable 2)) + (set-to-2 foobar) + (print foobar) + +Macros don't understand keyword arguments like functions do. Rather, the :ref:`keyword objects ` themselves are passed in literally. This gives you flexibility in how to handle them. Thus, ``#** kwargs`` and ``*`` aren't allowed in the parameter list of a macro, although ``#* args`` and ``/`` are. + +On the inside, macros are functions, and obey the usual Python semantics for functions. For example, :hy:func:`setv` inside a macro will define or modify a variable local to the current macro call, and :hy:func:`return` ends macro execution and uses its argument as the expansion. + +Macros from other modules can be brought into the current scope with :hy:func:`require`. + +.. _macro-pitfalls: + +Pitfalls +-------- -where ``obscure-name`` is an attempt to pick some variable name as not to -conflict with other code. But of course, while well-intentioned, -this is no guarantee. +Macros are powerful, but with great power comes great potential for anguish. There are a few characteristic issues you need to guard against to write macros well, and, to a lesser extent, even to use macros well. -The method :hy:func:`gensym ` is designed to generate a new, unique symbol for just -such an occasion. A much better version of ``nif`` would be:: +Name games +~~~~~~~~~~ - (defmacro nif [expr pos-form zero-form neg-form] +A lot of these issues are variations on the theme of names not referring to what you intend them to, or in other words, surprise shadowing. For example, this macro was intended to define a new variable named ``x``, but it ends up modifying a preexisting variable:: + + (defmacro upper-twice [input] + `(do + (setv x (.upper ~input)) + (+ x x))) + (setv x "Okay guys, ") + (setv salutation (upper-twice "bye")) + (print (+ x salutation)) + ; Intended result: "Okay guys, BYEBYE" + ; Actual result: "BYEBYEBYE" + +If you avoid the assignment entirely, by using an argument more than once, you can cause a different problem: surprise multiple evaluation. :: + + (defmacro upper-twice [input] + `(+ (.upper ~input) (.upper ~input))) + (setv items ["a" "b" "c"]) + (print (upper-twice (.pop items))) + ; Intended result: "CC" + ; Actual result: "CB" + +A better approach is to use :hy:func:`hy.gensym` to choose your variable name:: + + (defmacro upper-twice [input] (setv g (hy.gensym)) `(do - (setv ~g ~expr) - (cond (> ~g 0) ~pos-form - (= ~g 0) ~zero-form - (< ~g 0) ~neg-form))) + (setv ~g (.upper ~input)) + (+ ~g ~g))) + +Hyrule provides some macros that make using gensyms more convenient, like :hy:func:`defmacro! ` and :hy:func:`with-gensyms `. + +Macro subroutines +~~~~~~~~~~~~~~~~~ + +A case where you could want something to be in scope of a macro's expansion, and then it turns out not to be, is when you want to call a function or another macro in the expansion:: + + (defmacro hypotenuse [a b] + (import math) + `(math.sqrt (+ (** ~a 2) (** ~b 2)))) + (print (hypotenuse 3 4)) + ; NameError: name 'math' is not defined + +The form ``(import math)`` here appears in the wrong context, in the macro call itself rather than the expansion. You could use ``import`` or ``require`` to bind the module name or one of its members to a gensym, but an often more convenient option is to use the one-shot import shorthand :hy:class:`hy.I` or the one-shot require shorthand :ref:`hy.R `:: + + (defmacro hypotenuse [a b] + `(hy.I.math.sqrt (+ (** ~a 2) (** ~b 2)))) + (hypotenuse 3 4) + +A related but distinct issue is when you want to use a function (or other ordinary Python object) in a macro's code, but it isn't available soon enough:: + + (defn subroutine [x] + (hy.models.Symbol (.upper x))) + (defmacro uppercase-symbol [x] + (subroutine x)) + (setv (uppercase-symbol foo) 1) + ; NameError: name 'subroutine' is not defined + +Here, ``subroutine`` is only defined at run-time, so ``uppercase-symbol`` can't see it when it's expanding (unless you happen to be calling ``uppercase-symbol`` from a different module). This is easily worked around by wrapping ``(defn subroutine …)`` in :hy:func:`eval-and-compile` (or :hy:func:`eval-when-compile` if you want ``subroutine`` to be invisible at run-time). + +By the way, despite the need for ``eval-and-compile``, extracting a lot of complex logic out of a macro into a function is often a good idea. Functions are typically easier to debug and to make use of in other macros. + +The important take-home big fat WARNING +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Ultimately it's wisest to use only four kinds of names in macro expansions: gensyms, core macros, built-in Python objects that are in scope by default, and ``hy`` and its attributes. It's possible to rebind nearly all these names, so surprise shadowing is still theoretically possible. Unfortunately, the only way to prevent these pathological rebindings from coming about is… don't do that. Don't make a new macro named ``setv`` or name a function argument ``type`` unless you're ready for every macro you call to break, the same way you wouldn't monkey-patch a built-in Python module without thinking carefully. This kind of thing is the responsibility of the macro caller; the macro writer can't do much to defend against it. There is at least a pragma :ref:`warn-on-core-shadow `, enabled by default, that causes ``defmacro`` and ``require`` to warn you if you give your new macro the same name as a core macro. + +.. _reader-macros: -This is an easy case, since there is only one symbol. But if there is -a need for several gensym's there is a second macro :hy:func:`with-gensyms ` that -basically expands to a ``setv`` form:: +Reader macros +------------- - (with-gensyms [a b c] - ...) +Reader macros allow you to hook directly into Hy's parser to customize how text is parsed into models. They're defined with :hy:func:`defreader`, or, like regular macros, brought in from other modules with :hy:func:`require`. Rather than receiving function arguments, a reader macro has access to a :py:class:`HyReader ` object named ``&reader``, which provides all the text-parsing logic that Hy uses to parse itself (see :py:class:`HyReader ` and its base class :py:class:`Reader ` for the available methods). A reader macro is called with the hash sign ``#``, and like a regular macro, it should return a model or something convertible to a model. :: -expands to:: + (defreader matrix + (.slurp-space &reader) + (setv start (.getc &reader)) + (assert (= start "[")) + (setv out [[]]) + (while (not (do (.slurp-space &reader) (.peek-and-getc &reader "]"))) + (setv x (.parse-one-form &reader)) + (if (= x '|) + (.append out []) + (.append (get out -1) x))) + (if (= out [[]]) [] out)) + + (print (hy.repr #matrix [1 2 3 | 4 5 6 | 7 8 9])) + ; => [[1 2 3] [4 5 6] [7 8 9]] + +Note that because reader macros are evaluated at parse-time, and top-level forms are completely parsed before any further compile-time execution occurs, you can't use a reader macro in the same top-level form that defines it:: (do - (setv a (hy.gensym) - b (hy.gensym) - c (hy.gensym)) - ...) + (defreader up + (.slurp-space &reader) + (.upper (.read-one-form &reader))) + (print #up "hello?")) + ; LexException: reader macro '#up' is not defined -so our re-written ``nif`` would look like:: +.. _macro-namespaces: - (defmacro nif [expr pos-form zero-form neg-form] - (with-gensyms [g] - `(do - (setv ~g ~expr) - (cond (> ~g 0) ~pos-form - (= ~g 0) ~zero-form - (< ~g 0) ~neg-form)))) +Macro namespaces and operations on macros +----------------------------------------- -Finally, though we can make a new macro that does all this for us. :hy:func:`defmacro/g! ` -will take all symbols that begin with ``g!`` and automatically call ``gensym`` with the -remainder of the symbol. So ``g!a`` would become ``(hy.gensym "a")``. +Macros don't share namespaces with ordinary Python objects. That's how something like ``(defmacro m []) (print m)`` fails with a ``NameError``, and how :hy:mod:`hy.pyops` can provide a function named ``+`` without hiding the core macro ``+``. -Our final version of ``nif``, built with ``defmacro/g!`` becomes:: +There are three scoped varieties of regular macro. First are **core macros**, which are built into Hy; :ref:`the set of core macros ` is fixed. They're available by default. You can inspect them in the dictionary ``bulitins._hy_macros``, which is attached to Python's usual :py:mod:`builtins` module. The keys are strings giving :ref:`mangled ` names and the values are the function objects implementing the macros. - (defmacro/g! nif [expr pos-form zero-form neg-form] - `(do - (setv ~g!res ~expr) - (cond (> ~g!res 0) ~pos-form - (= ~g!res 0) ~zero-form - (< ~g!res 0) ~neg-form))) +**Global macros** are associated with modules, like Python global variables. They're defined when you call ``defmacro`` or ``require`` in a global scope. You can see them in the global variable ``_hy_macros`` associated with the same module. You can use ``_hy_macros`` to list, add, delete, or get help on macros, but be sure to use :hy:func:`eval-and-compile` or :hy:func:`eval-when-compile` when you need the effect to happen at compile-time, which is often. (Modifying ``bulitins._hy_macros`` is of course a risky proposition.) Here's an example, which also demonstrates the core macro :hy:func:`get-macro `. ``get-macro`` provides some syntactic sugar for getting all sorts of macros as objects. :: + + (defmacro m [] + "This is a docstring." + `(print "Hello, world.")) + (print (in "m" _hy_macros)) ; => True + (help (get-macro m)) + (m) ; => "Hello, world." + (eval-and-compile + (del (get _hy_macros "m"))) + (m) ; => NameError + +**Local macros** are associated with function, class, or comprehension scopes, like Python local variables. They come about when you call ``defmacro`` or ``require`` in an appropriate scope. You can call :hy:func:`local-macros ` to view local macros, but adding or deleting elements in this case is ineffective. + +Finally, ``_hy_reader_macros`` is a per-module dictionary like ``_hy_macros`` for reader macros, but here, the keys aren't mangled. There are no local reader macros, and there's no official way to introspect on Hy's handful of core reader macros. So, of the three scoped varieties of regular macro, reader macros most resemble global macros.