Autology: A Lisp with access to its own interpreter

Are there any examples?

This in turn, is based on black[0] which is an extension to scheme, which in turn is inspired by [1]

[0] http://pllab.is.ocha.ac.jp/~asai/Black/

[1] https://www.lirmm.fr/~dony/enseig/MR/notes-etudes/Reflective...

There's no relationship - tower of interpreters is related to lightweight modular staging

https://dl.acm.org/doi/10.1145/1868294.1868314

This is just lisp or more broadly "I'm an interpreted language" since you know... Python has access to much of its interpreter runtime too . And if you think you can't do wild stuff in Python you can (hint: you can use ctypes on libpython to do way more than you think you can do).

Typically, Lisp is quite flexible, via macros and reader macros, but the smallest core is still fixed in function (the part that implements special forms, order of evaluation, etc., basically eval and apply). This makes even that core replaceable on a form by form basis. Interesting from a theoretical perspective but not very useful in practice, and performance killing as the author points out.

Once upon a time that was true. The original Lisp interpreter could be extended by writing new FEXPR routines, which are given their argument code as source to interpret.

In the FEXPR era, Lisp didn't have lexical scopes, so FEXPR routines did not require an environment parameter.

FEXPRs were eventually abandoned in favor of macros. That was largely due to practical reasons; Lisp people were pressuring themselves or else being pressured to make Lisp perform better. It was not because FEXPRs were exhaustively researched.

A modern treatment of FEXPRs will almost certainly be lexically scoped, so that FEXPRs will have a lexical environment parameter to pass to the recursive eval calls.

Autology makes a different choice here by passing a parameter *i* (implicitly) which is not simply the lexical environment, but an object for interpretation, which can be selectively customized. This makes a categorical difference.

It's immediately obvious to me that this allows possibilities that are impossible under FEXPRs.

Here is why: when a FEXPR recursively evaluates code, whether through the standard eval provided by the host Lisp, or its own custom evaluation routine, it will encounter invocations of other FEXPRs. And those FEXPRS are "sealed off": they are coded to invoke a certain eval and that's it.

Concretely, suppose we are writing an if operator as a FEXPR. We get the if form unevaluated, and destructure it into the pieces: condition, consequent, alternative. What our FEXPR does then is recursively call eval to evaluate the condition. If that is true, we recursively call eval to evaluate consequent, otherwise we call eval on the alternative. (In all eval calls we pass down the lexical environment we were given).

Now suppose someone is writing a FEXPR in which they want to heavily customize the Lisp dialect with their custom evaluator, custom-eval. Problem is what, happens when custom-eval sees an if form that is programmed using our if FEXPR? It just calls that FEXPR, whose body calls eval; it doesn't know that the code is in a customized dialect which requires custom-eval!

It looks as if that the approach in Autology handles (or if not, could easily handle) the situation. The FEXPR which customizes the Lisp dialect can pass down a modified interpreter via *i*. Then when interpretation encounters if, it will call the if FEXPR, and what FEXPR will use the customized interpreter on the condition, then and alternative.

Now (non-hygienic) macros can already support this, in theory. If you've written a DSL as a macro, and in that DSL you choose to expand regular macros defined in the host Lisp, at least some of them with simple behaviors like the if example will also Just Work. This is because macros don't (usually!) call eval, but generate code. That code just ends up being pasted into the expression of whatever dialect is being processed, and will therefore be interpreted as that dialect. A macro arranges for evaluation by inserting the pieces of code into the template such that they are in the right kind of syntactic context to be evaluated. For this to work across dialects, the dialects have to be similar. E.g. an if macro might rewrite to cond. If the target dialect has a sufficiently compatible cond construct, then everything is golden: the if to cond rewrite will work fine.

So how we can look at dynamic interpreter customization is that it brings one aspect of FEXPRs closer to macros. Macros can do some things that FEXPRs cannot, and vice versa.

Access to eval and a generic recursive data type as internal structure means the language is closed on itself, can introspect, manipulate, create more through the same means it's programmed (see norvig lispy with python [lists] instead of parens)

So the best thing I could come up with (assuming a theoretical perfect implementation that wasn't horribly slow and expensive) was the idea that since many languages have their own niches and categories of problems that they are best used to express/solve, you could conceivably work on a complex problem where different parts of the computation are most elegantly expressed in different languages. So having a metalanguage that allows you to switch syntax in a lexical scope would be useful!?

Highly doubtful that this benefit would outweigh the egregious complexity cost of using a language like Autology, but a fun thought experiment.

Calling code across languages at least is very widespread. And typically requires custom, often clunky, solutions for all language pairs.

You could be inspired by FORTH and not have an AST at all ..

This is basically just Forth or Factor.