Zig's comptime is bonkers good

I'm a pretty big fan of Zig--I've been following it and writing it on-and-off for a couple of years. I think that comptime has a couple of use-cases where it is very cool. Generics, initializing complex data-structures at compile-time, and target-specific code-generation are the big three where comptime shines.

However, in other situations seeing "comptime" in Zig code has makes me go "oh no" because, like Lisp macros, it's very easy to use comptime to avoid a problem that doesn't exist or wouldn't exist if you structured other parts of your code better. For example, the OP's example of iterating the fields of a struct to sum the values is unfortunately characteristic of how people use comptime in the wild--when they would often be better served by using a data-structure that is actually iterable (e.g. std.enums.EnumArray).

Interesting points.

> Implementing generics in this way breaks parametricity. Simply put, parametricity means being able to reason about functions just from their type signature. You can't do this when the function can do arbitrary computation based on the concrete type a generic type is instantiated with.

Do you mean reasoning about a function in the sense of just understanding what a functions does (or can do), i.e. in the view of the practical programmer, or reasoning about the function in a typed theoretical system (e.g. typed lambda calculus or maybe even more exotic)? Or maybe a bit of both? There is certainly a concern from the theoretical viewpoint but how important is that for a practical programming language?

For example, I believe C++ template programming also breaks "parametricity" by supporting template specialisation. While there are many mundane issues with C++ templates, breaking parametricity is not a very big deal in practice. In contrast, it enables optimisations that are not otherwise possible (for templates). Consider for example std::vector<bool>: implementations can be made that actually store a single bit per vector element (instead of how a bool normally is represented using an int or char). Maybe this is even required by the standard, I don't recall. My point is that in makes sense for C++ to allow this, I think.

Hi, article author here. I was motivated to write this post after having trouble articulating some of its points while at a meetup, so that's why the goal of this post was focused on explaining things, and not being critical.

So at least address your points here:

* I do agree this is a direct trade-off with Zig style comptime, versus more statically defined function signatures. I don't think this affects all code, only code which does such reasoning with types, so it's a trade-off between reasoning and expressivity that you can make depending on your needs. On the other hand, per the post's view 0, I have found that just going in and reading the source code easily answers the questions I have when the type signature doesn't. I don't think I've ever been confused about how to use something for more than the time it takes to read a few dozen lines of code.

* Your specific example for recursive generic types poses a problem because a name being used in the declaration causes a "dependency loop detected" error. There are ways around this. The generics example in the post for example references itself. If you had a concrete example showing a case where this does something, I could perhaps show you the zig code that does it.

* Type checking happens during comptime. Eg, this code:

  pub fn main() void {
      @compileLog("Hi");
      const a: u32 = "42";
      _ = a;
      @compileLog("Bye");
  }

  when_typecheck.zig:3:17: error: expected type 'u32', found '*const [2:0]u8'
   const a: u32 = "42";
                  ^~~~
  Compile Log Output:
  @as(*const [2:0]u8, "Hi")

* I'm not sure what you mean by hygiene, can you elaborate?

> being able to reason about functions just from their type signature.

This has nothing to do with compile-time execution, though. You can reason about a function from its declaration if it has a clear logical purpose, is well named, and has well named parameters. You can consider any part of a parameter the programmer can specify as part of the name, including label, type name, etc.

> There is a good discussion of some issues here: https://typesanitizer.com/blog/zig-generics.html

That's actually not a great article. While I agree with the conclusion stated in the title, it's a kind of "debate team" approach to argumentation which tries to win points rather than make meaningful arguments.

The better way to frame the debate is flexibility vs complexity. A fixed function generics system in a language is simpler (if well designed) than a programmable one, but less flexible. The more flexibility you give a generics system, the more complex it becomes, and the closer it becomes to a programming language in its own right. The nice thing about zig's approach is that the meta-programming language is practically the same thing as the regular programming language (which, itself, is a simple language). That minimizes the incremental complexity cost.

It does introduce an extra complexity though: it's harder for the programmer to keep straight what code is executing at compile time vs runtime because the code is interleaved and the context clues are minimal. I wonder if a "comptime shader" could be added to the language server/editor plugin that puts a different background color on comptime code.

> type Example = Something[Example]

You can't use the binding early like this, but inside of the type definition you can use the @This() builtin to get a value that's the type you're in, and you can presumably do whatever you like with it.

The type system barely does anything, so it's not very interesting when type checking runs. comptime code is type checked and executed. Normal code is typechecked and not executed.

comptime is not a macro system. It doesn't have the ability to be unhygienic. It can cleverly monomorphize code, or it can unroll code, or it can omit code, but I don't think it can generate code.

And I would add:

* Documentation. In a sufficiently-powerful comptime system, you can write a function that takes in a path to a .proto file and returns the types defined in that file. How should this function be documented? What happens when you click a reference to such a generated type in the documentation viewer?

* IDE autocompletions, go to definition, type hinting etc. A similar problem, especially when you're working on some half-written code and actual compilation isn't possible yet.

I think the reason people are so in love with zig comptime is because even rust lovers realize that rust macros are a pile of poo poo

Scheme has a "hygienic" macro system that allows you to do arbitrary computation and code alteration at compile time.

The language doesn't see wide adoption in industry, so maybe its most important lessons have yet to be learned, but one problem with meta-programming is that it turns part of your program into a compiler.

This happens to an extent in every language. When you're writing a library, you're solving the problem "I want users to be able to write THIS and have it be the same as if they had written THAT." A compiler. Meta-programming facilities just expand how different THIS and THAT can be.

Understanding compilers is hard. So, that's at least one potential issue with compile-time programming.

Here's one of my favorite uses for it. I used to write a separate program to generate static tables. With compile time function execution, this was no longer necessary. Here's an example:

    __gshared uint[256] tytab = tytab_init;
    extern (D) private enum tytab_init =
    () {
        uint[256] tab;
        foreach (i; TXptr)        { tab[i] |= TYFLptr; }
        foreach (i; TXptr_nflat)  { tab[i] |= TYFLptr; }
        foreach (i; TXreal)       { tab[i] |= TYFLreal; }
        /* more lines removed for brevity */
        return tab;
    } ();

A link to the full glory of it:

https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ba...

Another common use for CTFE is to use it to create a DSL.

Walter, I'll take any chance I can get to say: thank you for creating D! One thing I was wondering about is the limits of compile time execution.

How does the D compiler ensure correctness if the machine the compiler runs on is different from the machine the program will execute on?

For example, how does the compiler know that "int s = sum(100000, 1000000)" is the same value on every x86 machine?

I'm thinking there could be subtle differences between generations of CPU, how can a compiler guarantee that a computation on the host machine will result in the same value on the target machine in practice, or is it assuming that host and target are sufficiently similar, as long as the architecture matches? (which is fine, I'm wondering as to what approaches exist)

D's ImportC also can do CTFE with C code!

    int sum(int a, int b) { return a + b; }

    _Static_assert(sum(3, 4) == 7, "look ma, check at compile time!");

Tbf, Zig allows that too (calling the same function in a runtime and comptime context):

    fn square(num: i32) i32 {
        return num * num;
    }

    pub fn main() void {
        _ = square(2);
        _ = comptime square(3);
    }

You don't know the amout of magic that goes behind the scenes in python and php with the __ functions. I think zig's approach is refreshing. Being able to follow the code trumps the seconds wasted typing the extra code.

In my humble opinion, a lot of the dislike of operator overloading is related to unexpected runtime performance.

My ideal solution would be for the language to introduce custom operators that clearly indicate an overload. Something like a prefix/postfix (e.g. `let c = a |+| b`). That way it is clear to the person viewing the code that the |+| operation is actually a function call.

This is still open to abuse but I think it at least removes one of the major concerns.

I feel like the ocaml solution would fit zigs usecase well.

In ocaml you can redefine operators... but only in the context of another module.

So if I re-define + in some module Vec3, I can do:

  Vec3.(a + b + c + d)

  let open Vec3 in
  a + b + c + d

  a.add(b).add(c).add(d)

Maybe such operators for basic linear algebra (for arrays of numbers) should be just built into the language instead of overloading operations. I'm not sure if such a proposal doesn't already exists.

Yeah I never got the aversion to operator overloading either.

"+ can do anything!" As you said, so can plus().

"Hidden function calls?" Have they never programmed a soft float or microcontroller without a div instruction? Function calls for every floating point op.

At some point there was a discussion about compile time reflection, which I guess could include functionality like that, but I think the topic died along with some kind of drama around it. Quite a bummer, cause things like serde would have been so much easier to imeplement with compile time reflection

With comp-time reflection you can build frameworks like ORMs or web frameworks. The only trade-off is that you have to include such a library in the form of source code.

>I don't really get the big deal about needing meta programming in the language itself. If I want to generate structs, serialization, properties, instrumentation, etc, I just write a regular C program that processes some source files and output source files and run that first in by build script.

This describes exactly what people don't want to do.

C# (strictly, Roslyn/dotnet) provides this in a pretty nice way: because the compiler is itself written in the language, you can just drop in plugins which have (readonly!) access to the AST and emit C# source.

Debugging .. well, you have to do a bit more work to set up a nice test framework, but you can then run the compiler with your plugin from inside your standard unit test framework, inside the interactive debugger.

Yes, this is the same approach Ryan Fleury and others advocate, and it's perfectly good:

> Arbitrary compile-time execution in C:

> cl /nologo /Zi metaprogram.c && metaprogram.exe

> cl /nologo /Zi program.c

> Compile-time code runs at native speed, can be debugged, and is completely procedural & arbitrary

> You do not need your compiler to execute code for you

https://x.com/ryanjfleury/status/1875824288487571873

I don't know about zig bit the power of lisp is that youre manipulating the s-expressions or to put it another way, you're manipulating the ast. To do that in C you'd need to write a full C parser for your C program that processes source files.

I used to do that in Python with the numba jit. Write Python code that generates a Python code that then gets compiled.

It's a fragile horrible mess, and the need to do this was a major reason for me to switch away from Python. It's a bit like asking why we don't just pass all arguments to functions as strings. Yeah, people write stringly typed code, but it should rarely be necessary, and your language should provide means to avoid it.

Whether you consider it a big deal or not is up to you, but with zig's approach you don't have to write/maintain a separate parser, nor worry about whether it's complete enough to process your source files.

I don't know a lot about debugging zig comptime, though. I use printf-style debugging and the built-in unit test blocks. That's all I've needed so far. (Perhaps that's all there is.)

Well put. I always have the feeling that any language which has an `eval` function or an invokable compiler can do meta program. That said, I think the "big deal" is in UX/DX. It's really nice to have meta programming support built-in to the language when you need it.

> How do you people debug and test these meta programs?

I couldn't find any other answer than using @compileLog to print-debug [1]. In lisp, apparently some implementations allow to trace macros [2]. Couln'd find anything about Nim's macro debugging capabilities.

This whole thing looks like a severe limitation that is not balanced by the benefit of having all code in the same place. Do you know other languages that provide sensible meta-programming facilities?

[1] https://www.reddit.com/r/Zig/comments/jkol30/is_there_a_way_... [2] https://stackoverflow.com/questions/44872280/macros-and-how-...

I've done this sort of thing by writing a code generator in python instead of using comptime. I'm not confident that comptime zig is particularly fast, and I don't want to run the json parser that generates the struct all the time.

How does this affect the compile times?

sure, and hygienically, it's not a preprocessor thing.

Nim is a fun language but I wouldn't consider it for "serious" work. It has the same issues as other niche languages (ie. ecosystem), plus: a polarising maintainer (most core contributors don't seem to last long) and primarily funded by a crypto company (if you care about that). Then again, 10 years ago none of that would have bothered me.

Zig has the feature of not having exceptions. I see that Nim is trying to move away from them, but exceptions color functions, which means that you have to account for them even if you don't use functions that throw them[1]. Life is too short to deal with invisible control flow.

[1]: https://github.com/status-im/nim-stew

I Agree. I tried briefly Zig and quickly gave up because, as someone used to Rust, the compiler wasn't helping me find those issues at compile time. I know that Zig doesn't make those promises, but for me, it's a deal breaker, so I suppose Zig isn't the language for me.

On the other hand, I do like the concept of comptime vs Rust macros.

Please keep the Rust community away from Zig. (I joke. Mostly...)

[flagged]

IIRC Andrew Kelley's original goal for developing Zig was to build a DAW.

I'm using D for audio plugins and we do use CTFE extensively (named comptime in Zig). Zig might be a bit more fit maybe because of the easier C and C++ interop and targetting, but I'm not sure about the COM and OOP story.

What can you do in Mojo that you can't do in Zig?