You Are in a Box

Just this past week I've been working on a toy/experimental web DSL [0] that uses dynamically loaded shared libraries as middleware that pass per request arena-allocated Jansson json objects between steps in a pipeline. It's extensible in that new middleware can be created. Influenced by bash and OCaml/F#, here is some kind of demo of the syntax:

  POST /api/users
    |> validate: `
      name: string(3..50)
      email: email
      age?: number(18..120)
      team_id?: number
    `
    |> jq: `{ sqlParams: [.body.name, .body.email, .body.age] }`
    |> pg: `INSERT INTO users (name, email, age) VALUES ($1, $2, $3) RETURNING *`
    |> result
      ok(201):
        |> jq: `{ success: true, user: .data.rows[0] }`
      validationError(400):
        |> jq: `{
          error: "Validation failed",
          field: .errors[0].field,
          rule: .errors[0].rule,
          message: .errors[0].message
        }`

[0] https://github.com/williamcotton/webpipe

This really reminds me of what Plan 9 was aiming for — breaking out of the 'box' by making everything a file, using per-process namespaces, and cleanly exposing system and network resources with proper permissions. It had that same idea: your environment shouldn't be a prison, it should be a flexible, composable space. (https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs) (https://fqa.9front.org/fqa0.html)

The key point of the article is "your data is trapped inside your program", i.e. data models can't generally be shared between programs. One thing that has improved my life has been using apache arrow as a way to decrease the friction of sharing data between different executables. With arrow (and it's file based compressed cousin parquet), the idea is that once data is produced it never needs to be deserialized again as you would with json or avro.

Computers are boxes, therefore all software is literally (and figuratively) "in a box", are they not? This might seem like a frivolous jest, but it is not. For example, the author points out that clojure, java, kotlin can interoperate, but notes they are stuck in the same jvm 'box'. This generalizes and recurses, so you must find a specific place to stop, and then motivate that.

One likely place to stop is at "processes". But this must be motivated since ultimately processes are as synthetic a convention as a language thread - it's just that the runtime is called a "kernel" instead of a "runtime".

Ultimately I think what the author is getting at is a data problem, not a code problem. Or rather, it's yearning toward a world where data and code are strongly decoupled, conventions around data are standardized, so that processes written in disparate tooling can successfully interoperate locally. However I doubt there is much appetite for a "model of everything" registry (such things have been tried, and failed, in the past). That said we might take another stab at this, since LLMs make likely that software will become more dynamic in terms of translating data structures at runtime such that one program can examine a peer program and its data structures, and how to map them to local data structures, thus achieving interoperability without a centralized agreement on representation.

Zawinski's Law, when taken literally, argues that programs all eventually need to be communicated with by people and other programs using a generic protocol and without using program-specific or domain-specific libraries to do so.

Unix shells (local), I'll add in HTTP (remote), and Email (remote and asynchronous) are the only forms that are ubiquitous, precisely because they enforce no structure for the payload. The structured alternatives are only popular in specific domains because they create their own ecosystem. As long as input can clearly be parsed, which goes hand in hand with being human-readable as long as you're not trying to be too efficient, you get schema implicitly transmitted out of band (by having output that you can visually inspect) and interoperability for anything that can read and write text.

I'd be interested in other directions this might go, but I remain skeptical of anything that will introduce and enforce a new paradigm that requires adding complexity to programs to accommodate it.

I like this post but the whole thing is a tease for an unwritten next article

Ever heard the cliche about “designing a game with no rules”? Seems pretty similar to “structured data with no boxes.” I think data boxes as defined by the author are not inherently bad. Optimization and specialization go hand-in-hand.

> there is no interop between powershell and nushell

FWIW I wrote a post about this design issue:

Oils Is Exterior-First (Code, Text, and Structured Data) - https://www.oilshell.org/blog/2023/06/ysh-design.html#survey...

That is

- Powershell and nushell have an "interior" design (within a process/VM)

- while POSIX shell, bash, OSH, and YSH have an "exterior" design (between processes)

And I'll claim that the exterior design is the glue you need in large, heterogeneous systems. Making the shell "interior" and "easy to use" is at odds with the role as essential glue -- it creates pressure for something else to be used instead.

---

Maybe the more pithy statement is here:

A Sketch of the Biggest Idea in Software Architecture - https://www.oilshell.org/blog/2022/03/backlog-arch.html

The lowest common denominator between a PowerShell, Elvish, Rash, and nushell script is a Bourne shell script (and eventually a YSH script)

I also claim this isn't theoretical -- there are probably a non-trivial number of bash scripts gluing together PowerShell and other shells. IMO it's better to have 1 type of glue, than 2 or more types, which I call "Unix sludge / Cloud sludge".

---

And I disagree with this part, which references protocol buffers:

> how do you get a schema? well, you establish in-band communication. RPC is ...

Protocol buffers transmit schemas OUT of band, usually via a monorepo. The data sent over the wire can't be interpreted without the schema information compiled into the binary.

The monorepo works well enough within Google, but even there it failed to scale (probably around the time of "Alphabet", e.g. Waymo and other companies)

Also, protobufs are biased toward C++; users of other languages feel this friction to varying degrees. In general, they'd rather use .gob for Go, pickle for Python, JSON for JS, Java serialization, etc.

I feel this the most on mobile platforms where the phone really should be acting as your agent but instead we're stuck with all these apps.

>

    Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
    —Zawinski's Law of Software Envelopment

It's a highly-dimensional box, isn't it? This age-old tension between interface standards and business/innovation speed.

Everything is a file is a good example of a fundamental and major standard that lasts till today and even though IPC kind of didn't make it all the way, I think about the core UNIX philosophy and Alan Kay's thoughts around as very, very accurate in terms of where we've ended up and what the likely ways out look like to me.

The trend of typing all lowercase in articles is annoying.