Pipelining might be my favorite programming language feature

> The author keeps calling it "pipelining", but I think the right term is "method chaining". [...] You get a similar effect with coroutines.

The inventor of the shell pipeline, Douglas McIlroy, always understood the equivalency between pipelines and coroutines; it was deliberate. See https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf It goes even deeper than it appears, too. The way pipes were originally implemented in the Unix kernel was when the pipe buffer was filled[1] by the writer the kernel continued execution directly in the blocked reader process without bouncing through the scheduler. Effectively, arguably literally, coroutines; one process call the write function and execution continues with a read call returning the data.

Interestingly, Solaris Doors operate the same way by design--no bouncing through the scheduler--unlike pipes today where long ago I think most Unix kernels moved away from direct execution switching to better support multiple readers, etc.

[1] Or even on the first write? I'd have to double-check the source again.

I don’t find your “seasoned developer” version ugly at all. It just looks more mature and relaxed. It also has the benefits that you can actually do error handling and have space to add comments. Maybe people don’t like it because of the repetition of “data =“ but in fact you could use descriptive new variable names making the code even more readable (auto documenting). I’ve always felt method chaining to look “cramped”, if that’s the right word. Like a person drawing on paper but only using the upper left corner. However, this surely is also a matter of preference or what your used to.

In most debuggers I have used, if you put a breakpoint on the first line of the method chain, you can "step over" each function in the chain until you get to the one you want.

Bit annoying, but serviceable. Though there's nothing wrong with your approach either.

> The author keeps calling it "pipelining", but I think the right term is "method chaining".

Allow me, too, to disagree. I think the right term is "function composition".

Instead of writing

  h(g(f(x)))

We can use whatever syntax we want for that, but I like Elm syntax which would look like:

  x |> f >> g >> h

If you add in a call to “.lazy“ it won’t create all the intermediate arrays. There since at least 2.7. https://ruby-doc.org/core-2.7.0/Enumerator/Lazy.html

I think the best term is "function composition", but with a particular syntax so pipelining seems alright. Method chaining is a common case, where some base object is repeatedly modified by some action and then the object reference is returned by the "method", thus allowing the "chaining", but what if you're not dealing with objects and methods? The pipelined composition pattern is more general than method chaining imho.

You make an interesting point about debugging which is something I have also encountered in practice. There is an interesting tension here which I am unsure about how to best resolve.

In PRQL we use the pipelining approach by using the output of the last step as the implicit last argument of the next step. In M Lang (MS Power BI/Power Query), which is quite similar in many ways, they use second approach in that each step has to be named. This is very useful for debugging as you point out but also a lot more verbose and can be tedious. I like both but prefer the ergonomics of PRQL for interactive work.

Update: Actually, PRQL has a decent answer to this. Say you have a query like:

    from invoices
    filter total > 1_000
    derive invoice_age = @2025-04-23 - invoice_date
    filter invoice_age > 3months

    from invoices
    filter total > 1_000
    into tmp
    
    from tmp
    derive invoice_age = @2025-04-23 - invoice_date
    filter invoice_age > 3months

> The author keeps calling it "pipelining", but I think the right term is "method chaining".

I believe the correct definition for this concept is the Thrush combinator[0]. In some ML-based languages[1], such as F#, the |> operator is defined[2] for same:

  [1..10] |> List.map (fun i -> i + 1)

0 - https://leanpub.com/combinators/read#leanpub-auto-the-thrush

1 - https://en.wikipedia.org/wiki/ML_(programming_language)

2 - https://fsharpforfunandprofit.com/posts/defining-functions/

3 - https://github.com/typelevel/mouse?tab=readme-ov-file

Shouldn’t modern debuggers be able to handle that easily? You can step in, step out, until you get where you want, or you could set a breakpoint in the method you want to debug instead of at the call site.

We might be able to cross one more language off your wishlist soon, Javascript is on the way to getting a pipeline operator, the proposal is currently at Stage 2

https://github.com/tc39/proposal-pipeline-operator

I'm very excited for it.

I prefer Scala. You can write

``` params.get("user") |> create_user |> notify_admin ```

Even more concise and it doesn't even require a special language feature, it's just regular syntax of the language ( |> is a method like .get(...) so you could even write `params.get("user").|>(create_user) if you wanted to)

> I would be lying if I didn't secretly wish that all languages adopted the `|>` syntax from Elixir.

This is usually the Thrush combinator[0], exists in other languages as well, and can be informally defined as:

  f(g(x)) = g(x) |> f

I've been using Elxir for a long time and had that same hope after having experienced how clear, concise and maintainable apps can be when the core is all a bunch of pipelines (and the boundary does error handling using cases and withs). But having seen the pipe operator in Ruby, I now think it was a bad idea.

The problem is that method-chaining is common in several OO languages, including Ruby. This means the functions on an object return an object, which can then call other functions on itself. In contrast, the pipe operator calls a function, passing in what's on the left side of it as the first argument. In order to work properly, this means you'll need functions that take the data as the first argument and return the same shape to return, whether that's a list, a map, a string or a struct, etc.

When you add a pipe operator to an OO language where method-chaining is common, you'll start getting two different types of APIs and it ends up messier than if you'd just stuck with chaining method calls. I much prefer passing immutable data into a pipeline of functions as Elixir does it, but I'd pick method chaining over a mix of method chaining and pipelines.

I'm a big fan of the Elixir operator, and it should be standard in all functional programming languages. You need it because everything is just a function and you can't do anything like method chaining, because none of the return values have anything like methods. The |> is "just" syntax sugar for a load of nested functions. Whereas the Rust style method chaining doesn't need language support - it's more of a programming style.

Note also that it works well in Elixir because it was created at the same time as most of the standard library. That means that the standard library takes the relevant argument in the first position all the time. Very rarely do you need to pipe into the second argument (and you need a lambda or convenience function to make that work).

Agree. This is absolutely my fave part of Elixir. Whenever I can get something to flow elegantly thru a pipeline like that, I feel like it’s a win against chaos.

R has a lovely toolkit for data science using this syntax, called the tidyverse. My favorite dev experience, it's so easy to just write code

Yes, a small feature set is important, and adding the functional-style pipe to languages that already have chaining with the dot seems to clutter up the design space. However, dot-chaining has the severe limitation that you can only pass to the first or "this" argument.

Is there any language with a single feature that gives the best of both worlds?

I find the threading operators in Clojure bring much joy and increase readability. I think it's interesting because it makes me actually consider function argument order much more because I want to increase opportunities to use them.

Yeah, I found this when I was playing around with Hy a while back. I wanted a generic `->` style operator, and isn't wasn't too much trouble to write a macro to introduce one.

That's sort of an argument for the existence of macros as a whole, you can't really do this as neatly in something like python (although I've tried) - I can see the downside of working in a codebase with hundreds of these kind of custom language features though.

Yes threading macros are so much nicer than method chaining, because it allows general function reuse, rather than being limited to the methods that happen to be defined in your initial data object.

I like partial application like in Standard ML, but it also means, that one must be very careful with the order of arguments, unless we get a variant of partial application, that is flexible enough to let you specify which arguments you want to provide, instead of always assuming the first n arguments. I use "cut" for this in Scheme. Threading/Pipelines are still very useful though and can shorten things and make them very readable.

Sure. But how do you write that in a way that is expressive, terse, and readable all at once? Nothing beats x | y | z or (-> x y z). The speed of both writing and reading (and comprehending), the sheer simplicity, is what makes pipelining useful in the first place.

for loops are also gotos with less power, yet we usually prefer them.

... and then recreate the scripting language...

I've always found magrittr mildly hilarious. R has vestigial Lisp DNA, but somehow the R implementation of pipes was incredibly long, complex and produced stack traces, so it moved to a native C implementation, which nevertheless has to manipulate the SEXPs that secretly underlie the language. Compared to something like Clojure's threading macros it's wild how much work is needed.

And base R has had a pipe for a couple years now, although there are some differences between base R's |> and tidyverse's %>%: https://www.tidyverse.org/blog/2023/04/base-vs-magrittr-pipe...

R, specifically tidyverse, has a special place in my heart. Tidy principles makes data analysis easy to read and easy to use new functions, since there are standards that must be met to call a function "tidy."

Recently I started using Nushell, which feels very similar.

Base R as well: |> was implemented as a pipe operator in 4.1.0.

R + tidyverse is the gold standard for working with data quickly in a readable and maintainable way, IMO. It's just absolutely seamless. Shoutout to tidyverts (https://tidyverts.org/) for working with time series, too

At the risk of over generalized pronouncements, ease of debugging is usually down to how well-designed your tooling happens to be. Most of the time the framework/language does that for you, but it's not the only option.

And for exceptions, why not solve it in the data model, and reify failures? Push it further downstream, let your pipeline's nodes handle "monadic" result values.

Point being, it's always a tradeoff, but you can usually lessen the pain more than you think.

And that's without mentioning that a lot of "pipelining" is pure sugar over the same code we're already writing.

Pipelining simplifies debugging. Each step is obvious and it is trivial to insert logging between pipeline elements. It is easier to debug than the patterns compared in the article.

Exception handing is only a problem in languages that use exceptions. Fortunately there are many modern alternatives in wide use that don't use exceptions.

Yes, certainly!

I've encountered and used this pattern in Python, Ruby, Haskell, Rust, C#, and maybe some other languages. It often feels nice to write, but reading can easily become difficult -- especially in Haskell where obscure operators can contain a lot of magic.

Debugging them interactively can be equally problematic, depending on the tooling. I'd argue, it's commonly harder to debug a pipeline than the equivalent imperative code and, that in the best case it's equally hard.

I don't know what you're writing, but this sounds like language smell. If you can represent errors as data instead of exceptions (Either, Result, etc) then it is easy to see what went wrong, and offer fallback states in response to errors.

Programming should be focused on the happy path. Much of the syntax in primitive languages concerning exceptions and other early returns is pure noise.

Established debugging tools and logging rubric are not suitable for debugging heavily pipelined code. Stack traces, debuggers rely heavily on line based references which are less useful in this style and can make diagnostic practices feel a little clumsy.

The old adage of not writing code so smart you can’t debug it applies here.

Pipelining runs contrary enough to standard imperative patterns. You don’t just need a new mindset to write code this way. You need to think differently about how you structure your code overall and you need different tools.

That’s not to say that doing things a different way isn’t great, but it does come with baggage that you need to be in a position to carry.

Pipelining is just syntactic sugar for nested function calls.

If you need to handle an unhappy path in a way that isn’t optimal for nested function calls then you shouldn’t be nesting your function calls. Pipelining doesn’t magically make things easier nor harder in that regard.

But if a particular sequence of function calls do suit nesting, then pipelining makes the code much more readable because you’re not mixing right-to-left syntax (function nests) with left-to-right syntax (ie you’re typical language syntax).

Pipelining is also nice until you have to use it for everything because you can't do alternatives (like default function arguments) properly.

Rust chains everything because of this. It's often unpleasant (see: all the Rust GUI toolkits).

> Pipelining can become hard to debug when chains get very long. The author doesn't address how hard it can be to identify which step in a long chain caused an error.

Yeah, I agree that this can be problem when you lean heavily into monadic handling (i.e. you have fallible operations and then pipe the error or null all the way through, losing the information of where it came from).

But that doesn't have much to do with the article: You have the same problem with non-pipelined functional code. (And in either case, I think that it's not that big of a problem in practice.)

> The author uses examples that function very differently.

Yeah, this is addressed in one of the later sections. Imo, having a unified word for such a convenience feature (no matter how it's implemented) is better than thinking of these features as completely separate.

You can add peek steps in pipelines and inspect the in between results. Not really any different from normal function call debugging imo.

I think you may have misinterpreted his motive here.

Just before that statement, he says that it is an article/hot take about syntax. He acknowledges your point.

So I think when he says "semantics beat syntax every day of the week", that's him acknowledging that while he prefers certain syntax, it may not be the best for a given situation.

the paragraph you quoted (atm, 7 mins ago, did it change?) says:

>Let me make it very clear: This is [not an] article it's a hot take about syntax. In practice, semantics beat syntax every day of the week. In other words, don’t take it too seriously.

It's just as difficult to debug when function calls are nested inline instead of assigning to variables and passing the variables around.

Agreed that long chains are hard to debug. I like to keep chains around the size of a short paragraph.

The article also clearly points that that it's just a hot-take, and to not take it too seriously.

You flipped SELECT and WHERE, which probably just solidifies your point. I can't count the number if times I've seen this trip up analysts.

I don't see how LINQ provides an especially illuminating example of what is effectively method chaining.

It is an exemplar of expressions [0] more than anything else, which have little to do with the idea of passing results from one method to another.

[0]: https://learn.microsoft.com/en-us/dotnet/csharp/language-ref...

Agreed. It would be nice if SQL databases supported something similar.

I don't like it either, because it promotes method `b` to the global namespace. There may be many such `b` methods on different, unrelated types. I think that the latter should be prefixed with the typename or module name.

   a.b(c) == AType.b(a, c)   (or AType::b(a, c) , C++ style)

What were their reasons?

To me, the cool (and uncommon in other languages' standard libraries) part about C++ ranges is that they reify pipelines so that you can cut and paste them into variables, like so:

    auto get_ids(std::span<const Widget> data)
    {
        auto pipeline = filter(&Widget::alive) | transform(&Widget::id);
        auto sink = to<std::vector>();
        return data | pipeline | sink;
    }

This looks awesome!

I'm really want to start playing with some C++23 in the future.

This is not functionally different from operator<< which std::cout has taught us is a neat trick but generally a bad idea.