Dagger: A shell for the container age
123 comments
·March 26, 2025randomtechguy
shykes
If I may offer a different perspective: Dagger has always been a general-purpose composition engine, built on container tech. Its most successful use case is CI - specifically taking complex build and test environments, and making them more portable and reproducible. But we never claimed to replace Jenkins or any other CI platform, and we've always been very open with our community about our desire to expand the use of Dagger beyond CI. We also never claimed to replace Docker, or to "be a shell" (note that the title of this HN page doesn't reflect the title of our post in that regard).
Every feature we ship is carefully designed for consistency with the overall design. For example, Dagger Shell is built on the same Dagger Engine that we've been steadily improving for years. It's just another client. Our goal is to build a platform that feels like Lego: each new piece makes all other pieces more useful, because they all can be composed together into a consistent system.
randomtechguy
It's a fair point - My opinions and use case are my own, I didn't mean to imply or assume there were promises not kept. The dagger team has been nothing but supportive and I do think has built a great community.
That said, in the early days it was definitely pitched for CI/CD - and this how we've implemented it.
> What is it? > Programmable: develop your CI/CD pipelines as code, in the same programming language as your application.
> Who is it for? > A developer wishing your CI pipelines were code instead of YAML
https://github.com/dagger/dagger/blob/0620b658242fdf62c872c6...
Edit: This functionality/interaction with the dagger engine still exists today, and is what we rely on. The original comment is more of an observation on the new directions the project has taken since then.
shykes
Yes it's a fair observation. In terms of use cases, we did focus exclusively on CI/CD, and only recently expanded our marketing to other use cases like AI agents. It's understandable that this expansion can be surprising, we're trying to explain it as clearly as possible, it's a work in progress.
I just wanted to clarify that in terms of product design and engineering, there is unwavering focus and continuity. Everything we build is carefully designed to fit with the rest. We are emphatically not throwing unrelated products at the wall to see what sticks.
For example, I saw your comment elsewhere about the LLM type not belonging in the core. That's a legitimate concern that we debated ourselves. In the end we think there is a good design reason to make it core; we may be wrong, but the point is that we take those kinds of design decisions seriously and we take all use cases into account when we make them.
clvx
I feel Nix is eating their lunch. Even though Nix as a language could be tough, it's a simpler solution than Dagger. It provides a shell, almost complete reproducibility and isolation can be added through containers. Working with dependencies to build source code using nix is usually straightforward but having the right binary version that doesn't have the same love as major languages (i.e. terraform, flux, etc) is one thing Nix really needs to implement. Marcelo's package version search[1] is a way to discover the particular hash but then you need to explicitly use that nixpkgs version for that particular binary and do a lot of mental gymnastics in your nix files. Nix could implement a syntax where you define the package versions as attribute set and then internally does a discovery of the nixpkgs hash for that versions and installs it. Flox and DevBox follow this pattern but I don't see why you need and external tool where this can be embedded in Nix (the cli).
[1] http://lazamar.github.io/download-specific-package-version-w...
randomtechguy
IMO dagger isn't really comparable to nix.
We tried to fit dagger where we had jenkins - not just for binary builds, but for the other stuff. Mounting secrets for git clones / NPM installs, integration tests, terraform execution, build notifications and logging.
Caching is great, and dagger/nix both have interesting options here, but that's more of a bonus and not the core value prop of a build orchestrator.
verdverm
Yeah, turning a lot of our Jenkins Groovy into Dagger one-liners. Makes it super easy to "run" CI locally before pushing, but you still need the orchestrator / scheduler. Also really loving the Dagger Cloud UI for build logs over Jenkins UI
code_biologist
We tried to fit dagger where we had jenkins
"Tried", implying it didn't go well and isn't a fit for replacing Jenkins?
verdverm
I've found the Nix ecosystem to be lacking, missing packages, wrongly built packages (from the official upstream), and out of date versions. Homebrew still outclasses Nix in this regard (quality over quantity).
After taking Nix for a spin, I cannot be bothered to learn another custom tool with a bespoke language when I already have containers for doing the same things.
For Dagger, I can choose from a number of languages I already know and the Docker concepts map over nearly 1-1
clvx
> I've found the Nix ecosystem to be lacking, missing packages, wrongly built packages (from the official upstream), and out of date versions. Homebrew still outclasses Nix in this regard (quality over quantity).
That's also the case with the Docker ecosystem. On top of that, you need to take into account the base image, versions, etc.
At the end what I look for is for a project being able to build my source code with runtime dependencies and supporting tools that won't change overtime for the architecture that I need.
kiliankoe
This is an interesting take, especially since I've had exactly the opposite impression. Since you mention Homebrew, I assume you're talking about running Nix on macOS, which has been eye-opening for me. The ecosystem is much larger, I very rarely don't find what I'm looking for in nixpkgs, and if it's not there, it _definitely_ won't be in homebrew. I only use homebrew for casks nowadays (and even that is managed through nix). There's also been times when I've tried to install something from homebrew on a colleague's machine, and it was a huge pain to set it up correctly (having to manually install some dependencies) whereas everything worked without effort using nix. The only downside imho is that package authors tend to go for updating homebrew first if anything, since that's considered the default. Nix will typically get updated version a few days later, but that's fine for me.
zamalek
> I've found the Nix ecosystem to be lacking, missing packages, wrongly built packages
This is an interesting problem you've faced, their marketing claim is that they have the most comprehensive catalog of packages (and I'm inclined to believe it). I very rarely run into broken packages, and that's usually resolved by using a stable release for that specific package - and it's not like my usage of packages is lightweight (7292 lines of nix config). That's on NixOS (and Silverblue, and Ubuntu) at least.
Imustaskforhelp
theoretically you could always use the previous version / iterate on them.
I think this approach of see what sticks and them trying out a lot of different things can be nice but not in its current form.
Although I have installed dagger to give it a try and I am just not sure how to make it work , the quickstart / hello world doesn't' work.
No I don't want to make an AI , I just want to see what you really are.
And so I do agree with your statement!
randomtechguy
Yup - Ever since the introduction of llm as a core API we're basically going to stop upgrading / using the project unless things start moving in a more sane direction.
levlaz
Why is that? It’s just an extra type that has no impact on any other types. It’s a tool in your toolbox that you can use when you need it and ignore otherwise.
a_t48
This is very cool, I often find myself composing together a mixture of Dockerfiles and shell scripts for various images (build X Dockerfile for Y arch, save it as a base, build Z Dockerfile on top of the last thing, with W args). And then having to run it different ways with different mounts, based on if you're on a developer machine/robot/CI, running as a service/adhoc/attaching. This _looks_ like it could solve some of that messiness. I like that it can just reference the output of a build without having to muck around with tags (which are system global and can collide if you're not careful).
mikepurvis
I have a particular hate on for the sandwich pattern where you have a Dockerfile to a "build" environment that then gets entered, and a bunch of stuff done in there, and then the results ADDed into a different Dockerfile for the publishable asset. I know multi-stage builds sort of address this need, but a lot of people aren't using them or have found cases where it's desirable to have the "running" context for the container, for example if a bunch of stuff is being volumed-in (which is now possible at build time but is janky).
Or there's the copy my ansible playbooks into the container and run them there as localhost approach, which can be mitigated by what Packer does with at least running the playbook over SSH. But then Packer has no concept of multi-stage or copying in assets from other containers.
There's a lot of room to do better in this space.
a_t48
I don't like the sandwich either, it's too easy to accidentally miss a dependency. What I find myself doing more is building one Dockerfile as a base, and then doing `FROM $BASE` in the second to ease readability, caching, image pinning, etc.
mikepurvis
I think it often comes about because there's a desire for developers to be able to locally reproduce what is happening on CI (good!) and having an end-to-end Dockerfile can be a great piece of that (yay!) but as soon as devs are debugging a problem or wanting to change a dependency, attach a debugger, or whatever else, it becomes too burdensome to rebuild the whole thing every time. So you end up with the setup and finalization of the build in Dockerfiles, and then some nebulous semi-manual stuff occurring in the middle.
This is workable-ish for the base developer + ship a product use cases, but it creates a bit of a crisis of repeatability when you've got a complicated CI config implementing all this mystery meat that falls in the middle of the sandwich, and there's separate documentation somewhere else that explains what the dev workflow is supposed to be, but no one is explicitly tasked with keeping the CI script and dev workflow in sync, so they inevitably drift over time.
Plus, long and complicated CI-specific scripts are a smell all on their own, as was discussed here on HN a few days ago.
mdaniel
> But then Packer has no concept of multi-stage or copying in assets from other containers.
I don't know about containers specifically, since I've never bothered to use packer for that process, but it does seem that packer supports multi-step artifact production and their example even uses docker to demonstrate it https://github.com/hashicorp/packer/blob/v1.9.5/website/cont...
a_t48
OMG https://docs.dagger.io/features/debugging - Docker hasn't supported shelling into a broken build for a while now, this sounds so nice. I see there's some Dockerfile support, I wonder if I can just use this as a drop in replacement for `docker build` and get a nicer experience.
shykes
Yes you can.
Dagger is built on the same underlying tech as docker build (buildkit). So the compatibility bridge is not a re-implementation of Dockerfile, it's literally the official upstream implementation.
Here's an example that 1) fetches a random git repo 2) builds from its dockerfile 3) opens an interactive terminal to look inside 4) publish to a registry once you exit the terminal:
git https://github.com/goreleaser/goreleaser |
head |
tree |
docker-build |
terminal |
publish ttl.sh/goreleaser-example-imagephilsnow
> 3) opens an interactive terminal to look inside 4) publish to a registry once you exit the terminal
It seems like it would be good to be able to prevent the pipeline from publishing the image, if the inspection with 'terminal' shows there's something wrong (with e.g. 'exit 1'). I looked a little bit into the help system, and it doesn't seem that there's a way from inside the 'terminal' function to signal that the pipeline should stop. Semantics like bash's "set -e -o pipefail" might help here.
with-exec lets you specify that you want a command to succeed with e.g.
container | from alpine | with-exec --expect SUCCESS /bin/false | publish ...
a_t48
If that docker-build command fails, will the interactive debugger do something sensible?
levlaz
You sure can :)
Dagger can read dockefiles as is, https://docs.dagger.io/cookbook#build-image-from-dockerfile
verdverm
Being able to drop into failed container builds / runs has been game changing, saved me mucho time. If you are being programmatic, you can also inject the drop into interactive mode anywhere you want, without needing an error. You can use this to build up working environments into which you drop developers (kind of like dev containers)
levlaz
Yeah for sure, this was specifically designed to solve the problems you describe!
CBLT
I solve most of those issues with a Docker Bakefile; I'm confident I could solve the rest with Bakefiles if I had to. Reasonable developer experience.
a_t48
The last time I tried something more than Buildx that Docker itself put out the experience was bad - something about not properly caching. I'll have to give this another shot sometime, though.
ledauphin
I had missed the fact that Dagger is apparently trying to... replace? Docker for container definitions. That's a pretty big vision.
I really like the ambition here; it's very hard for me to translate this into belief that I could start using this now and actually replace some of my existing tooling.
At the same time, I kind of hate that they went for bash-compatible. I know everybody thinks bash is unbeatable, but at some point _surely_ we're going to move past its awful syntax and footguns (and SQL, and can I have a pony?)...
shykes
In theory you could replace much of Docker's low-level tooling with Dagger. In practice, that's not what we're trying to do. There's a lot of inertia to ripping out existing tools, and even if the replacement is better, it may still not be worth the effort.
What we're focusing on is green field application of container tech. Things that should be containerized, but aren't. For example:
- Cross-platform builds
- Complex integration testing environments
- Data processing pipelines
- AI agent workflows (where you give a LLM the ability to perform tasks in a controlled environment)
In those kinds of workflows, there is no dominant tool to replace - Docker or otherwise. Everyone builds their own unique snowflake of a monolith, by gluing together a dozen tools. Dagger aims to replace that glue with a modular, composable system.
ledauphin
this is helpful. and i appreciate the intellectual/technological humility.
i think there's a decent chance we end up giving Dagger a spin this year.
riedel
Does it replace Kabuki in non priviledged CI builds? Can one exchange lower / independent layers like with nix container builds?
shykes
> Does it replace Kabuki in non priviledged CI builds?
I have never heard of Kabuki, and couldn't find it in a quick web search. Did you mean Kaniko?
> Can one exchange lower / independent layers like with nix container builds?
Yes.
vito
The Bash compatible syntax is just one representation of an underlying cross-language API. You can also write code in your preferred language, provided there's an SDK for it. (Currently: Python, TypeScript, Go, PHP, Java, Elixir, Rust, .NET)
But if you're talking specifically about shell or a "native" REPL in a safer language - I also want to scratch that itch. Some day I'll dust off https://github.com/vito/dash :)
esafak
They even supported CUE at one point. https://dagger.io/blog/ending-cue-support
verdverm
I worked with the Dagger team on that. It was super hard to marry two DAG engines that work in opposite directions. The team eventually heard from users they wanted to write in the language of their choice, so there are now a bunch of SDKs.
I think one could wrap the Go SDK with CUE, and have oft considered doing it, but the work has not migrated from the backlog yet. I've been toying with a CUE powered monorepo DX CLI that is more akin to BuildPacks, using Dagger as the containerization layer.
riedel
I feel this is much more powershell like than bash. The container thing looks much more than a typed object than a plain character device/pipe. Also what is the bourne again part?
levlaz
It’s def closer to powershell in all the best ways IMO. But the key point is that the container thing is actually a typed object. So is every other primitive in there including once you make yourself.
ledauphin
okay, I think what I ultimately missed is that this is just a new language for the existing engine. I may not be the world's most careful reader.
topspin
> That's a pretty big vision.
Is it? Docker is quite long in the tooth and this point and is a long way from perfect. Docker's design is rife with poor choices and unaddressed flaws. One I've been tracking for years: there is no straightforward way to undo several things that get inherited by derived images, such as VOLUME and EXPOSE, and Docker Inc. doesn't care no matter how long the github threads get.
So I think there is ample room for alternatives.
WhyNotHugo
I see this more as a complement. I wouldn’t stop using Dockerfile for standalone definitions, but for shell+docker+shell sandwiches (the second shell being inside the container), this looks very handy.
levlaz
What kind of tooling do you have in mind to replace?
Also what would you prefer to see instead of bash compatible?
ledauphin
i want to move away from GitHub Actions and I also want to move away from some bespoke monorepo tooling for something that's a better overall orchestrator of 'things'. Dagger scratches some of those itches, or seems to, so I've had my eye on it for a while. Knowing that they're trying to also help me define and use containerized environments - I sort of assumed I'd have to bring those myself.
verdverm
I'm using Dagger in a monorepo setup (actually several). For the main one, I've gone the route of wrapping the Dagger Go SDK with a custom Go CLI (plus CUE as another centerpiece). For sure, I've been making the team aware of the challenges and rough edges. (of course there are a lot of ways people organize monorepos, we are polyglot with one go.mod but multiple package.json...)
verdverm
I'm largely replacing Dockerfiles and Makefiles with a custom CLI that wraps Dagger. Being able to use a proper language, functions, packages, imports is so much better by comparison. (ymmv)
mikepurvis
I mean if we're going full pie in the sky, fully declarative containers exist today:
https://github.com/nlewo/nix2container#nix2container
Dagger looks nifty, but it's too bad that it's still basically an imperative "start from thing, do stuff to it, publish" kind of model, rather than just expressing a desired end state and letting an underlying engine find its way there.
shykes
Dagger is fully declarative. It's just built on a dynamic declarative API, instead of a static declarative DSL.
So, if you took Nix, and replaced the static scheme-like DSL with a proper API, and then built SDKs in 5 languages for that API; and then built a bash-like shell also for easy scripting; then you would start to have something that approximates Dagger.
shykes
Someone already built a web UI for composing Dagger Shell scripts in a notebook format: http://docs.runme.dev/guide/dagger
It's really neat, I recommend checking it out.
Imustaskforhelp
woah , that actually looks really nifty.
I am imagining this with a simple cloudflare tunnel and self hosting gitlab and I am really really seeing an open source way that developers can REALLY scale.
I mean docker is really great but dagger in notebook formats just seems really cool ngl.
sourishkrout
love the enthusiasm. cocreator of Runme here.
the best part with the Dagger + Runme combo is that it runs entirely local. this isn't just a huge with for portability. it also cuts down development cycle times significantly.
oulipo
Nice! Would be lovely to have a few "workflow examples" and tutorials which really showcase the capabilities of this setup on your website, I'm not exactly sure I understand yet in what kind of cases it really shines
jatins
Was curious what Dagger does and this is what the homepage says
> The cross-platform composition engine
> Build powerful software environments from modular components and simple functions. Perfect for complex builds and AI workflows.
This is generic to point of being useless.
Everything is a composition engine. Javascript is a composition engine. macOs is a composition engine.
levlaz
I would lump dagger in with JS (programming language, ecosystem, runtime) and MacOS (operating system) before comparing it to any single tool.
Answering "what does dagger do" can be tough, because there are very broad applications.
Its always hard to describe general purpose platforms like these. Dagger is an open platform for building composable software. It has SDKs, sandboxed execution environment, observability, and now this shell interface.
jbverschoor
Somewhat related, self promo
Docker shell container - https://github.com/jrz/container-shell
shykes
That looks very neat! Let me see if we can integrate it with Dagger :)
mrbluecoat
Is the objective to get inside a container to do dev stuff? Reminds me of https://www.jetify.com/devbox and https://flox.dev/
remram
In clear terms, what would I do with this? What activity does it help me with? What program do I replace with it?
"The devops operating system" does what?
shykes
The article gives a few examples of concrete things you can do:
- Build, run, and publish containers
- Containerize language-specific builds
- Run integration tests in ephemeral environments, in a repeatable way
- Run your tools in controlled sandboxes with only the files, secrets and network access they need
It also mentions what you might replace with it:
> When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.
remram
I already build containers with Docker. I already have Makefiles that build containers and run commands in them. I already replace my complex shell scripts with Python scripts.
How does Dagger help? I just can't get anything concrete from their site.
levlaz
If you’re working on your own you may not need dagger.
If you’re working on a team often your shell scripts, python programs, and makes files are not always portable between local and CI, but even worse between your local machine and your colleagues local machine. This is where dagger shines because it lets you do all that stuff in a fully portable way.
hamandcheese
My initial impression (looking at the examples, not actual usage) is that this feels like a weird half-way between Dockerfile and using actual code to define and compose software (i.e. like Nix). Not super appealing to me as a heavy Nix user.
elorm
Did Dagger pivot their product ? I seem to recall the main selling point was an Independent pipeline-as-code service. Looks like they’re trying to rebuild Docker now?
shykes
It's the same product. Dagger Shell is a new client for the same Dagger Engine and its API. It allows you to do more from the command-line, before you have to switch to code. But the SDKs and "pipeline as code" are still there, unchanged.
GONE_KLOUT
It would be better to not confront potential users with a long text about unix history, nobody has time to read that. Instead just write a few bullet points about what it actually does.
However, I am very happy somebody finally put the prompt on TOP of a terminal window, since years I am wondering why people like to bow their head to the bottom of the screen.
I guess it is the expected gesture for IT devs and admins nowadays to follow and bow your head, so this is a great help for people to understand what is the right posture in life!
imiric
> We still use the shell, but it's no longer the universal composition system it was meant to be.
Huh? When did that change?
Dagger seems interesting, but this opinion that the shell and standard Unix tools are archaic is a dubious starting principle.
Every now and then a project with similar ideas comes along, and whether it rejects the notion of passing unstructured data between commands, addresses the footguns and frustrations of shell scripting, or is built with a more modern and "safe" programming language, it ultimately never seems to catch on as much as traditional Unix shells and tooling has.
The reality is that these tools and the design choices made 50 years ago are timeless in nature. They're deliberately lacking in features and don't attach themselves to any specific tech du jour. It's this primitive nature and the "do one thing well" philosophy that makes them infinitely composable. The same pipelines Unix nerds were building 50 years ago to solve problems are still useful today, which is remarkable when you consider how quickly technology moves.
Sure, new tools are invented all the time, and they might do things better than old ones. I use `eza` instead of `ls`; `fd` instead of `find` (mostly); `rg` instead of `grep` (mostly); `fzf` is a pretty essential addition to my workflow, and so on. But the underlying principles of these tools are still the same as the tools they're replacing. They're just slightly modernized and ergonomic versions of them.
Whether or not we need a `container` command, `from alpine`, or an entire new shell for that, is a separate topic. It could be argued that this could be accomplished with a few functions or standalone commands. Even if we do need this new tooling, that's great, but don't tell me that it's meant to replace a proven set of tools and workflows[1]. When containers stop being popular, will we still need this?
Also, "Daggerverse" and "modules"? Great, let's bring in the npm mentality to the shell, just what I needed.
[1]: Ah, they don't, it's meant to serve as a complement. Alright, fair enough. I'm lowering my pitchfork.
manno23
[dead]
I feel like it's getting harder to tell what dagger is _actually_ for these days.
At first we'd hoped it could replace jenkins - it provided an alternative way to run and debug CI pipelines - right on your machine! You could write in golang and just import what you needed. The dev direction feels more scattered now, trying to replace docker, be a new shell(?), and weirdly trying to be some kind of langchain? Doing something different doesn't imply better. A new set of complicated CLI args is no better than what we started with (shell scripts, or jenkinsfiles to integrate docker builds). I'm a little bummed that the project has seemingly drifted (in my view) from the original mission.