The Pain That Is GitHub Actions

I came to the exact same conclusion accidentally in my first role as a Tech Lead a few years back.

It was a large enterprise CMS project. The client had previously told everyone they couldn't automate deployments due to the hosted platform security, so deployments of code and configs were all done manually by a specific support engineer following a complex multistep run sheet. That was going about as well as you'd expect.

I first solved my own headaches by creating a bunch of bash scripts to package and deploy to my local server. Then I shared that with the squads to solve their headaches. Once the bugs were ironed out, the scripts were updated to deploy from local to the dev instance. Jenkins was then brought in an quickly setup to use the same bash scripts, so now we had full CI/CD working to dev and test. Then the platform support guy got bored manually following the run sheet approach and started using our (now mature) scripts to automate deployments to stage and prod.

By the time the client found out I'd completely ignored their direction they were over the moon because we had repeatable and error free automated deployments from local all the way up to prod. I was quite proud of that piece of gorilla consulting :-)

Strongly isolated systems like Nix and Bazel are amazing for giving no-fuss local reproducibility.

Every CI "platform" is trying to seduce you into breaking things out into steps so that you can see their little visualizations of what's running in parallel or write special logic in groovy or JS to talk to an API and generate notifications or badges or whatever on the build page. All of that is cute, but it's ultimately the tail wagging the dog— the underlying build tool should be what is managing and ordering the build, not the GUI.

What I'd really like for next gen CI is a system that can get deep hooks into local-first tools. Don't make me define a bunch of "steps" for you to run, instead talk to my build tool and just display for me what the build tool is doing. Show me the order of things it built, show me the individual logs of everything it did.

Same thing with test runners. How are we still stuck in a world where the test runner has its own totally opaque parallelism regime and our only insight is whatever it chooses to dump into XML at the end, which will be probably be nothing if the test executable crashes? Why can't the test runner tell the CI system what all the processes are that it forked off and where each one's respective log file and exit status is expected to be?

I came from the semiconductor industry, where everything was locally hosted Jenkins + bash scripts. The Jenkins job would just launch the bash script that was stored in perforce(vcs), so all you had to do to run things locally was run the same bash script.

When I joined my first web SaaS startup I had a bit of a culture shock. Everything was running on 3rd party services with their own proprietary config/language/etc. The base knowledge of POSIX/Linux/whatever was almost completely useless.

I'm kinda used to it now, but I'm not convinced it's any better. There are so many layers of abstraction now that I'm not sure anybody truly understands it all.

This.

I heavily invested in a local runner based CI/CD workflow. First I was using gogs and drone, now the forgejo and woodpecker CI forks.

It runs with multiple redundancies because it's a pretty easy setup to replicate on decentralized hardware. The only thing that's a little painful is authentication and cross-system pull requests, so we still need our single point of failure to merge feature branches and do code reviews.

Due to us building everything in go, we also decided to have always a /toolchain/build.go so that we have everything in a single language, and don't need even bash in our CI/CD podman/docker images. We just use FROM scratch, with go, and that's it. The only exception being when we need to compile/rebuild our ebpf kernel modules.

To me, personally, the Github Actions CVE from August 2024 was the final nail in the coffin. I blogged about it in more technical detail [1] and guess what was the reason that the TJ actions have been compromised last week? Yep, you guessed right, the same attack surface that Github refuses to fix, a year later.

The only tool, as far as I know, that somehow validates against these kind of vulnerabilities, is zizmor [2]. All other tools validate schemas, not vulnerabilities and weaknesses.

[1] https://cookie.engineer/weblog/articles/malware-insights-git...

[2] https://github.com/woodruffw/zizmor

Whenever possible I now just use GitHub actions as a thin wrapper around a Makefile and this has improved my experience with it a lot. The Makefile takes care of installing all necessary dependencies and runs the relevant build/Test commands. This also enables me to test that stuff locally again without the long feedback loop mentioned in other comments in this thread.

After years of trial and error our team has come to the same conclusion. I know some people might consider this insanity, but we actually run all of our scripts as a separate C# CLI application (The main application is a C# web server). Effectively no bash scripts, except as the entry point here and there. The build step and passing the executable around is a small price to pay for the gain in static type checking, being able to pull in libraries as needed, and knowing that our CI is not going to down because someone made a dumb typo somewhere.

The other thing I would add is consider passing in all environment variables as args. This makes it easy to see what dependencies the script actually needs, and has the bonus of being even more portable.

Am I an outlier in that not only do I find GitHub actions pleasant to use, but that most folks over complicate their CI/CD pipelines? I've had to re-write alot of actions configurations over the last few years, and in every case, the issue was simply not thinking through the limits of the platform, or when things would be better to run as custom docker images (which you can do via GitHub Actions) etc.

It tends to be that folks want to shoehorn some technology into the pipeline that doesn't really fit, or they make these giant one shot configurations instead of running multiple small parallel jobs by setting up different configurations for different concerns etc.

The way that gitlab shines is just fundamentally better than GitHub actions.

It's really easy to extend and compose jobs, so it's simple to unit test your pipeline: https://gitlab.com/nunet/test-suite/-/tree/main/cicd/tests?r...

This way I can code my pipeline and use the same infrastructure to isolate groups of jobs that compose a relevant functionality and test it in isolation to the rest of the pipeline.

I just wish components didn't have such a rigid opinion on folder structure, because they are really powerful, but you have to adopt gitlab prescription

It really depends on what you do?

GitHub CI is designed in a way which tends to work well for

- languages with no or very very cheap "compilation" steps (i.e. basically only scripting languages)

- relatively well contained project (e.g. one JS library, no mono repo stuff)

- no complex needs for integration tests

- no need for compliance enforcement stuff, especially not if it has to actually be securely enforced instead of just making it easier to comply then not to comply

- all developers having roughly the same permissions (ignore that some admin has more)

- fast CI

but the moment you step away from this it just falls more and more and more apart and I every company which doesn't fit the constraints above I have seen so far has non stop issues with GitHub Actions.

But the worst part, which maybe is where a lot of hatred comes from, is that it's there for cheap maybe even free (if you anyway pay for GitHub) and it doesn't need an additional contract, billing, etc. Not an additional vetting of 3rd party companies. Doesn't need managing your own CI service etc. So while it does cause issues non stop it also seems initially still "cheaper" solution for the company. And then when your company realizes it's not and has to setup their own GitHub runner etc. it probably isn't. But that is if you properly account dev time spend on "fixing CI issues" and even then there is the sunk cost fallacy because you already spend so much time to make github actions work and you would have to port everything over etc. Also, realistically speaking, a lot of other CI solutions are only marginally better.

I use GH actions. You should treat it like all build systems: let them do what they are good at and nothing else. The rest should be shell scripts or separate docker containers. If it gets complicated, dumb it down to "run this script". Scripts are a lot easier to write and debug than thousands of lines of yaml doing god knows what.

The problem isn't github actions but people overloading their build and CI system with all sorts of custom crap. You'd have a hard time doing the same thing twenty years ago with Ant and Hudson (Jenkin's before the fork after Oracle inherited that from Sun). And for the same reason. These systems simply aren't very good as a bash replacement.

If you don't know what Ant is. That was a popular build system for Java before people moved the problem to Maven and then to Gradle (without solving it). I've dealt with Maven files that were trying to do all sorts of complicated things via plugins that would have amounted to two or three lines of bash. Gradle isn't any better. Ant at least used to have simple primitives for "doing" things. But you had to spell it out in XML form.

The point of all this, is that build & CI systems should mainly do simple things like building software. They shouldn't have a lot of conditional logic, custom side effects, and wonky things that may or may not happen depending on the alignment of the moon and stars. Debugging that stuff when it fails to work really sucks.

What helps with Yaml is using Yaml generators. I've used a Kotlin one for a while. Basically, you get auto complete, syntactical sanity, type checking and if it compiles it runs. Also makes it a lot easier to discover new parameters, plugin version updates, etc.

We see quite a lot of organisations inside because of the business we have, and, while usually this is not our task, when I hear these stories and see people struggle with devops stuff in reality, the first thing we push for is to do anything to dumb it down and remove all the dependencies on 3rd party providers so we are back to having everything run again like, in this case, the hello world of github actions. It is literally always the case that the people who complain have this (very HN, so funny you say that) thing of absolutely grossly overarchitecting and writing things that are just there because they read it on HN/some subreddits/discord. We sometimes walk into struggling teams where we check the commits / setup only to find out they did things like switch package manager/bundler/etc 5x in the past year (this is definitely a HN thing where a new packagemanager for js pops up every 14 minutes). Another terrible thing looking at 10+ year codebases, we see js, ts, py, go, rust and when we ask wtf, they tell us something something performance. Of course the language was never the bottleneck of these (people here would be pretty scared to see how bad database setups are even for multi million$ projects in departmental or even enterprise wide; the DBA's in the basement know but they are not consulted for various reasons), mostly LoB, apps. And the same happens with devops. We only work for large companies, almost never startups, and these issues are usually departmental (because big bad Java/Oracle IT in the basement doesn't allow anything so they have budgets to do their own), but still, it's scary how much money is being burnt on these lame new things that won't survive anyway.

Sounds like you have the same pain points as everyone else; you're just more willing to ignore them.

I am with the author - we can do better than the status quo!

Unless I'm misunderstanding, you can use workflow_dispatch to avoid having to make useless commits to trigger actions.

I have a small gripe that I think exemplifies a bigger problem. actions/upload-artifact strips executable permissions from binaries (1). The fact they fucked this up in the first place, and six years later haven't fixed it, gives me zero confidence in the team managing their platform. And when I'm picking a CI/CD service, I want reliability and correctness. GH has neither.

When it takes all of a day to self host your own task runner on a laptop in your office and have better uptime, lower cost, better performance, and more correct implementations, you have to ask why anyone chooses GHA. I guess the hello-world is convincing enough for some people.

(1) https://github.com/actions/upload-artifact/issues/38

You must have simple, straightforward flow touched only by a handful of folks max.

The world is full of kafkaesque nightmares of Dev-ops pipeline "designed" and maintained by committees of people.

It's horrible.

That said, for some personal stuff I have Google Cloud Build that has a very VERY simple flow. Fire, forget and It's been good.

You might like 'git commit --allow-empty' to make your somewhat-useless commits.

But honestly, doesn't github now have a button you can press to retrigger actions without a commit?

GitHub Actions are least hassle, when you don't care about how much compute time you are burning through. Either because you are using the free-for-open-source repositories version, or because your company doesn't care about the cost.

If you care about the compute time you are burning, then you can configure them enough to help with that, but it quickly becomes a major hassle.

Feel this pain so much. If you are debugging Github Action container builds, and each takes over ~40 minutes to build.. you can burn through a whole work day only testing six or seven changes.

There has to be a better way. How has nobody figured this out?

With GitLab, I have found https://github.com/firecow/gitlab-ci-local to be an incredible time-saver when working with GitLab pipelines (similar to https://github.com/nektos/act for GitHub)

I wish GitLab/GitHub would provide a way to do this by default, though.

This is exactly a big piece of our frustration -- the terrible feedback loop and how much mental space it wastes. OP does talk about this at the end (babysitting the endless "wip" commits till something works).

Highly recommend nektos/act, and if it's something complex enough, you can Ssh into the server to investigate. There are many action that facilitate this.

I use LLMs for a lot of things these days, but maybe the most important one is as a focus-preserving mechanism for exactly these kinds of middle-ground async tasks that have a feedback loop measured in a handful of minutes.

If the process is longer than a few minutes, I can switch tasks while I wait for it. It's waiting for those things in the 3-10 minute range that is intolerable for me: long enough I will lose focus, not long enough for me to context switch.

Now I can bullshit with the LLM about something related to the task while I wait, which helps me to stay focused on it.

Glad I’m not the only one. GitLab runners just make sense to me. A container you run scripts in.

I have some GitHub actions for some side projects and it just seems so much more confusing to setup for some reason.

> I'm beyond confused how it got to that state.

A few years back I wanted to throw in the towel and write a more minimal GHA-compatible agent. I couldn't even find where in the code they were calling out to GitHub APIs (one goal was to have that first party progress UI experience). I don't know where I heard this, so big hearsay warning, but apparently nobody at GitHub can figure it out either.

> Sad to see Earthly halting development and Dagger jumping on the AI train :(. Hopefully we'll get a proper alternative.

Hi, Dagger CEO here. We're advertising a new use case for Dagger (running AI agents) while continuing to support the original use case (running complex builds and tests). Dagger has always been a general purpose engine, and our community has always used it for more than just CI. It's still the exact same engine, CLI, SDKs and observability stack. It's not like we're discontinuing a product, to the contrary: we're getting more workloads on the platform, which benefits all our users.

A lot of GH actions teams were impacted by layoffs in November.

Example:

https://github.com/actions/runner/pull/2477#issuecomment-244...

Presumably the issue is that GH underpriced Actions such that it's not worth improving because driving more usage won't drive revenue, and that then forced prices down for everyone else because everyone fixed on the Actions pricing.

I might have missed the news, but I did not find anything in regards to earthly stopping development

What happened there?

We switched to Depot last week. Our Rust builds went down from 20+ minutes to 4-8 minutes. The easy setup and their docker builds with fast caching are really good.

I think this is where things went off the rails for him. Commiting back to the same branch that is running CI has too many gotchas in any CI system. You touched on the first issue, the remote branch immediately deviates unexpectedly from the local branch. Care has to be taken not to trigger additional CI runs from that commit.

I do such things with pre-commit.

Doing it in CI sounds like making things more complicated by resetting to remote branches after pushing commits. And, in the worst case, something that actually brakes code that works locally.

That part immediately made me short circuit out of the piece. That sounds like a recipe for disaster and an unnecessary complexity that just brings loads of new failure modes. Not a best practice.

Trivial mistakes in PRs are almost always signs of larger errors.

I'm new to CI auto-fixes. My early experience with it is mixed. I find it annoying that it touches my code at all, but it does sometimes allow a PR to get further through the CI system to produce more useful feedback later on. And then a lot of the time I end up force-pushing a branch that is revised in other ways, in which case I fold in whatever the CI auto-fix did, either by squashing it in or by applying it in some other way.

(Most of the time, the auto-fix is just running "cargo fmt".)

By what measure is this "exponentially faster"? Surely GH doesn't take an exponential time in the number of steps of the workflow...

Depot looks nice, but also looks fairly expensive to me. We're a small B2B company, just 10 devs, but we'd be looking at 200+500 = $700/mo just for building and CI.

I guess that would be reasonable if we really needed the speedup, but if you're also offering a better QoL GHA experience then perhaps another tier for people like us who don't necessarily need the blazing speed?

Depot is fantastic. Can heavily recommend it. It’s like magic when your builds suddenly take 1m instead of 5+ just by switching the runner.

> Things like 10 GB cache limits in GitHub

10,000,000,000 bytes should be enough for anyone! It really is a lot of bytes...

I think the HN community at large had a bit of a learning experience a couple of days ago.

"Defaults matter" is a common phrase, but equally true is: "the pattern everyone recommends including example documentation matters".

It is fair to criticise the usage of GH Actions, just like it's fair to criticise common usage patterns of MySQL that eat your data - even if smarter individuals (who learn from deep understanding, or from being burned) can effectively make correct decisions, since the population of users are so affected and have to learn the hard way or be educated.

Pinning dependencies is trading one problem for another.

Yes, your builds will work as expected for a stretch of time, but that period will come to an end, eventually.

Then one day you will be forced to update those pinned dependencies and you might find yourself having to upgrade through several major versions, with breaking changes and knock-on effects to the rest of your pipelines.

Allowing rolling updates to dependencies helps keep these maintenance tasks small and manageable across the lifetime of the software.

But there is no transitive locking like package manager lockfiles. So if I depend on good/foo@hash, they depend on bad/hacked@v1 and V1 gets moved to malicious version I get screwed.

This is for composite actions. For JS actions what if they don't lock dependencies but pull whatever newest package at action setup time? Same issue.

Would have to transitively fork everything and pin it myself, and then keep it updated.

I have been using renovate, which automatically pins, and updates, hashes. So I can stay lazy, and only review the new hash when a renovate PR gets opened: https://docs.renovatebot.com/modules/manager/github-actions/...

> It's just that function calls can only appear in very limited places in the program (only inside `steps`), and to define a function, you have to create a Git repository.

FYI there is also `on: workflow_call` which you can use to define reusable jobs. You don't have to create a new repository for these

https://docs.github.com/en/actions/writing-workflows/workflo...

This is what I do. I've written 90% of the logic into a Go binary and GitHub Actions just calls out to it at certain steps. It basically just leaves GHA doing the only thing it's decent at...providing a local UI for pipelines. The best part is you get unit tests, can dogfood the tool in its own pipeline, and can run stuff locally (by just having the CLI nearby).

Makes migrations easier too; better to let gitHub or gitlab etc to just be the platform to host source code and trigger events which you decide how to deal with. Your CI itself should be another source controlled repo that provides the features for the application code's thin CI layer to invoke and use. That allows you to be able to run your CI locally in a pretty realistic manner too.

I have done something similar with Jenkins and groovy CI library used by Jenkins pipeline. But it wasn't super simple since a lot of it assumed Jenkins. I wonder if there is a more cleaner open source option that doesn't assume any underlying platform.

> Usually if you’re using it, it’s because you’re forced to.

Like teams.

It's funny, I've used them all too. . . I like GHA overall but it sure has its quirks.

Anyone who claims that GHA is garbage and any of the others are amazing is either doing something very basic or is crazy, or lying.

At the end of the day, you run shell scripts and commands using a YAML based config language (except for Jenkins). Amazingly, it's hard to build something that does that with the right abstractions and compromises between flexibility and good hygiene.

> GHA is great because of things like Dependabot [...] so you don't have to work hard to get anything advanced or specific to work.

That may have been true before GitHub decided that PRs can't access repository secrets anymore. Apparently now you can at least add these secrets to Dependabot too (which is still duplicate effort for setup and any time you rotate secrets), but at the time when the change was introduced there were only weird workarounds.

Well but that’s the problem. You cannot fully automate this. You have to manually check the diff of each dependency and only accept the dependabot PR if the changes are safe.

The only automation that I know of is cargo vet. Although it doesn’t work for GitHub Actions, the idea sounds useful. Basically, vet allows people who trust each other to vet updates. So one person verifies the diff and then approves the changes. Next, everyone who trusts this person can update the dependency automatically since it has been “vetted”.

[1]: https://github.com/mozilla/cargo-vet

Wasn’t part of the problem though that renovate was automatically upgrading people to the compromised hash? Or is that just the fault of people configuring it to be too aggressive with upgrades?