Disk I/O bottlenecks in GitHub Actions

Today we (Depot) are not, though some of our customers configure this. For the moment at least, the ephemeral public IP architecture makes it generally unnecessary from a rate-limit perspective.

From a performance / efficiency perspective, we generally recommend using ECR Public images[0], since AWS hosts mirrors of all the "Docker official" images, and throughput to ECR Public is great from inside AWS.

[0] https://gallery.ecr.aws/

Exactly, a ramdisk-backed writeback cache for the root volume for Linux. For macOS we wrote a custom nbd filter to achieve the same thing.

CI containers should probably run entirely from tmpfs.

You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection.

edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html

I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.

> Docker is a big culprit

Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it.

A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mo...

This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.

unsafe-io eliminates most fsyncs, but not all of them.

Let's start from the assumption that dpkg shouldn't commit to its database that package X is installed/updated until all the on-disk files reflect that. Then if the operation fails and you try again later (on next boot or whatever) it knows to check that package's state and try rolling forward.

> If you can't trust the kernel to close() then you can't trust it to fsync() or anything else either.

https://man7.org/linux/man-pages/man2/close.2.html

       A successful close does not guarantee that the data has been
       successfully saved to disk, as the kernel uses the buffer cache to
       defer writes.  Typically, filesystems do not flush buffers when a
       file is closed.  If you need to be sure that the data is
       physically stored on the underlying disk, use fsync(2).  (It will
       depend on the disk hardware at this point.)

Of course none of this matters much on an ephemeral Github Actions VM. There's no "on next boot or whatever". So this is one environment where it makes sense to bypass all this careful durability work that I'd normally be totally behind. It seems reasonable enough to say it's reached the page cache, it should continue being visible in the current boot, and tomorrow will never come.

"durability" isn't voodoo. Consider if dpkg updates libc.so and then you yank the power cord before the page cache is flushed to disk, or you're on a laptop and the battery dies.

> Kernel-level crashes, the only kind of crash that risks half-written files [...]

You can get half-written files in many other circumstances, eg on power outages, storage failures, hw caused crashes, dirty shutdowns, and filesystem corruption/bugs.

(Nitpick: trusting the kernel to close() doesn't have anythign to do with this, like a sibling comment says)

What's your experience developing a package manager for one of the world's most popular Linux distributions?

Maybe they know something you don't ?????

this is actually an interesting topic and it turns out kernel never made any promises about close() and data being on the disk :)

and about kernel-level crashes: yes, but you see, dpkg creates a new file on the disk, makes sure it is written correctly with fsync() and then calls rename() (or something like that) to atomically replace old file with new one.

So there is never a possibility of given file being corrupt during update.

> Kernel-level crashes, the only kind of crash that risks half-written files, are no more likely during dpkg than any other time. A bad update is the same bad update regardless, no better, no worse.

Imagine this scenario; you're writing a CI pipeline:

1. You write some script to `apt-get install` blah blah

2. As soon as the script is done, your CI job finishes.

3. Your job is finished, so the VM is powered off.

4. The hypervisor hits the power button but, oops, the VM still had dirty disk cache/pending writes.

The hypervisor may immediately pull the power (chaos monkey style; developers don't have patience), in which case those writes are now corrupted. Or, it may use ACPI shutdown which then should also have an ultimate timeout before pulling power (otherwise stalled IO might prevent resources from ever being cleaned up).

If you rely on sync to occur at step 4 during the kernel to gracefully exit, how long does the kernel wait before it decides that some shutdown-timeout occurred? How long does the hypervisor wait and is it longer than the kernel would wait? Are you even sure that the VM shutdown command you're sending is the graceful one?

How would you fsync at step 3?

For step 2, perhaps you might have an exit script that calls `fsync`.

For step 1, perhaps you might call `fsync` after `apt-get install` is done.

[flagged]

Thanks for sharing. I have a custom bash script which does the docker builds currently and swapping to useblacksmith/build-push-action would take a bit of refactoring which I don't want to spend the time on now. :-)