How to handle people dismissing io_uring as insecure? (2024)

I don't work at Google anymore and don't have any special insight into the internal adoption of io_uring, but I think it stands to reason that Google would benefit tremendously from rolling out a higher-performing way to do IO across their fleet. I mean, having myself done some lowish-level performance/optimization work and knowing that the impact of these kinds of changes is measurable and the scale is almost fleetwide, I wouldn't be surprised if the benefits - after major internal libraries/tools are also updated to use io_uring - are O(Really Big Money)

Having talked to members of their prodkernel team about other subjects, I also think they are competent enough to know the difference between "not ready" and "acceptably flawed". And believe me, the incentives are such that O(Really Big Money) optimization projects get staffed unless there is something making them infeasible.

Not everybody has the same threat model and security stance as Google and that's ok. But personally I would take their internal adoption of io_uring very seriously as a measure of whether it's safe for me to adopt it, especially if I'm running untrusted or third party software (including certain kinds of libraries).

Without going into the weeds, there has be some vendor support, and that vendor is obviously not google. How to convince people: Get it into RHEL.

yes but what this isn't telling you is that android has a long history of running hopelessly outdated kennels and it being very common that Linux kernel related android cves related to newish features have already been fixed upstream by generic improvements to that feature code

I like how someone helpfully added

> Although initial async offload design in io_uring could be problematic, later kernels changed the thread model. After such improvements, there were no known inherent problems with it and its development is very careful with new features. Considering that a performant async framework with a user facing API is complex, it was to be expected that issues would be found initially. After initial issues have been addressed, it is not any less secure than anything else in the kernel and io_uring acceptance quickly grew in production. Some of its criticism are also based on wrong or outdated assumptions.[14]

...but the only citation is a link to this GH thread, which doesn't support the claims made.

Jens Axboe replies on the very first line of the thread:

> As I'm sure you know, this is all mostly centered around a) google using an old kernel on android

People are saying: "Oxygen is blue and that's why the sky is blue." Someone is replying: "The sky isn't blue because of the oxygen." You are then saying: "Well what if the people who are saying that the sky is blue because of oxygen are right."

Although it gets a bit more complicated, the statement `io_uring` is insecure might be true, that's not really in dispute here. The people who are saying it, aren't saying it because they know it to be true, they are saying it because they heard about security issues in the context of `io_uring` and assumed that using `io_uring` would make your code less secure.

This is incorrect, the security issues are in security features in Linux which have not been updated to handle `io_uring`. This means that your application won't be any less/more secure when using `io_uring`. But your system might be less secure if you have support for `io_uring` enabled and applications can make use of it.

Moreover, the "security issues" are only undoing security related hardening you would have put in place over the baseline, they're not putting you below baseline.

That's why a statement such as `io_uring` is insecure isn't very useful.

If these people make the argument that: "I don't want to use `io_uring` because that would mean that security conscious system administrators would not want to run my software as a precaution." then it would make sense and nobody would be disputing it.

> Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong.

I think this is the right approach. We know that io_uring has a somewhat significant history of critical security issues. It's not enough just to point out that "these 3 critical CVEs were fixed in the last 12 months, it's secure now!"

Reputation and trust has to be built over a long period of time.

> Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong. If you start there, you will fail to convince people of anything, because you automatically dismiss their claim, without thinking about what they might have seen and what they might think.

Bingo. This is the correct approach. Very well said!

> you could look at https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring

CVE count by year:

  2019: 1
  2020: 1
  2021: 10
  2022: 15
  2023: 19
  2024: 21
  2025: 10

CVE statistics are also pretty hard to interpret in light of the kernel team’s willingness to assign CVE numbers for most any bugfixes.

This just reinforces the (maybe unfounded) impression that security is a secondary consideration, and performance is primary.

I'd use io_uring in a heartbeat on a dedicated system where the job is only I/O and security isolation isn't a concern. But multiuser/multiapplication/networked? Not a chance.

Isn't the issue here just that io_uring needs to be enhanced such that, when a seccomp-bpf filter is installed, the filter gets called to approve each SQE, before it gets executed?

I believe you can deny io_uring altogether with the syscalls io_uring_enter, io_uring_register, io_uring_setup?

This would be useful if you want to boot with io_uring but deny it for some sensitive workloads.

Is that a true limitation that cannot be overcome? Are solutions possible and/or available, but require further work to be shipped?

What regular filter for syscalls do you use?

Maybe it is (mine is for sure) but that's a "bitten by the dog once" case.

You got bitten and everyone around you assures you won't get bitten again, but the pain was real and you still have a scar from the event. Why bother or invest risking another bite if there are other places to be which had never bitten before.

Over my career I hated technology 3 times. First was PHP, second was Python during Python 2/3 fiasco and third was CoffeeScript.

Edit: till this day one of my favorite meme is titled "PHP: Training wheels without a bike"

It's with so many things! Some evergreens just from last week here on HN:

- Rails is horribly slow!

- Python is still stuck migrating to Python 3!

- MySQL doesn't scale!

- Haskell tooling sucks!

- io_uring is insecure!

- ... and dozens more just in the programming world. These are just off the top of my head. Probably hundreds more in the wider engineering world.

Programming languages are different. They usually can't be "repaired" once broken, because they need to maintain backwards compatibility.

the work that the people in PHP did for 7 is criminally underrated

It is kind of like Perl. Their views of Perl is still stuck at CGIs of the 90's.

I still think that Perl was and is still a better language than PHP and Python. And people never had a serious look at raku because of the Perl heritage.

But that ship has sailed.

It’s true of most languages that have been around for a while. E.g. people who don’t write Javascript think of it as it was pre-ES6, not to mention that many people don’t write JS directly and use Typescript instead.

Same with Java, it’s somehow stuck in time in Java 6 in some people’s minds.

Wow. My view is stuck at php 4. Didn't even know 5 existed. I only remember Facebook making a big deal of hhvm and php 7 and laravel being considered great but not as great as rails or django.

It's tragic, and it bothers me that Jens Axboe's work is suffering due to this. Obviously, with the clarity of hindsight, this might have been avoided. Now the cost of the damage it high.

My idea: consider formal verification. A rigorous formal proof of behavior is capable of solving actual flaws and probably capable of overcoming the bad PR.

That would require a great and ongoing effort. However, given the nature of io_uring, it's likely rather amenable to formal verification, and ultimately it is probably necessary. Perhaps our new "AI" tools could greatly reduce the effort.

Anyhow, that's my brilliant thought...