lsr: ls with io_uring
53 comments
·July 18, 2025tln
The times seem sublinear, 10k files is less than 10x 1k files.
I remember getting in to a situation during the ext2 and spinning rust days where production directories had 500k files. ls processes were slow enough to overload everything. ls -F saved me there.
And filesystems got a lot better at lots of files. What filesystem was used here?
It's interesting how well busybox fares, it's written for size not speed iirc?
SillyUsername
Love it.
I'm trying to understand why all command line tools don't use io_uring.
As an example, all my nvme's on usb 3.2 gen 2 only reach 740MB/s peak.
If I use tools with aio or io_uring I get 1005MB/s.
I know I may not be copying many files simultaneously every time, but the queue length strategies and the fewer locks also help I guess.
tyingq
Probably historical preference for portability without a bunch of #ifdef means platform+version-specific stuff is very late to get adopted. Though, at this point, the benefit of portability across various posixy platforms is much lower.
Retr0id
Has anyone written an io_uring "polyfill" library with fallback to standard posix-y IO? It could presumably be done via background worker threads - at a perf cost.
vlovich123
Seems like a huge lift since io_uring is an ever growing set of interfaces that is encompassing more and more of the kernel surface area. Also, the problem tends to not necessarily be that the io_uring interface isn’t available at compile time but a) the version you distribute to has a kernel with it disabled or you don’t have permission to use it meaning you need to do LD_preload magic or use a framework b) the kernel you’re using supports some of the interfaces you’re trying to use but not all. Not sure how you solve that one without using a framework.
But I agree. It would be cool if it was transparent, but this is actually what a bunch of io-uring runtimes do, using epoll as a fallback (eg in Rust monoio)
elcapitan
iirc io_uring also had some pretty significant security issues early on (a couple of years ago). Those should be fixed by now, but that probably dampened adoption as well.
Thaxll
io_uring is a security nightmare.
pjc50
How so?
Thaxll
This is a good read on the topic: https://chomp.ie/Blog+Posts/Put+an+io_uring+on+it+-+Exploiti...
sim7c00
you give process direct access to a piece of kernel memory. its a reason why there is separation. thats all.
tln
Thats a great speed boost. What tools are these?
never_inline
Poe's law hits again.
superkuh
One reason is so that they work in all linux environments rather than just bleeding edge installs from the last couple years.
quibono
Lovely, I might try doing this for some other "classic" utility!
A bit off-topic too, but I'm new to Zig and curious. This here: ``` const allocator = sfb.get();
var cmd: Command = .{ .arena = allocator };kristoff_it
Allocator is an interface so you write library code only once, and then the caller decides which concrete implementation to use.
There's cases where you do want to change your code based on the expectation that you will be provided a special kind of allocator (e.g. arenas), but that's a more niche thing and in any case it all comes together pretty well in practice.
IggleSniggle
Caveat emptor, I don't write Zig but followed its development closely for awhile. A core design element of zig is that you shouldn't be stuck with one particular memory model. Zig encourages passing an allocator context around, where those allocators conform to a standardized interface. That means you could pass in different allocators with different performance characteristics at runtime.
But yes, there is a default allocator, std.heap.page_allocator
danbruc
Why does this require inventing lsr as an alternative to ls instead of making ls use io_uring? It seems pretty annoying to have to install replacements for the most basic command line tools. And especially in this case, where you do not even do it for additional features, just for getting the exact same thing done a bit faster.
tiagod
You don't have to install it. You can modify ls yourself too.
nailer
`ls` is in C, `lsr` is in Zig. The `lsr` programmer probably doesn't want to make new code in C.
null
mschuster91
> Why does this require inventing lsr as an alternative to ls instead of making ls use io_uring?
Good luck getting that upstreamed and accepted. The more foundational the tools (and GNU coreutils definitely is foundational), the more difficult that process will be.
Releasing a standalone utility makes iteration much faster, partially because one is not bound to the release cycles of distributions.
WorldMaker
In the history of Unix its also a common way to propose tool replacements, for instance how `less` became `more` on most systems, or `vim` became the new `vi` which in its day became the new `ed`.
nailer
> instance how `less` became `more` on most systems
How `more` became `less`.
The name of 'more' was from paging - rather than having text scroll off the screen, it would show you one page, then ask if you wanted to see 'more' and scroll down.
'less' is a joke by the less authors. 'less is more' etc.
s1mplicissimus
> Releasing a standalone utility makes iteration much faster, partially because one is not bound to the release cycles of distributions.
which certainly is a valid way or prioritizing. similarly, distros/users may prioritize stability, which means the theoretical improvement would now be stuck in not-used-land. the value of software appears when it's run, not when it's written
KPGv2
> the value of software appears when it's run, not when it's written
Have you ever tried to contribute to open source projects?
The question was why wouldn't someone writing software not take the route likely to end in rejection/failure. I don't know about you, but if I write software, I am not going to write it for a project whose managers will make it difficult for my PR to be accepted, and that 99% likely it never will be.
I will always contribute to the project likely to appreciate my work and incorporate it.
I'll share an anecdote: I got involved with a project, filed a couple PRs that were accepted (slowly), and then I talked about refactoring something so it could be tested better and wasn't so fragile and tightly coupled to IO. "Sounds great" was the response.
So I did the refactor. Filed a PR and asked for code review. The response was (after a long time waiting) "thanks but no, we don't want this." PR closed. No feedback, nothing.
I don't even use the software anymore. I certainly haven't tried to fix any bugs. I don't like being jerked around by management, especially when I'm doing it for free.
(For the record, I privately forked the code and run my own version that is better because by refactoring and then writing tests, I discovered a number of bugs I couldn't be arsed to file with the original project.)
kmeisthax
[dead]
ninkendo
I wonder how it performs against an NFS server with lots of files, especially one over a kinda-crappy connection. Putting an unreliable network service behind blocking POSIX syscalls is one of the main reasons NFS is a terrible design choice (as can be seen by anyone who's tried to ctrl+c any app that's reading from a broken NFS folder), but I wonder if io_uring mitigates the bad parts somewhat.
loeg
> as can be seen by anyone who's tried to ctrl+c any app that's reading from a broken NFS folder
Theoretically "intr" mounts allowed signals to interrupt operations waiting on a hung remote server, but Linux removed the option long ago[1] (FreeBSD still supports it)[2]. "soft" might be the only workaround on Linux.
[1]: https://man7.org/linux/man-pages/man5/nfs.5.html
[2]: https://man.freebsd.org/cgi/man.cgi?query=mount_nfs&sektion=...
rkangel
This was more interesting for the tangled.sh platform it's hosted on. Wasn't aware of that!
Imustaskforhelp
Really interesting, the difference is real though I would just hope that some better coloring support could be added because I have "eza --icons=always -1" command set as my ls and it looks really good, whereas when I use lsr -1, yes the fundamental thing is same, the difference is in the coloring.
Yes lsr also colors the output but it doesn't know as many things as eza does
For example .opus will show up as a music icon and with the right color (green-ish in my case?) in eza whereas it would be shown up as any normal file in lsr.
Really no regrets though, its quite easy to patch I think but yes this is rock solid and really fast I must admit.
Can you please create more such things but for cat and other system utilities too please?
Also love that its using tangled.sh which is using atproto, kinda interesting too.
I also like that its written in zig which imo feels way more easier for me to touch as a novice than rust (sry rustaceans)
johnisgood
As for coloring support, I think the best way would be to implement LS_COLORS / dircolors. My GNU ls looks nice.
iknowstuff
til u segfault
ReDress
I've been playing around with io_uring for a while.
Still, I am yet to come across a some tests that simulate typical real life application workload.
I heard of fio but are yet to check how exactly it works and whether it might be possible to simulate real life application workload with it.
Bender
I am curious what would happen if ls and other commands were replaced using io_uring and kernel.io_uring_disabled was set to 1. Would it fall back to an older behavior or would the ability to disable it be removed?
adgjlsfhk1
It's a shame to see uutils doing so poorly here. I feel like they're our best hope for an organization to drive this sort of core modernization forward, but 2x slower than GNU isn't a good start.
the8472
io_uring doesn't support getdents though. so the primary benefit is bulk statting (ls -l). It'd be nice if we could have a getdents in flight while processing the results of the previous one.
This seems more interesting as demonstration of the amortized performance increase you'd expect from using io_uring, or as a tutorial for using it. I don't understand why I'd switch from using something like eza. If I'm listing 10,000 files the difference is between 40ms and 20ms. I absolutely would not notice that for a single invocation of the command.