Towards fearless SIMD, 7 years later

You started with this... take:

> Rust feels like a Python developer’s idea of a high-performance computing language. It’s a great language for many kinds of applications — just not when you need to squeeze out every bit of performance from advanced hardware.

And went on to say that Rust in particular is problematic for:

> byte-level processing

It's particularly odd for you to say this given that the memchr Rust crate is just as fast as stringzilla for substring search. And is generally faster in cases where the needle is invariant, because stringzilla doesn't have APIs for amortizing searcher construction.

We've had a discussion about this before where I provided receipts[1] and we have not had a meeting of the minds on this point. The thing I'm trying to achieve here is to point out that your claims are contested and there is evidence that you're wrong. And so I'd caution readers to also in turn question your higher level claims about Rust being a "Python developer's idea of a high-performance computing language."

[1]: https://old.reddit.com/r/rust/comments/1ayngf6/memchr_vs_str...

Ah yeah, gather/scatter are indeed a rather problematic thing for autovectorization. That said, with no-alias info (which Rust has a lot of) it's possible: https://rust.godbolt.org/z/zTfo9nxhd.

Unfortunately it doesn't get autovectorized without the unsafes, but theoretically it should be possible-ish for bounds checking to be autovectorized (most problematic aspect being that it might be hard to annoying-to-impossible to ensure that in the case of multiple panic/UB sources the proper one happens first).

I'd imagine in any non-trivial situation you'd want a custom layer of abstractions over whatever the language provides for all languages. For that a portable-simd thing is actually a rather good base, on which you could add custom arch-specific abstractions/ops as desired.

Not sure what's problematic with mixed-precision (I know SVE is rather weird for mixed-width elements, but that's about it?), though I primarily don't care about float stuff generally. Also no clue what's problematic with byte-level stuff.

Indeed there are still a bunch of things that you want proper manual SIMD for (hell the SIMDful project I work on has an entire DSL for doing nice SIMD), but autovectorization still covers a good amount.

> except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers.

Counterexamples: Chromium's byte-level HTML scanning, several var-len bit packing codecs, and Gemma.cpp's matmul is mixed-precision (fp8->bf16->fp32->fp64). All written with the Highway general-purpose SIMD wrapper. Please revise your post or expand upon the structure/dispatch concern.

Don't think any language has standardized SIMD that's particularly nice; Highway is probably a quite nice library on C++, though I haven't used it enough to get comfortable.

The thing I use for my projects is Singeli[1], a DSL specifically made for SIMD stuff (though it's capable of generally sanely doing abstractions over types/operations/loops; it's just a fancy code generator). Obligatory disclaimer that I'm one of the two people working on its design. It's far from a nice experience starting from nothing, but it's pretty nice for what I do.

CBQN's the main place it's used, can click around its source: https://github.com/dzaima/CBQN/tree/develop/src/singeli/src

Its goal isn't necessarily to unify architectures, but rather make it as easy as possible to make abstractions that do; as such its built-in includes for x86 don't have arbitrary shuffling, but do provide a sane interface over the cases that are supported in a single instruction (not including constant creation/loading), and those can run on NEON unchanged (assuming they're ran on 128-bit vectors, of course, as NEON doesn't support larger ones); and, with Singeli just generating C/C++ currently, you can just map in __builtin_shufflevector if desired. e.g. here's your AVX2 `pre`:

    include 'skin/c' # defines infix a+b & a*b etc to run __add/__mul/... (yes, those aren't here by default, and you can define custom infix/prefix ops)
    include 'arch/c' # defines __add & __mul to do C ops
    include 'arch/iintrinsic/basic' # not necessary for a shuffle, but provides basic x86 arith ops
    include 'arch/iintrinsic/select' # x86 shuffles; there's similar 'arch/neon_intrin/basic' & 'arch/neon_intrin/select' for NEON

    fn pre(inp: [32]i8, out: *[32]i8) : void = {
      store{out, 0, vec_shuffle{16, inp, merge{ # 16 specifies to repeat per 16-elt lane
        range{8}*2+1,            # lower half: 8 Y components; compile-time index calculations
        range{4}*4, range{4}*4+2 # upper half: (4 * U), (4 * V).
      }}}
    }

    fn sigmoid{E}(r:*E, x:*E, n:ux) : void = {
      def V = arch_preferred_vector{E}
      @fancy_loop{V,4}(r in tup{'dst',r}, x, M in 'mask' over n) {
        # this loop body is generated 3 times for x86 & ARM - with x being a 4-elt tuple (core unrolled loop); a 1-elt tuple and no masking; a 1-elt tuple and masking
        if (any_hom{M, ...(x!=x)}) {
          emit{void, 'abort'}
        }
        r{x / __sqrt{1 + x*x}}
      }
      # were it not for a bug in tuple loop var mutation in Singeli having undesired pervasion, this would be possible:
      # @fancy_loop{V,4}(r, x, M in 'mask' over n) {
      #   if (...) ...
      #   r = x / __sqrt{1 + x*x}
      # }
    }
    export{'sigmoid', sigmoid{f32}}

(I'm not actually particularly expecting interest in Singeli; I just like writing stuff :) )

[1]: https://github.com/mlochbaum/Singeli

While it's true that par_iter() uses a concurrent data structure under the hood, it's specifically designed to use work-stealing to avoid needing threads to communicate.

Why would putting a lock over a global workqueue be faster than per-thread workqueues that don't require inter-thread communication (except in the case where work-stealing is required)?

The rayon library uses work stealing for this. Its parallel iterators offer some control of splitting strategies and granularity, so you can tune the trade-off between full utilization and cost of moving things between threads.

Additionally, in Bevy, independent queries (systems) are executed in parallel, so there's always something else to do, except your one worst loop.

> I'm curious what makes this so different from my experience. I rarely ever have to write "unsafe", and I'm writing quite low-level engine code that certainly uses bit manipulation.

Perhaps your usecase is similar enough to eg JavaScript engines? Because that's a usecase that browser writers would at least have in mind?

That's not a negative aspect of Rust, and I'm not picking out a list of things I dislike about Rust. Making a statement that isn't unequivocally positive does not equal criticizing something.

Cargo is, however, a similarity with JS. On the whole a good one. Also, cargo works much more like npm than maven, for example.

What's the problem of using toolchain-specific features instead of behavior defined by the standard? Isn't this the bread and butter of embedded development and the reason why some behavior is purposely left undefined in the standard?

Exactly this. Most of the art of it is avoiding UB and sticking to implementation-defined behavior.

I've found it much easier to use from writing a web service to writing a high performance DB that outperforms RocksDB and clearly people are using it for things like writing operating systems as well as game engines. I'm not sure what in your mind falls under "advanced projects" but I suspect it's something like number crunching (although you link StringZilla so not sure).

I'm still not seeing any description of specific challenges you feel are harder in Rust than in C/C++. In my mind Rust is completely equivalent in being able to accomplish the same tasks.

I don’t think Rust is worse at SIMD code, though.

In my experience Rust is much easier for advanced projects. Once you get into high performance code, C++ takes much more time, for two reasons.

1. With C++ you have to think a lot. Like, a lot. To convince yourself that what you are doing is safe, and that the lifetimes and races of various objects are safe. Rust takes a huge load off, by failing to compile your mistakes. You know that for the code that doesn't use "unsafe", if it compiled, then you don't even have to think about races or lifetimes.

2. C++ has so many places where copies sneak in. Tracking down needless copies in C++, for object types where it's not as simple as "just disable copy construct & copy assign", can be tricky and is extremely brittle to future changes of code. And not just for the CPU cost of copies, but RAM costs too.

And I say that as someone who's been coding C++ on a daily basis since the 1990s.

I'm not proficient in Rust, but API wise I'd conceptually define types like f32xn or f32s, which have the number of elements that fit into a vector register for your target architecture, so 4 for NEON/SSE, 8 for AVX and 16 for AVX512.

I can recommens lookig at the highway library: https://github.com/google/highway

I guess being unpriviliged this still guarantees the dragons are restricted your process, otherwise the chip would have a security problem. So as a heuristic you could fork your process, and try to execute SIMD operations (detecting known nonstandard versions and failing them), and if results seems fine send a "ok" flag up a pipe. (or compile this into a separate test executable)

Just because it's reserved doesn't mean anything can happen. On x86_64, you get a clearly defined error when you use reserved bits and the like. I don't know if RISC-V is the same, but if it isn't it's because they chose to be vague, not because that's what it means to be reserved.