How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

What happened to him?

He's a total legend, yet apparently he's never met Bill Gates in person from what he said in an interview in the Dave's Garage YouTube channel a few years ago. You'd think that someone who's been that prominent for so long in the company would have been invited to a company dinner where he was present or something.

Small thing but I love the effort he puts into actually coding up his examples instead of screenshots. For example: https://devblogs.microsoft.com/oldnewthing/20250414-00/?p=11...

He has many better ones but that's the latest one I've seen

Raymond knows everything. From microcode bugs on Alpha AXP to template meta programming to UI.

There are fast instructions (e.g., REP STOSx, AVX zero stores, dc zva) and tricks (MTE, zero pages), but no magic CPU instruction exists that transparently and efficiently randomizes or zeros the stack on function calls. You think there would be one and I bet there are on some specialized high-security systems, but I'm not sure even where you would find such a product. Telecom certainly isn't it.

I'm a bit distant from this stuff, but it looks like C++26 will have something like -ftrivial-auto-var-init enabled by default. See the "safe by default" section of [1].

For reference, the actual proposal that was accepted into C++26 is [2]. It discusses performance only in general, and it refers to an earlier analysis [3] for more details. This last reference describes regressions of around 0.5% in time and in code size. Earlier prototypes suggested larger regressions (perhaps even "horrendous") but more emphasis on compiler optimizations has brought the regression down considerably.

Of course one's mileage may vary, and one might also consider a 0.5% regression unacceptable. However, the C++ committee seems to have considered this to be an acceptable tradeoff to remove a frequent cause of undefined behavior from C++.

[1]: https://herbsutter.com/2024/08/07/reader-qa-what-does-it-mea...

[2]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2024/p27...

[3]: https://open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2723r1...

Microsoft's Visual C++ compiler has the /Ge compiler option ( see https://learn.microsoft.com/en-us/cpp/build/reference/ge-ena... ) Deprecated since VC2005.

This compiler option causes the compiler to emit a call to a stack probe function to ensure that a sufficient amount of stack space is available.

Rather than just probe once for each stack page used, you can substitute a function that *FILLS* the stack frame with a particular value - something like 0xBAADF00D - one could set the value to anything you wanted at runtime.

This would get you similar behaviour to gcc/clang's -ftrivial-auto-var-init

Windows has started to auto-initialize most stack variables in the Windows kernel and several other areas.

    The following types are automatically initialized:
    
        Scalars (arrays, pointers, floats)
        Arrays of pointers
        Structures (plain-old-data structures)
    
    The following are not automatically initialized:
    
        Volatile variables
        Arrays of anything other than pointers (i.e. array of int, array of structures, etc.)
        Classes that are not plain-old-data


    During initial testing where we forcibly initialized all types of data on the stack we saw performance regressions of over 10% in several key scenarios.

    With POD structures only, performance was more reasonable. Compiler optimizations to eliminate redundant stores (both inside basic blocks and between basic blocks) were able to further drop the regression caused by POD structures from observable to noise-level for most tests.

    We plan on revisiting zero initializing all types (especially now that our optimizer has more powerful optimizations), we just haven’t gotten to it yet.

I had to Google to find the tid bit that I read about Perl years ago. I think this will affect iteration order of dicts.

    > Nov 22, 2012 — Perl 5.18 will introduce per process hash randomization and almost certainly will feature a new hash function.

it probably shouldn’t be a “release” thing. actually, certainly. i do wonder how many bugs would never have seen the light of day, if someone’s “set” actually turned out to be a sequence (i.e. allowed duplicate values) resulting in a debug build raising an assert.

But the linux equivalent here would be glibc, not the kernel

Semi-related: this type of thing is actually covered in the Site Reliability Engineering book by Google. They highlighted a case of a system that outperformed its SLO, so people depended on it having 100% uptime. They "fixed" this by injecting errors to go closer to their SLA, forcing downstream engineers to deal with the fact that the dependent services would sometimes fail for no reason.

I know it's easier said than done everywhere, just found it to be an interesting parallel.

> If you promise randomization

You don't. You say the order is undefined.

You can pay for those features in debug mode or in chaos monkey mode. It's okay to continue to not pay for them in release mode. Heck, Rust has this approach when it comes to handling integer overflow - fully checked in debug mode, silent wraparound in release mode.

However, the compiler does not tell you this. We're back to the problem that it's possible to have a "working" C program that relies on UB and will therefore break at some point, but the tools will not yell at you for doing this. Whereas in Java or C# you get warnings or errors for using maybe-uninitialized variables.

Also, scanf should be deprecated. Terrible API. Never use scanf or sscanf etc. We managed to get "gets()" deprecated, time to spread that to other parts of the API.

atoi() or atof() etc. work OK, but really you need a parser.

Not necessarily; you can do a thing where it's randomized during development, testing and fuzzing but not in production builds or benchmarks so that the obvious "I rely on internal map order" bugs are spotted right away.

You can get it pretty much for free by using a random salt with your hash function. This is also useful for avoiding DOS attacks using deliberate hash collisions to trigger quadratic behavior in your hash tables.

Any sane language would design a list iterator to follow the order of the list. No, the difference is when you're iterating over orderless hash-based sets or maps/dictionaries. Many languages choose to leave the iteration order undefined. I think Python did that up to a point, but afterward they defined dictionaries (but not sets) to be iterated over in the order that keys were added. Also, some languages intentionally randomize the order per program run, to avoid things like users intentionally stuffing hash tables with colliding keys.

When I was doing gamedev about 5 years ago, we were still debugging with optimisation on. You get a class of bugs just from running in lower frame rates that don't happen in release.

The compiler has no way of knowing that the memory would be undefined, not unless it somehow can verify the data file. The most I think it can do is flag the program for not checking the return value of scanf, but even that is unlikely to be true since the program probably was checking for end of file which is also in the return value. It was failing to check the number of matched parameters. This is the kind of error that is easy to miss given the semantics of scanf.

Yeah, the debugging here is great, but the actual cause is super mild.

If your whole career will be using higher order languages with very little data stored on stack (vs heap), why should those programmers care? It seems like normal progression of more abstraction in the tools that we use. Similarly, I have programmed a lot of C and C++ in my career and I never once need assembly language. (I am expecting someone to pop in the convo here and tell me about how I am a terrible C/C++ programmer because I don't know any assembly.)

We have been seeing this shift for a while, where "software engineers" graduate from 3 month bootcamps. Except now most likely they will not be earning 500k making crud apps.

What about the incredible front end Devs that only know JS/CSS/HTML? They can still be true craftspeople in their art, be it cross-browser/platform issues or performance tweaking.

Compare python devs of today to fortran devs of the 60s. Something like that distance. Maybe more. But the trend isnt new.

When I worked at Microsoft and I had downtime I would sometimes read the code for app compatibility shims out of pure curiosity.

Win9x video games that made bad assumptions about the stack were a theme I saw. One of the differences between win9x and NT based windows is that kernel32 (later kernelbase) is a now user mode wrapper atop ntdll, whereas in the olden days kernel32 would trap directly into the kernel. This means that kernel32 uses more user mode stack space in NT. A badly behaving app that stored data to the left of the stack pointer and called into kernel32 might see its data structures clobbered in NT and not in 9x. So there were compatibility hacks that temporarily moved the stack pointer for certain apps.

IIRC the whole parsing performance issue was because the original code was written for the SP campaign of GTA5 that only had a handful of objects to parse data for. That was barely a blip in terms of performance impact and AFAIK was written years before GTAOnline was made (where it became an issue - and even then only became an issue much after GTAOnline was first made).

Writing some simple code that works with the data you expect to have without bothering with optimizations is fine, if anything it is one of the actual cases of "premature optimization": even with profiling no real time is spent on that code, your data wont make it spend any time and you should avoid wild guesses since chances are you'll be wrong (even if in this case it could be a correct guess, it'd be like a broken clock guessing the time is always 13:37).

The actual issue with that code was that, after they reused it for GTAOnline and started becoming a performance issue after some time as they added more objects, nobody thought to try and see what is wrong.

They were not the only one to make that mistake e.g. rapidjson had to fix the same error, few people expect parsing one token out of sscanf to strlen the entire input (not only that but there are c++ APIs which call sscanf under the hood).

The second error of deduplicating values by linear scanning an array was way more egregious.

The article discusses how that turns into an infinite loop and causes a hang.

When you subtract a small float from a very large float, the value doesn't change. This is because the "steps" between float values increase with the size of the value (i.e. floats have coarser resolution for larger magnitudes)

To see this in action, try running the following in a JavaScript interpreter:

console.log(1_000_000_000_000_000_000 - 1);

If m_fBladeAngle is really large (>2.2e8 back of the envelope), the subtraction will have no effect, and that would be an infinite loop.

If the code runs every frame, it's probably always small and does just one iteration once in a while when it wraps over the value.

fmod takes in the order of 30+ cycles, probably more in year 2003 CPUs, vs 1 for cmp, 1 for sub, 1 for jmp.

I really can't remember if it was this one, I'll have to check if I saved the link somewhere at work.. but maybe it was Hyrum's law https://www.hyrumslaw.com/

Just bring back spacebar heating

https://xkcd.com/1172/

But it's this and many other features of C/C++ that make it faster than Java. C/C++ developers really don't want to "pay" for something for safety.

Though, I really like the _mm_undefined_ps() intrinsics for SSE that make it clear that you're purposefully not initialising a variable. Something like that for ints and floats would be pretty sweet.

We don't have enough information to claim it's the "right decision" only that this choice did work, not that other choices couldn't have been better.

In video games you can go back and try another option but life isn't like that and so we can only suppose what might have happened.

There'd be a lot less Rust evangelism on this site if there were less UB bug outcomes on this site.

I would have picked Elixir, but it's unsuitable for game dev (at least thus far)

I wouldn't consider it "Rust evangelism" as much as "not C/C++/any language that makes it trivially easy to write undefined-behavior bugs, evangelism".

I'd be just as much a fan of Roc, but they're not yet mature and actually in the middle of a compiler rewrite (as it so happens, from Rust to Zig, lol) https://www.roc-lang.org/

While they did mention rust, the actual suggestion was "functional programming and immutability", which to me suggests several other languages first and makes it not really rust evangelism.

If your attitude is just "I'm not going to use abc because too many people say it's good", without even just trying that out first hand to verify those claims, I don't think you can go very far in your technical skills.

The best engineers I know are open to everything and played with almost every tool/language/whatever to form (sorry) informed opinions about them. They often know what they are talking about, and they choose the best tool for the job.

I don't think mentioning Rust on an article specifically talking about a memory safety bug count as "constant". This is Rust's core strength.

The code would have failed because you can't use an uninitialized variable, so you would have had to set it to a default. You don't just get random garbage from the stack.

It would have forced you to either specify a default or fail pretty loudly as soon as you launched the game, both much better than leaving a bug there just for it to resurface 20 years later.

Most popular languages would prevent this. In this case it’s as simple as having more sensible reader API than sscanf in standard library and forcing variables to be initialized.

Of course not, but this here was a memory access error and rust would have prevented this.

You didn't actually understand what the post is about. Maybe read it again.

It's more that functional languages just happen to be stricter in various ways that would've mitigated against this. You could quite happily design a functional language that has an unsafe equivalent to sscanf in its stdlib, or has big parts of the spec which are "undefined behaviour" that may differ depending on the underlying OS/compiler/runtime/stdlib in use. But the more popular functional languages have gained traction in part because they tend to have a "if you model the types correctly, the program basically works" philosophy around them. I don't think things like Haskell, Ocaml or F# would allow this if you wrote idiomatic code, you'd probably need to do something a little hacky or sketchy.

It simply would not have allowed you to write code which did that. And you wouldn't have a function like sscanf() either. You'd probably end up with a much more normal looking parser function that returned a value-or-error type.

I've never heard of a functional language that would allow you to initialize a value to whatever value the system memory already had in that memory location. In languages that allow nil, it would at least be nil; in languages that don't, you would have gotten an error about an uninitialized and undefaulted value. In any typed language, you would have also gotten an error.

It's true that C may be unique-ish in this regard though- this bug also couldn't happen in Ruby, which is not a functional language, but Ruby certainly still makes undefined behaviors much more possible than in other languages like Elixir.

The most successful videogame of all time was written for Java as an applet in the browser.

Unity+C# is now a pretty common combo.

My only issue with defensive codding is that often it doesn't play nice with code coverage requirements. I've been in situations where I would like to add defensive coding just in case, but then the PR doesn't pass the coverage checks. The best is when you can ensure via th compiler (e.g. via the type system) that a case is impossible, but C++ (in my case) isn't perfect for this.