Writing a Self-Mutating x86_64 C Program (2013)
16 comments
·May 25, 2025oxcabe
It's impressive how well laid out the content in this article is. The spacing, tables, and code segments all look pristine to me, which is especially helpful given how dense and technical the content is.
alcover
I often think this could maybe allow fantastic runtime optimisations. I realise this would be hardly debuggable but still..
vbezhenar
I used GNU lightning library once for such optimisation. I think it was ICFPC 2006 task. I had to write an interpreter for virtual machine. Naive approach worked but was slow, so I decided to speed it up a bit using JIT. It wasn't a 100% JIT, I think I just implemented it for loops but it was enough to tremendously speed it up.
userbinator
Programs from the 80s-90s are likely to have such tricks. I have done something similar to "hardcode" semi-constants like frame sizes and quantisers in critical loops related to audio and video decompression, and the performance gain is indeed measurable.
alcover
> "hardcode" semi-constants
You mean you somehow avoided a load. But what if the constant was already placed in a register ? Also how could you pinpoint the reference to your constant in the machine code ? I'm quite profane about all this.
ronsor
> Also how could you pinpoint the reference to your constant in the machine code?
Not OP, but often one uses an easily identifiable dummy pattern like 0xC0DECA57 or 0xDEADBEEF which can be substituted without also messing up the machine code.
belter
I guess in OpenBSD because of W ^ X this would not work?
mananaysiempre
Not as is, but I think OpenBSD permits you to map the same memory twice, once as W and once as X (which would be a reasonable hoop to jump through for JITs etc., except there’s no portable way to do it). ARM64 MacOS doesn’t even permit that, and you need to use OS-specific incantations[1] that essentially prohibit two JITs coexisting in the same process.
[1] https://developer.apple.com/documentation/apple-silicon/port...
rkeene2
In Linux it also needs mprotect() to change the permissions on the page so it can write it. The OpenBSD man page[0] indicate that it supports this as well, though notes that not all implementations are guaranteed to allow it, but my guess is it would generally work.
Retr0id
It's not required on linux, if the ELF headers are set up such that the page is mapped rwx to begin with. (but rwx mappings are generally frowned upon from a security perspective)
akdas
I was thinking the same thing. Usually, you'd want to write the new code to a page that you mark as read and write, then switch that page to read and execute. This becomes tricky if the code that's doing the modifying is in the same page as the code being modified.
timewizard
The way it's coded it wouldn't; however, you can map the same shared memory twice. Once with R|W and a second time with R|X. Then you can write into one region and execute out of it's mirrored mapping.
I had a great experience writing self modified programs is a single instruction programming game SIC-1: https://store.steampowered.com/app/2124440/SIC1/