How Copyover MUD Servers Worked

Doing exec() to a new process but keeping the file descriptors relies on none of the file descriptors having the CLOEXEC (close-on-exec) flag. This flag is, IIUC, considered best practice for all file descriptors now, and in Python, all file descriptors are now by default created “non-inheritable” (i.e. with the CLOEXEC flag). A working model would instead be to create one inheritable file descriptor, do the fork() (and exec the new process in the child), and then sendmsg() all the normal file descriptors to the new process.

This is a fun blast from the past. One of my first real programming experiences was working as a (the) programmer on a MUD and I remember learning about this technique somewhere (I think just reading about how forking worked) and then figuring out how to implement it on our MUD.

It felt absolutely magical to be able to hot deploy changes without kicking everyone off of the server - but felt even more magical to have gotten it to work.

The MUD community was a fun early introduction to open source. People (many of them probably "kids" like I was at the time) sharing various patches and features. It felt so cool to release something and have other people use it and provide feedback. Like the author says - at some point the MUD itself became a lot less interesting than the programming.

The way my mud did it, and I believe most muds (considering I took my implementation from another popular codebase), was not by forking. Sockets are just files, so all we did was save everything including players, execl() another instance of the MUD, and exit. Same port wasn't an issue with SO_REUSEADDR.

On boot as it's loading players, their file descriptor ID was saved, so it loads it up and resumes talking to the socket. It also loads the ID of the server socket descriptor. No need to send information to a new process as it just loads the previous game state.

If you're curious: https://github.com/borlak/acmud/blob/7c2442dfccfc28364fc399a...

And: https://github.com/borlak/acmud/blob/7c2442dfccfc28364fc399a...

I actually run a MUD that stills uses this copyover method (primarily C code) for major code changes and migrations. We have about 100 players online at any given time! We've actually built in some more modern automatic copyover triggers and hooks related to recovering from crashes, capturing backtraces, performing database migrations, and sending/receiving systemd signals (among a lot of other modernization upgrades).

This is a pretty interesting approach to replacing your own executable, though it's really akin to a spicy spawn where you (the parent) exec rather than they (the child) exec. Not to minimize the coolness of this approach -- it's certainly a good way to do it, and doubly so on older Unix.

These days you could probably spawn your child process, test-boot it and then mmap bits of it back into your process and then unmap your old code pages once you're certain it boots. Or given how cheap cycles are, test-boot it and then throw that away and spawn the new executable if you're happy with the results.

Irssi has a command that does what I assume is something similar to this to upgrade in place without disconnecting https://irssi.org/documentation/help/upgrade/

No need for past tense! The most active MUD today (AFAIK, 250 logged in at most times) uses a version of this to automatically reboot every 2 days.

I played on a fantastic MUD as a kid. It was based on the code for some other MUD, and the admins were unhappy about some things, so they decided to write their own MUD engine from scratch. They got as far as establishing a network connection to a client - all network code written in scratch from C - before they burned themselves out and quit, which pretty much killed the MUD too.

this is why languages such as LPC were developed. LPMuds allowed developers to code rooms and objects while the game is running and simply reload them whenever they wanted without admin intervention. if loading failed then the old version was kept and you'd just fix your code and try again. it was incredibly robust and powerful. for a player to get access to new versions of objects or rooms they had to drop the objects and pick them up again or reenter the room so that the object references could be updated to the new versions.

even better, LPC has been rewritten into pike, a general purpose programming language that retains the same capability. in the roxen webserver i can write and reload modules at runtime. this works efficiently because the lifetime of any object instance is limited to each http request. when a module is reloaded http requests already in progress are not affected, only new ones.

in the object storage server open-sTeam also written in pike a more advanced method was developed using proxy objects that can update object references and thus allow the updating of code without breaking the references. i am still using that to host my own websites.

to this day i have yet to find any other language with this power. smalltalk can do it and i believe lisp too, but that's it.

I looked up my favorite mud awhile back. Just this post of other people looking for it for over 20 years!

https://groups.google.com/g/alt.mud/c/Hn5EOqDfcyg

Maybe a dumb question here. Why is a child process needed? Can the parent process not open a pipe, write its own state into the pipe, and then call exec, allowing the new binary to read from the pipe? Will exec destroy the pipe if there is no child process?