How Copyover MUD Servers Worked
83 comments
·February 11, 2025ydant
jonmarkgo
MUDs were how I really learned to code (in C) and how I fell in love with programming and online communities, almost 25 years ago now.
ZeWaka
> The MUD community was a fun early introduction to open source.
They still are these days. There's plenty around, and there's even more MUD-likes (GUI) that are open-source and played/developed by hundreds & thousands.
dominicrose
I was using Ruby for hot reload, but then of course it was after the first MUDs. When everything is a function, you can just reload them. Doing it in Typescript (for fun) was also possible but quite challenging.
Erwin
That was an amusing post to see pop up here, as I believe I came up with the "copyover" name when I copied Melvin Smith (aka "Fusion") idea about "hot reboot" from his "MUD++" code base to the popular Diku-based MERC/Envy etc. bases -- that was 2000 or probably earlier. Whether Melvin originally got the idea from somewhere else I don't know.
That version just used exec, and closed all files but network descriptors already logged in, the mapping of fds -> login names was saved in a file. When the new copy started up, it would log the users on existing file descriptors. Today, using explicit file descriptor passing (so you don't accidentally keep files open) or a long-running proxy would be preferable.
Back then C/C++ were often used by the developers, and we were at best CS students. There were surprisingly few segmentation faults, but I remember a few mysterious memory corruptions...
em-bee
2000 or probably earlier
according to the patches in the source posted on your website the main work was done 1996-1997 ;-)
i am actually surprised that this didn't happen earlier, given that Diku itself was inspired by LPMuds which could do live code update already in the early 90s. of course the motivation for Diku was to produce something more stable than LPMud, so maybe they didn't think that live updates were a good idea in the first place. (that said, i don't remember LPMuds being unstable myself, but i only played from about 1992 at which time it may have improved)
_jackdk_
So the original versions were even simpler than I knew! I'd like to add this detail to the article. How would you like to be credited?
tdeck
This kind of reminds me of a story I heard about how they passed around raw function pointers between servers as an RPC mechanism at Yahoo back in the day. They'd disabled ASLR to make that possible. I guess there were few enough segmentation faults :).
jonmarkgo
99% sure I used your copyover snippet when we first added it to our MUD like 25 years ago
johnbellone
I definitely used his copy over snippet when implementing it.
Multicomp
Only on HN. Thanks for sharing!
jonmarkgo
I actually run a MUD that stills uses this copyover method (primarily C code) for major code changes and migrations. We have about 100 players online at any given time! We've actually built in some more modern automatic copyover triggers and hooks related to recovering from crashes, capturing backtraces, performing database migrations, and sending/receiving systemd signals (among a lot of other modernization upgrades).
_jackdk_
Very cool - I thought it would be a forgotten technique by now. Which MUD, and have you written up the details anywhere? I'm very interested to know what a modern version of this looks like, particularly with the systemd integration.
jonmarkgo
https://www.legendsofthejedi.com/
It originated (more than 20 years ago) as a fork of SWReality, which is itself a fork of SMAUG, which comes from DIKU, and so on and so on.
We (myself and a lot of other awesome volunteer coders over the years) have made some pretty major modifications to the codebase. It all runs on PostgreSQL now (instead of flat text files with a custom parser), it has a built-in Lua scripting interpreter for extensions and in-game interactions, it has deep integrations to our Discord community, it has sidecar web services, and much much more...
FajriFadli
This is awesome, thanks for sharing! I used to play lots of MUDs and also the browser-based MMORPG life sim Star Wars Combine, so I don't know how I missed this back then.
By the way, you have broken link on the `Learn More About the Economy` link in the home page. Should just have it redirect to `/crafting-the-galaxy`
_jackdk_
That's so cool. Have you written about the MUD's history and technical evolution anywhere? If not, would you be interested in collaborating on an article like that?
johnbellone
Ha, blast from the past - I have bunch of code that is in mainline SMAUG.
draculero
is still online? I would like to play to remember my good old days!
mmastrac
This is a pretty interesting approach to replacing your own executable, though it's really akin to a spicy spawn where you (the parent) exec rather than they (the child) exec. Not to minimize the coolness of this approach -- it's certainly a good way to do it, and doubly so on older Unix.
These days you could probably spawn your child process, test-boot it and then mmap bits of it back into your process and then unmap your old code pages once you're certain it boots. Or given how cheap cycles are, test-boot it and then throw that away and spawn the new executable if you're happy with the results.
em-bee
this is why languages such as LPC were developed. LPMuds allowed developers to code rooms and objects while the game is running and simply reload them whenever they wanted without admin intervention. if loading failed then the old version was kept and you'd just fix your code and try again. it was incredibly robust and powerful. for a player to get access to new versions of objects or rooms they had to drop the objects and pick them up again or reenter the room so that the object references could be updated to the new versions.
even better, LPC has been rewritten into pike, a general purpose programming language that retains the same capability. in the roxen webserver i can write and reload modules at runtime. this works efficiently because the lifetime of any object instance is limited to each http request. when a module is reloaded http requests already in progress are not affected, only new ones.
in the object storage server open-sTeam also written in pike a more advanced method was developed using proxy objects that can update object references and thus allow the updating of code without breaking the references. i am still using that to host my own websites.
to this day i have yet to find any other language with this power. smalltalk can do it and i believe lisp too, but that's it.
relistan
I ran and maintained a CircleMUD for some years but always felt that we should have run LPMud (or MudOS). Those were clearly the superior technology. And playing them was so cool because they _changed_ much more often because it was painless.
The last maintained LPMud fork that I know of is LDMud https://github.com/ldmud/ldmud . Would be cool if there are others.
The BEAM vm has a lot of the functionality you mention from LPC and Pike. Erlang being from the 80s—predating LPC—and also from Sweden, I imagine that Lars Pensjö and later Fredrik Hübinette et al may have taken some inspiration there.
cotillion
I still maintain the CD LPMud driver for Genesis MUD at https://github.com/cotillion/cd-gamedriver. There is not very much activity though since most critical issues have been fixed over the years and the game is very stable.
relistan
Awesome!
em-bee
we should have run LPMud (or MudOS). Those were clearly the superior technology.
which is quite ironic given that CircleMUD is based on DikuMUD which was created to be more stable than LPMUD.
Erlang is one of the languages i am still very curious about. hopefully some day i'll get an opportunity to work with it
_jackdk_
Thanks for bringing this up. I actually had a section on LPMuds in an earlier draft, but took it out because I didn't think I could do it justice. I also wanted to focus more on the Unix stuff. My friends and I tended to play DIKU derivatives, so I was never intimately familiar with LPMuds' capabilities.
I hear that class/skill variety was often greater on LPMuds because it was so much easier to code up custom behaviour. Do you know of any especially good articles on LPMuds class (in the OO sense)/object hierarchies and the software architecture of mudlibs?
lyu07282
I believe when regular users were able to create the world around them in a MUD they were called "wizards", this is like as if in a MMORPG some players gain the privilege to add mods to the official game world for all players to experience.
em-bee
that is correct. players who essentially completed the game were promoted to wizards with editing privileges. in the muds where i was active though eventually not all players were interested in that and new challenges were created where the players could continue, while the bar to become a wizard was not raised so that at one point players could choose whether they became wizards or remained players.
borlak
The way my mud did it, and I believe most muds (considering I took my implementation from another popular codebase), was not by forking. Sockets are just files, so all we did was save everything including players, execl() another instance of the MUD, and exit. Same port wasn't an issue with SO_REUSEADDR.
On boot as it's loading players, their file descriptor ID was saved, so it loads it up and resumes talking to the socket. It also loads the ID of the server socket descriptor. No need to send information to a new process as it just loads the previous game state.
If you're curious: https://github.com/borlak/acmud/blob/7c2442dfccfc28364fc399a...
And: https://github.com/borlak/acmud/blob/7c2442dfccfc28364fc399a...
hoseja
That sounds so much nicer than that horrific hack in the OP.
empathy_m
I spent a fair chunk of the mid-90s to early 2000s on a moderately busy MUD with a group of people who, it turns out, were all about the same age. No idea in retrospect how I juggled this with school/friends/work - I guess kids just have a ton of free time.
Kind of drifted away for a couple decades during college and after, as other things filed up the time.
I came back decades later, after going to a memorial service for a friend who died untimely of a serious medical condition, and seeing that a bunch of the people there were from her online community. They talked about the MUSH she hung out on had been a real lifeline when she was bedbound for immune-system reasons -- it was really, really cool to see, and I went back and checked out my own place and met up with folks again.
During 2020 a TON of people all had the same idea and all logged in to my place again. There was a brief resurgence of activity (from dozens of people online to a hundred+). Very few new players, but very cool to see people who were all, more or less, the same cohort -- just grown up now. Folks have slowly drifted away again in the past couple years, and that's fine too. I'm glad it's there.
It's nice to have these subcritical, human-scale online communities. Not everything has to be a subreddit or even a 10,000+ person discord - you can just hang out on a server!
AStonesThrow
The MUDLine: timeline of MUD history, by Lauren P. Burka (1995)
https://www.linnaean.org/~lpb/muddex/mudline.html
See also: Burka's MUDDEX (1993), curating listings of the erstwhile MUD servers as NSFnet ruled the USA
https://www.linnaean.org/~lpb/muddex/index.html
Burka was known to her fellow players as "ashne" - stylized in lowercase - on Tinymud Classic & Islandia, hosted by CMU prof Jim "Fuzzy" Aspnes, among other servers
https://en.wikipedia.org/wiki/James_Aspnes
several interesting hacks we collaborated on:
A TCP port concentrator, implemented by "leet". This add-on front end multiplexed connections with separate processes, allowing us to overcome the 64 file descriptor hard limit, per process, on Unix. Nearly 256 players could participate!
Na Choon Piaw, while working for Bell Labs Research on the East Coast, added a programming language that resembles Forth, releasing TinyMUCK 1.x and TritonMUCK test bed server. Piaw worked with a VAX server, 7800 I believe.
https://en.wikipedia.org/wiki/TinyMUCK?wprov=sfla1
also, Jon "Stinglai" Blow from Berkeley designed a string interning method for TinyMUCK 2.x that saved plenty of memory, by deduplicating ASCII strings in memory.
Me? I assumed many interesting names over the years. Primarily "ChupChup" or his plural counterparts, "chupchups", or also "Lucretia", the statuesque goth woman wearing "boots of the deepest black". But in Real Life, (RL), I remain Robert Earl.
jisnsm
Irssi has a command that does what I assume is something similar to this to upgrade in place without disconnecting https://irssi.org/documentation/help/upgrade/
skulk
No need for past tense! The most active MUD today (AFAIK, 250 logged in at most times) uses a version of this to automatically reboot every 2 days.
somenameforme
Wow. You made me check. I can't believe Medievia finally 'died'. I mean it's still alive, but on life support at the minimum.
cdr
I played on a fantastic MUD as a kid. It was based on the code for some other MUD, and the admins were unhappy about some things, so they decided to write their own MUD engine from scratch. They got as far as establishing a network connection to a client - all network code written in scratch from C - before they burned themselves out and quit, which pretty much killed the MUD too.
cheschire
There’s a lot to learn here. Many developers fall into this trap halfway through their careers. Spolsky would probably call these folks you’re talking about architect astronauts.
Nuzzerino
As someone who has (for several years) developed, supported, and operated my own dialect of a multiplayer game server engine (which includes an embedded physics engine), that sounds absolutely brutal. It took several years before I saw my first user and it was after I thought my project was long dead. Then it started catching on. You probably won’t see results if you expect more than that.
Hopefully they gained some appreciation for how much work goes into these.
fragmede
As the adage goes, you can make a game, or you can make a game engine. Choose one.
jeffrallen
It is also fun to engineer servers which can be upgraded without ever stopping listening, even for a nanosecond on the listen socket. Would be a fun senior C programming job interview question, but unfortunately those don't exist anymore. Only kubernetes pod wranglers do, where the answer is "destroy it all and create it all and hope that the ingress somehow hides the mess".
Sigh.
anacrolix
Amen. Lost art. All those years of shared libs and inetd, brilliant concepts, clobbered and forgotten by big daddy containers.
This is a fun blast from the past. One of my first real programming experiences was working as a (the) programmer on a MUD and I remember learning about this technique somewhere (I think just reading about how forking worked) and then figuring out how to implement it on our MUD.
It felt absolutely magical to be able to hot deploy changes without kicking everyone off of the server - but felt even more magical to have gotten it to work.
The MUD community was a fun early introduction to open source. People (many of them probably "kids" like I was at the time) sharing various patches and features. It felt so cool to release something and have other people use it and provide feedback. Like the author says - at some point the MUD itself became a lot less interesting than the programming.