The hidden complexity of scaling WebSockets

My SaaS has been using WebSockets for the last 9 years. I plan to stop using them and move to very simple HTTP-based polling.

I found that scalability isn't a problem (it rarely is these days). The real problem is crappy network equipment all over the world that will sometimes break websockets in strange and mysterious ways. I guess not all network equipment vendors test with long-lived HTTP websocket connections with plenty of data going over them.

At a certain scale, this results in support requests, and frustratingly, I can't do anything about the problems my customers encounter.

The other problems are smaller, but still annoying, for example it isn't easy to compress content transmitted through websockets.

This is all true, but it also serves to remind us that Rails gives developers so much out of the box, even if you're not aware of it.

ActionCable is Rails' WebSockets wrapper library, and it addresses basically every pain point in the post. However, it does so in a way that all Rails developers are using the same battle-tested solution. There's no need for every project to hack together its own proprietary approach.

Thundering herds, heartbeat monitoring are both covered.

If you need a messaging schema, I strongly recommend that you check out CableReady. It's a powerful library for triggering outcomes on the client. It ships with a large set of operations, but adding custom operations is trivial.

https://cableready.stimulusreflex.com/hello-world/

While both ActionCable and CableReady are Rails libraries, other frameworks would score huge wins if they adopted their client libraries.

The key to managing this complexity is to avoid mixing transport-level state with application-level state. The same approach for scaling HTTP requests also works for scaling WebSocket connections:

* Read, write and track all application-level state in a persistent data store.

* Identify sessions with a session token so that application-level sessions can span multiple WebSocket connections.

It's a lot easier to do this if your application-level protocol consists of a single discrete request and response (a la RPC). But you can also handle unidirectional/bidirectional streaming, as long as the stream states are tracked in your data store and on the client side.

Many years ago, we used to start a streaming session with an http request, then upgrading to websockets after obtaining a response (this was our original "StreamSense" mechanism). In recent years, we changed StreamSense to go websocket first and fallback to http streaming or http long polling in case of issues. At Lightstreamer, we started streaming data 25 years ago over http, then moving to websockets. We've seen so many different behaviors in the wild internet and got some much feedback from the fieldsl in these decades that we believe our current version of Lightstreamer includes heuristics and mechanisms for virtually every possible aspect of websockets that could go wrong. From massive disconnections and reconnections, to enterprise proxies with deep inspections, to mobile users continuously switching networks. I recall when a big customer required us to support one million live websocket connections for each server (mid-sized) keeping low latency. It was challenging but forced us to come up with a brand new internal architecture. So many stories to tell covering 25 years of evolution...

I am really unsure why devs around the world keep defaulting to websockets for things that are made for server sent events. In 90% of the usecases i see, websockets are just not the right fit. Everything is simpler and easier with SSE. Some exceptions are high throughput >BI<directional data streams. But even if eg. your synced multiplayer cursors in something like figma use websockets don't use it for everything else eg. your notification updates.

I wrote about the way we handle WebSocket connections at Canva a while ago [1]. Even though some small things have changed here and there since the post was published, the overall approach has held up pretty well handling many millions of concurrent connections.

That said, even with great framework-level support, it's much, much harder to build a streaming functionality compared to plain request/response if you've got some notion of a "session".

[1]: https://www.canva.dev/blog/engineering/enabling-real-time-co...

The only complexity I have found with regards to scaling WebSockets is knowing the minimum delay between flush event completion and actual message completion to destination. It takes longer to process a message, even on IPC routing, than it does to kill a socket. That has upstream consequences with consideration of redirection and message pipes between multiple sockets. If you kill a socket too early after a message is flushed from the socket there is a good chance the destination sees the socket collapse before it has processed the final message off the socket and that processing delay is not something a remote location is easily aware of.

I have found for safety you need to allow an arbitrary delay of 100ms before killing sockets to ensure message completion which is likely why the protocol imposes a round trip of control frame opcode 8 before closing the connection the right way.

I recall another complication with websockets: IIRC it's with proxy load balancers, like binding a connection to a single connection server, even if the backend connection is using HTTP/2. I probably have the details wrong. I'm sure someone will correct my statement.

> WebSocket connections can be unexpectedly blocked, especially on restrictive public networks.

What? How would public network even know you’re running a websocket if you’re using TLS? I dont think it’s really possible in general case

> Since SSE is HTTP-based, it's much less likely to be blocked, providing a reliable alternative in restricted environments.

And websockets are not http-based?

What article describes as challenges seems like very pedestrian things that any rpc-based backend needs to solve.

The real reason websockets are hard to scale is because they pin state to a particular backend replica so if the whole bunch of them disconnect at scale the system might run out of resources trying to re-load all that state

The comment about Render/Railway gracefully tranferring connections seems weird? I am pretty sure it just kills the service after the new one is alive which will kill the connections. Not some fancy zero downtime reconnect.

for me, the most important lesson i've learned when using websockets is to not use them whenever possible.

i don't hate them, they're great for what they are, but they're for realtime push of small messages only. trying to use them for the rest of your API as well just throws out all the great things about http - like caching and load balancing, and just normal request/response architecture. while you can use websockets for that it's only going to cause you headaches that are already solved by simply using a normal http api for the vast majority of your api.