Understanding Round Robin DNS
42 comments
·October 26, 2024jgrahamc
delusional
> I'll let HN know when I get an authoritative answer
Please remember to include a TTL so I know how long I can cache that answer.
jgrahamc
Thank you for appreciating my lame joke.
metadat
> This allows you to share the load between multiple servers, as well as to automatically detect which servers are offline and choose the online ones.
To [hesitantly] clarify a pedantry regarding "DNS automatic offline detection":
Out of the box, RR-DNS is only good for load balancing.
Nothing automatic happens on the availability state detection front unless you build smarts into the client. TFA introduction does sort of mention this, but it took me several re-reads of the intro to get their meaning (which to be fair could be a PEBKAC). Then I read the rest of TFA, which is all about the smarts.
If the 1/N server record selected by your browser ends up being unavailable, no automatic recovery / retry occurs at the protocol level.
p.s. "Related fun": Don't forget about Java's DNS TTL [1] and `.equals()' [2] behaviors.
[1] https://stackoverflow.com/questions/1256556/how-to-make-java...
[2] https://news.ycombinator.com/item?id=21765788 (5y ago, 168 comments)
encoderer
We accomplish this on Route53 by having it pull servers out of the dns response if they are not healthy, and serving all responses with a very low ttl. A few clients out there ignore ttl but it’s pretty rare.
ChocolateGod
I once achieved something similar with PowerDNS, which you can use LUA rules to do health checks on a pool of servers and only return health servers as part of the DNS record, but found odd occurrences of clients not respecting the TTL on DNS records and caching too long.
tetha
You usually do this with servers that should be rock-solid and stateless. HAProxy, Traefik, F5. That way, you can pull the DNS record for maintenance 24 - 48 hours in advance. If something overrides DNS TTLs that much, there is probably some reason.
teddyh
One of the early proposed solutions for this was the SRV DNS record, which was similar to the MX record, but for every service, not just e-mail. With MX and SRV records, you can specify a list of servers with associated priority for clients to try. SRV also had an extra “weight” parameter to facilitate load balancing. However, SRV did not want the political fight of effectively hijacking every standard protocol to force all clients of every protocol to also check SRV records, so they specified that SRV should only be used by a client if the standard for that protocol explicitly specifies the use of SRV records. This technically prohibited HTTP clients from using SRV. Also, when the HTTP/2 (and later) HTTP standards were being written, bogus arguments from Google (and others) prevented the new HTTP protocols from specifying SRV. SRV seems to be effectively dead for new development, only used by some older standards.
The new solution for load balancing seems to be the new HTTPS and SVCB DNS records. As I understand it, they are standardized by people wanting to add extra parameters to the DNS in order to to jump-start the TLS1.3 handshake, thereby making fewer roundtrips. (The SVCB record type is the same as HTTPS, but generalized like SRV.) The HTTPS and SVCB DNS record types both have the priority parameter from the SRV and MX record types, but HTTPS/SVCB lack the weight parameter from SRV. The standards have been published, and support seem to have been done in some browsers, but not all have enabled it. We will see what browsers will actually do in the near future.
jsheard
> The new solution for load balancing seems to be the new HTTPS and SVCB DNS records. As I understand it, they are standardized by people wanting to add extra parameters to the DNS in order to to jump-start the TLS1.3 handshake, thereby making fewer roundtrips.
The other big advantage of the HTTPS record is that it allows for proper CNAME-like delegation at the domain apex, rather than requiring CNAME flattening hacks that can cause routing issues on CDNs which use GeoDNS in addition to or instead of anycast. If you've ever seen a platform recommend using a www subdomain instead of an apex domain, that's why, and it's part of why Akamai pushed for HTTPS records to be standardized since they use GeoDNS.
teddyh
Oh yes¹. This is an advantage shared by all of MX, SRV and HTTPS/SVCB, though.
hypeatei
The browser behavior is really nice, good to know that it falls back quickly and smoothly. Round robin DNS has always been referred to as a "poor mans load balancer" which it seems to be living up to.
> Curl also works correctly. First time it might not, but if you run the command twice, it always corrects to the nearest server.
This took two tries for me, which begs the question how curl is keeping track of RTT (round trip times), interesting.
latchkey
> "It's an amazingly simple and elegant solution that avoids using Load Balancers."
https://www.cloudflare.com/learning/dns/glossary/round-robin...
toast0
Skipping an unnecessary intermediary is worth considering.
Load balancing isn't without cost, and load balancers subtly (or unsubtly) messing up connections is an issue. I've also used providers where their load balancers had worse availability than our hosts.
If you control the clients, it's reasonable to call the platform dns api to get a list of ips and shuffle and iterate through in an appropriate way. Even better if you have a few stablely allocated IPs you can distribute in client binaries for when DNS is broken; but DNS is often not broken and it's nice to use for operational changes without having to push new configuration/binaries everytime you update the cluster.
If your clients are browsers, default behavior is ok; they usually use IPs in order, which can be problematic [1], but otherwise, they have good retry behavior: on connection refused they try another IP right away, in case of timeout, they try at least a few different IPs. It's not ideal, and I'd use a load balancer for browsers, at least to serve the initial page load if feasible, and maybe DNS RR and semi-smart client logic in JS for websockets/etc; but DNS RR is workable for a whole site too.
If your clients are not browsers and not controlled by you, best of luck?
I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds. And that clients behind those resolvers will have trouble when you update DNS, but if your loadbalancer is behind a DNS name, when it needs to change addresses, you'll deal with that then, and you won't have experience.
[1] one of the RFCs suggests that OS apis should sort responses by prefix match, which might make sense if IP prefixes were heirarchical as a proxy to get to a least network distance server. But in the real world, numerically adjacent /24s are often not network adjacent, but if your servers have widely disparate addresses, you may see traffic from some client ips gravitate towards numerically similar server ips.
ectospheno
> I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds.
I’ve run a min ttl of 3600 on my home network for over a year. No one has complained yet.
wongarsu
An clients tested in the article behaved correctly and chose one of the reachable servers instead.
Of course somebody will inevitably misconfigure their local DNS or use a bad client. Either you accept an outage for people with broken setups or you reassign the IP to a different server in the same DC.
latchkey
If you know all of your clients, then you don't even need DNS. But, you don't know all of your clients. Nor do you always know your upstream DNS provider.
Design for failure. Don't fabricate failure.
zamadatix
Why would knowing your clients change whether or not you want to use DNS? Even when you control all of the clients you'll almost always want to keep using DNS.
A large number of services successfully achieve their failure tolerances via these kinds of DNS methods. That doesn't mean all services would or that it's always the best answer, it just means it's a path you can consider when designing for the needs of a system.
tetha
> As you can see, all clients correctly detect it and choose an alternative server.
This is the nasty key point. The reliability is decided client-side.
For example, systemd-resolved at times enacted maximum technical correctness by always returning the lowest IP address. After all, DNS-RR is not well-defined, so always returning the lowest IPs is not wrong. It got changed after some riots, but as far as I know, Debian 11 is stuck with that behavior, or was for a long time.
Or, I deal with many applications with shitty or no retry behavior. They go "Oh no, I have one connection refused, gotta cancel everything, shutdown, never try again". So now 20% - 30% of all requests die in a fire.
It's an acceptable solution if you have nothing else. As the article notices, if you have quality HTTP clients with a few retries configured on them (like browsers), DNS-RR is fine to find an actual load balancer with health checks and everything, which can provide a 100% success rate.
But DNS-RR is no loadbalancer and loadbalancers are better.
nerdile
It's putting reliability in the hands of the client, or whatever random caching DNS resolver they're sitting behind.
It also puts failover in those same hands. If one of your regions goes down, do you want the traffic to spread evenly to your other regions? Or pile on to the next nearest neighbor? If you care what happens, then you want to retain control of your traffic management and not cede it to others.
latchkey
> It's an acceptable solution if you have nothing else.
I'd argue it isn't acceptable at all in this day and age and that there are other solutions one should pick today long before you get to the "nothing else" choice.
toast0
Anycast is nice, but it's not something you can do yourself well unless you have large scale. You need to have a large number of PoPs, and direct connectivity to many/most transit providers, or you'll get weird routing.
You also need to find yourself some IP ranges. And learn BGP and find providers where you can use it.
DNS round robin works as long as you can manage to find two boxes to run your stuff on, and it scales pretty high too. When I was at WhatsApp, we used DNS round robin until we moved into Facebook's hosting where it was infeasible due to servers not having public addresses. Yes, mostly not browsers, but not completely browserless.
latchkey
Back in 2013, that might have been the best solution for you. But there were still plenty of headlines... https://www.wamda.com/2013/11/whatsapp-goes-down
We're talking about today.
The reason why I said Anycast is cause the vast majority of people trying to solve the need for having multiple servers in multiple locations, will just use CF or any one of the various anycast based CDN providers available today.
zamalek
Take a look at SRV records instead - they are very intentionally designed for this, and behave vaguely similarly to MX. Creating a DNS server (or a CoreDNS/whatever module) that dynamically updates weights based on backend metrics has been a pending pet project of mine for some time now.
V__
This seems like a nice solution for zero-downtime updates. Clone the server, add a the specified ip, deny access to the main one, upgrade and turn the cloned server off.
urbandw311er
What a great article! It’s often easy to forget just how flexible and self-correcting the “official” network protocols are. Thanks to the author for putting in the legwork.
cybice
Cloudflare results with worker as a reverse proxy can be much better.
easylion
But won't it add an additional hop hence additional latency to every single request ?
rodcodes
Nah, because the Cloudflare Workers run at closest edge location and are real fast.
The real solution with Cloudflare is to use their Load Balancing (https://developers.cloudflare.com/load-balancing) which is a paid feature.
specto
Chrome and Firefox use the OS dns server by default, which in most OS' have caching as well.
easylion
did you try running a simple bash curl loop instead of manually printing. The data and statistics will be become exactly clear. Because i want to understand how to ensure my clients get the nearest edge data center
Hmm. I've asked the authoritative DNS team to explain what's happening here. I'll let HN know when I get an authoritative answer. It's been a few years since I looked at the code and a whole bunch of people keep changing it :-)
My suspicion is that this is to do with the fact that we want to keep affinity between the client IP and a backend server (which OP mentions in their blog). And the question is "do you break that affinity if the backend server goes down?" But I'll reply to my own comment when I know more.