Skip to content(if available)orjump to list(if available)

When Reverse Proxies Surprise You: Hard Lessons from Operating at Scale

whstl

It's nice to see someone else preaching this:

> Production Lesson: Never let exceptions dictate the norm. Handle them explicitly, in isolated paths or tiers, instead of polluting the mainline logic. What looks like "flexibility" is often just deferred fragility waiting to surface at scale.

I've seen this pattern far too often in production systems. In the name of "covering edge cases", a huge amount of complexity is moved over to configuration languages, interfaces, APIs, etc, to be more flexible. Not only this doesn't free up the developers time (because it overcomplicates it all), it also makes things worse on the other side for the users of such structures. We already have something "flexible": source code itself, no need to reinvent the wheel.

stacktrace

Very interesting read! But I want to point out a small correction - the DNS collapse issue at HAProxy, along with O(N^2), also had some O(N^3) code paths, which is just mind-blowing.

Also, I believe this should be the correct GitHub issue link - https://github.com/haproxy/haproxy/issues/1404

> Production Lesson: Code that "works fine" at small scale may still hide O(N²) or worse behavior. At hundreds or thousands of nodes, those costs stop being theoretical and start breaking production.