Escaping surprise bills and over-engineered messes: Why I left AWS

It's like, the complexity has to live somewhere.

I've seen successful serverless designs but the complexity gets pushed out of code and into service configuration / integration (it becomes arch spaghetti). These systems are difficult to properly test until deployed to cloud. Also, yeah, total vendor lock in. It works for some teams but is not my preference.

If you have a low traffic or % of server ultiziation such as B2B applications. "Full Container on Serverless" can be insanely cheap. Such as FastAPI, Django, Rails etc running all of these on Lambda when you've only got a few hits during the day and almost none at night, is very cost effective.

What it's also surprising is people getting excited and "certified" on AWS (and attending AWS conferences, lol), job postings requiring you to "know" AWS for a developer position. Why on earth do I have to know AWS to develop software? Isn't it that supposed to be covered by DevOps or sysadmins? If one word could define AWS that would be: overengineer. The industry definitely does not need all that machinery to build things, probably a fraction of what is offered and way way simpler.

Our team kinda thinks the same thing about serverless but despite that we have some things built with it. And the paradoxical thing is that this issues have just never materialized, the serverless stuff is overwhelmingly the most stable part of our application. It's kinda weird and we don't fully trust it but empirically serverless works as advertised.

My experience - system went down every time we have significant load. Reasons were various, but all triggered by load. Switched to ECS + Aurora - problem gone, bill has slightly increased.

As with many of these things, I've seen it time and time again; the initial setup can be simple, perhaps that's a good thing for an org that needs to get moving quickly, but very soon you start to see the limitations and the more advanced stuff just isn't possible and now you're stuck, locked in so have to rebuild from the ground up.

Sometimes you have to do the (over) engineering part yourself, so that someone else isn't making the decisions on your behalf.

Just a counterpoint. But my experience has been the opposite.

Taking a legacy application that runs on a single server and is struggling to scale with spaghetti of config files all over the box and security issues that have gone unpatched because no one wants to touch the box as it’s too dangerous to change anything, as everything is intertwined. No backups or unverified backups. No redundancy. No reproducible environments so things work on local but not on remote. Takes devs days to pin the issues.

Then break it down and replace with individual fully managed AWS services that can scale independently and I don’t need to worry about upgrades.

Yeah, the total bill for services is usually higher. But that’s because now there’s actually someone (AWS) maintaining each service, rather than completely neglecting it.

They key to all this is to use IaC such as AWS CDK to have code be the single source of truth. No ClickOps!

That’s still way too much. In my previous job I ran the site on $10 a month dual core cpu. It had decent traffic, live status updates, a live stream during 2 hours of the day, a successful e-commerce shop. The site was getting thousands of visits an hour and I was always surprised at how much bandwidth we would chew. It even survived a could have ddos attacks without issues. I also cached the hell out of it with memcached.

My past two previous projects have been on $5 vps with out issue. Some one really tried to bring one of them down with a ddos but gave up when it didn’t work.

A $5 VPS can take you very far if your project is well optimized.

At that lower end you get two of these machines, https://www.hetzner.com/sb/#price_from=100, plus another hundred bucks per month for fun and CDN and some DDOS protection if that's your thing and a backup solution. Pick a 256 GB RAM machine and one with 3-4 TB SSD.

It'll reliably cost exactly that and you'll be able to handle quite a lot of traffic. Probably enough that the same load would land you around 3k/month on AWS or GCP or whatever.

For 3k/month you can get hundreds of cores and 1-2 TB RAM on a couple of machines physically separated in a redundant data center setup. That's with markup from the data center selling it to you and the credit cost. After two-three years you own the hardware and can sell it if you want. A rig like this makes it rather easy to build HA-systems, have your devs experiment a bit with Ansible and keepalived and ProxySQL and they'll figure it out.

The beauty of Unix-like systems is that they're trivially discoverable and well documented. The clown platforms are not, and if your systems go down for whatever reason it's typically much harder and more labour intensive to get them back up again when they're clowned. This is especially the case if they get transfered to new management, e.g. due to bankruptcy proceedings. Paying a premium to get locked-in is a bad deal, unless you're the kind of organisation that is likely to also have mainframes in production.

There's a "sweet spot" between startup and enterprise where the cloud makes zero sense.

Startups will benefit from the cloud, as it allows them to easily scale, and maintain infrastructure that would otherwise be very costly.

Then at some point, around 50-100 employees (depending on business), it makes more sense to have a server park sitting at home, and you probably have the revenue to hire a couple of system administrators to handle stuff.

Then you have enterprise customers with 500+ employees, who have compliance and governance to follow, who maybe has huge spikes during a month (banks for one experience a lot more load during payday), which you will have to pay for in your local server farm, but you can dynamically scale in a cloud data center.

Not that you can just "throw everything in the cloud", it still requires planning for the best cost optimizing, but once you reach enterprise level you can probably afford a couple of cloud architects or similar to facilitate this.

I'd say about $10k.

For that cost, you can get multiple production regions (for failover), dev/alpha/beta environments, a reasonable amount of CDN traffic, a bunch of databases and some halo services (like X-Ray/CloudWatch).

You have to be joking, there are massive Number of household name companies spending orders of magnitude more, on cloud. Companies that can afford to dedicated finOps people

Seems like a pointless thing to talk about, "most web projects today". You don't know what specific requirements people have, and people have been making "web projects" with limited resources since the web started.

Yet, for some reason, I see this repeated everywhere, always. Does it make the people who repeat it feel better? Are they actually informing anyone of anything? Who knows.

It depends. If you want a service that doesn't have downtime because of a hardware issue, then you should have at least 2 machines. And then, you probably want to do deployments in a way where you spin up 2 new machines and then, only once confirmed that the deployment works, shut down the old 2.

That is super easy to do on AWS (and many other providers) but not so easy (or even impossible) anymore when you do things on-prem.

EDIT: due to the replies, let me clarify; this is not an argument for AWS specifically but for services that make it easy to spin up machines and that handle deployments and failover for you.

Also, there are more reasons to have more than one machine, e.g. being able to deploy with zero downtime and therefore without having to think when to deploy and what to do if a deployment fails etc.

I just went down the rabbit hole of trying to host a small app for free(ish) on cloud solutions, only to remember that I have a 2GB fiber line and a raspberry pi sitting idle - if it ever gets so popular that I outgrow this, it'll be a nice problem to have.

You're absolutely right. A little off-topic, but I've seen a similar problem on the on-prem side of things. Their server setups are a lot of the time completely overblown "because this is what the nice vendor recommended, they are great partners!"

Awesome, your 5 node Nutanix setup with 15 whole VMs is peaking at 5% of its total compute, 5% of its RAM, and 8% of its disk. Granted, I do work with medium sized businesses in the manufacturing sector, but so many get tricked into thinking they need 5 9s, when 3 is enough, and their setup isn't even providing 5 anyway because nothing else is downstream. Then 5 years later their warranty is up and they scrap it all and replace. It's crazy.

Kills me how many businesses contract this shit out and get tricked into spending a million on infra when they could just run 2 servers and pay a mid level engineer to manage it. "Not in the budget." Yeah, I wonder why.

Isn't that a security risk? Having to open a port for the pi and then always keeping an eye on it in case the next heartbleed, etc. is discovered?

Yeah but OTOH I’ve just set up a vm host + guest on a N100 minipc for home assistant (migrating from a bare metal pi 4) and while I’m happy with the result, I’ve spent more hours than I’d like or planned to get it working reliably vs click ‘make a vm please’ and have it waiting for an ssh connection after a coffee break.

(Started with proxmox but the installer’s kernel doesn’t have the Ethernet driver and they don’t provide WiFi because it’s stupid for the installed system - but they allow upgrading to a kernel which supports my adapter, so got a bit catch 22’d. Moved to Ubuntu server that would stop booting after creating a bridge interface because ‘waiting for network. no timeout’ and then a few more interesting issues.)

If you have enough users that you need to scale to more machines ... run the rpis in a 3d printed 1u rack adaptor as a kubernetes cluster.

Hetzner is good to me, but I am EU based.

Loving that German logic that even emergency maintenace has 2-week notification.

Been using OVH for bare metal for a few years now and no major hiccups other than scheduled maintenance.

Enshittification at cloud providers guarantees that in order for some department to hit the numbers ownership demands, an ever increasing number of customers must get screwed with overage bills.

It's a change. It might not be much to you. But there's lots of people who $6 extra will make their day

In AWS you could have quite complex things as stuff like Lambda and EC2 have free tiers.

But it's also not really relevant to the main point of the article which is the risk of large surprise bills from eg runaway feedback loops and DoS attacks.

Funny. I'm planning to move from Azure to AWS because Azure is planning to raise my ~$10 bill to ~$50 later this year for no good reason.

The worst case for a traffic spike on some hard-limited service is the service becoming unavailable. The worst case for a traffic spike on an unlimited service is financial calamity.

It is possible, since spending has an API. I’ve recently implemented it on Azure for a company, to ensure our storage costs didn’t run wild.

Couldn’t find an out-of-the-box solution, maybe, as you say because it’s not in their interest.

Would contacting one's credit card union to deny the billing work as a kind of a hard shutdown?

A perfectly workable N100 box runs you like $150 (that's on the expensive side I'd say). If you can find a cheap Windows key or maybe get a box with an OEM license it'll be an even better deal.

> I never want to use windows again.

Can't blame you :) I'm nearing the point of never wanting to use any computer again.