Escaping Surprise Bills and Over-Engineered Messes: Why I Left AWS
36 comments
·February 4, 2025bschwindHN
ZYbCRq22HbJ2y7
Seems like a pointless thing to talk about, "most web projects today". You don't know what specific requirements people have, and people have been making "web projects" with limited resources since the web started.
Yet, for some reason, I see this repeated everywhere, always. Does it make the people who repeat it feel better? Are they actually informing anyone of anything? Who knows.
addicted
I don’t see a refutation or even disagreement with the claim in your comment. Are you saying that this is too obvious to restate?
ziddoap
The implied refutation of "You don't know what specific requirements people have" is that there will be people with specific requirements that a Pi/Mini PC obviously doesn't meet.
YetAnotherNick
Not only that, people always assume that you need complicated dependencies if they use AWS and compare any of the other company's stack with their raspberry pi. They can use AWS lightsail just fine for $5 if they just need a VM.
valenterry
It depends. If you want a service that doesn't have downtime because of a hardware issue, then you should have at least 2 machines. And then, you probably want to do deployments in a way where you spin up 2 new machines and then, only once confirmed that the deployment works, shut down the old 2.
That is super easy to do on AWS (and many other providers) but not so easy (or even impossible) anymore when you do things on-prem.
snackbroken
Most web projects can tolerate an hour of scheduled downtime a week and the occasional (< 1/year) unscheduled hour of downtime to replace broken hardware. Running on a single machine with 2 disks in RAID1 right up until the point where the ~50 hours of downtime a year equal one engineer's salary in lost revenue makes sense if you pick a DC that will sell/rent you replacement hardware and do replacements for you. Just make sure you have offsite backups in case the DC burns down. If your office/home is near a well stocked computer store, most web projects can skip the DC altogether and just put the server in the closet under the stairs or whatever. The company will likely die before the server does anyways.
esquire_900
Super easy? Getting that setup for the first time from zero knowledge will probably take a few days. And that's before understanding all the intricacies like your AWS bill, the hidden costs of cloud, properly securing IAM and setting up your VPC etc etc
wontondisregard
Not once in my entire career have I seen people successfully avail themselves of the cloud's purported benefits. I have, however, seen a lot of happy account managers in Las Vegas during Re:Invent, oh yes.
freeone3000
It is exactly as easy, or easier, to do this in proxmox than it is in AWS console, and would have a price breakeven point of less than a year (you only need two physical nodes).
2030ai
Digital Ocean I believe is pay as you go and supports this for anyone who can click "link my Github account" and auth themselves.
wontondisregard
I'm sorry, but the ten billionth CRUD web app really doesn't need nine nines.
2030ai
Just a blue green! But I agree somewhat. Single machine running Docker compose is pretty decent. Pets! Not cattle. (I'd rather care for a dog than a 100 sheep)
2030ai
If you have enough users that you need to scale to more machines ... run the rpis in a 3d printed 1u rack adaptor as a kubernetes cluster.
faizshah
Personally I use lightsail on AWS and cloudflare cause there is always an off ramp to try some of the fancy stuff but then you can always go back to just using cheap VMs behind cloudflare. You can also put it all behind a VPC and you can use CDK/CloudFormation so that’s also nice.
I gave up on using GCP even though the products like BigQuery are way better just because I got burned too many times like with the Google Domains -> Squarespace transition.
I’m thinking of switching back to a bare metal provider now like Vultr or DO (would love to know what people are using these days I haven’t used bare metal providers since ~2012).
Also, completely unrelated does anyone know what the best scraping proxy is these days for side projects (data journalism, archiving etc.)?
choilive
Been using OVH for bare metal for a few years now and no major hiccups other than scheduled maintenance.
wahnfrieden
OVH VPS in Canada is also a great deal
smatija
Hetzner is good to me, but I am EU based.
Loving that German logic that even emergency maintenace has 2-week notification.
deathanatos
I've used NearlyFreeSpeech for years (as registrar & DNS), and I've loved their service. Their site is plain, and you just trade money for a plain, simple product, with basically 0 bullshit between you and that exchange. Their site is so refreshing in today's landscape of upsells and other corporate dark patterns.
The article implicates AWS, but AFAICT the other major cloud, GCP, behaves similarly. The docs for "budget alerts"[1] do call it out directly,
> Setting a budget does not automatically cap Google Cloud or Google Maps Platform usage or spending. Budgets trigger alerts to inform you of how your usage costs are trending over time. Budget alert emails might prompt you to take action to control your costs, but they don't automatically prevent the use or billing of your services when the budget amount or threshold rules are met or exceeded.
But still. But wait, you say, those docs go on to suggest,
> One option to automatically control spending is to use budget notifications to programmatically disable Cloud Billing on a project.
And the linked page states,
> Following the steps in this capping example doesn't guarantee that you won't spend more than your budget.
sigh "Over-Engineered Messes", TFA hits it on the nose.
There's also limiting API usage, but that's on requests … not on cost.
I avoid it all for personal stuff.
At work, we pipe all these cloud bills into a BigQuery account which then pipes into graphs in Grafana, which all tells us that engineers have no idea what the actual horsepower of 3 GHz * 32 cores is when they request a bajillion more cores.
It's probably also reasonably categorized as an "Over-Engineered Mess".
(We also import Azure's billing data, and boy do they make that obnoxious. CSV files dumping into a bucket, and they might go back & edit CSVs, or drop new ones, and if there is there a schema for those CSV files … I've yet to find it. Some columns you might think were guaranteed non-"" are not. Dates in American. Severely denormalized. Etc.)
rectang
Enshittification at cloud providers guarantees that in order for some department to hit the numbers ownership demands, an ever increasing number of customers must get screwed with overage bills.
wordofx
Enshittification of software that people create guarantees that they will get overage bills and blame cloud providers.
rectang
The cloud products could be created with hard stops, but the vendors choose not to do so. It's a constant, exhausting fight to engineer projects to avoid the risk of business-destroying overages when those projects don't need to scale infinitely — e.g. when it wouldn't be great if some misbehaving, malicious AI bot takes down the site but that's still better than the alternative of serving some insane number of requests and paying for the traffic.
Insanity
That is pretty cool. I tend to default to AWS, luckily not too expensive for my side projects (about $15/month) - nothing accessible to the public though so my cost is relatively predictable.
That said, I do wish you could hard shutdown at a certain budget limit.. but guess that is not in AWS’s best interested.
rectang
The worst case for a traffic spike on some hard-limited service is the service becoming unavailable. The worst case for a traffic spike on an unlimited service is financial calamity.
darthrupert
Would contacting one's credit card union to deny the billing work as a kind of a hard shutdown?
cmeacham98
No - most cloud providers are postpaid, you use their services and then get a bill at the end of the month.
Making your payment method invalid doesn't absolve you of this debt and will just result in being sent to collections and/or sued if the total is large enough.
wao0uuno
Reads like an ad.
placardloop
AWS isn’t and has never been economical for side projects or hobby tinkering, unless you specifically want to tinker with AWS.
I’m a big AWS fan. I’d recommend any company of decent size to use AWS. But seriously, if your project is just a personal blog or some rails app you tinker with on the weekend, just get a $5/mo instance on Digital Ocean or a raspberry pi.
CoughlinJ
Are we entirely discounting the free tier that's provided?
ufmace
I don't think "surprise bills" is a good reason to avoid AWS. Yes, you avoid them on other providers by doing simple bare-metal Linux boxes. But doing a bare-metal Linux box with no other services is also fixed-cost at AWS. You only get into tricky to determine in advance variable costs when you string together a bunch of AWS's extra services in ways that you don't understand well or don't/can't set limits on.
On the other hand, doing single bare-metal boxes being much more complex and usually more expensive is a good reason to skip AWS for simple projects. In addition to the profusion of instance types and billing/usage options.
I also think the lack of options on limiting max billing for flexible services is pretty reasonable actually. For most of them, there's no single obvious reasonable thing to do when the money hits the limit. Storage costs money too, but I don't think much of anyone really wants to have their data get deleted when the cost hits the limit, for example.
endgame
Aren't you still going to be on the hook for data transfer out?
wontondisregard
Don't worry, there are plenty of other reasons to avoid AWS, too.
x0x0
> I also think the lack of options on limiting max billing for flexible services is pretty reasonable actually.
It's really scummy. There's no reason not to allow me to set, eg, a 5-10x normal spend hard limit where they shut things off. For things like lambdas with accidental reflection (stop lambdaing), someone using your bandwidth to serve files (stop serving), a bug in a your Athena scripts that downloads a far broader date range than expected (stop all athena), etc.
sema4hacker
Ironically, NearlyFreeSpeech still didn't exactly provide the "simple monthly max spend option" you were looking for, because now you're maintaining multiple prepaid accounts. It's silly that online services won't provide a billable account with a hard max.
wontondisregard
The hard max is however much money is in your account. Not sure what else you want.
robocat
Your comment is misinformed.
You can simply configure a single account, or multiple accounts, as you wish. You link sites to accounts so you can control both spending and risk (commonly you don't want site-A slashdotting to affect site-B). With a single account you can limit your financial exposure across multiple sites.
Perhaps refrain from commenting about things you are ignorant about.
I've been a NearlyFreeSpeech customer for many years and its a great service that I can highly recommend.
Truth is, most web projects made today can run on a raspberry pi or mini PC and be just fine. If you have enough users that you need to scale to more machines, you'll be in a position to know what to do to handle it, or hire someone who does.