The $1k AWS Mistake
34 comments
·November 19, 2025dabiged
rikafurude21
Thats a year salary but hey think about how much more complicated your work would be if you had to learn to self-host your infra!
andrewstuart
You could have been running a 24 core $60/month unlimited bandwidth server on IONOS.
krystalgamer
Ah, the good old VPC NAT Gateway.
I was lucky to have experienced all of the same mistakes for free (ex-Amazon employee). My manager just got an email saying the costs had gone through the roof and asked me to look into it.
Feel bad for anyone that actually needs to cough up money for these dark patterns.
mgaunard
Personally I don't even understand why NAT gateways are so prevalent. What you want most of the time is just an Internet gateway.
CjHuber
Does Amazon refund you for mistakes, or do you have to land on HN frontpage for that to happen?
Dunedan
Depends on various factors and of course the amount of money in question. I've had AWS approve a refund for a rather large sum a few years ago, but that took quite a bit of back and forth with them.
Crucial for the approval was that we had cost alerts already enabled before it happened and were able to show that this didn't help at all, because they triggered way too late. We also had to explain in detail what measures we implemented to ensure that such a situation doesn't happen again.
thecodemonkey
Hahaha. I'll update the post once I hear back from them. One could hope that they might consider an account credit.
Aeolun
I presume it depends on your ability to pay for your mistakes. A $20/month client is probably not going to pony up $1000, a $3000/month client will not care as much.
viraptor
They do sometimes if you ask. Probably depends on each case though.
merpkz
> AWS charges $0.09 per GB for data transfer out to the internet from most regions, which adds up fast when you're moving terabytes of data.
How does this actually work? So you upload your data to AWS S3 and then if you wish to get it back, you pay per GB of what you stored there?
0manrho
That is the business model and one of the figurative moats: easy to onboard, hard/expensive (relative to on-boarding ) to divest.
Though important to note in this specific case was a misconfiguration that is easy to make/not understand in the data was not intended to leave AWS services (and thus should be free) but due to using the NAT gateway, data did leave the AWS nest and was charged at a higher data rate per GB than if just pulling everything straight out of S3/EC2 by about an order of magnitude (generally speaking YMMV depending on region, requests, total size, if it's an expedited archival retrieval etc etc)
So this is an atypical case, doesn't usually cost $1000 to pull 20TB out of AWS. Still this is an easy mistake to make.
pavlov
Yes…?
Egress bandwidth costs money. Consumer cloud services bake it into a monthly price, and if you’re downloading too much, they throttle you. You can’t download unlimited terabytes from Google Drive. You’ll get a message that reads something like: “Quota exceeded, try again later.” — which also sucks if you happen to need your data from Drive.
AWS is not a consumer service so they make you think about the cost directly.
embedding-shape
"Premium bandwidth" which AWS/Amazon markets to less understanding developers is almost a scam. By now, software developers think data centers, ISPs and others part of the peering on the internet pay per GB transferred, because all the clouds charge them like that.
plantain
Try a single threaded download from Hetzner Finland versus eu-north-1 to a remote (i.e. Australia) destination and you'll see premium bandwidth is very real. Google Cloud Storage significantly more so than AWS.
Sure you can just ram more connections through the lossy links from budget providers or use obscure protocols, but there's a real difference.
Whether it's fairly priced, I suspect not.
blitzar
Made in California.
We are programmed to receive. You can check out any time you like, but you can never leave
hexbin010
Yes uploading into AWS is free/cheap. You pay per GB of data downloaded, which is not cheap.
You can see why, from a sales perspective: AWS' customers generally charge their customers for data they download - so they are extracting a % off that. And moreover, it makes migrating away from AWS quite expensive in a lot of circumstances.
speedgoose
Yes. It’s not very subtle.
null
ilogik
the statement is about aws in general, and yes, you pay for bandwith
viraptor
The service gateways are such a weird thing in AWS. There seems to be no reason not to use them and it's like they only exist as a trap for the unaware.
mgaunard
If you want to avoid any kind of traffic fees, simply don't allow routing outside of your VPC by default.
andrewstuart
Why are people still using AWS?
And then writing “I regret it” posts that end up on HN.
Why are people not getting the message to not use AWS?
There’s SO MANY other faster cheaper less complex more reliable options but people continue to use AWS. It makes no sense.
harel
You probably saved me a future grand++. Thanks
thecodemonkey
That was truly my hope with this post! Glad to hear that
belter
Talking how the Cloud is complicated, and writing a blog about what is one of the most basic scenarios discussed in every Architecture class from AWS or from 3rd parties...
cube00
A Gateway Endpoint is a special type of VPC endpoint that allows you to privately route traffic to S3 without going through your NAT Gateway or Internet Gateway. It's essentially a direct pipe from your VPC to S3.
A paragraph later.
The solution is to create a VPC Gateway Endpoint for S3. This is a special type of VPC endpoint that creates a direct route from your VPC to S3, bypassing the NAT Gateway entirely.
nrhrjrjrjtntbt
NAT gateway probably cheap as fuck for Bezos & co to run but nice little earner. The parking meter or exit ramp toll of cloud infra. Cheap beers in our bar but $1000 curb usage fee to pull up in your uber.
tecleandor
I think it's been calculated that data transfer is the biggest margin product in all AWS catalog by a huge difference. A 2021 calculation done by Cloudflare [0] estimated almost 8000% price markup in EU and US regions.
And I can see how, in very big accounts, small mistakes on your data source when you're doing data crunching, or wrong routing, can put thousands and thousands of dollars on your bill in less than an hour.
--
0: https://blog.cloudflare.com/aws-egregious-egress/null
ukoki
I don't think its about profits, its about incentivising using as many AWS products as possible. Consider it an 'anti-lock-in fee'
fragmede
Just $1,000? Thems rookie numbers, keep it up, you'll get there (my wallet won't, ow).
thecodemonkey
Haha, yep we were lucky to catch this early! It could easily have gotten lost with everything else in the monthly AWS bill.
bravetraveler
Came here to say the same, take my vote
- DevOops
I made the same mistake and blew $60k.
I have never understood why the S3 endpoint isn't deployed by default, except to catch people making this exact mistake.