Migrating from AWS to Hetzner
584 comments
·October 17, 2025adamcharnock
torginus
Yup, I hope to god we are moving past the age of 'everything's fast if you have enough machines' and 'money is not real' era of software development.
I remember the point in my career when I moved from a cranky old .NET company, where we handled millions of users from a single cabinent's worth of beefy servers, to a cloud based shop where we used every cloud buzzword tech under the sun (but mainly everything was containerized node microservices).
I shudder thinking back to the eldritch horrors I saw on the cloud billing side, and the funny thing is, we were constantly fighting performance problems.
dematz
Tangential point but why is it that so often these leaving the cloud posts use the word "beefy" to describe the servers? It's always you don't need cloud because beefy servers handle pretty much any bla bla bla
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
If anyone from oxide computer or similar is reading, maybe you should rebrand to BEEFY server inc...
torginus
Because that is a casual word in the English language to describe an object with substantial power?
If you would suggest a word that would make a better substitute in this case, that could move the conversation forward, and perhaps you could improve the aesthetic quality of posts about leaving the cloud.
fsckboy
>Tangential point... rebrand to BEEFY server
idea for an ad campaign:
"mmm, beefy!" https://www.youtube.com/watch?v=j6ImwKMRq98&t=21s
i don't know how "worldwide" is the distribution of Chef "Boyardee" canned spaghetti (which today is not too good), but the founder's story is fairly interesting. He was a real Italian immigrant chef named Ettore Boiardi and he gets a lot of credit for originally evangelizing and popularizing "spaghetti" and "spaghetti sauce" in America when most people had never tried it.
https://en.wikipedia.org/wiki/Ettore_Boiardi
you know, "spaghetti code..."?
earthnail
Because the server types you get for the price of a single Heroku dyno are incredibly beefy. And suddenly you need a lot less dynos. Which is quite important if you start managing them yourself.
bcantrill
Noted!
gwking
The servers are always beefy and the software is always blazingly fast. Blazingly beefy is my new joke trademark.
bombcar
My conspiracy theory is that "cloud scaling" was entirely driven by people who grew up watching sites get slash dotted and thought it was the absolute most important thing in the world that you can quickly scale up to infinity billion requests/second.
colechristensen
No, cloud adoption was driven by teams having to wait 2 years for capex for their hardware purchase and then getting a quarter of what they asked for. You couldn't get things, people hoarded servers they pretended to be using because when they did need something they couldn't get it. Management just wouldn't approve budgets so you were stuck using too little hardware.
On the cloud it takes five seconds to get a new machine I can ssh into and I don't have to ask anyone for the budget.
You can save a lot of money with scaling, you have to actually do that though and very few places do.
neonsunset
However old, .NET Framework was still using a decent JIT compiler with a statically typed language, powerful GC and a fully multi-threaded environment :)
Of course Node could not compete, and the cost had to be paid for each thinly sliced microservice carrying heavy runtime alongside it.
rightbyte
What is old is new again.
My employer is so conservative and slow that they are forerunning this Local Cloud Edge Our Basement thing by just not doing anything.
radu_floricica
> What is old is new again.
Over the years I tried occasionally to look into cloud, but it never made sense. A lot of complexity and significantly higher cost, for very low performance and a promise of "scalability". You virtually never need scalability so fast that you don't have time to add another server - and at baremetal costs, you're usually about a year ahead of the curve anyways.
hibikir
A nimble enough company doesn't need it, but I've had 6 months of lead time to request one extra server in an in-house data center due to sheer organizational failure. The big selling point of the cloud really was that one didn't have to deal with the division lording over the data center, or have any and all access to even log in by their priesthood who knew less unix than the programmers.
I've been in multiple cloud migrations, and it was always solving political problems that were completely self inflicted. The decision was always reasonable if you looked just at the people the org having to decide between the internal process and the cloud bill. But I have little doubt that if there was any goal alignment between the people managing the servers and those using them, most of those migrations would not have happened.
odie5533
Complexity? I've never set up a highly available Postgres and Redis cluster on dedicated hardware, but I can not imagine it's easier than doing it in AWS which is only a few clicks and I don't have to worry about OS upgrades and patches. Or a highly available load balancer with infinite scale.
kitd
People are usually the biggest cost in any organisation. If you can run all your systems without the sysadmins & netadmins required to keep it all upright (especially at expensive times like weekends or run up to Black Friday/Xmas), you can save yourself a lot more than the extra it'll cost to get a cloud provider to do it all for you.
Lalabadie
I'm a designer with enough front-end knowledge to lead front-end dev when needed.
To someone like me, especially on solo projects, using infra that effectively isolates me from the concerns (and risks) of lower-level devops absolutely makes sense. But I welcome the choice because of my level of competence.
The trap is scaling an org by using that same shortcut until you're bound to it by built-up complexity or a persistent lack of skill/concern in the team. Then you're never really equipped to reevaluate the decision.
f1shy
If everything is properly done, it should be next to trivial to add a server. When I was working on that we had a written procedure, when followed strictly, it would just take less than an hour
binary132
It’s kinda good if your requirements might quadruple or disappear tonight or tomorrow, but you should always have a plan to port to reserved / purchased capacity.
ep103
The benefit of cloud has always been that it allows the company to trade capex for opex. From an engineering perspective, it trades scalability for complexity, but this is a secondary effect compared to the former tradeoff.
throwaway894345
If you’re just running some CRUD web service, then you could certainly find significantly cheaper hosting in a data center or similar, but also if that’s the case your hosting bill is probably a very small cost either way (relative to other business expenses).
> You virtually never need scalability so fast that you don't have time to add another server
What do you mean by “time to add another server?” Are you thinking about a minute or two to spin up some on-demand server using an API? Or are you talking about multiple business days to physically procure and install another server?
The former is fine, but I don’t know of any provider that gives me bare metal machines with beefy GPUs in a matter of minutes for low cost.
Aissen
As an infrastructure engineer (amongst other things), hard disagree here. I realize you might be joking, but a bit of context here: a big chunk of the success of Cloud in more traditional organizations is the agility that comes with it: (almost) no need to ask permission to anyone, ownership of your resources, etc. There is no reason that baremetal shouldn't provide the same customer-oriented service, at least for the low-level IaaS, give-me-a-VM-now needs. I'd even argue this type of self-service (and accounting!) should be done by any team providing internal software services.
abujazar
The permissions and ownership part has little to do with the infrastructure – in fact I've often found it more difficult to get permissions and access to resources in cloud-heavy orgs.
rcxdude
I think also this was only a temporary situation caused by the IT departments in these organisations being essentially bypassed. Once it became a big important thing then they have basically started to take control of it and you get the same problems (in fact potentially more so because the expense means there's more pressure cut down resources).
michaelt
"No need to ask permission" and "You get the same bill every month" kinda work against one another here.
ambicapter
I'm at a startup and I don't have access to the terraform repo :( and console is locked down ofc.
blibble
don't underestimate the ability of traditional organisations to build that process around cloud
you keep the usual BS to get hardware, plus now it's 10x more expensive and requires 5x the engineering!
kccqzy
That's a cultural issue. Initially at my workplace people needed to ask permissions to deploy their code. The team approving the deployment got sick of it and built a self-service deployment tool with security controls built in and now deployment is easy. All it matters is a culture of trusting other fellow employees, a culture of automating, and a culture of valuing internal users.
alexchantavy
> no need to ask permission to anyone, ownership of your resources, etc
In a large enough org that experience doesn’t happen though - you have to go through and understand how the org’s infra-as-code repo works, where to make your change, and get approval for that.
darkwater
> What is old is new again.
I think there is a generational part as well. The ones of us that are now deep in our 40s or 50s grew up professionally in a self-hosted world, and some of us are now in decision-making positions, so we don't necessarily have to take the cloud pill anymore :)
Half-joking, half-serious.
olavgg
I'm in my 40s and run my own company. We deliver a data platform, our customers can choose between our self-hosted solution or run it on AWS/Azure for 10x higher cost.
Damogran6
As a career security guy, I've lost count of the battles I've lost in the race to the cloud...now it's 'we have to up the budget $250k a year to cover costs' and you just shrug.
The cost for your first on-prem datacenter server is pretty steep...the cost for the second one? Not so much.
marcosdumay
> What is old is new again.
It's not really. It just happens that when there is a huge bullshit hype out there, people that fall for it regret and come back to normal after a while.
Better things are still better. And this one was clearly only better for a few use-cases that most people shouldn't care about since the beginning.
kccqzy
My employer also resisted using cloud compute and sent staff explanations why building our own data centers is a good thing.
HPsquared
"Do nothing, Win"
Thicken2320
Using the S3 API is like chopping onions, the more you do it, the faster you start crying.
scns
Less to no crying when you use a sharp knive. Japanese chefs say: no wonder you are crying, you squash them.
Esophagus4
Haha!
My only “yes, but…” is that this:
> 50k API calls per second (on S3 that is $20-$250 _per second_ on API calls!).
kind of smells like abuse of S3. Without knowing the use case, maybe a different AWS service is a better answer?
Not advocating for AWS, just saying that maybe this is the wrong comparison.
Though I do want to learn about Hetzner.
wredcoll
You're (probably) not wrong about the abuse thing, but it sure is nice to just not care about that when you have fixed hardware. I find trying to guess which of the 200 aws services is the cheapest kinda stressful.
mike_hearn
Why would it be abuse? Serving e.g. map tiles on a busy site can get up to tens of thousands of qps, I'd have thought serving that from S3 would have made sense if it weren't so expensive.
wiether
They conveniently provide no detail about the usecase, so it's hard to tell
But, yeah, there's certainly a solution to provide better performances for cheaper, using other settings/services on AWS
realitysballs
Ya but then you need to pay for a team to maintain network and continually secure and monitor the server and update/patch. The salaries of those professionals , really only make sense for a certain sized organization.
I still think small-midsized orgs may be better off in cloud for security / operations cost optimization.
esskay
You still need those same people even if you're running on a bunch of EC2 and RDS instances, they aren't magically 'safer'.
lnenad
I mean, by definition yes they are. RDS is locked down by default. Also if you're using ECS/Fargate (so not EC2) as the person writing the article does, it's also pretty much locked down outside of your app manifest definitions. Also your infra management/cost is minimal compared to running k8s and bare metal.
abenga
This implies cloud infrastructure experts are cheaper than bare metal Linux/networking/etc experts. Probably in most smaller organizations, you have the people writing the code manage the infra, so it's an "invisible cost", but ime, it's easy to outgrow this and need someone to keep cloud costs in check within a couple of years, assuming you are growing as fast as an average start-up.
ldoughty
I think it's completely different ballparks to compare the skill sets...
It is cheaper/easier for me to hire cloud infrastructure _capable_ people easier and cheaper than a server _expert_. And a capable serverless cloud person is MUCH cheaper and easier to find.
You don't need to have 15 years of a Linux experience to read a JSON/YAML blob about setting up a secure static website.. of you need to figure out how to set up an S3 bucket and upload files... And another bucket for logging... And you have to go out of your way now to not be multi-az and to expose it to public read... I find most people can do this with minimal supervision and experience as long as they understand the syntax and can read the docs.
The equivalent to set up a safe and secure server is a MUCH higher bar. What operating system will they pick? Will it be sized correctly? How are application logs offloaded? What are the firewall rules? What is the authentication / ssh setup? Why did we not do LDAP integration? What malware defense was installed? In the event of compromise, do we have backups? Did you setup an instance to gather offloaded system logs? What is the company policy going to be if this machine goes down at 3am? Do we have a backup? Did we configure fail over?
I'm not trying to bash bare metal. I came from that space. I lead a team in the middle of nowhere (by comparison to most folks here) that doesn't have a huge pool of people with the skills for bare metal.. but LOTS of people that can do competent severless with just one highly technical supervisor.
This lets us higher competent coders which are easier to find, and they can be reasonably expected to have or learn secure coding practices... When they need to interact with new serverless stuff, our technical person gets involved to do the templating necessary, and most minor changes are easy for coders to do (e.g. a line of JSON/YAML to toggle a feature)
adamcharnock
I very much understand this, and that is why we do what we do. Lots of companies feel exactly as you say. I.e. Sure it is cheaper and 'better', but we'll pay for it in salaries and additional incurred risk (what happens if we invest all this time and fail to successfully migrate?)
This is why we decided to bundle engineering time with the infrastructure. We'll maintain the cluster as you say, and with the time left over (the majority) we'll help you with all your other DevOps needs too (CI/CD pipelines, containerising software, deploying HA Valkey, etc). And even after all that, it still costs less than AWS.
Edit: We also take on risk with the migration – our billing cycle doesn't start until we complete the migration. This keeps our incentives aligned.
DisabledVeteran
That used to be the case until recently. As much as neither I nor you want to admit it -- the truth is ChatGPT can handle 99% of what you would pay for "a team to maintain network and continually secure and monitor the server and update/patch." Infact, ChatGPT surpasses them as it is all encompassing. Any company now can simply pay for OpenAI's services and save the majority of the money they would have spent on the, "salaries of those professionals." BTW, ChatGPT Pro is only $200 a month ... who do you think they would rather pay?
tayo42
You have a link to some proof that chat gpt is patching servers running databases with no down time or data loss?
parliament32
I would pay you 100x that amount monthly to perform those services, as long as you assume the risk. If you're convinced this is viable, you should start a business :)
null
parliament32
If you haven't had to fight network configuration, monitoring, and security in a cloud provider you must have a very simple product. We deploy our product both in colos and on a cloud provider, and in our experience, bare-metal network maintenance and network maintenance in a PaaS consumes about the same number of hours.
dorkypunk
Then you have to replace those professionals with even more specialized and expensive professionals in order be able to deploy anything.
rightbyte
Isn't most vulnerabilities in your own server software or configs anyways?
rixed
I do not disagree, but just for the record, that's not what the article is about. They migrated to Hetzner cloud offering.
If they had migrated to a bare metal solution they would certainly have enjoyed an even larger increase in perf and decrease in costs, but it makes sense that they opted for the cloud offering instead given where they started from.
traceroute66
> on S3 that is $20-$250 _per second_ on API calls!
It is worth pointing out that if you look beyond the nickle & diming US-cloud providers, you will very quickly find many S3 providers who don't charge you for API calls and just the actual data-shifting.
Ironically, I think one of them is Hetzner's very own S3 service. :)
Other names IIRC include Upcloud and Exoscale ... but its not hard to find with the help of Mr Google, most results for "EU S3 provider" will likely be similar pricing model.
P.S. Please play nicely and remove the spam from the end of your post.
themafia
> We typically see a doubling of performance
The AWS documents clarify this. When you get 1 vCPU in a Lambda you're only going to get up to 50% of the cycles. It improves as you move up the RAM:CPU tree but it's never the case that you get 100% of the vCPU cycles.
rgrieselhuber
We moved DemandSphere from AWS to Hetzner for many of the same reasons back in 2011 and never looked back. We can do things that competitors can’t because of it.
dhruv_ahuja
Can you please explain what are some of those things? Curious to know and learn.
lisperforlife
I think you can get much farther with dedicated servers. I run a couple of nodes on Hetzner. The performance you get from a dedicated machine even if it is a 3 year old machine that you can get on server auction is absolutely bonkers and cannot be compared to VMs. The thing is that most of the server hardware is focused towards high core count, low clock speed processors that optimize for I/O rather than compute. It is overprovisioned by all cloud providers. Even the I/O part of the disk is crazy. It uses all sorts of shenanigans to get a drive that sitting on a NAS and emulating a local disk. Most startups do not need the hyper virtualized, NAS based drive. You can go much farther and much more cost-effectively with dedicated server rentals from Hetzner. I would love to know if they are any north-american (particularly canadian) companies that can compete with price and the quality of service like Hetzner. I know of OVH but I would love to know others in the same space.
ozim
As mentioned multiple times in other comments and places people think that doing what Google or FB is doing should be what everyone else is doing.
We are running modest operations on European VPS provider where I work and whenever we get a new hire (business or technical does not matter) it is like a Groundhog day - I have to explain — WE ALREADY ARE IN THE CLOUD, NO YOU WILL NOT START "MIGRATING TO CLOUD PROJECT" ON MY WATCH SO YOU CAN PAD YOUR CV AND MOVE TO ANOTHER COMPANY TO RUIN THEIR INFRA — or something along those lines but asking chatgpt to make it more friendly tone.
PeterStuer
The number of times I have seen fresh "architects" come in with an architectural proposal for a 10 user internal LoB app that they got from a Meta or Microsoft worldscale B2C service blueprint ...
kccqzy
> doing what Google or FB is doing
Google doesn't even deploy most of its own code to run on VMs. Containers yes but not VMs.
ozim
Well I think that’s the point people think if we run VPS and not containers or some app fabric, serverless so PaaS we are “not using real cloud”. But we use IaaS and it is also proper cloud.
dijit
Yeah, the irony being Google runs VMs in Containers but not the other way around.
jwr
I actually benchmarked this and wrote an article several years back, still very much applicable: https://jan.rychter.com/enblog/cloud-server-cpu-performance-...
kees99
Did you "preheat" during those tests? It is very common for cloud instances to have "burstable" vCPUs. That is - after boot (or long idle), you get decent performance for first few minutes, then performance gradually tanks to a mere fraction of the initial burst.
fakwandi_priv
> The total wall clock time for the build was measured. The smaller the better. I always did one build to prime the caches and discarded the first result.
The article is worth the read.
vicarrion
I also did a benchmark between cloud providers recently and compared performance for price
https://dillonshook.com/postgres-cloud-benchmarks-for-indie-...
fireant
That isn't the same as parent through, you are comparing VMs instead of dedicated servers
eahm
I recently rediscovered this website that might help: https://vpspricetracker.com
Too cool to not share, most of the providers listed there have dedicated servers too.
CaptainOfCoit
Great website, but what a blunder to display the results as "cards" rather than a good old table so you can scan the results rather than having to actually read it. Makes it really hard to quickly find what you're looking for...
Edit: Ironically, that website doesn't have Hetzner in their index.
dizhn
That is weird indeed. But I bet you are getting Hetzner results indirectly through resellers :) (Yeah I checked one Frankfurt based datacenter named FS1 - probably for Falkenstein. They might be colo or another datacenter there of course)
chromehearts
Amazing website, glad to know that I already have a super great offer! But will definitely share this
aantix
What a great site. Thanks for sharing!
63stack
This is an amazing site
ta12653421
++1
excellent website, thanks.
codethief
> I would love to know if they are any north-american (particularly canadian) companies that can compete with price and the quality of service like Hetzner
FWIW, Hetzner has two data centers in the US, in case you're just looking for "Hetzner quality but in the US", not for "American/Canadian companies similar to Hetzner".
CaptainOfCoit
IIRC, Hetzners dedicated instances are only available in their German and Finnish data centers, not anywhere else sadly :/
joshstrange
This is correct, they only offer VPS in the US.
ccakes
latitude.sh do bare metal in the US well
g8oz
Similarly OVH is French and has bare metal in their US and Canadian data centers.
matt-p
Yeah no dedicated severs in the US sadly. I'm not aware of anyone who can quite match hetzners pricing in the US (but if someone does I'd love to know!). https://www.serversearcher.com throws up clouvider and latitiude at good pricing but.. not hetzner levels by any means.
MrPowerGamerBR
I haven't checked Hetzner's prices in a while, but OVHcloud has dedicated servers and they do have dedicated servers in the US and in Canada (I've been using their dedicated servers for years already and they are pretty dang good)
shlomo_z
I have been considering colocating at endoffice (I saw the suggestion once at codinghorror.com)
wongarsu
> I would love to know if they are any north-american (particularly canadian) companies that can compete with price and the quality of service like Hetzner.
In a thread two days ago https://ioflood.com/ was recommended as US-based alternative
amelius
But I'm looking more for "compute flood" ...
deaux
On a similar note, I'm looking for a "Hetzner, but in APAC, particularly East Asia". I've struggled to find good options for any of JP, TW or KR.
b0ner_t0ner
LayerStack is very fast in APAC:
https://www.layerstack.com/en/dedicated-clouddeaux
Going to try this out, looks very much like what I was looking for.
citrin_ru
VMs are middle ground between AWS and dedicated hardware. With hardware you need to monitor it, report problems/failures to the provider, make necessary configuration changes (add/remove node to/from a cluster e. t. c.). If a team is coming from AWS it may have no experience with monitoring/troubleshooting problems caused by imperfect hardware.
pwmtr
We’ve been seeing the same trend. Lots of teams moving to Hetzner for the price/performance, but then realizing they have to rebuild all the Postgres ops pieces (backups, failover, monitoring, etc.).
We ended up building a managed Postgres that runs directly on Hetzner. Same setup, but with HA, backups, and PITR handled for you. It’s open-source, runs close to the metal, and avoids the egress/I/O gotchas you get on AWS.
If anyone’s curious, I added here are some notes about our take [1], [2]. Always happy to talk about it if you have any questions.
[1] https://www.ubicloud.com/blog/difference-between-running-pos... [2] https://www.ubicloud.com/use-cases/postgresql
normie3000
This is one key draw to Big Cloud and especially PaaS and managed SQL for me (and dev teams I advise).
Not having an ops background I am nervous about:
* database backup+restore * applying security patches on time (at OS and runtime levels) * other security issues like making sure access to prod machines is restricted correctly, access is logged, ports are locked down, abnormal access patterns are detected * DoS and similar protections are not my responsibility
It feels like picking a popular cloud provider gives a lot of cover for these things - sometimes technically, and otherwise at least politically...
ozim
Applying security patches on time is not much problem. Ones that you need to apply ASAP are rare and for DB engine you never put it on public access, most of the time exploit is not disclosed publicly and PoC code is not available for patched RCE right on day of patch release.
Most of the time you are good if you follow version updates for major releases as they come you do regression testing and put it on prod in your planned time.
Most problems come from not updating at all and having 2 or 3 year old versions because that’s what automated scanners will be looking for and after that much time someone much more likely wrote exploit code and shared it.
recroad
Exactly this. For a small team that's focused on feature development and customer retention, I tend to gladly outsource this stuff and sleep easy at night. It's not even a cost or performance issue for me. It's about if I start focusing on this stuff, what about my actual business am I neglecting. It's a tradeoff.
DanielHB
There must be SaaS services offering managed databases on different providers, like you buy the servers they put the software and host backups for you. Anyone got any tips?
swiftcoder
to be fair, AWS' database restore support is generally only a small part of the picture - the only option available is to spin an entirely new DB cluster up from the backup, so if your data recovery strategy isn't "roll back all data to before the incident", you have to build out all your own functionality for merging the backup and live data...
matt-p
I think the "strategy" for most people is to do it manually, or make the decision to just revert wholesale to a particular time.
baobun
In the adjacent category of self-managed omakase postgres: https://www.elephant-shed.io/
bdcravens
While I'm sure it's a great project, a few issues in the README gave me pause to think about how well it's kept up to date. Around half of the links in the list of dependencies are either out of date or just plain don't work, and referencing Vagrant with no mention of Docker.
baobun
It's indeed undermaintaned so it's not a case of only plug-and-play and automated pulls for production. Still a solid base to build from when setting up on VMs or dedicated and I'm yet to find something better short of DIYing everything.
slig
Also, Pigsty [1]. Feels too bloated for my taste, but I'd love to hear any experience from fellow HNers.
andybak
I love how few comments on this and similar posts give much context along with their advice. Are you hosting a church newsletter in your spare time or a resource intensive web app with millions of paying enterprise customers and a dedicated dev ops team in 3 continents?
Any advice on price / performance / availability is meaningless unless you explain where you're coming from. The reason we see people overcomplicating everything to do with the web is that they follow advice from people with radically different requirements.
cube00
> The reason we see people overcomplicating everything to do with the web is that they follow advice from people with radically different requirements.
Or they've had cloud account managers sneaking into your C-suite's lunchtime meetings.
Other comments in this thread say they get directives to use AWS from the top.
Strangely that directive often comes with AWS's own architects embedded into your team and even more strangely they seem to recommend the most expensive server-less options available.
What they don't tell is you you'll be rebuilding and redeploying your containerised app daily with new Docker OS base images to keep up with the security scanners just like patching an OS on a bare metal server.
casparvitch
IDK mate, my personal pastebin needs to run on bare metal or it can't keep up
DarkNova6
Tech industry in a nutshell
Hasz
Different requirements, different skillsets, different costs, different challenges. AWS is only topically the same product as Hetzner, coming from someone who has used both quite a bit.
Terretta
Strong agree. I hadn't seen your comment when I wrote this, below: https://news.ycombinator.com/item?id=45616366
TL;DR: Think of hosting providers like a pricing grid (DIY, Get Started, Pro, Team, Enterprise) and if YAGNI, don't choose it.
sergiotapia
> a dedicated dev ops team in 3 continents
you don't need that in 99.9999% of cases.
jwr
I've been running my SaaS on Hetzner servers for over 10 years now. Dedicated hardware, clusters in DE and FI, managed through ansible. I use vpncloud to set up a private VPN between the servers (excellent software, btw).
My hosting bill is a fraction of what people pay at AWS or other similar providers, and my servers are much faster. This lets me use a simpler architecture and fewer servers.
When I need to scale, I can always add servers. The only difference is that with physical servers you don't scale up/down on demand within minutes, you have to plan for hours/days. But that's perfectly fine.
I use a distributed database (RethinkDB, switching to FoundationDB) for fault tolerance.
withinboredom
Similar setup to me (including rethinkdb). Why choose FoundationDB? RethinkDb is still maintained and features added occasionally (I'm on the rethinkdb slack and maintain an async php driver). It just is one guy though, working on it part time.
jwr
RethinkDB is somewhat maintained, and while it is a very good database and works quite well, it is not future-proof. But the bigger reason is that I need better performance, and by now (after 10 years) I know my data access patterns well, so I can make really good use of FoundationDB.
The reason for FoundationDB specifically is mostly correctness, it is pretty much the only distributed database out there that gives you strict serializability and delivers on that promise. Performance is #2 on the list.
vjerancrnjak
How will you deal with lack of 3 AZ or FI to DE latency?
boobsbr
Nice to see someone still using RethinkDB.
da02
You use vpncloud to connect across different Hetzner data centers (DE + FI)? I thought/assumed Hetzner provided services to do this at little-to-no cost.
GordonS
Not the GP, but I also use Hetzner, but use Tailscale to connect securely across different Hetzner regions (and indeed other VPS providers).
Hetzner does provide free Private Networks, but they only work within a single region - I'm not aware of them providing anything (yet) to securely connect between regions.
jwr
No, I use vpncloud for a local (within a datacenter) VPN. This lets me move more configuration into ansible (out of the provider's web interfaces), avoid additional fees, and have the same setup usable for any hosting provider, including virtual clouds. Very flexible.
js4ever
We've helped quite a few teams move from AWS to Hetzner (and Netcup) lately, and I think the biggest surprise for people isn't the cost or the raw performance, it’s how simple things become when you remove 15 layers of managed abstractions.
You stop worrying about S3 vs EFS vs FSx, or Lambda cold starts, or EBS burst credits. You just deploy a Docker stacks on a fast NVMe box and it flies. The trade-off is you need a bit more DevOps discipline: monitoring, backups, patching, etc. But that's the kind of stuff that's easy to automate and doesn't really change week to week.
At Elestio we leaned into that simplicity, we provide fully managed open-source stacks for nearly 400 software and also cover CI/CD (from Git push to production) on any provider, including Hetzner.
More info here if you're curious: https://elest.io
(Disclosure: I work at Elestio, where we run managed open-source services on any cloud provider including your own infra.)
pqdbr
Would like to know more about your postgres offering: does it offer streaming replicas and streaming backup? Or just dump stored to s3?
js4ever
Yes we offer clusters with auto failover and replicas can be in multiple regions and even in multiple providers.
We support postgres but also MySQL, redis, opensearch, Clickhouse and many more.
About backups we offer differential snapshots and regular dumps that you can send to your own S3 bucket
https://docs.elest.io/books/databases/page/deploy-a-new-clus...
breadislove
Hetzner is really great until you try to scale with them. We started building our service on top of Hetzner and had couple 100s of VMs running and during peak time we had to scale them to over 1000 VMs. And here couple of problems started, you get pretty often IPs which are black listed, so if you try to connect to services hosted by Google, AWS like S3 etc. you can't reach them. Also at one point there were no VMs available anymore in our region, which caused a lot of issues.
But in general if you don't need to scale crazy Hetzner is amazing, we still have a lot of stuff running on Hetzner but fan out to other services when we need to scale.
jakewins
> Also at one point there were no VMs available anymore in our region, which caused a lot of issues.
I'm not sure if this is a difference between other clouds, at least a few years ago this was a weekly or even daily problem in GCP; my experience is if you request hundreds of VMs rapidly during peak hours, all the clouds struggle.
dinvlad
Right now, we can’t request even a single (1) non-beefy non-GPU VM in us-east on Azure. That’s been going on for over a month now, and that’s after being a customer for 2 years :(
GordonS
I don't use Azure much anymore, but I used to see this problem regularly on Azure too, especially in the more "niche" regions like UK South.
antonvs
We launch 30k+ VMs a day on GCP, regularly launching hundreds at a time when scheduled jobs are running. That’s one of the most stable aspects of our operation - in the last 5 years I’ve never seen GCP “struggle” with that except during major outages.
At the scale of providers like AWS and even the smaller GCP, “hundreds of VMs” is not a large amount.
Macha
If you’re deploying something like 100 m5.4xlarge in us-east-1, sure, AWS’s capacity seems infinite. Once you get into high memory instances, GPU instances, less popular regions etc, it drops off.
Now maybe after the AI demand and waves of purchases of systems appropriate for that things have improved, but it definitely wasn’t the case at the large scale employer I worked at in 2023 (my current employer is much smaller, so doesn’t have those needs, so I can’t comment)
dinvlad
Not a single VM possible to request on Azure us-east for over a month now though :-(
jamesblonde
The blocking of services on Hetzner and Scaleway by Microsoft is well known -
https://www.linkedin.com/posts/jeroen-jacobs-8209391_somethi...
I didn't know AWS and GCP also did it. Not surprised.
The problem is that European regulators do nothing about such anti-competitive dirty tricks. The big clouds hide behind "lots of spam coming from them", which is not true.
lossyalgo
First comment on that post claims that according to Mimecast, 37% of EU-based spam originates from Hetzner and Digital Ocean. People have been asking for 3 days for a link to the source (I can't find it either).
On the other hand, someone linked a report from last year[0]:
> 72% of BEC attacks in Q2 2024 used free webmail domains; within those, 72.4% used Gmail. Roughly ~52% of all BEC messages were sent from Gmail accounts that quarter.
[0] https://docs.apwg.org/reports/apwg_trends_report_q2_2024.pdf
jwr
Note that we might be talking about two different things here: some of us use physical servers from Hetzner, which are crazy fast, and a great value. And some of us prefer virtual servers, which (IMHO) are not that revolutionary, even though still much less expensive than the competition.
CaptainOfCoit
Worth noting that this seems to be about Hetzners cloud product, not the dedicated servers. The cloud product is relatively new, and most of the people who move to Hetzner do so because of the dedicated instances, not to use their cloud.
drcongo
Hetzner's cloud offering is probably a decade old by now - I've been a very happy customer for 8 years.
watermelon0
Hetzner was founded in '97, so cloud offering could technically still be considered relatively new. :D
CaptainOfCoit
You're right! I seem to have mixed it with some other dedi provider that added "cloud" recently. Thanks for the correction!
My point of people moving to Hetzner for the dedicated instances rather than the cloud still remains though, at least in my bubble.
V__
This sound really intriguing, and I am really curious. What kind of service do you run where you need a 100s of VMs? Was there a reason for not going dedicated? Looking at their offering their biggest VM is (48 CPU, 192 GB RAM, 960 GB SSD). I can't even imagine using that much. Again, I'm really curious.
breadislove
we have extremely processing heavy jobs where user upload large collection of files (audios, pdfs, videos etc.) and expect to get fast processing. its just that we need to fan out sometimes, since a lot of our users a sensitive to processing times.
GordonS
I've ran into the IP deny list problem too, but for Windows VMs - you spin them up, only to realise that you can't get Windows Updates, can't reach the Powershell gallery etc.
And just deleting it and starting again is just going to give you the exact same IP again!
I ended up having to buy a dozen or so IPs until I found one that wasn't blocked, and then I could delete all the blocked ones.
matt-p
I think they're great but it's unfortunate they don't have more locations which would at least enable you to spin VMs up in different locations during a shortage. If you rely on them it might be wise to have a second cloud provider that you can use in a pinch, there's many options.
FBISurveillance
We scaled to ~1100 bare metal servers with them and it worked perfectly.
atonse
Username checks out.
CaptainOfCoit
Best feature of (some) the dedicated servers Hetzner offers is the unmetered bandwidth. I'm hosting a couple of image-heavy websites (mostly modding related) and since moving to Hetzner I sleep much better knowing I'll pay the same price every single month, and have been for the ~3 years I've been a happy Hetzner customer.
999900000999
Long long ago, at the start of my career I was at a great company. We were using a Postgres DB version not supported by RDS. So I had to manually set up postgres over and over again on EC2 instances. This was before Docker was reliable/standard.
I wasted hours on this, and the moment RDS starts to support the postgres version we need it everything was much easier.
I still remember staying up till 3:00 a.m. installing postgres, repeatedly.
While this article is nice, they only save a few hundred dollars a month. If a single engineer has to spend even an hour a month maintaining this, it's probably going to be a wash.
And that's assuming everything goes right, the moment something goes wrong you can easily wipe out a year saving in a single day ( if not an hour depending on your use case).
This is probably best for situations where your time just isn't worth a whole lot. For example let's say you have a hobbyist project, and for some reason you need a very large capacity server.
This can easily cost hundreds of dollars a month on AWS, and since it's coming out of your own pocket it might be worth it to spend that extra time on bare metal.
But, at a certain point you're going to think how much is my time really worth. For example, and forgive me for mixing up terms and situations, ghost blog is about $10 a month via their hosted solution. You can probably run multiple ghost blogs on a single Hetzner instance.
But, and maybe it was just my luck, eventually it's just going to stop working. Do you feel like spending two or three hours fixing something over just spending the $20 a month to host your two blogs ?
CuriouslyC
I use Hetzner for this reason, but there are caveats. They're great but their uptime isn't as good as AWS and they don't have great region coverage. I strongly advise people to pair them with Cloudflare. Use Hetzner for your core with K8s, and use R2/D1/KV with Container Durable Objects to add edge coprocessing. I also like to shard customer data to individual DOs, this takes a ton of scaling pressure off your data layer, while being more secure/private.
geenat
AWS has certainly had some pretty public facing downtime ;) I'd say its been roughly the same in my experience- the only way to avoid it IMHO is multi-region.
CaptainOfCoit
I do this too. Hetzner dedicated servers for the "core" and data-storage basically, and thin/tiny edge-nodes hosted at OVH across the globe as my homebrew CDN.
BoredPositron
That's exactly how we do it we have Gcore in the mix for GPU compute though.
likium
If customer data is considered edge, then what’s core?
CuriouslyC
Everything that's shared between customers, internal system state and customer metadata. I use Postgres with FDWs + Steampipe + Debezium to integrate everything, it's more like a control plane than a database. This model lets you go web scale with one decently sized database and a read replica, since you're only hitting PG for fairly static shared data, Cloudflare Hyperdrive gives insane performance.
sergioisidoro
I really liked Hetzner but I got burned by one issue. I had some personal projects running there and the payment method failed. Automated email communications also failed among so much spam and email notifications I receive, and when I noticed the problem they had wiped all my data without possibility of recovery.
It was a wake up moment for me about keeping billing in shape, but also made me understand that a cloud provider is as good as their support and communications when things go south. Like an automated SMS would be great before you destroy my entire work. But because they are so cheap, they probably can't do that for every 100$/month account.
I've had similar issues with AWS, but they will have much friendlier grace periods.
dotancohen
> It was a wake up moment for me about keeping billing in shape
sergioisidoro
Yep. And importantly - backups on different cloud providers, with different payment methods.
roflmaostc
Sorry to hear that.
But if you do not pay and you do not check your e-mails, it's basically your fault. Who is using SMS these days even?
oefrha
I had payment issues with Hetzner too, that was back in 2018, haven’t used them since. At least back then, and at least for me, they were unlike any other provider I’ve used which would send you plenty of warnings if they fail to bill you. The very first email I got from them that smelt of trouble was “Cancellation of Contract”, at which point my account was closed and I could only pay by international bank wire. (Yes I just checked all my correspondence with them to make sure I’m not smearing them.) Amusingly they did send payment warning after account closure. Why not before? No effing clue. That was some crazy shit.
sergioisidoro
Yes, absolutely my fault. But these problems happen. Credit cards expire, people change companies or go on leaves, off boarding processes are not always perfect, spam filters exist.
Add to that the declining experience of email with so much marketing and trash landing in the inbox (and sometimes Gmail categorizing important emails as "Updates")
That's why grace periods for these situations are important.
Who uses SMS? This might be a cultural difference, but in Europe they are still used a lot. And would you be ok if your utility company cut your electricity bill just with an email warning? Or being asked to appear to court by email?
account42
> Add to that the declining experience of email with so much marketing and trash landing in the inbox (and sometimes Gmail categorizing important emails as "Updates")
This is also something under your control - you don't have to use Gmail as your email provider for important accounts and you can whitelist the domains of those service providers if you don't rely on a subpar email service.
amelius
How long after shutting you down did they delete your data?
That period should definitely be longer than a few days.
matdehaast
I've had billing issues, and they have let it be resolved a couple of weeks later.
null
1a527dd5
Love it!
We are unfortunately moving away from self-hosted bare metal. I disagree with the transition to AWS. But it's been made several global layers above me.
It's funny our previous AWS spend was $800 per month and has been for almost 6 years.
We've just migrated some of our things to AWS and the spend is around $4,500 per month.
I've been told the company doesn't care until our monthly is in excessive of five figures.
None of this makes sense to me.
The only thing that makes sense is our parent company is _huge_ and we have some really awesome TAMs and our entire AWS spend is probably in the region of a few million a month, so it really is pennies behind the sofa when global org is concerned.
Terretta
There are many other costs besides that AWS bill. Naming two it's hard to put a number on, but get discussed at board room or senior exec level:
- client confidence
- labor pool
aunty_helen
And to add to that second one, ability to bring in a third party contractor to reduce headcount when needed.
Sammi
I read some many stories like this and every time I think of the "your margins are my opportunity" quote, and think there must be so many inefficient enterprises that are ripe for disruption by a small efficient team.
cube00
> None of this makes sense to me.
OpEx good, CapEx bad.
1a527dd5
Now you mention it, the other thing we are being forced to do is categorise our work (e.g. commits/PRs) as Cap/Op. And then once a year a bunch of us are randomly selected by one of the big four auditing companies to talk about why that piece of work was Cap/Op.
marcosdumay
What could make sense if the OP was talking about a less than 30% difference.
What country is it that applies a 400% income tax to companies?
(Well, seriously, it makes sense in a larger than 80% tax rate. Not that impossibly high, but I doubt any country ever had it.)
I cannot overstate the performance improvement of deploying onto bare metal. We typically see a doubling of performance, as well as extremely predictable baseline performance.
This is down to several things:
- Latency - having your own local network, rather than sharing some larger datacenter network fabric, gives around of order of magnitude reduced latency
- Caches – right-sizing a deployment for the underlying hardware, and so actually allowing a modern CPU to do its job, makes a huge difference
- Disk IO – Dedicated NVMe access is _fast_.
And with it comes a whole bunch of other benefits:
- Auto-scalers becomes less important, partly because you have 10x the hardware for the same price, partly because everything runs 2x the speed anyway, and partly because you have a fixed pool of hardware. This makes the whole system more stable and easier to reason about.
- No more sweating the S3 costs. Put a 15TB NVMe drive in each server and run your own MinIO/Garage cluster (alongside your other workloads). We're doing about 20GiB/s sustained on a 10 node cluster, 50k API calls per second (on S3 that is $20-$250 _per second_ on API calls!).
- You get the same bill every month.
- UPDATE: more benefits - cheap fast storage, run huge Postgresql instances at minimal cost, less engineering time spend working around hardware limitations and cloud vagaries.
And, if chose to invest in the above, it all costs 10x less than AWS.
Pitch: If you don't want to do this yourself, then we'll do it for you for half the price of AWS (and we'll be your DevOps team too):
https://lithus.eu
Email: adam@ above domain