It's five grand a day to miss our S3 exit

Apple may have markup, but the part is for sure more likely to be higher quality: https://www.cultofmac.com/news/apple-thunderbolt-4-cable-com...

It's just people conflating popularity with <every positive attribute>.

If <service> is popular, it must also be cheap, beautiful, well documented, have every feature that exists and make you popular with your friends.

I once had a Product Manager try to start an argument with me: "Explain to me how it is possible that the service we pay 25k a month doesn't have <feature>. You don't know what you are saying.". It just didn't do what he wanted, and getting angry with them over the phone didn't magically made the feature appear.

Yeah, I used to be asked to price out a move to AWS every year at one position. After several years Hetzner finally got cheaper than operating our own colo's, but only basically because we were in London and London real-estate is expensive, and so colo space is accordingly expensive, while Hetzner's DC space is dirt cheap.

AWS, however, remained 2x-3x as expensive, with the devops time factored in.

> The perception seems to be "Cloud exists, therefore it must be cheaper, because if it wasn't cheaper why would it exist?

People are also blithely unaware that large customers get significant discounts, and so I regularly has to explain that BigCo X being hosted in AWS means at most that it is cost-effective for them because their spend means they're getting a significant discount over the already highest volume published pricing, and my clients usually are nowhere close to spend enough to be able to get those discounts.

> If we'd done that once I'd feel like well management weren't to know, but it recurs with about a 3-5 year periodicity.

So basically every time management changes[1]?

[1]: https://maexecsearch.com/average-c-suite-tenure-and-other-im...

Regarding apple parts, I recently replaced a broken screen on a MacBook pro with an OEM part. I can’t get the color to look right. Not to mention the one vertical row where pixels look off (not dead, but not normal either). The guy at the shop said I would not notice. I am now kicking myself for not going with the real thing.

I think management is just prone to wanting to believe the grass is greener on the other side. If you are already a cloud org with negotiated pricing and cost-optimization management would ask about building a data center and you would show them how much you would need to expand your IT staff in order to acquire the skills to operate the new data center never mind the upfront cost.

[flagged]

I can provide an example where cloud, despite its vastly higher unit costs, makes sense. Analytics in high finance (note: not HFT). Disclosure: my employer provides systems for that.

A fair number of our clients routinely spin up workloads that are CPU bound on hundreds-to-thousands of nodes. These workloads can be EXTREMELY spiky, with a baseload for routine background jobs needing maybe 3-4 worker nodes, but with peak uses generating demand for something like 2k nodes, saturating all cores.

These peak uses also tend to be relatively time sensitive, to the point where having to wait two extra minutes for a result has real business impact. So our systems spin up capacity as needed, and once the load subsides, terminates unused nodes. After all, new ones can be brought up at will. When the peak loads are high (& short) enough, and the baseload low enough, the elastic nature of cloud systems has merit.

I would note that these are the types of clients who will happily absorb the cross-zone networking costs to ensure they have highly available, cross-zone failover scenarios covered. (Eg. have you ever done the math on just how much a busy cross-zone Kafka cluster generates in zonal egress costs?) They will still crunch the numbers to ensure that their transient workload pools have sufficient minimum capacity to service small calculations without pre-warm delay, while only running at high(er) capacity when actually needed.

Optimising for availability of live CPU seconds can be a ... fascinating problem space.

> paying more since it's flexible, so we can scale up/down quickly

I’ve heard this argument too and I think I’ve seen exactly one workload where it actually made sense and was tuned properly and worked reliably.

I've noticed this too, freelancing/consulting around in companies. I'm not sure where this idea even comes from

Internal company accounting can be weird and lead to unintuitive local optima. At companies I've worked at, what was objectively true was that cloud was often much cheaper than what the IT department would internally bill our department/project for the equivalent service.

I think it's because people think their workloads are extremely spiky, and so assume they will spin up/down loads enough to save money, and that has translated into cloud being perceived as cheap.

But devs rarely pay attention to metrics. I've had clients with expensive Datadog setups where it was blatantly obvious that nobody had ever dug into the performance data, because if they did they'd have noticed that key metrics were simply not fed to it.

If they did pay attention, most of them would realise that their autoscaling rarely kicks in all that much, if at all. Often because it's poorly tuned, but also because most businesses see small enough daily cycles.

Factor in that the cost difference between instances vs. managed servers is quite significant, and you need to have significant spikes much shorter in duration than most businesses day/night variation to save money.

It can make sense to be able to spin up more capacity quickly, but then people need to consider that 1) a lot of managed hosting providers has hardware standing by and can automatically provision it for you rapidly too - unless you insist on only using your own purchased servers in a colo, you can get additional capacity quickly, 2) a lot of managed hosting providers also have cloud instances so you can mix and match, 3) worst case you can spin up cloud instances elsewhere and tie it into your network via a VPN.

Some offer the full range from colo via managed servers to cloud instances in the same datacentres.

Once you prep for a hybrid setup, incidentally, cloud becomes even less competitive, because suddenly you can risk pushing the load factor on your own/managed servers much closer to the wire, knowing you can spin up cloud instances as a fallback. As a result, the cost per request for managed servers drops significantly.

I also blame a lot of this on business often shielding engineering from seeing budgets and costs. I've been in quite senior positions in a number of companies where the CEO or CFO were flabbergasted when I asked for basics costing of staff and infra, because I saw it as essential in planning out architecture. Engineers who aren't used to seeing cost as part of their domain will never have a good picture of costs.

You don't have to buy the hardware. It's very common to rent it.

But that too is based on people not knowing the alternatives, as renting managed servers can be close to a wash vs. leasing hardware for a colo (often to the point that relatively cost of land near your preferred managed hosting providers vs. colos that work with access to staff etc. might be what makes one or the other cheaper). Buying outright can be cheaper but isn't necessary.

None of the colo'd setups I've worked on bar one used purchased servers - it's all been leased. But the majority of non-cloud workloads I've worked on have not even been leased, but rented.

Cloud made sense for the startup I worked for previously. If you are a startup then a $1M per year expense makes much more sense than a $5M up front purchase with 5-10 years of life - in five years you might be billionaires or you might be bankrupt and until then the Cloud was better.

For smaller businesses it's often "the only thing Joe knew when he was building it".

> when finance oversees IT despite not understanding it

... and when IT often do not even get to see the spend, and/or isn't expected to.

I've had clients where only finance had permissions to get at the billing reports, and engineering only ever saw the billing data when finance were sufficiently shocked by a bill to ask them to dig into it - at which point they cared for long enough to get finance off their backs, and then stopped caring again.

This was more a response to the comment I replied to, that cloud is always more expensive. And saying it more for everyone, not OP.

It's almost always less expensive at the start, which is super important for the early stages of a company (your capital costs are basically zero when choosing say AWS).

Then after you're established, it's still cheaper when considering opportunity costs (minor improvements in margin aren't usually the thing that will 10x a company's value, and adding headcount has a real cost).

But then your uniqueness as a company will come into play and there will be some outsized expense that seems obscene for the value you get. For the article writer, it was S3, for the OP, it's bandwidth. For me it's lambdas (and bizarrely, cloud watch alarms). That's when you need to have a hard look and negotiate. Sometimes the standard pricing model really doesn't consider how you're using a certain service, after all it's configured to optimize revenue in the general case. That doesn't mean the provider isn't going to be willing to take a much lower margin on that service if you explain why the pricing model is an issue for you.

So obviously this is an extreme, but I worked for a company that had long dismissed third party cloud providers as too expensive (customers would be routing all of their network traffic through our data centers, so obviously the bandwidth costs would just be too dang high). Then that company got purchased by a certain mega corporation who then negotiated an exclusive deal with GCP, and the math flipped. It was now far too expensive to run our own set of datacenters. Google was willing to take such a low margin on bandwidth that it made no sense not to.

So in this case, hundreds of billions. But the principle stands at lower company sizes, just with different numbers and amounts of leverage.

I don’t remember if our first enterprise agreement was at $1M or $2M, but it was low and in that neighborhood [but also 10 years ago, well before cloud was the default and had growth baked into it].

Cloud providers are looking for multi-year term, commitment to growth as much as/more than exact spend level now.

In my experience with GCP, go through a Google partner (that will aggregate multiple clients to get discounts) and you'll be able to get commitment discounts with $500K/year or even less. But don't save too much money during your commitment period: if you don't expend your commitment, you'll pay for it anyway, and you might even lose some discounts.

Also, one trick to inflate your commitment expenses is asking your SaaS providers if it's possible to pay them through AWS or GCP marketplaces: it often counts against your commitment minimum expense, so not everything has to be instances and storage.

S3 is beyond impressive, but how many workloads truly need that? I’ve never had a single instance of data loss on a NetApp or Pure array.

On the other hand you have transient failures in the cloud (at least on Azure - this behavior is even documented) so does that count towards the 99.99999%?

> How many are actually multi-region?

The fact half the internet seems to fall over whenever us-east-1 has a hiccup is quite telling.

A lot more than you’re giving them credit for.

This idea that their basic users go down entirely when the AZ has some issues is ridiculous, a standard autoscaling group and load balancer basically forces you to be multi-AZ. Very much unlike a VPS.

Using RDS instead of self-installing SQL eliminates the need for an entire full time role for DB admin. So that’s kind of a big deal despite it being a “basic” use case.

A lot of services like ECS, elastic beanstalk, can make it so that you can wait longer to hire operations people and when you do they can migrate to more scalable solutions without having to do a major migration to some other provider or build up a self hosted solution custom. If you outgrow a VPS you have to do a major migration.

And if you take a look at the maturity and usefulness of the terraform providers SDKs, and other similar integrations of VPS and bare metal providers they are very basic when comparing to BOTO and the terraform provider.

I struggle to replicate the level of automation I can achieve with these cloud tools on my own homelab with Proxmox.

I can’t speak too much for small companies. But there are a lot of large enterprises and smaller businesses and government agencies that do use more AWS services than just compute + storage + web services. Do need the elasticity etc.

For instance, I was surprised how large the market was for Amazon Connect - Amazon’s hosted call centers. It’s one of the Amazon services I have some experience in and I still get recruiters contacting me for those jobs even though I don’t really emphasize that specialty.

My experience is from 7 years of working with AWS. First at a startup with a lot of complex ETL and used a lot of services. But the spend wasn’t that great.

My next 5 years was between working at AWS (Professional Services) and two years at a a third party consulting company (full time) mostly as an implementation lead.

Even though my specialty is “cloud native application development” and I avoid migrations like the plague, most of the money in cloud consulting are large companies deciding to move to the cloud because they decided that the redundancy, lower maintenance overhead, and other higher level services were worth it.

> Write off is am accounting term, not operational.

That's the point. I've just decommisioned 10 year old servers for a client. They were still working fine, but the system had finally been replaced.

If you're calculating break-even based on the rate at which you're writing off the accounting value of the servers, you'll end up with a far longer time to break-even than if you amortise the hardware cost over the projected actual lifetime of the hardware.

If you built up that domain knowledge while being paid top dollar per hour by the same company, then I understand their reluctance to go along with your offer. It feels a little bit extortionate to be honest. I wouldn't go along with it either, not because it's a bad deal in isolation, but because it sets a bad precedent. It basically tells every employee/contractor that if they know a way to add a lot of measurable value, they can use that as a bargaining chip to 3x their pay. This also discourages trying to add any value that isn't as easily expressed in dollars (which is the case for many important things, such as product quality improvements).

I think part of the expectation when contracting somewhere long-term (or just being an employee, for that matter) is that the amount of value you add per hour worked increases sharply over time, and slower than your fee. In other words, initially you're overpaid wrt your value-add, and then that corrects itself over time as you figure out what the company is all about.

I've made similar pitches to clients many times, and one thing I've learned is that ironically the problem is promising the actual saving, vs. offering a much smaller saving.

The challenge is that people don't believe you when you tell them they can save that much, no matter how evidence you prepare. I'm starting a sales effort for my agency right now, and one of the things we've worked on is to promise less than what we determine we can deliver after reviewing the clients costs, and raising our prices, because it's ironically easier to close on the basis of a promise to deliver 20%-30% savings at a relatively high cost than a promise to deliver 50%+ with little effort.

> I know it's very petty

No it isn't.

My current and last jobs had monthly RDBMS bills in excess of $1 million/month. It is staggering. We could buy two fully-loaded 42U racks in separate DCs and be net positive after a few months. I’ve done the math, in great detail.

No go. “It’s hard to hire for that skill set.” Is it $9 million/year hard?! You already have a team lead – me. This shit is not that hard; people will figure it out, I promise.

It's not petty. It's the profit margin average solutions get to make over expertise with individual tools, or groups of tools.

Considering it was about 100 developers, it was horrible.

The two major problems were:

1. The volume of data itself was not that that big (I had a backup on my laptop for reproductions), but it was just too heavy for even the biggest things in AWS. Downtimes were very frequent. This is mostly due to decisions from 10 years ago.

2. Teams constantly busy putting out fires but still getting only 1-2% salary increases due to lack of new features.

EDIT: Since people like those war stories. The major cause for the performance issues was that each request from an internal user would sometimes trigger hundreds of queries to the database. Or worse: some GET requests would also perform gigantic writes to the Double-Entry Accounting system. It was very risky and very slow.

This was mostly due to over-reliance on abstractions that were too deep. Nobody knew which joins to make in the DB, or was too afraid, so they would instead call 5 or 6 classes and joining manually causing O(N^2) issues.

To give a dimension of how stupid it was: one specific optimization I worked on changed the rendering time of a certain table from 25 seconds to 2 miliseconds. It was nothing magic.

I'm glad I left.

My experience from offering devops services on retainer to a number of clients is that the ones that host in cloud environments spend more money on me for similar scale setups than the ones that host on managed setups.

And even if you don't want the hassle of storing the data yourself, there are many far cheaper outsourced options than S3.

I run multiple SaaSes, some popular, some not so much, off a single $20/mo Hetzner VPS with Dokku on it. It works great, and I've never needed to worry about anything.

Meanwhile, I see friends working on MVPs with 1-2 non-paying customers who already have costs in the thousands of dollars a month, but "it's fine because we got free money for a year". Yes, but that means that your company now has an expiration date of a year.

Were the files lots of smaller files or lots of large files? I imagine if lots of small files requires a lot of concurrent threads to utilize a 10Gbps connection.

>Actually if Intel were to push 18A on server it would make performance / density even better

Don't you end up needing pretty insane electric capacity not to mention cooling for that kind of density?

Sure, but making something redundant is not really that difficult. HAProxy in front N nodes across M racks, ideally in separate DCs, and then a floating IP in front of your HAProxies. Set up hot standby for your DB.

I used to joke that my homelab almost had better reliability than any company I’d been at, save for my ISP’s spotty availability. Now that I have a failover WAN, it literally is more reliable. In the five years of running a rack, I’ve had precisely one catastrophic hardware failure (mobo died on a Supermicro). Even then, I had a standby node, so it was more of an annoyance (the standby ran hotter and louder) than anything.

Got any recommended providers?

You would not need to have extended downtime. Every major RDBMS that I’m aware of supports standby nodes, and if you want, a full active-active cluster (not recommended, personally).

The downtime is as long as you have your health check monitoring interval set up for.

When using colocation, nothing is stopping people from storing the database data externally from the server running the database like some cloud services do. But doing so, either in cloud or not, does have a serious downside: greatly increased latency.