Grafana: Why observability needs FinOps, and vice versa
44 comments
·February 6, 2025brunoarueira
sda2
ha, we use NewRelic for our application but the company is so cheap, they won’t even buy the infra team a license!
brunoarueira
Haha appears some managers change only the address but the behavior is the same, I already worked on a place where we had to use free heroku addons besides the paid PostgreSQL and the dynos
jsiepkes
One of the advantages of self-hosting is that you don't need this level of FinOps. You also don't have to live in fear of bill-mageddon.
danpalmer
And one of the disadvantages is that you can't solve problems by just spending more. It's a real trade-off, and too often is simplified to one option being obviously better.
m1keil
You can't spend more fast, but you can always spend more.
danpalmer
Why would you spend more if it's not solving problems?
You can certainly spend more, but on-prem/self-hosted, time is usually the limiting factor, either directly or through opportunity cost. Contrary to popular belief, time does not always equal money, if you need more storage and the blocker is that you need to build out an S3 equivalent (rather than just paying for S3), then you'll be blocked by hiring, by hardware lead times, etc.
jterrys
lol.
We recently switched from Grafana to Prometheus. Reason being that a license refresh took longer to process on their end. What happens when a license expires on Grafana? They fucking shut down all your shit cold turkey. Don't care if you're in prod or have a dedicated guy on their end for support or whatever. So you're happily churning along and then suddenly you're blind. Nice FinOps. With Prometheus there's a grace period where they'll happily overcharge you. But we've never had a product absurdly blow up on us like this before. It's truly mind boggling that they're out here talking about 'FinOps' now.
m1keil
So next time someone says "in cloud you don't need anyone to manage it for you" you can link them this article.
NotGMan
You know you have a problem when you're afraid to add a few more metrics because the bill might get too high.
danpalmer
This was the reason that we ended up choosing Datadog over Grafana Cloud at my previous place. Most metrics came from "integrations", and Datadog doesn't charge extra for any of those (they just curate them so that the cardinality can't get too high), whereas Grafana charges (charged?) for each metric and didn't provide anything to reduce cardinality. Their solution was to suggest we did more engineering and ran more infrastructure to aggregate before sending to them, not something we wanted to invest in given that the whole point was to not self-host Grafana anymore.
Datadog is expensive, but at least we were only making these decisions for the ~hundreds of custom business metrics, and not the ~tens of thousands of metrics from our infrastructure.
kozikow
I am big fan of "cost monitoring".
In my previous company I had a good setup for costs monitoring - including release to release comparisons, drill downs, statistics, etc.
After each release I looked at this data. It saved a lot of $, by simple fixes like "why we are calling this API twice?".
It also quite some issues that weren't strictly customer related, but weren't apparent from other type of data (you will always have some "unknown unknowns" in your monitoring, and costs data seem to be pretty wide net to catch some of those)
m1keil
What levels of observability did you have for costs of data transfer and how did you do it?
lukaslalinsky
I'm a fan of Grafana, both the main tool and also the infrastructure projects they have been working on, they did a great service to the IT world. However, the pricing of their cloud service, that's some shady business. It goes from reasonable pricing to oh-my-god pricing so fast. I really wish they would introduce limits on their paid plans, but that's against their business practice.
floating-io
Seriously? Not everything needs to be xxxOps. And if you need a "FinOps" team to manage your cloud cost, I would argue that there's something wrong with your whole damn paradigm.
StratusBen
I'm the author of the blog post and the CEO of https://vantage.sh/
I kind of agree that I don't love the term. That being said it has become the de-facto way that people refer to the space and practice. The FinOps Foundation hyped up the term and space quite a bit which they deserve credit for but do wish there was a better name :)
mvdtnz
> if you need a "FinOps" team to manage your cloud cost, I would argue that there's something wrong with your whole damn paradigm.
How would you manage your cloud costs if you ran a company of, say, 4,000 engineers? Balancing the needs of delivery teams to build their technology with the needs of the business to manage costs. Do you think every single team should directly report their cloud costs to the CFO? Or at that scale does it make more sense to report costs to another individual? And when that needs to scale, maybe we give that individual a team?
moandcompany
The _Ops team needs an _Ops' team.
The game here is defining parts of the job away to be someone else's job or responsibility.
arccy
we invent new functions to use to decorate our CVs with
j0rdans
I think for most smaller orgs you can get away with an off the shelf product to surface some more basic cost stuff. In relatively large engineering orgs, you're looking at optimising on stuff like cross-region calls to save millions a year so yeah there's a good reason to invest in cloud cost management.
sebazzz
> And if you need a "FinOps" team to manage your cloud cost, I would argue that there's something wrong with your whole damn paradigm.
FinOps isn't a dedicated job but something a cloud engineer can do as part of its job. In the same way that DevOps doesn't need to be a dedicated function itself.
And as for the cloud... Yes that turned out to be a whole lot expensive for companies than predicted - and you share compute with anyone so that dualcore CPU isn't always that fast. But cloud is also flexible and that is where FinOps comes in.
svilen_dobrev
> And if you need a "FinOps" team to manage your cloud cost,
somehow i misread that as "massage your cloud costs", and it.. sticks..
m1keil
FinOps isn't even new (my definition of new is "once you have an O'Reilly book on the topic it is not new anymore").
aledalgrande
I had the same thought. Why do we need to keep coming up with weird names/acronyms/portmanteau
MortyWaves
Sadly it’s one of the only ways of getting through the thick skulls of dumb middle management that seem to always be leading software projects despite having no technical background.
If it’s not got a familiar marketing thing going on, they’ll refuse to acknowledge it even if their devs are practically begging for it.
aledalgrande
> dumb middle management that seem to always be leading software projects despite having no technical background
Or you change to a company with capable & technical middle management :)
It always blows my mind when people say "management does not have to be technical".
EfficientDude
What's Grafana? I see it mentioned here a lot, nowhere else though. Is it a YC property?
brunoarueira
Grafana is mostly knows for the most used interface to query Prometheus and create dashboards for collected metrics
MortyWaves
And more recently inventing and reinventing Prometheus alternatives. It’s a bit much trying to keep up.
YZF
Very popular open source dashboard/monitoring software: https://grafana.com/
Nothing to do with YC AFAIK.
rajin112
Observability tool
On my last job, the company was using NewRelic (for two environments we was using at the time) which had an ok cost and "suddenly" we'd been forced to use Datadog which costs way over for our budget and after the person responsible for the change and integration see the estimated high costs, started to cut everything possible to keep it low. So, our tools degraded and we wasn't able to test things on staging and collect metrics like we was when using NewRelic. FinOps is certainly a good approach, but we need it from the start!