Open Source LLMOps Stack
4 comments
·February 26, 2025mrbonner
It was just Ops, then DevOps, MLOps, AIOps and now LLMOps. I lost count of how many promote cases I reviewed over the last 10 years with those "Ops" as the sole reason to promote.
keyle
I read LLMOps for the first time today and I couldn't stop laughing. Glad someone else picks on it.
We're getting closer and closer to my idyllic goal: RoLLMOps! [1]
remram
Oh man, they went crazier than that with "secops", "itops", "finops", "dataops". Every Foozler person is now "doing FooOps", every dev is "devops".
Titles have never been about clarity or distinctiveness but how cool that sounds, and apparently this is the fashion. Nevermind that "ops" used to stand for something. Me wants shiny word.
Some background: I work on Langfuse and we've been collaborating with LiteLLM.
(LiteLLM is a Python library and proxy/gateway that handles cost management, virtual keys, caching, and rate-limiting for OpenAI or other LLM APIs. Langfuse manages LLM tracing, evaluation, prompt management, and experiments.)
We’ve each been building our open-source projects since early 2023 and learned that many devs and especially platform teams use the two together, so we created an integrated “OSS LLMOps stack.”
This is a fully self-hostable, technology-agnostic setup that lets you (1) Use LLMs via a standardized interface without adding complexity to the application; (2) Keep LLM Tracing, Evaluation, Prompt Management in-house for compliance; (3) Track cost and usage via a single interface, create virtual API keys for attribution of costs
It also enables direct transfer of LLM traces from the LiteLLM proxy to Langfuse. This simplifies the rollout of LLMOps practices (observability and evaluations) across multiple projects—you don't need to instrument all applications.
Additionally, the LiteLLM proxy can fetch and cache prompts from Langfuse's prompt management system, using them as templates for requests made through the proxy.
Both of these workflows can function without the integration, but are easier to manage with it!
We’d love your feedback!