Skip to content(if available)orjump to list(if available)

Show HN: A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)

Show HN: A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)

4 comments

·June 29, 2025

I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp).

It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency

Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test

The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including the growing number of third-party “proxy” or “cheap LLM API” services.

It supports: • OpenAI-compatible APIs (official + proxies) • Claude (via Anthropic) • Local endpoints (custom/self-hosted)

You can also self-host it with docker-compose. Config is clean, adding a new provider only requires a simple plugin-style addition.

Would love feedback, PRs, or even test reports from APIs you’re using. Especially interested in how some lesser-known services compare.

Visit

swyx

idk what it is but buying that domain made it seem more commercial and therefore less trustworthy. also most people prob want to just use artificialanalysis' numbers rather than self run benchmarks (but this is ok if want to run your own)

mdhb

In what universe is a post created by a new account with zero comments and a grand total of 2 votes over the course of 2 hours doing on the front page?

null

[deleted]

iRomain

LLM