Jared Siegel argues vendor AI dashboards hide the data engineers need

Open the usage page on your paid AI subscription and you will see a bar chart that tells you how much work is happening. Jared Siegel, an engineer who posted on his blog this weekend, says that chart is a distraction and that nobody paying for a model should settle for it as proof of anything. The post reached 182 points on Hacker News.

The provider is selling you inference. What they show you is a picture of the inference happening, framed as proof of work. Siegel wants to answer other questions. Why did a request cost what it did. Why did a cache miss. Which of three agents using the same key spent half the budget. He posts a screenshot of the dashboard he was shown and writes "I can't make a decision with this bullshit" twice in a row. There are OpenTelemetry export options, he writes, which is some help, and then lists five reasons that still are not enough.

The five reasons a dashboard will not cover

A proxy you run yourself gives you one endpoint for every provider and every model, Siegel writes. That alone changes the shape of the problem. Every call carries a timestamp, a key, a model name, a client identifier and a status code, written to a log you own and can query. Second, you can hand out virtual API keys per project, per agent and per harness, which stops you from spreading one main key across every tool you own. Third, the infrastructure is yours. Fourth, you can switch provider or model in minutes instead of standing a new setup up for the third time. Fifth, the dashboard a vendor shows you is their product, so what it exposes can change when they decide to change it.

Siegel is deliberate that he will not recommend a specific proxy. The point is that you run it yourself somewhere you control, and modify it when you need to. His own has custom MCP server endpoints and extra features he put in by hand.

What a usable view looks like

A second screenshot shows what Siegel means. A time series by request, each row tagged with the key that made the call, the model that answered and the result status. When he sees a cache miss during a coding session, he writes, he can trace it to a reason in twenty seconds by hand. The proxy exposes an API, so the same lookup can be automated, though he has not done it himself.

The deeper point sits in his last paragraphs. He writes that he runs his own proxy at home because learning this way is enjoyable, and at work because he is not doing his job if he does not. "I may never fully understand how the fuck a large language model works well enough to white board it in an interview. But I can wrassle it down and make sure it's instrumented while serving inference."

For a team releasing an agent or a model-backed feature to real users, the operational question is the same whether you accept the argument or not. A dashboard the vendor controls will not tell you who spent money on what, which agent is calling itself in a loop that will not stop, or why your cache hit rate dropped at 3am. One proxy sitting in front of every call is a small amount of code, and the one place every one of those questions can be answered without asking the provider.

Source

If you don't run your own LLM proxy, how the fuck do you even know whats going on?, Jared Siegel. Hacker News discussion.