MCPJam launches a paid testing and evals platform for MCP servers with Swarms, User Testing and CI/CD

MCPJam launched a paid platform on 17 September for testing, evaluating and running CI on MCP servers before they reach production. It adds four capabilities on top of the company's open-source Inspector: Swarms, which simulates many users against a server; User Testing, which records real chat sessions for review; Evals, which grades tool calls and responses; and CI/CD, which runs the whole set on every commit. The platform launched today on Product Hunt.

What each part does

Swarms lets a team define a population of simulated users, each with a persona and a set of goals, and runs them against an MCP server in parallel. The purpose is to catch tool calls that fail on prompts the developer never tried, not the ten scripted queries in the demo. User Testing captures real chats against the same server, so a team can review what a live user actually asked for and where the server got it wrong. Evals runs graders against those transcripts, so a change to a prompt or a tool schema can be compared against a labelled corpus rather than by eye. CI/CD runs the same set on every pull request.

Where this sits

The company also maintains the MCPJam Inspector, an open-source project at 2,207 stars on GitHub with active pushes as of 17 September. Inspector is the local tool for chatting with an MCP server, inspecting its tools, prompts and resources, and stepping through its calls. The paid platform is the hosted version of the same work: the tests you ran on your laptop against one server, run continuously against every branch by MCPJam's servers.

MCPJam says on its homepage that 106,000 developers and more than 300 enterprises use MCPJam to see how their servers behave in apps such as ChatGPT and Claude. That is the company's own count and there is no independent audit of it. The Inspector star number is verifiable on GitHub.

Why anyone would buy it

A working MCP server has a testing problem that a working REST API does not. A tool call is text a model chose, so a change to the prompt, the tool name or the tool description can silently change which calls fire. The demo may pass while a real user's chat, phrased differently, sends the wrong tool with the wrong arguments and never surfaces a stack trace. A team that wants to answer "did that change to the prompt make anything worse" has to record enough real prompts and grade them, which is what Evals and User Testing are for. Whether a team pays for a platform to do it or writes the same fixtures in a repository is a build-or-buy call, and this launch is the paid option.

Source

• MCPJam, Swarms, User Testing, Evals, and CI/CD for MCP servers.
• MCPJam Inspector, github.com/MCPJam/inspector.
• Product Hunt, MCPJam, 17 September 2026.