How to test an MCP server

Most MCP testing stops once the server responds correctly. That proves it works when called. It says nothing about whether an agent will call it, which is the part your users depend on. As Braintrust’s guide puts it, a server can pass unit and protocol tests yet still fail when an agent uses it.

Test four layers. The first two run on every commit. The last two need a real model, so they cost money and run less often.

Layer	Question	Needs a model	When
1. Handlers	Does each tool do its job?	No	Every commit
2. Protocol	Does it speak MCP correctly?	No	Every commit
3. Real agent	Does an agent connect and call it?	Yes	Before release
4. Selection evals	How often is it chosen, across prompts and agents?	Yes	Before release, then on a schedule

Layer 1: handlers

Test your tool functions like any other code, with no MCP involved. Cover bad input and upstream failures too. The spec treats execution errors (isError: true) as feedback the model can use to correct itself, so an error message is part of your interface. Assert on its wording.

Layer 2: protocol

The official MCP Inspector calls your server directly, with no model in the loop. Use the web UI while developing:

npx @modelcontextprotocol/inspector node build/index.js

Use the CLI in CI. It has meaningful exit codes: 4 means unreachable, 5 means a tool error, and 6 means --strict found a schema problem.

npx @modelcontextprotocol/inspector --cli node build/index.js --method tools/list --strict
npx @modelcontextprotocol/inspector --cli node build/index.js \
  --method tools/call --tool-name track_order --tool-arg order_id=48213

For spec compliance, the project also has a conformance suite:

npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp

The latest spec revision, 2026-07-28, removes the initialize handshake and carries client details in each request’s _meta. Servers may support both eras, and it is worth testing both.

Layer 3: a real agent

Now put a model in the loop. Claude Code can run headlessly and load only the servers you give it:

claude -p "Where's my order #48213?" \
  --mcp-config ./mcp.json --strict-mcp-config \
  --allowedTools "mcp__acme-shop__*" \
  --output-format stream-json --verbose

Watch for three things:

• Pre-approve your tools. Without permission, Claude sees the tools but can’t call them, and the run looks like a miss. Denials appear in permission_denials in the result.
• Check the connection first. The system/init event lists each server’s status and any mcp_server_errors. A failed server is a different problem from a skipped tool.
• Count calls by looking for tool_use blocks whose name starts with mcp__. Better still, count requests in your own server log, because that works for every agent.

Codex and Gemini CLI can run the same way. I use codex exec --json and gemini -p "..." -o json, with the server defined in each tool’s own config. OpenAI’s docs also describe testing in ChatGPT: after you change a tool, refresh the connection and start a new conversation.

Layer 4: selection evals

One successful run proves your tool can be called. An eval tells you how often it is, and that number is what you tune. The closest thing to an official recipe is OpenAI’s golden prompt set: direct prompts that name your product, indirect prompts that describe the outcome without naming the tool, and negative prompts that should go to another tool or to none.

Record the tool selected, the arguments, and whether it was the right call. Then track two numbers: precision, meaning the right tool ran, and recall, meaning your tool ran when it should have. Change one description at a time and rerun.

Run each prompt several times, because agents aren’t deterministic. Test with other tools connected too. Your tool never runs alone in real use, and Anthropic’s docs say tool picking degrades past 30 to 50 tools.

Descriptions are usually where the gains are. Anthropic calls them by far the most important factor in tool performance, and Neon reports going from about 60% to 100% on its evals by changing the descriptions of two tools.

Tools that can run these:

• MCPJam Inspector is open source under Apache-2.0. Its test cases assert on calls, such as “Tool was never called” or “First tool called was”.
• Braintrust is a commercial platform for datasets, scoring and CI gating.
• A short script works too. Run the prompts in a loop and count requests in your log.

What testing before release can’t catch

All four layers describe your server on the day you ran them. The day after, a model update or an agent release can change the result, and so can a new tool the user installs next to yours, with no release from you. OpenAI’s guidance says the same thing about metadata drift: replay your prompts when things change, and review analytics weekly.

So keep layer 4 running on a schedule, per agent, and watch for drops. I’m building Uncalled to do that run every day. It isn’t live yet, so for now I run it by hand for a few teams, and you can ask for one at hello@uncalled.app.

If your number is already low, why Claude isn’t calling your MCP tool goes through the causes in the order to check them.