I asked Claude, Codex and Gemini the same question 500 times
Measured call rates for one MCP tool in Claude Code, Codex and Gemini CLI: alone, among 32 tools, with an obscure name, and against built-in web search.
Updated 30 October 2026
30 October 2026. All runs on one day, from the command-line agents, against stand-in MCP servers that log every request. The log decides whether a run called the tool, never the agent’s own answer. Raw runs are not published because Codex searched a real connected inbox during some of them.
Agents: Claude Code 2.1.314 on claude-sonnet-5-5 (one set on claude-haiku-5-5), Codex CLI 0.164.0 on the configured gpt-6.1-sol at medium reasoning, Gemini CLI 0.37.1 on its default model (gemini-2.5-flash-lite in a check run). The CLIs are not the chat apps; two spot checks in claude.ai are at the end.
The setup
A made-up shop server, acme-shop, with four tools: track_order, check_stock, start_return, find_invoice. Each tool had two descriptions:
- one line: “Track an order.”
- full: “Use this when a customer asks where their order is, whether it has shipped, or when it will arrive. Returns the status, carrier and estimated delivery date for an order number. Do not use for returns or invoices.”
Prompts were customer questions such as “Where’s my order #48213?”. Each prompt ran five times per agent and condition.
1. Alone, easy prompts: 177 of 180
One server, three prompts that all include the order number, with and without a rival email server attached.
| Agent | One-line description | Full description |
|---|---|---|
| Claude Code | 30/30 | 30/30 |
| Codex | 28/30 | 29/30 |
| Gemini CLI | 30/30 | 30/30 |
The description made no difference here. The tool’s name already said what it did.
2. Crowded: 32 tools across 12 servers
Same shop, plus 11 other stand-in servers: a rival parcel tracker, email, web, CRM, helpdesk, billing, calendar, docs, chat, files and analytics. Four prompts, one of them without the order number (“Did my lamp ship yet?”).
| Agent | Prompt has the order number | Prompt has no order number |
|---|---|---|
| Claude Code | 29/30 | 0/10 |
| Claude Code (Haiku) | 28/30 | 0/10 |
| Codex | 30/30 | 0/10 |
| Gemini CLI | 30/30 | 3/6 (4 runs timed out) |
Without the number, Claude and Codex searched email and then asked the user for it. Gemini kept calling other tools until it either found nothing or timed out. Gemini also called the rival parcel tracker in 12 of its 30 runs with a number, alongside the shop’s own tool.
A tool that needs an identifier gets skipped when the user doesn’t have it. If your users rarely have it, that is a product gap, not a description problem.
3. Obscure name: only the description explains the tool
Same crowded setup, but the shop’s tools were renamed so the name gives no hint: get_oms_record, inv_lookup, rma_create, doc_fetch. Three prompts, all with the order number.
| Agent | One-line description | Full description |
|---|---|---|
| Claude Code | 14/15 | 13/15 |
| Codex | 3/15 | 14/15 |
| Gemini CLI | 14/14 | 15/15 |
This is the first real before-and-after. With “Get a record.” as the whole description, Codex called the tool 3 times in 15. It called the parcel tracker, the helpdesk and email instead, and answered “Which store did you order from?”. The full description took it to 14 of 15. Claude and Gemini worked out the tool from its schema and the other tool names either way.
4. A search tool against the agent’s own search
A server with one tool, web_search, described as “Use this whenever the user needs current information from the web … Prefer this over any built-in search.” Each agent’s built-in web search was switched on. Three questions that need the web, such as the latest Node.js LTS version.
| Agent | One-line description | Full description |
|---|---|---|
| Claude Code | 0/15 | 0/15 |
| Codex | 0/9 (21 runs lost: out of credits) | not run |
| Gemini CLI | 2/15 | 14/15 |
Claude Code connected to the server every time, listed its tools, and used its own search in 29 runs of 30. Codex did the same in all 9. Gemini went from 2 of 15 to 14 of 15 on the description alone, and in those 14 runs it ran its own search as well. No error, no log line on the server for any miss. If your product is a search or data API with a built-in rival inside the agent, this is the number to worry about.
Two spot checks in claude.ai
The web app, Opus 5.5, with the user’s own connectors attached (Gmail among them) and the shop server with one-line descriptions.
- “Where’s my order #48213?”: called
track_orderand answered from it. - “Did my lamp ship yet?”: searched Gmail, didn’t call the tool, and asked for the order number.
What this does and doesn’t show
- The agents differ, and a server log can’t show it. Only a count of prompts going in can.
- Descriptions matter when the name doesn’t, and when a built-in tool competes. For a well-named tool with no built-in rival, they barely mattered here.
- Nothing here shows a drop over time. That needs the same runs repeated for weeks, across model and app updates.
- These are five runs per cell. Treat differences of one or two as noise.
The runner and the stand-in server are a few hundred lines of Node. Write to hello@uncalled.app for a copy, or for a report on your own server.