Fitness Report

• SCORED the gates passed and the run produced numbers.
• REFUSED a named gate stopped the run. The gate, its counts and its reason are shown in place of a score.
• could not check a probe did not apply to this connection. Reported as unknown, never as a pass.

Rows are listed by the first-try point estimate, but a row only takes a lower position when another interval sits entirely above it. Rows that nothing separates share a position and are marked as tied, and where two rows still overlap across positions the lower one says so on its own row. Servers scored under different runner models are never ranked against each other.

What this measures, and what it does not

Conformance asks whether the server speaks MCP correctly. Fitness Report asks whether an agent can actually get the job done with it.

The word in that first sentence belongs to the official suite at github.com/modelcontextprotocol/conformance, which is a different project with a different job. Run it too. This table is about task outcomes.

MCP Atlas asks which model is best at using tools. Fitness Report asks which server is worth using.

The runner model is pinned into every score record and printed on every ranked group. Two servers driven by different runner models are reported side by side but never ranked against each other, because the token accounting is not comparable across tokenizers.

Gate order

Gates run cheapest first and stop at the first failure, so a broken eval never spends model budget: free structural checks, an answer-leak string check and the zero-token protocol probes, then cheap null-model baselines (no tools, stubbed-empty, random valid arguments), then the paid construct gate where a reference agent with full information must reach the answer key at a rate of at least 0.90, and only then the full drive. An indeterminate baseline halts exactly like a failure. On any gate failure the report carries no score field at all, plus the failing gate with its counts, threshold, alpha, p-value and reason string. Those are the strings printed in the table.

With a median of about 2.5 tools across the open server roster, a suite below 8 tasks is refused as INSUFFICIENT_SURFACE rather than published as a two-task 100%.

Two documented divergences from evalgate

The gate math is ported from evalgate. It differs in exactly two places, and we own both here.

1. Construct denominator: oracle errors count

   evalgate silently shrinks the denominator when an oracle call throws, so three successes out of three completed calls reads as 100% even when forty calls errored. That is wrong for flaky remote servers, which is every remote server. Errors are gated on their own instead: errors / (n + errors) > 0.05 resolves the run to COMPROMISED. A compromised run is not a verdict about the server. It is a refusal to score, because the measurement did not complete.

2. Published PASS is symmetric with published FAIL

   A pass on this page additionally requires that the Wilson 95% lower bound sits at or above the threshold, or that the run's own n achieves the planned power against the detectable rate. Exact binomial power is sawtoothed in n, so clearing the planned sample size is not the same claim as being powered at it, and the run's actual n is what gets tested. Otherwise the verdict is downgraded to EXTEND and no pass is published. A small sample that happens to land above the line is not evidence, and publishing it while publishing failures at full rigour would be asymmetric.

Reading the interval

First-try success is shown as its Wilson 95% interval with the point estimate marked inside it. The interval is the finding, not the point. Listing order uses the point estimate, but the position is a dominance position: a row sits below another only when that other interval lies entirely above it. Rows nothing separates share a position and are marked tied, and any remaining overlap between neighbours is printed on the lower row, because at these sample sizes the order between two overlapping rows carries no information. The intervals are computed once, in the run record. This page only renders the stored numbers, so there is no second copy of the statistics to drift.

Protocol probes

The probe columns are deterministic, cost zero tokens and run before anything else, so a refused run still reports them. Spec currency is the revision the connection actually negotiated. Protocol hygiene counts the probe checks that passed, including whether the server accepts an impossible protocol version, whether a mismatched method or name header is rejected, whether the modern server discovery method is implemented, whether cache hints are present and correctly scoped, and what the deprecation surface looks like. A probe that does not apply to a connection is reported as unknown, never folded into the pass count.

Credential context

Every score is stamped with the credentials it was collected under: anonymous, free-key or owner-key. Tool surfaces differ by credential on real servers, so a number collected anonymously is not the same measurement as one collected with an owner key, and the two are labelled rather than merged.

Evidence

Every finding on this page links to the recorded session that justifies it. Each run publishes two tapes, the MCP wire plane and the agent plane, served with permissive cross-origin headers and an immutable cache, and the replay link opens both in the viewer in one merged timeline. Nothing is embedded or loaded until you click. There are no bare counts here: if a claim has no recording behind it, the row says so instead.

Prior art: MCPEval for model-driven task synthesis with machine-checkable predicates, evalgate for the validity gate math, mcp-tape for the tape format and pairing statistics. What is added here is the refusal, the signed replay behind every claim, and rewrite proposals tied to the recorded sessions they were derived from.