Evaluating MCP research quality

Purpose¶

Every page of this book makes behavioural claims: discovery stays within the UN System Data Commons corpus (Statistical scope), observations exist for countries and supranational geographies (Geographic scope), a facet is chosen by its id, research runs discovery → assessment → retrieval (research rules). The evaluation framework described here exists to make those claims testable. It is deliberately small, and it is not a general evaluation manual for Data Commons.

What we evaluate¶

Levels 1 to 3 run on every trunk push and on the daily schedule through make test-e2e. The invariants alone run without Docker: uv run pytest -m e2e tests/e2e/test_platform_invariants.py.

Benchmark catalog and expectation types¶

The catalog, tests/benchmark/catalog.json, records for each research question what a correct outcome looks like. A query carries an id, a category, the user_query a person would ask, a why naming what the query is meant to catch, a run block with the tool and arguments a client would derive from the question, a variables block, and a list of expectations. The variables block lists acceptable UN variables explicitly, or describes them by metadata (acceptable_metadata: population type and constraint values) where a whole family is acceptable, and lists rejected variables each with a reason: the near neighbour that would be a semantic error.

Every expectation carries a type, and the type says what a failure means:

A quality expectation that already failed when the catalog was validated carries a dated known_gap. A later run reports it as still failing or as fixed, never as a new regression. Two expectations apply to every query by default: discovery succeeds, and every candidate is a UN variable or UN topic.

The checks an expectation can name, and the layer that can run them:

The catalog is validated offline by tests/benchmark/test_catalog.py on every commit: ids, categories, check names and their fields, the rule that acceptable variables are UN variables, and that a rejected variable outside the corpus appears only in the statistical-scope category. The header records the server version and date the expectations were last confirmed against.

Running the benchmark¶

make benchmark runs every query against MCP_URL (the production server by default) through tests/benchmark/test_discovery.py, one test per query, and writes _benchmark/results.json and _benchmark/report.md. The runner is plugins/benchmark.py, stdlib only. The pipeline runs it on the daily schedule, and on demand from the web UI, as a non-gating job that keeps both files as artifacts. Nothing generated is committed.

Every evaluated expectation ends in one status, and the expectation’s type decides what the status means for the run:

Rephrasing robustness¶

A query may carry two or three rephrasings: alternative wordings of the same concept that a client might send, run as fresh requests with the same places and nothing carried over from one phrasing to the next. For each such query the report shows how many phrasings found an acceptable variable within the top N, the rank of the first acceptable candidate per phrasing, how many phrasings ended with an acceptable variable found, with candidates but none acceptable, or with no candidates at all, whether the top candidate agreed across phrasings, the overlap of the first five candidates with the canonical phrasing, and for how many phrasings a rejected variable outranked the acceptable one. Where an expectation is a known gap, the report says whether the gap is stable across phrasings, fixed in all of them, or phrasing-sensitive. This is descriptive evidence, classified as quality, never a contract: exact candidate order need not be identical, the question is whether the intended UN statistical concept stays discoverable and the semantic traps stay controlled under natural changes of wording. Thresholds, if any, come after a baseline has been observed.

Endpoint comparison¶

The same catalog can be run independently against production and a candidate endpoint (MCP_URL), each run writing its own results.json, and the two artifacts compared without any further call:

make benchmark BENCHMARK_DIR=_benchmark/baseline
MCP_URL=https://candidate/mcp make benchmark BENCHMARK_DIR=_benchmark/candidate
make benchmark-compare BASELINE=_benchmark/baseline/results.json CANDIDATE=_benchmark/candidate/results.json

The comparison classifies each query as unchanged, improved, regressed, mixed, changed or not comparable. Direction is stated only where it is unambiguous: a contract that flips, an acceptable variable that crosses the top-N line, a semantic trap that starts or stops outranking the acceptable candidate, a known gap that closes or reopens. Everything else, a different top candidate among acceptable ones, a reshuffled list, a current-state difference, a duration, is listed for a maintainer to judge. It is a regression-analysis tool, not a competition score: there is no winner and no total. The pipeline offers it as a manual job that takes CANDIDATE_MCP_URL; two endpoints are never run on the schedule.

The report opens with the counts by type and status, then an “attention required” section: contract failures, execution errors, known gaps now fixed, current-state changes, and quality expectations that newly fail. Known gaps still open are listed separately so they read as standing findings, not news. The two gaps found while validating the seed, the under-18 count ranking below percentage variables and the origin-based refugee rate outranking the host-side UNHCR variables, stay visible there until the search changes. They are search-quality observations, not contractual behaviour and not launch blockers.

Query categories¶

Reserved for later phases: temporal, exploratory, data-availability, multi-entity, adversarial and playbook-routing.

Findings recorded while validating the catalog, all dated 2026-09-11 and kept in the catalog header as search-quality observations rather than user rules:

• The embeddings search has no similarity threshold, so a query about a concept the corpus does not hold still returns candidates. Search results alone are therefore not sufficient to establish that the UN System Data Commons lacks relevant data: a genuine zero-state is an agent-level conclusion reached after discovery and qualification, and the runner never evaluates a no_data outcome from a candidate list.

• The corpus has no host-total refugee variable: UNHCR publishes one variable per country of origin, observed about the host, while the SDG rate of refugees per 100,000 by country of origin ranks first for every phrasing of “refugees in Uganda” tried. That is the kind of near neighbour the semantic category exists for.

• The bare concept population ranks census-registration and median-age variables above the population count for France and for the SIDS grouping, while total population ranks the count first. Recorded for the search owners; the book does not tell users which words to use.

• A place name inside the query text degrades discovery: “how many people live in Rwanda” surfaced a UNHCR variable and “how many people live in France” returned no candidates. This one did become guidance, as the concept-and-place rule on the tools page, because the tools offer dedicated place arguments for exactly this.

Platform invariants¶

tests/e2e/test_platform_invariants.py asserts the following against the live server. Each test is marked contract or current_state. In the pipeline the two markers run as two jobs on every trunk push and on the daily schedule: the contract tests gate a release; the current_state tests, which deliberately record known defects of the hosted platform, run in a separate non-gating job, so a change in hosted behaviour is visible the day it happens without blocking a book release.

• Discovery returns only UN variables and UN topics, for concepts the wider graph covers heavily, with and without places (contract).

• Shared places resolve to their common identifiers: a country, a UN region and a geographic grouping (contract).

• The countries within a region carry observations (current_state).

• Places below country level resolve but carry no observations (current_state).

• Representative UN variables carry only facets whose provenance is under undata/p/ with the United Nations as source (contract).

• A variable can carry several facets of one UN provenance (current_state).

• source_override takes a facet id, echoed back as sourceId (contract); a provenance DCID passed there selects nothing (current_state).

• A place-aware search for total population by sex returns UNICEF’s terminal topic Total population, by Sex with the female and male variables as its members (current_state, because it depends on search ranking; the pairing of that topic with its peer group is asserted structurally at the REST edge).

• date="range" with both bounds returns only observations inside the bounds (contract); bounds supplied without date="range" currently come back as the single latest value (current_state, a known defect: when the server applies or rejects them instead, the Date modes warning is softened).

• A World-level get_child_observations answer lists every observed place in entityMetadata with a value for each (contract); today the names are blank for the tail of that list (current_state, a known defect: when every place is named, the notes on the tools page and Walkthrough B are dropped).

• A DCID in a search tool’s place field fails with a server error while the same search by name resolves (current_state, a known backend error-handling issue; the third-party wording of the error is not asserted).

Deliberately not asserted: whether a retrieval tool serves a variable from outside the corpus when handed its DCID. Today it does; the intended contract is that it must not. Asserting the present behaviour would freeze the wrong thing, so that check waits for the runtime hardening it belongs to.

The REST edge¶

Inspecting the graph with the REST API documents five recipes against the deployment’s V2 node endpoint. The book never calls that endpoint at build: the examples on the page are static and dated, and tests/e2e/test_rest_edge.py keeps them honest against the live deployment (uv run pytest -m e2e tests/e2e/test_rest_edge.py, stdlib only). The five documented traversals — the UNICEF population family through its series node and its groups, the Age vocabulary behind a constraint, Rwanda’s containment and the LDC grouping’s countries, the UN topic hierarchy from its roots, and the peer group behind UNICEF’s Total population, by Sex with the terminal topic it is paired with, in both directions — are contract: a failure means the page is wrong. The page’s boundary warning rests on safety evidence marked current_state: the same API answers for a base-graph variable, and REST indicator resolution returns base-graph candidates unless scoped. If that evidence stops holding because the deployment becomes UN-scoped by default, the test asks for the warning to be reviewed and softened; it does not fail because contamination disappeared. Pagination behaviour is deliberately not asserted.

Future evaluation layers¶

• Agent trajectories (deferred). The evaluation cycle stopped after the benchmark and the endpoint comparison; no agent runner exists and none is planned until a separate decision has been taken on model provider, credentials, cost and runtime. It should be revisited after the retrieval-boundary hardening is implemented, when there is a contract to test rather than a documentation rule to observe. What such a layer would test: whether an agent carries a discovered undata/ variable into retrieval rather than an identifier from elsewhere; whether the source it names is the provenance of the facet it actually retrieved; whether it performs discovery before concluding that data are absent, with no tool call warranted only for adversarial or out-of-scope requests; and whether it follows the appropriate playbook routing. Direct factual claims should be deterministically traceable to MCP tool results where possible, and derived claims reproducible from the retrieved observations. Which playbooks the client read would be telemetry first; whether reading the right one before the first call should become a contract is a question for after real clients have been observed, since clients ingest and cache resources differently.

Not planned: ranking formulas, browser automation of the website, or judge scoring as the primary signal.