Section 01
Research question
How often does an ordinary web search, of the kind an AI browsing agent would also run, return the answer to a research question outright — and what distinguishes the questions that survive it from the questions that do not?
Section 02
Method
- 01Ten benchmark questions were written first, each with one short, objectively verifiable answer.
- 02Every search run while building each benchmark was logged at the time: the query, where it was run, what came back, and whether the answer was exposed.
- 03A search was marked exposed when the answer was visible in the results without opening a source, and partially exposed when the result narrowed the field to a small candidate set.
- 04The counts on this page are taken directly from those published logs. Each benchmark page shows its own log, and the log can be exported.
Section 03
The dataset
| Benchmark | Domain | Searches logged | Answer exposed | Case study |
|---|---|---|---|---|
| AIBRC-001 (Ghana) | Public records and administrative history | 5 | Yes3 | Open |
| AIBRC-001 | Nineteenth-century communications history | 4 | Yes2 | Open |
| AIBRC-002 | Space exploration and mission records | 3 | Yes2 | Open |
| AIBRC-003 | Law reports and court records | 3 | Yes2 | Open |
| AIBRC-004 | United States Supreme Court records | 3 | Yes1 | Open |
| AIBRC-005 | International politics and treaty records | 4 | Yes2 | Open |
| AIBRC-006 | Computing and network history | 4 | Yes2 | Open |
| AIBRC-007 | Art history and museum records | 4 | Yes1 | Open |
| AIBRC-008 | Science and publication records | 4 | Yes2 | Open |
| AIBRC-009 | Economic history and central banking records | 5 | Yes2 | Open |
| Total | All ten domains | 39 | 19 (49%) | |
Section 04
Findings
Finding 01
19 of 39 recorded searches (49%) returned the answer, or enough of it to guess, on the first page of results.
Finding 02
Every one of the ten benchmarks contained at least one query that shortened the work. No question in this set was fully resistant to a single well-phrased search.
Finding 03
The queries that gave the answer away were the ones quoting a unique identifier — an instrument number, a case name, a mission name. Identifiers are indexed, so they act as lookup keys rather than clues.
Finding 04
Queries built from a combination of ordinary attributes — a region, a year, a population figure — returned candidate sets rather than answers, and required an official source to resolve.
Finding 05
Where a question turned on a distinction rather than a fact — a signing date versus an entry-into-force date, a founding statute versus an opening day — search results routinely returned the wrong one of the pair with confidence.
Finding 06
The distinction cases were only settled by opening the issuing body's own record. Summaries agreed with each other and were still wrong about which date answered the question asked.
Section 05
What follows from it
For benchmark design
Unique identifiers belong in the answer key, not the question. A question survives contact with search when it requires assembling several ordinary attributes and then confirming them against an official record.
For research practice
The risk is not that search fails to answer; it is that it answers confidently with the adjacent fact. Distinction questions need the issuing body's own document before the answer is written down.
For anyone commissioning research
A finding is only as good as the record behind it. Ask which body issued the document, and ask to see the search log — including the searches that made the work easier.
Section 06
Limitations
- This is a self-collected sample of 39 searches on ten questions I wrote myself. It is not a controlled experiment and the questions are not a random sample of research tasks.
- The searches were run and judged by one researcher. Exposure was recorded by my own reading of the results page, not by a scored rubric.
- No AI system was tested or benchmarked here. This study measures what ordinary web search returned for my queries — it makes no claim about the performance of any particular AI model or product.
- Results pages change over time and by location, so re-running the same queries may not reproduce the same counts.
Section 07
Where the underlying evidence lives
Section 08
Work with me
If you need a question answered from the record rather than from summaries, send it over with any deadline and I will tell you what can and cannot be established.