AI Research: Agentic Research and Synthesis

Status: built, with a limitation serious enough to lead with. We generated 190 structured role-knowledge documents. A substantial portion were produced without live web grounding, and are therefore recall rather than research. We label them that way internally and here.

What we tried#

Generate a deep knowledge document for each role in an internal AI organisation — the frameworks that role uses, the metrics it is judged on, its failure modes, and decision rules it can act on. Roughly 190 documents, produced in waves, resumable, only ever picking work not yet done.

The mechanics worked. Generation at that scale is a solved problem: batch it, make it resumable, never repeat completed work, and write output to a stable location. None of that is where the difficulty lives.

Where it actually broke: grounding#

The generator requested live web grounding on every call. The grounding quota was not available, so the calls returned content anyway — assembled from the model's internalised knowledge, bounded by its training cutoff, and not verified against anything.

The output was indistinguishable in quality, structure and confidence from grounded research. That is the finding, and it is the one worth carrying to any similar project:

A research system without grounding does not fail. It produces confident, well-structured, plausible documents that nobody can distinguish from researched ones by reading them.

We record grounding status per document and surface it wherever the documents are used. Anything else would present recall as research, which in a knowledge system is not a quality issue but a correctness one.

The second finding: depth varies invisibly#

The documents were produced in two phases with different generators. The later set is substantially deeper — named frameworks, metrics with warnings about how they get gamed, failure modes with early signals, decision rules that can actually be applied. The earlier set is thinner.

Both look like completed documents. Depth is not visible without reading closely and knowing what depth would look like for that subject, which is precisely the knowledge a reader consulting them does not have.

The practical consequence: if you generate a corpus in phases, record which phase each item came from, and prioritise regeneration by the consequence of that item being shallow rather than by its position in the list.

What worked well#

Batch generation with resumption. Long runs fail partway. A generator that only ever picks unfinished work turns a failed run into a delay rather than a restart.

A shared corpus rather than per-item copies. Common context injected from one source means a correction reaches everything. The alternative is 190 places to fix a fact.

Structured output requirements. Asking for named sections — frameworks, metrics, failure modes, decision rules — produced far more usable documents than an open brief. The structure also makes gaps visible: an empty section is a signal, whereas a fluent paragraph is not.

Provenance recorded at generation time. Cheap when you do it, impossible afterwards.

What we would do differently#

Verify grounding before generating at scale, not after. We generated a corpus and then established that a portion of it was not what it appeared to be. The check costs one call.

Sample-review early. Twenty documents read properly after the first wave would have exposed the depth problem before it was replicated across a hundred more.

Define what "good" looks like for one item first. Without a reference, quality drifts and there is nothing to compare against.

Plan for regeneration. Any generated corpus will need parts redone. Design for replacing individual items from the start.

Where this fits against retrieval#

A generated knowledge corpus and a retrieval system solve different problems and are frequently confused.

Retrieval answers from documents that exist and can cite them. Generation produces documents that did not exist, and can cite nothing unless grounding was real. If the knowledge already exists in your organisation, retrieval is almost always the better tool — see RAG.

Generation earns its place where no document exists and a structured starting point is genuinely useful — provided everyone downstream knows what they are reading.

FAQ#

Is this available as a service?#

No. It is an internal experiment. This page reports what we learned, not a capability you can buy.

Why publish a result that is partly negative?#

Because it is the useful part. That an ungrounded research system produces confident, plausible, unverifiable output at scale is a finding worth more than a description of the parts that worked — and it is a mistake that is easy to repeat.

How can you tell recalled content from researched content?#

By reading, largely you cannot — which is the problem. You can tell by recording provenance at generation time, by spot-checking specific claims against sources, and by noticing an absence of anything time-specific. Anything recent tends to be missing rather than wrong.

Does a bigger or newer model fix this?#

No. A better model produces better recall, which raises the quality of the unverifiable output without making it verifiable. Grounding is an architecture question, not a model question.

What is the corpus actually used for?#

Internally, as role context for an AI organisation — see AI PMO. It is not published as research, and it is not what powers anything customer-facing.

Would you generate a corpus this way again?#

Yes, with grounding verified first, a reference document defining quality, and a sample review after the first wave. The approach is sound; we ran it in the wrong order.

How does this relate to your Research Center?#

Directly opposed in method. The Research Center reports carry sourced, dated figures and state where sources disagree. This experiment is what happens without that discipline, which is exactly why the two are kept separate.

Related Articles#

Compare with RAG for answering from documents that exist, the Research Center for sourced work, and AI PMO for where this corpus is used.

What else is coming for AI Research