AI test generation without hallucinated steps: a grounding checklist

Ask a language model to write test cases for a feature and it will produce something plausible within seconds. That is the problem. Plausible test cases that reference a settings toggle which does not exist, a confirmation dialog that was removed two releases ago, or a validation rule someone imagined are worse than no test cases at all, because a human now has to verify each one against reality.

The engineering answer is grounding: constraining generation so that every step traces back to something in your actual source material. It is not one technique but a stack of them, and skipping any layer tends to reintroduce the failure.

Why models invent steps

Two mechanisms, and they need different fixes.

The first is retrieval failure. The model was not given the relevant requirement, so it filled the gap from its training distribution — which contains an enormous quantity of generic web application behaviour. Your login flow gets a “remember me” checkbox because most login flows have one.

The second is instruction ambiguity. The model was given the right context but was asked for “comprehensive test coverage”, which it correctly interprets as an invitation to be exhaustive rather than accurate. Told to produce twenty cases when the requirement supports eight, it will produce twenty.

The grounding stack

1. Ingest the right sources, and only those

Requirements, user stories, acceptance criteria, API contracts and existing test cases. Not general documentation, not marketing pages, not the wiki page last edited in 2021. Corpus discipline does more for output quality than any prompt change, and it is the cheapest intervention available.

2. Chunk along semantic boundaries

Fixed-size chunking splits an acceptance criterion across two chunks and retrieval returns half a rule. Chunk on document structure — story, criterion, endpoint — so a retrieved unit is a complete thought. Keep the parent document reference in metadata so you can expand context when the retrieved chunk is not sufficient on its own.

3. Retrieve, then verify retrieval

Before generating anything, evaluate the retrieval step in isolation. If the correct requirement is not in the top results for a query, no amount of prompt engineering downstream will save the output. This is the layer teams most often skip, and it is where most hallucination originates.

Build a small labelled set — fifty queries with known-correct source chunks — and measure whether retrieval finds them. It is a day of work and it makes every subsequent problem diagnosable.

4. Force structured output with source references

Do not ask for prose test cases. Ask for a schema: title, preconditions, steps, expected result, and — critically — a source reference field for each case, identifying the chunk it derives from. Validate against the schema and reject anything malformed.

The source reference does most of the work. A model that has to cite the requirement supporting each case invents far less, and when it does invent, the invention is immediately visible because the citation will not support the step. This single change reduces hallucinated steps more than any other intervention we apply.

5. Let a planner decide the count

Rather than asking for “comprehensive coverage”, run a planning step that reads the retrieved requirements and decides how many cases of which types are warranted — positive, negative, boundary, permission. Then generate against that plan. You get eight cases when eight is right, and you get a rationale for the distribution that a test lead can argue with.

6. Evaluate the generation, every time

Groundedness is measurable. For each generated case, check whether its steps are supported by the cited source. This can be automated with an LLM-as-judge scoring against the retrieved chunk, validated periodically against human review. Track the score over time and gate on it — if groundedness drops after a model or prompt change, you want to know before the output reaches a tester.

Untested test generation is an odd thing for a quality function to ship.

What good output looks like

A well-grounded generator is noticeably less impressive on first look. It produces fewer cases. It declines to cover areas where the requirements are silent, and it says so. It occasionally returns “the source material does not specify the behaviour for expired sessions” — which is not a failure but the most useful output it can give you, because it has found a genuine requirements gap.

Teams accustomed to the demo experience sometimes read this as underperformance. It is the opposite. Volume was never the constraint in test design; correctness and traceability were.

Where this sits in the program

Grounded generation is the second guided project, built immediately after the first LLM agent, because everything downstream depends on getting it right. You build the retrieval pipeline, design chunking strategies, implement source-referenced structured output, and construct the evaluation harness that measures whether retrieval is actually working.

It also happens to be the most transferable thing in the curriculum. The same stack — ingest, chunk, retrieve, verify retrieval, structure with citations, evaluate — is what any enterprise RAG application needs, whether it is generating test cases, answering policy questions or explaining defects. Learn it properly once on a problem you understand deeply, and it generalises everywhere.