Agent or script? A decision framework for AI in test automation

The single most common mistake we see in AI-assisted testing is not a bad prompt or a poorly tuned retriever. It is reaching for an agent when a deterministic script would have been cheaper, faster, more reliable and easier to debug.

This matters commercially. An agent loop that costs a fraction of a cent per invocation looks free in a demo and looks very different multiplied across a nightly regression run. It matters technically too: every agent you add is a source of non-determinism inside a system whose entire purpose is to detect change.

Start with the question the task is actually asking

A useful first filter: does the task require a decision, or does it require an action?

Clicking through a checkout flow is an action. The steps are known, the order is fixed, and the only uncertainty is whether the application behaves. That is a script. Wrapping an LLM around it adds latency, cost and a new failure mode without adding capability.

Deciding which forty test cases to generate from a two-hundred-page requirements document is a decision. The input is unstructured, the output space is large, and the quality of the result depends on judgement about coverage and risk. That is where a model earns its place.

If you can write the rule down completely, write the rule. Models are for the cases where you cannot.

The four-part test

When we teach this in the agents module, we work through four questions in order. If the answer to any of the first three is no, stop — you are looking at a script.

1. Is the input genuinely unstructured or variable?

If your input is a well-formed API payload or a stable DOM, you do not need a reasoning engine to parse it. Variability in the input is the primary justification for a model. A requirements document written differently by each business analyst qualifies. A JSON response with a published schema does not.

2. Does the task require choosing between paths?

Agents earn their cost when the next step depends on the result of the previous one in a way you cannot enumerate. Investigating a failure is a good example: the second thing you check depends entirely on what the first thing told you. Executing a known regression suite is not — the path is the same every time.

3. Is a slightly different answer each run acceptable?

This is the question teams skip, and it is the one that causes the most damage. Non-determinism is tolerable in generation, where a human reviews the output. It is corrosive in assertion, where the entire value of the test is that it fails for exactly one reason. An agent deciding whether a screen “looks correct” will eventually disagree with itself, and you will spend a week finding out why.

4. Does the value exceed the cost, at your volume?

Only now do you get to the arithmetic. Multiply the per-invocation cost by your actual run frequency, not your demo frequency. Add the latency to your pipeline's critical path. Then ask whether the alternative — a person spending two hours a sprint — is genuinely more expensive. Often it is. Sometimes it very much is not.

Where the line usually falls in quality engineering

In practice, we see a consistent split across enterprise engagements.

Good agent territory: generating test cases from requirements and user stories; triaging and clustering failures across a large run; investigating a flaky test's history to propose a root cause; converting natural-language steps into automation code; summarising the risk profile of a release for a non-technical stakeholder.

Bad agent territory: executing a known test path; asserting on values that have a correct answer; anything on the critical path of a merge gate where an extra thirty seconds of latency compounds across hundreds of runs a day; anything where an unexplained behaviour change would be attributed to the application rather than to the model.

The interesting middle: self-healing locators. This is genuinely a decision task — the element moved, and something must judge which candidate is the right one. But it is also the place where a wrong decision hides a real defect, which is why we teach healing with a mandatory heal history and a review loop rather than as a silent runtime fix.

The interview version of this question

Every AI engineering panel we have prepared candidates for asks some form of “when would you not use an agent?” It is a filter question. Candidates who have only followed tutorials answer with benefits. Candidates who have built systems answer with constraints: cost at volume, latency on the critical path, non-determinism in assertions, and debuggability when something goes wrong at 3am.

The second-order version follows immediately: “you have an agent doing X and it is too slow — what do you cut first?” The answer is almost always the same shape. Move the deterministic parts out of the loop, cache what is stable, reduce the number of tool round-trips, and only then consider a smaller model. If you have never had to do this on something real, it is difficult to fake.

A practical starting point

If your team is early in this, resist the platform instinct. Pick one task that clearly passes all four tests — test-case generation from requirements is usually the best candidate — and build it properly, with grounding and an evaluation dataset, before you build anything else. A single well-evaluated generator that your team trusts is worth considerably more than five agents nobody is sure about.