You cannot make every generated sentence identical. You can make the decision to accept, reject, or review it much more consistent. That is the practical meaning of turning generative work into a repeatable process.
The useful takeaway
Define fixed acceptance rules around variable AI output; measure failures and keep a person in charge of exceptions.
A realistic example
Example: a weekly market brief with one suspicious number
An assistant drafts a short market brief from approved public sources. It says a provider has 10,000 paying customers and links to the provider’s update.
What goes wrong
The link exists, but the update says 10,000 trial sign-ups—not paying customers. A machine can find the citation and still miss the change in meaning. The recommendation built on that claim now overreaches the evidence.
A better check
Automatic checks require a date, source link, and reproducible calculations. An editor opens the source, corrects “paying customers” to “trial sign-ups,” and decides whether the recommendation still follows.
The flow
What gets checked, and when
Follow the numbered steps in order. Each answers a different question; one green check cannot stand in for the others.
- 01
Shape
Require the same fields in every brief.
Catches: Missing date, claim, or source.
- 02
Compute
Recalculate numbers from named inputs.
Catches: Confident arithmetic error.
- 03
Read
An editor checks whether cited passages really support the claims.
Catches: Trial sign-ups misreported as paying customers.
- 04
Release
Editor signs off on interpretation.
Catches: A plausible but unjustified conclusion.
Keep creative drafting; make acceptance less mysterious
Generative systems can produce different wording from the same instruction. A deterministic check, by contrast, gives the same pass or fail result for the same inputs and rule. Examples include “a publication date is present,” “each figure has a source link,” and “the quoted percentage matches a calculation from verified inputs.” These checks keep the basic contract from drifting. They do not establish that a linked passage supports the claim.
For the market brief, define a small output shape: claim, figure, date, source link, source passage, and editorial interpretation. If a field is missing, the draft waits. A person still reads the passage and its context. In the example, a link to 10,000 trial sign-ups is not evidence of 10,000 paying customers.
- Use a fixed structure for facts and sources before formatting the final prose.
- Calculate numbers in ordinary code or a spreadsheet from stored inputs; do not ask a language model to be the calculator of record.
- Keep a separate human decision for tone, fairness, and what the evidence actually means.
Build a small set of known cases
Start with five examples with expected outcomes: a clean brief, a missing source, a percentage calculated from the wrong group, a stale figure, and a case where the evidence is genuinely mixed. Run them whenever the instructions, model, source set, or acceptance rule changes. Repeat each case several times because a generative answer can vary between runs. Add new cases from real misses; five are a starting set, not proof the system is safe.
Include a case the model has not seen while you were writing its instructions. If you only test the examples you used to teach it, you may be measuring memory. Put another way: a student who knows the answer sheet is not necessarily ready to run the newsroom.
Measure misses, not just passes
Track what was caught before release, what escaped after release, and what humans had to correct. If a new rule blocks half of the good drafts, it may be too blunt. If the rule passes nearly everything but customers keep finding errors, it is guarding the wrong failure. Review a sample of passes as well as failures.
A practical first scorecard has four lines: total drafts, drafts stopped by automatic checks, drafts changed by reviewers, and errors found after publication. Add a note about the type of error. The goal is not to celebrate a high pass rate; it is to make the process safer without burying the team in rework.
Change one part at a time
When you change a model, a prompt, a source feed, and the scoring rule at once, a better result tells you very little about why it improved. Hold the test cases steady, change one element, compare the failures, and keep the prior version available until the new one earns its place.
The same rule applies to AI-built software. A stable browser test, a check of the saved record, and a clear human review help a team change the implementation without silently changing the promise to users. The final product may be delightfully flexible; the release decision should be boring in the best possible way.
Try this
Turn one draft into a small release contract
- 1Pick one repeatable output: a brief, support reply, proposal, or software change.
- 2Write three checks a machine can apply exactly and one judgment a person must still own.
- 3Start with five known cases, including two deliberate failures. Run each several times, then add cases from errors you actually find.
You are ready when: You can state what passes automatically, what needs review, and what would stop release.
Sources and further reading
The examples above are constructed teaching scenarios. These primary sources support the testing methods and evaluation principles.
- NIST Generative AI Profile
Evaluation against known ground truth, output quality, and human oversight.
- Playwright: test assertions
Repeatable checks for expected browser behavior.