A second AI assistant can be a useful reviewer. It can also agree with the first one with the confidence of two interns who copied the same answer. Give it a specific job: find what might be wrong and show the evidence.
The useful takeaway
Ask an AI reviewer to find failures against a written standard, not to declare the work “good.” Require sources for claims that would change a decision.
A realistic example
Example: the refund answer with a made-up exception
A support assistant drafts a reply about a customer refund. It says the customer qualifies for an exception to the 30-day policy.
What goes wrong
The exception sounds kind and plausible. It is not in the approved policy. A general “review this answer” agent praises the tone and misses the invented promise.
A better check
An AI reviewer receives the policy text and a rule: identify every promise in the draft, quote the policy passage that allows it, and mark unsupported promises for human review.
The flow
What gets checked, and when
Follow the numbered steps in order. Each answers a different question; one green check cannot stand in for the others.
- 01
Draft
Assistant writes the proposed reply.
Catches: Nothing yet; this is a candidate.
- 02
Extract
AI reviewer lists factual claims and promises.
Catches: Claims hidden inside polite wording.
- 03
Check
Reviewer pairs claims with policy passages; a person verifies support.
Catches: Invented policy exception.
- 04
Decide
A person approves, edits, or rejects.
Catches: Bad judgment that rules alone miss.
An AI reviewer is a role, not a new authority
An AI reviewer—sometimes called a quality-assurance (QA) agent—is simply an AI assistant given a separate review task. It may read a draft reply, compare a report with its sources, or inspect a software test. The separation helps it look for problems the drafter may overlook. It is not independent evidence; both assistants can share the same blind spot.
Give the reviewer written pass/fail rules it can apply. “Is this high quality?” invites an essay. “List every claim about price, eligibility, timing, and customer rights; show the approved document for each; mark anything unsupported” produces something a person can check.
- Give it the exact draft, software change, or report version to review.
- Give it the approved policy, source document, or written rule to compare against.
- Ask for a short list of claim, quoted evidence, concern, and proposed action.
- Set the limit: the assistant can recommend; it cannot send, publish, change a record, or approve its own finding.
Ask for sources when the answer would move money, rights, or trust
Not every sentence needs a citation. A brainstorming list or an invented training scenario can be labeled as such. Ask for a source when a statement would affect a purchase, a policy decision, a customer promise, a legal or safety obligation, or a number in a business case. Also ask when the claim could have changed recently—prices, product features, dates, and rules are moving targets.
A link is only a starting point. Open it. Does it say the claimed thing? Is it current? Is it the original source or someone repeating it? In the refund example, the approved policy wins over a confident summary. If the source does not resolve the case, say so and route it to the policy owner.
Test the reviewer with a planted mistake
Before trusting the reviewer, give it a harmless test draft with a deliberate error. Add a fake refund exception, a wrong date, or a number that does not add up. If it misses the error, revise the task or reduce the responsibility. If it catches only the planted mistake, that still does not prove it will catch every new one.
Keep a small set of review examples: ordinary cases, unusual cases, and a few bad answers that previously slipped through. Run the same set when the instructions, source collection, or AI model changes. This helps you see old errors returning without treating the assistant’s score as an objective measure of truth.
Escalate disagreement instead of forcing a verdict
If the drafting assistant and reviewer disagree, do not ask them to debate forever. Save the claim, source, and competing interpretations for the human owner. The owner may discover an outdated policy, an ambiguous customer case, or a source that was never approved. Those are process findings, not merely instruction problems.
This is also where humor helps: the agent can be the energetic detective, but it should never be the judge, jury, and refund department. Give it the magnifying glass, not the company credit card.
Try this
Build a reviewer for one risky claim type
- 1Pick one claim type, such as eligibility, price, or an external statistic.
- 2Write the approved source and the exact format in which the reviewer must report support or uncertainty.
- 3Plant one unsupported claim in a safe test draft and see whether the reviewer catches it.
You are ready when: A person can trace each finding to a source and can see exactly what the agent did not verify.
Sources and further reading
The examples above are constructed teaching scenarios. These primary sources support the testing methods and evaluation principles.
- NIST Generative AI Profile
Source history, output evaluation, known ground truth, and human oversight.
- Playwright: best practices
Separating what a test does from what it proves.