Run a two-week pilot that can tell you something

Start with the same question for old and new work: what is a finished acceptable result? A polished draft may still omit the one term the customer needed. A fast summary may send the reviewer back through every source. Define the pass condition and the costly errors before anyone watches a model perform.

Collect a small set of real-shaped cases: common, messy, stale-source, missing-information, and high-consequence examples. Replace confidential details where necessary, but preserve the shape of the work. Record which source is authoritative, what a correct result contains, and what should happen when the tool cannot know. The right answer to a missing contract clause may be “ask the account owner,” not an invented clause.

In week one, keep the output internal and ask reviewers to record changes. In week two, let the revised process handle a limited real queue only after the owner accepts the safety checks. Neither week needs an organization-wide license. The team needs enough access to perform the task and enough comparison cases to see whether its apparent speed survives review.

If preparation falls from 30 minutes to 10 but review rises from 5 minutes to 20, the gain is smaller than the demo suggests. Record elapsed time and hands-on time separately. Count the tasks that pass on the first review, not only the drafts created. For customer-facing work, track whether customers come back because the first answer was incomplete.

Do not translate every released hour into a payroll saving. If a manager uses freed time for higher-value customer work, that may be valuable capacity; cash savings require an actual spending or staffing change. Keep the categories apart in the business case. Include license cost, training, expert review, integration, and error repair.

Ask reviewers what changed in their work. Did AI remove tedious preparation or create a new fact-checking burden? A tool that helps a new employee may have little effect on an expert; some tasks may even degrade if the expert follows a plausible but inferior suggestion. Segment the test by role and task type instead of averaging unlike work together.

Name one error that triggers an immediate pause: a wrong customer record, unsupported contractual claim, private information in the wrong workspace, or a result no reviewer can verify. Record who can stop the pilot and who investigates. No amount of time saved should quietly override a high-consequence failure.

At day 14, make a plain decision. Expand only if quality holds and the operating owner will continue to sponsor the workflow. Redesign if the problem is a fixable source, permission, or review step. Stop if the value disappears after correction or the tool cannot be made safe for the task. “Everyone liked it” is feedback; it is not the decision.