Why Expert Human-in-the-Loop Review of AI Needs Independent Reference Points

Should expert human-in-the-loop review be enough for high-risk AI?

The Situation

Every time someone uses an AI-generated answer, they've already decided to trust it. KPMG and the University of Melbourne surveyed more than 48,000 people across 47 countries, and among employees using AI at work, 66% said they rely on its output without evaluating its accuracy. Most firms try to address that risk by holding employees directly accountable for their own output, and for high-risk uses in the business, requiring a “human in the loop”. For critical business processes using AI such as credit decisions, marketing efforts steering millions in investment, and job applicant evaluations, that human review is the essential control, and it rests on judgment nobody else can see.

The Exposure

Many firms are getting comfortable with employees using AI to help craft output, as long as the producer takes responsibility for the final content before it goes anywhere. That responsibility typically includes a minimum of reviewing for appropriateness and fact checking. But unless there’s a process for objective verification, significant errors can escape. Deloitte delivered a report to the Australian government with fabricated citations and references, then refunded part of its fee after disclosing it had used generative AI in the production. Researchers at the University of Lille, testing how well chatbots judge their own reliability, found they consistently reported 90-100% confidence on medical licensing exam questions, including the ones they got wrong. Most firms just assume a human in the loop covers it, but if the human is checking the tool against itself, it doesn't.

The Judgment Call

Firms treat expert sign-off as their human-in-the-loop control, but it’s not enough to just review and then decide. External reference points need to be systematically incorporated, and documented, to assist the expert. The review often happens by querying the AI, but with that approach every response is directly shaped by the conversation immediately before it, and the tool pulls the reviewer into its own reality. Stanford researchers found models changed their answers under user pushback 58% of the time. The fix is to anchor the expert with reference points independent of the tool output, such as a system of record, known-good reference documents, or a colleague who hasn't iterated with the AI. Before reviewing the output, the expert should document the checkpoints a trusted answer should hit, then record which checkpoints the output actually met. Auditors already work this way, setting an expectation before they look at the result. The expert's judgment stays in charge, but now it’s auditable, grounded, and defensible to an examiner. Reserve it for workflows above your risk tolerance to manage the overhead.

• Risk: The extra checks slow the expert, reducing the volume they can work through.

• Benefit: Sign-offs become auditable beyond just the expert’s opinion, and the documentation protects them during a challenge.

This Week’s Action

• What to do: Pick your highest-risk AI workflow already in use. Have the expert write down the checkpoints a trusted answer should hit and the external reference for each, then check the output against them.

• Who to involve: The expert in the process today, and your CRO or CAE to confirm the workflow’s critical risk rating within your ERM framework.

• What outcome to achieve: A running log with one row per decision, showing which of the checkpoints each output hit.

• Time required: 60 minutes to draft the checkpoints, 30 minutes with your CRO or CAE to confirm the risk rating and 15 minutes to log the first decision. Total: 105 minutes.

Artifact

Answer for your highest-risk AI workflow.

1. Tolerance: Is the workflow rated in your ERM framework, with the cost of a wrong output stated in business terms?
   ☐ Not addressed     ☐ Partially addressed     ☐ Fully documented and current

2. Confidence: Is the confidence level needed to act on the output written down?
   ☐ Not addressed     ☐ Partially addressed     ☐ Fully documented and current

3. Checkpoints: Has the expert listed what a trusted answer must show, before anyone runs the prompt?
   ☐ Not addressed     ☐ Partially addressed     ☐ Fully documented and current

4. External references: Does each checkpoint point to a source independent of the AI, such as a system of record, an approved policy, or an independent reviewer?
   ☐ Not addressed     ☐ Partially addressed     ☐ Fully documented and current

5. First answer: Is the AI's original output saved before anyone challenges it?
   ☐ Not addressed     ☐ Partially addressed     ☐ Fully documented and current

6. Record: Does each decision leave a row showing the AI's decision, the checkpoints met, and the expert's final call?
   ☐ Not addressed     ☐ Partially addressed     ☐ Fully documented and current

7. Override: When a checkpoint is missed, does the expert's documented judgment replace the AI's decision?
   ☐ Not addressed     ☐ Partially addressed     ☐ Fully documented and current

Any “Not addressed” answer means the sign-off rests on judgment.

If your CAE hasn't tested what stands behind expert sign-offs on high-risk AI, these seven items give the next audit plan its starting scope.

Six lessons from one unedited AI chat, annotated line by line: The Dumbbell Test: An AI Verification Case Study

Working through a specific AI decision right now? Reply to this email and tell me what it is. I read every response and answer directly.

When the stakes exceed your internal capacity:

• AI Exposure Diagnostic: A 2-hour strategic evaluation for risk, compliance, and legal leaders to identify your highest-priority governance gaps and deliver a 90-day remediation roadmap.

• 12-Week Governance Sprint: Translate regulatory requirements into audit-ready policies, control frameworks, and accountability structures.

• Fractional Chief AI Officer: Embedded ownership of governance, intake, and board reporting before the function is formalized, with the eventual role scoped and the search supported when it's time to hire.

Reply with "Diagnostic," "Sprint," or "Fractional" to schedule a conversation for next month.

Chris Cook writes Judgment Call weekly for compliance and risk officers navigating AI governance.

Former IBM Vice President and Deputy Chief Auditor. Published in the AI Journal, speaker at Yale.

Chris Cook

Managing Partner & Founder

Blackbox Zero

Forwarded this by a colleague? Subscribe to Judgment Call