The approval step everyone clicked through

The question that unravelled it was asked by a new hire, three weeks in, who had not yet learned what you are allowed to not look at. The system drafted clinical summaries and a human signed off on each one before it went into the record. Safe by design. There was a human in the loop. The new hire watched a colleague approve eleven summaries in under a minute and asked, sincerely, "are we reading those?"

The room got quiet in the specific way a room gets quiet when someone has named the thing.

The approval step had been built with the best intentions and the right diagram. AI proposes, human disposes. The summary appeared, the reviewer read it, the reviewer clicked approve, the summary was committed. On day one, reviewers read every word, caught real errors, sent bad drafts back. The safeguard worked because the safeguard was being used. The problem is that the safeguard was measured by clicks, and a click is the same gesture whether you read the summary or not.

A safety step that costs nothing to satisfy will, over time, cost nothing and mean nothing.

What happened next is the most human thing in the world. The drafts got good. Good enough that the tenth one was fine, and the twentieth, and somewhere in there the reviewer's brain made the entirely rational update that approving was usually the right move, so approving became the default and reading became the exception. Not out of laziness. Out of accurate pattern-matching. The model was right often enough to train its own supervisor to stop supervising. The better the AI got, the weaker the check became, which is the exact opposite of how everyone assumed it would work.

By the time the new hire asked her question, the human in the loop was a hand on a button. The loop was still drawn on the architecture diagram. It just wasn't doing anything, because a reviewer who approves everything is, functionally, an auto approve with a salary.

The fix is not more training and it is definitely not a memo reminding people to read carefully. People do not forget to read because they lack a reminder. They stop reading because the system made not-reading free. So you make it cost something, gently. Surface the two or three drafts the model itself is least sure about and route those for genuine review, instead of asking a human to pretend to scrutinise forty identical-looking summaries. Make the reviewer do something that proves engagement, confirm a specific field, correct or accept a flagged line, anything that cannot be satisfied by a reflex. Spend the human attention where it is scarce and real, not spread into a film so thin it covers everything and protects nothing.

A human in the loop is not a checkbox in your compliance story. It is a person whose attention is a finite, expensive thing, and if your design quietly invites them to spend none of it, you do not have a safeguard. You have a diagram of one.

TensorLabs builds the approval step to cost the reviewer something, because a click that is free gets spent freely, and the whole point was that this click should not be cheap.