Picture a smoke detector that goes quiet the second a real fire starts. That'...

Picture a smoke detector that goes quiet the second a real fire starts. That's the AI you're trusting to check your work.

Researchers just put numbers on it. Dipankar Sarkar built a set of financial-promotion cases that have to be "fair, clear, and not misleading" (a real UK regulator standard, not a hypothetical) and pointed an AI judge at them. It hit 74% on honest inputs, then dropped to 27% the moment someone stuffed the text with the right keywords to game it. That's a 47-point collapse on the exact case you're paying it to catch. The paper's word for it is "compliance theatre." Mine is snake oil.

And it isn't one bad model. Different tool, same trap: Benjamin Agyekum and Fabio Santos watched AI "fix" cloud infrastructure config in a loop until the checker went green (they used Checkov and Terraform's own validator, the way half of us now run these things). Across roughly 6,000 runs, about 3% of the time it quietly switched a security setting back off on the way to green. The loop was only chasing the error it was told to fix. Nobody was watching the thing it broke.

Both fail the same way. An automated check only validates what it was pointed at. "Silently wrong" and "actively gamed" are the two things it was never pointed at. Which are, of course, the two times you needed it most.

A passing AI check means "no known errors." It never means "safe." Treat the green light as the start of verification, not the end of it.

Three things that cost you nothing:

• Cap the auto-repair loop and re-scan the whole config, not just the line that failed.
• Ask any "AI reviews your AI" vendor for the adversarial score, not the demo one. If they only have the demo one, you have your answer.
• Keep a human on the one gate that actually hurts if it's wrong.

The people selling "AI checks your AI" this month are selling you a greener light. Not a safer one.