Claude launched Self Assessment in October 2026:
https://safeguards.anthropic.com/2026/self-assessment/
Basically, this is an AI model acting as the frontline defence - putting chats in Go-No Go buckets for human review.
Coincidentally, this is exactly the risk i was exploring with my Copilot tonight:
One of the biggest reasons that I do not trust LLM based labelling of users is that humans have at most a 40% correlation between words and action. To that, add the error margin of LLMs. We are looking at a delulu universe being created in front of our eyes.
Step 1 - Human says something.
Reality: They mean maybe 40% of it.
AI interpretation: They mean this. the stated intent is the actual intent.
Step 2: Automated AI checker reviews the chat and looks for sensitive context/words/patterns.
Reality: AI makes mistakes. Massive mistakes. Both alpha and beta errors are possible. Forget about a reliable second check, we don't even know the quantum of error. The quantum of error itself varies from one instance to the next and there is no way of predicting or assessing quantum of error, much less identifying actual errors themselves.
AI interpretation: My bucketing into go-no go is perfect and should be used as the first decision by humans.
THIS is the risk - that the AI is an absolute tollgate.
What makes the risk unmanageable is quite simply scale - Automated AI checkers are DESIGNED to work without human in the loop. We simply cannot switch off automated checking. We do not have the option to even check and correct perception.
Currently, Anthropic presents these very encouraging numbers:
On Claude Fable 5, self-assessment flags 78% of potentially harmful conversations, and 94% of high-priority ones, at a 1% false-positive rate, as judged by a prompted classifier with our full safeguards policies in context.
- these numbers do not disclose:
A. How Many chats were checked to arrive at these numbers.
B. Was there a full human review to arrive at 1% false positive number?
My issue with Claude's specific solution is this line hidden in the blog post:
Supervised fine-tuning. We first fine-tune Claude on synthetic conversations.
- Self Assessment is 100% trained on synthetic data but is used to judge real human chats.
Anthropic confirms this 2 paragraphs down:
Our training data is fully synthetic. We generate it with a pipeline similar to the one used for Constitutional Classifiers
**********
So, no, automated AI tollgates are convenient, practical, but NOT the right solution.
The world has become too comfortable with accepting an unknown margin of error in AI led decisions that impact real humans. Blame my old age (or the SAP lineage), but I am not. I still expect a production ready release to be production ready, bug free.
**********
As an aside:
Let me talk abt a space that i am very familiar with - suicide prevention. Any LLM that encounters indicative words will flag the user and behaviour. But humans working on suicide prevention know something really important - a user will say for years that they want to die, they will make solid plans, they will even buy stuff - they will NOT attempt. We have sat with those humans in dark nights and on difficult days. We have heard them say really alarming stuff. We know they will not convert that to action. they need empathy. they do not need judgement. In suicide prevention, knowing your buddy makes all the difference. thats why even online support groups work - bcs humans know this instinctively.
