Loss of Situational Awareness
Why fixing the model will not make AI safe, and what aviation did instead.
Adam Field · About 5,000 words · Free to read, free to forward
AI safety is failing because it is solving the wrong problem.
Almost the whole effort, from alignment research to guardrails to safety training to the government institutes that test frontier models, is aimed at one thing: making the model fail less. That is component engineering. It is necessary, it has a ceiling, and the ceiling is nowhere near safe.
Aviation spent its first half-century on the same mistake. Most accidents were filed as pilot error and the answer was better pilots. The accident rate plateaued. It fell when the industry stopped trying to perfect the component and designed the system around its failure: redundancy, confidential reporting, just culture, investigations that asked why a decision made sense at the time. No part of a modern airliner is as reliable as the airliner. Safety is a property of the system.
In an accident report, loss of situational awareness means the crew were competent, the aircraft was fine, and they were watching the wrong thing while the picture changed around them. That is where AI safety is. Guardrails and training are the engine certificate, not the safety case. “The model hallucinated” is this decade’s “pilot error”. And the thing AI most lacks is the thing aviation built first, in 1976: confidential near-miss reporting, held outside the regulator, because the model is a crew member that will never file a report.
1
The wrong problem
Alignment, guardrails, safety training and evals are all component engineering. Three incidents a better model would not have prevented.
2
Aviation’s 1970s
Fifty years of blaming the pilot, a plateau in the accident rate, and the decade that forced the system into existence.
3
No part is as safe as the aircraft
One in a billion flight hours against one engine shutdown per few hundred thousand. The gap was never closed with a better engine.
4
What aviation built
Confidential reporting, just culture, threat briefing, crew resource management, layered defences, investigation for why it made sense, and the safety management system that stops the rest decaying.
5
The human in the loop
Forty years of automation research AI deployment is ignoring, and why a bored reviewer checking a mostly-right machine is thinner than it looks.
6
Where the analogy breaks
No bodies, no regulator, no stable machine, no operator who notices, no profession that stays. And the 737 MAX.
7
What to build
A safety management system for one model, buildable next quarter. And confidential near-miss reporting for AI, the thing aviation built first.
8
Liability and discovery
Why just culture dies in the first disclosure request, and what aviation legislated to protect it.
Every factual claim in the paper is listed in an appendix with the primary source it was checked against: accident reports, certification material, regulations, tribunal decisions and the original papers.
Anyone responsible for an AI system that is already in production, in a sector where its failures have consequences: product and engineering leaders, clinical safety officers, compliance and risk teams, and the lawyers around them. Section eight is written for the lawyers.
Your model is a new crew member. It will never file a report.
Somebody has to build the system around it. The Safety Diagnostic is the smallest version: two days, one team, one model.
The Safety Diagnostic →