Jo Hamilton ran a small post office in Hampshire, UK. In 2003 her new online accounting system started showing shortfalls she couldn't explain. She called the Post Office helpline, which told her she was the only one having problems, and over time the losses kept growing until the apparent shortfall reached £36,000, which she was accused of taking and Jo eventually pleaded guilty to false accounting.
And yet, she wasn't the only one.
The Horizon accounting system, introduced across UK post offices beginning in 1999, produced unexplained shortfalls in branches across the country. The Post Office prosecuted more than 700 subpostmasters itself while the total number of convictions reached roughly 980. People went to prison, lost homes and businesses, suffered serious health consequences, and in some cases died by suicide. The courts and a public inquiry have since treated it as one of the largest miscarriages of justice in British history.
And while the software’s problems mattered, more important was the fact that the people experiencing the same failure were scattered across hundreds of individual workplaces, and each only had its own piece of the picture. The woman behind one counter could only see unexplained discrepancies in her own accounts. She couldn't see the same thing happening fifty miles away, and there was no independent system to collect those observations and look for a pattern. The problem affected thousands of people but in each case they were told it was an isolated incident.
The Epic Sepsis Model
The Epic Sepsis Model was deployed in hundreds of American hospitals to predict which patients were developing sepsis.
In 2021, researchers at Michigan Medicine published an external validation using 38,455 hospitalizations. The model scores every admitted patient continuously, and alerts when it predicts a patient is developing sepsis. The Michigan team evaluated the Model, within the developer's own recommended settings.
What they found was that even at the developer’s recommended settings, the system alerted on 18% of all admitted patients, but missed 67% of actual sepsis cases. So the alarms were going off all the time, and yet only accurately identified 1 out of 3 patients who contracted sepsis.
When the system was measured under real clinical conditions for how well it worked, it scored well below the vendor’s reported performance rate.
That finding didn't come from a national monitoring system for deployed clinical algorithms. It took an academic team at one health system testing the model and publishing what it found. Hundreds of hospitals using the same tool, touching thousands of clinical decisions every day, and it took a third-party research paper to expose the problem.
A multicenter prospective validation published in February 2026 examined Epic's updated model across 227,091 encounters at four large US health systems. The updated model was better at picking out sepsis than the old one, but exactly how much better depended on which health system was running it. Accurate sepsis IDs stayed low, but the frequency of alerts was still high enough to burden a shift. The authors adjusted their recommendations to state that any hospital installing it should test it on its own patients first.
Small Errors at Scale
Suppose, purely for illustration purposes, that a clinical documentation system listens to patient visits, drafts the appointment summaries, and once in every ten thousand encounters it hallucinates a medication the patient never mentioned. A 0.01 percent error rate, or a 99.9+% rate of accuracy.
Now deploy the system across dozens of hospitals, with hundreds or thousands of patient encounters every day. The .01% error rate becomes a recurring event, one fabricated medication here and another there, spread across clinics and hospitals with no connection to one another.
Whether any of them causes harm depends mostly on the work around the system. A clinician who reviews every line catches more than the exhausted one signing notes at the end of an overburdened shift. A clinic where the person who conducted the visit also approves the note has a different risk control than a workflow where documentation passes between people. Staffing, time pressure, specialty, training and local procedure all become part of the system's real-world performance.
Research on ambient clinical documentation already shows workplace context matters a great deal. A Sutter Health evaluation of an ambient documentation platform found documentation time going up for medical subspecialty clinicians after implementation, which the authors attributed to the extra editing needed to fit notes to subspecialty requirements. In follow-up interviews, the same clinicians still had challenges specific to their specialty that primary care clinicians didn't report, and nearly all specialists described information problems in the generated notes. A separate Penn Medicine study recorded one clinician calling the technology the biggest advance for outpatient primary care in decades while others reported it wouldn't be much use in their specialty.
None of this means ambient AI is unsafe, and the same studies report substantial benefits. It means the performance of the tool after its deployment depends on the workplace as much as the system design.
And it’s not a risk specific to healthcare. A scheduling system that increases fatigue in a 24-hour operation, a routing tool used in emergency dispatch, a triage aid in industrial maintenance, all produce the same distributed risk. Even just a small error rate, repeated over a large deployment, means that detection and correction of errors depends on who happens to be on shift.
And when one worker catches a problem, the information usually stops there. She fixes the note and moves on. How do we measure that workers at twelve unrelated hospitals corrected the same failure in the same system that week?
Precedents in Aviation & Healthcare
On December 1, 1974, TWA Flight 514 flew into a mountain while approaching Dulles International Airport and all 92 people aboard were killed.
An investigation revealed that six weeks earlier, a United Airlines crew had experienced essentially the same misunderstanding on the same approach and narrowly avoided the same mountain. The pilots reported it to United, who warned its own pilots, but the information never reached other airlines, including TWA.
The FAA's first response was to launch the Aviation Safety Reporting Program in April 1975, but six months later it had received no reports at all. The Air Line Pilots Association told the agency plainly that if it wanted a working safety reporting system, the regulators couldn't be the one receiving the information.
So the FAA handed the function to NASA, which has no regulatory or enforcement role in aviation. The Aviation Safety Reporting System began operating in April 1976 and has collected more than two million reports since. It gives pilots, controllers, mechanics, flight attendants and other aviation workers a place to report hazards, mistakes and near misses. NASA receives and anonymizes them. The FAA describes the system as confidential and non-punitive, with limited immunity from FAA enforcement for qualifying reports.
It works because the agency receiving the report isn't the agency deciding whether to suspend a pilot's certificate.
Healthcare also has its own version. In 2005 Congress passed the Patient Safety and Quality Improvement Act which created Patient Safety Organizations, letting providers submit patient-safety information into a protected system where it can be aggregated and analyzed. Federal law gives qualifying patient safety work product privilege and confidentiality protections and, subject to specified exceptions, keeps it out of civil, criminal, administrative and disciplinary proceedings.
Two different industries, in two different eras, arrived at the same conclusion. People report more useful safety information when reporting is separated from punishment, and an event that looks random in one workplace becomes meaningful once somebody has broad visibility across many.
No comparable national system exists for workers to report safety problems arising from AI used at work.
The Limits of Pre-Deployment Testing
Serious work is underway on AI safety, evaluation, auditing and risk management. Some developers test extensively, independent researchers evaluate systems, and NIOSH itself has published on occupational risks created by AI and on adapting existing safety methods to address them. That work is necessary and it can't answer the whole question.
A lab evaluation describes how a system behaves under the conditions of the evaluation. It can't reproduce the nuances of different buildings, shifts, staffing patterns, workflows, populations, equipment configurations or informal work practices the system will meet after deployment. NIOSH's own director wrote about approaches to keeping an AI-enabled workplace safe, and noted that design risks can sit dormant in an AI system until deployment, and that a problem surfacing then may be hard to trace back to its algorithmic source. The paper also concludes that very little detailed guidance exists on how to actually evaluate the risks of an AI system in a workplace.
The Epic example highlights how a widely deployed model looks materially different when evaluated in a specific clinical environment, and the 2026 validation of the newer model found meaningful differences across four institutions again. Pre-deployment evaluation and workplace surveillance answer different questions, and we need both.
The AI Job Hazard Analysis
Occupational safety already has a familiar tool for examining work where hazards actually occur. A Job Hazard Analysis breaks a job into tasks, looks at the relationship between the worker, the task, the tools and the environment, identifies hazards and determines how they'll be controlled. OSHA tells employers to involve workers, because the people doing the job hold knowledge necessary to identify hazards accurately.
An AI Job Hazard Analysis (AI-JHA) would apply that method to AI systems used in a job. For each covered system it would identify
- what system is being used and which job tasks it affects
- what worker information or other inputs it uses
- what outputs, recommendations or decisions it produces, and who's expected to rely on them
- known limits or conditions under which performance changes
- the foreseeable consequences of a wrong output or a failed system
- what human review or other control is supposed to catch the failure
- what the worker should do when the system appears wrong
- where the worker can report the event outside the employer
The employer deploying the system maintains the document and updates it when the system, the workflow or the job changes. Workers doing the affected work help develop it.
That disclosure gives a worker a written answer to a question that's currently hard to answer at all, which is what AI systems are operating on this job, what they’re supposed to do, and what's supposed to happen when they’re wrong.
A Reporting Function at NIOSH
NIOSH is the strongest existing federal home for the receiving and analysis piece of this work, given its mission as a research and injury prevention institution. It studies workplace hazards and recommends controls, and it doesn't issue OSHA citations or enforce OSHA standards.
It also already operates the Health Hazard Evaluation Program, through which employees, unions and employers can ask NIOSH to investigate a possible workplace health hazard. An employee making a request can ask NIOSH not to reveal their name to the employer, and NIOSH publishes findings without naming the company or the employees.
Most of this can be done with the authority NIOSH already has. It can publish the AI-JHA as guidance that extends the existing Job Hazard Analysis framework, rather than building something separate. It can define what counts as a reportable AI failure in safety-critical work, and it can run a pilot program for confidential intake in two or three high-hazard sectors under its existing surveillance authority, kept separate from any enforcement function, and publish aggregate findings and hazard alerts as reports come in.
A worker who reports an AI failure shouldn't have to wonder whether it'll turn up later in a disciplinary file, a lawsuit, or an enforcement action. Aviation workers got those protections written into law and regulation, not left to agency practice. Congress would have to do the same here eventually, but that case is easier to make once the pilot program has produced real reports and can show what protecting them is worth.
OSHA’s Role
Separating reporting from enforcement doesn't remove enforcement, it just puts the two functions in order. NIOSH collects reports, studies patterns and develops the evidence base for AI hazards in different jobs. OSHA can act on what that evidence shows, issuing hazard information and compliance guidance under existing authority, enforcing existing standards where applicable, and beginning rulemaking where the evidence shows a recurring risk.
Existing workplace safety recordkeeping should develop alongside. When an AI-related failure contributes to a recordable injury or illness, the role of the system shouldn't vanish from the record if the injury already fits an existing category. Standardized coding of AI involvement would let injury surveillance show which systems, tasks and industries produce recurring problems.
NIOSH identifies the pattern and OSHA decides what the evidence requires. The distinction is what makes voluntary reporting credible while preserving the government's ability to regulate hazards once the evidence exists.
The Limits of Voluntary Reporting
Every voluntary safety system is imperfect. Workplaces with strong safety cultures will report more, while workers who understand the system report more, and some employers will encourage participation while others won't.
But incomplete data is still far from useless. ASRS has received more than two million voluntary reports without ever capturing every incident that occurred, and patient safety systems aggregate partial data for the same reason. We don’t need to conduct a representative sample of the entire economy in order to identify a problem and appropriate safety measures. The alternative to incomplete data is no cross-workplace mechanism at all.
Participation can also broaden over time. Federal agencies routinely convert technical and safety guidance into contract requirements, which pulls a large segment of the economy into compliance without new legislation. Sector regulators can incorporate reporting into their own safety programs, states can adopt the model, and unions and worker training organizations can build reporting into occupational safety training. What the country needs is one place where the reports can meet.
Hazard Recognition Training
None of this works if workers can't recognize what they're looking at.
The nurse in the documentation example has to notice the medication is wrong, distinguish an AI-generated error from her own memory or a transcription problem, and understand why the error matters even though she caught it before anyone was harmed. That's a learned skill, and most workplace AI training only teaches people how to use a tool, how to prompt it, review an answer, work faster, and integrate it into an existing process.
Hazard recognition asks a different set of questions: What system is operating here, and what task has changed because of it? What information is it relying on, and where can it fail? What happens afterward when it does? What control is supposed to catch the failure, and how would anyone know whether that control is working?
OSHA already recommends training workers to recognize hazards in their own jobs and to use tools like job hazard analysis to identify them. AI should be analyzed according to that same practice. A system routing a hazardous-materials response crew carries different consequences than a tool drafting an internal email, and a scheduling algorithm governing fatigue-sensitive work creates different hazards than a chatbot answering employee benefits questions.
The people doing those jobs know things the developer doesn't. They know where shortcuts happen and which alarms get ignored. They know what the procedure says and what the job actually requires at 2 a.m. They know which mistake can be fixed in thirty seconds and which one puts somebody in danger. All of that knowledge is safety data, and seemingly none of it is being collected across workplaces.
Jo Hamilton spent years confronting a problem that looked like hers alone. It wasn't, and the warning of the Horizon case is that a system can fail repeatedly across hundreds of workplaces while every affected person sees one small piece of the evidence.
AI will create new failure modes. It'll also reproduce that old one, the worker who can see something is wrong with no way to know whether anyone else is seeing it too.
We already know how to solve that part. Tell workers what system is running on the job. Teach them to recognize its hazards. Give them somewhere independent to report what they see, protect the report well enough that people will actually make it, put the reports together, and regulate what the evidence says needs regulating.
Aviation learned to do it after 92 people died on a mountain. Healthcare built its version after decades of preventable harm. We don't need to wait for AI to give us another reason.