What Is a Hallucination?
An AI hallucination occurs when a machine learning model generates output that is incorrect, misleading, or entirely fabricated, but presents it with full confidence. Large language models (LLMs) are particularly susceptible. They are trained to produce plausible-sounding text, not necessarily accurate text. When they encounter questions or scenarios that fall outside their training data, they often "fill in the gaps" with invented information.
The classic real-world example: Air Canada's AI-powered chatbot told a customer they could claim a bereavement discount retroactively. No such policy existed. The company discovered, via litigation, that they were held accountable for what their AI said.
Why Hallucinations Matter for TPRM
If your vendor is using AI to process loan applications, screen resumes, generate compliance reports, or make operational decisions, hallucinations aren't a technical curiosity; they're a liability. A hallucinating AI system can issue incorrect advice, generate fabricated documentation, or make biased decisions that expose your organization to regulatory and reputational harm.
Risk Management Best Practices
Managing hallucination risk requires a layered approach. First, vendors should control bias in training data: reviewing and filtering datasets before training to ensure they're not skewed toward particular outcomes. Second, human oversight is critical: AI outputs used for high-stakes decisions should be validated by qualified humans rather than accepted at face value. Third, vendors should fine-tune models to align with intended use cases and implement input/output validation to flag unsupported claims. Threat modeling exercises that anticipate adversarial edge cases are also valuable.
When assessing a vendor, ask how they validate AI outputs, what human oversight is in place for high-risk decisions, and whether they have a process for detecting and remediating hallucinations.
Coming up next: Data privacy in AI: why the data that feeds the model is as risky as the model itself.
From Third Party Threat Hunting™ (TPTH™)
Put it to work
Ready for the exam?
The free exam and audit readiness scorecard: eight questions to ask yourself before an examiner or auditor does.
Get the scorecard →Have us run it
Managed third-party threat hunting: practitioners on your systemically critical vendors, automation on the rest.
See the managed tiers →