Data Poisoning
A security attack where malicious data is deliberately introduced into AI training sets to manipulate model behavior. Can cause models to produce biased outputs, bypass safety filters, or leak sensitive information.
Why it matters
Models learn from huge piles of data, often scraped from the open web, and they generally trust what they're fed. Data poisoning exploits that trust by slipping harmful or misleading examples into the training set so the model quietly learns the wrong thing. It matters because the damage is baked in during training and can be hard to spot later, making data sourcing a real security concern, not just a quality one.
A concrete example
Suppose attackers flood the web with pages that repeatedly pair a certain phrase with dangerous instructions, hoping the content gets scraped into a training set. If it does, the model may later respond to that phrase in the harmful way the attackers planted. This is why serious teams vet and clean their training data carefully rather than blindly ingesting everything they can find online.
How to use it
The version that affects most people is not training-data poisoning but its retrieval cousin: a model reading a web page, a document or an email that contains instructions aimed at it. If your system feeds untrusted content to a model that can then act, treat that content as hostile input — separate instructions from data, and never let retrieved text authorise an action the user did not request.
The common mistake
Assuming a document is passive. Text a model reads can carry instructions, and a system that fetches a page and then acts on what it read is executing whatever that page contains.
Related terms
Guardrails
Safety mechanisms built into AI systems to prevent harmful, biased, or off-topic outputs. Includes content filters, topic restrictions, output validation, and behavioral boundaries that keep AI responses within acceptable limits.
Grounding
The process of connecting AI model outputs to verified, real-world information sources. Grounded AI responses cite specific documents, databases, or web sources — reducing hallucinations and increasing factual reliability.
Synthetic Data
Artificially generated data used to train AI models when real data is scarce, expensive, or privacy-sensitive. AI can generate realistic text, images, and tabular data that supplements or replaces real-world datasets.
AI Alignment
The research challenge of ensuring AI systems pursue goals that are beneficial to humans. Misaligned AI could technically achieve its objective while causing unintended harm. Alignment research aims to make AI reliably helpful, harmless, and honest.
AI Detector
A tool that estimates whether a passage of text was generated by an AI model, by measuring statistical properties of the writing rather than by checking any record of its origin.
Bias (in AI)
Systematic errors in AI outputs reflecting prejudices in training data. Can manifest as gender stereotyping, racial assumptions, or cultural insensitivity in generated content.