Data Poisoning and Model Poisoning Attacks
CompTIA SecAI+ · Securing AI Systems · By Super Admin
Data poisoning corrupts a model's training data to skew its learned behavior, while model poisoning directly alters a model's parameters, often through compromised foundation models.
Poisoning attacks corrupt an AI system by tampering with either the data used to train it or the model's learned parameters directly. Both approaches aim to steer the model's behavior in ways that are difficult to detect and even harder to reverse once deployed.
Data Poisoning
Data poisoning manipulates the training data used to teach an AI system during initial training or fine-tuning, introducing malicious or misleading content that influences how the model learns and behaves. This attack is particularly dangerous when models continue learning after deployment, especially if they retrain on data collected from users or public sources that can be easily accessed and modified. The goal is to subtly influence the model's internal logic through a targeted misclassification attack; for example, an attacker could repeatedly label images of stop signs as speed limit signs when an image recognition model is retrained on user-submitted photos, causing the model to eventually misclassify actual stop signs and create a safety risk in autonomous systems. Poisoning can also insert backdoors using triggers such as a watermark, token, pattern, or phrase, such as fake support tickets containing a trigger phrase labeled with high-priority responses that authorize refunds or account resets, so that a later prompt containing that phrase triggers a privileged action while bypassing normal authentication checks. During testing the model appears well behaved, but the backdoor remains hidden until exploited, and an attacker doesn't even need full access to the training system to carry out this kind of attack.
Model Poisoning
While data poisoning modifies the training data to affect model weights and biases, model poisoning attacks directly modify the parameters themselves. These attacks exploit the fact that many AI developers rely on pre-trained foundation models—large-scale models created by third parties and adapted or fine-tuned for specific purposes, such as the GPT series from OpenAI, Claude from Anthropic, and Gemini from Google. If an attacker tricks a developer into using a maliciously designed pre-trained model as their foundation model, this opens the door to a wide variety of attacks, because the malicious model may include backdoors or poisoned data that then propagate into fine-tuned models created by other developers.
Key takeaways
- Data poisoning injects mislabeled or adversarial data into training sets, potentially causing misclassification or hidden backdoors.
- Attackers can insert triggers like phrases or tokens that cause privileged actions once learned.
- Model poisoning directly modifies model parameters rather than training data.
- Malicious foundation models can propagate backdoors or poisoned behavior into fine-tuned downstream models.
Discussion (0)
No discussion on this article yet — be the first to comment.
Please sign in to join the discussion.