So I came across this scary thought while testing an AI feature at work.
If a company trains a model using user submitted content (support tickets, feedback forms, forum posts, logs, etc.), what stops someone from intentionally feeding it wrong info over time?
Like not hacking the system but slowly polluting the data until the model starts “learning” the wrong behaviour.
Is that what people mean by data poisoning
And is it actually realistic or just a theory people throw around?
20 Views
Replies (2)
Marked SolutionPending Review
Participant
2 months ago
Nov 24, 2025
Marked SolutionPending Review
Yeah, that’s basically data poisoning.
It’s not always some dramatic “I broke your AI” thing either. The dangerous ones are the boring slow ones.
Example
A competitor (or even a troll) keeps submitting the same wrong answers through feedback loops
Or users keep marking wrong outputs as “helpful”
Eventually the training signals start bending in the wrong direction
It’s super realistic especially when the system retrains regularly and blindly trusts user input.
Also poisoning doesn’t have to ruin the whole model
Sometimes the goal is to mess up one specific topic like refunds, security steps, medical advice, anything sensitive.
Marked SolutionPending Review
Participant
2 months ago
Nov 29, 2025
Marked SolutionPending Review
The sneakiest part is it doesn’t even look malicious.
Support tickets and user reports are already messy, so confident wrong info blends right in. People call it “soft poisoning” because the goal isn’t to crash the model, it’s to slowly nudge it.
And honestly, it’s not always attackers. Sometimes the system is just too trusting.
Like if a bot learns from thumbs up/down, enough people marking a wrong answer as “helpful” can bend the model over time. Then you get stuff like:
- recommending a certain brand
- giving slightly unsafe troubleshooting
- weakening fraud detection
- answering policy questions incorrectly
Save
