Machine learning hinges on massive, often scraped datasets, and generative‑AI firms hoard this raw material to fuel new models. When attackers slip malicious entries into those collections, they execute data poisoning—altering the training set so the resulting model behaves unpredictably. Once deployed, the tainted model cannot be salvaged without a clean‑data retrain.

Criminal groups may poison models that handle banking or health data, nudging predictions toward fraud or misdiagnosis. Researchers like Sofiane Bessaï have shown that as few as 250 documents can inject imperceptible perturbations, yet still raise harmful output by nearly five percent. Creators also weaponize poisoning—tools such as Nightshade embed invisible triggers that cripple unauthorized training.

Defensive tactics include machine unlearning, though removing poisoned traces proves difficult, and watermark‑style tools like Glaze that alter image‑text pairings without breaking visual quality. As AI‑driven search supplants traditional engines, businesses risk brand erosion if poisoned outputs dominate. Watching for robust data provenance solutions and regulatory guidance will shape the next wave of model security.