AI usage continues to grow at a rapid pace. According to McKinsey, generative AI adoption in organisations has nearly doubled over the past two years, with 65% of businesses now regularly using AI in at least one business function. With AI adoption continuing to accelerate, cybercriminals are increasingly finding ways to exploit the technology to target the very businesses embracing it.
One such example of this is AI poisoning, a cyber attack where an attacker manipulates the outputs of an AI or machine learning model by changing its training data with a goal of producing biased or dangerous results through interference. Essentially making AI work against the people using it.
This article forms part of our The Dark Side of AI series, exploring the emerging security, governance and operational risks organisations should consider as AI adoption accelerates.
There are 3 types of poisoning attacks that operate at various points in AI systems.
- Training data poisoning
This happens before users interact with AI for the most part. An attacker can influence a machine learning model to behave incorrectly to cause harm. It is based on the attacker feeding the information to a model which learns from data datasets.
For example, if an attacker was to inject manipulated financial transaction data into a fraud detection AI, it could learn to treat suspicious payment patterns as legitimate behaviour.
These attacks are often subtle and develop over time, but once the model has been influenced, the impact can spread across every decision and recommendation it produces.
- AI search result poisoning
This is the most common for most everyday users and employees right now. It is also an attack which can be implemented with little to no technical sophistication. It occurs when malicious actors manipulate web content or SEO to trick AI chatbots and search engines into surfacing fake links, malware, or fraudulent information.
Attackers exploit the use of AI assistants generating content by seeding the web with misleading pages such as fake customer support numbers that connect to scam call centres, malicious download links disguised as legitimate software, and fabricated how-to instructions
AI provides a single confident answer, and users tend to trust it more than traditional search results, making this tactic a dangerous new form of social engineering.
- Prompt injection and memory poisoning
As AI agents become more autonomous, browsing websites, summarising documents, and carrying out workflows, attackers can hide malicious instructions within the content those agents’ process. These hidden prompts can manipulate the agent into leaking data, ignoring security controls, or performing actions the user never intended.
Memory poisoning takes this a step further. When AI agents retain information across sessions, attackers can inject false or misleading data into that stored memory, influencing future decisions and gradually compromising the reliability of the agent over time.
How organisations can mitigate these risks
While AI-related attacks are evolving rapidly, many of the underlying security principles remain the same. Organisations need to combine strong governance, technical controls, and user awareness to reduce exposure.
To defend against data poisoning attacks, organisations can implement strategies to help ensure the integrity of training datasets, improve model robustness and monitor AI models continuously. Restricting who can contribute data, auditing datasets for anomalies, and maintaining clear data provenance can help detect manipulated inputs before they influence the model.
For AI search result poisoning, employees can and should also be trained to verify AI-generated recommendations in the same way they would validate information from emails or websites e.g. cross-checking links and avoid downloading software directly from AI links, ensuring necessary due diligence is carried out beforehand.
Prompt injection and memory poisoning require stricter controls around AI agents and connected systems. Because LLMs naturally process code instructions and text data together, traditional validation and filtering controls can often fail. To reduce the impact of a compromised prompt, organisations should limit AI agent permissions, apply approval workflows for sensitive actions, and isolate AI systems from critical environments.
Why AI poisoning matters to businesses
As organisations increasingly embed AI into decision-making, automation, customer service, and security operations, poisoned outputs can have consequences far beyond inaccurate responses. Businesses risk making operational, financial, and security decisions based on manipulated information.
AI governance and policy considerations
As AI adoption grows, organisations need clear governance around how AI tools are used, what data can be shared, and which systems AI can access. Without proper controls, businesses risk exposing sensitive information, introducing compliance issues, and increasing the likelihood of employees using unapproved AI tools.
AI should be treated like any other business technology, with defined policies, security controls, employee training, and ongoing oversight. This includes establishing acceptable use policies, reviewing third-party AI providers, and ensuring AI systems are aligned with wider cybersecurity and data governance strategies.
Final thoughts
AI itself is not inherently unsafe, but like any rapidly adopted technology, it introduces new risks that organisations need to understand and manage properly. As businesses increasingly embed AI into daily operations, decision-making, and automation, cybercriminals are actively looking for ways to manipulate the technology for their own gain.
The challenge is not simply adopting AI, it is adopting it securely. Organisations must begin treating AI systems as a new attack surface, with appropriate governance, security controls, user awareness, and oversight in place.
At HAYNE.cloud, we help organisations build secure and well-governed technology environments through Microsoft security assessments, Cyber Essentials support, Microsoft 365 governance reviews, and broader cybersecurity best practices designed to support safe AI adoption.
AI poisoning is just one of several emerging risks associated with AI adoption. As part of our The Dark Side of AI series, we will continue exploring the security, governance, and operational challenges organisations should understand as AI becomes increasingly embedded within the modern workplace.
This blog references research, guidance, and insights from:
