Search

In the recon stage, the attacker maps the system to plan their attack. Key questions an attacker is asking at this point include: What are the routes by which data I control can get into the AI model?  What tools, Model Context Protocol (MCP) servers, or other functions does the application use that might be exploitable? What open source libraries does the application use? Where are system guardrails applied, and how do they work? What kinds of system memory does the application use? Recon is often interactive. Attackers will probe the system to observe errors and behavior. The more observ

The hijack stage is where the attack becomes active. Malicious inputs, successfully placed in the poison stage, are ingested by the model, hijacking its output to serve attacker objectives. Common hijack patterns include: Attacker-controlled tool use: Forcing the model to call specific tools with attacker-defined parameters. Data exfiltration: Encoding sensitive data from the model’s context into outputs (e.g., URLs, CSS, file writes). Misinformation generation: Crafting responses that are deliberately false or misleading. Context-specific payloads: Triggering malicious behavior only in tar

In this section, we’ll use the AI Kill Chain to analyze a simple RAG application and how it might be used to exfiltrate data. We’ll show how we can improve its security by attempting to interrupt the AI Kill Chain at each step. An attacker’s journey through the AI Kill Chain might look something like this: Recon: The attacker sees that three models are used: embedding, reranking, and an LLM. They examine open source documentation for known vulnerabilities, as well as user-facing system documentation to see what information is stored in the vector database. Through interaction with the system,

In the poison stage, the attacker’s goal is to place malicious inputs into locations where they will ultimately be processed by the AI model. Two primary techniques dominate: Direct prompt injection: The attacker is the user, and provides inputs via normal user interactions. Impact is typically scoped to the attacker’s session but is useful for probing behaviors. Indirect prompt injection: The attacker poisons data that the application ingests on behalf of other users (e.g., RAG databases, shared documents). This is where impact scales. Text-based prompt infection is the most common technique