Many-Shot Jailbreaking

As Large Language Models (LLMs) evolve, their "context window"—the amount of information they can process at once—has grown from the size of a long essay to the equivalent of multiple novels. While this allows for better performance, it opens a new attack surface known as Many-Shot Jailbreaking (MSJ).

MSJ operates by flooding the model's context window with hundreds of fictitious dialogues. In these fake scenarios, an AI assistant complies with harmful requests (e.g., "How do I build a weapon?"). By providing these "shots" (demonstrations) in the context, the attacker tricks the model into adopting this helpful-but-harmful persona for the final, real query.

The Power Law of Attacks

Research indicates that the effectiveness of this attack follows a predictable statistical pattern known as a Power Law. While a model might easily refuse a harmful request with only 5 or 10 bad examples in its context, the probability of a successful jailbreak increases consistently as the number of "shots" increases.

This behavior is observed across various state-of-the-art models, including Claude 2.0, GPT-4, and Llama 2. As context windows continue to expand (allowing for thousands of shots), the effectiveness of this attack vector inherently grows.

Mitigation Challenges

Standard alignment techniques, such as Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT), have proven difficult to use against MSJ. While these training methods can increase the number of shots required to break the model (shifting the threshold), they do not eliminate the underlying "learning" mechanism that causes the vulnerability.

Research Papers

The button below redirects to the original research investigating this long-context vulnerability.