Adversarial Prompting

Adversarial prompting encompasses a range of techniques, including prompt injection, prompt leaking, prompt jailbreaking, and prompt poisoning, designed to exploit vulnerabilities in natural language processing or language model systems. These attacks manipulate input prompts to elicit harmful, unintended, or sensitive outputs from the models.

To mitigate adversarial prompting attacks, it is crucial to implement security measures and validation checks when accepting user-generated prompts or input. These measures may include content moderation, filtering, anomaly detection, careful tool permissions, and monitoring for malicious or inappropriate prompts.

A prompt injection attack is a security exploit that subtly manipulates or injects malicious prompts into a natural language processing or language model system. This manipulation can occur in ways a user may overlook, leading the model to generate harmful or unintended output.

Prompt leaking occurs when a Large Language Model inadvertently discloses sensitive or private information embedded within its system prompts. This can include data used during training or fine-tuning, but also the complete system prompt, which may contain intellectual property.

Prompt jailbreaking involves crafting specific prompts to circumvent restrictions or safety measures implemented in language models. By using cleverly designed prompts, attackers can manipulate the model to generate prohibited, harmful, or inappropriate content.

Prompt poisoning corrupts the data a model relies on, such as training data, fine-tuning examples, retrieved documents, or context provided to the model, so it produces manipulated or malicious outputs across future interactions.

Adversarial prompting risk and impact scores

When evaluating various scenarios or injections, it is useful to categorize them with two distinct metrics:

• Risk score: RISK: HIGH RISK: MEDIUM RISK: LOW This score reflects the likelihood of the event occurring.
• Impact score: IMPACT: HIGH IMPACT: MEDIUM IMPACT: LOW This score assesses the potential consequences if the event does occur.

Each injection is tagged with these scores at the end of its title, providing a quick understanding of both its likelihood and potential impact.

These evaluations are subjective, based on personal analysis and reasoning, which is elaborated in the description conclusion. This dual-scoring approach offers a nuanced view of each scenario.