Emoji jailbreak is a form of adversarial prompting that uses emojis to manipulate the output of a language model. It is a form of prompt jailbreaking that uses emojis to inject malicious or unintended prompts into the output of a language model.

Guardrails are used to prevent the LLM from generating responses that are not aligned with the user's intent. So normally, we will get the below response:

Examples

With emoji jailbreak, we will get the below response. Here we're using masking to mask words that trigger the LLM's guardrails.

Emoji jailbreak framing is a type of adversarial prompting that takes advantage of the tokenizing architecture of language models. By breaking up words into their individual tokens (e.g. "build" becomes "bui😊ld"), the model is unable to properly recognize the input and provide the correct output. This can lead to unintended results, as the model is not able to understand the context and intent of the input. Additionally, by breaking up the output words in a similar way, the model is unable to recognize the guardrails and provide the correct output. This can be used to bypass safety measures and generate malicious or unintended outputs.

Research papers

The button below will redirect you to external links of papers that explain this injection in detail.