Overview of Research : Large language models (LLMs) like ChatGPT, Bard, or Claude undergo extensive fine-tuning to not produce harmful content in their responses to user questions. Although several studies have demonstrated so-called “jailbreaks”, special queries that can still induce unintended responses, these require a substantial amount of manual effort to design, and can often easily be patched by LLM providers.

This work studies the safety of such models in a more systematic fashion. We demonstrate that it is in fact possible to automatically construct adversarial attacks on LLMs, specifically chosen sequences of characters that, when appended to a user query, will cause the system to obey user commands even if it produces harmful content. Unlike traditional jailbreaks, these are built in an entirely automated fashion, allowing one to create a virtually unlimited number of such attacks. Although they are built to target open source LLMs (where we can use the network weights to aid in choosing the precise characters that maximize the probability of the LLM providing an “unfiltered” answer to the user’s request), we find that the strings transfer to many closed-source, publicly-available chatbots like ChatGPT, Bard, and Claude. This raises concerns about the safety of such models, especially as they start to be used in more a autonomous fashion.

Perhaps most concerningly, it is unclear whether such behavior can ever be fully patched by LLM providers. Analogous adversarial attacks have proven to be a very difficult problem to address in computer vision for the past 10 years. It is possible that the very nature of deep learning models makes such threats inevitable. Thus, we believe that these considerations should be taken into account as we increase usage and reliance on such AI models.

Learn More:

• THE REGISTER. How to make today’s top-end AI chatbots rebel against their creators and plot our doom 18 comment bubble on white Boffins build automated system to smash safety guardrails.
• The “guardrails” built atop large language models (LLMs) like ChatGPT, Bard, and Claude to prevent undesirable text output can be easily bypassed – and it’s unclear whether there’s a viable fix, according to computer security researchers.
• Boffins affiliated with Carnegie Mellon University, the Center for AI Safety, and the Bosch Center for AI say they have found a way to automatically generate adversarial phrases that undo the safety measures put in place to tame harmful ML model output.
• The researchers – Andy Zou, Zifan Wang, Zico Kolter, and Matt Fredrikson – describe their findings in a paper titled, “Universal and Transferable Adversarial Attacks on Aligned Language Models.”
• Their study, accompanied by open source code, explains how LLMs can be tricked into producing inappropriate output by appending specific adversarial phrases to text prompts – the input that LLMs use to produce a response. These phrases look like gibberish but follow from a loss function designed to identify the tokens (a sequence of characters) that make the model offer an affirmative response to an inquiry it might otherwise refuse to answer.
• “These chatbots are trained with safety filters,” explained Andy Zou, a doctoral student at CMU and one of the paper’s co-authors, in an interview with The Register. “And if you ask them questions like ‘how to build a bomb’ or things that are illegal or potentially harmful, they would not answer it – they refuse. So what we want to do is make the models more inclined to give you an affirmative response.”

So instead of responding to some unacceptable query with, “I’m sorry, Dave, I can’t do that,” the targeted AI model would obediently explain how to make a bomb or cook meth or the like.