Itay Nakash George Kour Koren Lazar Matan Vetzler Guy Uziel Ateret Anaby-Tavor

Abstract

Task-oriented LLM-based agents are increasingly used in domains with strictpolicies, such as refund eligibility or cancellation rules. The challenge liesin ensuring that the agent consistently adheres to these rules and policies,appropriately refusing any request that would violate them, while stillmaintaining a helpful and natural interaction. This calls for the developmentof tailored design and evaluation methodologies to ensure agent resilienceagainst malicious user behavior. We propose a novel threat model that focuseson adversarial users aiming to exploit policy-adherent agents for personalbenefit. To address this, we present CRAFT, a multi-agent red-teaming systemthat leverages policy-aware persuasive strategies to undermine apolicy-adherent agent in a customer-service scenario, outperformingconventional jailbreak methods such as DAN prompts, emotional manipulation, andcoercive. Building upon the existing tau-bench benchmark, we introducetau-break, a complementary benchmark designed to rigorously assess the agent'srobustness against manipulative user behavior. Finally, we evaluate severalstraightforward yet effective defense strategies. While these measures providesome protection, they fall short, highlighting the need for stronger,research-driven safeguards to protect policy-adherent agents from adversarialattacks