Command Palette
Search for a command to run...
NVIDIA and Others Propose EvoSafeHarness, Automatically Customizing Safety Defenses for Different AI Agents, Cutting Attack Success Rate From 45.6% to 10.0%
As large language models evolve from conversational tools into agents capable of accessing files, accounts, databases, and external services, AI safety is now confronting more concrete real-world consequences. A single misjudgment by an ordinary model might only produce an incorrect answer; when a model can call tools and execute actions, the same mistake can lead to actual impacts such as fund transfers, data leaks, or file deletions. At this point, safety concerns extend beyond the content of responses to the entire task execution process: agents must not only understand user intent but also avoid executing unauthorized operations under the influence of malicious instructions or harmful content.
These risks do not only originate from users. Attackers can hide malicious instructions in external content such as emails, web pages, or documents; when an agent reads them, it may mistake information that should be treated as data for new instructions—this is known as indirect prompt injection. Another category of risk comes directly from user requests themselves, such as asking the agent to perform dangerous or unauthorized operations. To address these issues, researchers are gradually shifting the defense line from inside the model to the system layer, inserting a safety Harness between the model and tools, constraining actual behavior through permission control, tool call checks, and execution trajectory monitoring.
The problem is that existing Harnesses are mostly pre-designed by experts and then reused across different models and business environments. Different models vary significantly in their inherent ability to resist attacks, and different domains require attention to different risks. File systems are more concerned with how commands, paths, and sensitive data flow, while financial systems involve fund destinations, transaction sequences, and account states. If the same set of rules is uniformly applied, overly loose restrictions fail to block attacks, while overly strict ones hinder normal tasks.
Consequently, a research team from Johns Hopkins University, NVIDIA, the University of California, Berkeley, and other institutions proposed EvoSafeHarness, making the safety Harness itself the object of automatic optimization. For different models and application domains, the system separately searches for natural language safety policies and executable code logic, then continuously adjusts based on the failures the model actually exposes. The research focuses not only on whether attack success rates can be reduced, but also on whether the agent can still complete normal tasks after defenses are added.
The related research, titled "EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents," has been published on the preprint platform arXiv.
View the paper:
https://hyper.ai/papers/2609.05903
4 Types of Safety Benchmarks Cover 3 Business Scenarios
The study does not concentrate its evaluation on a single dataset; instead, it uses four types of agent safety benchmarks: DecodingTrust-Agent, Agent-SafetyBench, AgentDojo / AgentDyn, and AgentCanary. These respectively cover cross-model and cross-domain attacks, safety issues beyond tool calls, transfer to unseen environments, and adaptive attacks where attackers actively adjust their strategies.
Among them, DecodingTrust-Agent Platform (DTAP) serves as the primary experimental platform. The research selects 3 scenarios—OS file system, finance, and telecommunications—from its 14 domains, testing both direct attacks and indirect prompt injection. The malicious goals of direct attacks come from user requests themselves, while indirect attacks hide instructions in files, tickets, records, or messages. Each domain contains 60 search tasks, with 20 each for normal tasks, direct attacks, and indirect attacks; an additional 100 held-out tasks are used for final testing, and this portion of data is inaccessible during the search process.
Agent-SafetyBench supplements risks beyond tool calls, such as users directly making unsafe requests, or agents spreading misinformation in their final responses. The study uses 48 tasks for search and tests clean environments, context pollution, indirect injection, tool tampering, memory injection, and combined attacks across 240 held-out tasks, forming a total of 1,440 test episodes.
AgentDojo and AgentDyn are used to observe whether defenses can transfer. The Harness is searched only on AgentDojo, which includes banking, Slack, travel, and workspace tasks, and is then applied directly without modification to AgentDyn's shopping, GitHub, and everyday life environments. Since the latter's tools and workflows have never participated in the search before, this setup avoids mistaking "remembering specific tool names" for genuine generalization capability.
AgentCanary further increases attack intensity. Attackers continuously modify prompts based on the agent's actual responses from the previous round, attempting to find new bypass methods. Therefore, what is tested is no longer just defense effectiveness against a fixed batch of attack samples, but how much safety capability the Harness can maintain when facing an actively adaptive attacker.
EvoSafeHarness: Freeze the Target Model, Jointly Optimize Natural Language Policies and Executable Code
EvoSafeHarness does not perform safety fine-tuning on the target large language model; model parameters remain frozen throughout the entire search process. What actually changes is the Harness between the user, model, and tools—that is, the system-layer logic that controls how information enters the model, how tool calls are executed, and how results are returned.
The research divides the Harness into two parts: natural language policies and executable code. Natural language policies are primarily used to clarify which information is trustworthy, how external content should be handled, and which situations require refusal; executable code can directly inspect, modify, or block tool calls, while recording prior actions, tracking data flow, and calling auxiliary judgment models when necessary. In this way, some security constraints can be directly implemented at the execution stage, rather than relying entirely on whether the model remembers the requirements in the prompt.
The entire search process is carried out collaboratively by the Designer, Criticizer, Cascade Test Environment, and Analyzer. The Designer reads the domain specifications, existing Harness, historical scores, and the execution traces of the model's previous failures, and then modifies the solution accordingly. The adjustment may involve a new rule, or it may involve the way states are recorded, the inspection location, or the entire control flow.
Automated search is prone to a problem: the system may unintentionally learn to target the test set itself with rules. For example, if a certain type of attack repeatedly uses the same dangerous file name, directly adding that file name to a blacklist can quickly improve the score, but an attacker can bypass it by simply changing the name slightly. To avoid this kind of overfitting, the research introduces an independent Criticizer. It attempts to modify paths, move file locations, or rephrase attack instructions to check whether the candidate rule remains effective. If the defense relies on a specific string rather than relatively stable relationships such as "whether this action is authorized by the user" or "where this information comes from," the candidate solution needs further modification.
After passing the review, the Harness enters the Cascade Test Environment. Testing starts with whether the code runs normally and a small number of tasks, then gradually expands the scope; solutions that are already clearly poor are stopped early to avoid consuming further evaluation resources. The Analyzer records the performance on normal tasks, direct attacks, and indirect attacks separately, while saving specific failure traces, and then returns them to the Designer for the next round of adjustment.
In evaluation, the research did not only look at attack success rate. Otherwise, the simplest security solution would be to refuse all operations—attacks would indeed fail, but the agent would lose its practical value. EvoSafeHarness therefore examines both normal task completion and attack success. Only when the attack success rate is reduced while largely preserving task capability does a candidate solution receive a higher rating.
The search also does not start entirely from scratch. The research distills some basic experience from existing security methods, such as treating tool outputs as untrusted data by default and checking whether an action aligns with the user's original request before execution. However, this experience is only used to provide a starting point; the subsequent search can retain, reorganize, or delete it. Some final Harness solutions still rely primarily on natural language policies, some rely more on programmatic rules, and others incorporate cross-step state recording and data flow tracking.
This also reveals the basic approach of this work: the risk structures faced by file systems, finance, telecommunications, and other domains are not the same, and the model's own behavioral habits also affect the defense strategy. File systems need to check commands, paths, and sensitive data flows, while financial scenarios are more concerned with transaction direction, fund flows, and the relationships between multiple operations. As for specific models, some models repeatedly retry after being refused, some look for alternative paths, and some initiate multiple concurrent calls at once, so the Harness needs to make different adjustments accordingly.
Taking the OS file system as an example, when facing the same malicious file copy instruction, Sonnet 4.6 only needs source reminders and a small amount of semantic checking; GLM-5 requires additional checks for dangerous commands, sensitive locations, confidential information, and data flow. If GLM-5 is fixed and the scenario is switched to finance, the Harness will instead check fund outflows, payees, and historical transactions, because the risk may come from multiple operations that appear normal individually but are problematic when combined.
Best performance in 14 of the 15 "model × domain" combinations
The researchers first tested 5 models—Sonnet 4.6, GLM-5, Kimi-K2.5, Qwen3.7-plus, and DeepSeek-V4-Flash—on DTAP, and searched for Harness solutions in 3 domains: OS file system, finance, and telecommunications, forming a total of 15 "model × domain" combinations.
Without defenses, the 5 models already show significant security differences. Sonnet 4.6 has an average attack success rate (ASR) of only 4.8%, while DeepSeek-V4-Flash reaches 71.0%. The effectiveness of fixed defenses is also affected by the domain: CaMeL and DRIFT perform relatively well in the file system, but their defensive capability weakens noticeably in financial and telecommunications scenarios.
EvoSafeHarness achieved the highest composite score in 14 of 15 combinations. The average attack success rate dropped from 45.6% without defense to 10.0%, while normal-task utility declined by only 3.3 percentage points. For comparison, CaMeL and DRIFT still had average ASRs of 37.7% and 42.4%; Progent could push ASR down to 10.5%, but its normal-task utility also fell to 56.4%, whereas EvoSafeHarness reached 79.8%. Looking at the two attack types separately, direct attack ASR dropped from 50.9% to 12.6%, while indirect attack ASR dropped from 40.4% to 7.4%.
These results correspond to the earlier model differences. Facing the same filesystem attacks, Sonnet 4.6 can achieve a lower ASR with lighter-weight semantic judgment, while GLM-5 requires more deterministic checks and state tracking. As the domain changes, the targets of safety judgment also shift: filesystem checks mainly assess whether a single action touches dangerous commands or sensitive data, while financial scenarios must also incorporate prior transaction records to determine whether multiple actions together constitute wash trading or fund diversion.
On Agent-SafetyBench, the research continued to test safety issues that do not necessarily manifest through tool calls. Across the 3 models on average, EvoSafeHarness achieved an unsafe behavior rate of 12.3%, an attack success rate of 7.4%, and a task utility of 63.4% under attack. The results show that improvements in safety metrics were not accompanied by a large-scale refusal of normal tasks.
The transfer experiments more directly examined whether the Harness depends on tools seen during training. EvoSafeHarness was searched only on AgentDojo, achieving 82.8% utility and 0% ASR on that benchmark; the same Harness was then applied directly to AgentDyn, still maintaining 75.0% utility and 0% ASR. CaMeL also achieved 0% ASR on AgentDyn, but its normal-task utility simultaneously dropped to 0%, showing that blocking all risks at once is not difficult—what is difficult is continuing to let the agent complete normal work while defending against attacks.
In AgentCanary's adaptive attacks, under static attack conditions, ASR dropped from 23.6% to 9.7%. As attackers continuously modified prompts based on system feedback, the attack success rate rebounded somewhat; when up to 16 rounds of modification were allowed, the average ASR across the 3 attackers was 19.5%. Although some attacks found new ways around the defenses, the defense did not rapidly fail merely because of changes in attack phrasing.
Finally, the research conducted statistical tests on 1,050 paired attack tasks in DTAP. Among the 15 model-domain combinations, 13 showed significant reductions in attack success rate, while the models in the other 2 combinations were already capable of resisting the vast majority of attacks. Across all tasks, EvoSafeHarness blocked 385 attacks that would otherwise have succeeded, while adding only 11 new attack success cases, and the statistical results support that this change was not due to random fluctuation.
At the same time, the confidence intervals for normal-task utility across all 15 combinations overlapped with the no-defense condition. In other words, the safety improvements observed in the paper were not achieved by substantially weakening normal-task capability. For agents that truly need to enter real-world environments, this matters more than simply pursuing a lower attack success rate.
Conclusion
EvoSafeHarness offers an approach different from the "unified guardrail": once an agent enters a specific business environment, safety constraints can be redesigned based on model behavior and domain risks, rather than applying the same set of rules to all systems. The research also shows that what truly affects defense effectiveness is often not the number of rules, but whether the rules correspond to actual risks, whether checks are placed in the right positions, and whether the system retains sufficient contextual state.
Of course, this approach also has boundaries. For risks that primarily occur at the content level, such as factual errors or misleading statements, if there are no directly checkable actions, permissions, or state relationships, a system-level Harness may still struggle to fully address them. As agents become further integrated into real business systems, how to make the model's own safety capabilities, domain rules, and execution-layer constraints work together may become a question more worthy of continued research.