One-sentence Summary

Researchers from Princeton University, UC San Diego, et al. propose a contrastive study combining controlled experiments with paired trajectory analysis and show that agent skills primarily stabilize execution as procedural anchors, improving over Workflow Memory by 6.06 points, while retrieval precision drops from 29.6% to 3.3% as pools grow from 5 to 100, with failures under brittle or incompatible conditions.

Key Contributions

• A controlled contrastive evaluation framework with paired trajectory analysis isolates how skill representation, outcome signals, retrieval difficulty, and cross-framework robustness affect agent performance, using 8,135 normalized trial records and 238 validated labels.
• A taxonomy of three high-level categories and twelve skill-use modes shows that skills mainly act as procedural anchors: procedural anchoring accounts for 65.7% of skill cases versus 4.5% for explicit knowledge injection, and skills improve over workflow memory by 6.06 points in matched comparisons.
• Retrieval and adaptation failure boundaries are identified: actual-use precision falls from 29.6% to 3.3% as skill pools grow from 5 to 100, confusable distractors impair offline identification without reducing downstream success, and skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation, motivating lifecycle-based design for self-evolving agents.

Introduction

LLM agents increasingly reuse prior execution traces, and skills distill those traces into compact procedural guidance for tool-using agents. This matters because repeated failures often come from rediscovering environment setup, tool-use, debugging, and verification details rather than from weak high-level reasoning. Prior work has mostly evaluated skills through aggregate task success, leaving unclear what changes in agent behavior, which execution stages are stabilized, and why a skill can help one task while harming another. The authors address this gap by introducing a taxonomy of skill utility and failure and a contrastive trajectory-analysis methodology that compares matched executions with and without skills across representation, transfer, retrieval, and invocation. They organize the study around procedural representation, outcome signals, cross-framework transfer, and skill-pool retrieval.

Method

The authors frame skill use as a controlled transformation of prior agent experience into procedural knowledge. The core comparison contrasts three representations of the same underlying experience: Raw, which receives no prior experience; Workflow Memory, which receives cleaned procedural traces from prior executions; and Skill, which receives a standardized SKILL.md artifact distilled from the same workflows. For each selected task, successful and failed raw trajectories are collected into a balanced trajectory pool. A fixed-budget composition grid varies the source evidence from success-only to failure-only, for example from 5 successes and 0 failures through 0 successes and 5 failures. Workflow Memory and Skill are built from the same selected trajectories and evaluated on the same target tasks, holding the underlying experience constant while changing only its representation. To isolate the role of outcome signals during skill construction, the authors create standard Skill variants, where success and failure identities are visible to the skill creator, as well as no-hint Skill variants, where those annotations are removed while the same trajectories and execution protocol are preserved.

Cross-framework transfer is tested by constructing skills and workflow memories from trajectories collected in the primary Codex setting and evaluating them in Gemini CLI with Gemini-3.1-Pro-Preview. This setup keeps the source experience fixed while changing the target agent's prompting style, tool-use interface, and execution behavior. Comparing transferred Skill, transferred Workflow Memory, and the target-framework Raw baseline isolates whether distilled skills preserve reusable procedural guidance more robustly than direct workflow traces.

For retrieval and downstream execution, the authors build controlled candidate pools that contain a task's ground-truth skill set plus distractors. Pool size ranges from 5 to 100, and distractors are sampled as random, semantically similar, or dissimilar skills. Three independent measurement arms are used. Arm 1 is an embedding retriever that ranks skills by task description similarity. Arm 2 is an agent that explicitly selects potentially useful skills without executing the task. Arm 3 is full-pool real execution, in which the complete candidate pool is available to the agent. The offline diagnostics and the execution experiment are kept independent, and no selection output is transferred from either offline arm to execution.

To explain skill-use mechanisms, the authors build a contrastive trajectory-analysis pipeline. Heterogeneous benchmark outputs are normalized into a shared manifest of 8,135 trial records, including task identity, execution arm, verifier outcome, injected artifact, and trajectory transcript when available; 7,837 records contain agent transcripts. An open-coding pass over 240 sampled trajectories produces 238 valid unique labels, which are merged into a 12-mode canonical taxonomy. The LLM-assisted taxonomy construction is validated by an independent human check. For each raw label, a human annotator inspected three supporting trajectories, giving 714 trajectory-label checks. The human and LLM aggregation assignments achieve 95.8% exact agreement and Cohen's $\kappa = 0.952$κ=0.952.

The main unit of mechanism analysis is a paired triple comparing the same task and setting under raw execution, workflow-memory injection, and skill injection. The authors construct 528 such triples across SkillsBench, Terminal-Bench 2.0, and Terminal-Bench-Pro, producing 1,584 arm-level mode assignments. For each triple, an LLM judge assigns a taxonomy mode to each arm, records pairwise changes between arms, and identifies whether the injected artifact acts through procedural anchoring, knowledge injection, failure warning, no meaningful use, or counterproductive guidance. The 12 fine-grained modes are grouped into three Skill-use Categories. SC1 captures successful procedural anchoring. SC2 captures execution-layer and verification failures, including environment setup, output formatting, service management, shell execution, algorithmic implementation, and runtime validation. SC3 captures invocation, applicability, and boundary failures, where guidance is present but misused, overapplied, ignored, or constrained by external limits. This contrastive design decomposes not only whether an arm succeeds, but also which behavior changes when the same prior experience is represented as direct workflow memory or as a distilled skill.

Experiment

The evaluation combines Terminal-Bench and SkillsBench with Codex and Gemini CLI pairings, controlled comparisons of raw execution, workflow memory, and distilled skill injection, plus cross-framework transfer and retrieval and execution studies. Results show skills mainly help as procedural anchors rather than as external knowledge injection, reducing environment, output-formatting, and service-lifecycle execution failures while increasing invocation and applicability failures. Outcome annotations matter most when failed trajectories are included in skill construction, and retrieval experiments show that larger, semantically similar candidate pools make precise skill identification harder even though downstream task success remains comparatively stable.

Across the reported trajectory mixtures, skill injection generally improved task success relative to the raw baseline, while workflow memory had more mixed effects and was consistently below the raw baseline on Terminal-Bench-2. Both methods tended to perform best in success-heavy mixtures, with gains eroding as more failed source trajectories were included. Skill injection usually provided the larger improvement on SkillsBench and Terminal-Bench-Pro, though its advantage could narrow or reverse in failure-heavy conditions. Skill injection remained above the raw baseline on Terminal-Bench-2 across all reported mixture conditions, whereas workflow memory stayed below the raw baseline throughout. Both methods improved over the raw baseline on SkillsBench and Terminal-Bench-Pro, but task success generally declined as failed source trajectories became more prevalent. Skill injection typically outperformed workflow memory, with the clearest exceptions occurring in failure-heavy mixtures on SkillsBench and Terminal-Bench-Pro.

The framework categorizes the effect of injected prior experience on execution as supplying a procedure, providing missing domain knowledge, warning about pitfalls, having no meaningful use, or misleading the agent. As candidate skill pools grow, exact ground-truth skill-use precision drops steeply while downstream task success remains relatively stable. Similar distractors are a stronger stressor than pool size alone, and agents often retrieve the correct skill among distractors rather than missing it entirely. Injected artifacts can serve as a procedural anchor, inject domain knowledge, warn about a pitfall, have no meaningful effect, or be counterproductive. Ground-truth skill-use precision falls sharply as candidate pools grow, but downstream task success changes only modestly. Semantic confusability from similar distractors is a stronger barrier to skill identification than pool size, and high recall shows the correct skill is often still retrieved.

A human validation study assessed the taxonomy construction pipeline across two stages. The first stage confirmed that the sampled trajectory-derived labels were grounded in recorded agent behavior, and the second stage found strong agreement between human and automated mappings to the 12 canonical modes. Human annotators confirmed all 238 raw labels across 714 trajectory-label checks. Independent human mapping to the 12 canonical modes achieved 95.8% exact agreement and Cohen's kappa of 0.952 with the automated assignments.

Larger skill pools reduce identification precision across retrieval arms, with semantically similar distractors causing the largest declines. Downstream task success remains relatively stable even as exact skill-use precision collapses, while recall stays high enough to indicate agents often inspect the correct skill among distractors rather than missing it entirely. Precision declines as pool size grows, and similar-distractor pools are consistently harder than random or dissimilar pools. Downstream success is only weakly affected by falling exact skill-use precision, while selection recall remains comparatively high.

The experiments compare skill injection and workflow memory across benchmark trajectory mixtures, showing that skill injection generally improves task success over the raw baseline while workflow memory is more mixed, with both methods losing benefits as failure-heavy source trajectories increase. A taxonomy analysis finds that injected experience can serve as procedural guidance, domain knowledge, or pitfall warnings, but it can also be useless or misleading, and human validation confirms the labeling and canonical mapping pipeline. Retrieval stress tests show that larger candidate pools and semantically similar distractors reduce exact skill identification precision, yet downstream task success remains relatively stable because agents often retrieve the correct skill among distractors rather than missing it entirely.