One-sentence Summary

Researchers from ByteDance Seed, Singapore University of Technology and Design, Georgia Institute of Technology, and collaborators introduce HarnessDev, a benchmark that shifts LLM evaluation from task outputs to the model's ability to create and evolve its own agent execution harness, demonstrating that while generated harnesses match or exceed human-engineered references in writing and machine-learning experimentation, they lag behind in code and search, with unstable evolution gains and limited cross-model transfer.

Key Contributions

• HarnessDev is a benchmark evaluating language model agents on building and improving runnable execution infrastructure through Creation from a minimal seed and Evolution via downstream feedback, spanning 2,207 instances across four domains and five hidden evaluation benchmarks.
• In Creation experiments, six frontier language models produce harnesses that match or exceed human-engineered references on writing and machine-learning tasks but lag on code and search, with large variation in execution cost.
• Evolution results show limited, unstable performance gains that transfer only partially to held-out tasks and depend strongly on the executing model, indicating weak cross-model generalization.

Introduction

As LLM-based agents move into real-world deployment, the surrounding execution infrastructure (the agent harness) heavily shapes their effectiveness; even with identical model weights, different harnesses can yield dramatically different downstream performance. Most agent evaluations treat the harness as a fixed experimental configuration rather than an artifact to be developed, leaving a gap in measuring a model’s ability to build and iteratively improve its own runnable execution systems. The authors introduce HarnessDev, a benchmark that shifts evaluation from task outputs to the harness itself by assessing whether a model can both create a harness from a weak seed and continuously evolve it using feedback, while measuring capability, efficiency, and transfer across runtime models.

Dataset

The HarnessDev benchmark is a collection of downstream task instances used to evaluate a model’s ability to build and improve execution harnesses, not to train the model itself. The dataset composition and usage are as follows:

• Overall scale and sources: The benchmark contains 2,207 unique downstream instances drawn from five benchmarks across four domains (detailed in Table 2 of the paper). The authors focus on two code-domain benchmarks for the Evolution stage: SWE-Pro and Terminal-Bench. These are mature, open-source task suites; the paper fixes their versions and licenses.

• SWE-Pro subset:

  • A public split of 731 instances is used during the Creation stage, where the model never sees the full set but receives only 1–3 development cases per task family.
  • From this public split, a fixed subset of 100 tasks is designated as the feedback set for the Evolution stage. The remaining 630 instances form a held-out split, used exclusively to measure generalization after harness development and never shown to the creator.

• Terminal-Bench subset:

  • All 89 tasks are used as the feedback set during Evolution. No separate held-out split is mentioned; these tasks serve as the development signal for that stage.

• How the data is used:

  • Creation (RQ1): The model receives a task-family specification, a weak seed harness, and 1–3 development cases. It must build a harness that generalizes to unseen tasks. The harness is frozen and evaluated on the full downstream benchmarks (including the SWE-Pro public split and other domain tasks).
  • Evolution (RQ2): Starting from its own frozen Creation harness, the model receives execution results on the 100-task SWE-Pro feedback set and all 89 Terminal-Bench tasks. It uses these results to improve the harness. After each improvement, the harness is frozen and evaluated on both feedback sets (for online adaptation scores) and, crucially, on the 630 held-out SWE-Pro instances (for generalization scores, never shown to the model). A budget of ten full-evaluation pairs is allowed, with limited diagnostic probes in between.

• Processing and metadata: No cropping or data augmentation is applied. The harness must produce structured artifacts (trajectories, logs, results) that are scored by a predefined evaluator J. The development environment provides a mutable workspace, and the seed harness supplies only a compatibility layer (parsing, tool access, artifact writing) with no agent logic, ensuring that any nonzero score comes from the creator’s additions.

Method

The authors propose HarnessDev, a benchmark designed to evaluate the execution system that a model develops rather than the answer it produces for a single task. The core pipeline involves a creator LLM $L_C$LC​ working inside a development environment $D$D to produce a runnable harness $H$H. Once $H$H is frozen, an executor LLM $L_E$LE​ runs inside it on a downstream task $x$x, and an evaluator $J$J scores the resulting output $y$y. This process is formalized as:

To measure whether a model can design an execution system from scratch, the authors provide every creator with a weak seed harness $H_{seed}$Hseed​. This seed acts as a runnable compatibility layer rather than a task-solving agent. It parses task and model configuration, exposes permitted low-level tools, and writes required results, trajectories, logs, and task artifacts. Crucially, the seed lacks an agent loop, task decomposition, tool policy, context management, persistent task state, verifier, retry or recovery logic, or stopping rule. Unmodified, it produces an empty or partial artifact and scores zero on downstream benchmarks.

As illustrated in the framework diagram above, the weak seed provides a unified input contract and a public development loop. The creator agent may revise the harness using visible feedback from public tasks, while hidden evaluation tasks, answers, and official scores remain inaccessible. The seed exposes passive primitives such as paths, files, search, process, LLM gateway, and artifact I/O, but these are available and not orchestrated.

To transform this weak seed into a functional system, the creator must implement a comprehensive control layer.

As shown in the figure below, the created runnable harness $H$H integrates a control plane implemented by the creator. This control plane consists of six key modules: Loop (E), Tools (T), Context (C), State (S), Lifecycle (L), and Verify (V). These six mechanisms jointly govern downstream task execution within the harness execution core. The finished harness reports through a unified audit contract, generating artifacts such as result.json, trajectory.json, response.md, and runtime logs. A domain finalizer then maps the run into the domain authoritative scorer-readable artifact, which varies by domain (e.g., code repository state, data or MLE submission, writing prose, or search answers with cited evidence).

The benchmark studies two stages of harness development. In the Creation stage, the creator receives the weak seed $H_{seed}$Hseed​, a task-family specification, tool and permission constraints, a design tutorial, and one to three development cases. It revises the harness using feedback from those cases to build infrastructure that generalizes to unseen tasks. The resulting harness $H$H is frozen before evaluation. In the Evolution stage, the creator starts from its own frozen Creation harness $H_0$H0​. During development, it receives results from a fixed feedback set. The controller evaluates $H_0$H0​ on benchmarks, and each official post $H_0$H0​ candidate is frozen and submitted as a pair of evaluations. The creator is provided a budget of post $H_0$H0​ full-evaluation pairs and may use fixed-subset probes for diagnostics between charged pairs. Generalization is measured separately after freezing on held-out instances that are never shown to the creator during the development loop.

Experiment

The HarnessDev benchmark evaluates LLMs on constructing and iteratively improving execution harnesses from a minimal seed, isolating harness design from single-task solving. Creation experiments show models can build functional harnesses but performance varies sharply by domain, with common pitfalls like dead code and executor-specific overfitting. Evolution from downstream feedback yields only modest, often overfit gains, and harness improvements rarely generalize across executors or held-out tasks, highlighting the difficulty of developing robust, reusable infrastructure.

Harness development is evaluated in two stages: Creation, where a model builds an execution system from a minimal seed, and Evolution, where it refines that system using feedback from downstream tasks. Creation performance varies sharply by domain, while Evolution is non-monotonic and noisy, with visible feedback often failing to predict held-out quality. Creation starts from a weak seed that provides only a compatibility layer, so any nonzero score comes entirely from the creator's added execution logic. Evolution uses results from a designated feedback set, but improvements are frequently erased by later changes and more updates do not guarantee a better final harness. Visible feedback scores and held-out performance move in the same direction only about half the time, and creator-declared best versions rarely match the actual held-out optimum. Under self-evaluation, models match or exceed human performance in writing and machine-learning experimentation but remain far behind in search, research, and code. Changing only the executor during evolution can substantially shift the baseline and alter which harness modifications prove useful.

Creation benchmarks span four domains with 2,207 total tasks across five downstream evaluations. The emphasis is on code via SWE-bench Pro and Terminal-Bench, though data analysis, writing, and research are also covered. Performance varies sharply by domain, with models matching or exceeding human references only in writing and ML experimentation while lagging elsewhere. Creation includes 2,207 unique instances distributed among code (731 SWE-bench Pro, 89 Terminal-Bench), data analysis (75 MLE-bench), writing (46 EQ-Bench3), and research (1,266 BrowseComp). Code tasks dominate the suite, using task success as the primary metric, while other domains employ accuracy, rubric scores, or medal scores. Under self-evaluation, models match the human reference in writing and exceed it in machine-learning experimentation, but remain far behind in search, research, and code.

Under self-evaluation, harness quality differs sharply by domain, with Opus 4.8 achieving the highest average score (67.8) but still trailing the human-engineered system (86.2). Writing harnesses approach the reference, while search and code tasks show the largest gaps, and over three-quarters of failed Data tasks are attributed to harness defects rather than executor limitations. Implementation style and edit volume do not determine success: Gemini 3.1 Pro attains the best Terminal-Bench score (68.8) with the fewest added lines, suggesting that focused, verified changes matter more than code quantity. Opus 4.8 leads self-evaluated creation with an average score of 67.8, yet remains well below the human system-level score of 86.2. Generated writing harnesses nearly match human performance, while search and code harnesses fall far behind, and 77.8% of Data-task failures stem from harness defects. Gemini 3.1 Pro adds only 1,006 lines but achieves the highest Terminal-Bench accuracy (68.8), showing that compact, verified edits can outperform larger rewrites.

Under a fixed Gemini executor, harness creation performance varies sharply across domains and creators. Gemini's own harness achieves the highest average success, while other creators show domain-specific strengths and some harnesses contain performance-hurting artifacts such as a collapsed replica. Harness defects are a major bottleneck, and small, verified edits can outperform larger code additions. Gemini's harness attains the top average success (55.6) under fixed-executor evaluation, followed by Opus (53.3) and Qwen (52.8). Removing a collapsed replica from Opus's SWE-Pro harness would raise its score from 33.0 to 49.1; for DeepSeek, SWE-Pro improves from 29.2 to 43.8 and Terminal-Bench from 38.2 to 57.3. GPT-5.5's EQ-Bench3 mean of 46.5 jumps to 69.7 after excluding a zero-valued first harness, showing high sensitivity to a single poor generation. Harness defects cause 77.8% of failed Data tasks, making harness design a more critical bottleneck than executor capability. An Opus Code harness that performs well under Self-Eval nearly collapses under the fixed Gemini executor due to a hard-coded 120-step limit, illustrating cross-executor fragility. Gemini adds the fewest code lines among creators yet leads Terminal-Bench, indicating that focused, verified changes can be more effective than large code additions.

The Code harness artifacts exhibit large variation in net lines added, but edit size does not predict downstream performance. Gemini 3.1 Pro wrote the least code yet achieved the top Terminal-Bench score, while Seed 2.0 Pro wrote far more yet scored lowest across both benchmarks. Gemini 3.1 Pro added only 1,006 net lines across its three Code artifacts and reached a Terminal-Bench score of 68.8, while Seed 2.0 Pro added over three times as many lines (3,294) and scored just 6.0. Among comparable high-line-count creators (net LOC around 3,200–3,500), SWE-Pro scores ranged from 10.8 to 33.5, confirming that the volume of code changes does not determine harness quality.

The evaluation separates harness development into Creation, where models build an execution system from a minimal seed, and Evolution, where they iteratively refine it using task feedback. Self-evaluated creation performance varies sharply by domain, with models matching or exceeding human references only in writing and machine-learning experimentation while remaining far behind in search, research, and code, and harness defects are a dominant failure source. Code edit volume does not predict downstream quality, as compact, verified changes can outperform larger rewrites, and harnesses exhibit cross-executor fragility that can collapse performance under different runtime conditions.