AI TrackerFrontier desk · AEST
Back to news
normalaitech

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

arXiv cs.AI · 16 Nov 2026, 05:39 AEST

SWE-CoWork introduces a benchmark for coding agents in software organizations, featuring 101 development assignments across various programming languages and application domains. The study reveals significant gaps in task delivery and highlights the challenges of requirement tracking in autonomous software engineering.

Key facts

Read aloud

SWE-CoWork introduces a benchmark for coding agents in software organizations, featuring one hundred one development assignments across various programming languages and application domains. The study reveals significant gaps in task delivery and highlights the challenges of requirement tracking in autonomous software engineering. A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising one hundred one development assignments across one hundred one real repositories, eleven programming languages, and twelve application domains. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-five point five directs a mean of ninety three percent of first questions about queried cards to their accountable owners across runs but obtains only sixty four percent of colleague-held requirements. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.

Sources in this cluster