AI TrackerFrontier desk · AEST
Back to news
normalaitech

Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

arXiv cs.AI · 7 Oct 2026, 03:57 AEST

Large language models (LLMs) can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. The study introduces BOTTLED, a benchmark for evaluating LLM agents' ability to create cheaper, task-specific solutions under fixed constraints.

Key facts

Read aloud

Large language models L L M s can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. The study introduces BOTTLED, a benchmark for evaluating L L M agents' ability to create cheaper, task-specific solutions under fixed constraints. We call this ability 'bottling': the ability to turn general capabilities into task-specific solutions that balance answer quality and amortised cost. Across ten models and three tasks, we find that strong zero-shot task performance does not reliably translate into strong bottling capabilities. Nevertheless, bottling can yield substantial savings: on query-product relevance classification, Opus five retains about eighty two percent of its zero-shot macro-F1 at roughly six hundred fifty seven times lower reported cost. BOTTLED provides a basis for evaluating and improving agents' ability to invest limited resources in reusable solutions for large, repetitive workloads.

Sources in this cluster