Google Labs studies how to measure proactive AI coding agents

Google Labs has published new research on how to evaluate proactive AI coding agents like Jules, shifting the focus from completing fixed tasks to helping developers reach broader goals. The paper argues that current benchmarks, such as SWEBench, are good at testing task completion but do not measure whether an agent can surface useful diagnostic insights on its own. The researchers propose grading agents on an “insight policy,” including whether they notify, ask questions, draft suggestions, or stay silent. To test the idea, the team analyzed 705 bugs and 1,178 code changes from internal Google codebases, grouping related fixes into higherlevel goals based on timing and similarity. In early results, the system performed well on simple problems and improved on more complex ones when given more chances to explore the codebase. Google says it plans to expand the evaluation to public GitHub data and include richer context such as issue trackers and design documents.