Published work

We tested GPT-4o, GPT-4o-mini, Claude Sonnet, and Claude Opus on 65 real Spring Framework bug tickets, scoring each against the actual commit that fixed the issue. Claude Sonnet achieves 69% pass rate and 0.227 token overlap — 29% better than GPT. Includes per-tier breakdown, cost comparison, and key findings.

Scientific-grade methodology document. Covers corpus construction (exact Jira REST API endpoint), outcome-based tier labelling with indicator function notation, git benchmark extraction algorithm, evaluation protocol with verbatim prompts, Jaccard scoring definition, per-model per-tier statistics with standard deviations, 7 documented limitations, and a shell-script replication guide. Designed for independent replication without access to private data or proprietary models.

Outcome-based tier classification of the full Spring Framework Jira archive using resolution time, watcher count, and assignee experience as ground-truth signals. 22% Automate · 30% Assist · 48% Escalate. Interactive chart showing how the Escalate fraction grows from 36% (2010) to 68% (2020) as easy work is consumed.