The Real‑SWE benchmark introduced a fundamental change: instead of pulling problems from public repositories, it drops coding agents into private, production‑grade codebases from real companies. This shift exposed a stark performance gap – the leading agent, Claude Fable 5.1, achieved only a 38.8% success rate, meaning it failed more than six out of ten attempts.
Why the Change Matters for Practitioners
Private code is unlikely to have been seen during model training, and Specific Labs estimates that 99% of enterprise tokens are hidden from frontier models. Consequently, the benchmark better reflects the conditions engineers face when integrating AI‑driven tools into existing services. The results suggest that high scores on public benchmarks may overstate real‑world reliability.
Private Code Benchmark Results
Each model received eight attempts per task and was evaluated with its native coding tool – Claude Code for Fable 5.1, Codex CLI for GPT‑6 Astra, and Gemini CLI for Gemini 3.8 Flash. The top three scores were:
Claude Fable 5.1(Claude Code) – 38.8%GPT‑6 Astra(Codex CLI) – 33.8%Gemini 3.8 Flash(Gemini CLI) – 31.2%
All other agents fell below 30%, with the lowest at 16.2% for GPT‑5.6 Sol. No model solved the “analytics stream reducer” task across 64 attempts, and six of the ten tasks posted success rates under 15%.
Operational Failure Patterns
Failure analysis revealed distinct patterns:
- Fable 5.1 missed requirements (36.7%) or hit integration errors (34.7%).
- Astra split failures evenly between integration errors and unverified assumptions (34% each).
- Gemini 3.8 Flash saw integration errors in roughly half of its failed runs.
- GPT‑5.6 Sol made unverified assumptions in 43.3% of failures.
These categories map directly to engineering concerns: requirement gaps demand better prompt engineering or specification extraction, while integration errors highlight the need for robust toolchain wrappers and environment validation.
Implications for Architecture and Tooling
The benchmark underscores that the surrounding scaffolding – the CLI, IDE plugin, or execution wrapper – materially influences outcomes. For example, Fable 5.1 performed differently when run in Cursor versus Claude Code. Practitioners should therefore treat the AI model and its execution environment as a coupled system rather than interchangeable components.
From an operational standpoint, the low success rates on tasks such as billing schedule migration (14.1%) and linearizable scans (4.7%) suggest that relying on AI agents for critical production changes without extensive validation could introduce regressions. Continuous integration pipelines that automatically apply AI‑generated patches would need additional safety gates, such as static analysis, integration testing, and human review.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Engineers should incorporate private‑code style evaluations into their model selection process, focusing on both raw model capability and the reliability of the surrounding tooling. Monitoring for requirement mismatches and integration failures can guide prompt refinement and wrapper development. Until AI agents demonstrate consistent success on real codebases, they are best used as assistive aids rather than autonomous code authors in production environments.
