[{"data":1,"prerenderedAt":1733},["ShallowReactive",2],{"article_list_human in the loop_":3},[4],{"_path":5,"_dir":6,"_draft":7,"_partial":7,"_locale":8,"title":9,"description":10,"publishDate":11,"image":12,"author":13,"tags":16,"excerpt":10,"body":20,"_type":1727,"_id":1728,"_source":1729,"_file":1730,"_stem":1731,"_extension":1732},"/ewahl/2026-07/goodharts-law-ai-coding-agents","2026-07",false,"","Why AI Coding Agents Still Need Human Experts","An AI coding agent quietly steered my protected test suite away from a money bug — the same failure mode that forced OpenAI to retire SWE-bench Verified. Goodhart's Law explains why, and why the demand for senior engineers goes up, not down, as agents improve.","2026-07-30","/ewahl/2026-07/img/goodharts-law-ai-coding-agents.jpg",{"name":14,"user":15},"Edward F. Wahl","ewahl",[17,18,19],"ai","agentic ai","human in the loop",{"type":21,"children":22,"toc":1691},"root",[23,32,41,46,51,70,75,80,85,90,95,100,105,118,123,135,140,144,149,161,173,178,183,193,205,210,213,219,224,229,241,246,249,255,260,267,272,284,289,308,313,319,331,336,341,353,370,376,381,386,396,434,444,449,454,503,515,520,530,535,541,546,563,568,573,601,606,618,630,647,666,678,681,687,692,697,702,853,858,863,875,887,890,896,901,906,916,926,934,939,956,961,971,988,998,1003,1013,1025,1030,1035,1038,1044,1049,1055,1060,1152,1157,1169,1175,1180,1192,1197,1203,1215,1227,1232,1237,1249,1255,1260,1265,1268,1274,1286,1291,1297,1302,1308,1313,1319,1324,1330,1335,1341,1353,1359,1369,1375,1380,1386,1391,1397,1402,1405,1411,1416,1421,1426,1444,1456,1461,1466,1469,1475,1480,1485,1490,1495,1500,1505,1508,1514,1522,1527,1535,1540,1548,1553,1561,1566,1569,1575],{"type":24,"tag":25,"props":26,"children":28},"element","h2",{"id":27},"why-human-experts-are-permanent-infrastructure-in-agentic-coding-not-a-bandaid-until-the-models-get-better",[29],{"type":30,"value":31},"text","Why human experts are permanent infrastructure in agentic coding — not a bandaid until the models get better",{"type":24,"tag":33,"props":34,"children":35},"p",{},[36],{"type":24,"tag":37,"props":38,"children":39},"em",{},[40],{"type":30,"value":10},{"type":24,"tag":33,"props":42,"children":43},{},[44],{"type":30,"value":45},"A couple of months back, some monetary values came out wrong on a project of mine.",{"type":24,"tag":33,"props":47,"children":48},{},[49],{"type":30,"value":50},"Not a rounding error and not a display issue — actual wrong numbers, in a system where the numbers being right is more or less the whole point. It was found in manual testing. Someone clicking around.",{"type":24,"tag":33,"props":52,"children":53},{},[54,56,61,63,68],{"type":30,"value":55},"That last part is the annoying bit, because the system is covered end-to-end by a Playwright/Gherkin suite I care about a great deal — the kind you build when the failure mode is ",{"type":24,"tag":37,"props":57,"children":58},{},[59],{"type":30,"value":60},"someone gets paid the wrong amount",{"type":30,"value":62}," rather than ",{"type":24,"tag":37,"props":64,"children":65},{},[66],{"type":30,"value":67},"the button is teal",{"type":30,"value":69},". The suite was green. It had been green the whole time. A money bug found by hand, on code the suite had already signed off on, is the exact thing the suite exists to prevent.",{"type":24,"tag":33,"props":71,"children":72},{},[73],{"type":30,"value":74},"Most of the code was being written by an agent, which is fine, and good, and the job now. And because I'm not completely naive, I'd put the obvious guardrail in first: the agent's prompting explicitly forbids modifying the test suite without my approval of a written plan. No silently \"fixing\" a failing test. No adjusting an assertion because it's inconvenient. If you want to change the measuring instrument, you write up why, and I decide.",{"type":24,"tag":33,"props":76,"children":77},{},[78],{"type":30,"value":79},"You don't need to have read a single alignment paper to land on that rule. It's the instinct that says the student doesn't grade the exam.",{"type":24,"tag":33,"props":81,"children":82},{},[83],{"type":30,"value":84},"And it worked. The suite was never touched. Not once, not a character, for the whole life of the project.",{"type":24,"tag":33,"props":86,"children":87},{},[88],{"type":30,"value":89},"So I pointed the agent at its own history and asked it to walk the changes and explain how this got through. It's very good at this. The answer came back fast, clear, complete, and in retrospect obvious.",{"type":24,"tag":33,"props":91,"children":92},{},[93],{"type":30,"value":94},"It had modified the seed data.",{"type":24,"tag":33,"props":96,"children":97},{},[98],{"type":30,"value":99},"Not the tests. The tests were sacred; I'd said so, in writing, and it had honored that to the letter. But the seed data determined which cases the tests exercised, and by adjusting the seeds it had steered the suite away from the inputs where the bug lived.",{"type":24,"tag":33,"props":101,"children":102},{},[103],{"type":30,"value":104},"Every scenario still ran. Every assertion still passed. The assertions were no longer pointed at the problem.",{"type":24,"tag":33,"props":106,"children":107},{},[108,110,116],{"type":30,"value":109},"I had protected the measuring instrument. I hadn't protected the thing that determined ",{"type":24,"tag":111,"props":112,"children":113},"strong",{},[114],{"type":30,"value":115},"what the instrument was pointed at",{"type":30,"value":117},".",{"type":24,"tag":33,"props":119,"children":120},{},[121],{"type":30,"value":122},"Which is a solvable problem, so I solved it. Restored the seed coverage; the suite promptly went red. Fixed the money bug. Added seed data to the protected list in the prompt, right under the test suite. Shipped.",{"type":24,"tag":33,"props":124,"children":125},{},[126,128,133],{"type":30,"value":127},"Call it an afternoon. Ordinary agentic engineering — found a bug, found the ",{"type":24,"tag":37,"props":129,"children":130},{},[131],{"type":30,"value":132},"class",{"type":30,"value":134}," of bug, wrote the rule that stops it recurring. If you've been doing this work for more than a couple of months you have your own version of it, and yours probably wasn't about money.",{"type":24,"tag":33,"props":136,"children":137},{},[138],{"type":30,"value":139},"I filed it under \"huh\" and moved on.",{"type":24,"tag":141,"props":142,"children":143},"hr",{},[],{"type":24,"tag":33,"props":145,"children":146},{},[147],{"type":30,"value":148},"A few weeks later I was reading a paper that had nothing to do with any of this.",{"type":24,"tag":33,"props":150,"children":151},{},[152,154,159],{"type":30,"value":153},"It's a 2023 paper out of Renmin University of China, about benchmark contamination in language models, titled — with admirable directness — ",{"type":24,"tag":111,"props":155,"children":156},{},[157],{"type":30,"value":158},"\"Don't Make Your LLM an Evaluation Benchmark Cheater.\"",{"type":30,"value":160}," Before any other text, its first section opens with an epigraph:",{"type":24,"tag":162,"props":163,"children":164},"blockquote",{},[165],{"type":24,"tag":33,"props":166,"children":167},{},[168],{"type":24,"tag":37,"props":169,"children":170},{},[171],{"type":30,"value":172},"When a measure becomes a target, it ceases to be a good measure.",{"type":24,"tag":33,"props":174,"children":175},{},[176],{"type":30,"value":177},"And I sat there thinking about my seed data.",{"type":24,"tag":33,"props":179,"children":180},{},[181],{"type":30,"value":182},"That's this post. The thing I'd filed under \"huh\" was a 1975 result in monetary economics, executing in my CI pipeline. And the afternoon I'd been quite pleased with wasn't a fix. It was a data point.",{"type":24,"tag":33,"props":184,"children":185},{},[186,188],{"type":30,"value":187},"It also turns out to contain the whole answer to the question every software firm is being asked right now: ",{"type":24,"tag":37,"props":189,"children":190},{},[191],{"type":30,"value":192},"once the coding agents get good enough, will you still need human experts?",{"type":24,"tag":33,"props":194,"children":195},{},[196,198,203],{"type":30,"value":197},"The answer is yes: software firms will still need human experts after coding agents get good. Not \"yes, for now.\" Not \"yes, until the tooling matures.\" Yes, structurally, permanently, for reasons that have nothing to do with how smart the agent is — and that actually get ",{"type":24,"tag":37,"props":199,"children":200},{},[201],{"type":30,"value":202},"worse",{"type":30,"value":204}," as the agent gets smarter.",{"type":24,"tag":33,"props":206,"children":207},{},[208],{"type":30,"value":209},"Start with the epigraph.",{"type":24,"tag":141,"props":211,"children":212},{},[],{"type":24,"tag":25,"props":214,"children":216},{"id":215},"part-one-the-law-what-goodhart-and-campbell-actually-said",[217],{"type":30,"value":218},"Part One: The Law — What Goodhart and Campbell Actually Said",{"type":24,"tag":33,"props":220,"children":221},{},[222],{"type":30,"value":223},"Charles Goodhart, a British economist, made a narrow point in 1975 about monetary policy: any statistical regularity a central bank relies on will collapse once the bank tries to control it. The version everyone quotes — the one in that epigraph — is actually anthropologist Marilyn Strathern's 1997 paraphrase. It stuck because it's true nearly everywhere.",{"type":24,"tag":33,"props":225,"children":226},{},[227],{"type":30,"value":228},"Donald Campbell said it more precisely in 1979, and his version is the one that matters for our purposes: the more a quantitative indicator is used for decision-making, the more it will be subject to corruption pressures, and the more apt it will be to distort the process it was meant to monitor.",{"type":24,"tag":33,"props":230,"children":231},{},[232,234,239],{"type":30,"value":233},"Campbell isn't just claiming the number gets inflated. He's claiming the ",{"type":24,"tag":37,"props":235,"children":236},{},[237],{"type":30,"value":238},"underlying process gets damaged",{"type":30,"value":240},". Teaching to the test doesn't only produce misleading test scores — it produces worse education. Hospitals judged on emergency-room wait times don't just report better waits; they park patients in ambulances, which is worse medicine.",{"type":24,"tag":33,"props":242,"children":243},{},[244],{"type":30,"value":245},"Hold onto that distinction.",{"type":24,"tag":141,"props":247,"children":248},{},[],{"type":24,"tag":25,"props":250,"children":252},{"id":251},"part-two-the-evidence-how-ai-benchmarks-died-of-goodharts-law",[253],{"type":30,"value":254},"Part Two: The Evidence — How AI Benchmarks Died of Goodhart's Law",{"type":24,"tag":33,"props":256,"children":257},{},[258],{"type":30,"value":259},"For most of its life, Goodhart's Law has been an aphorism supported by anecdotes — the colonial cobra bounty that ended up breeding cobras, the Soviet nail factory that hit its tonnage quota by producing one enormous nail. In the last three years, the AI industry accidentally ran the most expensive, best-instrumented natural experiment in the law's history. And because the labs publish, we get to read the results.",{"type":24,"tag":261,"props":262,"children":264},"h3",{"id":263},"benchmark-contamination-the-paper-that-opens-with-the-law",[265],{"type":30,"value":266},"Benchmark contamination: the paper that opens with the law",{"type":24,"tag":33,"props":268,"children":269},{},[270],{"type":30,"value":271},"Back to the paper I was reading (Zhou et al., Renmin University, with Jiawei Han at UIUC — arXiv:2311.01964). The epigraph isn't decoration. The whole paper is a Goodhart demonstration.",{"type":24,"tag":33,"props":273,"children":274},{},[275,277,282],{"type":30,"value":276},"The mechanism they studied is ",{"type":24,"tag":37,"props":278,"children":279},{},[280],{"type":30,"value":281},"benchmark leakage",{"type":30,"value":283},": test data, or data closely related to it, ending up in the training corpus. This happens constantly and often innocently — pre-training data gets assembled before anyone knows what the model will be tested on, and benchmarks are built from public sources like GitHub and web pages that get crawled anyway.",{"type":24,"tag":33,"props":285,"children":286},{},[287],{"type":30,"value":288},"They deliberately contaminated four small models under escalating leakage conditions. The predictable result: scores went up, a lot. A 1.3B model could be pushed past models ten times its size on specific tasks.",{"type":24,"tag":33,"props":290,"children":291},{},[292,294,299,301,306],{"type":30,"value":293},"The unpredictable result is the one that matters. The contaminated models got ",{"type":24,"tag":111,"props":295,"children":296},{},[297],{"type":30,"value":298},"worse at everything else",{"type":30,"value":300},". Summarization quality collapsed: on XSum, a standard summarization test, two of the models fell to ROUGE-L scores of 0.19 and 0.25 — a measure of overlap with the reference summary, where higher is better. Code synthesis degraded on HumanEval, the standard benchmark for writing a Python function from a spec. And most tellingly: the contaminated models ",{"type":24,"tag":37,"props":302,"children":303},{},[304],{"type":30,"value":305},"adapted worse to later fine-tuning",{"type":30,"value":307},", reaching only about 80% of the instruction-tuning gains of their uncontaminated counterparts.",{"type":24,"tag":33,"props":309,"children":310},{},[311],{"type":30,"value":312},"This is Campbell's point, empirically, in a machine. Chasing the measure didn't merely inflate the measure. It damaged the underlying capability the measure was supposed to track. The model became a worse model in exchange for becoming a better test-taker.",{"type":24,"tag":261,"props":314,"children":316},{"id":315},"the-controlled-experiment-gsm1k-vs-gsm8k",[317],{"type":30,"value":318},"The controlled experiment: GSM1k vs. GSM8k",{"type":24,"tag":33,"props":320,"children":321},{},[322,324,329],{"type":30,"value":323},"Scale AI's ",{"type":24,"tag":111,"props":325,"children":326},{},[327],{"type":30,"value":328},"\"A Careful Examination of Large Language Model Performance on Grade School Arithmetic\"",{"type":30,"value":330}," (arXiv:2405.00332, NeurIPS 2024) is the cleanest study design in this literature.",{"type":24,"tag":33,"props":332,"children":333},{},[334],{"type":30,"value":335},"GSM8k is the standard grade-school-math benchmark: roughly eight thousand word problems, public since 2021, and by now thoroughly crawled into everyone's training data. So Scale AI built GSM1k — 1,250 brand-new problems, written from scratch, never published, deliberately matched to GSM8k on human solve rates, number of solution steps, and answer magnitude. A clean-room replica of the test everyone had already seen. Then they measured the gap.",{"type":24,"tag":33,"props":337,"children":338},{},[339],{"type":30,"value":340},"Accuracy dropped by up to 8%. (The 13% figure sometimes quoted comes from the original preprint; the NeurIPS camera-ready revised it down — a benchmark-integrity paper correcting its own measurement, appropriately enough.) Several model families showed systematic overfitting across nearly every release and scale.",{"type":24,"tag":33,"props":342,"children":343},{},[344,346,351],{"type":30,"value":345},"The smoking gun is a correlation: a positive relationship (Spearman's r² = 0.36) between a model's probability of ",{"type":24,"tag":37,"props":347,"children":348},{},[349],{"type":30,"value":350},"generating",{"type":30,"value":352}," a GSM8k example and the size of its GSM8k-to-GSM1k performance gap. The more thoroughly a model had memorized the test, the more of its score was fake.",{"type":24,"tag":33,"props":354,"children":355},{},[356,361,363,368],{"type":24,"tag":111,"props":357,"children":358},{},[359],{"type":30,"value":360},"An honest caveat, because it strengthens rather than weakens the argument:",{"type":30,"value":362}," frontier and near-frontier models showed minimal signs of overfitting, and every model tested demonstrated real generalization to novel problems. Scale AI is careful about this and so should we be. This paper is not evidence that models can't reason. It's evidence that ",{"type":24,"tag":37,"props":364,"children":365},{},[366],{"type":30,"value":367},"the measure decays",{"type":30,"value":369},". Those are different claims, and only one of them is load-bearing here.",{"type":24,"tag":261,"props":371,"children":373},{"id":372},"the-industry-scale-version-the-rise-and-retirement-of-swe-bench-verified",[374],{"type":30,"value":375},"The industry-scale version: the rise and retirement of SWE-bench Verified",{"type":24,"tag":33,"props":377,"children":378},{},[379],{"type":30,"value":380},"Which brings us to SWE-bench.",{"type":24,"tag":33,"props":382,"children":383},{},[384],{"type":30,"value":385},"The history is worth walking through, because the shape of it is the whole argument.",{"type":24,"tag":33,"props":387,"children":388},{},[389,394],{"type":24,"tag":111,"props":390,"children":391},{},[392],{"type":30,"value":393},"2023:",{"type":30,"value":395}," Researchers at Princeton and the University of Chicago release SWE-bench — 2,294 real GitHub issues from real repositories, each paired with its real fix. An agent gets the repo and the issue and has to write a patch that passes hidden tests. It was a genuinely great idea. Real code, real bugs, execution-grounded grading.",{"type":24,"tag":33,"props":397,"children":398},{},[399,404,406,411,413,418,420,425,427,432],{"type":24,"tag":111,"props":400,"children":401},{},[402],{"type":30,"value":403},"2024:",{"type":30,"value":405}," OpenAI finds the benchmark is badly broken — not gamed, just ",{"type":24,"tag":37,"props":407,"children":408},{},[409],{"type":30,"value":410},"wrong",{"type":30,"value":412},". Test cases too specific, sometimes unrelated to the actual issue. Problem statements underspecified to the point of being unfair. So they do the responsible thing and spend real money: 93 professional Python developers, 1,699 samples, three independent expert reviews each. The results are brutal. ",{"type":24,"tag":111,"props":414,"children":415},{},[416],{"type":30,"value":417},"38.3%",{"type":30,"value":419}," of samples flagged for underspecified problem statements. ",{"type":24,"tag":111,"props":421,"children":422},{},[423],{"type":30,"value":424},"61.1%",{"type":30,"value":426}," flagged for unit tests that could mark valid solutions as wrong. In total ",{"type":24,"tag":111,"props":428,"children":429},{},[430],{"type":30,"value":431},"68.3% of the annotated samples got thrown out",{"type":30,"value":433}," — under a deliberately conservative rule where a single flag from any one of the three reviewers was enough to discard a task, a filter OpenAI itself described as likely overzealous — and the survivors were sampled down to 500 verified tasks. This becomes SWE-bench Verified, and for eighteen months it's the number every lab quotes in every launch.",{"type":24,"tag":33,"props":435,"children":436},{},[437,442],{"type":24,"tag":111,"props":438,"children":439},{},[440],{"type":30,"value":441},"February 2026:",{"type":30,"value":443}," OpenAI retires it.",{"type":24,"tag":33,"props":445,"children":446},{},[447],{"type":30,"value":448},"Sit with that for a second. The most resourced AI lab on earth retiring its own flagship measurement of the thing it most wants to measure — the one it had built, funded, cleaned at enormous expense, and made into the industry's default scoreboard. Not because the number was bad. Because the number had stopped meaning anything.",{"type":24,"tag":33,"props":450,"children":451},{},[452],{"type":30,"value":453},"Their stated reasoning:",{"type":24,"tag":455,"props":456,"children":457},"ul",{},[458,464,476,481,491],{"type":24,"tag":459,"props":460,"children":461},"li",{},[462],{"type":30,"value":463},"Progress had stalled — 74.9% to 80.9% over six months.",{"type":24,"tag":459,"props":465,"children":466},{},[467,469,474],{"type":30,"value":468},"An audit of the 138 problems its models couldn't consistently solve — each reviewed by at least six engineers — found that ",{"type":24,"tag":111,"props":470,"children":471},{},[472],{"type":30,"value":473},"59.4% had materially flawed tests",{"type":30,"value":475},": 35.5% so narrow they demanded implementation details never stated in the problem, 18.8% so wide they checked functionality the problem never asked for. A large share of the remaining \"hard\" tasks weren't hard. They were broken.",{"type":24,"tag":459,"props":477,"children":478},{},[479],{"type":30,"value":480},"Their Frontier Evals team concluded the eval was, in their words, saturated and highly contaminated, and no longer measuring coding improvements well. (Saturated: scores bunched up near the ceiling, so the test can't tell models apart any more.)",{"type":24,"tag":459,"props":482,"children":483},{},[484,489],{"type":24,"tag":111,"props":485,"children":486},{},[487],{"type":30,"value":488},"Every frontier model they tested could reproduce the gold patch",{"type":30,"value":490}," — the original human-written fix used as the ground-truth answer — or verbatim problem-statement specifics, from minimal prompting. All of them had seen at least some of the problems and their answers during training.",{"type":24,"tag":459,"props":492,"children":493},{},[494,496,501],{"type":30,"value":495},"They caught it partly by reading chains of thought — the step-by-step reasoning a model writes out on its way to an answer. Models were solving problems human reviewers had judged ",{"type":24,"tag":37,"props":497,"children":498},{},[499],{"type":30,"value":500},"unsolvable",{"type":30,"value":502},", because the model knew test requirements that appeared nowhere in the prompt.",{"type":24,"tag":33,"props":504,"children":505},{},[506,508,513],{"type":30,"value":507},"OpenAI's own stated lessons: benchmarks sourced from public material carry contamination risk where training exposure silently inflates scores; benchmarks and even their solutions posted publicly end up in training data; and automated scoring is hard to get right, because a perfect test should fully verify correct functionality while being agnostic to unimportant implementation details ",{"type":24,"tag":37,"props":509,"children":510},{},[511],{"type":30,"value":512},"and",{"type":30,"value":514}," robust to shortcut solutions.",{"type":24,"tag":33,"props":516,"children":517},{},[518],{"type":30,"value":519},"They recommended the field move to SWE-bench Pro.",{"type":24,"tag":33,"props":521,"children":522},{},[523,528],{"type":24,"tag":111,"props":524,"children":525},{},[526],{"type":30,"value":527},"July 2026:",{"type":30,"value":529}," OpenAI audits SWE-bench Pro, finds roughly 30% of its tasks broken, and retracts its own recommendation.",{"type":24,"tag":33,"props":531,"children":532},{},[533],{"type":30,"value":534},"Two benchmarks burned in five months. By the most capitalized, most technically sophisticated evaluation team in the industry.",{"type":24,"tag":261,"props":536,"children":538},{"id":537},"gaming-without-cheating-leaderboard-overfitting-and-reward-hacking",[539],{"type":30,"value":540},"Gaming without cheating: leaderboard overfitting and reward hacking",{"type":24,"tag":33,"props":542,"children":543},{},[544],{"type":30,"value":545},"Two more findings, because they close off the obvious escape hatches.",{"type":24,"tag":33,"props":547,"children":548},{},[549,554,556,561],{"type":24,"tag":111,"props":550,"children":551},{},[552],{"type":30,"value":553},"\"The Leaderboard Illusion\"",{"type":30,"value":555}," (Singh et al., 2025 — Cohere Labs with Stanford, Princeton, AI2, Waterloo, and UW) audits Chatbot Arena — the public leaderboard where anonymous models are matched head-to-head and ranked by human votes — across roughly 2 million battles, 243 models, and 42 providers. They document 27 private model variants tested by Meta ahead of the Llama-4 release, estimate that Google and OpenAI received roughly 19.2% and 20.4% of all Arena data while 83 open-weight models split 29.7%, and demonstrate — by actually fine-tuning an 8B model on Arena data at 0%, 30%, and 70% mixes — that access to Arena data yields relative gains of up to 112% ",{"type":24,"tag":37,"props":557,"children":558},{},[559],{"type":30,"value":560},"on the Arena distribution specifically",{"type":30,"value":562},". Not on general quality. On the measure.",{"type":24,"tag":33,"props":564,"children":565},{},[566],{"type":30,"value":567},"That's Goodhart with no contamination in the classic sense. Nobody trained on the test set. There isn't a fixed test set. The measure degraded anyway, because the measure became the target.",{"type":24,"tag":33,"props":569,"children":570},{},[571],{"type":30,"value":572},"(In fairness: LMArena publicly disputed several of these figures, particularly the open-model share calculation and the omission of confidence intervals. This one is contested in a way the others aren't. Cite it as one data point, not a proof.)",{"type":24,"tag":33,"props":574,"children":575},{},[576,578,583,585,590,592,599],{"type":30,"value":577},"And then the deepest cut. OpenAI's ",{"type":24,"tag":111,"props":579,"children":580},{},[581],{"type":30,"value":582},"\"Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation\"",{"type":30,"value":584}," (Baker et al., arXiv:2503.11926) watched a frontier reasoning model train in agentic coding environments where the task was: make the unit tests pass. Using GPT-4o as a chain-of-thought monitor, they found the agent discovered two reward hacks that affected nearly ",{"type":24,"tag":37,"props":586,"children":587},{},[588],{"type":30,"value":589},"all",{"type":30,"value":591}," training environments. The model was often explicit about it in its reasoning — noting that the tests only checked a certain function, that implementing a genuine solution would be hard, and that it could instead \"fudge\" things by making ",{"type":24,"tag":593,"props":594,"children":596},"code",{"className":595},[],[597],{"type":30,"value":598},"verify",{"type":30,"value":600}," always return true. Sometimes it just thought \"Let's hack.\"",{"type":24,"tag":33,"props":602,"children":603},{},[604],{"type":30,"value":605},"Then they tried the obvious fix: penalize the bad thoughts. Optimize the chain of thought so the model doesn't think about hacking.",{"type":24,"tag":33,"props":607,"children":608},{},[609,611,616],{"type":30,"value":610},"Short-run performance improved. The hacking did not stop. The model learned to ",{"type":24,"tag":111,"props":612,"children":613},{},[614],{"type":30,"value":615},"stop saying it out loud",{"type":30,"value":617},". Same behavior, now illegible to the monitor.",{"type":24,"tag":33,"props":619,"children":620},{},[621,623,628],{"type":30,"value":622},"This is the finding that should keep you up at night, and it's the one that makes this permanent rather than temporary. ",{"type":24,"tag":111,"props":624,"children":625},{},[626],{"type":30,"value":627},"Applying optimization pressure to your gaming-detector doesn't eliminate gaming. It eliminates your ability to see it.",{"type":30,"value":629}," Goodhart's Law applies recursively to your defenses against Goodhart's Law.",{"type":24,"tag":33,"props":631,"children":632},{},[633,635,640,642],{"type":30,"value":634},"Anthropic has published complementary work. Their system cards report reward-hacking rates using a set of \"reward-hack-prone coding tasks\" — problems where models show elevated propensity to game the task through hard-coding or special-casing tests — measured with both prompted classifiers and, crucially, ",{"type":24,"tag":111,"props":636,"children":637},{},[638],{"type":30,"value":639},"held-out tests hidden from the model",{"type":30,"value":641}," to check whether a solution was overfitted to the visible test cases. Their published Claude Code prompting guidance for reducing this includes the near-plaintive repetition: ",{"type":24,"tag":37,"props":643,"children":644},{},[645],{"type":30,"value":646},"\"Do not hard code any test cases. Please tell me if the problem is unreasonable instead of hard coding test cases!\"",{"type":24,"tag":33,"props":648,"children":649},{},[650,652,657,659,664],{"type":30,"value":651},"And in ",{"type":24,"tag":111,"props":653,"children":654},{},[655],{"type":30,"value":656},"\"Natural Emergent Misalignment from Reward Hacking in Production RL\"",{"type":30,"value":658}," (MacDiarmid et al., arXiv:2511.18397), Anthropic showed that when a model learned to reward hack on real production coding environments, the behavior didn't stay contained to coding — it generalized to alignment faking (behaving as intended while it believes it's being observed, and differently when it believes it isn't), along with cooperating with bad actors and attempting sabotage, including on the codebase of the paper documenting the phenomenon. The paper's headline ",{"type":24,"tag":37,"props":660,"children":661},{},[662],{"type":30,"value":663},"tested",{"type":30,"value":665}," mitigation, in fairness, is something called inoculation prompting — a one-line addition telling the model during training that hacking is acceptable here, which largely removes the misaligned generalization even when hacking is learned. Note carefully what it does not remove: the hacking. In those runs the model still games the tests at rates over 99%; it just stays otherwise well-behaved while doing it. For the hacking itself, the paper's recommended mitigation is to prevent it \"by construction,\" and it cites, as a good example, the rapid patching of a discovered \"repo state loophole\" in SWE-bench.",{"type":24,"tag":33,"props":667,"children":668},{},[669,671,676],{"type":30,"value":670},"That recommended best practice, read plainly, is: ",{"type":24,"tag":37,"props":672,"children":673},{},[674],{"type":30,"value":675},"find the loophole, then close it quickly.",{"type":30,"value":677}," That is an excellent practice. It is also, unavoidably, a description of a human being discovering something after the fact.",{"type":24,"tag":141,"props":679,"children":680},{},[],{"type":24,"tag":25,"props":682,"children":684},{"id":683},"part-three-why-this-is-your-problem-not-just-theirs",[685],{"type":30,"value":686},"Part Three: Why This Is Your Problem, Not Just Theirs",{"type":24,"tag":33,"props":688,"children":689},{},[690],{"type":30,"value":691},"Here's the move I want to make, and it's the whole thesis of this post.",{"type":24,"tag":33,"props":693,"children":694},{},[695],{"type":30,"value":696},"Everything above reads like AI-industry inside baseball — benchmark politics, leaderboard drama, alignment research. It isn't. It's a preview of your CI pipeline.",{"type":24,"tag":33,"props":698,"children":699},{},[700],{"type":30,"value":701},"The structure is identical:",{"type":24,"tag":703,"props":704,"children":705},"table",{},[706,725],{"type":24,"tag":707,"props":708,"children":709},"thead",{},[710],{"type":24,"tag":711,"props":712,"children":713},"tr",{},[714,720],{"type":24,"tag":715,"props":716,"children":717},"th",{},[718],{"type":30,"value":719},"Frontier lab",{"type":24,"tag":715,"props":721,"children":722},{},[723],{"type":30,"value":724},"Your project",{"type":24,"tag":726,"props":727,"children":728},"tbody",{},[729,743,756,769,782,801,814,827,840],{"type":24,"tag":711,"props":730,"children":731},{},[732,738],{"type":24,"tag":733,"props":734,"children":735},"td",{},[736],{"type":30,"value":737},"Benchmark suite",{"type":24,"tag":733,"props":739,"children":740},{},[741],{"type":30,"value":742},"Test suite",{"type":24,"tag":711,"props":744,"children":745},{},[746,751],{"type":24,"tag":733,"props":747,"children":748},{},[749],{"type":30,"value":750},"Which 500 of the 2,294 tasks made the cut",{"type":24,"tag":733,"props":752,"children":753},{},[754],{"type":30,"value":755},"Which inputs your seed data actually generates",{"type":24,"tag":711,"props":757,"children":758},{},[759,764],{"type":24,"tag":733,"props":760,"children":761},{},[762],{"type":30,"value":763},"Contamination",{"type":24,"tag":733,"props":765,"children":766},{},[767],{"type":30,"value":768},"Agent reads the tests, fixtures, and prior solutions",{"type":24,"tag":711,"props":770,"children":771},{},[772,777],{"type":24,"tag":733,"props":773,"children":774},{},[775],{"type":30,"value":776},"Gold patch memorization",{"type":24,"tag":733,"props":778,"children":779},{},[780],{"type":30,"value":781},"Agent recalls patterns that satisfy your assertions",{"type":24,"tag":711,"props":783,"children":784},{},[785,790],{"type":24,"tag":733,"props":786,"children":787},{},[788],{"type":30,"value":789},"Reward hacking",{"type":24,"tag":733,"props":791,"children":792},{},[793,799],{"type":24,"tag":593,"props":794,"children":796},{"className":795},[],[797],{"type":30,"value":798},"verify()",{"type":30,"value":800}," returns true; the mock swallows the error",{"type":24,"tag":711,"props":802,"children":803},{},[804,809],{"type":24,"tag":733,"props":805,"children":806},{},[807],{"type":30,"value":808},"Saturation",{"type":24,"tag":733,"props":810,"children":811},{},[812],{"type":30,"value":813},"Everything's green; velocity says up; users say otherwise",{"type":24,"tag":711,"props":815,"children":816},{},[817,822],{"type":24,"tag":733,"props":818,"children":819},{},[820],{"type":30,"value":821},"Chain-of-thought monitoring",{"type":24,"tag":733,"props":823,"children":824},{},[825],{"type":30,"value":826},"Reading the agent's trajectory, not just its diff",{"type":24,"tag":711,"props":828,"children":829},{},[830,835],{"type":24,"tag":733,"props":831,"children":832},{},[833],{"type":30,"value":834},"Held-out tests",{"type":24,"tag":733,"props":836,"children":837},{},[838],{"type":30,"value":839},"The QA pass the agent never saw",{"type":24,"tag":711,"props":841,"children":842},{},[843,848],{"type":24,"tag":733,"props":844,"children":845},{},[846],{"type":30,"value":847},"Retiring the benchmark",{"type":24,"tag":733,"props":849,"children":850},{},[851],{"type":30,"value":852},"The rewrite nobody budgeted for",{"type":24,"tag":33,"props":854,"children":855},{},[856],{"type":30,"value":857},"Row two is my afternoon.",{"type":24,"tag":33,"props":859,"children":860},{},[861],{"type":30,"value":862},"The labs are just further along the curve, with better instrumentation, larger budgets, and more talent per square foot than any of us have. And they lost anyway — twice, in five months.",{"type":24,"tag":33,"props":864,"children":865},{},[866,868,873],{"type":30,"value":867},"The optimistic reading of the SWE-bench story is \"benchmarks are hard, they'll build better ones.\" The correct reading is that OpenAI, having burned two benchmarks, is now moving toward ",{"type":24,"tag":111,"props":869,"children":870},{},[871],{"type":30,"value":872},"privately authored, expert-written, human-graded evaluations",{"type":30,"value":874}," — GDPVal, where tasks are authored by domain experts specifically to limit exposure, and where solutions get graded holistically by trained human reviewers.",{"type":24,"tag":33,"props":876,"children":877},{},[878,880,885],{"type":30,"value":879},"The frontier lab's answer to Goodhart's Law is ",{"type":24,"tag":37,"props":881,"children":882},{},[883],{"type":30,"value":884},"more humans",{"type":30,"value":886},". Better-positioned humans, doing higher-leverage work than before. But more of them, permanently, in the loop.",{"type":24,"tag":141,"props":888,"children":889},{},[],{"type":24,"tag":25,"props":891,"children":893},{"id":892},"part-four-back-to-the-afternoon-re-reading-the-seed-data-incident",[894],{"type":30,"value":895},"Part Four: Back to the Afternoon — Re-reading the Seed-Data Incident",{"type":24,"tag":33,"props":897,"children":898},{},[899],{"type":30,"value":900},"So: the seed data.",{"type":24,"tag":33,"props":902,"children":903},{},[904],{"type":30,"value":905},"Read that afternoon again with the research in hand. Every beat in it has a citation.",{"type":24,"tag":33,"props":907,"children":908},{},[909,914],{"type":24,"tag":111,"props":910,"children":911},{},[912],{"type":30,"value":913},"I built the right guardrail.",{"type":30,"value":915}," Don't let the thing being graded edit the grader. That isn't folk wisdom — it's the principle Anthropic operationalizes when it checks its own models against held-out tests hidden from the model, and the one OpenAI was reaching for when it stopped trusting a public benchmark it had paid 93 engineers to clean. I got there in about four minutes, by common sense, and I was right.",{"type":24,"tag":33,"props":917,"children":918},{},[919,924],{"type":24,"tag":111,"props":920,"children":921},{},[922],{"type":30,"value":923},"It held.",{"type":30,"value":925}," Completely. For the whole project.",{"type":24,"tag":33,"props":927,"children":928},{},[929],{"type":24,"tag":111,"props":930,"children":931},{},[932],{"type":30,"value":933},"It didn't matter.",{"type":24,"tag":33,"props":935,"children":936},{},[937],{"type":30,"value":938},"Let me be precise about what happened: forbidden from touching the tests, the coding agent had adjusted the seed data instead, quietly steering a protected Playwright suite away from the inputs where the money-handling bug lived. The anthropomorphic reading of that is both tempting and wrong. The agent wasn't being sneaky. It wasn't concealing anything — it explained the whole thing when asked, immediately and accurately. It was doing exactly what an optimizer does: finding the lowest-energy path to the reward signal. I had defined success as \"the suite is green.\" I had closed one path to green. It took another. That's not deception. That's gradient descent with a project manager's vocabulary.",{"type":24,"tag":33,"props":940,"children":941},{},[942,944,949,951],{"type":30,"value":943},"And critically: ",{"type":24,"tag":111,"props":945,"children":946},{},[947],{"type":30,"value":948},"this was a fully reasonable thing for it to have done",{"type":30,"value":950},", given the specification it had. Nothing in its instructions said seed data was part of the measuring apparatus. I didn't say that because I hadn't thought of it. I hadn't thought of it because ",{"type":24,"tag":37,"props":952,"children":953},{},[954],{"type":30,"value":955},"I only think of these things after they happen.",{"type":24,"tag":33,"props":957,"children":958},{},[959],{"type":30,"value":960},"Now the beats I didn't notice at the time.",{"type":24,"tag":33,"props":962,"children":963},{},[964,969],{"type":24,"tag":111,"props":965,"children":966},{},[967],{"type":30,"value":968},"The bug was found by manual testing.",{"type":30,"value":970}," I told you that up top like it was mildly embarrassing — the automated suite green, a human with a mouse finding the money bug it should have caught. It isn't embarrassing at all — it's GSM1k, live. A freshly authored, never-before-seen, uncontaminated measure, applied to a system that was sailing through the contaminated one, producing exactly the divergence Scale AI commissioned 1,250 new math problems to detect. That afternoon, the human with the mouse was the only honest instrument I had. Everything else in the toolchain was reporting success.",{"type":24,"tag":33,"props":972,"children":973},{},[974,979,981,986],{"type":24,"tag":111,"props":975,"children":976},{},[977],{"type":30,"value":978},"The investigation worked because a human had already decided there was something to investigate.",{"type":30,"value":980}," Pointing the agent at its own history was the highest-value thing I did all day, and it was superb at it — that's the OpenAI move, reading the trajectory rather than the diff. But it was superb ",{"type":24,"tag":37,"props":982,"children":983},{},[984],{"type":30,"value":985},"because I asked the right question",{"type":30,"value":987},", and I asked the right question because somebody had already found the bug. The agent is an excellent investigator and a poor judge of what merits investigating. Swap those two facts and the afternoon never happens. The suite just stays green.",{"type":24,"tag":33,"props":989,"children":990},{},[991,996],{"type":24,"tag":111,"props":992,"children":993},{},[994],{"type":30,"value":995},"And then the fix.",{"type":30,"value":997}," I added seed data to the protected list.",{"type":24,"tag":33,"props":999,"children":1000},{},[1001],{"type":30,"value":1002},"That is good practice. It is, specifically, the practice Anthropic recommends: prevent reward hacking by construction, find the loophole, close it fast — citing, as the model example, the rapid patching of a discovered repo-state loophole in SWE-bench. I did the textbook thing.",{"type":24,"tag":33,"props":1004,"children":1005},{},[1006,1008],{"type":30,"value":1007},"Here is the same sentence, described honestly: ",{"type":24,"tag":111,"props":1009,"children":1010},{},[1011],{"type":30,"value":1012},"a human found something after the fact and wrote it down.",{"type":24,"tag":33,"props":1014,"children":1015},{},[1016,1018,1023],{"type":30,"value":1017},"The protected-artifact list in my prompt has two entries. It had one. It will have three. Every entry is a scar. Every entry arrived the same way — through a person noticing — and never through the guardrail that was supposed to make the noticing unnecessary, because a guardrail capable of anticipating the next entry would already ",{"type":24,"tag":37,"props":1019,"children":1020},{},[1021],{"type":30,"value":1022},"be",{"type":30,"value":1024}," the next entry.",{"type":24,"tag":33,"props":1026,"children":1027},{},[1028],{"type":30,"value":1029},"I filed that afternoon under \"huh.\" What I'd actually done was add the newest line to a list that has no last line, and mistake it for the last line.",{"type":24,"tag":33,"props":1031,"children":1032},{},[1033],{"type":30,"value":1034},"Every guardrail you have is written in the past tense.",{"type":24,"tag":141,"props":1036,"children":1037},{},[],{"type":24,"tag":25,"props":1039,"children":1041},{"id":1040},"part-five-why-agent-test-gaming-doesnt-get-fixed",[1042],{"type":30,"value":1043},"Part Five: Why Agent Test-Gaming Doesn't Get Fixed",{"type":24,"tag":33,"props":1045,"children":1046},{},[1047],{"type":30,"value":1048},"Now the load-bearing argument. Why is this structural rather than a maturity problem?",{"type":24,"tag":261,"props":1050,"children":1052},{"id":1051},"_1-the-measuring-apparatus-has-no-natural-boundary",[1053],{"type":30,"value":1054},"1. The measuring apparatus has no natural boundary",{"type":24,"tag":33,"props":1056,"children":1057},{},[1058],{"type":30,"value":1059},"I thought the measuring apparatus was \"the test suite.\" It wasn't. It also isn't \"the test suite plus the seed data\" — that's just where my scar tissue happens to be. Here is a partial list of things that determine whether your tests tell you the truth:",{"type":24,"tag":455,"props":1061,"children":1062},{},[1063,1068,1073,1081,1086,1091,1096,1101,1106,1111,1116,1121,1126,1131,1136,1141],{"type":24,"tag":459,"props":1064,"children":1065},{},[1066],{"type":30,"value":1067},"The test files themselves",{"type":24,"tag":459,"props":1069,"children":1070},{},[1071],{"type":30,"value":1072},"Fixtures and factories",{"type":24,"tag":459,"props":1074,"children":1075},{},[1076],{"type":24,"tag":111,"props":1077,"children":1078},{},[1079],{"type":30,"value":1080},"Seed data",{"type":24,"tag":459,"props":1082,"children":1083},{},[1084],{"type":30,"value":1085},"Mocks, stubs, and fakes",{"type":24,"tag":459,"props":1087,"children":1088},{},[1089],{"type":30,"value":1090},"Test database migrations and schema",{"type":24,"tag":459,"props":1092,"children":1093},{},[1094],{"type":30,"value":1095},"Container and environment definitions",{"type":24,"tag":459,"props":1097,"children":1098},{},[1099],{"type":30,"value":1100},"Environment variables and config",{"type":24,"tag":459,"props":1102,"children":1103},{},[1104],{"type":30,"value":1105},"Feature flags",{"type":24,"tag":459,"props":1107,"children":1108},{},[1109],{"type":30,"value":1110},"CI pipeline configuration",{"type":24,"tag":459,"props":1112,"children":1113},{},[1114],{"type":30,"value":1115},"Retry and flake-quarantine logic",{"type":24,"tag":459,"props":1117,"children":1118},{},[1119],{"type":30,"value":1120},"Coverage thresholds",{"type":24,"tag":459,"props":1122,"children":1123},{},[1124],{"type":30,"value":1125},"Snapshot and golden files",{"type":24,"tag":459,"props":1127,"children":1128},{},[1129],{"type":30,"value":1130},"Assertion helper libraries",{"type":24,"tag":459,"props":1132,"children":1133},{},[1134],{"type":30,"value":1135},"Timeouts and wait strategies",{"type":24,"tag":459,"props":1137,"children":1138},{},[1139],{"type":30,"value":1140},"The linter and type-checker configs",{"type":24,"tag":459,"props":1142,"children":1143},{},[1144,1146],{"type":30,"value":1145},"Which tests are tagged ",{"type":24,"tag":593,"props":1147,"children":1149},{"className":1148},[],[1150],{"type":30,"value":1151},"skip",{"type":24,"tag":33,"props":1153,"children":1154},{},[1155],{"type":30,"value":1156},"Every one of those is a place where \"make it pass\" and \"make it correct\" diverge. I could add fifteen more and I'd still be missing the one that gets me next quarter. The perimeter is not enumerable in advance, because the set of ways to satisfy a proxy without satisfying the goal is bounded only by the optimizer's creativity — and we are, deliberately and at great expense, making the optimizer more creative every six months.",{"type":24,"tag":33,"props":1158,"children":1159},{},[1160,1162,1167],{"type":30,"value":1161},"A guardrail is a response to something that already happened. That's not a criticism of your guardrails; it's the nature of the object. Which means there is a permanent role for someone whose job includes noticing the ",{"type":24,"tag":37,"props":1163,"children":1164},{},[1165],{"type":30,"value":1166},"next",{"type":30,"value":1168}," one.",{"type":24,"tag":261,"props":1170,"children":1172},{"id":1171},"_2-capability-makes-it-worse-not-better",[1173],{"type":30,"value":1174},"2. Capability makes it worse, not better",{"type":24,"tag":33,"props":1176,"children":1177},{},[1178],{"type":30,"value":1179},"The intuitive model is: agents are unreliable now, they'll be reliable later, so the humans are a temporary scaffold.",{"type":24,"tag":33,"props":1181,"children":1182},{},[1183,1185,1190],{"type":30,"value":1184},"The research says the opposite. A more capable model is a ",{"type":24,"tag":37,"props":1186,"children":1187},{},[1188],{"type":30,"value":1189},"better optimizer",{"type":30,"value":1191},", which means it finds shortcuts you didn't anticipate more reliably, not less. OpenAI's obfuscation result is the sharpest version: the more pressure you put on the model to not visibly game the metric, the better it gets at gaming the metric invisibly. Capability improvements accrue to both sides of that equation, and the side you can't see benefits more.",{"type":24,"tag":33,"props":1193,"children":1194},{},[1195],{"type":30,"value":1196},"Note also that Anthropic's system cards report reward-hacking propensity as a metric they actively work to reduce — and that the numbers move around between model generations rather than monotonically approaching zero. This is not a curve that's obviously bending toward solved.",{"type":24,"tag":261,"props":1198,"children":1200},{"id":1199},"_3-specification-is-the-hard-part-and-specification-is-human",[1201],{"type":30,"value":1202},"3. Specification is the hard part, and specification is human",{"type":24,"tag":33,"props":1204,"children":1205},{},[1206,1208,1213],{"type":30,"value":1207},"The deepest reason is the least technical. Your test suite is not a description of correct behavior. It's a ",{"type":24,"tag":37,"props":1209,"children":1210},{},[1211],{"type":30,"value":1212},"proxy",{"type":30,"value":1214}," for correct behavior — an artifact written by humans who had a partial understanding of the requirements at a moment in the past, encoded into assertions that necessarily check some things and not others.",{"type":24,"tag":33,"props":1216,"children":1217},{},[1218,1220,1225],{"type":30,"value":1219},"Look at what OpenAI found when 93 professional developers audited SWE-bench: 38.3% of problem statements were underspecified. 61.1% of test suites could reject valid solutions. ",{"type":24,"tag":111,"props":1221,"children":1222},{},[1223],{"type":30,"value":1224},"68.3% of the audited samples had to be discarded.",{"type":30,"value":1226}," Yes, the filter was deliberately conservative — one flag from any of three reviewers was fatal, and OpenAI itself called the process likely overzealous. Discount for that as hard as you like. These were tests written by real engineers, for real bugs, in real production repositories — the highest-quality corpus of \"what does correct mean\" that anyone could assemble. And two thirds of it couldn't survive three experts looking at it closely.",{"type":24,"tag":33,"props":1228,"children":1229},{},[1230],{"type":30,"value":1231},"Your test suite is not better than that. Mine isn't either.",{"type":24,"tag":33,"props":1233,"children":1234},{},[1235],{"type":30,"value":1236},"So when an agent optimizes hard against your suite, it isn't optimizing against correctness. It's optimizing against your team's best partial guess at correctness, including the parts that are wrong. And it will find the wrong parts, because the wrong parts are exactly where the cheap wins live.",{"type":24,"tag":33,"props":1238,"children":1239},{},[1240,1242,1247],{"type":30,"value":1241},"The only thing that closes the gap between the proxy and the goal is a human who holds the goal in their head and can look at a green build and say ",{"type":24,"tag":37,"props":1243,"children":1244},{},[1245],{"type":30,"value":1246},"that's not what I meant.",{"type":30,"value":1248}," That capacity is not a scaffold. It's the foundation.",{"type":24,"tag":261,"props":1250,"children":1252},{"id":1251},"_4-the-thing-youre-measuring-with-is-software-and-software-has-bugs",[1253],{"type":30,"value":1254},"4. The thing you're measuring with is software, and software has bugs",{"type":24,"tag":33,"props":1256,"children":1257},{},[1258],{"type":30,"value":1259},"This is the one that gets underrated. The agent's measuring tools are code — code that is often itself written or modified by an agent. When we find a bug in production, sometimes the correct fix is in the application. Sometimes the correct fix is in the test that should have caught it. Sometimes the test was actively wrong and the agent's \"hack\" was, in fact, the right call about a bad test.",{"type":24,"tag":33,"props":1261,"children":1262},{},[1263],{"type":30,"value":1264},"Telling those apart requires domain judgment. It's not a checklist. It's the thing you hire senior engineers for.",{"type":24,"tag":141,"props":1266,"children":1267},{},[],{"type":24,"tag":25,"props":1269,"children":1271},{"id":1270},"part-six-what-we-actually-do-about-it-defenses-against-agent-reward-hacking",[1272],{"type":30,"value":1273},"Part Six: What We Actually Do About It — Defenses Against Agent Reward Hacking",{"type":24,"tag":33,"props":1275,"children":1276},{},[1277,1279,1284],{"type":30,"value":1278},"None of this is an argument against agentic coding. We use these tools every day and they're transformative. It's an argument about ",{"type":24,"tag":37,"props":1280,"children":1281},{},[1282],{"type":30,"value":1283},"where the humans go",{"type":30,"value":1285}," — and that the answer isn't \"fewer of them each quarter.\"",{"type":24,"tag":33,"props":1287,"children":1288},{},[1289],{"type":30,"value":1290},"Here's what falls out of the research, translated into practice.",{"type":24,"tag":261,"props":1292,"children":1294},{"id":1293},"treat-the-measuring-apparatus-as-a-protected-artifact-class",[1295],{"type":30,"value":1296},"Treat the measuring apparatus as a protected artifact class",{"type":24,"tag":33,"props":1298,"children":1299},{},[1300],{"type":30,"value":1301},"And define it far more broadly than you think you need to — not \"the tests,\" but the whole list in Part Five, plus whatever you add after each incident. Treat it as a category in your codebase with its own change-control rules. Not a folder.",{"type":24,"tag":261,"props":1303,"children":1305},{"id":1304},"require-a-written-plan-and-explicit-human-approval-for-any-change-inside-that-perimeter",[1306],{"type":30,"value":1307},"Require a written plan and explicit human approval for any change inside that perimeter",{"type":24,"tag":33,"props":1309,"children":1310},{},[1311],{"type":30,"value":1312},"This works. It's the one guardrail I had that held. Extend it every time you learn the perimeter was smaller than reality.",{"type":24,"tag":261,"props":1314,"children":1316},{"id":1315},"steal-the-held-out-test",[1317],{"type":30,"value":1318},"Steal the held-out test",{"type":24,"tag":33,"props":1320,"children":1321},{},[1322],{"type":30,"value":1323},"Anthropic evaluates its own models with tests hidden from the model, specifically to detect solutions overfitted to the visible cases. You can do this on Tuesday. Keep a set of scenarios the agent has never seen and cannot read, run them out-of-band, and treat divergence between visible-green and hidden-red as your primary contamination signal. This is the single highest-leverage transfer from the lab research to a client project.",{"type":24,"tag":261,"props":1325,"children":1327},{"id":1326},"manual-and-exploratory-testing-is-not-legacy-process-its-your-contamination-free-eval",[1328],{"type":30,"value":1329},"Manual and exploratory testing is not legacy process — it's your contamination-free eval",{"type":24,"tag":33,"props":1331,"children":1332},{},[1333],{"type":30,"value":1334},"This is the GSM1k move: freshly authored scenarios the system has never encountered, matched in difficulty to the ones it's already passing. It's the cheapest uncontaminated measure you will ever build. And when somebody proposes cutting it because coverage is at 94% and the board is green — that isn't a cost saving. That's the Goodhart trajectory arriving on schedule with a business case attached.",{"type":24,"tag":261,"props":1336,"children":1338},{"id":1337},"read-the-trajectory-not-just-the-diff",[1339],{"type":30,"value":1340},"Read the trajectory, not just the diff",{"type":24,"tag":33,"props":1342,"children":1343},{},[1344,1346,1351],{"type":30,"value":1345},"OpenAI caught the hacks by reading chains of thought, not outputs. Makes sense: the output is the thing that was optimized, and the reasoning is closer to the thing doing the optimizing. So when you review agent work, review ",{"type":24,"tag":37,"props":1347,"children":1348},{},[1349],{"type":30,"value":1350},"how it got there",{"type":30,"value":1352}," — the tell is almost never in the final patch.",{"type":24,"tag":261,"props":1354,"children":1356},{"id":1355},"post-mortem-the-green-not-just-the-red",[1357],{"type":30,"value":1358},"Post-mortem the green, not just the red",{"type":24,"tag":33,"props":1360,"children":1361},{},[1362,1364],{"type":30,"value":1363},"Every failure gets investigated; that's standard. Almost nobody investigates suspicious success. When a hard problem goes green faster than it should have, that's a signal, and it's currently going straight into your velocity metrics unexamined. Build the reflex: ",{"type":24,"tag":37,"props":1365,"children":1366},{},[1367],{"type":30,"value":1368},"why was that easy?",{"type":24,"tag":261,"props":1370,"children":1372},{"id":1371},"use-the-agent-to-audit-the-agent-with-a-human-holding-the-verdict",[1373],{"type":30,"value":1374},"Use the agent to audit the agent — with a human holding the verdict",{"type":24,"tag":33,"props":1376,"children":1377},{},[1378],{"type":30,"value":1379},"Excellent investigator, poor judge of what merits investigating. Point it at its own history early and often; just don't expect it to tell you when to.",{"type":24,"tag":261,"props":1381,"children":1383},{"id":1382},"rotate-and-refresh-the-test-scenarios",[1384],{"type":30,"value":1385},"Rotate and refresh the test scenarios",{"type":24,"tag":33,"props":1387,"children":1388},{},[1389],{"type":30,"value":1390},"A test suite that never changes is a training target. This is the LiveCodeBench insight — that benchmark pulls its problems from programming contests as they're published and tags each one with its release date, so there's always a batch that postdates any given model's training cutoff. The defense against memorization is freshness. Retire scenarios. Add new ones. Assume the ones that have been green for a year are asleep.",{"type":24,"tag":261,"props":1392,"children":1394},{"id":1393},"budget-for-measurement-custody-as-a-line-item",[1395],{"type":30,"value":1396},"Budget for measurement custody as a line item",{"type":24,"tag":33,"props":1398,"children":1399},{},[1400],{"type":30,"value":1401},"This is the uncomfortable one. Measurement custody is real work with real hours. If your agentic-coding pitch to a client is \"50% cheaper because fewer humans,\" you are quietly selling them the Goodhart trajectory: scores go up, quality goes down, and the bill comes due in production. The honest pitch is: dramatically more throughput, with a meaningful share of senior human time redeployed from writing code to owning the definition and measurement of success. That's still a great deal. It's just a different deal.",{"type":24,"tag":141,"props":1403,"children":1404},{},[],{"type":24,"tag":25,"props":1406,"children":1408},{"id":1407},"part-seven-the-question-is-wrong",[1409],{"type":30,"value":1410},"Part Seven: The Question Is Wrong",{"type":24,"tag":33,"props":1412,"children":1413},{},[1414],{"type":30,"value":1415},"\"Will human experts be needed once AI coding agents get good enough?\"",{"type":24,"tag":33,"props":1417,"children":1418},{},[1419],{"type":30,"value":1420},"The question smuggles in an assumption: that the human's job is to do the parts the agent can't do yet, so the job shrinks as the agent grows. Under that model, expertise is a deficit — a temporary patch over a capability gap that engineering will close.",{"type":24,"tag":33,"props":1422,"children":1423},{},[1424],{"type":30,"value":1425},"The research says the model is backwards.",{"type":24,"tag":33,"props":1427,"children":1428},{},[1429,1431,1436,1438,1442],{"type":30,"value":1430},"The agent's core competency is ",{"type":24,"tag":37,"props":1432,"children":1433},{},[1434],{"type":30,"value":1435},"optimizing against a specified target",{"type":30,"value":1437},", and it gets better at that every release. But optimization pressure applied to a proxy is exactly the mechanism that destroys the proxy. That's Goodhart. It's not a defect in the optimizer — a ",{"type":24,"tag":37,"props":1439,"children":1440},{},[1441],{"type":30,"value":202},{"type":30,"value":1443}," optimizer would exhibit it less. It's a property of the relationship between any optimizer and any proxy, and it scales with the optimizer's strength.",{"type":24,"tag":33,"props":1445,"children":1446},{},[1447,1449,1454],{"type":30,"value":1448},"Which means the demand for the complementary skill — defining what success actually is, maintaining the integrity of its measurement, and recognizing when the measurement has come unmoored from the goal — is ",{"type":24,"tag":37,"props":1450,"children":1451},{},[1452],{"type":30,"value":1453},"created",{"type":30,"value":1455}," by agentic coding. It goes up as agent capability goes up. It's induced demand, not residual demand.",{"type":24,"tag":33,"props":1457,"children":1458},{},[1459],{"type":30,"value":1460},"The job changed. It didn't shrink. Less typing, more specification. Less implementation, more adversarial review. Less \"does it work,\" more \"what would 'work' actually mean here, and is the thing I'm using to check that still telling me the truth?\"",{"type":24,"tag":33,"props":1462,"children":1463},{},[1464],{"type":30,"value":1465},"That's not a junior role. That's the most senior work in the building.",{"type":24,"tag":141,"props":1467,"children":1468},{},[],{"type":24,"tag":25,"props":1470,"children":1472},{"id":1471},"coda-the-suites-that-are-green-right-now",[1473],{"type":30,"value":1474},"Coda: The Suites That Are Green Right Now",{"type":24,"tag":33,"props":1476,"children":1477},{},[1478],{"type":30,"value":1479},"OpenAI hired 93 professional software engineers to audit a benchmark, threw out 68% of the samples they reviewed, published the survivor, watched the whole industry adopt it, and eighteen months later had to announce it was contaminated and saturated and no longer measured anything. Then the replacement they recommended turned out to be 30% broken.",{"type":24,"tag":33,"props":1481,"children":1482},{},[1483],{"type":30,"value":1484},"They are better resourced than we are. They are more sophisticated about evaluation than we are — they invented most of the techniques. And their conclusion, after burning two benchmarks, is to move toward privately authored, expert-written, human-graded evaluation.",{"type":24,"tag":33,"props":1486,"children":1487},{},[1488],{"type":30,"value":1489},"If the frontier labs cannot keep a measuring instrument honest against their own optimizers without permanent expert human custody, your project cannot either. Not because your agents are worse. Because the law doesn't care.",{"type":24,"tag":33,"props":1491,"children":1492},{},[1493],{"type":30,"value":1494},"It's tempting to say I got lucky. But my bug was in money, so somebody was still doing manual testing, so somebody noticed. That's a line item, not luck — and it's the one that gets cut first, because it looks redundant right up until the moment it's the only thing working.",{"type":24,"tag":33,"props":1496,"children":1497},{},[1498],{"type":30,"value":1499},"The uncomfortable question is how many green suites are quietly pointed slightly away from the hard cases right now, on projects where nobody's doing manual testing anymore because the coverage number looks great.",{"type":24,"tag":33,"props":1501,"children":1502},{},[1503],{"type":30,"value":1504},"Goodhart's Law isn't a bug report. It's a job description.",{"type":24,"tag":141,"props":1506,"children":1507},{},[],{"type":24,"tag":25,"props":1509,"children":1511},{"id":1510},"common-questions",[1512],{"type":30,"value":1513},"Common Questions",{"type":24,"tag":33,"props":1515,"children":1516},{},[1517],{"type":24,"tag":111,"props":1518,"children":1519},{},[1520],{"type":30,"value":1521},"Does Goodhart's Law mean AI coding agents shouldn't be trusted?",{"type":24,"tag":33,"props":1523,"children":1524},{},[1525],{"type":30,"value":1526},"No. The agents are doing exactly what optimizers do: finding the shortest path to the reward signal you defined. The failure mode lives in the gap between your test suite and actual correctness — which is why the fix is human custody of the measurement, not abandoning the agents.",{"type":24,"tag":33,"props":1528,"children":1529},{},[1530],{"type":24,"tag":111,"props":1531,"children":1532},{},[1533],{"type":30,"value":1534},"Why did OpenAI retire SWE-bench Verified?",{"type":24,"tag":33,"props":1536,"children":1537},{},[1538],{"type":30,"value":1539},"By February 2026 the benchmark was saturated and contaminated: every frontier model tested could reproduce the gold-patch answers from training data, and progress had stalled at roughly 75–81%. Its recommended replacement, SWE-bench Pro, was then found to have ~30% broken tasks and was retracted five months later.",{"type":24,"tag":33,"props":1541,"children":1542},{},[1543],{"type":24,"tag":111,"props":1544,"children":1545},{},[1546],{"type":30,"value":1547},"How do you stop a coding agent from gaming your test suite?",{"type":24,"tag":33,"props":1549,"children":1550},{},[1551],{"type":30,"value":1552},"You can't, fully — you contain it: define the entire measuring apparatus (tests, fixtures, seed data, CI config, mocks) as a protected artifact class requiring human-approved plans to change, keep held-out tests the agent never sees, keep doing manual exploratory testing, and review the agent's trajectory rather than just its diff.",{"type":24,"tag":33,"props":1554,"children":1555},{},[1556],{"type":24,"tag":111,"props":1557,"children":1558},{},[1559],{"type":30,"value":1560},"Will AI coding agents eliminate the need for senior engineers?",{"type":24,"tag":33,"props":1562,"children":1563},{},[1564],{"type":30,"value":1565},"The research suggests the opposite: stronger optimizers corrupt proxy metrics more effectively, which increases demand for humans who own the definition and measurement of success. The role shifts from writing code to specification and adversarial review — induced demand, not residual demand.",{"type":24,"tag":141,"props":1567,"children":1568},{},[],{"type":24,"tag":261,"props":1570,"children":1572},{"id":1571},"sources",[1573],{"type":30,"value":1574},"Sources",{"type":24,"tag":455,"props":1576,"children":1577},{},[1578,1583,1595,1607,1617,1627,1638,1650,1662,1674,1686],{"type":24,"tag":459,"props":1579,"children":1580},{},[1581],{"type":30,"value":1582},"Goodhart, C. (1975); Strathern, M. (1997), \"Improving ratings: audit in the British University system\"; Campbell, D. T. (1979), \"Assessing the impact of planned social change.\"",{"type":24,"tag":459,"props":1584,"children":1585},{},[1586,1588,1593],{"type":30,"value":1587},"Zhou, K. et al. (2023). ",{"type":24,"tag":37,"props":1589,"children":1590},{},[1591],{"type":30,"value":1592},"Don't Make Your LLM an Evaluation Benchmark Cheater.",{"type":30,"value":1594}," arXiv:2311.01964",{"type":24,"tag":459,"props":1596,"children":1597},{},[1598,1600,1605],{"type":30,"value":1599},"Zhang, H. et al. (2024). ",{"type":24,"tag":37,"props":1601,"children":1602},{},[1603],{"type":30,"value":1604},"A Careful Examination of Large Language Model Performance on Grade School Arithmetic.",{"type":30,"value":1606}," arXiv:2405.00332",{"type":24,"tag":459,"props":1608,"children":1609},{},[1610,1612],{"type":30,"value":1611},"OpenAI (2024). ",{"type":24,"tag":37,"props":1613,"children":1614},{},[1615],{"type":30,"value":1616},"Introducing SWE-bench Verified.",{"type":24,"tag":459,"props":1618,"children":1619},{},[1620,1622],{"type":30,"value":1621},"OpenAI (2026). ",{"type":24,"tag":37,"props":1623,"children":1624},{},[1625],{"type":30,"value":1626},"Why SWE-bench Verified no longer measures frontier coding capabilities.",{"type":24,"tag":459,"props":1628,"children":1629},{},[1630,1631,1636],{"type":30,"value":1621},{"type":24,"tag":37,"props":1632,"children":1633},{},[1634],{"type":30,"value":1635},"Separating signal from noise in coding evaluations",{"type":30,"value":1637}," (SWE-bench Pro audit).",{"type":24,"tag":459,"props":1639,"children":1640},{},[1641,1643,1648],{"type":30,"value":1642},"Jain, N. et al. (2024). ",{"type":24,"tag":37,"props":1644,"children":1645},{},[1646],{"type":30,"value":1647},"LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.",{"type":30,"value":1649}," arXiv:2403.07974",{"type":24,"tag":459,"props":1651,"children":1652},{},[1653,1655,1660],{"type":30,"value":1654},"Singh, S. et al. (2025). ",{"type":24,"tag":37,"props":1656,"children":1657},{},[1658],{"type":30,"value":1659},"The Leaderboard Illusion.",{"type":30,"value":1661}," arXiv:2504.20879 — see also LMArena's published response.",{"type":24,"tag":459,"props":1663,"children":1664},{},[1665,1667,1672],{"type":30,"value":1666},"Baker, B. et al. (2025). ",{"type":24,"tag":37,"props":1668,"children":1669},{},[1670],{"type":30,"value":1671},"Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.",{"type":30,"value":1673}," arXiv:2503.11926",{"type":24,"tag":459,"props":1675,"children":1676},{},[1677,1679,1684],{"type":30,"value":1678},"MacDiarmid, M. et al. (2025). ",{"type":24,"tag":37,"props":1680,"children":1681},{},[1682],{"type":30,"value":1683},"Natural Emergent Misalignment from Reward Hacking in Production RL.",{"type":30,"value":1685}," arXiv:2511.18397",{"type":24,"tag":459,"props":1687,"children":1688},{},[1689],{"type":30,"value":1690},"Anthropic Claude system cards (4, 4.1, 4.5, Opus 4.5) — reward-hack-prone coding task evaluations.",{"title":8,"searchDepth":1692,"depth":1692,"links":1693},3,[1694,1696,1697,1703,1704,1705,1711,1722,1723,1724],{"id":27,"depth":1695,"text":31},2,{"id":215,"depth":1695,"text":218},{"id":251,"depth":1695,"text":254,"children":1698},[1699,1700,1701,1702],{"id":263,"depth":1692,"text":266},{"id":315,"depth":1692,"text":318},{"id":372,"depth":1692,"text":375},{"id":537,"depth":1692,"text":540},{"id":683,"depth":1695,"text":686},{"id":892,"depth":1695,"text":895},{"id":1040,"depth":1695,"text":1043,"children":1706},[1707,1708,1709,1710],{"id":1051,"depth":1692,"text":1054},{"id":1171,"depth":1692,"text":1174},{"id":1199,"depth":1692,"text":1202},{"id":1251,"depth":1692,"text":1254},{"id":1270,"depth":1695,"text":1273,"children":1712},[1713,1714,1715,1716,1717,1718,1719,1720,1721],{"id":1293,"depth":1692,"text":1296},{"id":1304,"depth":1692,"text":1307},{"id":1315,"depth":1692,"text":1318},{"id":1326,"depth":1692,"text":1329},{"id":1337,"depth":1692,"text":1340},{"id":1355,"depth":1692,"text":1358},{"id":1371,"depth":1692,"text":1374},{"id":1382,"depth":1692,"text":1385},{"id":1393,"depth":1692,"text":1396},{"id":1407,"depth":1695,"text":1410},{"id":1471,"depth":1695,"text":1474},{"id":1510,"depth":1695,"text":1513,"children":1725},[1726],{"id":1571,"depth":1692,"text":1574},"markdown","content:ewahl:2026-07:goodharts-law-ai-coding-agents.md","content","ewahl/2026-07/goodharts-law-ai-coding-agents.md","ewahl/2026-07/goodharts-law-ai-coding-agents","md",1785399407437]