The verification tax

The mood around AI spend has shifted this year from "what can it do" to "show me the number." Budget that got approved on enthusiasm is now getting deferred pending something measurable, and a lot of teams are discovering they never defined what they were measuring.

I've spent the last few weeks building fairly hard with agents in the loop, so I want to do the accounting on myself while it's fresh. Not the pitch version. The version with the losses in it.

What I shipped

Ten days, roughly. A three-service system for screening options contracts, with a live market data feed, a rules engine, and an MCP gateway in front of it. A self-hosted analytics stack. A logging pipeline across two machines. A migration of this site off WordPress. Nine articles, most of them technical, several with working code in them.

That's more than I'd have produced in a month before, and I want to be clear the gain is real before I start complaining about it. Anybody telling you nothing changed is not building anything.

The first draft got free

The thing that collapsed is the cost of a first version.

Boilerplate went to zero, which everybody already knows. More interesting is what happened to exploration. Before, trying an approach I was only 30% confident in meant an afternoon I might throw away, so I'd think hard first and commit to one path. Now I can have two versions of a thing by lunch and pick. The cost of being wrong about a design dropped far enough that I stopped pre-optimizing my own decisions, and I make better ones as a result.

Same for the parts of a project I'm bad at. I can write a rules engine happily. I will avoid writing a log-shipping config for three weeks. That avoidance was a real drag on my throughput for years and it's mostly gone.

If the story ended there, the ROI question would be easy.

Verification did not get free

Here's what the accounting actually looks like. Producing a draft got roughly ten times cheaper. Confirming the draft is correct got no cheaper at all, and I now have ten times as much of it to confirm.

Three things from this month, all of which shipped or nearly shipped:

A security article opened with a statistic about how many public MCP servers run without authentication. The number was wrong by a wide margin. It was wrong because it came from a summary of a summary, and the actual study said something noticeably less dramatic. Catching it required opening the primary source, which took four minutes and which nothing in my process would have forced me to do.

The same article claimed that validating a token's audience mitigates two named attack classes. It mitigates one. The protocol spec is explicit that the other requires a different fix entirely, and I only found that because I went to link the spec and read the page I was linking. The wrong version was fluent, confident, and would have been quoted back at me by anyone who knew the material.

Then a smaller one. Text I wrote with plain ASCII punctuation kept rendering with typographic characters. I spent a round convinced the file was wrong before finding that the markdown pipeline had a smart-punctuation feature on by default and was rewriting the source at build time. That one cost me twice: once in the wrong diagnosis, once in the real fix.

None of those are exotic. They're the ordinary failure mode of generated work, which is that it's plausible. Wrong code usually announces itself by not running. Wrong prose and wrong numbers just sit there looking finished.

The tax has a shape

Verification doesn't parallelize the way production does.

I can have three things drafted at once. I cannot check three things at once, because checking is the part that requires me to hold the actual problem in my head, and there's one of me. So the bottleneck moved. It used to be typing. Now it's the rate at which I can genuinely understand output, and that rate has not improved since 2019.

Which means the naive version of this trade makes things worse. If you increase production tenfold and don't touch verification, you don't get ten times the work. You get ten times the surface area with the same amount of checking spread over it, which is a thinner layer of confidence on a much bigger pile. That's a great way to ship a subtle mistake into something that matters.

I think this is most of why the ROI conversation has soured. Teams measured output, got a big number, and then quietly spent the gains on incidents and rework that never got attributed back to the source.

The checking layer is the constraint

The highest-value thing I built this month never shipped to anyone. It's a hundred-line script that scans my writing for phrasing patterns I've decided I don't want, plus a habit of opening the primary source before anything with a number in it goes out.

That's the whole insight, as far as I have one. Anything that makes verification cheaper or more automatic pays back immediately and keeps paying, because it applies to everything downstream forever. Anything that only makes production faster moves work to the place that was already backed up.

Concretely, what earned its keep: automated checks that fail loudly, a rule that no statistic ships without me opening the source, tests around the parts where being wrong is expensive, and structured logs so that "what actually happened" is a query rather than an argument. Boring infrastructure, all of it, and it's the reason the output was usable rather than merely voluminous.

What didn't earn its keep: generating things I then had to read carefully in domains where I can't quickly tell right from wrong. That's a net loss every time and I've stopped doing it. If I can't verify it faster than I could have written it, I write it.

The number I'd give

If somebody asked me for a multiplier I'd say three, not ten, and I'd say the three is real and durable.

The gap between three and ten is the tax. It gets paid either in verification time up front or in incidents later, and my guess is that most of the disappointing-return stories are teams who assumed they could skip it. The ones getting the full three built something that checks the work.

That's a less exciting pitch than the one being sold. It's also the only version I've been able to reproduce.