Open Scaffold Templates: A Midstream Correction

Over the previous couple of months, I've invested time and effort on an idea I was quite proud of, and then talked myself out of. And finally talked myself back into. I think the talking-myself-out-of part is the most useful part of the story so far, so here it is.

The original concept

I started with what I considered a reasonable goal: a suite of AI driven templates that would build typical Access artifacts on demand. Repeatable, testable, maybe even boring. Point the tool at a problem, such as adding product-code scanning to an inventory database so a clerk can walk the shelves and complete a stock take without counting or writing anything down on paper. Another idea was to stand up a database to track umpire assignments across a season based on a general description of what it would requre. You get back a working database, built the same way every time.

The clever part, or what I first imagined was going to be the clever part, was using the suite for benchmarking. A develoloper could implement the entire tempate suite by hand, then have an AI agent run. They then compare the results. Measurable, a metric that means something. With the numbers produced by the comparison, you can have a conversation about using AI using evidence instead of a hunch.

I liked that. It felt rigorous. So I started building on it.

Why the benchmark plan fell apart

The benchmark plan didn't survive contact with the arithmetic. I should have seen that one coming. The problem is right there in the description.

The trouble with a race between a human and an AI agent over a suite of build tasks is that the end result is two of everything.

Two inventory modules. Two umpire databases. You only needed one working artifact; you now have to choose between a matched pair. One of them is going in the trash.

Even "winning" doesn't mean what you'd think it should mean. Let's just say the agent finishes the whole suite in half the time it would take a human developer. That's reasonable, maybe even on the conservative side. But hold on. No responsible developer would dare ship that output unread. They have to verify every line of code along with every other object it produced. Moreover, to call the race fair, they have to compare the two versions side by side. Maybe even run a test suite on both.

In the end, even though the AI wins on the stopwatch, the developer doesn't just get artifacts, they inherit a verification pass and a reconciliation pass that the human-only path probably never required. Whatever you gained on the build, you hand right back in bookkeeping and testing.

A test suite that manufactures disposable duplicate work isn't a business tool. It's a science-fair project. I changed course almost before I started.

The refocus

What came next was the part I still believed in: the concept and design of the templates themselves.

I narrowed the project down to that. I decided I wanted to create a library, not a competion. The rechristened Open Scaffold Templates would be structured descriptions of artifacts, each paired with a set of standards. The developer can specify an outcome, pick from the options on offer, and hand the whole package to the AI to build. The template carries the shape and the rules; the agent does the assembly. That felt like the right division of labor, and for a while it felt tractable.

I started to build and test templates.

This time, it wasn't the competition idea that killed the project. It was this next part.

A conceptual wall

Here's an undeniable statement of fact that I failed to incorporate in the second version of my template library. It underlies everything that follows.

An AI agent is not a compiler.

That's obvious in hindsight, but, at least for me, it wasn't part of my thinking at that point. That meant this attempt was also doomed.

The defining characteristic of a compiler is this: you can feed it the same source material and it makes the same decisions every time. That's the compiler contract. What goes in, comes back out.

AI agents don't work that way. They work in probabilities and predictions, and when you hand an Agent a template and ask it to build, it will not walk the same build path twice. We can count on AI to do the job despite that fact because, when properly defined and directed, the destination comes out the same, or very nearly the same every time. It was close enough, in fact, to bear a strong resemblance to a deterministic path and that helped lull me into thinking I was on the right path. The truth, however, is how it gets there wanders, and no amount of rule-writing pins it down.

I know this because I tried to pin it down. Over a week's time, I fell into a routine.

• Run the template.
• Note what drifted.
• Write a rule to close the gap.
• Run it again.
• Watch a different thing drift.
• Write another rule.
• Watch a different thing drift.
• Write another rule.
• Run it again.
• Take a long break and try again tomorrow.

I had the AI assistant writing and rewriting and revising rules several times a day. In one case, for example, the AI concluded from changes I requested, that one of our rules was actually backwards because I kept second-guessing its output. The AI even suggested reversing the rule. That's the point when I first began to suspect we were on the wrong path altogether. That feeling got stronger until I could no longer ignore it.

True, things did get better in some ways as we unearthed true gaps in the logic behind the template we were trouble-shooting, but ultimately, nothing ever converged. Every additional rule I added to account for one run's behavior was quietly invalidated by the next run's behavior. I began to feel like I was writing a specification to control the output of a series of dice rolls. I was on the verge of throwing in the towel instead of the dice because I could never quite finish writing rules.

In hindsight it was never going to work, and I should have seen it coming. The whole reason a language model is worth using is that it generalizes instead of memorizing. Demanding that an AI's output become 100% reproducible is demanding it stop being the thing that made it useful in the first place.

An AI agent is not capable of running a task consistently enough for a finite set of rules to describe every run, every time. You can't rule your way out of non-determinism. That's not a gap in the rules. It's the machine doing exactly what it is designed to do. I was fighting it, not harnessing it.

A deterministic, rules-bound AI template is a fool's errand, in other words. Like a lot of lessons I've had to learn over the years, learning it firsthand was painful, but effective.

Over the border

There was a second realization waiting to spring out at me after I'd digested that first one. The unrealistic rule-writing task was one problem.

The bigger problem was that I'd been trying to suppress the exact quality I adopted AI for in the first place. The ability to push me past the known frontier.

I've written before about AI letting a developer cross the frontier of their own knowledge — the value is in its ability to adapt, explore, and reach a solution you wouldn't have conceived of or reached on your own. And there I was, trying to build a cage for it. I wanted one controlled path out of a tool whose whole purpose is to leave the beaten one.

So I retreated and restarted on the opposite foot. This time I would specify some basic parameters and my goals and leave the work of getting there to the AI. That, it turns out, is what "scaffold" was supposed to mean all along. Don't try to script what happens when an error occurs — tell it to adopt an error-handling pattern and work through to the end state. Focus on outcomes, not procedures.

The user picks the outcome. The template carries the intent and the standards. The route is the agent's problem as long as the outcome is right.

It comes at a cost. You give up the comfort of a predictable path. Sometimes you'll see something you didn't expect mid-build — an error message no one could have headed off, a datatype choice you'd never stopped to consider. It's like watching the sausage get made. That's the price of letting the tool do the thing you hired it to do. After all, if you want to duplicate something, there's always copy-paste and where's the fun in that.

Code wadding prevented

For me, payoff was that the doom loop broke. When every surprising run earns one more rule bolted onto the last, you aren't designing. You're doing what I call code wadding in VBA: add one more line to swallow an exception, graft on one more branch for a condition you didn't foresee, until you've got a wad of interconnected code so fragile nobody dares touch it. Specifying goals instead of steps forces the procedure back to the big picture that code wadding always skips.

I didn't finish the earlier versions of Open Scaffold Templates. This one, I think, is the one that will finally be worth building. I'm doing that now.

I'm still working on it and probably will be for a while longer. This time, though, I'm looking at the process differently: Looking for ways to remove rules, not add them. Looking for better ways to describe outcomes and options to a user without telling the user or the AI agent how to achieve those outcomes.

Can I predict where this version will end up? No, I can't, and I'm reasonably sure my AI can't either. After all, it's a probability machine, not an Oracle.

Written with an assist from Claude Code, which deeply regrets its role in the initial false steps.