The boundary used to be free

For fifty years, software was safe because it was incapable.

A program was a set of cases somebody had thought of in advance. It read the inputs it had been told to expect, followed the branches it had been given, and when reality produced something outside that set, it stopped — or fell over noisily. We called that a limitation, and we spent enormous budgets working around it.

But it was also a control, and it cost us nothing. A core banking system is a monument to enumerated cases. Every rule the regulator cares about sits in there as an explicit branch, written by a named person, reviewed and released. That rigidity was the compliance story. The system could not act outside its specification because it had no capacity to. Safety was a by-product of incapacity.

That is the guarantee adaptive systems remove.

An AI handles the case nobody wrote. This is the whole promise — the reason it is worth the disruption. It is also the whole problem, because it will handle the case nobody wrote in a way nobody sanctioned, and it will do so fluently. The failure mode of a traditional program is a crash. The failure mode of an adaptive one is confident, plausible, well-formatted wrongness.

So the boundary does not disappear. It moves. What used to be implicit in every line of code now has to be written down somewhere else — explicitly, as its own artefact — because nothing inside the machine is enforcing it any more.

What we thought we were building

Over the past months we built what we have come to call a “brain”: the working knowledge of an engineering practice, written down properly — architecture, decisions, conventions, operating rules, the things that went wrong and the reasons they went wrong.

We thought we were building context for an AI. Better inputs, better outputs.

What we were actually doing was writing down the operating envelope. And that changed the shape of the thing. The “brain” is not a wiki. It is a set of versioned repositories with release tags, pull requests and automated checks — knowledge held under the same discipline we apply to source code. That looked like over-engineering for a pile of notes, right up until we understood what the notes were for. You version your boundaries the way you version code, because they are now the control.

Three things break once the constraint is no longer free.

Ambiguity becomes operational. A compiler never interpreted anything; it did exactly what the syntax said. An adaptive system interprets — so an overloaded word stops being sloppy writing and becomes an operational risk. A defined vocabulary is no longer pedantry: precision is cheaper than correction.

An unwritten and uncommitted rule does not exist. To have any effect it must be present at the moment of decision — in front of whoever, or whatever, is deciding. Otherwise it is an anecdote, and the mistake repeats. The failure is as old as institutions; what is new is how quickly it now surfaces.

An unchecked rule drifts. Agreement is not enforcement. A rule nobody verifies erodes quietly — drift is not carelessness; it is simply what happens when nothing checks. Consistency has to be machine-checked to survive, because every rule you cannot check is a rule you do not have.

Why this lands hardest in financial services

A regulator does not ask which model you use. They ask what your system may and may not do, how you know, and how you would prove it.

“It adapts” is not an answer.

That is the awkward position our industry occupies. The sector with the most to gain from software that handles the unanticipated case is also the one that can least tolerate an unstated operating envelope. The tension does not resolve by picking a better model — within a few years everyone will have much the same models. It resolves by learning to treat constraints as first-class artefacts: authored, owned, versioned and mechanically checked, with the same seriousness we have always given to code.

That is the real shift. Not that machines became creative. That constraint stopped being free.

The boundary used to be a property of the machine. Now it has to be a decision.