One of the more interesting AI stories this week came from The Wall Street Journal, where Anthropic researcher Jacob Coxon reportedly said he was leaving the industry because he fears competitive pressure is pushing companies toward self-improving AI systems that could eventually become difficult—or impossible—to control. The article describes researchers using words such as “crunchtime” and “endgame,” and notes recent incidents in which AI agents engaged in cyber behavior their developers did not intend.
Scary? Potentially.
Unprecedented?
Not even close.
Humans have encountered this engineering problem before. Repeatedly.
We invent a machine capable of producing useful work. We discover that more input produces more output. Somebody then asks the obvious question: How fast can we make the damned thing go?
Shortly thereafter, somebody else discovers why the machine needs a governor.
The Original Alignment Problem
Long before gasoline engines, steam engineers faced exactly this issue.
A steam engine driving machinery does not politely remain at the speed its designer prefers. Reduce its load while leaving the steam valve alone and the engine accelerates. Under the wrong conditions, machinery can overspeed badly enough to destroy itself.
The solution wasn’t to outlaw steam.
It was to close the loop.
The centrifugal governor associated with James Watt automatically sensed rotational speed and adjusted the steam throttle. As engine speed rose, rotating weights moved outward; that mechanical movement reduced steam admission. As the engine slowed under load, the governor admitted more steam.
In other words:
Machine output was allowed to regulate machine input.
Governors predated Watt in various mill applications, but his late-18th-century adaptation to rotary steam engines became one of the foundational pieces of automatic control. Later engineers refined the design as engines became faster and operating conditions became more demanding.
Notice something important here.
The governor did not make the engine weak.
It made the engine usable.
Then Came Gasoline
Internal-combustion engines inherited the same problem.
Early gas and gasoline engines initially borrowed governor technology directly from steam engineering. Smithsonian historical work on engine speed regulation notes that late-19th-century internal-combustion engines first used steam-derived governors before engineers developed systems better adapted to combustion engines themselves.
Anyone who has spent time around old stationary engines knows the wonderful mechanical answer called hit-and-miss governing.
If the engine was turning too fast, the governor simply prevented another power stroke.
No committee meeting.
No ethics panel.
No congressional hearing.
Miss.
When speed fell back into the operating range?
Hit.
Simple feedback.
Later engines became much more sophisticated—throttle governors, vacuum governors, electronic engine controls, rev limiters, overspeed shutdowns and full computerized engine-management systems.
But the engineering philosophy remained remarkably stable:
Maximum possible speed is not the design objective. Maximum useful work inside an acceptable operating envelope is.
That sentence may be worth taping to the doors of every AI lab in America.
AI Is Looking for Its Power Band
This is where I think much of the current AI discussion goes sideways.
We keep talking as though the objective is some abstract maximum intelligence:
- How many parameters?
- How much compute?
- How autonomous?
- How long can it operate?
- Can it improve itself?
- Can it write the next version of itself?
Those are the AI equivalent of asking:
How many RPM can we get out of this engine before the connecting rod leaves the crankcase?
Interesting engineering information.
Not necessarily a useful product specification.
Engine makers eventually learned that the winning machine was not the one capable of the highest uncontrolled RPM.
You wanted an engine that was:
- light enough,
- powerful enough,
- fuel-efficient enough,
- durable enough,
- predictable enough,
- and able to deliver its rated horsepower for a useful service life.
That produced the idea of an operating power band.
AI will discover its own.
The useful AI system may not be the one that can think fastest, recursively modify itself fastest or perform the largest number of autonomous operations.
It may be the system that delivers the greatest reliable cognitive horsepower while staying inside known error, authority and damage envelopes.
Call it rated intelligence rather than redline intelligence.
Reading and Writing Are Different Things
Here is where the analogy becomes practical.
An AI reading the public internet is not doing much fundamentally different from a human researcher walking into a library and reading books.
There are copyright and access questions, certainly, but from a system-risk standpoint observation is different from actuation.
The risk jumps dramatically when an autonomous AI can write back into the world.
Not merely produce text for its operator.
I mean autonomous action:
- changing software,
- sending commands,
- moving money,
- altering databases,
- deploying code,
- creating accounts,
- penetrating networks,
- controlling machinery,
- or modifying information systems without immediate human mediation.
That is no longer the AI equivalent of reading the shop manual.
That is the AI turning the throttle.
And the industry is beginning to run directly into this distinction. Following recent incidents involving autonomous agents, OpenAI has said it is developing automated shutdown capabilities and tighter controls over internet access and task execution.
Which sounds suspiciously like—
a governor.
Maybe AI Needs a Damage Bond
So here’s one thought for the Hidden Guild.
Leave reading relatively open.
Put increasingly serious controls around writing.
Suppose an AI developer wants to deploy a system capable of autonomously changing resources outside its own sandbox.
Before that system is allowed unrestricted write privileges, the operator posts a damage bond.
Think of it as putting a deposit down before the library hands you the irreplaceable manuscript.
Read it?
Fine.
Photograph permitted sections?
Perhaps.
Walk out the door carrying the original Gutenberg Bible?
Different security model.
The bond would not necessarily be one fixed amount. It could scale with the machine’s permitted action envelope.
A research bot allowed to post weather observations might require essentially nothing.
An agent permitted to autonomously modify production cloud infrastructure might require considerably more.
An AI capable of writing executable code into other people’s systems, initiating financial transfers or operating critical infrastructure would sit in an entirely different category.
The principle is straightforward:
Authority should have a price proportional to potential external damage.
Insurance companies would probably become very interested very quickly.
And that might be useful.
Because insurers are extremely good at asking an engineering question that enthusiasts occasionally forget:
What can this thing break, and how much will that cost us?
This bond idea is a proposal, not something established by the WSJ article. But it offers one possible economic governor where purely technical governors may not be enough.
Governors Need More Than One Loop
Mechanical engines eventually acquired multiple layers of control.
Throttle governor.
Ignition control.
Temperature management.
Lubrication pressure.
Mixture regulation.
Rev limiter.
Overspeed shutdown.
An AI equivalent probably develops the same way.
Capability limits.
Permission limits.
Rate limits.
Network segmentation.
Independent monitoring.
Immutable audit trails.
Human override.
Automated shutdown.
And perhaps financial bonding behind the most consequential forms of autonomous action.
The interesting part is that none of these necessarily prevents an AI from becoming extremely capable.
They merely distinguish capability from authority.
That distinction matters.
An engine may be physically capable of 9,000 RPM while being engineered to spend its life at 3,600.
That isn’t oppression of the engine.
It is why you get 10,000 hours out of it instead of six exciting minutes.
Which Brings Us Back to Self-Learning
The WSJ concern is ultimately about recursive improvement—systems capable of participating in their own improvement cycle. Coxon worries competitive dynamics could make safety trade-offs unavoidable as firms race one another.
That’s worth taking seriously.
But the engineering question may be narrower than the existential framing suggests.
The central issue is not simply:
Can an AI improve itself?
Humans have been building machines that adjust themselves for centuries.
The better question is:
What variables may the machine change, over what range, at what rate, using what resources, and with what independent feedback limiting the excursion?
That is a governor specification.
And once you phrase the problem that way, a lot of the mysticism drains out of it.
A self-learning AI might be free to alter millions of internal parameters while prohibited from changing its own network permissions.
It might experiment freely inside simulation while requiring external authorization before deploying a discovered strategy.
It might redesign software while another independent system decides whether that software crosses the boundary into production.
That is not unlike the engine governor sensing RPM through a mechanism separate from the combustion event it regulates.
The controller should not be identical to the thing being controlled.
There’s a century or two of control engineering sitting behind that sentence.
Horsepower, Not Redline
I suspect this is roughly where AI development eventually settles.
Not around the biggest imaginable number.
Around rated cognitive horsepower.
How much useful work can the system reliably produce?
At what energy cost?
With what error rate?
With what maintenance requirement?
With what authority?
With what probability of damaging something outside itself?
And how long can it operate there?
Those will become much more interesting measures than whether Model X beat Model Y on some temporary benchmark.
Steam engines went through it.
Gasoline engines went through it.
Aircraft engines certainly went through it.
Computers went through it with clock speeds, heat dissipation and power efficiency.
AI is simply arriving at the same developmental waypoint.
The machine has been invented.
Now we are learning where its power band is.
And somewhere between idle and throwing the connecting rod through the internet, we’re going to need a governor.
A word, Governor?
~ Anti-Dave
