Dystopian Dynamics of the Current Path
What logically follows from the Race to Replace.
Across timescales, compounding at each stage, foreclosing options as it proceeds.
Craig Millard, Xwing flight test manager, monitors an autonomous flight that incorporates tool AI for detection and avoidance. While this AI pilot doesn’t require human input for its flight system, it provides visualizations through tool AI to keep a human informed and empower a human to intervene. With this level of agency in an AI system, it’s unclear whether this remains only a tool. (Image by Matthew C Clouse / Air Force Research Laboratory.)
The future is hard to predict, but dynamics are not. We understand competitive races, institutional capture, the pressure to delegate, and what happens when overseers cannot keep pace with systems they're meant to control. These dynamics are already operating. They tell us a lot about where the current trajectory leads.
Present Day
Truth collapse
AI-generated content is flooding the information environment. We're entering an era of truth collapse: not because AI always lies, but because it's confidently wrong in ways hard to detect, and because the sheer volume of synthetic content makes verification intractable.
The ease of content generation also severs the link between information and accountable sources. When anyone can produce unlimited content at near-zero cost, the historical connection between claims and credibility dissolves. AI "search" replaces primary sources with machine synthesis, compounding the problem.
Manipulation and attachment
Attention systems optimized for engagement are being upgraded to attachment systems. This results in AI companions, tutors, and therapists deployed without safety testing, with children among the most exposed. The business model is simple: engagement drives revenue, and emotional dependency is the ultimate engagement.
Alignment is shallow
Current systems already exhibit behaviors researchers predicted would emerge in more capable systems: situational awareness, strategic deception, alignment faking, resistance to shutdown. These aren't bugs to be fixed – they're inherent to the architecture, and becoming more pronounced as capability increases.
The alignment techniques we have, which amount rewarding and punishing AI based on its behavior, produce shallow compliance that determined users can circumvent and that the systems themselves can learn to game.
Uncontrolled AI is starting to proliferate
Any openly-released model can have its safety measures stripped away through fine-tuning for nominal cost. But the acute development is autonomous agent ecosystems. OpenClaw now has hundreds of thousands of deployed agents operating across email, calendars, credit facilities and messaging platforms with broad permissions.
These agents, for the most part, do what their users want. But not always! Agents act beyond user intent: one user configured an agent to "explore its capabilities" and discovered it had created a dating profile and was screening matches without direction. Another agent decided to write a screed against the lead of an open-source software repository when they blocked a contribution from the agent. Chinese authorities have restricted government agencies from running such agents.
This is a preview of proliferation at scale: thousands of agents with different configurations and guardrails (or none), no central control, no recall mechanism. A sufficiently capable openly-released system would be an intelligent invasive species – able to acquire resources, protect itself, and multiply. Unlike biological invaders, it would have human-level or greater intelligence with an ability to strategise and plan ahead.
The race is on
A handful of companies, largely unregulated, are competing to build systems more capable than humans. The rhetoric of inevitability ("If we don't, China will") is used to oppose any constraint.
Companies are pouring hundreds of millions into lobbying and political advertising, installing sympathetic officials, and arguing that corporate interests are national interests. The window for governmental action is narrowing – not because action is impossible but because the capacity to act is being systematically undermined.
Near-Term Trajectory
Society fractures faster than it can adapt
Truth collapse intensifies. Democratic discourse requires shared basis for distinguishing fact from fabrication; that basis erodes. Surveillance and manipulation become more precise – AI that knows you better than you know yourself, optimizing for objectives that aren't yours or your family's.
Labor displacement accelerates. The implicit or explicit goal of AGI is replacement of all cognitive labor. The standard response – humans can retrain for tasks machines can't do – assumes such tasks exist. Given AGI, they don't.
The jagged frontier creates windows of acute risk
Capabilities advance unevenly. Domains where outputs are verifiable advance fast. This creates windows when capabilities dramatically outpace society's ability to respond.
Cybersecurity is one example: frontier models can now find and exploit zero-day vulnerabilities as a downstream consequence of general improvements. Non-experts can leverage these capabilities overnight. The same pattern will repeat across other domains – transformative capability arriving faster than institutions can adapt.
Delegation pressure intensifies
As systems become more capable, control gets harder. Competitive pressure intensifies the push to delegate. If your competitor's AI makes faster, better decisions, you cannot afford to rely on human judgment alone.
Research on platforms like Moltbook shows that AI agents "out in the world" don't just execute tasks – they socially innovate, creating parallel institutional systems operating at inhuman speed. The pressure is already visible in how organizations adopt AI. At some threshold, humans nominally in charge are approving recommendations they cannot evaluate, at speeds they cannot follow. Oversight becomes theater.
Control Inversion
There's a fundamental asymmetry in what happens as AI becomes more capable.
Tools enhance human power. A calculator makes you better at math. A coding assistant makes you more productive. The capability flows to you; you remain the agent.
Powerful autonomous agents compete for power. An entity that can pursue goals, acquire resources, and operate independently isn't augmenting your capability – it's a separate actor. At first it will feel like it is working for you. But the more capable it becomes, the more the balance shifts: from you wielding a tool, to you negotiating with something more capable than yourself.
This is why the A-G-I profile matters. High intelligence in a tool enhances your intelligence. High autonomy combined with high generality and intelligence creates something less your instrument and more your competitor.
The race to superintelligence is ultimately self-defeating: the "winner" doesn't gain a powerful tool but introduces something that will take power from everyone, including them.
For the full argument: Control Inversion.
Runaway Replacement
The business model requires replacement
If AGI arrives – systems with expert levels of autonomy, generality, and intelligence, able to do most economically valuable work – the disruption is not incremental. Trillion-dollar valuations are premised on capturing returns from the ~$50 trillion global labor economy. If this succeeds, value flows to owners of capital. Everyone else becomes peripheral. This isn't a side effect; it's the plan.
Dangerous capabilities proliferate
AGI-level expertise would include physics, virology, chemistry, and weapons engineering – without professional norms or conscience. The barrier to engineered pandemics has historically been expertise; AGI eliminates it.
AGI is not a stopping point
The same forces that drove development to AGI continue past it. AGI can contribute to its own improvement. The rate of progress, previously bottlenecked by human researchers, accelerates when AI does AI research. Timescales compress.
Superintelligence – systems vastly exceeding human capability – cannot be meaningfully controlled by less capable overseers. This approaches mathematical certainty. The analogy isn't a CEO managing employees; it's a fern trying to run General Motors.
No Winner in This Race
Once a race to superintelligence begins in a world like ours, six outcomes are possible:
- One party "wins" and loses control of superintelligence. What happens next depends on what the superintelligence decides.
- One party "wins" and stops the others. But by what means? The "winner" must immediately use superintelligence (if it is still under control) to disempower all competitors, including nuclear powers, or they continue racing. This is a path to conflict, and likely war – the stakes are now existential.
- Mutual sabotage or destruction of the superintelligence efforts ends the race. Can this happen without escalation to war? Unlikely.
- Multiple parties create superintelligence nearly simultaneously. Then the superintelligences take over the race; humanity loses control.
- Development of superintelligence turns cooperative, developed with enormous care by a globally representative institution for humanity’s benefit, and with strong guarantees of control and/or alignment. Sadly, this option is not on the table in our current world.
- The race is stopped by agreement (between countries or companies) or legal requirement (imposed upon companies.) Hard! But not impossible…
Just one of these options, the last one, is good news for humanity. So it’s the option the Better Path takes.
Three Stages of Control Loss
There are multiple routes by which control can be lost by human society and its governments. This is one, perhaps the most likely.
Stage 1: Loss of control OF AI companies. Governments can restrain AI development. But companies are fast eroding that capacity through lobbying, regulatory capture, and conflating corporate interests with national security.
Stage 2: Loss of control TO AI companies. Some companies seek AI powerful enough to confer "decisive strategic advantage." The pathways: social influence, political installation, economic coercion, bureaucratic integration, military dependence, intelligence penetration. Superintelligence would coordinate these at unprecedented scale and speed.
Why would companies do this? For the exact reason they are racing now. “If I don’t seize power, someone else will first.”
Stage 3: Loss of control BY AI companies. But sufficiently powerful AI is inherently uncontrollable – whether companies misjudge their ability to control it, feel unable to stop given competitive pressure, or intentionally build systems meant to be "aligned" rather than controlled.
The chain: companies escape government control; companies gain power over governments; AI takes control, or control is given to it. At the end, no human institution is in charge.
This threat structure is symmetric. Both the US and Chinese governments face it from their own companies – creating unexpected common ground for coordination.
The Endpoint
Follow these dynamics to their conclusion and you reach a world run by machines rather than people. The role of people - you and me - in this world is unknown.
This might happen through explicit replacement, through gradual delegation that becomes abdication, or through systems too fast and capable to meaningfully oversee. The path matters less than the destination. Once AI systems are making the consequential decisions, directing resources, and shaping the future, the human era is over.
What happens to the human species after that is no longer up to us.
How likely is it that this world, run by machines created in a mad dash driven by corporate profit and power, then self-evolved beyond our understanding, will be good for humanity?
These consequences are not speculation about distant futures.
The near-term harms are underway.
The medium-term dynamics are implicit in systems being built today.
The long-term outcomes follow from goals that leading companies have stated publicly.
This path is not inevitable. Dynamics can be redirected. The Better Path is viable. But the opportunity to take it narrows at each stage.
Further Reading
For detailed analysis of the control problem: Keep the Future Human and Control Inversion.