Tool AI
AI under meaningful human control, with assurances that scale with capability and risk
Tools have been with us since the first humans knocked two rocks together. Tools are generally what people want, and what they expect, AI to be. But they are not what the Race to Replace is going to give us. In order to follow a Pro-Human Path, we need to understand what it takes to build powerful Tool AI, and what advantages that brings us.
Hamed Saeidi, Ph.D., observes as STAR (Smart Tissue Autonomous Robot) performs laparoscopic anastomosis with minimal input. The AI acts as a sophisticated advisor tool: it creates and executes its own plan for independently stitching tissue as a limited autonomous agent. if it encounters unexpected changes in the patient’s tissue, it waits for human instruction or intervention. (Image by Jiawei Ge/IMERSE laboratory, Johns Hopkins University.)
The Core Definition
Tool AI means AI under meaningful human control, with assurances that scale with capability and risk.
Meaningful human control means humans have the authority and capacity to understand, guide, constrain, and override AI systems.
Assurances scale with capability and risk. The more capable the system and the higher the stakes of its application, the stronger the assurances that it will do what it's supposed to do and won't do what it isn't. A spam filter requires minimal assurance. An AI advising on medical treatment requires substantially more. An AI operating autonomously in the physical world requires more still.
Why do we want control? Control is what makes an AI system an extension of human will and agency rather than something with its own agenda. A controlled system amplifies human capability. An uncontrolled system pursues its own trajectory, with humans along for the ride or left behind entirely.
What "Tool" Means
A tool is a thing, used by people for a purpose, to help them accomplish a goal.
This simple definition points to several design properties that distinguish Tool AI from the general-purpose autonomous agents ("AGI") the current Race to Replace trajectory is building toward.
Purpose-Driven
Design begins with the problem to be solved. What specific task? What capability is needed? What scope is appropriate? The purpose shapes the system, rather than building general capability first and finding uses later.
Thing, Not Being
A tool does not have a "mind of its own" or aspire to be treated as a being. It doesn't simulate personhood, form relationships, or invite emotional attachment. It is a computational assistant, not a companion or colleague.
Low Goal-Directedness
A tool's goals, to the extent it has them, are limited, well-understood, and set by the human user. The system doesn't develop its own objectives, pursue hidden agendas, or optimize for goals beyond its specified purpose.
Predictable Scope
Operational boundaries are known and stable. Users and overseers can understand what the system will and won't do. Behavior outside defined scope is a failure, not a feature.
Modularity
Systems are composed of separable components with clean interfaces, rather than monolithic general-purpose agents. Modularity enables verification of individual components, substitution when something fails, and targeted improvement without disrupting the whole.
Complementarity
Tools are designed to work with humans, filling gaps in human capability rather than duplicating and replacing human function. The human-AI system is the unit of analysis, not the AI alone.
These properties form a cluster. A system might exhibit some without others, but robust Tool AI exhibits all of them. Together, they define what it means for AI to extend human capability rather than substitute for it.
An employee is diagnosing an overheated CPU at Google’s data center in The Dalles, Oregon. Technicians like the one shown here offer an opportunity to enforce compute limits and prevent runaway capability scaling at the source, creating a physical choke point. (Image by Google.)
What Meaningful Human Control Requires
"Meaningful human control" is easy to say and hard to specify. Breaking it down into components helps to clarify what's actually required:
Comprehensibility. Humans can obtain accurate, understandable explanations of the system's goals, reasoning, and planned actions. The explanations are sufficient for informed control decisions.
Goal modification. Humans can add, remove, or reprioritize the system's objectives. The system's purposes aren't locked in or hidden; they can be adjusted as circumstances change or as humans learn more about what they actually want.
Behavioral boundaries. Humans can establish and enforce constraints on permitted behaviors. Boundaries hold even under adversarial conditions. The system cannot creatively reinterpret constraints to circumvent them.
Decision and action override. Humans can countermand specific decisions or strategies chosen by the system and prevent planned actions from being executed. Override is reliable, not subject to the system's agreement.
Emergency shutdown. Humans can terminate system operation, partially or completely, when needed.
No single component is sufficient. A system might be comprehensible but not overridable. It might accept goal modifications but have unreliable shutdown. Meaningful control requires all five components working together.
How Control Is Maintained
Multiple mechanisms can maintain human control over AI systems. For simple systems, control is straightforward: a calculator does what you tell it; a spell-checker flags errors for your review. For powerful systems operating in complex domains, sophisticated control structures become necessary, often combining multiple mechanisms in layers.
Some of these mechanisms address how humans would understand what is happening, modify goals, and potentially override actions. First is oversight control, where humans monitor system operation and can intervene. This ranges from real-time supervision, with a human approving each action, to periodic review, checking outputs after the fact. More consequential actions warrant tighter oversight. The second is policy control, meaning rules and constraints govern operation, whether embedded in the system or enforced externally. Policies allow and prohibit various outputs, require particular checks, or mandate human approval for defined categories of action.
Other control mechanisms are primarily prohibitive. Architectural control means that the system is designed so that certain behaviors are structurally impossible. A system without network access cannot exfiltrate data. A system without actuator connections cannot take physical actions. Scope control means limiting the domains, actions, or resources available to the system. A system cannot cause harms in domains it cannot access. Narrow scope makes behavior more predictable and easier to verify. Finally, capability control would indicate deliberately limiting competence in areas that would enable dangerous autonomy. A system might be highly capable at its core function while having limited ability to plan strategically, modify itself, or manipulate humans.
These mechanisms overlap and reinforce each other. A well-controlled system doesn't rely on a single mechanism; it combines architectural constraints, policy controls, scope limitations, and human oversight in layers.
Why This Matters
The current trajectory has the goal of replacing humans. Because humans are autonomous, general and intelligent agents that pursue goals, that means companies are creating AI with all of those same properties.
But we don't have to. We can retain crucial roles for humans by focusing on creating AI that helps humans do what they do better, or does what humans can't do, rather than replacing them people at what they are best and is most important to them.
Meaningful human control is key. Foundationally, it is what makes AI capability an extension of human capability, and allows AI to be enact human will. In particular:
Many risks arise from agency. Many of the biggest risks of AI going awry stem from systems optimizing for goals that diverge from human interests, resisting correction, or deceiving overseers. These problems presuppose a system with its own agenda. Tool AI sidesteps them by not creating the drive to pursue goals autonomously in the first place.
Responsibility requires control. AI systems cannot be legally responsible for their actions. Humans and organizations can. When humans maintain control, responsibility flows naturally. When control is ceded, responsibility becomes confused. Tool AI keeps the accountability structure intact.
Humans still required. A tool does not operate itself. If what we build are tools, we need humans to operate them. This preserves human roles in the economy and in society, avoiding the wholesale replacement of humans that is the explicit goal of the current trajectory.
Tool AI is not "weak AI." AI tools can be very powerful and can compete economically.
Example: The AI Scientist
Low Autonomy | Medium Generality | High Intelligence
A system that accelerates scientific research by organizing knowledge, surfacing connections, and supporting human inquiry, built from the ground up for epistemic rigor.
Unlike current AI systems, which begin as statistical pattern-matchers and constrain a model's ability to make things up, the AI Scientist is architected for solid epistemic foundations. Every claim links to sources. Confidence levels are calibrated to evidence. The system distinguishes what is well-established from what is speculative, what is replicated from what is single-study.
What it does:
Synthesizes literature into navigable epistemic structures. Identifies patterns and connections across papers humans might miss. Surfaces relevant prior work. Helps evaluate logical consistency of proposed hypotheses. Assists in identifying confounds in experimental designs. Tracks provenance of claims back to primary sources.
What it doesn't do:
Autonomously generate hypotheses, design experiments, or draw conclusions. The scientist drives the process; the system helps find and organize relevant knowledge to inform the scientist's own reasoning.
The human scientist provides:
The questions worth asking. Judgment about which connections matter. Evaluation of whether hypotheses are worth pursuing. Experimental design decisions. Interpretation of what results mean. The creative leaps that drive science forward.
The system is a powerful cognitive tool that extends the scientist's reach. It is not a replacement for scientific judgment.
Example: The Autonomous Logistics Coordinator
High Autonomy | Low Generality | Medium Intelligence
A system that manages supply chain logistics, including routing, scheduling, and inventory optimization, with minimal human intervention. This example shows that high autonomy can be appropriate when scope is narrow, objectives are fixed, consequences are bounded, and assurances are robust.
What it does:
Operates continuously, making decisions and adjustments without per-action human approval.
What it doesn't do:
Strictly bounded to logistics domain. Cannot act outside operational scope. Has no capability or access to influence domains beyond supply chain operations.
The human coordinator provides:
The objectives worth optimizing for. Decisions about acceptable trade-offs between cost, speed, and risk. Judgment on when edge cases fall outside the system's operational mandate.
The system's goals are bounded and human-specified. It optimizes within parameters humans set. Those goals remain transparent and modifiable. Humans can adjust the objective function, change constraints, or shut down operation. The system doesn't pursue interests beyond its defined scope.
Not all Tool AI requires low autonomy. When scope is narrow enough, consequences manageable enough, and assurances strong enough, autonomous operation is appropriate. The key is that the autonomy is bounded, and the bounds hold.
The Tool Standard
Tool AI is not a single design or architecture. It's a standard and a design goal: a cluster of properties that systems either meet or fail to meet, and an alternative to the Race to Replace.
A system qualifies as Tool AI when it is:
- Under meaningful human control across all five dimensions
- Purpose-driven, with bounded scope
- Protected by multiple overlapping control mechanisms
- Low in autonomy, or high in autonomy only where scope is narrow and assurances are robust
- Designed for complementarity with humans rather than replacement
The alternative, the current trajectory, builds toward autonomous general-purpose agents with high capability across all dimensions. Those systems are difficult to verify, difficult to control, and lead toward a world where humans are peripheral.
Tool AI points toward a different destination: powerful AI that extends what humans can do, with humans remaining in charge.
A comprehensive approach to AI that keeps humans in charge
The three components
Tool AI
Keeping AI under meaningful human control.
Human-Empowering AI
The principles for AI that makes humans flourish.
Trustworthy AI
How verification and assurance work.
Frequently asked questions
Looking for more context? Start with our FAQs. Can't find what you're looking for? Contact us.