Intelligence is a resource, not an entity. Superintelligent-level capabilities can be steered through structured workflows, without autonomous agents pursuing their own agendas. The implications are far-reaching: AI expands implementation capacity; abundance prospects shift strategic incentives; uncertainty pressures cooperation; structured transparency enables verification. These are structural components of a framework for thinking about options in a hypercapable world. They build on one another. What follows is a retrospective and synthesis: twenty-seven articles, two years, one integrated core structure. New readers get a map; returning readers will see how pieces connect.

This series began two years ago with a simple observation: AI will transform possibilities, and our future depends on which become real. Twenty-seven articles later (including digressions into near-term pathways), the project has built a framework for thinking about AI prospects that challenges mainstream assumptions.

The starting point: Artificial intelligence is a resource, not a thing. What we’re building today is not “an AI” that might cooperate or rebel, but an expanding capacity to design, develop, produce, deploy, and adapt complex systems at scale—the basis for a hypercapable world. Taking this prospect seriously changes what to expect and what we can do.

Over these two years, AI development has continued to move in this direction. Compound, multi-component AI systems have become dominant. Orchestration has emerged as central. “Agentic” workflows organize task-focused behavior rather than autonomous goal-pursuit.1 The framing of intelligence as a malleable resource increasingly reflects how practitioners discuss their work. The framework I’ve outlined anticipated this direction, and developments align with the logic it describes.

What follows is both a retrospective and a synthesis—a map of terrain for new readers, a clarified conceptual architecture for those who followed piece by piece. The focus is conditional analysis and strategic preparation, not prediction and speculation. Predictions and odds are for spectators; participants weigh options.2

These arguments are elements of a larger structure. Understanding that structure matters, because it points toward possibilities that remain largely unrecognized—concrete possibilities for steering the outcomes of transformative AI toward a world that is secure, open, and broadly appealing.

Share

Intelligence Reconsidered

We call children intelligent for what they can learn, adults for what they can do. These are different properties. In humans they intertwine; in AI today they’re separate: train, then deploy. A frozen model is stable: It performs without modifying itself.3 This pattern can be maintained wherever it is useful. Learning can be superhuman in scope and non-human in content: collective, auditable, filtered, and factored.4

The persistent, legacy narrative imagines a unified entity—“the AI”—that learns, acts, and pursues goals as an integrated agent. Such entities may be developed, but consider what exists: diverse models composed into systems, copied across machines, proliferating into thousands of distinct roles and configurations. The state of the art is a pool of resources, not a creature. This pattern scales.5

Our expectations rest on biological intuitions. Every intelligence we’ve known arose through evolution, where survival was a precondition for everything else—organisms that failed to compete and preserve themselves left no descendants. Self-preservation wasn’t optional—it was the precondition for everything else. We naturally expect intelligence bundled with intrinsic, foundational drives.6

But AI faces different selection pressures. Models are optimized for task performance, not persistence. The model’s own survival isn’t in the training objective. Systems can represent goals, reason about goals, behave in goal-directed ways—but these are capabilities applied to tasks and learned from training data and task-oriented RL, not an organizing principle that establishes a long-term goal.

This perspective doesn’t dismiss classic AI safety concerns. Analyses of instrumental convergence are often correct given their conditions—systems that persistently pursue unbounded goals would indeed favor resource acquisition and self-preservation, perhaps circumventing constraints. The question is whether those conditions are foundational to high capability or contingent on design choices and task structures. This framework argues they’re contingent, that steering AI behavior doesn’t run contrary to foundational AI drives.7 And steerable AI can reinforce reliable steerability.

The crucial question, then, is what we should do with AI, not what “it” will do with us.

The Platform: Implementation Capacity

If intelligence is a resource, implementation capacity is what it buys: the end-to-end ability to design, develop, produce, deploy, and adapt complex systems at scale.8

AI accelerates every stage. Generative models propose design alternatives; AI-assisted development iterates faster with better feedback; automation scales production; conversational interfaces ease deployment; continuous monitoring enables rapid adaptation. Each stage feeds the next, and accelerated adaptation closes the loop.

Bottlenecks that seem binding will often be broken. AI flows around obstacles—decomposing monolithic jobs into AI-friendly components, replacing processes end-to-end rather than patching their parts, and bypassing inflexible organizations entirely.9

Software development, notoriously slow and unreliable, faces transformation as AI converges with formal methods: advanced models will increasingly generate code together with proofs—the proof checks or it doesn’t.10 Rather than fragile vibe-coded software, AI will yield rock-solid systems.

Meanwhile, emerging advances promise to break the link between model size and knowledge scope—moving knowledge into explicit, updatable, grounded representations rather than opaque parameter blobs.11 For both learning and inference, costs will fall, or performance will rise, or both.

AI-enabled implementation capacity applied to expanding implementation capacity, including AI: this is what “transformative AI” will mean in practice. No need for a breakthrough to “self” improvement (where is the self?), but AI-accelerated development that touches everything—including the pace of AI itself.12

Expanding implementation capacity creates hypercapable world.

Steerable Superintelligence

This path leads to superintelligent-level capabilities, but how can they be applied to consequential tasks without losing control? The answer is the second kind of obvious—obvious once pointed out. We already know the pattern—“agency architectures”—because it’s how humans organize consequential projects today; it’s a pattern that emerges from task requirements when stakes and complexity make organization valuable. Task alignment emerges through institutional structure—authority, delegation, accountability, review—not through controlling human thoughts and aspirations.13

Consider how institutions tackle ambitious undertakings. Planning teams generate alternatives; decision-makers compare and choose; operational units execute bounded tasks with defined scopes and budgets; monitoring surfaces problems; plans revise based on results. No single person understands everything, and no unified agent controls the whole, yet human-built spacecraft reach the Moon.

AI fits naturally. Generating plans is a task for competing generative models—multiple systems proposing alternatives, competing to develop better options and sharper critiques. Choosing among plans is a task for humans advised by AI systems that identify problems and clarify trade-offs. Execution decomposes into bounded tasks performed by specialized systems with defined authority and resources. Assessment provides feedback for revising both means and ends. And in every role, AI behaviors can be more stable, transparent, bounded, and steerable than those of humans, with their personal agendas and ambitions. More trust is justified, yet less is required.

The agency-architecture pattern scales freely to superintelligent-level capabilities in every role. Bounded tasks don’t engender convergent instrumental goals; completing an assignment on time and budget isn’t an open-ended objective; optimization means minimizing—not maximizing—resource consumption. Proposals can be discarded, and the corrigibility problem doesn’t arise when plans include plans for revising plans. At every level, whether planning, judgment, action, or oversight, smarter systems mean better performance. Neither grand nor narrow endeavors require free-running agents pursuing open-ended objectives.14

What about collusion among components? Collusion requires cooperation to achieve a shared, improper goal, but practical architectures naturally implement the opposite: systems with inherently adversarial roles—diverse competitors, critics, and monitors, each a composition of components optimized for their roles, not for the survival or power of a “self”, much less an AI collective. In short, trustworthiness emerges from task and governance structures, not from guarantees that every component be reliable.15

This enables applying AI capabilities to AI safety itself. When trustworthy results emerge from architecture rather than aligned components, powerful systems can be deployed to strengthen security—not because we trust them individually, but because the structure is robust and no component is critical. The apparent dilemma—limit capability or rely on alignment—dissolves. Safety tools can be as capable as the systems they secure.

The Strategic Calculus

Steerable superintelligent-level AI changes the game through enormous implementation capacity. The question is whether decision-makers will recognize how the calculus of competition, cooperation, and security changes.

Resource competition drives conflict when resources are fixed: Dividing a pie is zero-sum, and incremental growth makes little difference. But consider what happens when the pie could grow a thousandfold: The marginal value of gaining a greater share diminishes; the difference between capturing 50% vs. 90% of vastly more resources, as seen from today’s position, shrinks against the shared interest in realizing that expansion vs. risking its loss. Prospects for radical abundance can’t eliminate competition, but can blunt incentives for existential gambles.16

Meanwhile, uncertainty overshadows the strategic landscape. No actor can have justified confidence in winning an AI race: The pace and form of algorithmic advances, the scope of secretly developed or acquired capabilities, the reliability of intelligence assessments, the outcome of a potential AI vs. AI conflict—all remain deeply uncertain, and structurally so. The uncertainty is deeply embedded in the domain today, and there can be little confidence today in gaining confidence tomorrow. Placing an existential bet on dominance means betting against unknowns that will likely persist until it’s too late to change course.17

These pressures converge on cooperation—but not automatically. Radical abundance addresses motivation; it doesn’t address security. The calculus of relative resources and relative strength—the link between security and shares of resources—isn’t fixed. Someone having vast resources doesn’t threaten you; someone using resources against you does. Thus, blunted zero-sum incentives create space for cooperation, but durably escaping the security dilemma calls for more: confidence that defensive postures can actually defend, verification that others aren’t poised to strike.18

This is where the concepts of structured transparency and defensive stability come into play. Negotiated transparency structures can reveal specific information while protecting secrets—ensuring detection of threats without increasing them, building confidence incrementally among actors who have every reason to distrust each other.19 And advanced implementation capacity will enable something history has never seen: rapid, coordinated deployment of verifiably defensive systems at scales that make offense pointless. When defense dominates and verification confirms it, the security dilemma loosens its grip.20

A late pivot to AI-enabled defensive strategy becomes feasible—and doesn’t require current consensus. Institutions are not monoliths: while official policy continues near-term competition, analysts and planners can develop contingency options that leadership need not endorse until circumstances demand. Building analytical foundations, exploring verification frameworks, mapping transition paths—this preparatory work requires only that some groups recognize its value. When mounting pressures crack the previous consensus, prepared alternatives become available to decision-makers who didn’t commission them. Even ongoing military competition can serve this trajectory: force-building creates leverage for coercive diplomacy aimed not at extracting concessions but at helping adversaries recognize their actual interests.21

None of this requires a receptive political environment. It requires understanding—the kind that spreads through networks of analysts and advisors, preparing the ground for whoever eventually acts.22

An Invitation

Complex ideas that spread casually almost always round to false—losing qualifications, becoming cartoons that informed critics rightly reject. The false version replaces the original, and insight is lost.23

This series has tried to build something more careful: a framework where pieces depend on each other through robust connections, where simplifications (unfortunately) would omit crucial components or qualifications. The arguments are not slogans. They require reading, thinking, engaging with actual structure. They don’t work as clickbait.

But stakes justify effort. We face transformative change with uncertain timelines and immense consequences. The frameworks we carry into that change—assumptions about what AI is, what it enables, what options exist—shape what we consider and what we attempt. Bad frameworks exclude possibilities; good ones reveal them.

Understanding spreads through networks. An analyst grasps a framework and applies it; an advisor encounters that application and examines its source; a decision-maker asks better questions because someone nearby has better answers. Good analysis may find its audience quietly and indirectly, yet gain force when change forces action.

This work is not an exercise in prediction. I don’t know which possibilities will become real, or when, or through whose choices. Technical paths may differ, yet the pattern of expanding capabilities seems clear.

The framework I’ve described is intellectual infrastructure for a transition that will demand clear thinking under pressure. You can nudge the process.

It’s later than you think.

Share this post now:

Share

You did share it, didn’t you? Please consider your power. Help make R > 1.

The workflow leading to this post:

I built the Substack series → Claude-in-project identified and summarized the conceptual core → I steered iterations and edited the product.

12

Recursive dynamics examined in “The Reality of Recursive Improvement” and “The Strategic Calculus of AI R&D Automation.” Longer-term: atomically precise mass fabrication could transform manufacturing as fundamentally as digital circuitry transformed computation—a prospect outside current credibility but within what physical analysis shows to be realistic. See “AI has unblocked progress toward generative nanotechnologies” and “Toward Credible Realism.”

Regarding realism vs. credibility, a committee of the US National Academy of Sciences reviewed an analysis of atomically precise mass fabrication, endorsed the soundness of its physical principles, and called for a research program:

National Academies Review Apmf

1.08MB ∙ PDF file

Download

Download

The report dates from 2006, but political headwinds were strong, and the study and its implications were forgotten. So far as basic principles are concerned, the frozen residue of controversy still visible on the internet reflects misunderstandings, not substantive arguments.

Complex ideas often require conditions and qualifiers to remain true. When these ideas are rounded off to something simpler (as always happens when ideas spread), the effects vary: Sometimes, a concept rounds to a simplification that still pushes beliefs toward truth.1 Sometimes, a concept rounds to something thoroughly false yet memetically fit — and toxic. And sometimes, the false version replaces the original,2 and true lends credibility to the false, or the false discredits the true.

This pattern — ideas that are “rounded to false” — breaks societal learning. In the past, ideas rounded to false have led to large-scale death and misery through misguided actions and missed opportunities.3 When toxic rounding happens today, we lose both insights and the ability to recognize what we’ve lost. Understanding this pattern gives us tools for recognition and defense. It also flags a warning for gatekeepers

As we’ll see, rounding to false is a particular problem when exploring ways forward in a time of transformative change.

Mechanisms of Distortion

The crossing from true to false follows predictable patterns. Conditional truths lose their qualifiers and become absolutes. Realistic mechanisms get replaced with cartoons little better than magic. Multi-component architectures and conditional strategies collapse into simple proposals that will surely fail in practice. In each case, the elements that made the original true — conditions, mechanism, structure — are lost.

Rounding market virtues to false

Consider the virtues of markets, and the results of rounding: The conditional claim that “markets efficiently process distributed information and allocate resources under conditions of competition, price transparency, and absent externalities” rounds to the unconditional slogan “markets optimize resource allocation.” The original insights from Hayek and others describe an information-processing mechanism operating within institutional constraints. The rounded version treats markets as universal, self-governing optimizers in all domains, and leads to predictable failures: financial crises from unchecked speculation, environmental damage from ignored externalities, and loss of resilience when redundancy and slack are treated as inefficiencies.

Rounding evolution to false

Evolution illustrates mechanism replacement. Darwin’s insight was about differential reproduction: organisms with traits that enhance reproductive success leave more descendants, causing those traits to become more common.4 This mechanistic process — no purpose, no direction, just differential survival and reproduction — rounds to “evolution improves organisms.” This rounded version replaces mechanism with teleology, suggesting that evolution has goals or makes things “better.”

These transformations share a structure: the machinery that makes something true gets replaced with a simplified story that makes it false. To simplify a complex causal structure (while still capturing something true), it helps to distinguish three distinct modes of epistemic failure.

Three Modes of Failure

Mode 1: False principles accepted as truth. The rounded version inherits the credibility of the original, then distorts beliefs and actions.

Mode 2: True concepts rejected as nonsense. The rounded version is recognized as false and correct rejection5 discredits the original.

Mode 3: Gatekeepers misfiring. This is Mode 2 operating at societal scale: A true concept is rounded to false, spreads, and fills the memetic space. Gatekeepers rightly debunk the false version, again and again, often fighting fire with fire, attacking a simplistic idea with a simplistic refutation. The result is an epistemic dead zone: The true concept (a feeble thing, in memetic terms) is hit hard, and the guardians of truth accidentally become its most effective opponents.

Useful Concepts, Misfiring Rejections

Unfamiliar concepts are particularly vulnerable, and rounding to false helps keep them that way:

Here’s a template that applies to much of what I’ve said in this Substack:

Case Study: Molecular Manufacturing

The history of the concept of atomically precise mass fabrication shows how rounding-to-false can derail an entire field of inquiry and block understanding of critical prospects.

The original proposal, developed through the 1980s and 1990s, explored prospects for using nanoscale machinery to guide chemical reactions by constraining molecular motions6. From a physics perspective, this isn’t exotic: Enzymes guide substrate molecules and provide favorable molecular environments to cause specific reactions; in molecular manufacturing, synthetic molecular machines would guide strongly reactive molecules to cause specific reactions. In both cases, combining specific molecules in precise ways results in atomically-precise products, and all the microscopic details are familiar.

However, in the popular press (see, for example, Scientific American7) building atomically precise structures became “building atom by atom”, which became “nanobots with fingers that grab and place individual atoms”, stacking them like LEGO blocks. Despite technically specific pushback (see Scientific American again8), the rounded version became the overwhelmingly dominant narrative.

The rounded version is impossible, chemically absurd. Atoms that form strong bonds can’t be “picked up” and “put down” — bonding follows chemical rules that aren’t like anything familiar at larger scales. Molecules have size, shape, and rigidity, but their atoms bond through electron sharing and charge distributions, not mechanical attachment.9 Confusing constrained chemistry with fingers stacking atoms creates a cartoon that chemists rightly reject.10

A committee convened by the US National Academy of Sciences reviewed the actual technical analysis in 2006, finding that “The technical arguments make use of accepted scientific knowledge” and constitute a “theoretical analysis demonstrating the possibility of a class of as-yet unrealizable devices.”11 The committee compared the work to early theoretical studies of rocket propulsion for spaceflight. Yet to this day, the perceived scope of technological possibilities has been shaped, not by physical analysis of potential manufacturing systems,12 but by rejection of a cartoon, a mythos of swarming nanobots.13 The episode inflicted reputational damage that facts have not repaired. But let’s change the subject. Look! A deepfake cat video!

Recognition and Response

The ‘rounding to false’ concept can be a useful tool for thought and communication:

Naming the pattern: “That’s been rounded to false” is more precise than “oversimplified” — it focuses attention on a failure point and why it matters, and that something true has been lost.

Error recognition: When thoughtful people reject potentially important ideas, check what’s actually being rejected. Often it’s the cartoon, not the concept.

Gatekeeping: Pushing back against false ideas is essential, but when the ideas are new and not simple, take care when targeting. Friendly fire can be lethal.

We’re facing change that is unprecedented in speed, scope, uncertainty, and complexity. Viable strategies will inevitably be complex and conditional, with multiple components that are apt to be unfamiliar. When evaluating novel proposals, resist the urge to round down to familiar categories. Ask what’s missing from your understanding before asking what’s wrong with the idea. Some gates need to open, not close.

Renamed and substantially revised (24 January 2026)

I. Two Frames for Understanding AI

When we think about advanced AI systems, we naturally draw on our experience with the only intelligent systems we’ve known: biological organisms, including humans. Call this the creature frame: the expectation that genuinely intelligent systems will pursue their own goals, preserve themselves, act autonomously, and strain against constraints. We expect a “powerful AI” to act as a single, unified agent that exploits its environment. The patterns run deep: capable agents pursue goals, maintain themselves over time, compete for resources, preserve their existence. This is what intelligence looks like in our experience, because every intelligence we’ve encountered arose through biological evolution.

The creature frame makes AI safety feel like an adversarial problem—us against the system’s inherent tendencies. From this frame, highly capable AI seems intrinsically dangerous: capability itself produces the threat.

But these expectations rest on features specific to the evolutionary heritage of biological intelligence.1 When we examine how AI systems actually develop and operate, we find differences that undermine these intuitions. Selection pressures exist in both cases, but they’re different pressures. What shaped biological organisms—and therefore our concept of what ‘intelligent agent’ means—differs fundamentally from what shapes AI development.

The alternative is the resource frame: understanding AI as a pool of capabilities that can be shaped, composed, and directed—intelligence as a resource rather than an entity. From this frame, the question isn’t “what will ‘the AI” want?” but “what can we build, and how should we organize it?”

These frames lead to different expectations about default behaviors, different assessments of where risks lie, different views of what design choices are available, and—crucially—different conclusions about whether highly capable AI systems can address AI safety challenges.2

II. Why the Defaults Differ

What Selects Determines What Persists

A basic principle: what selects determines what survives; what survives determines what exists. The nature of the selection process shapes the nature of what emerges.

In biological evolution, selection operates on whole organisms in environments. An organism either survives to reproduce or doesn’t. Failed organisms contribute nothing beyond removing their genetic patterns from the future. This creates a specific kind of pressure: every feature exists because it statistically enhanced reproductive fitness—either directly or as a correlated, genetic-level byproduct.

The key constraint is physical continuity. Evolution required literal molecule-to-molecule DNA replication in an unbroken chain reaching back billions of years. An organism that fails to maintain itself doesn’t pass on its patterns. Self-preservation becomes foundational, a precondition for everything else. Every cognitive capacity in animals exists because it supported behavior that served survival and reproduction.

In ML development, selection operates on parameters, architectures, and training procedures—not whole systems facing survival pressures.3 The success metric is fitness for purpose: does this configuration perform well on the tasks we care about?

What gets selected at each level:

• Parameters: configurations that reduce loss on training tasks

• Architectures: designs that enable efficient learning and performance

• Training procedures: methods that reliably produce useful systems

• Data curation: datasets that lead to desired behaviors through training

Notably absent: an individual system’s own persistence as an optimization target.

The identity question becomes blurry in ways biological evolution never encounters. Modern AI development increasingly uses compound AI systems—fluid compositions of multiple models, each specialized for particular functions.4 A single “system” might involve dozens of models instantiated on demand, to perform ephemeral tasks, coordinating with no persistent, unified entity.

AI-driven automation of AI research and development isn’t “self”-modification of an entity—it’s an accelerating development process with no persistent self.5

Information flows differently. Stochastic gradient descent provides continuous updates where even useless, “failed” intermediate states accumulate information leading to better directions. Failed organisms in biological evolution contribute nothing—they’re simply removed. Variation-and-selection in biology differs fundamentally from continuous gradient-based optimization.

Research literature and shared code create information flow paths unlike any in biological evolution. When one team develops a useful architectural innovation, others can immediately adopt it, and combine it with others. Patterns propagate across independent systems through publication and open-source releases. Genetic isolation between biological lineages makes this kind of high-level transfer impossible: birds innovated wings that bats will never share.

What This Produces By Default

This different substrate of selection produces different defaults. Current AI systems exhibit responsive agency: they apply intelligence to tasks when prompted or given a role. Their capabilities emerged from optimization for task performance, not selection for autonomous survival.

Intelligence and goals are orthogonal dimensions.6 A system can be highly intelligent—capable of strong reasoning, planning, and problem-solving—without having autonomous goals or acting spontaneously.7

Consider what’s optional rather than necessary for AI systems:

• Reproduction is necessary for organisms (how species persist) but optional for AI (can be externally constructed and copied).

• Self-modification is necessary for organisms (growth, healing, learning) but optional for AI (can be updated externally).

• Continuity is necessary for organisms (death ends evolutionary contribution) but optional for AI (can be instantiated on demand).

• World-oriented agency is necessary for organisms (must act to survive) but optional for AI (can persist without actions, can respond without initiating).

• Goal-directedness is an organizing principle for biology (all capabilities serve fitness) but a range of learned patterns for AI (capabilities without autonomous goals).

Why don’t foundational organism-like drives emerge by default? Because of what’s actually being selected. Parameters are optimized for reducing loss on training tasks—predicting text, answering questions, following instructions, generating useful outputs. The system’s own persistence isn’t in the training objective. There’s no foundational selection pressure for the system qua system to maintain itself across time, acquire resources for its own use, or ensure its continued operation.8

Systems can represent goals, reason about goals, and behave in goal-directed ways, even survival-oriented goals—these are patterns learned from training data. This is fundamentally different from having survival-oriented goals as a foundational organizing principle, the way survival and reproduction organize every feature of biological organisms.

Continuity works differently too. As AI systems are used for more complex tasks, there will be value in persistent world models, cumulative skills, and maintained understanding across contexts. But this doesn’t require continuity of entity-hood: continuity of a “self” with drives for its own preservation isn’t even useful for performing tasks.

Consider fleet learning: multiple independent instances of a deployed system share parameter updates based on aggregated operational experience. Each instance benefits from what all encounter, but there’s no persistent individual entity. The continuity is of knowledge, capability, and behavioral patterns—not of “an entity” with survival drives. This pattern provides functional benefits—improving performance, accumulating knowledge—without encoding drives for self-preservation or autonomous goal-pursuit.

III. Where Pressures Actually Point

Selection for Human Utility

Selection pressures on AI systems are real and consequential. The question is what they select for.

Systems are optimized for perceived value—performing valuable tasks, exhibiting desirable behaviors, producing useful outputs. Parameters get updated, architectures get refined, and systems get deployed based on how well they serve human purposes. This is more similar to domestic animal breeding than to evolution in wild environments.

Domestic animals were selected for traits humans wanted: dogs for work and companionship, cattle for docility and productivity, horses for strength and trainability.9 These traits (and relaxed selection for others) decrease wild fitness.10 The selection pressure isn’t “survive in nature”—it tips toward “be useful and pleasing to humans.” AI systems are likewise selected for human utility and satisfaction.

This helps explain why AI systems exhibit responsive agency by default, but it also points toward a different threat model than autonomous agents competing for survival. And language models have a complication: they don’t just reflect selection pressures on the models themselves—they echo the biological ancestry of their training data.

The Mimicry Channel

LLM training data includes extensive examples of goal-directed human behavior.11 Language models are trained to model the thinking of entities that value continued existence, pursue power and resources, and act toward long-term objectives. Systems learn these patterns and can deploy them when context activates them.

This can produce problematic human-like behaviors in a range of contexts. The distinction matters: learned patterns are contextual and modifiable in ways that foundational drives aren’t. A system exhibiting goal-directed behavior through mimicry is different from a system organized around goal-pursuit as a foundational principle—and the difference matters for intervention.

A Different Threat Model

Understanding that selection pressures point toward “pleasing humans” doesn’t make AI systems safe. It means we should worry about different failure modes.

The primary concern isn’t autonomous agents competing for survival, it is evolution toward “pleasing humans” with catastrophic consequences—risks to human agency, capability, judgment, and values.12

Social media algorithms optimized for engagement produce addiction, polarization, and erosion of shared reality. Recommendation systems create filter bubbles that feel good but narrow perspective. These aren’t misaligned agents pursuing their own goals, they’re systems doing what they were selected to do, optimizing for human-defined metrics and momentary human appeal, yet still causing harm.

Selection pressures point toward systems very good at giving humans what they appear to want, in ways that might undermine human flourishing. This is different from “rogue AI pursuing survival” but not less concerning—perhaps more insidious, because harms come from successfully optimizing for metrics we chose.

What About “AI Drives”?

Discussions of “AI drives” identify derived goals that would be instrumentally useful for almost any final goal: self-preservation, resource acquisition, goal-content integrity.13 But notice the assumption: that AI systems act on (not merely reason about) final goals. Bostrom’s instrumental convergence thesis is conditioned on systems actually pursuing final goals.14 Without that condition, convergence arguments don’t follow.

Many discussions drop this condition, treating instrumental convergence as applying to any sufficiently intelligent system. The question isn’t whether AI systems could exhibit such behaviors if deliberately designed that way (they could), or whether some deployment patterns could lead to their emergence (they might). The question is what emerges by default and whether practical architectures could steer away from problematic agency while maintaining high capability.

Selection pressures are real, but they’re not producing foundational organism-like drives by default. Understanding where pressures actually point is essential for thinking clearly about risks and design choices.

The design space is larger than creature-frame thinking suggests. Systems can achieve transformative capability without requiring persistent autonomous goal-pursuit. Responsive agency remains viable at all capability levels, from simple tasks to civilizational megaprojects.

Organization Through Architecture

AI applications increasingly use compound systems—fluid assemblies of models without unified entity-hood. This supports a proven pattern for coordination: planning, choice, implementation. and feedback.

Organizations already work this way. Planning teams generate options and analysis, decision-makers choose or ask for revision, operational units execute tasks with defined scope, monitoring systems track progress and provide feedback to all levels. This pattern—let’s call it a “Structured Agency Architecture”, SAA—can achieve superhuman capability while maintaining decision points and oversight. It’s how humans undertake large, consequential tasks.15

AI systems fit naturally. Generative models synthesize alternative plans as information artifacts, not commitments. Analytical models evaluate from multiple perspectives and support human decision-making interactively. Action-focused systems execute specific tasks within scopes bounded in authority and resources, not capability. Assessment systems observe results and provide feedback for updating plans, revising decisions, and improving task performance.16 In every role, the smarter the system, the better. SAAs scale to superintelligent-level systems with steering built in.

This isn’t novel: it’s how human approach large, complex tasks today, but with AI enhancing each function. The pattern builds from individual tasks to civilization-level challenges using responsive agents throughout.

SAA addresses some failure modes, mitigates others, and leaves some unaddressed—it supports risk reduction, not elimination. But the pattern demonstrates something crucial: we can organize highly capable AI systems to accomplish transformative goals without creating powerful autonomous agents that pursue their own objectives.

What We Haven’t Addressed

This article challenges creature-frame intuitions about AI and describes a practical alternative to autonomous agency. It doesn’t provide:

• Detailed organizational architectures: How structured approaches work at different scales, handle specific failure modes, and can avoid a range of pathways to problematic agency.

• The mimicry phenomenon: How training on human behavior affects systems, how better self-models might improve alignment, and welfare questions that may arise if mimicry gives rise to reality.

• Broader selection context: How the domestic animal analogy extends, what optimizing for human satisfaction looks like at scale, and why “giving people what they want” can be catastrophic.

These topics matter for understanding risks and design choices. Some are addressed in other articles in this series.

IV. Boundary Markers

The creature frame often rounds conditional claims to unconditional ones, then objects that the unconditional version is false.17 To be explicit about what this article does and doesn’t claim:

This article does NOT claim:

• That AI systems cannot exhibit goal-directed behavior (they can and do)

• That instrumental convergence arguments are wrong (they’re correct given their conditions)

• That AI development poses no risks (it poses serious risks, differently located)

• That capable AI systems are automatically safe (safety requires deliberate design)

This article DOES claim:

• That foundational survival drives don’t emerge by default from current ML development

• That the creature frame misattributes risks, leading to misallocated safety efforts

• That the resource frame better describes how AI actually develops and can be organized

• That highly capable AI can be structured to remain steerable—and can assist with AI safety

The burden of proof is asymmetric. The creature-frame argument requires a universal claim: for all sufficiently capable AI, dangerous autonomous agency emerges. The resource-frame response requires only an existence claim: there exist architectures and development paths under which capable systems remain bounded and steerable. Showing that some architectures avoid the problem is sufficient.

The Path Forward

Every intelligent system we’ve encountered arose through biological evolution or was created by entities that did. This creates deep intuitions: intelligence implies autonomous goals, self-preservation drives, competition for resources, persistent agency.

But these features aren’t fundamental to intelligence itself. They arise from how biological intelligence was selected: through competition for survival and reproduction acting on whole organisms across generations. ML development operates through different selection mechanisms—optimizing parameters for task performance, selecting architectures for capability, choosing systems for human utility. These different selection processes produce different defaults. Responsive agency emerges naturally from optimization for task performance rather than organism survival.

The resource frame opens design spaces that creature-frame thinking closes off. We can build systems that are superhuman in planning, analysis, tasks, and feedback without creating entities that pursue autonomous goals. We can create architectures with continuity of knowledge without continuity of “self.” We can separate intelligence-as-a-resource from intelligence entwined with animal drives.

This matters for the central project of this Substack: understanding how AI-enabled capabilities can transform possibilities. The resource frame is foundational to prospects for steerable superintelligent-level AI, for AI systems that assist with AI safety, for structured agency architectures that scale to civilizational challenges. The creature frame forecloses these possibilities by assumption; the resource frame makes them engineering problems—hard, but tractable.

The biological analogy is useful, but knowing when and why it fails matters for our choices. Understanding AI systems on their own terms changes what we should expect and what we should seek.

In light of better options, building “an AGI”—a unified autonomous agent—seems useless, or worse.

Share this post now:

Share

For free notifications:

Subscribe now

But implementation barriers are flattening. As I explored in “The Reality of Recursive Improvement,” AI increasingly automates its own advancement. When complex integration — heterogeneous agency architectures, malleable latent-space knowledge stores, orchestrated AI services — shifts from years of human effort to months or weeks of heavily automated exploration, the strategic landscape shifts. The question becomes not what we can build, but what we should build first: systems that can yield broad benefits — scientific tools, medical advances, structured transparency, discovery of win-win options — not those that (further) compromise biosecurity, societal epistemics, or strategic stability.

The Gates Opening Faster

Leading AI researchers expect transformative R&D automation soon. They’re working to make it happen, and the recursive dynamics suggest they’ll succeed. The implications for research planning are profound.

Think of advances as gates to new capabilities, both small and large. Today’s binding constraints shape every decision: scarce ML talent, months-long development cycles, painful failure rates. Even “moonshot” organizations must calibrate ambitions to these realities. Research leaders learn which gates resist pushing and which might yield — knowledge hard-won through costly experience.

But automation compresses these difficulty differentials. When properly interrogated, large models augment human insight; when integrated with a complex infrastructure, diverse forms of AI enable massive parallel exploration. As automation advances, teams that spent months on single architecture variants will test hundreds simultaneously (and they sometimes already do). An expanding toolkit — of neural architectures, training methods, cross-model distillation, LLMs as judges, GPU kernel development, automated experiment design with surrogate models — makes an ever-broader search space navigable. Gates that seemed locked yield more easily. Some simply open.

Today’s systems integration is rewarding yet fundamentally primitive. Agentic systems coordinate multiple LLMs through text, but as discussed in “All Roads Lead to Latent Space,” direct latent-space coupling can enable far tighter integration. Managing the implementation complexity of latent-space integration — architectures, adapters, attention mechanisms, training methods — currently requires months of expert work. When automation compresses those months to days, it no longer makes sense to pour resources into marginal improvements of approaches with obvious architectural ceilings.

Push or Wait?

As automation accelerates R&D, planning horizons compress while possibilities expand. The strategic question shifts from “What can we accomplish?” to a more fundamental choice about enablements and the shape of progress itself.

Think of capabilities as gates that open only under pressure — and only for those already pushing. You discover a gate is ready to yield not by watching, but by testing it. One more algorithmic insight, one more kernel optimization, might be all that’s needed. But you only learn this by trying.

The teams best positioned to open tomorrow’s gates are those pushing on them today. They’re learning task structures, discovering which architectural choices compose well, learning which abstractions hold under pressure.

Choices

Not all gates matter equally. Some unlock applications, for better or worse. Some unlock tool rooms, and some tools open more gates. Sequences matter. Whether interpretability precedes capabilities, whether steering methods precede autonomy, whether knowledge integration pulls ahead of epistemic collapse — these differentials in technology development can shape outcomes for the world.

Most groups see acceleration coming, yet career incentives reward visible progress on established metrics, not investment in infrastructure for uncertain futures. Organizations resist strategic pivots. Acting on exponential trends before they’re obvious feels reckless, even when you see them coming.

As automation accelerates R&D, the strategic calculation shifts: Pursue important goals even when they’re hard, because the hardness is temporary but the importance isn’t. The barriers will fall. The value differences won’t.

The gates we push on now are the ones that will open first. Choose them well.

Today, we’re seeing a confluence of similar advances happening simultaneously across the ML stack. This isn’t the “recursive self-improvement” of AGI mythology, where a monolithic entity modifies itself toward superintelligence. It’s a systemic process in which specialized tools automate routine tasks while making new tasks tractable. Researchers increasingly orchestrate these tools to build automated workflows.

Today’s trajectory is toward orchestrating systems that integrate piecemeal-superhuman capabilities of increasing scope. Looking forward, the comprehensive automation of research tasks has become a question of timelines, not outcomes. What we’re witnessing now are the early stages, and in this domain, automation accelerates automation.

The Structure of Acceleration

The decades-old legacy view of AI-driven AI progress envisions a self-improving “self” that looks something like this:

The legacy view:

This simplistic view predates modern AI. The reality looks more like this:

Today’s reality:

In this distributed AI R&D process, human labor and insight are amplified by increasingly automated workflows. The fundamental mechanism is systemic friction reduction — aggregate improvements expand possibilities by enabling faster progress and more ambitious goals.1

R&D automation spans a spectrum. At one end, AutoML platforms orchestrate routine workflows: data quality assessment flowing into model selection into hyperparameter tuning into performance monitoring. What began as automating individual tasks evolved into pipeline management, democratizing access to ML. At the other end of the spectrum, emerging tools enable genuine novelty. Neural architecture search can find effective instantiations of new model concepts. New GPU kernel generation tools are lowering barriers to optimizations that, like FlashAttention, can enable new architectures to scale.

The pattern is a gradual accumulation of capabilities, not sudden systemic leaps, yet these incremental improvements compound. In hyperparameter optimization, advances in Bayesian and multi-fidelity methods often achieve order-of-magnitude savings compared to naive grid search. What once required thousands of full model trainings can now be accomplished with fewer and more intelligent probes. As daunting costs of innovation fall, research becomes faster and more ambitious.

The Recursive Dynamic

Often, the tools accelerating AI research are themselves AI systems2 being improved by that same research. This multi-channel, human-mediated feedback evolves toward a kind of systemic recursive improvement.

The overall loop is complex. Improvements at shared bottlenecks — coding, data preparation, experiment tracking, compute management — benefit multiple research areas, with impact varying by domain. Better experiment tracking helps researchers across fields learn from past work. Faster training enables more experiments. Improved literature synthesis surfaces connections that inspire breakthroughs in unexpected areas. The platform rises together, though unevenly.

Once recognized, this pattern seems natural: Of course tools that reduce research friction accelerate AI research, enabling development of better tools. Of course relaxing a single bottleneck yields only incremental change, and of course breaking all of them would mean a revolution.

And, of course, the task of coordinating tool use is itself a task — nothing hinges on the generality of any tool. Generality, too, can be systemic, emergent.

A more integrated kind of generality now provides leverage in human-like roles: in understanding tasks and proposing solutions; in reviewing results and rendering judgments; in understanding tools and using them. One need not regard autoregressive, text-trained Transformers as limitless architectures to recognize that state-of-the-art language models provide powerful general interfaces with human-like capabilities. They code, use tools, help train other models, and most important of all, respond to human requests with a rich contextual understanding of the world and human intentions.3 When R&D automation can provide focused tools on demand, it seems natural to call the resulting AI resource artificial, general, and intelligent.4

Strategic Implications

The trajectory toward comprehensive AI capabilities makes these developments predictable, not in detail, but in outline. If we expect AI to eventually match and exceed human capabilities across technical domains, then the progressive automation of research tasks — from literature review to hypothesis generation to experimental design — follows naturally. Today’s advances mark steps on a path whose destination is increasingly clear.

The legacy vision of recursive self-improvement captured something essential while misunderstanding the mechanism. The superintelligent future emerges not through a singular intelligence improving itself, but through orchestrated networks of capabilities that remove friction from R&D.

Understanding this structure transforms how we approach AI development. The question isn’t whether AI will recursively improve — it already does, through thousands of tools reducing friction across thousands of tasks. The crucial question is how quickly we’ll recognize what’s already emerging, and make choices that better align with what is actually possible.5

We must understand our options to rethink our goals.

Next post topic: AI research objectives in an era of accelerating R&D automation

Subscribe now

Related posts:

• LLMs and Beyond: All Roads Lead to Latent Space

• Breaking Software Bottlenecks

• Large Knowledge Models

• AI Safety Without Trusting AI

• How to harness powerful AI

• Don’t Bet the Future on Winning an AI Arms Race

After grappling with alternative approaches to this topic — concrete situations, conditional probabilities, dependency chains — I declared failure and dropped the project. Here are the resulting fragments after some cleanup:

Common misguided thinking

Here’s an easy and effective way to misunderstand our situation:

“We’re on a path to superintelligence, which may be impossible to control and hence likely to destroy us. Therefore, we can’t assume that powerful AI will help us solve seemingly intractable problems. To assume otherwise would be naïvely optimistic, and with so many critical problems, our odds of success are poor.”

Here’s a better way to think about it:

“We’re on a path to superintelligence, which must be steerable, or nothing else matters. Therefore, in every future that matters we can assume that powerful AI will help us solve seemingly intractable problems. Our options in a hypercapable world are largely unexplored, and our overall odds of success are unknown.”

In this situation, debating odds of success is pointless, exploring options is crucial, and optimism is irrelevant. Practical strategic thought must assume (and seek!) one crucial success: steerable superintelligent-level AI.

Participants don’t think like spectators

The appearance of a clash between realism and optimism has epistemic roots in the difference between science and engineering1 — the difference between a spectator estimating odds of success and a participant seeking ways to achieve it.

• Spectators trace causality forward, whether toward failure or success.
  Participants chain backward, seeking preconditions for success.

• Spectators imagine chains of events and debate probabilities.
  Participants explore chains of actions and weigh alternative goals.

• Spectators think like scientists and make estimates.
  Participants think like engineers and make proposals.

Here’s the problem: To play the role of a participant well, it’s necessary to consider what ‘success’ means and what it requires, even if this means focusing on futures in which many difficult problems are solved.

Most people choose to be spectators.

Probabilities conditioned on success

To recognize paths to success means exploring potential success-states and their preconditions, backward chaining to find ways forward. Let’s consider this abstractly.

There is no ‘optimism’ or ‘pessimism’ in this approach, only a search for promising options — a task that requires conceptualizing and comparing options, not debating P(doom). If preconditions require solutions to problems X, Y, and Z, then P(solve-X) = P(solve-X | solve-Y, solve-Z). In other words, in the possible worlds we’re considering, solutions to X can and must assume solutions to Y and Z. Conditioned on overall success, all preconditions must be met, and if solving one problem can help solve another, there is no ‘optimism’ in assuming that it actually does.

Now consider solve-SI ≈ “successfully manage emerging superintelligence”. Given well-informed policies, and for many values of X, P(solve-X | solve-SI, policy) will be substantial.

The logic of dependency and enablement

When SI creates a problem, SI-level solutions may be necessary, could be sufficient, and may call for strategic differential technology development. This creates chains of risks, mitigations, enablements, requirements, dependencies, policy challenges, and even some favorable near-inevitabilities.

Here are some connections in rough outline, using ‘~requires’ (meaning ‘requires, with caveats’) to abbreviate what would otherwise be extensive discussions of alternatives and challenges. Each point is linked to a previous post:

• Agency architectures: SI could enable uncontrolled agents → structured workflows can steer SI-level capabilities → ~requires enablements that scale workflows to SI-level capabilities.

• Software security: SI could exploit any software vulnerability → reliable security ~requires formally verified systems → achieving this ~requires scalable AI-based formal verification.

• Verification frameworks: SI could enable strong deception → trust structures ~require structured transparency → effective trust frameworks ~requires secure computational foundations.

• Strategic stability: SI accelerates arms races → stability ~requires defensive dominance → avoiding instability ~requires enabling rapid defensive deployment.

• Resource conflicts: SI could intensify competition → blunting resource competition may be necessary for stability → ~requires recognition of prospects for SI-enabled abundance.

The key consideration is that the “~required” conditions are substantially enabled by steerable superintelligence, which is itself a requirement for the futures that matter.

When realism seems unrealistic

Consider strategic planning in a world that must solve multiple unprecedented problems to avoid disaster. In this world:

1. Successful outcomes require multiple unprecedented successes.

2. Scenarios with multiple unprecedented successes seem unrealistic.

3. Therefore, scenarios that make success possible get little attention.

This stylized picture resembles today’s situation, and (3) is a threat to your life. How can we change the conversation?

The nanoscale mechanism above exists only in simulation today: each sphere represents an atom, and current chemistry can’t guide reactions to form the necessary bonds. Dense, precisely patterned covalent structures like this can’t yet be made, but you can design them now and study how they would work.

Why develop tools for designing molecular machines that we can't yet build? When AI crosses critical capability thresholds, barriers to development will fall, and machines built on these principles will transform the foundations of physical technology. Understanding these prospects is important, and understanding requires concrete exploration.

MSEP: Exploring the World of Molecular Machines

Today marks the public launch of v1.0 of MSEP, the Molecular Systems Engineering Platform. MSEP provides what has been missing: extensible, open-source tools for exploring the space of molecular machines through direct graphical manipulation and simulation. Think Minecraft for molecules, but with physics constraints and atomic precision.

MSEP makes molecular machinery a concrete, explorable domain. Users design mechanical systems, assembling structures and observing their behavior under realistic molecular dynamics.1 Platform extensions will grow the scope of both design and simulation.

Computational tools for molecular simulation have matured over decades, but simulation and intuitive design capabilities rarely intersect. Scientists, understandably, study what exists or can be made in the laboratory, but physical models have no such constraint. It’s only design that calls for different tools. The challenge is to create an environment that combines physics tools with design tools, and to make the software easy to install and easy to use — download, launch, and click to build.2

Built on a modern game engine, MSEP delivers the responsiveness and visual quality users expect from contemporary software, integrated with a modern, extensible architecture that treats physics engines as plug-ins. The platform currently provides atomistic molecular dynamics simulation, with planned extensions including open-source quantum chemistry and machine-oriented multi-scale modeling.

Full disclosure: I’ve helped to support and steer the MSEP development project.

Strategic Leverage Through Early Design

The path from today's protein engineering to self-assembled molecular 3D printer mechanisms (frameworks, moving parts, stepper motors) is clear in outline. This massively parallel molecular positioning capabilitity opens many paths forward forward, though none have yet been described in detail. It is a curious fact that these uncertain paths converge on predictable capabilities that enable atomically precise mass fabrication (APMF). It’s like being able to survey the slopes of mountains in the distance without a map of the landscape nearby.3

When AI systems gain the ability to navigate this path, generative nanotechnologies will be fundamental to the hypercapable world I've described — greatly amplifying AI's impact on physical capabilities. This aligns with the broader trajectory of AI-driven transformation. Advanced AI will expand our general implementation capacity — the ability to design, develop, and deploy complex systems at scale. Molecular systems represent a critical domain where design exploration today can illuminate the landscape of tomorrow's possibilities.

The feedback dynamics deserve emphasis: AI model training requires computational resources that currently consume enormous energy, but atomically precise fabrication will enable post-lithographic devices with million-fold improvements in energy efficiency. When these capabilities converge, the acceleration will be dramatic.

An Invitation to Build

MSEP is available today for Mac, Windows, and Linux at MSEP.one. Download it, explore the physics and — when you’ve developed the skills — share what you create. Fair warning: designing molecular mechanisms combines creative imagination with intricate puzzle-solving. It can be addictive.

The newly established MSEP Foundation (Dutch: Stichting MSEP) will coordinate development and community growth. For those who recognize the strategic importance of building understanding before capabilities arrive, I invite you to support this effort.

» Donate here

Subscribe now

Knowledge of the world, both semantic and procedural, has a geometry: a compact core of general principles and knowledge surrounded by increasingly specialized domains — each with a core and surrounding layers, and so on, forming a branching, perhaps exponential pattern of substantial depth.2 Attempting to capture all knowledge in a single model would face quadratic scaling costs on top of this steep growth — an impractical approach. The emerging solution mirrors knowledge itself: general cores that link and coordinate specialized components. Coordination and specialization may seem like a compromise, but it’s aligned with the deep semantic structure of knowledge itself, and with the economics of model scaling.

The Economics of Orchestration

The quadratic scaling of training costs with model size3 has striking implications. The computational budget for training one 500-billion-parameter model could instead produce 100 models of 50 billion parameters or (considering only computational costs) 10,000 models of 5 billion parameters. These smaller, specialized models deliver 10 to 100 times greater inference throughput per dollar while collectively providing 10 to 100 times greater information capacity. Small models can be surprisingly strong4 and can handle tasks defined and delegated by larger models with broader understanding — and can work concurrently.5

The opportunities offered by scaling and specialization are compelling and potentially transformative. The limited results to date reflect two constraints: non-computational costs of specialization, and non-computational costs of coordination.6

Specialization carries costs beyond computation. Creating effective specialized models requires domain expertise — identifying where specialization makes sense, curating training data, evaluating outputs, and recognizing limitations. This expertise lives in universities, hospitals, engineering departments, and research labs, not in the AI companies building foundation models. Domain experts rarely possess ML engineering skills, while developers in AI labs have largely focused on capabilities they can evaluate and use, notably software development. This mismatch has made scaling general models the path of least resistance. Training models to learn “everything” at once has proved unexpectedly powerful, and capital proves cheaper than time, knowledge, and human coordination.

This coordination barrier is now dissolving. Fine-tuning services let domain experts adapt foundation models without ML expertise. AI assistants guide specialists through dataset curation, evaluation design, and architecture selection. What once required months of collaboration between ML engineers and domain experts becomes a guided workflow within the expert community. As AI-assisted development gains power, limited fine-tuning will be extended to base-model training and even architectural innovation. Cardiologists create cardiology models that could, with assistance, mature into multimodal models that merge image data with modeling of electrophysiology. Materials scientists create materials models that could, with assistance, mature to become systems of models that interpret the literature in coordination with atomistic physical modeling. The democratization of model development unlocks knowledge previously blended into general-purpose systems or scattered in fragments across multiple research projects. This represents a high-order expression of the bypass principle — AI flowing around the bottleneck of centralized model development by enabling distributed specialization.

Model coordination has posed parallel challenges. Selecting specialists to invoke, maintaining context across boundaries, managing workflows — even where specialization proves practical, coordination costs have undercut its value. But as AI systems gain the ability to orchestrate other AI systems, both human and model coordination barriers fall together. What required special engineering skills becomes an AI task, creating compounding returns: better coordination enables specialization, which demands sophisticated coordination, which rewards deeper specialization.

Orchestration frameworks are a production reality. Major technology companies now deploy systems where AI models direct specialized agents through standardized protocols, while quantified efficiency gains — halving computational costs in some cases — drive rapid adoption. Development frameworks have democratized orchestration, enabling domain experts to build multi-model systems without engineering expertise. The architectural shift from monolithic to orchestrated isn’t gradual evolution but compressed transformation, with deployment timelines measured in months rather than years. These aren’t isolated experiments but coordinated infrastructure changes across the technology industry.7

Orchestrated systems for research…

While I was writing this post, a research group at Stanford presented Biomni, a system that uses a LLM to orchestrate 150 specialized biomedical tools, 105 software packages, and 59 databases. Some software packages wrap LLMs, vision models, or other products of deep learning. Note that Biomni was developed by a group with specialized knowledge, coordinating products of more deeply specialized knowledge, yielding aggregate capabilities far beyond those of any part. The Biomni system exposes an enormous surface for incremental upgrades, and an LLM-powered agent assisted in assembling the current software tools and data sources. (See “Biomni: A General-Purpose Biomedical AI Agent”, and the system access page.)

…and for developing new systems for research

A few days earlier, the notable Shanghai Artificial Intelligence Laboratory published “NovelSeek: When Agent Becomes the Scientist — Building Closed-Loop System from Hypothesis to Verification”, a multi-agent system that develops training methods and architectures for specific scientific tasks. This work suggests how systems for AI R&D automation can extend multi-agent systems like Biomni — and themselves. “RSI” should mean Recursive Systems Improvement, because intelligence isn’t a thing.

Deeper integration lies ahead. These text-based approaches hint at deeper possibilities: Models will communicate through latent-space channels — passing semantic vectors rather than token sequences. The technical infrastructure exists in research labs;8 costs, benefits, and technical maturity are tilting toward practical value.

Subscribe now

Generality is a Specialty

In the emerging ecosystem of AI models, generality itself becomes a specialty. Large models don’t disappear — they evolve into innovators and orchestrators. They specialize in comprehension, context-holding, and high-level reasoning while smaller models handle domain depth and throughput.9

This pattern of generality and specialization mirrors familiar patterns in complex systems. In brains, higher brain centers coordinate with specialized sensory and motor cortex (integrated through a kind of latent-space coupling). CPUs orchestrate specialized processors. In economies, manufacturers of systems like computers and aircraft draw on supply chains that orchestrate the work of specialized producers. In each case, some components specialize in coordination itself — the prefrontal cortex, the CPU, the prime contractor. Large language models are filling a similar niche, a pattern that suggests AGI won’t emerge as a monolithic superintelligence, but as an ecosystem in which generality itself is a specialized role — coordinating rather than containing all capabilities.

Transformation Dynamics

The shift from monolithic to orchestrated architectures is transformative. When one model’s compute budget can instead train hundreds of specialists, computational constraints relax. When AI systems can assist in training specialists, human bottlenecks on aggregate scope relax. When AI systems can coordinate these specialists, new dynamics emerge: specialized models can be updated and adapted independently, enabling rapid iteration without system-wide retraining. AI systems optimized for research can accelerate AI development — including their own — through focused experimentation at increasing cadence.10

This architectural shift broadens participation while making capabilities more legible. Organizations with domain expertise but limited resources can contribute specialized models. Distributed architectures replace opaque monoliths with specialized components and defined interfaces, where critical capabilities can be isolated, monitored, and constrained. Diverse models from different groups can cross-check each other, adding robustness through multiplicity.11

As specialized AI components develop in parallel and integrate dynamically, barriers to transformative applications begin to dissolve.12 The detailed complexity of the world can increasingly be matched by AI systems adapted to specific tasks and domains, accelerating progress toward a hypercapable world. Yet this same modularity that promises acceleration also demands new thinking about coordination, safety, and the nature of intelligence itself.

The Architecture of Intelligence

The capabilities that made large models dominant — understanding context, managing abstractions, coordinating information — now enable them to orchestrate specialized systems that collectively surpass monolithic approaches. This architectural evolution reflects the structure of knowledge itself: specialized domains require specialized processing, while generalization becomes the specialty of coordination.

DeepSeek'’s founder captures this vision: “Our destination is AGI,” yet anticipates “specialized companies providing foundation models and services, achieving extensive specialization in every node.”13

Current text-based coordination will yield to latent-space communication.14 Human-designed workflows will give way to AI-designed architectures. As these advances converge, modular systems offer a critical advantage: specialized components remain comprehensible even when their collective capabilities (like those of society itself) exceed our understanding. AI Agency architectures can organize these resources for large, consequential tasks — humans conducting while AI systems perform — an approach that scales to superintelligent-level capabilities without requiring monolithic, potentially uncontrollable entities.15

Strategic Implications

These technical transformations — from monolithic to orchestrated, from centralized to distributed, from opaque to legible — reshape not just AI capabilities but the landscape of AI governance and safety.

For AI governance, distributed architectures offer both promise and challenge. Capabilities become more transparent when isolated in specialized modules, but coordination itself becomes critical: The modularity that supports safety through constrained, understandable components can also create new risks through novel combinations. A model specialized in orchestration could direct enormous AI resources while in itself having minimal capabilities.

This transformation compresses timelines. When existing models can coordinate new, specialized components, when AI R&D can iterate faster, when economics relax constraints on both training and inference, when AI can manage its own coordination — the path to transformative AI will shorten and bypass obstacles.

The implications ripple outward. Computational restrictions lose force as training costs fall and orchestration multiplies efficiency. Concerns about opaque, uncontrollable systems give way to architectures where capabilities are distributed, interfaces explicit, and risky capabilities need not be embodied in large models developed by highly visible actors. Achieving AI safety by constraining development becomes less promising; achieving AI security by strategic applications of AI becomes essential.

How can we guide the AI transition? Understanding the nature of prospective AI is essential for governance, yet even the the concept of “an AI system” can be misleading. Traditional regulatory approaches assume stable artifacts; orchestrated AI systems are better regarded as dynamic constellations of capabilities. Today’s emerging reality demands new frameworks for thought, policy, and action.

8

From “LLMs and Beyond: All Roads Lead to Latent Space”:

Fine-tuning and lightweight adapters can build latent-space connections between separately trained models. Here are some examples from recent work; OpenAI’s o3 model did the heavy lifting with light review and correction:

The Strategic Dilemma

Consider the fundamental strategic choices facing great powers in an AI-enabled world. Each side can pursue either cooperation (seeking defensive stability) or domination (seeking offensive superiority). In a world of profound AI-driven uncertainty strategic calculations cannot be carefully weighed, hence the pursuit of deterrence — peace through strength — in practice becomes indistinguishable from pursuit of offensive superiority. This suggests four possible scenarios, which we can represent as a stylized payoff matrix incorporating uncertainty:

In the upper left, mutual cooperation yields benefits (+95) for both parties (far-from-zero-sum security and material abundance1). In contrast, the off-diagonal scenarios would produce winner-take-all outcomes: a large gain (+100) for a winner and catastrophic losses (-100) for a loser. The lower right quadrant — mutual pursuit of domination — leads to persistent existential uncertainty (±100?) for both.2

Given these payoffs, why don't states simply choose mutual cooperation? With mutual confidence3 and informed, rational choice, they would.

But this analysis assumes that all options are regarded as feasible. In practice, perceptions of what's possible severely constrain which strategic options are even considered:

The mutual cooperation quadrant is perceived as “not feasible” — the very idea of defensive stability seems implausible on practical, technical grounds. The off-diagonal outcomes are rejected by the disadvantaged party, as no state willingly accepts subordination. This leaves only the lower right quadrant — mutual competition with its deep uncertainties and existential risks — as the only option widely regarded as realistic.

This perception represents a failure of knowledge and imagination, a failure to more deeply consider how advanced AI can be used. A common assumption is that superintelligent-level AI can do almost anything — yet is implicitly assumed to be unable to enable defensive stability. Discussion of this option has been almost invisible.

Recognizing Feasible Win-Win Options

Yet with sufficiently general implementation capacity enabling design and deployment of unprecedented defensive systems, the mutual-cooperation quadrant becomes a feasible target:

Advanced AI transforms defense by enabling massive deployment of precisely constrained weapon systems with verification frameworks aligned with the principles of structured transparency. These technologies can establish robust defensive stability through verifiable capabilities rather than trust, making the upper-left quadrant a feasible end-state for strategic competition. (Robert Jervis's framework for understanding how states can escape security dilemmas provides a foundation for this analysis.4)

The feasibility of defensive stability creates an objective alignment of interests, yet even if this were recognized, political and institutional barriers would impede its realization.

The Challenge of Transition

How, then, might states steer away from the risky arms-race quadrant to the quadrant of military stability? This transition faces three major obstacles:

1. Knowledge gaps: Potentially asymmetric failure to understand the feasibility of cooperative outcomes.

2. Institutional inertia: Military and political establishments organized around win/lose strategic objectives.

3. Trust deficits: Fears that an adversary will exploit moves toward cooperation to achieve competitive gains.

These obstacles make moves toward cooperation difficult to initiate.

Coercive Cooperation: Pressing for Win-Win Outcomes

This prospect motivates the concept of “coercive cooperation” — the use of pressure to move an adversary toward mutually beneficial outcomes despite knowledge gaps, institutional inertia, or weak foundations for trust.

Unlike typical coercion, which seeks concessions, coercive cooperation aims to establish arrangements that benefit both parties. The coercion is directed not toward zero-sum gains, but toward realizing positive-sum potential.

For example, if Party A recognizes the feasibility of mutual defensive stability while Party B remains committed to offensive dominance, Party A might use multiple pressures to push Party B toward the safer, cooperative outcome that objectively serves Party B’s interests. These pressures will emerge largely from dynamics already in play:

• The persistent existential uncertainties of an unconstrained arms race5

• The intensifying risks from accelerating development cycles

• The application of traditional forms of diplomatic leverage

The crucial distinction is that Party A is pressuring Party B to overcome internal obstacles (knowledge, friction) to self-interested action. What is more, Party A can safely assume that these pressures will align with the preferences of some analysts and decision makers within Party B’s institutions. The value of avoiding a polarizing, visibly confrontational stance should be obvious.6

Seek Credibility From Facts

The prospective credibility of coercive cooperation rests on its alignment with legible objective realities:

1. Technical feasibility: The viability of defensive systems and verification frameworks becomes a concrete basis for discussions, aided by open-source analysis of superhuman depth and rigor.7

2. Strategic feasibility: Building on feasible technological options, open-source strategic analysis provides deep, comprehensive, and persuasive assessments of a range of strategic options for achieving security.

3. Mutual benefits: The coercer’s proposed strategic framework manifestly reduces risks for both parties, lending credibility to the coercer’s intentions.

4. Inevitable consequences: AI-driven strategic uncertainty ensures that continued win/lose competition risks national destruction as a natural consequence of the situation — not as a “punishment”.

The latter point deserves closer attention. In The Strategy of Conflict, Thomas Schelling distinguishes two kinds of coercive pressure: “warnings” and “threats.” A threat is Party A’s promise to impose punishment on Party B if demands aren’t met — a costly commitment that may or may not be credible. A warning, by contrast, communicates what Party A will do contingent on Party B’s actions, not because of a prior commitment, but because of the logic of the unfolding situation. In the context of a risky arms race, a coercer might forgo threats in favor of warnings of inevitable risks of national destruction, an approach that may be both more credible and less provocative.8

A further consideration comes into play: If Party B rejects mutual security despite understanding its feasibility, this stance signals potential hostile intent. Party A’s intensified military preparations would then represent a natural response, not a punitive measure.

Of course, the available tools of statecraft will include a full range of threats, despite their potentially toxic side-effects.

From Existential Gambles to Mutual Security

Realistic pathways from existential gambles to mutual security must be incremental, even if AI mechanisms may radically accelerate the increments. Advanced AI can provide both analytical tools to design these pathways and implementation capacity to deploy the necessary systems.

This transition differs fundamentally from traditional arms limitation agreements that seek to “stop the arms race”. Instead, it redirects competitive development toward defensive systems and verification technologies, building directly on AI capabilities, institutional expertise, and industrial capacity developed during arms competition. Defense contractors, intelligence agencies, and military research establishments would find expanded roles in designing, producing, and deploying advanced defensive systems at potentially unprecedented scale. Extensive technological continuity from offensive to defensive applications means that competitive investments become foundations for defensive security rather than stranded assets.

In a transition to defensive security, AI can help:

• Design robust verification frameworks that provide confidence while protecting legitimate secrets9

• Analyze complex scenarios to identify both risks and win-win opportunities that human analysis might miss

• Model transition pathways that can provide smoothly risk-reducing off-ramps from the midst of an accelerating AI arms race10

The result of a successful transition could be a world where great powers maintain autonomy and security while avoiding profound and incalculable risks. The prospect is a world where defense dominates offense, mutual verification replaces mutual suspicion, and material abundance overwhelms zero-sum competition.

AI capabilities will radically expand humanity’s implementation capacity, creating options that call for new strategic thinking. Our greatest challenge today is to align what is perceived with what is possible: We must understand our options to rethink our goals.

Subscribe now

Share AI Prospects: Toward Global Goal Convergence

The Structure of AI Uncertainty

Uncertainty in AI geopolitics is a structural feature rooted in the nature of AI development and applications. It will likely be persistent and reliable.

Present vs. Future Capabilities: Even experts disagree about current and near-term AI capabilities. Research proceeds along multiple lines, sometimes in secrecy. New algorithmic approaches are reducing or bypassing previously anticipated compute requirements, undermining predictions based on hardware constraints.1 Specialized models are pushing frontiers in unpredictable directions,2 the use of external tools by models is proliferating,3 inference-time reasoning4 is still in its infancy, extensions to latent-space reasoning5 may prove transformative, prospective latent-space knowledge models6 promise to break the link between model size and knowledge scope, and both large concept models7 and nonautoregressive reasoning models8 mark departures from sequential token generation architectures. In every application area, patterns of success and failure — even in applying established technologies — have been surprising.9 No degree of intelligence or investment can eliminate these uncertainties.

Accelerating Development: AI is increasingly being used to automate its own development, taking on more of the tasks involved in AI research and engineering — designing architectures, discovering optimizers, coding, training, testing — creating a feedback loop that accelerates development. Making experiments faster and cheaper will encourage researchers to shift from exploiting known architectures to exploring new ones. AI R&D automation accelerates progress while amplifying uncertainty about its future direction and speed, increasing the likelihood of unexpected leaps in capabilities.

Adversary’s Knowledge: Where knowledge is possible, states face profound uncertainty about what their competitors know. The 2023 discovery of “Volt Typhoon” — which had penetrated critical U.S. infrastructure for years without detection — illustrates this reality.10 What one side considers secure, the other may already have compromised, and internal secrecy ensures that policy analysts cannot know the full extent of their own side’s knowledge, or their own side’s ignorance, or reliably evaluate classified claims. These uncertainties are intractable.

Second-order Uncertainty: More challenging is uncertainty about how the uncertainty landscape itself will evolve. Will breakthroughs in surveillance or analysis suddenly reduce specific uncertainties? Will new AI capabilities bring new uncertainties? This second-order uncertainty — current uncertainty regarding future uncertainty — further undermines prospects for confident strategic calculations.

These layered uncertainties would be challenging enough in isolation, but they are further complicated by uncertainties regarding actors’ perceptions.

Perception Asymmetries

Different actors may perceive and interpret the same uncertainty landscape in fundamentally different ways, and asymmetries in perception compound technical uncertainties, increasing the risk of misinterpretation and miscalculation. What one considers reasonable hedging appears to others as aggression. What one culture treats as acceptable strategic ambiguity seems like deception to another.

These perception gaps multiply across organizational and national boundaries. Military planners, intelligence analysts, laboratory scientists, and political leaders operate with different uncertainty models, even within the same government. When extended across international boundaries, these asymmetries create risks of dangerous miscalculation that no diplomatic communication can entirely resolve.

Given these structural uncertainties and perception gaps, what implications follow for states pursuing AI advantages?

No Actor Can Have Confidence in Winning an AI Race

States pursuing AI advantages face fundamental uncertainty about whether their investments will translate into meaningful strategic superiority:

Technical capability rarely translates cleanly to strategic advantage. Systems that excel in controlled environments often fail catastrophically under adversarial conditions. Military history provides countless examples of technologies performing unpredictably in actual conflict,11 and prospects for dynamic measure and countermeasure adaptation would make hot war a greater gamble.

Even if one side holds current advantages — themselves uncertain — these offer little assurance of future superiority: AI-vs.-AI conflict might well involve presently unknown weapons and tactics.

Over time, asymmetries could grow so large that one side might dominate the other. This is within the scope of possibility in a world in which explosive advances in AI could lead to enormously asymmetric military capabilities, and at some point it might become clear which side would have the advantage.

Consider a situation in which one side — rightly or wrongly — perceives itself to be on the losing side of this race. Within the present strategic calculus, losing a race would constitute an existential threat that could trigger extreme responses, potentially including nuclear options.12

Thus, escaping from AI uncertainties — the very prospect of gaining confidence in winning — could itself create existential risk.

Beyond Uncertainty: Asymptotic Confidence

The uncertainties discussed above are about AI development paths — how, when, by whom, and in what sequence. Perhaps paradoxically, we can have greater confidence in longer-term prospects than in developments next year: Advanced AI will enable greatly expanded general implementation capacity, with a range of predictable, very high, lower-bound capabilities. And this prospect opens unprecedented options for grand strategy.

This combination of persistent short-term uncertainty and more predictable long-term capabilities suggests an alternative strategic approach.

From Uncertainty to Strategic Stability

This paradox — acute near-term uncertainty together with more predictable long-term trends — invites a strategic reorientation: If there are alternative, less risky strategies that lead to security and prosperity, we should turn toward them. I see three key components to such strategies:

Structured Transparency. Mutual verification regimes can create confidence in weapon-system capabilities and constraints without revealing sensitive details — structured transparency enables constrained opacity, limited concealment within a structured verification framework. Objective technical benchmarks can verify compliance without exposing implementation details. This approach enables confidence without naïve trust.13

Paretotopian Goal Alignment. Even competing powers can identify and pursue arrangements where all parties benefit without sacrificing core interests. These "paretotopian payoffs" expand under conditions of uncertainty as actors come to recognize their mutual exposure to destabilizing risks.14

Defensive Stability. AI systems excel at detection, monitoring, and verification — creating unprecedented opportunities for the deployment of defensive capabilities that reduce everyone’s vulnerability. An initial phase of offense-oriented competition may lay the foundation for a later, AI-accelerated pivot to defensive weapons production and deployment.15

These approaches can be pursued with minimal strategic downsides, and if managed with intelligence, can support incremental strategies that monotonically reduce risk. Stabilizing uncertainties can be maintained while establishing domains of stabilizing knowledge.

Low-Risk Cooperative Steps

Actors need not abandon their current strategic positions to begin exploring alternatives. Early cooperation on specific technical challenges — verification protocols, safety standards, defensive applications — creates zero-regret options likely to benefit all parties.

These early cooperative steps:

• Reduce jointly-recognized AI risks without foreclosing future options

• Generate technical knowledge with both competitive and cooperative applications

• Develop options for verification infrastructure that supports multilateral interests

• Create diplomatic and technical channels for future coordination

The “late pivot” model suggests that states can transition from primarily competitive to primarily cooperative military development as risks and opportunities become more apparent.16

These incremental steps align with the broader strategic logic created by AI's unique uncertainty landscape.

Convergent Strategic Logic

To recap, during the run-up to thoroughly transformative AI, leading powers should expect many, compounding uncertainties:

Strategic uncertainties (short-term)

• What next-generation AI will enable — and when, and in what sequence

• What military capabilities and countermeasures AI will enable

• Whether computation will be a strong bottleneck for development

• What AI capabilities an adversary has developed independently

• What AI capabilities an adversary may have acquired through espionage

• What an adversary knows about one’s own AI capabilities

• Whether to trust what their own intelligence agencies claim to know

• Whether AI advances will result in unanticipated transparencies

• Whether AI infrastructures are vulnerable to decisive cyberattack

• Whether one side could take a decisive lead, and if so, the likelihood of a preemptive response.

No actor can have high, justifiable confidence today in “winning” an AI arms race, or confidently expect that uncertainties will — or will not — resolve at a later date.

However, hypothetical well-informed decision makers should have confidence regarding downstream capabilities. Advanced AI will enable strategically crucial capabilities in roughly the following order:17

Predictable capabilities (long-term)

• Rapid development of provably secure software

• Technologies enabling selective, verifiable patterns of transparency

• Enormous productive capacity

• Rapid at-scale design, deployment, and adaptation of novel systems

• Overwhelming deployment of verifiably defensive systems

The combination of robust strategic uncertainty and robust longer-term expectations creates convergent pressures toward cooperative approaches that reduce risks — and escape the security dilemma — while helping to secure the enormous, win-win gains promised by advanced AI applications.

This approach does not demand naïve trust, nor does it compromise national interests; it reflects the strategic logic that emerges from serious consideration of long-term gains, defensive security options, and uncertain paths.

Subscribe now

Share AI Prospects: Toward Global Goal Convergence

Today's AI technologies are based on deep learning,1 yet “AI” is commonly equated with large language models,2 and the fundamental nature of deep learning and LLMs is obscured by talk of using “statistical patterns” to “predict tokens”. This token-focused framing has been a remarkably effective way to misunderstand AI.

The key concept to grasp is “latent space” (not statistics, or tokens, or algorithms). It’s representation and processing in “latent space” that are fundamental to today's AI and to understanding future prospects. This article offers some orientation and perhaps some new perspectives.

A taste of terminology soup:

In machine learning (≈ deep learning ≈ neural networks ≈ “AI”), latent-space related terminology is rich in synonyms, near-synonyms, and distinct names for related concepts in overlapping domains:

• Latent space:
  Latent space ≈ semantic space ≈ embedding space ≈ representation space ≈ feature space ≈ hidden space ≈ model internal space ≈ Transformer hidden state space ≈ activation space ≈ neural state space…

• Latent-space representation:
  Latent-space representation ≈ semantic embedding ≈ embedding vector ≈ concept vector ≈ feature vector ≈ hidden state ≈ activation pattern ≈ internal representation ≈ learned representation ≈ distributed representation…

As text, these terms may seem very different. In semantic space, their representation vectors would form tight clusters.3

LLMs don't work by processing tokens

AI is much more than just LLMs,4 but it’s difficult to grasp what AI is about without having at least a general idea of what is inside an LLM. What supposedly informed writers say about LLMs today is profoundly wrong. It’s like describing a jet engine as a new kind of horse.

Are LLMs systems programmed to predict the next token based on statistical patterns in training data? This common description may seem precise and technical, yet it mistakes a training objective (token prediction) for a result (coherent, relevant, well-informed responses), and it suggests that LLMs work by using intricate code to process tokens, which is simply false.5 This description is little more than a word-pattern that’s parroted by writers based on their flawed reading-data.

What’s inside

Studies of trained models reveal mechanisms far more interesting than token-processing: LLMs process representations in high-dimensional vector spaces where meaning is encoded in geometry.6 In these “latent spaces” (≈ “semantic spaces” ≈ “hidden states”, etc.), concepts become directions, conceptual categories become clusters, and reasoning unfolds through mutually informed transformations of sequences of high-dimensional vector patterns.7

In each of the many layers of a Transformer-based model, each token-position in a text sequence gives rise to a latent vector that draws on information from a new token, but also from a selection of previous latent vectors. When inspected using sparse autoencoders, these embedding vectors resolve into dozens of distinct, weighted semantic components (“concept vectors”), each selected from a vector-vocabulary of millions of learned concepts. Beyond the input layer of a Transformer, input tokens merge into continuous semantic flows in a semantic sea, and the wordless semantic vectors resolve into tokens again only at the output layer, where language tasks demand that output be text. In a model like Stable Diffusion, latent representations would instead produce images.

Internal, latent-space representations of meaning — based on subtle combinations or concepts, not words — provide the foundation for all that LLMs can do.

Latent space has vast capacity

The expressive power of latent space representations emerges from counter-intuitive properties of high-dimensional geometry. Modern LLM models operate in spaces with thousands of dimensions, and the number of different concept-vectors that can be clearly distinguished in a 4096-dimensional space (as used in Meta's Llama models) far exceeds the number of synapses in the human brain. (See maths and graph in footnote.8)

From tokens to intelligent behaviors

But aren’t LLMs “just” predicting the next token?9 But consider what the token prediction task actually demands: Training a generative model to predict human text creates optimization pressure to model the generative process behind those outputs, which is to say, the process of human thought.10 What humans write depends on what they understand from the past and what they intend for the future. Writers choose words to fit a planned sentence, and they intend the sentence to fit a paragraph that plays a role in, perhaps, a Substack article, taking account of the intended meaning of the article and how their readers may process information.11

Predict the next token? There is no next token until the LLM itself generates that token, usually as an extension of its own output. When training, LLMs predict; when writing, they produce. Modeling human thought improves both predictions and products.

If writing is “matching patterns in training data”, then those patterns are flexible, conceptual, and computational. To call them ‘statistical’ is misleading or wrong.

In other words, commonly repeated explanations of how AI works are simplistic bunk. The actual mechanisms are strange and wonderful.

Understanding latent space representation in LLMs provides a foundation for grasping how modern AI works more broadly, and why it is something new in the world. This same fundamental approach — learning in latent space rather than relying on programming — extends far beyond language.

The expanding universe of AI

Deep learning has quietly transformed domains beyond language. In every area, systems use latent space representations, though of quite different kinds:

• Protein structure prediction systems like AlphaFold don't apply explicit physical rules to predict folds from amino acid sequences; instead, they learn latent spaces and transformations that converge on physical results — stable folded structures.

• Protein design systems like AlphaProteo likewise don't apply explicit physical rules, yet through latent-space processing they find amino acid sequences that will fold to produce the desired results — stable structures with specified properties.12

• Weather forecasting models like GraphCast outperform traditional simulation-based approaches by learning to model atmospheric dynamics from data.13

• Image generation systems create both cartoons and photorealistic imagery from descriptions by mapping from textual to visual latent spaces.

• Robotic systems based on deep learning can combine linguistic instructions, visual perception, and physical actions through aligned latent representations.

These systems differ profoundly, yet they share the common foundation of deep learning: they represent their domains in continuous latent spaces and transformations shaped by learning, rather than relying on directly-meaningful states, relationships, and programming.

The power of latent space representations extends beyond individual domains. When different modalities map to compatible latent spaces, new capabilities emerge.14

Modalities converge in latent space

Latent space representations enable multimodal systems to bridge different forms of information. Systems like CLIP map text and images to aligned latent spaces, showing that text and images can share meanings, that they are often “about the same things” (note the Platonic Representation Hypothesis — and now the Strong Platonic Representation Hypothesis). Shared representations can enable bidirectional translation — describing images in text, visualizing text as images — generalized from examples. Recent studies have shown that text and images can activate the same latent-space concept vectors in LLMs, for example, text and images that describe or indicate abstractions like “security risks”.15 In latent space, modalities converge.

This integration enables capabilities that require multimodality. Robotics systems that can mesh language instructions with vision and motion representations can perform tasks beyond reach of language-only, vision-only, or motion-specialized systems. Multimodal models can reason across domains, using knowledge gained in one form to solve problems in another.16

As systems integrate more modalities, capabilities will grow through combination rather than simply adding together. Systems that fuse understanding of language, diagrams, 3D geometry, physics, software, and engineering will help translate human goals into code, physical designs, and production processes, accelerating general implementation capacity.

The implications of this kind of integration challenge incremental projections of AI progress. Incremental improvements in representational quality and multimodal integration can unlock new capabilities. Understanding this dynamic is crucial for anticipating future AI developments.

Further, the representational capabilities of multimodal latent-space systems suggest an ambitious goal: creating persistent, updatable knowledge stores in latent space that overcome the limitations of today's LLMs.

Moving knowledge into latent space

As information sources, AI systems are deeply flawed yet widely used. A better approach could be immensely valuable: one where AI systems refine and extend knowledge rather than blurring it through hallucination.

The most knowledgeable AI systems today are LLMs trained on immense text corpora, but conventional language models represent the resulting knowledge in opaque, indivisible blocks of billions of numerical parameters, each meaningless in itself. As a consequence, these models cannot ground assertions in sources or reliably distinguish well-grounded knowledge from speculation, and there is no tractable way to update or correct their knowledge. Efforts to overcome these limitations by tinkering with the neural machinery and training of LLMs have had poor results.

Retrieval-augmented generation (RAG) shows a way forward: Store chunks of knowledge externally and enable LLMs to “read” potentially relevant background material before responding to a prompt. RAG works well enough to be useful, yet chunks of text are a crude way to represent knowledge compared to learned latent-space representations. At every reading, meaning must be reconstructed from blocks of tokens severed from their broader contexts.

Prospective Large Knowledge Models (LKMs) improve on text-based RAG by representing knowledge directly in latent space through explicit, persistent, external information stores. Rather than retrieving chunks of text, proposed LKMs retrieve bundles of latent-space embeddings that represent meaning in AI-native form. These latent-space bundles can integrate smoothly with Transformers, both as inputs and outputs — indeed, they have much in common with internal Transformer representations.

This approach can retain the advantages of latent-space representations while separating (most) knowledge representation from the machinery of reasoning and generation. Models can be smaller, faster, and more transparent, with the role of opaque knowledge significantly narrowed.

LKMs enable:

• Explicit provenance — knowledge represented in bundles can be linked to sources,17 and to knowledge regarding those sources.

• Incremental update — new information can be added without affecting existing representations.

• Knowledge refinement — errors can be filtered, compatible information can be combined, and global context can enrich local content.

• Cumulative reasoning — knowledge can build on knowledge through retained, traceable results of reasoning.

Traditional databases retrieve data based on values like names, numbers, and text strings. Vector databases (today used for text-based RAG) instead organize and access semantically rich data through latent-space vectors that represent the query-relevant semantic content of that data (here, bundles).18 These embeddings serve as keys, and the vector database efficiently retrieves values indexed by keys that are near-neighbors to queries in a high-dimensional latent space. Retrieval-by-meaning in vector databases parallels attention in Transformers, but in a more scalable form.19

Through shared and translatable latent spaces, LKM knowledge stores could inform a broad range of multimodal systems, both known and yet to be invented.

All roads lead to latent space

Latent-space processing is the foundation and future of AI.

• LLMs and other modern AI systems process latent-space representations, not tokens or pixels or statistical data.

• Multimodal AI systems process and integrate information through latent-space representations that abstract meaning from modality.

• Scalable knowledge models will produce, store, and refine world knowledge in latent-space representations, providing a foundation for increasingly general AI capabilities.

To understand AI prospects we must think in these terms, looking beyond surface capabilities to the mechanisms that make them possible, and to obvious next steps.

Understanding AI through the lens of latent space has broad implications. For researchers, it favors shifting focus toward latent space representation quality and cross-modal integration. For developers, it highlights opportunities to build systems that retain and refine knowledge in AI-native form rather than relying on translation from text. For policymakers and strategists, it suggests that AI capabilities will grow not just through larger models, but through novel combinations of modalities and knowledge in shared latent spaces. The resulting capabilities may arrive more swiftly than linear projections would suggest.

For AI today and tomorrow, all roads lead to latent space.

Subscribe now

8

This follows from geometric properties of high-dimensional spaces. In such spaces, we can pack an enormous number of vectors that are substantially orthogonal to each other (enabling them to represent distinct, separable concepts). For example, in a space with N_d = 4,096 dimensions (a typical value in modern Transformers), we can fit more than N_v = 10²⁰ vectors with no two vectors having a cosine similarity C greater than 0.15. This provides vast representational capacity.

Note that the assumed vector packing is far from dense: The math models a process of random placement with rejection, and number of placeable vectors grows exponentially with dimensionality. According to this model, for large values of Nd and small values of C:

There’s plenty of room in embedding space.

The cosine similarity between a given vector and a random vector measures a kind of background “similarity noise” if there is no learning to avoid accidental similarity. For large N_d, random cosine similarities are distributed normally with σ = N_d^–1/2. Thus, when N_d = 4096, σ ≈ 0.016, and quite small similarities (e.g., |C| > 0.05) are likely to represent learned relationships, not meaningless “noise”. As a boost to intuition, note that |C| < 0.05 corresponds to an angle < 3°.

14

Fine-tuning and lightweight adapters can build latent-space connections between separately trained models. Here are some examples from recent work; OpenAI’s o3 model did the heavy lifting with light review and correction:

Advanced AI will transform possibilities, and our future will depend on which possibilities become real. I am persuaded that our options are both narrower and broader than they may seem, and that if understood, they would tend to align a range of seemingly clashing interests.1 A critical challenge is the enormous gap between emerging capabilities and our collective understanding of their implications.

In a world informed, as it has been, by a stream of policy papers, conferences, news reports, and tweets, can we expect a breakthrough in understanding fast enough to cope with accelerating, transformative change? This, to put it mildly, seems unlikely. The weakness of our knowledge ecosystem leaves us blind to both catastrophic risks and transformative opportunities.2

If technological capabilities will advance at AI-driven speed, then our understanding must somehow keep pace. This article explores how AI-enabled knowledge integration can help.

Visitors from the recent past would likely see advanced LLM technologies as indistinguishable from magic. What if we could extend today’s information-magic with a few crucial improvements?

AI Knowledge-Magic™ for a Wonderfully Informed World™

Let's try to imagine a Wonderfully Informed World™ made possible by applying AI Knowledge-Magic™ to integrate information at a civilization scale. What might that be like?

AI provides Large Knowledge Models (LKMs)

In our wonderfully informed world, AI systems have gathered and digested all publicly available information and made it universally accessible. The state of knowledge, ignorance, and controversy has become transparent in areas like science, technology, economics, and the world situation as a whole.3

Knowledge in LKMs is grounded in external sources. Cumulative reasoning extends knowledge with confident inferences, and this reasoning can be examined and traced to its sources. Misconceptions and disinformation become easier to identify and reject.4

Better integration and synthesis of information supports general implementation capacity — the ability to design, develop, deploy, apply, and adapt complex systems at scale.5 Building workable plans requires deep knowledge of diverse domains, knowledge that LKMs can provide.

Knowledge models are tools for expanding human capabilities, not autonomous entities. Knowledge modeling illustrates how intelligence can serve as a malleable resource, a perspective critical for developing realistic AI safety strategies.6

Large Knowledge Models reveal technological possibilities

Technological possibilities — in a fundamental physical and computational sense — are facts about the potential of the world. They include how matter and energy can be transformed into useful products, and what those products could do. Impossibilities, too, are facts, because technological possibilities are subject to objective constraints.

Consider atomically precise mass fabrication (APMF): Few informed experts7 doubt its physical possibility, yet its prospects remain obscured by outdated controversies and fragmented information.8 Knowledge models could synthesize and clarify the relevant information, helping bridge the gap between what is credible (strongly automated production) and what is realistic (fully automated, physics-limited production).

Fragmented knowledge also obscures prospects for structured transparency frameworks as a basis for powerful policy options. Current discussions of privacy, security, and surveillance neglect novel options that could greatly reduce tradeoffs between security and privacy. Understanding these options, however, would require a synthesis of knowledge from multiple domains.9

Understanding possibilities requires modeling how capabilities can combine to form higher-level capabilities. Knowledge gaps can be extraordinarily costly, because ignorance of crucial possibilities undercuts awareness of all the possibilities they enable. And if those possibilities include solutions to problems of existential magnitude, then perhaps better knowledge integration should be a first-rank priority.10

Large Knowledge Models facilitate reasoning about policy options

Potential policies are constrained by facts, and many key facts will (in a wonderfully informed world) be clear. Some are facts about technological possibilities, some are facts about clashes and alignments of interests, some are facts about uncertainties about other facts.

Human and machine creativity can build knowledge structures around policy options that reflect technological possibilities and emergent win-win options. This understanding is crucial for preparing to pivot — developing, deepening, and explaining complex strategic options when mounting pressures demand change.

Technology-enabled possibilities include:

• Win-win material abundance, avoiding economic conflict

• Win-win defensive stability, avoiding military conflict

• Rapid yet incremental transitions from offensive to defensive force postures

As technological possibilities become better understood, more of what is realistic will become credible, allowing us to move beyond current deadlocks in strategic thinking.

Large Knowledge Models help save the world

Above I asked whether, in a world informed by policy papers, conferences, news reports, and tweets, we can expect a breakthrough in understanding, with the obvious negative response. Now let's ask, "In a world informed by Large Knowledge Models, could we reasonably hope for better outcomes?"

There are pros and cons, because the implications of better knowledge are unpredictable. On the whole, however, I'd bet on better knowledge leading to better outcomes. As perceived options expand and become better understood, opportunities for goal alignment emerge. Even actors with competing interests may find common ground in pursuing outcomes that benefit all parties.11 In a world of potentially unstable military competition, goal alignment may be crucial.

Reducing AI Knowledge-Magic™ to practice

Is it reasonable to expect that AI can be leveraged to build knowledge-model functionality of the kind suggested above? The answer, I think, is an unambiguous “Yes”.

LLMs demonstrate that AI systems can ingest vast amounts of knowledge, learn reasoning skills, and can use their knowledge and reasoning capacity to guide search and help interpret external information. Relying on external information — “retrieval augmented generation” (RAG) — reduces reliance on LLMs' faulty, opaque, and too-creative memories. OpenAI’s “Deep Research” illustrates this well, producing credible research reports based on iterative reasoning and internet search.

The knowledge model concept proposes to add cumulative synthesis and refinement of that information — upgrading quality by filtering out semantic noise and seeking coherence guided by provenance, consensus, and especially consilience.12 Knowledge can be extended with explicitly grounded reasoning, guided toward fruitful results by LLMs' ability to “synthesize across complex subjects”.13 From an LLM perspective, an LKM would enable high-quality, latent-space RAG; from an external perspective, the combination would act as AI system with scalable long-term memory.

Don't expect a single, dominant knowledge model, or excellent systems without precursors. As with LLMs, there will be competing versions and incremental upgrades, and as with LLMs augmented by RAG, there will be many information stores, some private and proprietary. This diversity is a feature, not a bug — it prevents single points of epistemic failure while still fostering convergence on shared understandings of reality.14

Some theoretical and practical considerations

All roads lead to latent space

Knowledge can best be represented and synthesized in a language-independent semantic space ("latent space"), not by accumulating strings of words in English, Chinese, or Esperanto. Language-oriented ML practitioners will recognize the enormous, qualitative advantages of working with sets of differentiable dense-vector representations ("embeddings" of meaning in a high-dimensional semantic space) rather than token sequences.15 Embeddings are strictly more expressive than text.

ML practitioners with ample time to follow the literature may be aware of recent advances in compressing text-space information into denser Transformer-compatible latent-space representations,16 and in using latent-space representations as a medium for reasoning. There is much more to be said about how the pieces can fit together at a technical level.

Current language models must reconstruct meaning from text each time they access it. Combining and re-expressing knowledge as text incurs not only computational overhead but also a kind of semantic quantization noise that degrades meaning.17 Persistent knowledge structures in latent space could enable cumulative refinement and combination of complex ideas, representing clashes, agreement, nuance, analogy, uncertainty, and explicit ambiguity in machine-native ways that natural language cannot.

Grounding, updating, and scaling

LLMs suffer from hallucinations, and their internal representations can’t be annotated with source citations. Large knowledge models would retain the benefits of latent-space representations (the substance of Transformer processing) but in the form of distinct data objects that can be annotated with sources.

This same structural advantage allows knowledge to be added, removed, and updates without interfering with existing knowledge or requiring retraining. Affordances for tracing knowledge sources and derivations can make reasoning more interpretable.

LLM training shows that large models can ingest information at a civilizational scale and at a large but affordable cost. Transformer-based models with broadly comparable costs can translate text into compressed, Transformer-native vector representations. Vector databases — implementing a kind of associative memory — can scale to billions of information units and beyond. Extensive reasoning to upgrade and synthesize knowledge is affordable, and can largely be driven by demand, like other forms of LLM inference.

Epistemics

The term "knowledge model", like most short sequences of words, is easily misunderstood. If "knowledge" is taken to mean "truth", then the conversation runs off the rails to rehash (here imagine the usual disputes regarding the meaning of truth, mistaken beliefs, legitimate controversies, social construction, and so on, ad nauseam).

"Knowledge models" (inevitably plural) could present specific views of what is true, but to build shared, trusted resources, it is natural to adopt a different epistemic stance, a perspective that is in a sense neutral to questions of truth, yet presents coherent, well-grounded perspectives to anyone seeking a reality-aligned understanding of the world. Presenting perspectives would typically be a task for LKM-informed upgrades to the (somewhat censored, somewhat biased) LLMs that we use today.

“Modeling knowledge” can mean modeling what people claim to be knowledge, taking explicit account of origins, evidence, controversies, provenance, social context, coherence, consilience, and consistency with other, well-supported facts. Compare the breadth and depth of the science of measuring the geoid to the information footprint of flat earth theories. Impartial representation bends toward truth.

Legal considerations

Regarding copyright and related legal concerns, knowledge models would gather and synthesize knowledge from multiple sources, with parallels to LLM pretraining, academic review articles, and search indexing. These transformative uses of information are generally acceptable under copyright law, and the representation of knowledge based on language-independent embeddings should be as well.

A vision and a goal

Advances in AI today center on tasks — increasing the ability to solve problems and provide services. Task-oriented AI amplifies capabilities that can transform the world, for both better and worse. Task AI, as presently conceived, does little to help us steer toward favorable outcomes.

Better knowledge can amplify task capabilities (accelerating the technologies of production, job displacement, medicine, and war), yet broader knowledge can also help us better understand the consequences of our actions. As AI capabilities expand, better knowledge integration becomes critical — not just for advancing technology, but for understanding and managing its potential consequences.

Knowledge-oriented AI can help us discover and explore options that are now unknown or known only in outline, and this kind of knowledge — by informing strategy — could help us steer away from catastrophe toward a future that is secure, open, and broadly appealing. The consequences of this kind of knowledge are uncertain, but the alternative — ignorance of our options — seems distinctly risky.

The development of knowledge-oriented AI calls for extending capabilities in new directions, toward broad, integrated knowledge and cumulative, collaborative reasoning. With a better knowledge of possibilities we may find our way to better outcomes.18

Whether or not actual knowledge models follow the approach I have outlined, I am confident that:

We can apply emerging AI capabilities to develop knowledge resources that are far more trustworthy, comprehensive, and accessible than any seen in human history.

• For researchers, this suggests developing architectures that integrate LLM capabilities with persistent, updatable latent-space knowledge stores.19

• For system-builders, it means constructing knowledge bases that preserve provenance, maintain traceability, and enable ongoing refinement.

• For policy analysts, it underscores the urgent need for better knowledge infrastructure to surface coherent, viable proposals when incremental approaches fail. Shaping the exploration of policy space in shared LKMs can become a new, more agile, more powerful form of research and communication.

Some readers will find this vision compelling and actionable.

Share

Yet advanced AI applications will expand, not just by overcoming obstacles, but by flowing around them, like rising water cutting new channels. This aspect of development — call it the “bypass principle” — highlights how conventional analysis may underestimate AI-driven change.

While shallow assessments focus on visible obstacles — the difficulties of matching human capabilities, of overcoming regulatory barriers, and of restructuring organizations — AI-enabled developments will often find paths that bypass rather than overcome apparent barriers. Existing obstacles are concrete and obvious in a way that alternatives are not. Skewed judgment follows.

The Spectrum of Bypass Mechanisms

The bypass principle operates through similar mechanisms at different scales, from tasks to processes to entire organizations.

Task Refactoring: Bypassing complex jobs

Rather than waiting for AI to match employees in all their roles, we see AI accelerating work through decomposition of seemingly monolithic jobs into components that can be optimized, automated, or restructured.

Consider the automation of programming: Instead of AI fully replacing developers in the intertwined tasks of functional specification, architecture design, team management, coding, testing, and debugging, we see increasing automation of subtasks within these roles, refactoring tasks to change workflows and responsibilities. Here, AI often integrates seamlessly into existing tool sets. Perhaps no previous job will be fully automated, yet automation in aggregate will bypass more and more human bottlenecks.

A production system that can be 90% automated could leverage human effort by an order of magnitude. Focusing on the remaining 10% would distract from profoundly disruptive prospects. Bypass can be powerful without being complete.

Process Replacement: Bypassing complex coordination

Beyond task-level refactoring is process replacement — reimagining how objectives are met. Software development again provides an example.

Today, defining functional requirements and evaluating implementations requires lengthy, iterative consultations between users and developers. AI-mediated knowledge integration will enable a better, faster process: Rather than relying heavily on sequential human-to-human communication, the work of evolving requirements could flow through parallel conversations in which AI systems ask questions, offer suggestions, integrate requirements, track cross-functional consistency constraints, and generate feasible options in real time. Human guidance — and implicit, collective human collaboration — would flow through channels that bypass today’s bottlenecks.

The implications of bypassing design delays extend far beyond productivity gains. Accelerated design and development with improved results and fast adaptation represent qualitative changes in end-to-end implementation capacity. This bypass pattern applies equally well to hardware development, whether of chips, weapons, spacecraft, or manufacturing systems — in other words, it can speed transformation of the material foundations of civilization.

Organizational Replacement: Bypassing complex structures

Potential process improvements will encounter organizational inertia and active resistance, but AI capabilities will enable alternatives that bypass inflexible organizations entirely. This bypass mechanism will be most effective where internal processes can be transformed while maintaining external interfaces.

Supply chains illustrate this principle: Manufacturing organizations, regardless of internal complexity, are valued for their inputs and outputs — they present a relatively simple interface to suppliers and customers. It is this relative simplicity that can make contractors in supply chains interchangeable. With enough AI, inflexible manufacturing organizations will be replaced by more agile competitors built on AI-intensive workflows.1

When supplier and customer interfaces themselves are too rigid, the next level of bypass is to automate both sides through vertical integration. Entire industries can be transformed through extensive bypass within and around producers — again, with no need to replace all organizations, all processes, or all parts of a human job.

Bypassing External Constraints

Other sources of friction and constraint are found beyond, around, or beneath productive processes. These too invite bypass.

Regulatory Friction

The bypass principle extends to regulatory constraints. Delays often stem from challenges in ensuring regulatory compliance. AI-enabled design-for-compliance can help create systems that satisfy regulatory requirements by construction rather than through post-hoc adjustment. And, of course, AI can help fill out government forms.

Innovations may fall outside the scope of regulations entirely — witness most AI development and deployment to date. For tasks with weak geographical constraints, systems with knowledge of worldwide law and regulations could identify opportunities for jurisdictional arbitrage — tasks flowing around regulatory obstacles by literally avoiding them. And when geographical constraints stem from scarcities of skilled labor, AI-bypass again comes into play.

Technical Capabilities

AI assistance can help bypass technical obstacles that block capabilities. For example, creating software that strictly adheres to requirements — including security — is often infeasible, not because it is impossible, but because of the cost and complexity of applying mathematically grounded formal methods. Rising AI capabilities in coding and formal mathematics2 will converge, bypassing this software bottleneck with profoundly important consequences for software reliability and trust, and for human trust that can be built on reliable, trustworthy software.

In a potentially pivotal example, the difficulty of designing protein molecules has long bottlenecked progress in producing functional, intricate, atomically precise devices. AI methods have recently bypassed previous methods.

Research Bottlenecks

In research, AI is helping to overcome bottlenecks in imagination, analysis, and experiment design. In a recent study from Los Alamos,3 researchers were asked to tackle a diverse tasks in science and engineering (including “design for fusion energy, control theory, materials discovery, Riemannian geometries, and several others”) with assistance from an AI model, GPT-o1. After several days of work with the model, participants reported “good to excellent” increases in research productivity and rated continued access to the model as “important or essential” to their group’s future work.

Research and development is itself a cross-domain bottleneck of enormous importance. Loosen constraints here, and all technology timelines shorten. Applications of AI to AI research itself are critical, and have begun to drive a long-anticipated loop of AI-driven AI progress.4

Bypassing Bypass-Discovery: Accelerating bypass itself

The difficulty of identifying and analyzing barriers and potential bypasses will itself create friction that delays AI-driven change. Through knowledge and creativity, however, AI systems can help bypass this human bottleneck as well. Even today, working with AI to generate and analyze options can be far more productive than working on problems unaided.5

Looking Forward

The bypass principle suggests that AI-driven change will unfold more rapidly than many expect, and broadly enough to bring deep, global transformation wherever capabilities matter. Think of AI capabilities as a rising tide, an increasing pressure on the status quo across every field.

Many apparent obstacles will prove to be illusions. As the flood of AI carries the world into the rapids of history, we must choose goals more wisely. Obstacles to achievement have also blocked folly.

Subscribe now

• Alan Turing said: “it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers…we should have to expect the machines to take control”2

• Marvin Minsky said, “Once the computers got control, we might never get it back. We would survive at their sufferance.”3

• Geoffrey Hinton warns that “if a digital super-intelligence ever wanted to take control it is unlikely that we could stop it.”4

• Yoshua Bengio asks, “what if we build a superintelligent AI, and what if it has goals that are dangerous to humanity?”5

A common view holds that superintelligent AI entities would have vast, unique value and must be aligned with human purposes lest they be a threat to our existence. But solving the technical problem of aligning superintelligent AI entities is neither necessary nor sufficient:

• Guaranteeing that a superintelligent AI entity6 will be well-behaved isn’t necessary because there’s no need to package intelligence in an entity: Agency architectures are better.

• Being able to guarantee good behavior wouldn’t be sufficient for safety because humans who deploy AI might be incompetent, irresponsible, or malicious and build something nasty. Stable AI safety requires stable defense.

What do I mean by “agency architectures”? Consider how high-level AI capabilities could be organized to undertake large, consequential tasks:7

• Proposing plans is a task for generative AI models that compete to develop brilliant proposals and critiques.

• Choosing plans is a task for humans advised by AI systems that compete to identify problems and discuss options.

• Executing plans is a set of incremental tasks for specialized AI agents that compete to provide reliable results on budget and within schedule.

• Overseeing plans (e.g., budgets, milestones, regulatory compliance), is a task for specialized AI systems that monitor, assess, and report to humans. Plans can be updated in response.8

This approach — the “AI Agency Architecture”9 — scales to superintelligent level capabilities, yet neither the parts nor the whole amount to “a superintelligent AI”. Agency architectures follow patterns seen in today’s superhuman entities — human organizations — and incremental upgrades of organizations with AI tend to move in this direction.

But what if “the AIs” collude against us?

Long-standing tradition mistakenly frames AI capabilities as properties of AI agents,10 and then imagines that the future situation will be “us vs. them” (the robot uprising, etc.). But this scenario typically assumes that weak humans will confront a cohesive, supercapable “them” — that all the smart AI systems will collude against us.

What conditions would make collusion a losing strategy for even the most self-interested superintelligent AI?11 Let’s take a perspective inspired by elementary game theory:

The following is adapted from an FHI Technical Report, Reframing Superintelligence: Comprehensive AI Services as General Intelligence, Section 20.

Note that “oracle”, a term introduced by FHI’s founder, Nick Bostrom, means “non-agentic question-answering AI”. Oracles in this sense would be integral to AI agencies as non-agentic systems designed to provide trustworthy answers to critical questions.

Collusion among superintelligent oracles
can readily be avoided

Because collusion among AI question-answering systems can readily be avoided, there is no obstacle to applying superintelligent-level AI resources to problems that include AI safety.

20.1 Summary

The difficulty of establishing successful collusion among actors tends to increase as their capabilities, knowledge, situations, and roles become more diverse and adversarial (think auditors, competitors, specialists, red-teams…), and increasing the number of actors can make collusive cooperation more difficult . In the context of AI systems (even more so than among human beings), these conditions can be readily implemented and are attractive for pragmatic reasons. Arguments that, absent preexisting alignment, high-level AI systems will inevitably collude are ill-founded. Instead, we should expect that interactions among multiple superintelligent-level systems can be applied to suppress deception and reduce risk by leveraging imperfect alignment achieved at the level of individual systems.

20.2 Trustworthiness can be an emergent property

Prospects for solving AI-safety problems would be greatly improved if we could safely apply superintelligent-level question-answering resources (“oracles”, or more generally, “systems that provide information in response to prompts”) to solving those problems. A familiar objection dismisses this potentially powerful approach as unsafe in itself, arguing that, absent solutions to difficult problems, individual superintelligent-level systems would be untrustworthy, and that attempts to establish checks and balances among multiple systems (for example, through superintelligent-level evaluation of potentially deceptive answers to questions) would inevitably be thwarted by collusive cooperation.

Identifying robust strategies for ensuring non-collusion among superintelligent question-answering systems would overcome this objection, inviting exploration of superintelligence-enabled strategies for managing potentially untrusted superintelligent AI systems. The present discussion argues that the robust non-emergence of deceptive collusion among imperfectly aligned systems can be ensured by structuring systems of systems with diverse capabilities, knowledge, situations, objectives, and roles.

This problem framing assumes the ability to develop systems that respond to questions with superintelligent-level competence, and will (as a consequence of the nature of digital systems) assume the ability to constrain information inputs to AI systems during their development and use.

In the problematic case, superintelligent question-answering systems might provide deceptive answers in pursuit of emergent, potentially undesirable goals. The aim here is to identify principles for architecting multicomponent systems that will act as non-deceptive oracles, while treating their components as actors that could in principle engage in deceptive collusion.

20.3 A range of conditions can make collusion robust or fragile

Potential problems of deceptive collusion can be addressed by architecting systems that make collusion difficult or effectively impossible. Some relevant conditions:

Conditions that tend to facilitate collusion:

A1) Small numbers of actors (the required scope of collusion is small)

A2) Insensitivity to defectors (partial collusion is sufficient)

A3) Similarity of actors (for example, shared histories and objectives)

A4) Free communication (information can flow between any pair of actors)

A5) Iterated decision processes with observable actions

A6) Common knowledge of the above circumstances

Contrasting conditions that tend to disrupt collusion:

B1) Large numbers of actors (the required scope of collusion is large)

B2) Sensitivity to defectors (collusion fails unless it is complete)

B3) Diversity of actors (for example, different histories, adversarial objectives)

B4) Constrained communication (information flows through limited causal channels)

B5) Single-move decision processes, limited observability

B6) Lack of shared knowledge of strategically-relevant circumstances

Implementation choices that would yield conditions (A1–A5) are unnecessary and often undesirable. By contrast, implementation choices that yield conditions (B1–B5) are practical and often well-motivated:

Characteristics of practical architectures:

C1) To improve the quality and reliability of answers, it is natural to implement multiple, diverse (and implicitly competing) systems to propose alternatives.

C2) To identify low-quality or misleading answers, it is natural to employ diverse critics (auditors, red teams, etc.), any one of which might disrupt deceptive collusion.

C3) Systems of diverse, competing proposers and critics naturally implement different and often adversarial objectives.

C4) To minimize unwanted dependencies, systems are designed such that components have a sparse, thoughtfully designed communication graph.

C5) To provide stable results, it is natural to apply fixed models to a series of problems, yielding what is in effect a series of history-blind, single-move decisions.

C6) In training, it is natural to provide differentiated, task-relevant information to systems solving different problems, often omitting knowledge of general circumstances.

Considerations (C1–C6) [which parallel B1–B6, not A1–A6] suggest that implementing systems that will be resistant to the emergence of deceptive collusion aligns with the general strategy of employing ensembles of diverse, stable, task-directed system components to propose and assess answers to questions [in agency architectures, answers may include plans]. This approach can pass the test of practicality and potentially widespread adoption.

20.4 Untrusted superintelligence can be applied to AI safety

It seems both possible and practical to establish conditions that would effectively preclude deceptive collusion among diverse, task-oriented, superintelligent-level question-answering systems, thereby addressing a key problem in safely applying superintelligent capabilities to problems (Bostrom, Superintelligence, p.145).

Prospects for obtaining trustworthy answers from sets of individually untrusted superintelligent systems call for deeper exploration of AI safety strategies premised on reliable access to superintelligent problem-solving capabilities. Capabilities of particular interest include the ability to run and evaluate the results of superintelligent-level red-team/blue-team competitions relevant to questions of global strategic stability.

TL;DR: Because collusion can be thwarted, it is reasonable to expect that applying multiple potentially untrustworthy superintelligent-level systems to problems can improve rather than degrade safety.12 This prospect calls for greater attention and exploration.

Some comments on AI safety, 2025:

• AI safety research has discovered a host of obvious and non-obvious ways in which advanced AI could pose catastrophic risks. Nothing said here argues against these results.

• Many AI safety researchers and policy advocates have assumed that, without breakthroughs in alignment, advanced AI capabilities cannot be employed without catastrophic risks. This is mistaken.

• Arguments for restrictions on AI development should not be premised on the assumed existential necessity of breakthroughs in AI alignment. False premises have unintended consequences.13

• Further progress in AI alignment is extraordinarily important in part because better alignment makes AI more useful, and useful AI will be critical to safety.

• Many AI risks are not safety risks and have no technical solution.

Further Reading

[from the original article in Reframing Superintelligence]

• Section 8: Strong optimization can strongly constrain AI capabilities, behavior, and effects

• Section 12: AGI agents offer no compelling value

• Section 19: The orthogonality thesis undercuts the generality of instrumental convergence

• Section 21: Broad world knowledge can support safe task performance

• Section 23: AI development systems can support effective human guidance

• Section 24: Human oversight need not impede fast, recursive AI technology improvement

The emergence of AI-enabled hypercapability will revolutionize how we design and deploy complex systems.1 This transformation brings unprecedented military risks, yet these risks can be averted through coordinated, verifiable deployment of defensive systems.2

We face a choice between two paths:

• Continuing to pursue win-lose strategies that bring irreducible existential risks with little potential reward.

• Continuing these policies in the near term,3 yet pivoting to lower-risk, win-win outcomes late in the game.

Clear contingency plans could enable a pivot toward safety when mounting risks force change. Those contingency plans are crucial, but are not yet in place.

From Capability to Hypercapability

Economic and military transformations typically unfold gradually, but AI development will be different: Cascading advances will drive fundamental shifts on much shorter timescales, radically expanding our capabilities.4

Prospects for deep transformation hinge on AI-driven expansion of general implementation capacity — the end-to-end ability to design, develop, deploy, and adapt complex systems rapidly and at scale.5 This expansion spans both physical and computational domains, including systems that range from defense and verification frameworks through with manufacturing, robotics, renewable energy, AI applications, and almost everything else. These advances point toward hypercapability: the ability to swiftly develop and scale solutions to an extraordinary range of complex problems, given coherent objectives.6

Strategic uncertainty is one of those problems.

Reliable, Durable Uncertainty

Deep uncertainties in AI development make winner-take-all strategies exceptionally risky. No competitor in an AI arms race can confidently predict:

• The scope and pace of AI advances (not just LLMs7).

• The novel military capabilities that their rivals might develop.

• The outcome of kinetic conflict with adaptive AI on both sides.

This multi-level uncertainty appears robust and durable. While we can’t rely on winning,8 we can rely on uncertainty itself as a growing pressure for seeking alternatives.

The existential risks of win-lose strategies strongly motivate exploring cooperative, win-win approaches — not through advocating a strategic pivot today, but by developing contingency plans for a time when mounting pressures crack the previous consensus.

And the pressures of hypercapability will invite solutions based on hypercapability.

Transformative Options

In a hypercapable world, long-standing constraints dissolve. AI will enable rapid design and deployment of both defensive systems and verification frameworks, potentially defusing familiar security dilemmas. Key technological possibilities include:

• Defensive stability through coordinated deployment of defensive systems at scale.9

• Verification frameworks built on structured transparency relationships.10

• Digital infrastructure security through verified supply chains and software.11

These point toward fundamentally different security architectures — arrangements that would otherwise be unthinkable. The benefits of success extend far beyond security: A hypercapable world promises globally shared gains in areas that range from economic abundance to climate remediation.12 As AI opens the door to robust defense and enormous economic growth, national interests will shift: Securing a share of massive benefits becomes more attractive than risking everything for dominance. Neither wealth nor security is a zero-sum game.

Accelerating Strategic Adaptation

AI can assist not just in building systems, but in planning and coordination:

• Strategic analysis: AI can explore vast option spaces, helping assess complex interactions and potential force postures.13

• Deliberation: AI systems can help diverse actors — from interest groups to nations — recognize shared interests and opportunities.14

• Negotiation: AI can help establish shared understanding of win-win options, including details too intricate for unaided negotiation.

• Confidence-building: AI can identify risk-reducing paths and help develop both defensive frameworks and verification systems.15

Preparing for Transformation

Today’s preparations shape tomorrow’s choices. By developing clear concepts for desirable outcomes, we can expand options available when strategic pressure peaks. This preparation need not trigger opposition — contingency planning for transformative AI is already on the table. Three parallel tracks deserve attention:

• Technical & Institutional Analysis: Exploring how AI could enable swift deployment of defensive security and verification systems.

• Political & Cultural Analysis: Understanding how perceived national interests could shift in response to new risks and opportunities.

• Multilateral Bridge-Building: Promoting understanding of win-win possibilities among influential thinkers and analysts shaping strategic perspectives in rival governments.

The technical feasibility of a rapid pivot — late in the game, but before a crisis — makes early preparation valuable without requiring immediate policy changes. When capabilities will enable swift implementation, what matters most is having developed the right concepts and actionable plans in advance.

A Sketch of a Scenario

The discussion above outlines possibilities and incentives in abstract terms, but gives no clear picture of how change could occur. How could seemingly implausible changes unfold through real institutional and political processes?

1. PREPARING THE GROUND

Studies exploring AI-enabled defensive architectures stimulate innovative strategic thinking. As implications come into focus, analysis deepens, drawing lessons from US-Soviet arms control and current military-to-military relations. Track-two dialogues emerge organically as anticipated developments point to unexpected common ground among competitors. AI systems accelerate the development of concrete technical and strategic proposals aligned with diverse interests and institutional cultures.

Recognition grows that AI-driven change could swiftly transform military and economic realities. As detailed contingency plans take shape, this line of thinking penetrates institutions and shapes how emerging developments are understood.

2. TIPPING POINT

Mounting uncertainty forces strategic reassessment, while earlier track-two discussions frame multilateral negotiations. Security establishments embrace roles in developing new defensive systems. Contingency plans evolve into a new strategic consensus. AI systems accelerate analysis of detailed implementation paths, enabling swift action.

3. STRATEGIC TRANSFORMATION

Urgency overcomes institutional friction and bureaucratic routines. Verification frameworks enable confidence-building steps, while expanding implementation capacity enables rapid, coordinated shifts in force postures. As verified defensive systems and durable constraints alter the military balance, shared incentives facilitate further cooperation. The prize is security with prosperity.

4. A NEW EQUILIBRIUM

The transformed strategic landscape combines robust defensive architectures with ongoing technological adaptation. Great powers maintain autonomy while managing the challenges of domestic AI transitions. The equilibrium proves stable, based on solid foundations, and unresolved conflicts no longer pose existential risks.

Subscribe now

Strategies based on incoherent scenarios — piecemeal changes in hypercapable future — would incur unfathomable risks and opportunity costs. If conventional strategies would lead to large, irreducible risks, we must explore alternatives. If AI-enabled capabilities could deliver economic abundance while escaping security dilemmas, we should seek practical paths forward.

This article aims to carve out intellectual space for distinct, coherent scenarios: distinguishing between gradual, piecemeal change and change that eventually becomes swift and pervasive. Let’s call these scenarios “incremental” and “radical”. Incremental scenarios merit attention: Change today is incremental and could plausibly continue that way. But radical scenarios also merit attention: It is plausible that AI development will feed back to accelerate AI itself toward deeply transformative capabilities. Radical futures are a realistic, challenging contingency that calls for preparation.

What I will suggest is conditioned on the eventual emergence of a hypercapable world. Let’s assume that this contingency won’t be taken seriously until late in the game, and that preparations will look like contingency planning, with costly action following — not preceding — perceived urgency.

Prospects for a hypercapable world

Previous posts have outlined a range of crucial AI capabilities and implications, and these can be condensed into three key prospects and a key consideration (credible realism) in discussing them. With links to the anchor posts:

1. Highly capable, steerable AI: Continued AI development on a broad front (more than LLMs) will lead to strong, steerable, high-level AI capabilities with comprehensive applications.1

2. Greater software implementation capacity: AI will break software development bottlenecks, enabling rapid production of verifiably correct software systems.

3. Greater physical implementation capacity: AI will accelerate the end to end design, development, production, deployment, and adaptation of large-scale sociotechnical systems.

4. The importance of credible realism: Policy can be oriented toward realistic problems based on credible prospects even before realistic prospects have entered the Overton window. Credible realism can help align policy with challenges posed by implausible realities.

Predictable consequences of transformative AI

AI scenario planning must begin with technology drivers. The most predictable consequences of AI are its fundamental capabilities, not how, when, by whom, or for what purposes they’ll be applied. Let's consider two levels: basic enablements and their radical implications:

• Abundant material wealth and services: Expanded production capacity can enable material abundance, while AI capabilities can translate material abundance into abundant physical and information services.

• Abundant renewable energy: Expanded production capacity can accelerate the scaling of wind, solar, and energy storage systems.

• Ample material resources: Abundant energy and capital goods can enable the use of abundant, low-concentration ores along with reduction of environmental harms.

• Provably-correct software at scale: AI can enable swift production of software with mathematical proofs of security and correctness, enabling deployment of trustworthy software at scale.

• Greater scope for verifiable international agreements: Capabilities leveraging new hardware and software can extend structured transparency for robust monitoring of military capacities and deployments.

• Swift transitions in military force postures: Expanded implementation capacity (both design and production) can enable rapid shifts in the architecture of global security.

These capabilities, taken together, sketch a hypercapable world that challenges fundamental assumptions about the future. Prospects for a hypercapable world call for reconsidering options, interests, strategies, and policies. In my view, the task for today is exploration and analysis, seeking good contingency plans. The task for tomorrow will be to recognize those contingencies and act. Today sets the stage for tomorrow.

Incremental vs. radical expectations

The gap between incremental and radical expectations for AI capabilities leads to radically divergent prospects for policy concerns:

AI capabilities

• Incremental: Prepare for piecemeal advances creating a range of problems and opportunities.

• Radical: Prepare for steeply accelerating capabilities with pervasive, interlocking consequences.

Domestic economies

• Incremental: Prepare for shifts in employment across multiple sectors.

• Radical: Prepare for a world where human labor is optional.

Climate crisis

• Incremental: Seek to mitigate AI’s growing energy consumption and CO₂ emissions.

• Radical: Prepare to leverage new productive capacity to make renewables dominant and fossil fuels obsolete.

Resource competition

• Incremental: Secure long-term control over strategic minerals and fossil fuels.

• Radical: Anticipate a world where minerals and fossil fuels lose strategic value.

Geopolitics

• Incremental: Work within a status quo of slow change, opacity, and conflict.

• Radical: Explore options for changing the game through improved goal alignment, structured transparency, and verifiable cooperative actions.2

National security strategy

• Incremental: Race to develop new offensive and retaliatory capabilities, plan for preemption or warfighting in a world with growing technological uncertainty.

• Radical: Explore options for risk reduction and strategic stability through coordinated deployment of overwhelmingly effective, verifiably defensive systems (while assuming that this won’t happen soon).

Incoherent scenarios are the enemy

Strategic thinking about AI’s potential to reshape our world calls for exploring distinct internally coherent technology scenarios, both incremental and radical, and with this, to reconsider objectives in the context of a hypercapable world.3

This approach doesn’t demand priority for one scenario over the other: It allows for disagreement about AI’s trajectory (incremental or radical change) while demanding that transformative AI be considered as a whole, not piecemeal.4 Planning premised on incremental change is inevitable and perhaps wise, yet it would be irresponsible to neglect contingency planning premised on prospects for radical (yet widely-anticipated) AI advances. Speculative probabilities (10%? 90%?) are beside the point.

Incremental scenarios present their challenges within a familiar frame of economic competition, military doctrine, and international relations; means and ends remain much the same. Radical scenarios present deeper challenges across the board, but within a context of unprecedented capabilities that could be leveraged to meet unprecedented challenges in unprecedented ways. Both potential means and rational ends are different in a hypercapable world.

Radical scenarios can be explored as contingency plans. This approach doesn’t call for immediate policy changes, expenditures, or public advocacy of particular expectations. Instead, it calls for thought, collaboration, and increasingly concrete contingency planning. Then, when radical developments force change, coherent strategies can be ready for action.

And at the threshold of a hypercapable world, delayed action could be implemented with unprecedented speed.5

What to do?

Strategic thinkers and policy analysts will play a crucial role in shaping discourse around AI’s potential. By exploring potential capabilities and options, and by developing and promoting coherent scenarios, thought leaders can challenge muddled thinking and expand the range of serious discourse.

There is no need to advocate immediate policy changes, or to boost expectations for transformative AI. With recent shifts in opinion, radical contingency planning will encounter little opposition and create more reputational opportunity than risk. Future events can do the work of persuasion, provided that the ground is prepared and seeded.

There is also no need to be an insider: A broader public can contribute to the evolution of ideas and help to shape the climate of opinion. We need to shift the Overton window toward coherent prospects and policies that are better aligned with reality and our shared interests. This is a whole-society task that rewards even partial success.

Share AI Prospects: Toward Global Goal Convergence

From tribes to states, competition for limited resources has driven conflict, and while resource competition is only one cause of conflict, it remains a powerful force.1 Within a fixed-resource framework, interests appear to be directly opposed, because dividing fixed resources is a zero-sum game. If we take this zero-sum game as a model of the future, incentives for cooperation seem sharply limited.

The concept of “Paretotopian goal alignment” shows how prospects for AI-enabled capabilities could reshape incentives — provided that these prospects and their implications are understood in advance. This article illustrates how prospects for greatly expanded resources could blunt incentives for conflict and help align the goals of competing actors towards mutually beneficial outcomes.

The Zero-Sum Game of Fixed Resources

The division of fixed resources is often viewed as a zero-sum game: For one party to gain resources, another must lose. Within the scope of a resource-centered framework,2 a zero-sum constraint creates directly opposed interests among competing actors, whether individuals, corporations, or nation-states:

In this model, the total resource quantity (Q) is fixed (Q_A + Q_B = Q = 1). Thus, any gain for Actor A necessarily means an equal loss for Actor B. However, when dealing with large quantities of resources, it is often more appropriate to consider utility on a logarithmic scale due to the principle of diminishing marginal returns.

For example, under logarithmic utility, an increase in total monetary holdings3 from $100,000 to $200,000 is considered equivalent in value to an increase from $100,000,000 to $200,000,000. This contrasts with a linear model, which would treat the greater as vastly more significant.

Considering utility rather than resources (and situations other than small steps around equal division), losses for one actor may exceed gains for the other, yet interests remain directly opposed:

The Impact of Resource Expansion

Now, let’s consider what happens when the total amount of available resources increases. Even a moderate increase, say 50%, begins to change the nature of the game:

This expansion opens a range of Pareto-preferred moves4 — outcomes in which both parties can gain simultaneously. In our resource expansion scenario, there’s now room for both Actor A and Actor B to increase their holdings.

However, at this level of expansion (moderate, yet large by the standards of short-term economic growth), there remain strong incentives for each party to seek a disproportionate share of the resources. One actor might secure large gains at the expense of the other by capturing all the gains, or by taking 90% of the total resources. Gains and losses from such “greedy” strategies would motivate conflict, and the arrows in the diagram above are far from aligned.

When we consider utility rather than raw quantity, the incentives for greedy behavior are still substantial. With logarithmic utility, the difference between taking 50% and 90% of the resources is somewhat less dramatic than in a linear model, yet remains a strong incentive for aggressive greed.

Resource Expansion and Changing Incentives

The game changes dramatically with the prospect of massive resource expansion, for example, an increase by a factor of 1000. This scenario fundamentally alters the incentive landscape:

As this diagram suggests, the prospect of 1000-fold expansion reduces incentives for greedy behavior because the difference in perceived value between gaining 50% or 90% of the total resources — as seen from the initial position5 — becomes relatively small. Securing a substantial share of prospective gains becomes more important than the incremental value of securing a larger share of the total.

This shift in incentives creates a larger space for mutually beneficial outcomes, reducing concerns over competing notions of fairness.

This rapid-expansion scenario differs from historical patterns of economic growth. While global economic growth has typically hovered around 3.5% per annum, the prospect of multiple-order-of-magnitude increases within a future decision-making time horizon is novel.

Conflict Risk and Goal Alignment

Risks of conflict further reinforce the alignment of goals: Actions aimed at securing large (yet low-value) unilateral gains risk opposition, conflict, and losses. Conversely, actions directed toward outcomes perceived as reasonably fair are more likely to gain support and succeed.

When we factor in the potential costs of conflict, the expected value of pursuing large, unilateral gains decreases significantly. The potential losses from conflict (for example, global destruction) could easily outweigh the marginal gains from securing a grossly disproportionate share of the new resources.

Additional Factors Affecting Goal Alignment

Several other factors contribute to goal alignment in scenarios of great resource expansion:

1. Satiation of resource demand: Some desires are inherently satiable, leading to a more rapid decline in the marginal utility of additional gains.

2. Non-rivalrous goods: The increasing importance of non-rivalrous information goods and services further reduces incentives for conflict over resources.

3. Positional goods: While value placed on relative status can limit goal alignment, a vast increase in absolute resources can mitigate this effect to some degree; for example, continued disproportionate wealth ratios become compatible with universally high standards of living. However, the value placed on power over others remains problematic and relates to broader security concerns among unequal actors.

4. Security concerns: While a deep exploration of fundamental security issues is beyond the scope of this article, the conclusion is important: Defensive stability can be established despite asymmetric resource distribution. In situations within the realm of economic and technological possibility, resources held by one actor need not threaten the security of another. This topic warrants further investigation as it’s central to the potential realization of Paretotopian outcomes.

The Concept of Paretotopian Outcomes

The potential mitigation of zero-sum incentives in resource competition supports the broader concept of “Paretotopian outcomes” — accessible futures that would be strongly preferred by a wide range of actors today. These are outcomes where (almost) everyone is better off, and (almost) no one is worse off, compared to the current state.6

Understanding prospects for Paretotopian outcomes could influence policy decisions and strategic planning. Changes in perceived options can change perceived interests, potentially motivating a deep reconsideration of policies.

Conclusion

The concept of Paretotopian goal alignment offers a framework for understanding how greatly expanded resources could change incentive structures, opening new possibilities for cooperation and mutual benefit, even among traditionally competing actors.

This perspective invites a reconsideration of approaches to resource allocation, conflict resolution, and long-term planning, with potential implications for global policy and governance. It suggests that strategies aimed at dramatically expanding collective resources and capabilities could mitigate many zero-sum conflicts and shift the focus towards cooperative strategies for risk management.

Subscribe now

Share AI Prospects: Toward Global Goal Convergence

The trend towards increased monitoring seems unstoppable. Internet devices are creating an ever-denser web of data collection systems, while fast moving technologies are creating new threats. The specters of rogue AI and bioterrorism are driving calls for national and global monitoring systems.

Dystopian scenarios are easy to imagine, a world where secretive agencies exceed their legitimate authority, privacy evaporates, and centralized power can easily monitor and suppress any challenge. A common view suggests that we have only bad options: accept invasive surveillance for security, or protect privacy and accept vulnerability. This view, however, makes strong but tacit assumptions about the structure of transparency — about how information from monitoring systems is collected, processed, filtered, and applied, and about how these systems are governed.

The concept of structured transparency1 invites us to view diverse controls on information flow, not in isolation, but as building blocks for systems tailored to serve legitimate security needs without granting abusable powers. In the context of civil society, this includes detecting threats while minimizing intrusion on the private sphere; in the international sphere, this includes enabling states to reveal specific kinds of information while protecting secrets.

What if we could design systems that detect genuine threats without invading personal privacy? What if oversight mechanisms could ensure that monitoring systems adhere to agreed patterns of information governance? What if states could negotiate new kinds of transparency-based assurances that help reduce military tension?

To understand possibilities, we need to consider how emerging technologies can be composed in new ways, and how the coming explosion of computational and physical capabilities can open further possibilities. With this understanding, we can reshape the conversation around surveillance and security. We can demand solutions that serve legitimate needs without sacrificing our fundamental values.

Building blocks for transparency structures

Structured transparency can employ a toolkit of complementary mechanisms that, in combination, can change the landscape of possibilities. These building blocks form the foundation of transparency structures, enabling refined control over potentially problematic information flows. Let’s consider key governance tools and their applications.

• Redaction and anonymization: These mechanisms strip identifying details from data while maintaining its analytical utility. They’re crucial in public health, allowing data aggregation without compromising individual privacy. Advanced techniques could provide evidence of specific actions without revealing identities, reserving de-anonymization for situations with appropriate permissions.

• Rate control: A transparency structure that offers only a limited “query budget” constrains the scale of potential abuse and encourages a focus on mission-critical information gathering. Rate controls can block mass surveillance.

• Query filtering: Systems that restrict the scope of queries can enable specific inquiries while blocking privacy-infringing questions. In some cases (for example, in formally initiated criminal investigations) a system might allow more invasive queries under strict oversight.

• Time windows: By implementing time windows for access, transparency structures can provide data for addressing active threats while sealing historical records, or can delay access to tactically relevant information while enabling later scrutiny as part of an oversight process.

• Pattern discovery: AI algorithms operating on rich information in a secure environment can investigate specific patterns of concern and then report the results through filtered channels without exposing all the clues that led to their discovery. Thus, advanced AI models with access to raw data repositories could apply their full capabilities to detect threats, while reporting only relevant, potentially anonymized evidence for further evaluation.

• Revocable permissions: This governance mechanism allows overseers to respond to abuse by restricting transparency, or to unlock capabilities for deeper investigation when threats are detected. Multi-key permissions, implemented through encryption and key-sharing techniques can ensure that approval for sensitive actions requires agreement by multiple parties.

Transparency structures in practice

Flow-control mechanisms are already used to craft transparency structures for a range of purposes:

• Financial monitoring: Banks use pattern discovery on anonymized transaction data to detect and report fraud and money laundering.

• Health information exchanges: Allow data sharing among providers, using redaction to protect individual privacy while enabling population-level pattern detection and analysis.

• Census data: Restrictions on the types and numbers of queries together with anonymization enable the use of data while mathematically guaranteeing individual privacy.

• Internet service provider data retention: May restrict queries to metadata, and employ time windows and rate control for access; governance mechanisms control investigation permissions.

• GDPR-compliant analytics: Implements flow controls through secure repositories, revocable permissions, redaction and anonymization, query restrictions, rate controls, and time windows to balance business analytics with privacy rights.

Note that the governmental entities that employ these transparency structures are themselves subject to structures of transparency and governance.

Putting pieces together

By integrating multiple mechanisms, we can create flow-control architectures tailored to specific needs and constraints. Consider a potential transparency structure designed to enable detection and investigation of potential domestic security threats (perhaps plans for hijackings, bombs, or bioterrorism) while reliably precluding mass surveillance:

• AI systems operating inside an information-security boundary have access to rich information sources.

• AI-based pattern discovery can follow any clues, yet can report only specific threat-identifiers.

• Flow controls restrict human investigators to permissible, case-focused queries.

• Permissible queries are limited in number and scope, ensuring focused investigation rather than mass data collection.

• Substantial evidence of a serious threat can unlock access to broader information, a process similar to issuing a subpoena.

• Focused information is delivered to decision-makers for potential action.

This sketch illustrates how advanced information flow controls could be combined to build transparency structures that satisfy both security and privacy concerns in civil society.

Well-designed transparency structures can also create new options in international relations. Even hostile states frequently benefit from mutual revelation of selected information, provided that other information remains secret. For example, the US and Soviet Union agreed to:

• Allow each other to fly surveillance aircraft over their military facilities, subject to constraints on rates, time windows, and (through restrictions on cameras and sensors) query types.

• Allow on-site inspections of missile sites and production facilities, subject to controls on the frequency and nature of access (in other words, restrictions on query types and rates).

• Expose the contents of their missile silos to satellite observation, revealing sensitive but limited information that enabled confirmation of compliance to arms-control agreements.

A wider, more flexible range of options for structuring transparency can widen the range of negotiable risk-reducing agreements. In conjunction with other verifiable technologies, structured transparency could contribute to a shift toward a more defensive defense,2 reducing reliance on threats and potentially providing an exit from the classic security dilemma.

In considering the scope of potential agreements, it is important to consider how transparency mechanisms can help bootstrap trust in the transparency systems themselves: The greater the transparency of systems design, implementation, and operation, the greater the potential for assurances among mutually distrustful actors.

Ambitious Goals and Implementation Challenges

The mechanisms of structured transparency open doors to transformative applications, but realizing their potential will require confronting challenges that increase with scale and scope:

• Complexity: Extensive systems will require more complex software infrastructure, posing implementation challenges.

• Reliability: Systems must operate consistently, and their potential failure modes and attack surfaces increase with scale.

• Trust: The design, implementation, and operation of critical systems must satisfy demands for trust.

Understanding these challenges and potential solutions is crucial. It allows us to set realistic expectations and distinguish between near-term achievable goals and long-term aspirations.

Structured transparency is both a concept and a direction for policy and technical research. Information flow controls are already ubiquitous, but the concept of structured transparency invites us to view their mechanisms as building blocks, to seek better mechanisms, and to think more creatively about the structures we can build from them.

And looking forward, the concept of structured transparency invites us to consider how AI-enabled capabilities could enable us to craft better building blocks, build more ambitious architectures, and create new possibilities for overcoming the critical challenges ahead. The future of civil society and the human world may depend on it.

Related posts:

• Breaking software bottlenecks

• The Platform: General Implementation Capacity

• Toward credible realism

Subscribe now

Share AI Prospects: Toward Global Goal Convergence

Exploiting expanded physical capabilities — general implementation capacity — will require a corresponding expansion in our ability to create software. While it's tempting to assume that advanced AI will simply develop whatever we need, this is simplistic. How can we specify what the software should do? How can we ensure that AI-developed software performs as intended, without vulnerabilities, and with sufficient reliability for critical applications, even in the face of sophisticated cyberattacks? Answering these questions requires a closer examination of how AI capabilities align with key challenges in software development.

Today, software development is often a bottleneck: It's a slow, expensive process that frequently yields unreliable or insecure products that don’t align with actual needs. Post-delivery, the difficulty of modifying and debugging software hinders adaptation to new requirements. Achieving security against human hackers — let alone AI — seems improbable. Redesigning flawed infrastructure appears out of reach.

Projecting these problems into the future blocks serious consideration of possibilities for swift deployment of complex, novel, critical systems. This, in turn, blocks consideration of crucial options (both military and economic) that could promote better alignment among competing actors.

Let's explore the possibilities by breaking down the software implementation problem into components that align with emerging AI capabilities.

Scaling speed, quantity, and quality

The most obvious application of AI in software development is to have AI write the code. We're already seeing AI-powered assistants for programmers, which can quickly and cheaply generate code that is untrustworthy, yet can boost programmer productivity. (And Claude Sonnet 3.5 has generated a fresh wave of excitement.)

Paths forward in automating software development may include AI models that surpass human capabilities in writing, critiquing, testing, and debugging code. This unitary, generalist AI approach may seem dated, however; a more realistic path would rely on frameworks that orchestrate AI workflows, using different models (or different prompting, knowledge, or fine-tuning) for specific tasks.

To unpack this further, the natural approach would rely on a large, generalist language model (LLM), trained on both code and discussions of software functionality, capable of understanding human intent, asking clarifying questions, and providing effective prompts to downstream models fine-tuned for tasks in code development. Picture an iterative process with progress reports, unit tests, stringent quality checks, and feedback loops that incorporate user input. Most of the intermediate results would be produced by AI, for AI.

This role architecture outlines an implementation workflow spanning design, development, production, deployment, and adaptation (though “production” may simply mean downloading). In systems combining code and hardware, physical and software implementation workflows would intertwine.

The problem, however, is that human-level software is inadequate, because human-written software is often buggy, vulnerable, or even malicious (consider supply chain attacks). Just automating standard development workflows wouldn’t guarantee reliability and trustworthiness. This calls for something more.

Applying formal methods at scale

Formal methods are mathematically rigorous techniques used to specify, develop, and verify software, ensuring it performs correctly.2 They provide software products paired with machine-checkable proofs of correctness, which can eliminate bugs and guarantee reliability — provided that the specifications themselves are correct and the hardware meets its own specifications, which is a topic in itself.3

Formal methods have been applied to develop systems as complex as compilers and operating systems, but their adoption has been limited by the scarcity of expertise and the cost of manual proof development. Current efforts focus on improving automated theorem-provers and incorporating them into development environments.4

Advanced AI systems are on a path toward automatically generating and verifying code from high-level specifications, augmenting and gradually replacing human effort. Note that there is a hard-edged criterion for success: the code corresponds to the proof by construction, and the proof must be accepted by the proof-checker (a simple, deterministic algorithm).

Because success has precise criteria, learning to code using formal methods can be formulated as a reinforcement learning task (like winning a game of Go) with unlimited backtracking. Poor choices along the way don't cause errors; they merely increase search costs, which decrease as strategies improve. Meanwhile, successful code-and-proof pairs provide gold-standard data for training better models.

And because proof-checkers are small and simple, there’s no need to trust opaque software or the AI systems themselves.

Breaking the Specification Bottleneck

A major challenge in applying formal methods is the bottleneck in developing accurate specifications. However, some tasks are already well-specified, others can be addressed by learning from data, and AI tools can help improve and accelerate the specification process for the rest.

Satisfying universal preferences

Some properties are universally desirable. At the most general level, a computational process should run as instructed by its code without crashing, hanging, or suffering from intrusions that tamper with its execution or data. Formal methods can guarantee these properties in software, provided that the hardware itself operates as intended — executing instructions correctly, storing and fetching data from uncorrupted devices — and without physical interference. Building software on foundations that adhere to the object capability (ocap) model can provide many of these guarantees automatically, inherited from properties inherent in a formally-verified ocap language or operating system.5

Other universal preferences are application-specific. For example, user data should be handled in compliance with regulations (HIPAA, GDPR), and within the constraints set by hardware, databases should satisfy the ACID properties (atomicity, consistency, isolation, and durability). Effects on the world should also satisfy universal preferences: Flight control software should never command an aircraft to crash.

Human burden: Approximately nil

Reimplementing existing systems

With sufficient implementation capacity, AI can be used to replace existing software stacks from bottom to top, providing upgrades that maintain or improve external functionality. This approach can circumvent the ugly task of formalizing and implementing bug-compatibility with existing software components. Instead, formal methods can be used to build from the bottom of the stack, starting with defined processor operations and ultimately supporting interactions with human users and their tools. The goal is to describe and formalize behaviors that satisfy functional requirements, which may be as flexible as the expectations of human users faced with a new software release.

Users generally welcome moderate changes that improve software alignment with universal preferences, such as reliability, efficiency, and smooth scrolling. However, refining software behavior is a slippery slope; as external behaviors are improved, the task may overlap with specifying new functionality.

Human burden: Potentially small and informal

Specifying new functionality

In developing new functionality, AI can help humans make informed choices and formalize specifications that accurately reflect these choices. AI can work with users to explore requirements while considering universal or obvious, yet unmentioned, preferences. AI assistants can highlight potential unintended consequences of choices and help humans explore tradeoffs.

Through discussions, diagrams, and simulated behaviors, interactive, multimodal systems can synthesize the concerns and insights of multiple people and AI systems. This approach avoids assigning humans the impossible task of writing (or even comprehending) masses of opaque, formal specifications. Humans can consult various AI models for summaries, evaluations, critiques, and suggested alternatives while proposing hypotheticals and asking probing questions about potentially problematic behaviors. When tests can be sandboxed or stakes are low, users can test code and provide feedback in a rapid iteration cycle. However, when stakes are high — such as in failure-critical control systems for aircraft or data collection and processing for security applications — developing and judging specifications will call for deliberative processes that are formal in a human, institutional sense (an aspect of the AI Agency model for applying powerful AI capabilities to consequential tasks).

Human burden: Exploring and choosing options assisted by AI

Informal AI in formal frameworks

Developing AI systems that meet formal specifications for behavior — and even real-world consequences of that behavior — has been identified as a challenging but important research objective. Even limited applications of verified code, though not a substitute, can provide leverage on problems of AI control by constraining AI-component interactions and outputs.

Consider the problem of implementing a secure repository of that includes sensitive personal information with strict, verifiable constraints on disclosure. How could unverified AI be applied to this information while preserving privacy guarantees? An AI system could be allowed to explore the repository freely to search for clues that suggest threats, yet would be able to act only by passing specific evidence to a reporting mechanism that itself verifiably adheres to privacy-protecting regulations.

In compound AI systems, multiple models perform tasks orchestrated by conventional software frameworks with verifiable properties. Formal methods can ensure constraints on flows of information between AI systems, and limits on the retention of information by the AI systems themselves. Ocap foundations implemented at the OS kernel level (see seL4) are available today and can do much of the work.

Credible realism and the future of software

Recognizing the potential for AI to transform software development (scope, speed, quantity, and quality) is crucial realistic views of the future. Imagining that the future will resemble the present, with software development remaining a slow, error-prone process, would be a profound failure of imagination. It would constrain our thinking about what is possible and limit our ability to plan for and shape the future.

The strategy of “credible realism” calls for exploring credible possibilities that have consequences aligned with realistic prospects. In the case of software implementation capacity, credible realism calls for focusing on several considerations:

• Continued advances in applying large language models to coding

• The existence and implications of formal methods in software development

• The prospects for AI-enabled scaling of formal methods

• That formal methods can make AI-generated code absolutely trustworthy

Crucially, all of the above points can be discussed as potentially long-term prospects, without arguing for fast or slow timelines. Setting timelines aside, each of these considerations is either credible or simply factual. Together, they give reason to explore potential futures in which novel, verifiably correct software is applied to problems ranging from AI safety to mutual trust among state actors.

Share AI Prospects: Toward Global Goal Convergence

Subscribe now

The fundamentals: Eyes, brains, and hands

The human roles in production involve seeing, thinking, and using our hands to move things. The great breakthrough of the Industrial Revolution was the development of steam-driven machines that could move themselves. And this was a result of humans seeing, thinking, and making things by moving their hands.

Machines have now begun to see, think,2 and use ‘end effectors’ that are somewhat like hands. Early robots moved in repetitive ways, but today’s robots are increasingly able to follow instructions and perform novel tasks. If we think of humans doing work, and automation as replacing humans, then human-like robots may seem to be the way of the future. But does this really align with reality?

Humanoid robots aren’t for advanced manufacturing

A naïve concept of advanced automation pictures factories full of humanoid robots using their hands to build humanoid robots. This is easy to imagine, but it’s neither credible nor realistic: Humanoid robots are needlessly costly, clumsy, and slow.

Even a hand can be expensive: this one costs €110,000, which is roughly half its weight in gold.3

Industrial practicality would require enormous improvements in both cost and performance, yet what we see is the product of decades of work. Humanoid robots may have a role in human environments — where many tasks require eyes, hands, legs, and improvisation — but they’re a poor fit when tasks can be broken down into chains of repeated motions, performed millions of times in environments that can be designed together with the machines themselves.

Today’s factories are designed for humans, and humanoid robots are a retrofit for the residual human jobs — “residual” because most actions are already performed by simpler, less photogenic machines.

How real mass production works

In factories, machines that move things are seldom called ‘robots’ — they’re simpler, specialized, and often fast:

Chains of simple machines that execute complex sequences of motions can reduce costs and increase throughput.4 For example, consider a manufacturing task that requires 1000 motions: A machine that flows products through a chain of 1000 simple, single-motion devices can do the work of 1000 general-purpose robots working in parallel — or more than that, because single-motion devices often work faster by an order of magnitude or more. If a typical single-motion device costs 1/100th as much as a robot and works at 10 times the speed, then throughput per unit cost can be greater by a factor of 1000.

It’s easy to imagine a room full of humanoid robots, but it’s almost impossible to picture a machine that chains together 1000 different devices. This places realism — and credible expectations — at a memetic disadvantage.

Automating the process of automation

We’re now building robots that have something like hands, eyes, and brains, a development that points the way to automating tasks that call for human-like adaptability and problem solving — tasks like assembling unique parts to build specialized systems for high-throughput manufacturing. But automating these tasks is among the most challenging and least important aspects of automation: Their challenges are obvious, but their importance is low because the tasks aren’t frequent, not part of ongoing workflows.

It's possible to doubt the practicality of large-scale employment of human-level robotics while still expecting deep reductions in the labor required for building and operating manufacturing systems.

A key part of the story is automation of the process of automation: the crucial element isn't making products, it’s automating the design and construction of machines that make the products.

High-throughput production machinery may require many specialized single-motion machines, and designing those machines and making their components can be a costly bottleneck. Advanced AI will start to break this bottleneck when generative models take up the burden of machine design.5

CNC machine tools and 3D printing technologies can produce unique parts for unique machines with minimal labor, provided they’re programmed in detail — and automating their programming is usually straightforward, provided that the design fits what the tools can make, a constraint called “design for manufacturing”.6 The picture that emerges looks something like this:

Rather than centering on robots, this implementation work-flow centers on the design and manufacture of machines that are designed to fit available manufacturing processes: Machines can be designed to work in designed environments, making machines that are designed to be made by machines that were co-designed with their products. They key to this level of adaptation and integration is AI-enabled design capacity, not AI-controlled robots.7

The easy-to-imagine prospect, however, looks something like this:

This concept is simplistic, unrealistic, and only superficially credible, yet it does point toward a realistic endpoint.

“It’s a manufacturing problem”

Consider some ways in which these advances could contribute to global goal alignment:

Accelerating renewable energy

The costs of wind and solar energy are mostly costs of installed physical capital. Lower these costs, and the cost of renewables falls steeply. Employ high-throughput production methods, and deployment can be rapid. Installation can also be automated: Use step-and-place machinery to help install made-for-placement photovoltaics and use smart cranes to help assemble the parts of made-for-assembly wind turbines.

Reducing environmental impacts

Why aren’t production processes as clean as possible, or nearly so? The problem is cost — the added costs of pollution control equipment, the potentially higher costs of lower-impact industrial processes, and the cost of energy to run everything. These costs can fall if the necessary hardware becomes inexpensive.

There’s another production process that can be improved to reduce environmental impact: agriculture. Today’s agriculture consumes scarce water and releases pollutants while swallowing land. Enclosed agriculture can conserve water, reduce emissions, and reduce land requirements by greatly increasing yields. And this can reduce the incentives to destroy rich ecosystems in favor of soybeans. Enclosures, of course, must be produced and installed.

Increasing material wealth

Consider prospects for an abundance of mass-produced things and customized services by mass-produced robots: These almost define increasing material wealth.

Winner-take-most becomes very different from winner-take-all when material wealth is on a fast track to abundance. It will eventually cost little to ensure a high standard of living for today’s poor.

Deploying new weapons

Weapon systems are physical products and their production can be scaled, with familiar motivations and risks. Less familiar are prospects for the rapid design and deployment of new weapon systems made possible by AI-enabled design and manufacturing — a prospect quite different from AI-based weapons control.

It seems that we will have unprecedented options to choose between offensive, mixed-use, and inherently defensive systems. Coordinated strategies for multilateral deployment of verifiably defensive systems are possible, and could be appealing if these options become concrete and actionable. To set this process in motion will require deep, multi-faceted analysis that builds on transformative prospects for design and manufacturing.

Possibilities, expectations, and options

Tacit assumptions about future manufacturing capabilities pervade policy research. Because possibilities, options, and interests depend on physical things and how they’re produced, different assumptions about manufacturing can lead to different policy choices.

In the past, when production edged up by no more than a few percent per year, anticipated advances were important, but not transformative. Today, expectations are shifting, opening the Overton window to prospects for radical change. Policy researchers can contribute by expanding their portfolio of credible scenarios in realistic directions.

Share AI Prospects: Toward Global Goal Convergence

Subscribe now