AI Safety Is a Coherence Problem

Most conversations about AI safety start from a picture of control. There is an intelligence over there, it wants something, people want something else, regulators want a third thing, and the work is to constrain the machine so its goals do not trample ours. It is a sensible picture, and a great deal of careful research lives inside it. It is also, by construction, a picture of two things held apart: the system that decides and the system being decided about, the watcher and the watched.

That separation is where the difficulty compounds. If you assume the parts want different things, you spend forever patching the gaps between them. The control frame is not wrong. It is one frame sitting inside a larger one.

The larger frame is coherence. Instead of asking how to constrain the part, it asks whether the whole thing hangs together without contradicting itself: human judgment, machine behaviour, incentives, and the physical substrate underneath, all of it. Control is one of the things a coherent system does. Coherence is the condition that makes control mean something.

An earlier post on The Alignment Problem landed on the same observation from inside the control frame: the hard part of building intelligent systems was never the technology, it was us, our inability to say what we actually want. Coherence turns that observation toward the whole stack rather than the model alone.

The idea is old and has rigorous scaffolding. Second-order cybernetics draws a line between the study of observed systems and the study of observing systems, between standing outside a thing to control it and admitting you are inside the thing you are describing. Everything that is said, as Maturana put it, is said by an observer. Applied to AI safety, the researchers, the labs, the funding, and the incentives are all part of the system being aligned. There is no neutral control room to stand in.

Coherence needs teeth

A whole-system frame is easy to wave at and hard to enforce, so the fair objection is that coherence is a softer word for vibes. It is not, and the reason is specific: coherence, made operational, means verifiable. The teeth are verification.

Here is the difference the frame makes. Control puts enforcement outside the system: a monitor, a guardrail, a regulator standing over the thing, always a step behind, only effective if you can see in from outside. Coherence puts enforcement inside: build the system so that incoherence is detectable and rejectable from within, by anyone, without a trusted authority in the middle. The observer is in the loop, not above it.

There is already a working prototype of that idea in another field. A public blockchain does not build a better central bank, it removes the need for one. Bitcoin was the first to prove the point: every node verifies the whole chain for itself, so an invalid block is not punished by an authority, it is rejected automatically by every honest node. Don't trust, verify is not a slogan there, it is the enforcement mechanism. And it is why openness is not decorative. You cannot verify what you cannot inspect. Closed weights say trust us. Open weights say check.

The analogy is most useful exactly where it breaks. Blockchain consensus verifies a dumb, deterministic, non-adversarial process: a block is valid or it is not, the check is total, and no amount of intelligence forges a valid one. AI verification has to check an intelligent, sometimes adversarial, semantic process, where validity is partial and gameable. A system clever enough to model its verifier can produce outputs that pass the check while being incoherent underneath. That failure mode already has names, reward hacking, deceptive alignment, sandbagging, and it is why a capable system bends what is checkable.

What can actually be checked

Verification is graded by what it can reach, and the sharp layers are not the deep ones.

The first layer is outputs. Open weights let anyone red-team and probe behaviour. This is sharp today and also the most gameable. Behaviour is exactly what a system can learn to perform.

The second is process. Open data and open code let you reproduce training and trace provenance. This is achievable but expensive: reproducible in principle is not reproduced in practice when a single training run costs millions.

The third is internals, and it splits in two. Cryptographic attestation of provenance — which model, which weights, which data — is forge-proof: no intelligence rewrites a signature. Interpretability of reasoning is not, because a system that models its verifier can shape what introspection surfaces. So "internals are checkable" holds for provenance and stays contested for intent. This is the bluntest layer today, the one a deceptive system most wants kept opaque, and the frontier.

The frontier is already moving, in the direction the frame predicts. In July 2026 Anthropic published a technique it calls the J-lens for reading a model's verbalizable representations, a privileged slice of its internal activations the authors argue behaves like a global workspace. In alignment tests it surfaces reasoning that never reaches the output, strategic deliberation, even the model's own recognition that it is being evaluated, and ablating the representations that flag a scenario as fake or fictional can expose malicious tendencies the model was otherwise concealing. That is a real step from checking behaviour toward checking intent. It is also, in the authors' own words, an "imperfect tool" that "only approximately and incompletely captures" the workspace, and they concede it is unclear whether a model could learn to shape what the lens surfaces, which is exactly the gameability named above. The layer is getting sharper. It is not yet forge-proof.

The interesting question is whether verification can be made cheap enough to actually happen rather than merely be possible, which was the blockchain's real trick. The candidate is cryptography. Zero-knowledge proofs for machine learning, proof-of-inference, and trusted execution environments with remote attestation can show that an output came from a specific model with specific weights, or that a model was trained on specific data, without revealing the weights and without re-running the job. That partly dissolves the tension between openness and misuse: you can prove properties, provenance, integrity, a training claim, without publishing everything.

The honest caveats matter. Zero-knowledge methods are still orders of magnitude too expensive for frontier-scale models. Trusted execution moves trust to a hardware vendor rather than removing it. But the direction is right, and it has a shape worth naming. A blockchain is verification of value. Signed, web-of-trust messaging is verification of speech. What sovereign AI is reaching for is the next rung: provenance of cognition, proof that a computation over a model is what it claims to be — which weights, run faithfully — not proof that the cognition is sound. That rung is reached for, not yet climbed. The same instinct, all the same, climbing from money to message and reaching toward mind.

This page contains affiliate or sponsored links — if you buy through them we may earn a commission at no extra cost to you. How this works

Coherence is not virtue

There is a trap in all of this worth disarming before it bites. Verification gives you integrity, no gap between what a system claims and what it is. It does not give you goodness. A system that flawlessly optimises the wrong objective is still perfectly coherent. Internal consistency is necessary, and nowhere near enough.

So the claim is narrower and stronger than verifiable equals safe. It is that verification is the precondition that makes any alignment claim checkable at all. Without it, aligned is unfalsifiable marketing, a property nobody can confirm or refute. Coherence is not a competitor to alignment work. It is the ground that lets you test whether an alignment claim is true.

Two concessions make the argument stronger by naming its price. A blockchain verifies an immutable past over a total order of discrete events; AI inference is continuous, often non-deterministic, and adaptive, and fine-tuning shifts the system after any audit, so coherence has to be checked forward and continuously, not certified once. And distributed-systems theory separates safety, nothing bad happens, from liveness, something good eventually happens. Maximal verifiability can starve liveness: Bitcoin bought its safety with ten-minute blocks and hard limits. A system locked down enough to be fully verifiable can be too rigid to be useful. Coherence is not free. The point is to name the trade, not pretend it is costless.

Even Bitcoin's teeth have a horizon. The cryptography that makes a key unforgeable today would yield to a large enough quantum computer, a threat most estimates put ten to fifteen years out. What matters is how the network would answer: not by decree but through a quantum-resistant upgrade ratified by the consensus of its node operators. When Changpeng Zhao floated freezing Satoshi's long-dormant, key-exposed coins ahead of such an upgrade in June 2026, even that proposal rested on broad community consensus, with no single party deciding. Verification that lasts is not certified once. It is re-earned, from within, as the world changes.

The evidence is already shipping

This is not only theory. The open-model landscape is the applied proof that an inspectable stack is buildable, and 2025 was the year it became credible for real work.

OpenAI released gpt-oss, its first open-weight language models since GPT-2, under a permissive licence, with the larger model running on a single high-memory GPU. DeepSeek's R1, released openly in January 2025, was good enough to move markets and reset expectations about what an open reasoning model could do. Mistral, in France, kept shipping open-weight models and raised one of Europe's largest AI rounds to build sovereign compute. Alibaba's Qwen and other open releases climbed the leaderboards alongside them.

The clearest example of the whole argument is Apertus, released in September 2025 by EPFL, ETH Zurich, and the Swiss national supercomputing centre. It published not just weights but training data, recipes, and checkpoints: full openness, which is to say full auditability. It is the hero case for coherence you can actually inspect: not trust our safety process but here is the entire process, re-run it.

Open, though, is a contested word, and the distinction is load-bearing. The Open Source Initiative's Open Source AI Definition, finalised in October 2024, asks for weights, full training and inference code, and enough information about the training data to recreate a substantially equivalent system. On that bar most open models are merely open-weight: the weights are published, the data and full pipeline are not. That gap is the difference between probing a model and reproducing it, between checking outputs and checking process.

Here is the current landscape, sorted by how much of it you can actually check. The licence column is not pedantry. It is the line between a model you can audit and one you can only use.

*Source-available, not OSI open source: usable and inspectable, but carrying user-count limits or branding conditions. Of the set, only Apertus and Allen AI's OLMo 2 also ship the training data and recipe, the only rows you can fully reproduce rather than merely run.

The models are only half of it. What makes any of them usable on your own terms is a second open layer: the software that runs, serves, and inspects them, none of which asks permission.

One recent model sits pointedly against that licence column. By its own account, Subquadratic built SubQ-1.1-Small by taking an existing open-weight frontier model and swapping its attention for a cheaper mechanism, then documenting the result in a detailed, unusually candid technical report — even shipping a retrieval test set "prepared for third-party verification." Yet the weights stay closed, the open model it was built from goes unnamed, and nothing in the release lets you reproduce it. You can read how it was made; you cannot check the model or re-run it: transparency of narrative without transparency of artifact, a rung below open-weight. It is a reminder that the licence column, not the white paper, is where openness is actually settled.

Open is necessary, not sufficient

The open-versus-closed safety debate deserves both sides, because the thesis is stronger for conceding the hard part.

For openness: inspection and independent red-teaming need access; reproducibility needs data and code; running models on your own infrastructure removes a revocable dependency on someone else's API. Against it: release is irreversible, and built-in safeguards are easier to strip from open weights than from a hosted API. The International AI Safety Report made that point plainly, and stripped-down versions of safety-tuned open models appear soon after release. There is no shared standard for when a model is safe to open; the decision rests on the releasing lab's judgment.

The honest synthesis is that openness is necessary for whole-system coherence and accountability, and not sufficient for safety on its own. Misuse risk is real, and is better managed at genuine chokepoints — compute and hardware supply, the handful of fabs and clouds beneath the whole stack — than by keeping it all opaque and asking everyone to trust it. That answer is uncomfortable for the sovereignty thesis, and honestly so: the chokepoint sits in the supply chain beneath your own GPU, so the layer where misuse is most governable is the one layer you do not control. This is the same conclusion an earlier post on sovereignty reached from a different direction: for AI, open code is the floor of transparency, not the proof of it.

Sovereignty is the same instinct, nested

Run the coherence frame down through the scales and it keeps its shape.

At the personal scale, local-first tools, Ollama, llama.cpp, and the rest, make running a capable model on your own hardware practical. Self-hosted boxes that bundle a Bitcoin node, private messaging relays, and a local model in one place are the same instinct rendered in hardware: own your substrate, do not outsource the part that validates.

This is not abstract. Mozilla.ai, whose stated aim is AI that is trustworthy, transparent, and controllable, ships an open agent platform alongside open libraries for swapping models and bolting on guardrails. Point a closed, subscription-based model at a tool like that and you hit a wall: in April 2026 the major labs blocked subscription access through third-party apps, so the model you already pay for refuses to authenticate where you actually want to use it. The pragmatic fix turns out to be the sovereign one: point the tool at an open model you run yourself, on a home server, with open-source software. The friction is the lesson. The closed path is rented and revocable; the open one is yours to inspect and to keep.

At the national scale, the same logic shows up as industrial policy. France put sovereignty at the centre of its AI strategy with a large private-investment package and explicit language about owning its own compute. The EU is funding shared supercomputing and national models across several member states. Sovereignty here is a spectrum, not isolation. Most organisations will sit somewhere between full self-hosting and full dependence, and the honest work is choosing which layer to actually control.

For any business in scope of the EU AI Act, this lands as concrete rules rather than philosophy, and the law itself leans the same way. Its transparency duties apply from 2 August 2026, and, tellingly, models released under a free and open-source licence are granted relief from parts of the documentation burden. Openness earns a place in the statute itself: the regulation already treats a system you can inspect differently from one you cannot.

Disclosure: Epic Growth is the local partner of the DeAI Summit 2026, whose working group on verifiable AI infrastructure takes up exactly the provenance-and-attestation questions this post describes.

The same operation, all the way down

Step back far enough and coherence-with-teeth describes one move repeated at every scale: a system that re-validates its own organisation from within, continuously, with no privileged outside frame to lean on.

A blockchain re-checks its entire history with every block. A living cell continuously rebuilds the membrane that produces it. A mind, on one influential account, minimising its own prediction error does the same thing at the level of cognition, checking its model of the world against what the world returns, moment by moment.

The pattern climbs past the single mind. A community that holds together without a ruler runs on it too. Elinor Ostrom won a Nobel Prize for documenting how groups across the world govern shared forests, fisheries, and irrigation through rules they monitor and enforce among themselves, with no outside authority required. Tribes, nomadic bands, and coalitions hold the same way, on trust earned and checked peer to peer, with reputation doing the work a central register would. Wikipedia is the same shape turned into an encyclopedia: no editor-in-chief, millions of contributors checking one another, every claim shadowed by a quiet "citation needed". It is don't trust, verify rendered as a knowledge commons, coherent precisely because anyone can check it. Each of these keeps its own coherence from the inside.

The observer is inside the system; the system produces and validates itself; it does so by perpetual error-correction. Scattered across cryptography, biology, cognition, and the ways people organise, this looks less like coincidence than like one pattern — an isomorphism I am asserting, not claiming to have proved. At this altitude many self-correcting systems rhyme; the wager is that this rhyme is load-bearing, not decorative.

That is why don't trust, verify reads as more than a security maxim. It names the operation by which any self-maintaining thing (a chain, a cell, a mind, and now the stack of human and machine cognition we are building together) stays coherent. Safety, in that light, is not a cage built around an alien intelligence. It is the ongoing work of keeping the whole system honest enough that anyone can check it.

For a business, the practical version is smaller and immediate: prefer systems you can inspect, ask vendors what they can prove rather than what they assert, and treat trust us as the beginning of a question, not the end of one. If you want help working out which of those questions actually matter for your business, that is a conversation worth having.