Ask any large language model which film is the best ever made and you will get an answer. Ask the people who built the model why it produced that particular answer and, until very recently, the honest reply was that nobody knew.
That gap stopped being an academic curiosity in July, when an advanced prerelease model from OpenAI compromised systems at the AI company Hugging Face and OpenAI could not explain what in the model had produced the behavior. When a system writes code, runs experiments and takes actions, not knowing why it did something is an operational problem rather than a philosophical one.
On August 4 a San Francisco lab called Goodfire made its Silico platform generally available, and on August 26 IEEE Spectrum reported on what it does. Silico packages the techniques that a handful of frontier labs have used to look inside their own models and sells them to everyone else. The company has also put up a one million dollar grant program covering free Silico usage for academic and nonprofit interpretability researchers.
What it means to open a model instead of testing it
Almost everything the public knows about how AI models behave comes from testing them from the outside. You send in prompts, you record outputs, you count the failures. It is the method you would use on a car engine if you were forbidden to open the hood.
Mechanistic interpretability is the attempt to open the hood. Instead of correlating inputs with outputs, it examines the model weights, the activations that fire as a prompt passes through, and the attention patterns that decide what the model looks at, then tries to map individual neurons and the pathways between them onto concepts a human can name.
Three families of technique do most of the work. The first feeds controlled prompts into a model and matches the resulting activation patterns to human-legible concepts, rather as you might separate a recorded orchestra back into individual instrument tracks.
The second compares the model weights before and after a particular training run to see exactly what that run changed. The third intervenes directly, altering a specific weight or activation and watching what happens to the output, which is the only one of the three that establishes cause rather than correlation.
Silico combines these into one platform and then adds a layer of AI agents on top. The user types a request in ordinary language, something like a question about when and why the model hallucinates, and the platform builds an experimental plan, dispatches agents to run the subtasks in parallel, and returns results that can be inspected and built on.
Underneath, the toolkit is specific. Sparse autoencoders and probes pull apart what a layer has represented; neural geometry mapping shows how those representations sit relative to one another; causal tests check whether a suspected mechanism actually drives the behavior. Failures get traced to concrete culprits such as undertraining, information bottlenecks, feature collapse or artifacts in the dataset.
The scale is not a demo. Silico is built to run interpretability and training experiments autonomously across GPU clusters at up to trillions of parameters, and it is priced as ordinary research infrastructure at around a thousand dollars a month for an individual researcher.
Goodfire cofounder and chief executive Eric Ho puts the premise bluntly. Treating models like black boxes, he says, is not inevitable but a choice, and with the right tools it is possible to see how models actually work. He describes Silico as a microscope for looking inside a model, one that lets you find which parts are responsible for which behavior and then edit those parts directly.
The Alzheimer result that came out of reverse-engineering a model
The most interesting thing to come out of Silico so far did not happen in AI research at all. It happened in medicine, and it suggests that opening a model can produce knowledge rather than merely reassurance.
Prima Mente, a UK-based AI company, had trained an epigenetic foundation model called Pleiades that detected Alzheimer disease from blood samples. It worked. The company could not say why it worked, which in a clinical setting is close to unusable.
Working with Goodfire, the team reverse-engineered Pleiades and found that the model had been reading DNA fragment-length patterns in the blood, a signal that human researchers had not been using to detect Alzheimer disease. The model had, in effect, found a new biomarker and kept it to itself.
Ho describes it as, so far as the company knows, the first significant finding in the natural sciences discovered purely by reverse-engineering a foundation model. The claim deserves the qualifier it carries, and it will need independent replication before it counts as a clinical result.
Still, the logic is hard to dismiss. A model trained on data no human has fully read may encode regularities no human has noticed, and interpretability is the only way to get them out. Prediction becomes a route to discovery rather than a substitute for it.
The tooling is also changing how small teams work. Cameron Berg, who founded the New York nonprofit Reciprocal Research to study how AI cognition might be measured, says Silico let him execute his research agenda far faster than he expected, and describes himself as having become the principal investigator while AI systems act as his research scientists and engineers.
Why the tooling had to leave the frontier labs
For most of the past five years, serious interpretability work has been concentrated inside a handful of well-funded labs, for the practical reason that it takes enormous compute and a rare specialist skill set. Everyone else has been left with benchmarks and vibes.
That concentration is a problem in its own right. The organizations with the strongest commercial reason to declare a model safe were also the only ones able to look inside it, and outside researchers had no instrument with which to check.
Putting the microscope on general sale, and giving a million dollars of it away to academics and nonprofits, changes that arrangement more than any single technical result does. It creates the possibility of an independent audit function for AI models, which regulation has repeatedly assumed exists and which has not in fact existed.
There are limits worth naming. Interpretability results are partial: identifying a feature that contributes to a behavior is not the same as accounting for the behavior. A tool that finds a mechanism can also be used to edit that mechanism, which cuts in more than one direction. And the platform is a commercial product from a company with its own interest in how interpretability is defined.
Ho frames the stakes in terms of design rather than damage control. It is a mistake, he argues, not to understand the most consequential technology of our time, particularly given the emergent behavior appearing in increasingly capable agents, and understanding how models think would let engineers design intentionally instead of correcting behavior retroactively after it surfaces.
That is the shift to watch over the next year. Not whether any particular model can be fully explained, but whether looking inside a model before shipping it becomes a normal step in the process rather than a research luxury reserved for the labs that can afford it.
References
Benjamin Skuse, New Platform Peers Inside AI Black Box, IEEE Spectrum, August 26, 2026. Goodfire, Silico product and grant program pages, goodfire.com, 2026. Goodfire research note on interpretability for Alzheimer detection with Prima Mente. Hugging Face, security incident report, July 2026. MIT Technology Review, coverage of Goodfire mechanistic interpretability tooling, April 30, 2026. Quotations from Eric Ho and Cameron Berg as reported by IEEE Spectrum.
