Making “why” testable

I made a language model answer “India Rupee” to a question about Kyoto. The prompt still said Kyoto. I had changed a numerical representation inside the model, and its answer changed from Japanese yen to a different country's currency.

That seemed like a way to investigate how it reached the answer. If the model gets from Kyoto to yen through Japan, changing that intermediate should have a predictable effect. India should lead to rupees.

The difficulty is establishing whether that's what the intervention changed. I might have altered a step the model was already using. I might also have introduced a route it wouldn't otherwise have taken.

In Rabbit Hole, I explored opening branches from an AI answer so readers could follow its terms and assumptions. I wanted to take “why?” further: let a reader follow an explanation into something they could test. These experiments gave me a way to try that with access to the model's internals.

I used a coding agent to build and run two local experiments with Qwen3-1.7B on my M4 Pro. The method was the Jacobian lens, or J-lens. It maps intermediate activations, the numerical state inside the network, toward the model's output vocabulary. This gives us directions associated with particular tokens, including country names. We can inspect those directions and use them to construct an edit. The labels tell us how the directions were derived; their role in a particular answer still needs testing.

I used an existing fitted lens to construct the J-space edits below. The question was narrow: what happens to a currency answer when I edit country-related directions while keeping the input fixed?

The first experiment asked the model to complete sentences like this:

Fact: The currency used in the country containing Kyoto is the

The targeted edit produced the intended currency in 25 of 32 questions. Random edits produced it in 3 of 96 runs, using three random directions on the same questions. An unrelated-concept edit produced it once in 32 runs. The effect had some specificity: changes of comparable magnitude did not all produce the currency I predicted. First experiment

Two findings made the explanation less straightforward.

First, the country was easier to identify from the actual output probabilities than from the selected internal readout. At that readout location, its median vocabulary rank was 206 with J-lens and about 1,389 with an ordinary logit lens, which reads intermediate states directly through the model's output mapping. In the actual next-token distribution, the median rank was 7.5.

Most generated answers didn't spell out the country name. But a word can lose the competition to become the next token and still have a relatively high probability. Its absence from the sentence hadn't established that I needed internal access to find it.

Second, the successful edit was strong and late. It affected every original prompt position near the end of the model's 28 blocks. Some changes were larger, by vector length, than the activations being edited. Failed outputs included “Turkeylite.” This was a substantial disturbance.

Look again at “India Rupee.” The model writes the country before the currency. Perhaps the intervention pushed it to say “India,” and the next token simply followed from that word. That would explain the output without establishing that I'd changed a silent country lookup in the original computation.

The follow-up tested whether an effect survived before the model could write that cue. I used 32 fresh cities and asked for a three-letter currency code, such as JPY or INR. The intervention moved earlier, to block 13, and touched only the last token of the city name. Its actual size was capped at 5% of that activation vector's length.

Those settings came from a separate development set. I froze them before evaluation, along with the target countries, scoring rules and two versions of the question. Generation stayed unconstrained: the model could write whatever it chose.

The two wordings exposed a problem immediately. With “Return its three-letter ISO 4217 code and nothing else,” the unmodified model produced a bare currency code for 29 of 32 cities. With “State the currency used in {city} using only its three-letter ISO 4217 code,” it produced no bare answers at all. Twenty-nine responses still contained the correct code in prose. The wording changed how it answered, even when it knew the currency.

I kept those results. The failure to follow the requested format was part of what the experiment needed to show.

Across both wordings, the smaller J-space edit increased the log probability ratio of the target currency code to the source code by an average of 1.30 nats. Almost all the change, 1.25 nats, was already present in the probability distribution before the first answer token. It couldn't have been caused by a country word the model had already emitted. That first token can be a fragment such as “JP” in JPY; moving its probability doesn't establish that the model will choose the complete code.

For the paraphrase, that measurement concerns the probability of starting immediately with a code. The model generally chose to begin a prose explanation instead. A probability measurement can be valid while describing an answer form the model rarely chooses.

A simpler edit using ordinary country directions also moved the probabilities. J-space exceeded that control by 0.93 nats at the matched magnitude and position. Random and unrelated edits were near zero; reversing the targeted direction moved the ratio the other way. This supports a direction-specific effect, with an advantage for J-space over the particular simpler construction tested. The setting had been selected for J-space, so it wasn't an exhaustive comparison of methods. Follow-up report

The generated answers barely changed. Across the 52 observations where the baseline contained the correct code, the smaller edit never produced the intended target as a bare answer. Even on the main prompt alone, where the format mostly worked, the result was zero switches among 23 correct baselines.

The only bare target success was Ipoh. Its answer changed from SGD to IDR, as predicted for the intervention toward Indonesia. But the baseline was already wrong: Ipoh's correct code is MYR.

Hue made the gap between the metric and the answer especially clear. The intended target was Poland's PLN. The probability of its first token, “PL,” increased but remained below one in a million. Meanwhile, the probability of the correct source prefix fell sharply. The ratio improved, yet the generated answer changed from the wrong USD to the wrong EUR. Lowering the correct answer's probability can make that ratio look better without making the intended answer competitive.

I also repeated the old strong edit on the new task. It produced 13 bare target answers on the main prompt, eight from correct baselines. An emitted country name therefore wasn't necessary for every strong-edit success. That result limits my earlier alternative explanation. It doesn't isolate why the stronger intervention worked, because its strength, layer and number of edited positions all differed from the smaller edit.

What I can establish is an effect before the model writes an answer: the selected country-directed edit changes currency-code probabilities. What remains unresolved is whether that direction was a step the unmodified model naturally relied on. An intervention can introduce an association as well as alter an existing one. These runs don't recover a circuit or establish that the original answer required this representation.

Both studies used one small model and a constructed set of currency questions. The follow-up held out cities while reusing the previously studied countries. The results describe that experiment; they don't give a general success rate for steering models.

That distinction changes what I would build into Rabbit Hole.

A branch labelled “Japan” could look like an explanation of the Kyoto answer. I'd want the reader to see what gave that label its authority. Was Japan prominent in a readout? Did an edit change the probabilities? Did it change the generated answer? In these experiments, those questions had different answers.

For an ordinary chat product, I'd start with a claim we can test through the input. Suppose a model says a design survives a regional outage because a second region holds a replica. A “why” branch could turn that claim into a prediction: remove the replica from the scenario, and the model should withdraw the guarantee. The branch would retain the exact edit and compare a fresh response with the original. Repeated runs would show whether the result is consistent.

An unchanged answer would leave several possibilities open. Another fact might provide the same protection. The explanation might have named the wrong dependency. The model might have failed to account for the change. Those possibilities would guide the next question without pretending that a prompt edit reveals the internal mechanism.

With local model access, the same branch could hold the intervention, controls and measured probabilities. The interface would need room for the awkward result too: the country-directed edit moved the numbers, while the answer stayed the same. A reader should be able to follow that result back to the question, the exact change, and the saved run that produced it.