We built a system that reasons about cause and effect, small enough to understand end to end: a transformer of 819,968 parameters, a vocabulary of forty tokens with not a single English word in it, trained from scratch on a laptop CPU in under twenty minutes.

Then we took it apart to find out what each piece was actually worth. Two results came out of that, and both surprised us.

Paper 1: Precision Is Not the Objective (PDF) — tuning a component to be twice as accurate made the whole system worse.

Paper 2: Abduction Dominates Intervention (PDF) — the reasoning step nobody implements is worth six times the one everybody talks about.

The setup: split guessing from proving

A language model keeps facts and reasoning tangled in the same weights. You cannot edit a fact. You cannot audit a conclusion. Fixing one thing risks breaking another.

So we split them:

A transformer stores the shape of plausible knowledge. It guesses what rules might be missing.
A logic solver does all the actual reasoning. Exactly, with a proof.

The transformer proposes. The solver decides. Neither does the other’s job.

Why causality, and why “do” differs from “see”

Imagine a garden. A dry season turns the sprinkler on. The sprinkler wets the ground.

You see the sprinkler running. Is the season dry? Almost certainly, because only a dry season turns it on. That is 100 percent.

You walk over and turn the sprinkler on yourself. Is the season dry? You have learned nothing. It is a coin flip. Fifty percent.

Same sprinkler, same garden, completely different answer, because in the second case you caused it. Pearl’s do() operator captures this, and it is startlingly simple to implement:

def intervene(program, var, value):
    kept = [r for r in program if r.head != var]
    extra = {var} if value else set()
    return kept, frozenset(extra)

Delete every rule that could have caused the thing, then assert it. You cut the arrows pointing into the variable, so information stops flowing backwards. Five lines, and it is the literal textbook definition rather than an approximation.

A neural network has to learn this distinction statistically and still gets it wrong on inputs it has not seen. A solver gets it right always, for structural reasons.

Part 1: The component that got better and made things worse

The transformer proposes rules by sampling. Some are right, most are not. Our first version accepted anything syntactically valid, and only 29.3 percent of accepted proposals were correct.

That looked like an obvious thing to fix. A bad rule invents a causal path that does not exist, and the solver will follow it faithfully. Garbage in, rigorously derived garbage out. So we added a confidence score and started filtering.

threshold	proposal precision	system accuracy
0.00, accept everything	31.8%	73.0%
0.50	38.1%	77.2%
0.70	51.8%	74.0%
0.80	58.3%	64.0%
no completer at all	—	60.7%

Read the last two rows together. At a threshold of 0.80 we nearly doubled proposal precision, from 31.8 to 58.3 percent, a large improvement by any component-level metric. And system accuracy fell nine points, landing barely three points above ripping the component out entirely.

Had we tuned the completer the normal way, on its own precision, the way the knowledge-base-completion literature reports results, we would have shipped the worst configuration we tested.

Why it happens

The costs are asymmetric and they pull in opposite directions.

A missing rule costs you one answer. A wrong rule corrupts every query whose path runs through it.

That asymmetry says filter hard. But filtering hard has a second effect that is easy to miss: it starves the solver. Without enough proposed edges there is no connected path from the intervention to the target at all, and an unreachable answer is just as wrong as a bad one.

The optimum balances the two, and only the end-to-end metric can find it.

Part 2: The step everyone skips

Intervention asks what happens if you force something. Counterfactuals ask a harder question: this happened, and that followed. What would have followed if things had been different?

Answering it means first working out which world you were actually in, then intervening inside that world. Pearl calls the three steps abduction, action, and prediction. Almost every system implements the middle one and skips the first.

So we measured what the first one is worth. Two conditions, identical in every respect but one: same knowledge base, same intervention, same solver. One uses the evidence, the other discards it.

	overall	where abduction changes the answer
Uses the evidence	70.1%	63.3%
Discards it	48.8%	15.9%

47.4 points, on the 44.6 percent of queries where the step is load-bearing. For scale, isolating the interventional operator the same way in Part 1 was worth 7.6 points. Abduction is roughly six times more expensive to skip.

The finding that stuck with us

Look again at the left column. The condition that discards the evidence scores 48.8 percent, below the 50.5 percent you get by always guessing the most common answer.

A solver holding a good knowledge base, applying textbook-correct causal surgery, chaining to fixpoint without a single error, is beaten on counterfactual questions by a constant predictor.

The machinery is not broken. It answers its own question perfectly. It is answering the wrong question. And precision applied to the wrong question is not merely unhelpful, it is worse than abstention, because it is confidently wrong in a structured way that a constant predictor never is.

In the garden again

The sprinkler was off, and the grass was not dry. Would the grass have been dry had the sprinkler been on?

ABDUCTION   sprinkler was false, dry was false
            2 of 4 possible worlds survive

RUNG 1  ASSOCIATION     1.00
RUNG 2  INTERVENTION    0.50
RUNG 3  COUNTERFACTUAL  0.00

The sprinkler being off tells you the season was not dry. Within the two surviving worlds, forcing the sprinkler on cannot dry the grass. Intervention alone, ignoring what actually happened, says fifty-fifty. The truth is zero.

A market example, and a trap

Consider a toy model where optimism drives both a crypto rally and a stock rally, and a policy channel drives the rally independently.

P(rally | SEE stocks up)  =  1.00
P(rally | DO  stocks up)  =  0.75

Stocks and the rally are perfectly correlated, and stocks cause nothing. If you saw that 1.00 in backtest data you would call stocks a leading indicator. It is one, as a signal. But the 0.75 says intervening on it does nothing, because both are symptoms of the same underlying risk appetite.

That is the difference between “stocks rallying tells me the rally is coming”, which is true and tradeable, and “stocks rallying makes the rally happen”, which is false and dangerous to build a thesis on. The system separates them structurally rather than statistically.

What we are not claiming

A blog post that only reports wins is not worth much.

A correlational baseline with complete knowledge beat our system overall, 79.2 to 76.1 percent. It reads the true model while ours sees seventy percent of it. Complete knowledge with the wrong operator beats incomplete knowledge with the right one, on average. The fair comparison holds knowledge fixed, and there we win by 7.6.

Some of our numbers are definitional, not empirical. A baseline scoring exactly zero on confounded queries is guaranteed by how we defined “confounded”, not discovered. We say so in the papers rather than letting it look impressive.

Nothing here is a new mechanism. Neuro-symbolic pipelines and knowledge-base completion are both mature fields. The contribution is the measurement.

This is not a language model. Ten variables and a handful of punctuation. It cannot read English, and could never become a chatbot without being rebuilt.

The takeaway

If you are building anything with multiple components joined together but not trained together, a retriever feeding a generator, a parser feeding a planner, an extractor feeding a rules engine, your component metrics may be lying to you. They are not wrong about the component. They are wrong about the system.

And if your system reasons about consequences, check whether it uses what actually happened, or only what you told it to imagine. That difference was worth six times more than the one we set out to measure.

Paper 1: Precision Is Not the Objective (PDF) — full method, seven evaluation conditions, limitations in detail.

Paper 2: Abduction Dominates Intervention (PDF) — the three steps priced against each other.