How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding
Problem Statement
It remains unclear whether the natural-language 'thoughts' generated during CoT prompting faithfully reflect a model's true internal reasoning or are post-hoc text layered on unchanged computations. Prior CoT-faithfulness studies were largely behavioral/black-box, lacking causal, feature-level mechanistic evidence. This work fills that gap by directly manipulating SAE-derived features via activation patching to test causal influence and interpretability changes across model scales.
Key Novelty
- First feature-level causal study of CoT faithfulness, combining sparse autoencoders with activation patching
- Introduces patch-curves and random-feature-patching baselines to test whether CoT-relevant causal information is concentrated in top features or widely distributed
- Reveals a scale-dependent emergence of CoT's causal and interpretability benefits by contrasting Pythia-70M and Pythia-2.8B
- Provides direct evidence that CoT increases activation sparsity and monosemantic feature interpretability, indicating more modular internal computation in larger LLMs
Evaluation Highlights
- Patching CoT-reasoning features into noCoT runs significantly raises answer log-probabilities in Pythia-2.8B, with no reliable effect in Pythia-70M
- Model confidence (logit-based) in Pythia-2.8B improves from 1.2 to 4.3 after CoT-feature patching
- Activation sparsity and feature interpretability scores are significantly higher under CoT vs noCoT in the 2.8B model
- Patch-curve analysis with random-feature baselines shows useful CoT information extends beyond the top-K features and is widely distributed
Signal Assessment
Methodology
- Train and apply sparse autoencoders on Pythia-70M and Pythia-2.8B activations to extract monosemantic, human-interpretable features
- Run both models on GSM8K math problems under CoT and plain (noCoT) prompting to obtain paired activation traces
- Identify features most associated with CoT reasoning and use activation patching to swap them into the noCoT run
- Measure causal impact via changes in answer log-probability/confidence, activation sparsity, and feature interpretability scores
- Build patch-curves across varying numbers of patched features and compare against random-feature patching baselines to assess how CoT-relevant information is distributed
System Components
Decomposes dense model activations into a larger set of sparse, monosemantic features that are more human-interpretable than raw neurons
Causal intervention that swaps activations/features from a CoT run into a noCoT run to measure the resulting effect on model outputs
A novel diagnostic plotting model performance/confidence as an increasing number of top CoT-features are patched, revealing how causal information is distributed
Control condition patching randomly selected features instead of top-ranked CoT features, verifying that observed effects are specific rather than generic
Two open-source LLMs of different scale used to study how CoT's causal and interpretability effects vary with model capacity
Grade-school math word problem dataset used as the multi-step reasoning task for evaluating CoT effects
Results
| Metric | noCoT / Baseline | With CoT Features | Delta |
|---|---|---|---|
| Answer confidence (logit), Pythia-2.8B | 1.2 | 4.3 | +3.1 |
| Answer log-prob after CoT-feature patching, Pythia-2.8B | noCoT baseline | Significant increase | Positive & significant |
| Answer log-prob after CoT-feature patching, Pythia-70M | noCoT baseline | No reliable change | ~None |
| Activation sparsity & interpretability score, Pythia-2.8B (CoT vs noCoT) | Lower (noCoT) | Significantly higher (CoT) | Increased modularity |
Key Takeaways
- CoT's causal benefit on internal computation and interpretability may be an emergent property of scale—small models (e.g., 70M) may not undergo the same mechanistic restructuring as larger ones (2.8B+)
- Sparse autoencoders combined with activation patching offer a practical toolkit for auditing whether prompting strategies like CoT produce genuine internal changes rather than superficial output differences
- CoT-relevant information is broadly distributed across many features rather than concentrated in a few, so interpretability or pruning methods should not assume a sparse, localized reasoning circuit
- For high-capacity LLMs, CoT prompting appears to genuinely induce more modular, sparse, and interpretable internal representations, supporting its use as more than a superficial prompting trick
Abstract
Chain‑of‑thought (CoT) prompting boosts Large Language Models accuracy on multi‑step tasks, yet whether the generated ``thoughts'' reflect the true internal reasoning process is unresolved. We present the first feature‑level causal study of CoT faithfulness. Combining sparse autoencoders with activation patching, we extract monosemantic features from Pythia‑70M and Pythia‑2.8B while they tackle GSM8K math problems under CoT and plain (noCoT) prompting. Swapping a small set of CoT‑reasoning features into a noCoT run raises answer log‑probabilities significantly in the 2.8B model, but has no reliable effect in 70M, revealing a clear contrast for these two scales. CoT also leads to significantly higher activation sparsity and feature interpretability scores in the larger model, signalling more modular internal computation. For example, the model's confidence in generating correct answers improves from 1.2 to 4.3. We introduce patch‑curves and random‑feature patching baselines, showing that useful CoT information is not only present in the top-K patches but widely distributed. Overall, our results indicate that CoT can induce more interpretable internal structures in high-capacity LLMs, validating its role as a structured prompting method.