Do LLMs grow internal “modules” during post‑training?

The most directly relevant paper here. It compares base and post-trained models and studies internal changes in knowledge, truthfulness, refusal, and confidence. It reports that truthfulness and refusal can be represented by vectors in hidden space, while factual knowledge locations are not simply overwritten.

A broad survey of SFT, RL, test-time scaling, reasoning post-training, and alignment. Useful background for how post-training changes model behavior, though less mechanistic than the papers below.

Shows that alignment/fine-tuning can improve reasoning behavior, but can also create self-assessment or “assessment misalignment” issues. Good reminder that better outputs do not necessarily mean cleaner internal cognition.

Frames interpretability around population-level representations rather than individual neurons. Demonstrates monitoring/manipulation of safety-relevant phenomena such as honesty, harmlessness, and power-seeking.

Finds directions across selected attention heads that improve TruthfulQA performance when applied at inference time. Strong evidence that “truthfulness” is partly encoded in steerable internal directions.

Finds truth-like directions in activations using logical consistency properties, without labeled outputs. Important for the idea that models can internally represent facts they do not reliably say.

Shows that in-context input-output tasks can be represented as compact vectors transported by a small number of attention heads. This is one of the clearest “module-like” findings: a task can become a reusable internal object.

Finds task-specific neurons after prompt tuning and shows perturbing them hurts task performance. Important caveat: the authors argue these structures are mostly generated in pretraining rather than fine-tuning.

Attempts neuron-level attribution for predictions. Useful for the localization question: are skills and knowledge stored in identifiable neurons, or only in distributed populations?

The ROME paper. Finds that middle-layer MLP computations mediate factual associations and can be directly edited. Good evidence for semi-local factual-memory mechanisms.

MEMIT extends direct model editing to many factual memories. Helpful for thinking about where “knowledge structures” live and how localized they are.

Explains why many concepts can be packed into shared neurons or dimensions. This is the central reason “modules” are hard to find cleanly: internal representations are often superposed and polysemantic.

Uses sparse autoencoders to decompose messy activations into more interpretable features. A major step toward mapping the “feature ecology” inside models.

Scales sparse-autoencoder feature extraction to a production-scale model. Finds many meaningful features for entities, concepts, contexts, and behaviors.

A concrete example of an emergent transformer circuit supporting in-context pattern copying and induction. Useful as a clear case where a real computation was reverse-engineered.

A cautionary paper: many safety-relevant behaviors are not simple token-aligned circuits. It argues for representation-level interpretability alongside circuit-level work.