StructLens: A Structural Lens for Language Models via Maximum Spanning Trees
Nara Institute of Science and Technology (NAIST)
The idea
See how tokens organize.
How does a language model organize the tokens in a sequence? StructLens turns residual-stream representations into trees, providing a structural view of token relationships within each layer and how they change across training.
The method
Representations → similarities → trees
At each layer, StructLens measures pairwise token similarity using reciprocal L2 distance. Under a forward constraint, each non-root token connects to its most similar preceding token. The resulting directed maximum spanning tree summarizes the geometry of token relationships.
01 / Across layers
Local organization peaks in the middle.
Connected tree fragments concentrate on contiguous token spans most strongly in middle layers. Shuffled-input comparisons indicate sensitivity to natural token order.
02 / Across training
Small spans first. Larger spans later.
OLMo2 checkpoints reveal scale-dependent emergence: smaller local units become detectable earlier, while later training consolidates denser, larger-span structures in middle layers.
Stage 1 · 1,000 steps · checkpoint 1 of 8
- Stage 1 · 1,000 steps (current)
- Stage 1 · 10,000 steps
- Stage 1 · 101,000 steps
- Stage 1 · 500,000 steps
- Stage 1 · 750,000 steps
- Stage 1 · 928,646 steps
- Stage 2 · 1,000 steps
- Final checkpoint
OLMo2 7B · MMLU · original token order · saved mean ratios. Checkpoints advance at equal time intervals; training intervals differ. Earlier curves remain faded. Axes stay fixed for each subtree size; final layer normalization is excluded.
View the published figure
03 / Between layers
Structural islands within the model.
Edge-edit similarity reveals groups of layers with similar token relationship trees. These structural comparisons offer a complementary perspective on layer similarity.
An exploratory application
Structure-aware pruning improves results in some tested settings. Results vary across models and tasks, leaving room for further work.
Build on this work
Citation
@misc{sakajo2026structlens,
title={StructLens: A Structural Lens for Language Models via Maximum Spanning Trees},
author={Haruki Sakajo and Frederikus Hudi and Yusuke Sakai and Hidetaka Kamigaito and Taro Watanabe},
year={2026},
eprint={2603.03328},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2603.03328},
}