NeurIPS 2026 · Representation analysis

StructLens: A Structural Lens for Language Models via Maximum Spanning Trees

Haruki Sakajo, Frederikus Hudi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Nara Institute of Science and Technology (NAIST)

The idea

See how tokens organize.

How does a language model organize the tokens in a sequence? StructLens turns residual-stream representations into trees, providing a structural view of token relationships within each layer and how they change across training.

StructLens pipeline: hidden states become inter-token similarity matrices, then maximum spanning trees.
Figure 1 · From hidden representations to a structural view.

The method

Representations → similarities → trees

At each layer, StructLens measures pairwise token similarity using reciprocal L2 distance. Under a forward constraint, each non-root token connects to its most similar preceding token. The resulting directed maximum spanning tree summarizes the geometry of token relationships.

A tree is a summary of representation geometry. Its edges do not establish causal connections or ground-truth linguistic dependencies.

01 / Across layers

Local organization peaks in the middle.

Connected tree fragments concentrate on contiguous token spans most strongly in middle layers. Shuffled-input comparisons indicate sensitivity to natural token order.

Contiguous subtree and token ratios across Llama 3.1 8B layers, comparing original and shuffled inputs.
Figure 4 · Local-span connectivity across Llama 3.1 8B layers.

02 / Across training

Small spans first. Larger spans later.

OLMo2 checkpoints reveal scale-dependent emergence: smaller local units become detectable earlier, while later training consolidates denser, larger-span structures in middle layers.

Stage 1 · 1,000 steps · checkpoint 1 of 8

Contiguous subtree ratioContiguous subtree ratio0.0000.0040.0080.0120.01608162432Layer depth (0 = input embeddings)Stage 1 · 1,000 stepsContiguous token ratioContiguous token ratio0.000.050.100.140.1908162432Layer depth (0 = input embeddings)Stage 1 · 1,000 steps
  1. Stage 1 · 1,000 steps (current)
  2. Stage 1 · 10,000 steps
  3. Stage 1 · 101,000 steps
  4. Stage 1 · 500,000 steps
  5. Stage 1 · 750,000 steps
  6. Stage 1 · 928,646 steps
  7. Stage 2 · 1,000 steps
  8. Final checkpoint

OLMo2 7B · MMLU · original token order · saved mean ratios. Checkpoints advance at equal time intervals; training intervals differ. Earlier curves remain faded. Axes stay fixed for each subtree size; final layer normalization is excluded.

View the published figure
Three-, four-, and five-node local-span connectivity across OLMo2 7B layers and training checkpoints.
Figure 5 · Local-span organization during OLMo2 7B pre-training.

03 / Between layers

Structural islands within the model.

Edge-edit similarity reveals groups of layers with similar token relationship trees. These structural comparisons offer a complementary perspective on layer similarity.

An exploratory application

Structure-aware pruning improves results in some tested settings. Results vary across models and tasks, leaving room for further work.

Edge-edit inter-layer similarity heatmap for Llama 3.1 8B on MMLU, with bright diagonal clusters.
Figure 6e · Edge-Edit similarity; brighter means more similar.

Build on this work

Citation

@misc{sakajo2026structlens,
      title={StructLens: A Structural Lens for Language Models via Maximum Spanning Trees}, 
      author={Haruki Sakajo and Frederikus Hudi and Yusuke Sakai and Hidetaka Kamigaito and Taro Watanabe},
      year={2026},
      eprint={2603.03328},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.03328}, 
}