Bag of dims: Training-free mechanistic interpretability via dimension-level sign patterns

We show that the standard basis of transformer hidden states already provides a training-free, architecture-general feature basis. Individual dimensions encode semantic content via their signs (±1) and confidence via their magnitudes, functioning as independent binary registers. A feature is simply a subset of dimensions with a consistent sign pattern, readable by counting sign agreements with no learned rotation. We validate this Bag of Dims framework across seven models spanning language (Qwen 3.5-4B, Gemma 3-4B, Mistral 7B, Qwen3-32B), vision (DINOv2, ViT-Base), and audio (AST). Sign patterns alone carry predictive content: replacing all magnitudes with unity preserves 60–93% top-5 next-token accuracy through the LM head, and pure Hamming scoring with no decoder on either side reaches 80–90% top-4096. These patterns organize into semantic features: from a single-token cache (one forward pass per vocabulary token, no context, no labels), we detect 175 categories at AUC 0.97–0.99 by counting sign agreements. A trained probe adds only +0.018 AUC and converges to axis-aligned weights: the rotation that autoencoders and probes learn buys almost nothing. These features are causally operative, not merely readable: they survive the K and V attention projections, trace back to the FFN neuron coalitions that write them (random-weight controls never reproduce this), and are already present—axis-aligned and sign-readable at residual parity—in the FFN’s own activation space, surviving the nonlinear SwiGLU gate. Flipping a feature’s sign pattern during the live forward pass suppresses its concept across four language models, magnitude-matched and concept-specific. Dimensions stay independent throughout (pairwise MI < 0.006 bits). Given only random seeds and no labels, discovery scales to 1500 features per language model. And the structure is not specific to language: the same per-dimension signs appear in self-supervised vision (DINOv2, 9/12 ImageNet superclasses), supervised vision (ViT-Base, 11/12), and aud