The authors investigated whether part‑of‑speech (PoS) information is encoded in the latent space of Sparse AutoEncoders (SAEs) applied to language model representations. They treated PoS tags as a controlled linguistic probe and examined whether individual SAE latents correspond to specific categories or whether PoS structure emerges from distributed groups of features. Using SAE activations extracted from a transformer‑based language model, they trained classifiers to predict PoS labels and assessed recoverability across tags. Results show that PoS distinctions are highly recoverable from the SAE latent representations, yet the mapping is not one‑to‑one; no single latent uniquely predicts a PoS tag. Instead, each PoS category is supported by a compact, sparse set of latents, with the size and composition of these sets varying considerably across tags. The study further distinguishes between open and closed PoS classes, finding substantial differences in how each class is represented. These latent groups remain stable when evaluated on held‑out data, while also exhibiting overlap between semantically or syntactically related categories, indicating that related PoS tags share subsets of features. Overall, the work demonstrates that SAEs localise morpho‑syntactic information in a distributed, category‑dependent manner rather than through isolated, atomic grammatical features.

Read original