The authors personalize three open‑weight Mixture‑of‑Experts (MoE) large language models by further training them on a large‑scale personalization dataset. To make the resulting modifications interpretable, they train crosscoders on paired activation patterns from the personalized and baseline (non‑personalized) models, yielding a set of disentangled personalized latents. Mapping these latents back to the MoE architecture reveals which experts are activated for specific personalization features, taking advantage of the MoE’s sparse routing that sends each token to only a few specialized experts. Empirically, the personalized MoE LLMs achieve performance comparable to dense LLMs while activating only a fraction of their total parameters, and they exhibit lower polysemanticity than dense models with a similar active‑parameter budget. Within the same model family, the learned personalized latents share less information with the non‑personalized model’s latents, indicating greater specialization. Causal intervention experiments demonstrate that manipulating these latents alone can reproduce the personalized model’s behavior, and amplifying the experts associated with particular latents shifts the baseline model toward the personalized output, thereby exposing distinct routing pathways for different personalization attributes.
Read original
huggingface/daily-papers