Function Over Form: Distributional Orthogonalization in Mixture-of-Experts with Replica Expert Mechanism

The scaling of LLMs increasingly relies on MoE architectures to decouple active computation from total parameter count. However, the efficacy of MoE is often constrained by expert collapse and representation redundancy, both leading to underutilization of model capacity. To address these challenges,