Breaking the Curse of Multilinguality with Cross-lingual
Expert Language Models

As a preprocessing step, we partition the multilingual corpus into $k$ clusters to train each x-elm. We consider learning TF-IDF clusters as well as a new clustering method that groups documents by language identity and linguistic typology (§3.2).

A preliminary stage of shared, dense pretraining is important for ensembling expert language models Li et al. (2022). Therefore, the first step of BTM is to initialize (branch) each expert with the parameters from a partially trained model. For this work, we initialize our x-elms with an existing multilingual pretrained model, XGLM Lin et al. (2022).

After initialization, we assign each expert a data cluster and train for a fixed number of steps with an autoregressive LM objective. Expert training is independent, with no shared parameters between models.

We collect the $k$ x-elms into a set and perform inference with them. We consider several methods of inference and expert ensembling in §3.3.

We partition the multilingual corpus automatically into $k$ components with k-means clustering. First, we encode each document into a word-level TF-IDF representation²²2Data tokenization is independent of the downstream model. Here, we use the sklearn text-vectorizer tokenizer.; we then perform balanced k-means clustering on these representations to obtain approximately balanced subsets of the data on which to train each x-elm. Further details on the balanced k-means clustering method can be found in Gururangan et al. (2023). This allocation method uses no language information outside of what is inherent in the text (e.g., script, vocabulary).

We also consider segmenting the corpus by language identity.³³3This requires knowledge of the language of each document. We use the language tags provided with mC4. Rather than balancing the amount of data allocated to each cluster in this setting, we instead keep the number of languages per cluster fixed. Specifically, we learn a balanced hierarchical clustering of the languages (Figure 2). We build this hierarchy using the language similarity metrics in lang2vec Littell et al. (2017), which represents languages based on linguistic features in resources such as WALS⁴⁴4World Atlas of Language Structures, https://wals.info/ and estimates language similarity with distance in this feature space. We first initialize each cluster with a single language; at each step, we merge each cluster with exactly one other based on the minimum distances between the cluster centroids. We then group languages according to the resulting hierarchy and the desired number of experts. When the number of languages equals the number of experts, typological clustering results in monolingual training, where every language is assigned a separate expert.

Figure 3 shows the difference in language distributions between the TF-IDF and Linguistic Typology clusters. While TF-IDF allows language data to spread across experts, we find that, in practice, the distributions remain relatively sparse. The main exception is at $k=16$, when the highest-resourced languages in the data (e.g., English or Russian) are split across clusters due to the constraint that balances the amount of data per cluster.

This method performs inference with a single expert chosen prior to evaluation; therefore, it incurs the same inference cost as the dense baselines. When evaluating the Typology experts on a particular language $\ell$, we choose the expert that included $\ell$ in the set of languages on which they continued pretraining. Similarly, when evaluating TF-IDF, we choose the x-elm trained on the highest percentage of $\ell$’s data.

We also consider ensembling TF-IDF experts by adapting the c-BTM ensemble routing method. Here, we calculate ensembling alphas, or weights, over these experts for each evaluation step based on the proceeding context’s TF-IDF distance from the experts’ k-means centroids. These weights are then used to ensemble the output probabilities from each expert.

More specifically, given a probability from each expert LM $p_{e}(x_{t}|x_{<t})$ and the corresponding ensemble weight $\alpha_{e}=p(e|x_{<t})\propto\textrm{exp}(-\textrm{dist}(x_{<t},c_{e})^{2}/T)$, the probability of the ensemble $p_{E}(x_{t}|x_{<t})=\sum_{e\in E}\alpha_{e}\cdot p_{e}(x_{t}|x_{<t})$. Here, $\textrm{dist}(x_{<t},c_{e})$ is obtained by embedding $x_{<t}$ with the learned TF-IDF vectorizer and calculating the Euclidean distance from $c_{e}$ (the centroid over the data representations allocated to expert $e$), and $T$ is a temperature parameter over the ensemble weight distribution. Further details and motivation for this setting are provided in Gururangan et al. (2023).

Ensembling x-elm outputs increases the cost of inference relative to the dense model or top-1 inference. However, it can potentially better fit different subsets of data in a diverse evaluation set. We also do not assume we know the identity of each example when ensembling, which makes this approach more flexible than the top-1 setting. In most cases, we ensemble all $k$ experts; however, we can also reduce computational costs by sparsifying the ensemble weights and only activating the $m$ ($<k$) experts that most contribute to an example: $p_{E}(x_{t}|x_{<t})=\sum_{e\in E}\alpha_{e}\cdot p_{e}(x_{t}|x_{<t}):\alpha_{e}\in\textrm{top-m}(\bf{\alpha}_{E})$. Table 8 presents the performance tradeoff with sparser TF-IDF ensembles.

For most languages (10 of 16), typologically clustered experts are the best-performing setting. For some high-resource languages (EN and ES), ensembling the TF-IDF experts works better than a single expert. However, this inference setting requires more parameters, as it uses all x-elms instead of just the single best expert per language. Furthermore, training x-elms for longer unsurprisingly outperforms lower compute settings. All of our experimental settings outperform the seed XGLM model; similarly, the experiments with the 21.0B token compute budget perform better than the respective experiment trained with 10.5B tokens.

We also show that multilingual language modeling with x-elms does not disproportionally benefit languages with more pretraining data (Figure 5). Instead, perplexity improvements over both the seed LM and the dense LM baseline may slightly favor low-resource languages ($\rho=-0.19,-0.26$, respectively).

We evaluate the existing dense baseline and ensembled TF-IDF clustered experts from the 21B token compute budget (§6.2) to test whether continued pretraining with x-BTM improves performance on unseen languages (x-elm Training). We also compare these results to XGLM. We note these models never trained on the target languages.

Table 2 presents the unseen target language perplexities in the XGLM and x-elm Training columns. We find that the original XGLM model performs poorly on the new languages, particularly those less related to XGLM’s highest-resourced ones (i.e., AZ and HE). While these perplexities remain high in the dense model and TF-IDF ensembles, training (on other languages) with x-BTM provides some performance improvements over the seed model.

We now consider how well Hierarchical Multi-Round training (HMR) works for language adaptive pretraining (LAPT, Chau et al., 2020), which incorporates new target languages into the continued pretraining process. Here, we group each target language with a higher-resource donor language already in our pretraining set; these are assigned with the language similarity metric used for typological clustering. We seed each new language’s expert with an expert specialized to that language’s donor; the new expert is then trained on the donor/target language pair. For HMR inference, we evaluate perplexity with the expert trained on that target language.⁹⁹9We also evaluate the donor languages to see what benefit, if any, they receive from the adaptation process.

We compare HMR against jointly continuing training on all four new languages and their respective donors in a single model (Dense). Each setting builds on models from the 10.5B compute budget: we continue training on the dense baseline for dense LAPT and branch from the donor languages’ $k$=8 typological experts for HMR training.

All of the LAPT settings provide considerable improvements on the new target languages over the unseen language experiments (Table 2, LAPT columns). The HMR setting outperforms continued dense training on every new language. Furthermore, HMR training removes the risk of catastrophic forgetting Yogatama et al. (2019) in other LAPT schemes, as this process adds new experts to x-elm rather than changing existing ones.

We also find that this setting provides performance gains on two donor languages over the experiments in §6.2. This is likely due to further training with more closely related languages for these languages (e.g., performing training on Arabic with Hebrew rather than French), consequently providing a more informative training signal for the higher-resource donor language as well.

We compare the perplexity of each expert model on all pretraining languages to that of the seed model, XGLM-1.7B (Figure 6 for $k=8$; other settings given in Appendix C). Across the considered values of $k$, we see less forgetting in the x-elms trained on TF-IDF clusters than in those clustered typologically. For the $k=8$ expert setting, the TF-IDF experts only forget on 47.7% of settings, and when forgetting occurs, the perplexity increase over the baseline is 3.10 on average. For typologically clustered experts, these measures are 83.6% and 3.14, respectively; we observe similar trends for the $k=4$ and $k=16$ x-elms. This implies that though in some cases only small quantities of data are shared across TF-IDF clusters, these data mitigate forgetting over the hard cluster assignments made by typological clustering.

For example, English is rarely forgotten, with only 25% of experts performing worse than the baseline. In comparison, 94.6% of experts perform worse on Urdu than XGLM. One potential cause of this discrepancy is the frequency with which the language was seen during seed training: languages that are more common in the XGLM pretraining corpus see fewer cases of forgetting and have smaller perplexity increases when it does occur (Figure 7). Another likely factor is inaccurate language classification in the BTM training data, which is a common issue when training language models on specific languages Blevins and Zettlemoyer (2022); this could lead to related, higher-resourced languages contaminating the datasets for lower-resourced ones Kreutzer et al. (2022).