Data and parameter scaling laws for neural machine translation
Proceedings of the 2021 Conference on Empirical Methods in Natural …, 2021•aclanthology.org
We observe that the development cross-entropy loss of supervised neural machine
translation models scales like a power law with the amount of training data and the number
of non-embedding parameters in the model. We discuss some practical implications of these
results, such as predicting BLEU achieved by large scale models and predicting the ROI of
labeling data in low-resource language pairs.
translation models scales like a power law with the amount of training data and the number
of non-embedding parameters in the model. We discuss some practical implications of these
results, such as predicting BLEU achieved by large scale models and predicting the ROI of
labeling data in low-resource language pairs.
Abstract
We observe that the development cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.
aclanthology.org
Showing the best result for this search. See all results