- Description:
A benchmark corpus to be used for measuring progress in statistical language modeling. This has almost one billion words in the training data.
Additional Documentation: Explore on Papers With Code
Homepage: http://www.statmt.org/lm-benchmark/
Source code:
tfds.datasets.lm1b.Builder
Versions:
1.1.0
(default): No release notes.
Download size:
1.67 GiB
Dataset size:
4.40 GiB
Auto-cached (documentation): No
Splits:
Split | Examples |
---|---|
'test' |
306,688 |
'train' |
30,301,028 |
- Feature structure:
FeaturesDict({
'text': Text(shape=(), dtype=string),
})
- Feature documentation:
Feature | Class | Shape | Dtype | Description |
---|---|---|---|---|
FeaturesDict | ||||
text | Text | string |
Supervised keys (See
as_supervised
doc):('text', 'text')
Figure (tfds.show_examples): Not supported.
Examples (tfds.as_dataframe):
- Citation:
@article{DBLP:journals/corr/ChelbaMSGBK13,
author = {Ciprian Chelba and
Tomas Mikolov and
Mike Schuster and
Qi Ge and
Thorsten Brants and
Phillipp Koehn},
title = {One Billion Word Benchmark for Measuring Progress in Statistical Language
Modeling},
journal = {CoRR},
volume = {abs/1312.3005},
year = {2013},
url = {http://arxiv.org/abs/1312.3005},
archivePrefix = {arXiv},
eprint = {1312.3005},
timestamp = {Mon, 13 Aug 2018 16:46:16 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/ChelbaMSGBK13},
bibsource = {dblp computer science bibliography, https://dblp.org}
}