TinyStories: How Small Can Language Models Be and Still Speak

The paper investigates the potential of small language models (SLMs) to generate coherent English text, addressing the question of whether such capabilities only emerge in large models with complex architectures. The authors introduce TinyStories, a synthetic dataset of short stories using simple vocabulary, designed to train and evaluate SLMs with fewer than 10 million parameters or simpler architectures, while still producing fluent and consistent narratives.

TinyStories is generated using GPT-3.5 and GPT-4, and the dataset is evaluated using a novel framework where GPT-4 grades the content as if it were student-written stories. This approach provides a multidimensional assessment of the models' capabilities, including grammar, creativity, and instruction-following, overcoming limitations of traditional benchmarks.

The study finds that SLMs trained on TinyStories can produce diverse and coherent stories, demonstrating reasoning capabilities and factual knowledge. The models are also more interpretable, with attention heads and neurons showing meaningful functions. The paper suggests that TinyStories can aid in developing LMs for low-resource domains and understanding language capabilities in LMs.

The authors explore the trade-offs between model size and training resources, finding evidence for a polynomial scaling law between model size and learning budget. They also investigate the impact of the number of attention heads on model performance, suggesting that increasing the number of heads improves performance across metrics.

The paper highlights the potential of TinyStories to facilitate research on language models, particularly in understanding the emergence of language capabilities and optimizing model architectures and hyperparameters. Future work could explore the creativity of models and their understanding of stories, as well as the applicability of refined datasets in practical settings.