Training Data Size

The datasets used to train frontier large language models are massive, often consisting of trillions of tokens. For example, the dataset for Llama 3 was over 15 trillion tokens, sourced from a vast collection of publicly available text and code from the internet.