Today we are excited to introduce Alephn 1, our first fully open-weight language model it consists of 111M parameter trained on 2.2B tokens from our internal dataset 'KnowSpread-1', crawled using our proprietary crawler. Alephn 1 is trained compute-optimally, following the Chinchilla scaling law at approximately 20 tokens per parameter. It is not meant to be a replacement for the frontier Open/Closed-source models, its sole purpose is to show how data can affect a model's performance. Aleph 1 is set to be released in the coming days on Hugging Face, accompanied by 100 checkpoints.
We believe that better models are not made solely from better architectures, but also from the quality of the data they are trained on.
Alephn 1 has the exact same architecture, training dataset size, and hyperparameters as Cerebras-GPT 111M. This was done to eliminate most confounding variables in the data-quality analysis process, thus ensuring a fair comparison.
Our model was able to outperform Cerebras-GPT on 5 of the 7 benchmarks (Hella-Swag, Wino-Grande, Open-BookQA, ARC-c, ARC-e) reported in the Cerebras-GPT paper. In addition, Alephn 1 was also able to outperform Pythia-70M, a model trained on 299B tokens (approximately 135× the number of tokens we used) on 2 benchmarks (Wino-Grande, Open-BookQA)
Figure 1: Performance comparison of Alephn 1.