Aurora-80K releases! A modern tiny language model.

Excerpt

I'm introducing Aurora-80K, a small language model with exactly 80 thousand parameters. It uses a factorized 4,096-token vocabulary despite having only 80K parameters. The benchmarks: Wikitext-2 BPB: 3.2902 BLiMP: 52.31% Arc-Easy: 26.05% More information about the model is available on the model page on Huggingface. if there's any questions I'll happily answer them! submitted by /u/Tall_Abrocoma_3533 [link] [comments]