mini-AGI: Continual-learning dynamically looped transformer with evolutionary grown (on a laptop)

Excerpt

Saw this today and found it very intriguing. Lots of interesting design choices here, and it's cool to see someone doing something different. Here's a few highlights: Looped transformer: dynamic recurrent depth on a per-token basis, up to 24 cycles Self-supervised learning: trains itself on new material constantly Weights stored on SSD and paged in on-demand Mixture of Experts: 8 active, 32 routed held in VRAM, smart caching of 96 more Dynamic size: builds new experts and increases parameter cou