Engram gone wild! 2b model update...

Excerpt

Update from : https://old.reddit.com/r/LocalLLaMA/comments/1wis23s/update_small_model_engram/ It's been about a week so I'm back. People were asking me about the model. People wanted code, or models, etc. Most of that is useless to you right now because you're not going to use an under trained model. So let's get to the details. The spec locked to the following after a LOT of testing : 2.6b model all up. Embedding, LM head, AttnRes, etc. 2.2b are trained. Embedding / LM Head are frozen (~205m ea