Experimenting with hypersurface-constrained dynamic weight updating [P]

Excerpt

Hey everyone! I want to share the results of a small language model architecture experiment I’ve been running as a side project. The ultimate goal is to test an approach to reduce the number of the model's training parameters, since the main bottleneck in training is VRAM. The core idea is very similar to Universal Transformer. Let's take just a single decoder block and iteratively pass the input through it L times in a loop. But instead of updating the input with the loop iteration depth inform