No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse

Excerpt

arXiv:2610.01493v1 Announce Type: cross Abstract: Iterative fine-tuning on synthetic data causes \emph{model collapse}: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator $h_k$, co