Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

Excerpt

arXiv:2609.39338v1 Announce Type: new Abstract: Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not only confidence in the correct class, but also relations among incorrect alternatives. Yet closer imitation does not necessarily yield a better student. A student may already distinguish the correct class more sharply than its teacher, so further imitation can require giving back discrimination it h