How Do Large Language Models Learn Concepts During Continual Pre-Training?

Excerpt

arXiv:2601.03570v2 Announce Type: replace Abstract: Human beings primarily understand the world through concepts (e.g., dog), abstract mental representations that structure perception, reasoning, and learning. However, how large language models (LLMs) acquire, retain, and forget such concepts during continual pretraining remains poorly understood. In this work, we study how individual concepts are acquired and forgotten, as well as how multiple concepts interact through interference and synergy.