A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic

Excerpt

arXiv:2610.07990v1 Announce Type: new Abstract: Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spa