Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA

Excerpt

arXiv:2610.06621v1 Announce Type: cross Abstract: Spectral variants of low-rank adaptation (LoRA) choose both a subspace and which factor to freeze. We separate these choices by freezing the input factor A or output factor B on the top or bottom singular directions of pretrained weights, with learning rates selected separately. At rank 2, the same-band advantage of freezing A is larger than either within-factor band difference on all four task-model pairs with complete comparisons. Freezing B al