QK-Wanda: Coupling Queries and Keys for Unstructured Pruning

Excerpt

arXiv:2610.01554v1 Announce Type: new Abstract: Wanda (Sun et al., 2024) prunes large language models by scoring weights independently within each linear projection, although queries and keys interact through dot products. We introduce QK-Wanda, which scores query and key weights by their individual deletion costs under an unmasked pre-RoPE reconstruction objective. It augments Wanda scores with information from the opposite projection (keys for query weights, and queries for key weights), allow