CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

Excerpt

arXiv:2610.02039v1 Announce Type: new Abstract: Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization.