Evaluating Decision Models for Text Annotation in Computational Social Science

Excerpt

arXiv:2609.24574v1 Announce Type: new Abstract: Computational social science increasingly relies on large language models for text annotation, and the validity of published findings now rests on the labels generated by such models. Decision models, a new model class built for categorical question answering, answer typed questions with a choice, a probability distribution over the label set, and a confidence score rather than free text, at a small fraction of frontier inference prices. Whether th