How many labelers do you have? A closer look at gold-standard labels

Excerpt

arXiv:2206.12041v3 Announce Type: replace-cross Abstract: The construction of most supervised learning datasets revolves around collecting multiple labels for each instance, then aggregating the labels to form a type of "true" label. We question the wisdom of this pipeline by developing a (stylized) theoretical model of this process and analyzing its statistical consequences, showing how access to non-aggregated label information can make training well-calibrated models more feasible than it is