Spider Bench: comparing 9 vision models on 2,000 spider photos [P]

Excerpt

I tested nine vision models on the same 2,000 spider photos. The highest exact-species accuracy was 49.85%. The tasks, predictions, scoring code, and run settings are public. The setup: Data: 2,000 photos covering 671 species and subspecies, sampled from a filtered collection of research-grade iNaturalist observations. The species list came from a Polish checklist; the photos were taken worldwide. Task: Pick one of 20 scientific names. Each list contains the expected species, up to nine alternat