I analyzed 31,352 hourly LLM benchmark scores: within-day variation was 2.8 points, while between-day variation was 8.4 [P]

Excerpt

https://preview.redd.it/42s57e5oqamh1.png?width=1903&format=png&auto=webp&s=69958a72e22276534b3605d11f3e1721f76e59c9 Disclosure: I developed AIStupidLevel, the open-source system used to collect and analyze this data. Both the frontend and backend are MIT-licensed. Most LLM evaluations measure performance at a single point in time. I wanted to investigate a different question: How stable are the models behind production APIs over time, and how can sustained performance changes be separated from