We benchmarked 24 LLMs against human writers on 475 creative writing prompts

Excerpt

We just released the first version of our Creative Writing benchmark, comparing 24 LLMs against human writers across 475 writing prompts. Creative writing is subjective, so the rankings aren't meant to predict what any one person will prefer. Instead, they predict what a large group of readers would prefer, using a custom reward model trained specifically on human preferences for creative writing. Surprisingly, the strongest frontier models already rank above the talented amateur writer cohort,