An AI Wattpad platform called Narrator has been built to assess LLMs on their fiction-writing capabilities. The project addresses the challenge of evaluating AI-generated stories, moving beyond traditional benchmarks. It uses real reader engagement metrics to rank models. The goal is to provide a more accurate evaluation of the models' ability to create compelling narratives.

Narrator's architecture employs a persistent agent loop to maintain context across chapters. Before generating, the agent accesses character sheets, plot outlines, and world-building notes. This approach significantly improves consistency in long-form fiction. The platform also features story forking, enabling readers to alter narrative paths, and a visual LitRPG interface.

The platform's granular filtering allows for specific genre comparisons, such as which model excels in Spanish comedy. The project aims to gather more reader engagement data and explore improved methods for maintaining narrative consistency. The creator is also interested in how other developers are tackling similar challenges.

This project is important because current LLM evaluation methods often fall short in assessing creative writing. The focus on reader engagement provides a more realistic measure of a model's ability to produce enjoyable fiction. As LLMs become more integrated in content creation, tools like Narrator will become crucial.