One-sentence Summary

UCLA researchers introduce the Psychological Depth Scale (PDS), a literary-theory-based framework that measures how well LLM-generated stories provoke emotion and empathy, and demonstrate through a Mixture-of-Personas prompting strategy for automated evaluation that GPT-4 stories match or surpass highly-rated human-written stories on this dimension.

Key Contributions

• The paper introduces the Psychological Depth Scale (PDS), a framework grounded in literary theory for evaluating stories through readers' responses, including empathy, engagement, emotional provocation, authenticity, and narrative complexity. Human evaluation of PDS is validated with an average Krippendorff's alpha of 0.72.

• Automated evaluation of PDS is developed using a Mixture-of-Personas prompting strategy and constrained decoding. GPT-4o achieves an average Spearman correlation of 0.51 with human judgments, while Llama-3-70B with constrained decoding reaches 0.68 for empathy.

• PDS is used to compare LLM-generated and human-authored stories, showing that GPT-4 stories surpass or are statistically indistinguishable from highly rated human-written Reddit stories. These results suggest that, for shorter texts with richer premises and prompting, LLMs can produce psychologically deep narratives comparable to skilled human authors.

Introduction

As large language models are increasingly used for story generation and narrative design, it matters not only that their text is fluent and stylistically acceptable, but also that it can resonate with human readers. Earlier evaluations of LLM-generated stories often focus on objective features such as structure, style, toxicity, or creativity, while studies of reader experience tend to address only narrow aspects such as empathy or engagement and may rely on expert raters or simple writing prompts. The authors leverage reader-response criticism and text world theory to introduce the Psychological Depth Scale, a framework that captures five reader-centered dimensions: empathy, engagement, emotion provocation, authenticity, and narrative complexity.

Dataset

The authors describe PSYCHDEPTH, an English short-story dataset built from premise-response pairs with authorship metadata.

• Dataset composition and scale: The full dataset contains 495 short stories, averaging about 450 words each. It includes 45 human-written stories and 450 LLM-generated stories. Each item pairs a prompt premise with a story response and metadata describing authorship and generation conditions.

• Human story source and filtering: Human stories are collected from Reddit’s r/WritingPrompts, an online writing community where users respond to prompt premises. The authors group stories by community ranking into Human-Advanced, Human-Intermediate, and Human-Novice, with average upvote counts of 1,434, 263, and 9. They also select human stories posted within 24 hours of one another to reduce nonquality influences on upvotes.

• LLM story source and metadata construction: LLM stories are generated using five model settings: Llama-2 7B, Llama-2 13B, Llama-2 70B, Vicuna-33B, and GPT-4. Vicuna-33B is included to fill the size gap between Llama-2 13B and 70B. The authors use two prompting strategies, WRITERPROFILE and PLAN+WRITE, and sample three generations per setting. WRITERPROFILE prepends a seasoned writer profile to the prompt, while PLAN+WRITE first generates character portraits and then composes the story. These generation choices are recorded as metadata to support comparison across model size, prompting method, and generation variance.

• Data use in the paper: The full dataset is used to analyze psychological depth in creative writing. The authors do not describe a training split or model fine-tuning mixture; instead, the data supports human evaluation.

• Study subset and processing: For the human study, the authors create a smaller 97-story subset using stratified sampling balanced across prompt premises, authorship type, and generation strategies. Five trained annotators evaluate every story in this subset. The annotation process produces 2,425 psychological depth ratings, 485 authorship likelihood ratings, and 1,128 free-form justifications.

• Annotation schema: Annotators rate five psychological depth components on 1 to 5 Likert scales and also rate the likelihood of human authorship from 1, indicating LLM authorship, to 5, indicating human authorship. Optional free-text justifications are also collected. No cropping strategy is involved because the dataset is text based.

Method

The authors leverage a method that couples an assessment framework with a controlled narrative generation protocol. At the core is the Psychological Depth Scale, a reader-centered instrument for evaluating human and machine authored stories. Instead of asking only what a sentence means, the scale asks what a narrative does to a reader. It also treats comprehension as the dynamic construction of mental text worlds, meaning that story quality is assessed by how a narrative shapes the reader's cognitive and emotional experience during reading. The final metric set is produced by reducing a broad literature-derived candidate space through thematic analysis, yielding five reader-observable dimensions: emotion provocation, empathy, engagement, authenticity, and narrative complexity.

The first metric, emotion provocation, evaluates whether a story reliably induces strong affective states with positive or negative valence. The method emphasizes that emotional impact is not simply proportional to the amount or intensity of emotional language. A more effective narrative should align valence and arousal, because emotionally congruent expressions are treated as more cognitively effective than conflicting combinations. The empathy metric evaluates whether readers temporarily adopt a character's perspective. This involves immersive identification, cognitive participation in narrated experiences, and perspective taking. Stories can support empathy through suggestive structures, including metaphor and metonymy, that connect character experience to shared human themes. The engagement metric captures attentional transportation, or the degree to which a story maintains focus, sustains interest, and reduces awareness of distractions. Engagement is treated as both a predictor and an outcome of other narrative achievements, since a deeply affecting story tends to keep readers absorbed while absorbed readers are more available to emotional and perspectival responses.

The authenticity metric assesses whether a narrative expresses recognizable, genuine aspects of human experience. The method treats authenticity as a resonance mechanism, because readers can feel their way into depictions even when those depictions involve material or mental differences. This metric is positioned as contributing to belonging, interest, and persuasive impact. The narrative complexity metric evaluates whether the story presents intricate plotting, layered character development, and structures that invite puzzled attention, such as nonlinear narration, plot twists, and double perspectivation. Complexity is defined as narrative depth rather than narrational difficulty. A simple story may be narrated in an obscure style, and a complex story may remain stylistically clear. Complexity is also treated as dynamic, because its perceived depth depends on context and the reader's situated experience.

For model-generated stories, the authors implement a controlled generation pipeline designed to isolate the influence of model scale while holding prompts and sampling design fixed. They restrict the architecture family to comparable open Llama models at 7B, 13B, and 70B parameter sizes, add a 33B fine-tuned Llama variant to fill the size gap, and include GPT-4 as a high-capability reference model. The protocol uses five models, two prompting strategies, and three sampled generations per condition, creating comparable narrative outputs for psychological depth evaluation. Quality control is applied to the generated stories to remove malformed or undesired samples.

The prompting module includes two strategies. The WriterProfile strategy prepends a persona description of a seasoned author whose work is known for psychologically deep and engaging storytelling. This profile primes the model to prioritize complex psychological states and stronger emotional expression before the story prompt is processed. The Plan+Write strategy decomposes short story creation into two phases. In the first phase, the model generates character portraits that expand the initial premise with main character details, including emotional states and inner thoughts. In the second phase, the model composes the final story by integrating the premise with those character portraits. The pipeline avoids adding further planning phases for setting, plot, or outline because additional stages were found to reduce coherence and consistency in short-form output. This design gives the system a lightweight but structured reasoning path, where character psychology is planned before composition.

Experiment

The experiments use a human annotation study to validate the Psychological Depth Scale, then apply it to LLM-based evaluation and comparisons of human- and LLM-written stories. Human annotators show substantial agreement across the scale’s components, supporting its coherence, while LLM judges align more closely with human judgments when prompted through multiple personas, though no single model performs best on every component. The story comparison suggests that advanced LLM output can rival or exceed highly rated human writing on several depth dimensions, even though authorship remains difficult to distinguish and model size alone does not guarantee psychological depth.

Rater agreement is consistently high across psychological depth components for all model judges, suggesting that MoP prompting produces stable and coherent judgments. Human raters show more moderate but meaningful agreement, supporting the framework as a practical way to evaluate short stories. The results indicate that persona variation introduces some diversity of opinion while still improving alignment with human judgments. Model ratings show high agreement across all psychological depth components under MoP prompting. Human ratings show noticeable but lower agreement than LLM persona judgments. MoP prompting increases diversity of opinion while improving correlation with human judgments.

Across the evaluated judge models, Mixture-of-Personas prompting generally increased the average zero-shot correlation with human psychological-depth judgments relative to the vanilla zero-shot baseline. The gain was not uniform across all depth components, with some components showing smaller or mixed changes depending on the judge model. Larger and proprietary judges tended to have higher baseline agreement, while the smallest judge showed a pronounced improvement after persona prompting. Mixture-of-Personas prompting increased average agreement with human judgments compared with the zero-shot baseline. Larger and proprietary judge models generally showed higher baseline correlation than the smallest judge model. The benefit of persona prompting varied across components, with engagement showing less consistent improvement.

GPT-4 generally receives higher average psychological-depth ratings than other LLMs and several human groups, with notable advantages in empathy and narrative complexity. Human stories are judged more human-like on average, especially the intermediate group, while GPT-4 remains the most human-like LLM and is perceived close to human performance. Model size does not reliably predict depth, because smaller Llama variants can score similarly to novice humans and remain below GPT-4. GPT-4 obtains the highest average ratings on most psychological depth components while also having relatively low standard deviations. Human authors are rated higher on perceived human authorship than LLM authors, with the intermediate human group rated most human-like. Smaller Llama models score closer to novice human writers than to GPT-4, weakening any simple size-depth relationship.

The experiments evaluate short-story psychological-depth judgments by comparing human raters with multiple model judges, testing whether Mixture-of-Personas prompting yields reliable, coherent, and human-aligned assessments. MoP prompting generally improves stability and correlation with human judgments while allowing more diverse opinions, although the benefit varies across depth components and judge models. Larger proprietary judges tend to agree more strongly with humans, GPT-4 receives the highest depth ratings, human stories are judged more human-like, and model size alone does not reliably predict psychological depth.