One-sentence Summary

The University of Melbourne introduces Conversational Information Gain (CIG), a framework that extracts atomic claims into an evolving semantic memory to score each utterance along Novelty, Relevance, and Implication Scope, showing that memory-derived dynamics correlate more strongly with human-perceived information gain in deliberative dialogues than traditional heuristics such as TF–IDF.

Key Contributions

• The paper introduces Conversational Information Gain, a framework for evaluating how each utterance advances collective understanding in deliberation, operationalized through an evolving semantic memory of atomic claims updated by LLM extraction and NLI-guided consolidation and scored for Novelty, Relevance, and Implication Scope.

• The paper contributes an annotation study of 80 segments from INSQ debates and FORA community discussions, demonstrating moderate-to-high reliability for CIG ratings and showing that memory-derived signals such as claim-update counts correlate more strongly with human-perceived informativeness than length or TF-IDF heuristics.

• The paper presents LLM-based CIG predictors, with GPT-5 reproducing aggregate human judgments under matched information conditions, and provides diagnostic moderation analysis showing immediate informational surges in INSQ debates and gradual accumulation across turns in FORA community discussions.

Introduction

Public deliberation in community forums, moderated discussions, and debates matters because democratic decision-making depends on exchanges that move participants toward shared understanding. Prior evaluation methods often focus on structural formality, civility, or shallow argument features, but these surface-level signals can reward polite or bureaucratic talk while missing substantive progress. The authors introduce Conversational Information Gain, a metric that measures how much an utterance advances a shared knowledge state, decomposing it into novelty, relevance, and implication scope. They operationalize this through a lightweight semantic memory of consolidated claims and show that both human annotators and LLMs can assess CIG with reasonable reliability, with memory-based signals tracking perceived progress more closely than common heuristics.

Dataset

• The authors use annotated transcripts from two deliberative discussion settings: ten TV debates from INTELLIGENCE SQUARED (INSQ) and ten community discussions from the FORA corpus about Durham’s community future vision.

• They manually curated the dataset by excluding topics that are highly sensitive or recently polarizing, in order to reduce annotator bias and potential harms.

• Both corpora contain moderator-led, multi-party discussions, but they differ in intent and participation style. INSQ debates are competitive motion-based events with domain experts and tighter moderation, while FORA assemblies are collaborative brainstorming meetings where facilitators often participate as peers.

• The authors use Table 2 statistics to show speaker participation differences: INSQ has more even participation, indicated by lower speaker Gini values, while FORA has more unequal participation due to a freer community discussion style.

• Both datasets follow a comparable introduction, discussion, and conclusion structure. For INSQ, the authors use the predefined structure and analyze only the discussion phase. For FORA, they use GPT-5 to detect phase boundaries and then manually verify them.

• Conclusions are discarded, while introductions are used to initialize the initial knowledge used by the memory modules.

• The discussion phase of each episode is segmented into subtopical units using GPT-5 with a shifting window and majority vote.

• For each segment, the authors construct a prior memory summary by retrieving relevant existing memories through semantic similarity and summarizing them with GPT-5. The summarization prompt was developed using pilot data annotated by one paper author.

• To focus annotation on substantive participant contributions, moderator turns and very short utterances of fewer than five words are skipped. These skipped turns account for 9–18% of tokens, depending on the setting.

• From each episode, four segments are selected for annotation. Selection balances estimated reading time, skip ratio, and segmentation confidence.

• Annotation was performed by 88 Prolific annotators who were native English speakers with at least a bachelor’s degree. Ten annotators were removed because of consistently low agreement with peers.

• Each assignment included a tutorial, a prescreen test, and four sampled segments from the same session presented in chronological order.

• The authors use two annotation variants over the same material. One variant collects a single contribution-in-group score per utterance on a 1 to 4 scale. The other collects separate ratings for Novelty, Relevance, and Implication Scope, also on 1 to 4 scales.

• Annotation metadata includes the discussion topic, a long-term memory summary built from prior memories, three preceding utterances as short-term context, and the target utterance to be rated.

• Inter-annotator agreement is reported as moderate to high across settings and annotation aspects.

Method

The authors design an evaluation pipeline for Conversational Information Gain, or CIG, by combining a multidimensional rubric with an incrementally updated semantic memory. The method treats dialogue not as a sequence of isolated utterances, but as a process that either expands or narrows the group’s shared knowledge. Refer to the framework diagram.

The framework begins with the CIG construct itself. The authors define CIG as the extent to which a response advances the group’s understanding of the topic or moves the discussion toward its goal, relative to prior knowledge and the preceding dialogue. To make this construct measurable, they decompose it into three aspects: Novelty, Relevance, and Implication Scope. Novelty asks whether an utterance introduces information that changes or extends the existing conceptual framing, rather than merely differing in lexical form from previous turns. Relevance asks whether the utterance is substantively connected to the main topic or goal, including whether it requires bridging inference or refers to a recognized subtopic. Implication Scope measures how broadly a contribution generalizes beyond the immediate speaker, case, or procedural exchange. In practice, a local procedural remark has narrow scope, a case-specific statement remains tied to one situation, inductive generalization lifts evidence into a broader pattern, and principle-level claims project universal applicability.

All aspects are rated on four levels. This design allows the method to separate informational contribution from surface conversational quality. For example, a short coordination move such as organizing the next turn may be helpful for dialogue flow, but it does not necessarily raise CIG because it does not expand the group’s shared understanding.

The core methodological contribution is the use of a dynamic semantic memory to represent what the conversation has already established. The authors maintain this memory through two modules, Extraction and Consolidation. In the Extraction stage, an LLM decomposes each utterance into atomic claims. These claims are written to be self-contained, meaning the extractor resolves references and removes context-dependent ambiguity. Compound statements are split into separate propositions, while hedging and filler material are normalized away to recover the underlying claim content. For example, an indirectly stated concern may be converted into a direct factual proposition that can later be compared with other claims. At the same time, the memory preserves speaker attribution, so the system knows which claim was introduced by which participant. This is important because public deliberation often involves parallel or conflicting framings, and those distinctions should not be erased by a naive merging step.

In the Consolidation stage, the system compares a newly extracted claim against previously stored memory items. Let the new claim be $A$A. The system first retrieves the top $k$k semantically similar memory items $\{B_i\}_{i=1}^{k}${Bi​}i=1k​ using vector search. An LLM-based natural language inference judge then evaluates each pair bidirectionally and assigns one of five relations: equivalent, forward entailment, backward entailment, contradiction, or neutral. This inference layer gives the memory a logical structure that pure vector retrieval alone would not provide.

The memory update policy is deterministic. The authors restrict comparisons to claims produced by the same speaker, using the subset

The action assigned to the new claim is

If $S(A)$S(A) is empty or contains only neutral relations, the action is ADD, and the claim is inserted as a new memory entry. This allows the memory to preserve parallel framings and inter-speaker disagreements. If $S(A)$S(A) is not empty, equivalence or backward entailment triggers NONE, meaning the existing memory already captures the information. Contradiction or forward entailment triggers UPDATE on the most similar prior claim $B^\star$B⋆, where

A contradiction replaces the outdated claim, while forward entailment merges the prior claim and the new claim into a more specific consolidated representation. This design makes the memory both semantically dense and speaker sensitive.

For scoring, the LLM judge does not evaluate an utterance from scratch. Instead, it receives a compact prior context constructed from the semantic memory. During preprocessing, discussion phases are segmented into sub-topical units, introductions are used to initialize background knowledge, and retrieved memory items are summarized into prior memory summaries. When a target segment is evaluated, the model receives the topic, a short window of recent utterances, the memory-based prior summary, and exemplar demonstrations. The model then outputs CIG and aspect ratings. This matched-context setup is central to the method: the model is asked to reason over the same evolving knowledge state that human annotators would need to judge what counts as new, relevant, or publicly consequential.

The authors also compare this memory-based representation with alternative prior contexts. Providing the full preceding transcript or the verbatim retrieved memory items produces predictions close to those based on summarized memory, while providing only the last few utterances or no prior context leads to larger deviations. This indicates that the memory summary is not merely a compression artifact; it preserves the information most relevant for judging conversational information gain.

The final stage links claim-level analysis to utterance-level impressions. After claims are extracted, the model can assign Novelty, Relevance, and Implication Scope scores to each claim. These claim-level scores are then aggregated to predict the CIG of the full utterance. The authors test multiple aggregation operators and find that the most robust predictions come from selective pooling over a small number of salient claims, followed by a conjunctive combination across aspects. This pattern implies that an utterance is not judged as informative merely because it contains many average claims. Instead, perceived informativeness appears to depend on a few strong claims while still requiring that no essential aspect be critically weak.

Experiment

The experiments evaluate whether the proposed CIG dimensions, automated prediction, and heuristic memory signals can explain and estimate perceived informativeness, using ordinal models, GPT-5 annotations, correlation analyses, and claim-level aggregation tests. They show that novelty and relevance capture most of the human CIG judgment while implication scope adds little, and that GPT-5 can reproduce aggregate human ratings when given comparable prior context, with memory-derived dynamics and summaries serving as strong, faithful proxies. A case study on moderator dynamics further shows that CIG can reveal how interaction styles shape information gain, producing immediate peaks in adversarial settings and more distributed gains in collaborative discussions.

The rubric defines Conversational Information Gain as the extent to which a contribution advances a group's shared understanding, using prior knowledge and preceding dialogue as context. It separates low-value repetitions from minimal clarification, incremental elaboration within an existing frame, and insightful reframing that changes the discussion direction. The related evaluation context suggests that CIG can be annotated and approximated at scale, with memory-based signals aligning strongly with human judgments and model estimates near human consensus. CIG focuses on informational advancement rather than general conversational quality Higher levels distinguish adding details from introducing new framings Novelty is rated independently of topical relevance and impact size Memory-derived context helps produce CIG estimates close to human ratings

The annotated FORA and INSQ sessions are broadly similar in conversation size, with comparable numbers of utterances per session, words per utterance, and segment-level utterance counts. INSQ sessions contain more speakers on average than FORA sessions, reflecting a more multi-party debate format, while the four annotated segments in each setting have similar numbers of speakers. Segment-level participation measures exclude skipped moderator turns and very short fragments, which account for a noticeable but limited portion of tokens. Both settings have broadly comparable utterance counts and segment lengths. INSQ sessions show higher speaker counts than FORA sessions, while segment-level speaker counts remain similar. Moderator turns and short fragmentary utterances are excluded from segment-level participation measures, and they represent a nontrivial token share.

Agreement on overall informativeness is moderate to high and fairly comparable across corpora, but the three aspect ratings show a more uneven pattern. Relevance is the most consistently judged aspect in one corpus, while novelty is relatively less stable there; in the other corpus, novelty reaches the highest agreement and scope the lowest. This suggests that perceived informativeness is rated consistently overall, but its component dimensions depend more on the setting. Overall CIG agreement is broadly similar across the two corpora. Aspect agreement varies, with relevance showing strong consistency in one setting and scope showing the weakest consistency in the other.

Ordinal regression model comparisons show that perceived CIG is predicted less well by utterance length alone than by the proposed semantic aspects. Adding novelty yields the strongest single aspect gain, relevance also improves fit, while implication scope provides little consistent benefit. Across corpora, models without implication scope are competitive or better, suggesting informativeness is not simply driven by broader public reach. Word count alone fits poorly compared with models adding semantic aspects Novelty is the most useful single aspect, with relevance also improving fit Implication scope is not required and can reduce predictive performance in one corpus

Across both corpora, GPT-5 remains comparable to the human leave-one-out baseline when given only the memory-based prior summary for CIG prediction. In several rating categories, GPT-5 matches or falls below individual human deviation from consensus, suggesting that the summary captures much of the information used by human annotators. GPT-5 aligns closely with human consensus on overall CIG in both corpora. For overall CIG and novelty, GPT-5 error is lower than the human leave-one-out baseline. The memory summary alone serves as a compact approximation of the context relied on for CIG ratings.

The experiments evaluate Conversational Information Gain using annotated FORA and INSQ dialogues, examining whether prior dialogue context can support reliable ratings and scalable estimation. The results show that overall informativeness is consistently judged across corpora, although ratings for novelty, relevance, and scope depend more on the setting. Model comparisons further indicate that semantic aspects, especially novelty and relevance, explain perceived CIG better than utterance length, and that a GPT-5 model using only a memory-based summary reaches human-like agreement, particularly for overall CIG and novelty.