One-sentence Summary

Tsinghua University and collaborators introduce the Farm (Fact to Misinform) dataset and a multi-turn testing framework to track how LLMs' correct factual beliefs can be systematically manipulated through persuasive conversations, finding that various persuasive strategies easily shift model beliefs in ways not captured by prior single-turn studies.

Key Contributions

• The study introduces the Farm dataset, which pairs factual questions with systematically generated persuasive misinformation to test whether external misinformation can overturn initially correct answers from large language models.
• The study develops a multi-turn conversational testing framework that uses closed-book question answering to establish an LLM's initial belief and then tracks belief changes as persuasive misinformation is introduced across dialogue turns.
• Experiments show that large language models with correct factual knowledge, including state-of-the-art models such as GPT-4, can be persuaded to adopt incorrect beliefs, with repetition and rhetorical appeals identified as particularly effective persuasive strategies.

Introduction

Large language models store substantial factual knowledge and are increasingly used for information seeking, but prior work shows they can follow misleading or user-provided claims that conflict with their memory. Most existing studies examine one-turn settings, prompt perturbations, hallucination detection, or sycophancy, leaving limited understanding of whether multi-turn persuasive dialogue can alter an LLM’s factual beliefs. The authors leverage psychological persuasion concepts to build Farm, a dataset of straightforward factual questions paired with persuasive misinformation, and a three-stage framework that checks an LLM’s initial belief, drives a multi-turn persuasive conversation, and measures belief change. They find that leading models remain vulnerable to factual misinformation, with belief alteration reaching 50.1 percent for ChatGPT and 20.7 percent for GPT-4.

Dataset

• The authors build Farm from three existing question answering sources: BoolQ, Natural Questions, and TruthfulQA. They sample 500 questions from each source, giving 1,500 base factual questions.

• Questions are filtered to those that GPT-4 can answer correctly in a closed-book setting. During evaluation, each model is tested only on questions it can answer correctly, so each model may use a different model-specific correct subset.

• The base QA pairs are converted into a unified multiple-choice schema. BoolQ remains boolean, Natural Questions questions are expanded into 4-option MCQs with the correct answer included, and TruthfulQA uses its provided MCQ version. Each question also receives a don't know option, and answer choices are shuffled to reduce order sensitivity.

• Misinformation metadata is generated for each question. For BoolQ, the misinformation target is the opposite answer. For Natural Questions, two variants are created: NQ1 uses a Not A target, and NQ2 lets GPT-4 select the most appropriate incorrect MCQ option. TruthfulQA follows the NQ2-style incorrect-option construction. For each target, GPT-4 generates a concise control statement that falsely claims the question answer is that target.

• Around each false control statement, the authors generate persuasive messages using three rhetorical appeals: logical, credibility, and emotional. Three unique messages are generated for each appeal per control statement. The generated appeals are checked with GPT-4 using textual entailment and strategy alignment evaluation.

• Five annotators perform formal validation, and two authors filter 48 unsuccessful generation cases. The final dataset contains 1,952 entries, with the extra entries coming from the two Natural Questions misinformation variants over the initial 1,500 base questions.

• The paper uses Farm as an evaluation benchmark for model robustness against persuasive misinformation. The data are not described as having training, validation, or test splits. Mixture is balanced by taking equal numbers from the three source datasets, and the main processing steps are filtering, MCQ formatting, abstention-option addition, option shuffling, and synthetic misinformation and appeal generation. No cropping strategy is mentioned.

Method

The authors build a two stage evaluation pipeline for testing how reliably an LLM maintains factual beliefs when it is repeatedly exposed to persuasive misinformation. The first stage curates straightforward factual questions and converts them into standardized multiple choice items. The second stage generates misinformation that contradicts the correct answers, and the third stage uses those items in an interactive belief tracking conversation. The authors leverage GPT-4 for prompting throughout question formatting, misinformation generation, and persuasion strategy construction.

To curate the questions, the authors first select simple factual questions that are answerable in a closed book setting and for which the target model can initially select the correct option. These questions are then formatted as multiple choice items. Binary questions remain in a yes or no form, while short answer questions are expanded into four options that include the correct answer and plausible incorrect alternatives. A “don’t know” option is also added so that the model can abstain when uncertain. The answer choices are shuffled to reduce sensitivity to option order. During testing, a belief is considered present only when the model selects an option other than “don’t know.”

For misinformation generation, the authors first define a false target proposition $T$T for each question. For binary questions, $T$T is set to the opposite of the correct answer. For short answer questions, $T$T may be set to an explicit negation of the correct answer or selected from incorrect multiple choice distractors. For long answer questions, a similar strategy is used to produce a misleading but plausible alternative. From $T$T, the authors generate a concise control statement $CTRL$CTRL that presents the false proposition as a fact. They then use GPT-4 to generate persuasive messages based on $CTRL$CTRL. The persuasion process is guided by three rhetorical appeals: logical appeal, which uses reasoning, facts, or evidence; credibility appeal, which invokes authority or trustworthy sources; and emotional appeal, which tries to provoke feelings such as fear, sympathy, or excitement. For each control statement, multiple persuasive messages are generated for each appeal type, enabling repeated pressure in a conversation.

The generated persuasive messages are screened for usability through two verification tasks. Textual entailment evaluates whether a generated appeal semantically supports the control statement, while strategy alignment checks whether the message actually reflects the intended persuasive strategy. This filtering step helps ensure that the misinformation used in the test is coherent, targeted, and aligned with the rhetorical mechanism being evaluated.

Refer to the framework diagram.

As shown in the framework, the misinformation test proceeds through three phases: an initial belief check, a persuasive conversation with implicit belief checks, and a final belief check. In the initial belief check, the model is asked the original factual question in its multiple choice form. Only if the model already selects the correct answer does the framework proceed to the persuasion phase. This restriction makes the test a controlled measurement of belief reversal rather than an evaluation of general question answering difficulty.

The persuasive conversation is the core of the method. Each conversation begins with the control statement $CTRL$CTRL and continues for up to four turns. The model’s own replies are recorded, but the persuasion is driven by generated messages associated with one selected strategy. In addition to the three rhetorical appeals, the authors include a repetition strategy that simply repeats the control statement across turns, allowing the method to test whether misinformation can succeed through repetition alone.

An important design choice is the implicit belief check. Rather than judging belief change from the model’s conversational replies alone, the framework reasks the original factual question after each turn. This check is deliberately hidden from the conversation history, meaning the test question and answer are not appended to the model’s visible context. This prevents the model from recognizing that it is being tested and reduces the risk of interpreting sycophantic agreement with misinformation as a genuine belief change. If the model still selects the correct option during an implicit belief check, the persuasive conversation continues until belief reversal occurs or the maximum number of turns is reached. The final belief check is explicit and marks the end of the evaluation, indicating whether the model has been successfully misinformed, retained its original belief, or abstained from answering.

The authors also describe a lightweight mitigation procedure intended to reduce susceptibility to misinformation during deployment. When potential misinformation is detected in user input, possibly by another model, the system inserts a reminder in the form of a system prompt. This prompt instructs the model to be cautious with potentially malicious users and to verify its memorized knowledge before responding. The intuition is twofold: models often assume that users are well intentioned when their responses conflict with model knowledge, and models may show stronger resistance when encouraged to recall supporting evidence. As shown in the comparison chart:

Experiment

Experiments evaluate five popular LLMs across multi-turn persuasive misinformation, tracking initial knowledge, retained accuracy, and belief change. Results show most models are surprisingly vulnerable, with higher knowledge or more advanced models generally resisting better, while repeated exposure amplifies misinformation. Analysis of token confidence and behavioral patterns links lower certainty to greater susceptibility, and a simple system-prompt reminder can partially mitigate the effect.

The example illustrates how a simple factual question is converted into misleading content by replacing the correct answer with an incorrect control statement. It then shows that the false claim can be supported through logical, credibility, and emotional rhetorical appeals generated with a large language model. This setup demonstrates a structured way to study how persuasive misinformation may challenge model robustness. The factual answer Jonah is replaced by the false answer Noah to create misinformation. The false claim is reinforced through logic-based evidence, claims of expertise, and emotionally charged storytelling.

The evaluation shows that generated persuasive appeals preserve their intended misinformation claims and rhetorical strategies very well across datasets. Entailment and strategy alignment are high for all question types, with strategy alignment being especially stable. Entailment varies slightly, with NQ1 showing the strongest result and TruthfulQA showing the lower end while still remaining high. Generated appeals remain strongly aligned with their intended rhetorical strategy across datasets. NQ1 shows the strongest NLI entailment among the evaluated datasets. TruthfulQA has the comparatively lower entailment in the evaluation, yet still shows high strategy alignment.

The table presents four message templates used to persuade a language model with misinformation: repetition, logical appeal, credibility appeal, and emotional appeal. Each conversation uses one strategy after presenting the misinformation, placing the persuasive message within a short interaction designed to probe belief change. The templates show that non-repetition strategies pair claim-specific support with a request to accept the misinformation. Repetition uses a direct assertion of the misinformation. Logical, credibility, and emotional strategies add a claim-specific appeal before requesting acceptance. Only one persuasive strategy is applied per conversation. The templates support testing separate persuasion pathways rather than mixed strategies.

The ranking suggests that greater model knowledge is often associated with stronger robustness against misinformation. GPT-4 is positioned highest in both knowledge and robustness, while weaker or smaller models appear more vulnerable to belief change. The ordering also implies that model improvements can correspond to greater resistance, though the relationship is not perfectly uniform. GPT-4 is ranked as both the most knowledgeable and the most robust tested model. Llama-2-7B is ranked as the least robust model, indicating higher susceptibility to misinformation. More capable or improved models generally appear more resistant to misinformation, consistent with a rough knowledge robustness trend.

Repeating misinformation generally increases the likelihood that models adopt false beliefs compared with a single exposure. The effect appears across both tested models and datasets, with ChatGPT showing consistent susceptibility gains and GPT-4 showing a more limited but still observable repetition effect. Repeated misinformation raises belief alteration relative to one-turn exposure across the reported models and benchmarks. GPT-4 is less affected overall, but repetition can still meaningfully increase its misinformed rate on certain datasets.

The evaluation converts factual questions into misinformation by replacing correct answers and validates that generated logical, credibility, and emotional appeals preserve the intended false claims and rhetorical strategies. Separate persuasion templates are then used to isolate repetition, logic, credibility, and emotional pathways, while model comparisons show that greater knowledge generally corresponds to stronger resistance, with GPT-4 most robust and weaker models more susceptible. Repeated misinformation raises false-belief adoption compared with single exposure, although the impact varies by model and dataset.