One-sentence Summary

Researchers at USC ISI, UCLA, and Bloomberg present a case study comparing LLM planning decisions with human journalists on press release coverage, finding that LLMs align better with humans on recommending article angles than informational sources, yet both angles and sources suggested by LLMs remain significantly less creative than those chosen by human writers.

Key Contributions

• A corpus of 650,000 news articles covering 250,000 press releases across 10 years is assembled to study journalists' planning decisions.

• Contrastive summarization is introduced to identify effective coverage that challenges press releases; on 6,000 article and press release pairs, this coverage correlates with more creative angles (r = .29) and more informational sources (r = .5).

• LLM recommendations align better with human angle choices than source choices, reaching 63.6 and 27.9 F1-score respectively, while manual evaluation finds both suggested angles and sources less creative than human plans.

Introduction

The authors study LLM assistance for journalism planning, where in-depth news coverage requires identifying meaningful story angles and selecting appropriate sources beyond simple summarization. Prior work showed LLMs can suggest coverage angles, but it remained unclear whether these suggestions align with human journalistic values, especially in terms of creativity and source selection. To address this gap, the authors build a large corpus of press releases and associated news articles, define effective coverage through contrastive summarization, and create a benchmark that compares LLM-generated planning decisions with real human journalistic choices.

Dataset

• The authors construct PressRelease, a corpus of news article and press release pairs. The final dataset contains 656,523 news articles and 250,224 press releases, intended to support analysis of how journalists cover press releases.

• Data are collected from two complementary directions to reduce collection bias:

  • News outlets to press releases:

    • The authors query Common Crawl for all URLs from 9 major financial newspapers in all scrapes since 2021, yielding 114 million URLs.
    • They use the supervised model from Welsh (2022) to identify 940,000 URLs that are news articles rather than other page types such as login pages.
    • They find hyperlinks from these articles to known press release websites.
    • This yields 247,372 articles covering 117,531 press releases.
    • For each matched press release, they retrieve the most recent version published before the news article from the Wayback Machine.
    • This direction is biased because it uses only 9 major financial outlets, may introduce noise through hyperlink filters, and likely favors press releases with more coverage.

  • Press releases to news articles:

    • The authors collect press release sources from subdomains associated with PR offices for all 500 S&P 500 companies, plus other organizations of interest such as OpenAI, SpaceX, and Theranos, and selected notable press releases.
    • They query a backlinking service for webpages that link to these press release sources.
    • They again use Welsh (2022)’s model to identify backlinks that correspond to news articles.
    • They retrieve 587,464 news articles and 176,777 press releases from the Wayback Machine.
    • This direction broadens outlet coverage but can overrepresent press releases from major companies and may miss releases that are not directly posted on company websites.
    • Combining both directions is intended to reduce one-sided popularity bias.

• The authors apply filtering heuristics to keep pairs where the press release is likely central to coverage:

  • They exclude article and press release pairs when the press release link appears in the bottom 50 percent of the article.
  • They exclude pairs whose publication dates are more than one month apart.
  • These rules are designed to remove articles where the press release is unlikely to be the main topic.

• The resulting corpus includes multi-article coverage of the same press release:

  • 70,062 press releases, or 28 percent of the dataset, are covered by more than one news article.
  • These multiply-covered stories account for 509,820 article-level records.
  • The authors note that this structure is not used in the current work but supports future analysis of coverage decisions.

• The authors use the dataset first to train and evaluate a contrastive-summary detector on a manually annotated subset:

  • They annotate 1,100 article and press release pairs.
  • One annotator labels all documents, and a second annotator labels 50 documents to measure agreement, yielding agreement greater than 0.8.
  • The annotated subset is split into 80/10/10 train, validation, and test portions.
  • During input construction, the authors test adding coreference resolution to make document representations more contextually complete.
  • After training the detector, they apply it to the full PressRelease corpus to obtain document-level NLI scores for article and press release pairs, which are then used to analyze press release coverage.

Method

The authors design the method as a multi-stage pipeline that first discovers article–press-release pairs, then converts the vague idea of “effective coverage” into a quantitative scoring problem, and finally uses the resulting pairs to evaluate how well large language models can support journalistic planning. The pipeline has three coupled components: bidirectional corpus construction, contrastive coverage scoring, and supervised detection of contrastive summaries.

The corpus construction stage is built as a reciprocal linking workflow rather than a single scraping operation. In one direction, the authors collect news article HTML, identify pages that are actual news articles rather than peripheral website pages, and then search those pages for hyperlinks pointing to known press-release domains. For each linked press release, they retrieve the archived version that was published most recently before the article appeared. This route captures how articles cite or link to releases, but it can overrepresent outlets included in the crawl, popular releases, and releases that are explicitly linked in article bodies.

To offset those biases, the authors also traverse links in the opposite direction. They identify press-release web properties and organization pages of interest, then query a backlinking service for pages that link to those press-release sources. The returned pages are filtered to retain news-article backlinks, and the corresponding press-release pages are again retrieved from historical archives. This second route widens outlet coverage, though it can overrepresent releases from major companies and miss releases not posted directly on company-owned domains. A filtering module then removes weak pairs by excluding links that appear in the latter half of an article and by excluding pairs with large temporal gaps. The intuition is that the press release should be the central subject of the article, not a footnote or old reference.

After forming candidate pairs, the authors define effective coverage through the concept of contrastive summarization. The input to this module is a press release represented as a sequence of sentences, $\vec{P}=p_1,\dots,p_n$P=p1​,…,pn​, and a news article represented as another sequence of sentences, $\vec{N}=n_1,\dots,n_m$N=n1​,…,nm​. A valid contrastive summary must satisfy two conditions. First, the article must reference the press release sufficiently often:

Second, the article must challenge the press release sufficiently often:

Here, “references” is treated as the union of entailment and contradiction in a natural language inference setting, while “contradicts” corresponds directly to the contradiction relation. This formulation operationalizes the journalistic practice of not merely repeating a release, but placing it in context and questioning it.

To estimate these relations at document scale, the authors use a sentence-level natural language inference module. For every press-release sentence and every article sentence pair, they compute an NLI probability $p(y \mid p_i,n_j)$p(y∣pi​,nj​) for relations such as entailment, contradiction, or no contradiction. They then aggregate these local signals into a press-release sentence score by keeping the strongest inner top-$k$k relations for each $p_i$pi​:

This $s_i$si​ score represents whether a given press-release sentence is sufficiently covered or challenged. The authors then aggregate across sentences by averaging the outer top-$k$k sentence scores to obtain a document-level coverage signal. The $k_{inner}$kinner​ parameter controls how many article sentences must support a press-release sentence before it is considered covered, while $k_{outer}$kouter​ controls how many press-release sentences must be covered before the article is considered substantially about the release. This aggregation is intentionally cheap and scalable because it relies on pairwise sentence scoring rather than full document generation.

The authors then train a supervised detector to identify contrastive summaries. They annotate article–release pairs with two labels: whether the article is substantially about the press release, and whether the article challenges the press release. The detector uses aggregated NLI scores as its primary representation. One important preprocessing choice is coreference resolution, because pronouns and named entities can otherwise fragment the evidence between a release and an article. The authors compare simple classifiers, including logistic regression and multilayer perceptron variants. They also evaluate a binned histogram-style classifier, which bins the distribution of NLI scores rather than relying only on pointwise features. The binned histogram classifier with coreference resolution gives the strongest detection performance, likely because it reduces sensitivity to outlier NLI scores while preserving the overall shape of the entailment and contradiction distributions.

Once the detector is trained, it is applied across the larger article–release corpus to assign document-level coverage and challenge scores. These scores are then used to select pairs in which journalists appear to have performed strong contrastive coverage. That selection forms the basis for the LLM planning module.

For the planning evaluation, the authors compare two kinds of preparatory documents: plans written by human journalists and plans generated by LLMs. The comparison centers on two concrete journalistic decisions that occur before article drafting: choosing an angle and selecting sources. An angle specifies the focus or interpretive lens of the story, while a source list specifies the people or documents needed to support that angle. Refer to the framework diagram for the high-level contrast between human plans and LLM plans. As shown in the figure below:

In the illustrated comparison, the human plan treats the press release as evidence for a broader negative pattern and proposes sources such as a CEO quote, an advocacy group, and a person directly affected by the event. The LLM plan, by contrast, frames the task more narrowly as describing the event and proposes sources such as a spokesperson or subject-matter professors. This comparison makes the planning module operational: rather than asking only whether an LLM can summarize a press release, the authors ask whether its generated angle and source choices resemble the investigative planning decisions seen in human coverage of press releases.

Experiment

The experiments first validate a NLI-based contrastive summarization analysis of press-release coverage, showing that effective journalism combines contextualization, challenging statements, creative angles, and diverse, resource-intensive sources. They then evaluate closed and open LLMs, in zero-shot, few-shot, and fine-tuned settings, by comparing LLM-generated angles and source plans with human journalist plans and rating plan creativity. LLMs remain poor at reproducing human planning, especially for sources and creativity, although fine-tuning reveals some learnable patterns and improves angle suggestion. Overall, the findings position LLMs as imperfect standalone planners and suggest retrieval-grounded and creativity-enhancing approaches with human oversight.

The corpus pairs large numbers of press releases with news articles that rewrite or evaluate them through distinct angles and cited sources. Many articles go beyond neutral restatement by adding critical or contextual material, and a substantial share of releases receive repeated coverage from different outlets. These examples show how news coverage can turn the same source material into different narratives through choice of angle, evidence, and sources. News articles often reshape press releases by adding neutral, critical, or more creative angles rather than repeating the release. Coverage quality is associated with source use, including people, unions, employees, studies, and other reported information. A substantial portion of press releases is covered by multiple articles, creating opportunities to compare different reporting decisions.

Document-level NLI classifiers detect whether news articles cover press releases more strongly than whether they challenge press releases. The histogram-based classifier with coreference resolution performs best for both detection tasks, while coreference resolution generally improves performance, especially for the histogram model. Coverage detection is easier than challenge detection for the evaluated classifiers. Coreference resolution helps produce sharper predictions and gives the largest gain to the histogram classifier.

The correlation results indicate that articles with more distinct sources are more likely to be labeled as contradicting or entailing the press release. The strongest positive association appears for contradiction, while entailment also shows a positive but weaker relationship. In contrast, neutral coverage is associated with using fewer distinct sources. Articles that contradict the press release tend to use more distinct sources. Articles that entail the press release also tend to use more sources, though the association is weaker than for contradiction. Neutral articles tend to use fewer distinct sources.

Articles with more creative planning angles show positive associations with NLI contradiction and entailment relations to press releases. Contradiction has a slightly stronger positive association than entailment, while neutral relations show a weak negative association. Contradiction and entailment are positively associated with creative planning angles. Neutral coverage is weakly inversely associated with creativity.

Source types are associated with the level of contradiction between a news article and a press release. Person-derived quotes have the strongest positive association, indicating that more contradicting coverage tends to involve information obtained from people. Published work and press reports are also positively associated, while court proceedings are the weakest among the listed source types. Person-derived quotes correlate most strongly with higher contradiction. Published work and press reports are the next most positively associated source types. Email and social media posts, statements and public speeches, and proposals, orders, or laws fall below quotes and published work.

The experiments evaluate a corpus of press releases paired with news articles to examine how coverage is framed and whether articles extend, challenge, or remain neutral toward the source material. Classifier results show that detecting whether a news article covers a press release is easier than detecting challenge, with coreference resolution improving predictions, especially for the histogram model. Correlation analyses further indicate that articles using more distinct sources and more creative angles are more likely to contradict or entail the release, while neutral coverage is associated with fewer sources, and person-derived quotes are particularly linked to contradicting reporting.