One-sentence Summary

Researchers from Facebook AI Research and Harvard SEAS propose a local attention-based neural model for abstractive sentence summarization that generates each summary word conditioned on the input sentence and, trained end-to-end, significantly outperforms strong baselines on the DUC-2004 shared task.

Key Contributions

• The paper proposes a local attention-based neural model for abstractive sentence summarization that generates each summary word conditioned on the input sentence, enabling simple end-to-end training and scalability to large datasets.
• On the DUC-2004 shared task, the model achieves significant performance gains over several strong baselines.
• Training on four million article-title pairs allows fully abstractive decoding without monotonic alignment, extending earlier headline generation methods with a scalable neural architecture.

Introduction

The authors address the challenge of sentence-level abstractive summarization, where the goal is to produce a condensed sentence that may involve paraphrasing, generalization, or reordering rather than simply deleting words. Most prior systems rely on extractive compression or impose linguistically motivated syntactic constraints, limiting their flexibility and scalability. The authors introduce a fully data-driven Attention-Based Summarization (ABS) model that combines a neural language model with an attention-based encoder, learning a soft alignment over the input. The model makes no assumptions about output vocabulary or sentence structure, allowing it to be trained directly on large-scale document-summary pairs (4 million articles from Gigaword). Jointly trained with a beam-search decoder and additional extractive features, this approach outperforms traditional syntax-based systems, integer linear programming methods, and a statistical machine translation baseline, achieving significant improvements on headline-generation benchmarks.

Dataset

The authors use two main datasets for training and evaluation of headline generation:

• Training data (headline generation)

• Source: Annotated Gigaword corpus (around 9.5 million news articles) preprocessed with Stanford CoreNLP for tokenization and sentence separation.

• Pair construction: The headline of each article is paired with its first sentence, forming an input-summary pair.

• Filtering: Heuristic rules remove noisy pairs:

• No non-stop-words in common between title and first sentence.

• Title contains a byline or extraneous editing marks.

• Title contains a question mark or colon.

• Final size: Roughly 4 million title-article pairs after filtering.

• Preprocessing: PTB tokenization, lowercasing, digits replaced with #, word types occurring fewer than 5 times replaced with UNK. All articles from the DUC evaluation time period excluded.

• Statistics:

• Input vocabulary: 119 million tokens, 110K unique types; average sentence length 31.3 words.

• Headline vocabulary: 31 million tokens, 69K unique types; average title length 8.3 words.

• Average overlapping word types between headline and input: 4.6 (2.6 within the first 75 characters of the input).

• Evaluation data

• DUC-2004 shared task set (evaluation-only): 500 news articles from New York Times and Associated Press, each with 4 human-written reference summaries capped at 75 bytes (roughly 14 words). Only the first sentence of each article is used.

• Heldout Gigaword subset: Randomly selected articles not used in training, paired with their headlines as single-reference summaries. This set, closer to the training task, is larger and will be released with the code. Systems are tuned to produce output of average title length.

All datasets are used for recall-oriented evaluation with ROUGE-1, ROUGE-2, and ROUGE-L, with outputs truncated at 75 characters for the DUC task.

Method

The authors propose a fully data-driven approach for abstractive sentence summarization by directly parameterizing the conditional distribution of the next word given the input sentence and the current context. Instead of relying on a noisy-channel approach that splits the estimation of a language model and a conditional summarization model, the framework integrates a neural probabilistic language model with a contextual input encoder. This joint parameterization allows the system to incorporate the input text directly into the generation process.

At the core of the architecture is a neural language model adapted from a standard feedforward neural network language model. The contextual probability of the next word is estimated as: $p(\mathbf{y}_{i+1} | \mathbf{y}_c, \mathbf{x}; \theta) \propto \exp(\mathbf{Vh} + \mathbf{Wenc}(\mathbf{x}, \mathbf{y}_c))$p(yi+1​∣yc​,x;θ)∝exp(Vh+Wenc(x,yc​)) where $\mathbf{h} = \tanh(\mathbf{U}\tilde{\mathbf{y}}_c)$h=tanh(Uy~​c​) represents the hidden layer derived from the context embeddings, and $\mathbf{enc}$enc is a contextual encoder function. The parameters include word embedding matrices and weight matrices mapping the hidden states to the vocabulary space.

To incorporate the input text, the authors design several instantiations of the encoder module. The most basic variant is a Bag of Words Encoder, which embeds the input sentence and applies a uniform distribution over the input words, ignoring word order. To capture local interactions between words, a Convolutional Encoder utilizes a time delay neural network architecture that alternates between temporal convolution layers and max pooling layers.

To further enhance the capacity of the model to align the input with the generated summary, the authors introduce an Attention-Based Encoder. This module learns a latent soft alignment over the input text, replacing the uniform distribution of the bag of words approach with a learned alignment matrix. As shown in the figure below:

This attention mechanism computes a weighted representation of the smoothed input embeddings based on the current generation context, allowing the model to focus on relevant input words when predicting the next summary token.

The training process focuses on minimizing the negative log-likelihood of the reference summaries. The objective function factors into a sum of token-level conditional probabilities across the dataset: $\mathrm{NLL}(\theta) = - \sum_{j=1}^{J} \sum_{i=1}^{N-1} \log p(\mathbf{y}_{i+1}^{(j)} | \mathbf{x}^{(j)}, \mathbf{y}_c; \theta)$NLL(θ)=−∑j=1J​∑i=1N−1​logp(yi+1(j)​∣x(j),yc​;θ) The authors optimize this objective using minibatch stochastic gradient descent. The training is conducted on GPUs with shuffled minibatches grouped by input length, and the embedding tables are renormalized after each epoch.

For summary generation, the authors employ a beam-search decoder to approximate the optimal summary sequence. Unlike phrase-based machine translation, the absence of explicit hard alignment constraints simplifies the decoding process, allowing the algorithm to maintain a fixed number of hypotheses at each step without tracking source word usage.

To address the limitation of purely abstractive models in transferring exact phrases or rare words from the input, the authors introduce an extractive tuning extension. They modify the scoring function into a log linear model that combines the neural conditional probability with additional indicator features. These features capture unigram, bigram, and trigram matches with the input, as well as word reordering. After training the primary neural model, the authors fix the neural parameters and tune the feature weights using minimum error rate training on a development set.

Experiment

The evaluation compares an attention-based abstractive summarization model against multiple baselines on headline generation using the DUC-2004 and Gigaword datasets. The full model (ABS+) significantly outperforms previous systems like TOPIARY and a tuned phrase-based MT system, with attention encoding and beam search providing the largest improvements. Qualitative analysis shows the model effectively identifies key words and sometimes rephrases, though it often makes syntactic errors and remains far from human performance.

On DUC-2004 and Gigaword summarization, the tuned abstractive model ABS+ achieves the best ROUGE scores, significantly outperforming both the TOPIARY system and the MOSES+ phrase-based baseline. Simple baselines like PREFIX perform well on ROUGE-1 due to high input–output overlap, while IR and COMPRESS are weak, showing that summarization needs both content selection and language modeling. Additional extractive features increase the reuse of input words and boost metric scores. The full ABS+ model is the top performer, scoring significantly higher than TOPIARY on all DUC metrics and outperforming MOSES+ on ROUGE-1 and ROUGE-L for Gigaword. The PREFIX baseline, which copies the first tokens of the article, achieves a high ROUGE-1 (e.g., 22.43 on DUC, 23.14 on Gigaword) thanks to 100% token overlap with the source. IR and COMPRESS baselines obtain the lowest ROUGE scores, confirming that neither sentence retrieval nor compression alone is sufficient for summarization. Incorporating extractive features raises the summary's token overlap with the input (MOSES+ reaches 70.5% on Gigaword) and consistently improves ROUGE results.

On the Gigaword validation set, a feed-forward neural language model without an encoder reduces perplexity compared to a KN-smoothed 5-gram baseline. Incorporating a simple bag-of-words encoder yields a large further drop, and more powerful encoders—convolutional and attention-based—push perplexity progressively lower, with the attention-based model achieving the best result. A feed-forward neural LM alone improves over the n-gram baseline, but adding a bag-of-words encoder lowers perplexity below 50, a much larger relative gain. The attention-based encoder delivers the lowest perplexity (below 30), outperforming both the convolutional and bag-of-words encoders by a substantial margin.

Ablation experiments on the DUC-2003 development set show that the largest gains in ROUGE scores come from replacing a bag-of-words encoder with an attention-based encoder and from switching greedy decoding to beam search. Abstractive generation yields the best overall results, though a purely extractive variant remains competitive. Beam search substantially improves all ROUGE metrics over greedy decoding for the same abstractive model. Moving from a bag-of-words encoder to an attention-based encoder dramatically raises ROUGE-1, ROUGE-2, and ROUGE-L. Purely extractive generation achieves strong ROUGE, but adding abstractive capability further lifts scores across all metrics.

The evaluation uses DUC-2004 and Gigaword summarization tasks to compare the abstractive ABS+ model against extractive baselines, finding that ABS+ yields the best ROUGE scores and that adding extractive features increases input word reuse and metric gains. Language modeling experiments on the Gigaword validation set show that an attention-based encoder dramatically reduces perplexity compared to simpler encoders like bag-of-words or convolutional ones. Ablation on DUC-2003 confirms that switching to an attention encoder and using beam search decoding provide the largest improvements, while abstractive generation surpasses a competitive extractive variant.