Benchmark	Methodology	Metrics
common-sense-reasoning-on-record	BERT-Base (single model)	EM: 54.040 F1: 56.065
common-sense-reasoning-on-swag	BERT-LARGE	Dev: 86.6 Test: 86.3
coreference-resolution-on-winograd-schema	BERT-large 340M	Accuracy: 62.0
cross-lingual-natural-language-inference-on-1	BERT	Accuracy: 74.3%
cross-lingual-natural-language-inference-on-3	BERT	Accuracy: 70.5%
emotion-recognition-in-conversation-on-cped	BERT_{utt}	Accuracy of Sentiment: 48.96 Macro-F1 of Sentiment: 45.18
linguistic-acceptability-on-cola	BERT-LARGE	Accuracy: 60.5%
multimodal-intent-recognition-on-photochat	BERT	F1: 53.2 Precision: 56.1 Recall: 50.6
natural-language-inference-on-multinli	BERT-LARGE	Matched: 86.7 Mismatched: 85.9
natural-language-inference-on-qnli	BERT-LARGE	Accuracy: 92.7%
natural-language-inference-on-rte	BERT-large 340M	Accuracy: 70.1%
natural-language-inference-on-wnli	BERT-large 340M	Accuracy: 65.1
natural-language-understanding-on-glue	BERT-LARGE	Average: 82.1
natural-language-understanding-on-pdp60	BERT-large 340M	Accuracy: 78.3
paraphrase-identification-on-quora-question	BERT-LARGE	F1: 72.1
question-answering-on-coqa	BERT Large Augmented (single model)	In-domain: 82.5 Out-of-domain: 77.6 Overall: 81.1
question-answering-on-coqa	BERT-base finetune (single model)	In-domain: 79.8 Out-of-domain: 74.1 Overall: 78.1
question-answering-on-mrqa-2019	BERT (large)	Average F1: 78.5
question-answering-on-multirc	BERT-large(single model)	EM: 24.1 F1: 70.0
question-answering-on-multitq	BERT	Hits@1: 8.3 Hits@10: 48.2
question-answering-on-piqa	BERT-Large 340M	Accuracy: 66.7
question-answering-on-squad11	BERT-LARGE (Ensemble+TriviaQA)	EM: 87.4 F1: 93.2
question-answering-on-squad11	BERT (ensemble)	EM: 87.433 F1: 93.160
question-answering-on-squad11	BERT (single model)	EM: 85.083 F1: 91.835
question-answering-on-squad11	BERT-LARGE (Single+TriviaQA)	F1: 91.8
question-answering-on-squad11-dev	BERT-LARGE (Ensemble+TriviaQA)	EM: 86.2 F1: 92.2
question-answering-on-squad11-dev	BERT-LARGE (Single+TriviaQA)	EM: 84.2 F1: 91.1
semantic-textual-similarity-on-mrpc	BERT-LARGE	F1: 89.3
semantic-textual-similarity-on-sts-benchmark	BERT-LARGE	Spearman Correlation: 0.865
sentiment-analysis-on-sst-2-binary	BERT-LARGE	Accuracy: 94.9
stock-market-prediction-on-astock	Bert Chinese	Accuray: 59.11 F1-score: 58.99 Precision: 59.07 Recall: 59.20
text-classification-on-dbpedia	Bidirectional Encoder Representations from Transformers	Error: 0.64
type-prediction-on-manytypes4typescript	BERT	Average Accuracy: 57.52 Average F1: 54.10 Average Precision: 54.18 Average Recall: 54.02

Build AI with AI

From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.

AI Co-coding

Ready-to-use GPUs

Best Pricing

Get Started

Hyper Newsletters

Subscribe to our latest updates

We will deliver the latest updates of the week to your inbox at nine o'clock every Monday morning

Command Palette

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova

Abstract

Code Repositories

Benchmarks

Build AI with AI

Hyper Newsletters