Command Palette
Search for a command to run...
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Abstract
Benchmark datasets have a significant impact on accelerating research inprogramming language tasks. In this paper, we introduce CodeXGLUE, a benchmarkdataset to foster machine learning research for program understanding andgeneration. CodeXGLUE includes a collection of 10 tasks across 14 datasets anda platform for model evaluation and comparison. CodeXGLUE also features threebaseline systems, including the BERT-style, GPT-style, and Encoder-Decodermodels, to make it easy for researchers to use the platform. The availabilityof such data and baselines can help the development and validation of newmethods that can be applied to various program understanding and generationproblems.
Code Repositories
Benchmarks
| Benchmark | Methodology | Metrics |
|---|---|---|
| cloze-test-on-codexglue-ct-all | CodeBERT(MLM) | Go: 83.31 JS: 81.77 Java: 80.63 PHP: 85.05 Python: 87.21 Ruby: 80.17 |
| cloze-test-on-codexglue-ct-maxmin | CodeBERT(MLM) | Go: 90.79 JS: 86.4 Java: 90.46 PHP: 88.21 Python: 82.2 Ruby: 86.84 |
| code-repair-on-codexglue-bugs2fix | CodeBERT | Accuracy (medium): 5.2 Accuracy (small): 16.4 BLEU (medium): 91.07 BLEU (small): 77.42 CodeBLEU (medium): 87.52 CodeBLEU (small): 75.58 |
| code-search-on-codexglue-advtest | CodeBERT | MRR: 27.19 |
| code-search-on-codexglue-webquerytest | CodeBERT | Accuracy: 47.8 F1: 58.95 |
| code-translation-on-codexglue-codetrans | CodeBERT | Accuracy (C#→Java): 58 Accuracy (Java→C#): 59 BLEU (C#→Java): 72.14 BLEU (Java→C#): 79.92 CodeBLEU (C#→Java): 79.41 CodeBLEU (Java→C#): 85.1 |
| defect-detection-on-codexglue-devign | CodeBERT | Accuracy: 62.08 |
| text-to-code-generation-on-codexglue-concode | CodeGPT-adapted | BLEU: 32.79 CodeBLEU: 27.74 EM: 20.1 |
Build AI with AI
From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.