HyperAIHyperAI

Command Palette

Search for a command to run...

3 months ago

Text is no more Enough! A Benchmark for Profile-based Spoken Language Understanding

Xiao Xu Libo Qin Kaiji Chen Guoxing Wu Linlin Li Wanxiang Che

Text is no more Enough! A Benchmark for Profile-based Spoken Language Understanding

Abstract

Current researches on spoken language understanding (SLU) heavily are limited to a simple setting: the plain text-based SLU that takes the user utterance as input and generates its corresponding semantic frames (e.g., intent and slots). Unfortunately, such a simple setting may fail to work in complex real-world scenarios when an utterance is semantically ambiguous, which cannot be achieved by the text-based SLU models. In this paper, we first introduce a new and important task, Profile-based Spoken Language Understanding (ProSLU), which requires the model that not only relies on the plain text but also the supporting profile information to predict the correct intents and slots. To this end, we further introduce a large-scale human-annotated Chinese dataset with over 5K utterances and their corresponding supporting profile information (Knowledge Graph (KG), User Profile (UP), Context Awareness (CA)). In addition, we evaluate several state-of-the-art baseline models and explore a multi-level knowledge adapter to effectively incorporate profile information. Experimental results reveal that all existing text-based SLU models fail to work when the utterances are semantically ambiguous and our proposed framework can effectively fuse the supporting information for sentence-level intent detection and token-level slot filling. Finally, we summarize key challenges and provide new points for future directions, which hopes to facilitate the research.

Code Repositories

looperxx/proslu
Official
pytorch
Mentioned in GitHub

Benchmarks

BenchmarkMethodologyMetrics
intent-detection-on-prosluGeneral SLU Model w/ Profile
Accuracy: 0.8531
slot-filling-on-prosluGeneral SLU Model w/ Profile
F1: 0.8327

Build AI with AI

From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.

AI Co-coding
Ready-to-use GPUs
Best Pricing
Get Started

Hyper Newsletters

Subscribe to our latest updates
We will deliver the latest updates of the week to your inbox at nine o'clock every Monday morning
Powered by MailChimp
Text is no more Enough! A Benchmark for Profile-based Spoken Language Understanding | Papers | HyperAI