Benchmark of public intent recognition services.
Most of the conversational AI platforms use the natural language understanding pipeline. An essential algorithm of the pipeline is intent recognition. It is responsible for classifying input messages into classes (intents), and this way, controlling the dialog flow. These days, many intent recogniti...
| Published in: | Language Resources & Evaluation Vol. 56; no. 3; pp. 1023 - 1042 |
|---|---|
| Main Authors: | , , , , |
| Format: | Article |
| Published: |
Springer Nature
Sep2022
|
| Subjects: | |
| Online Access: | View this record in EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=158609434&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 158609434 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 1574020X 179V jtl: Language Resources & Evaluation issn: 1574020X maglogo: N pubinfo: dt: Sep2022 vid: 56 iid: 3 pid: 237 pub: Springer Nature artinfo: ui: 158609434 10.1007/s10579-021-09563-3 ppf: 1023 ppct: 19 formats: fmt: – @attributes: type: T – @attributes: type: P size: 279KB tig: atl: Benchmark of public intent recognition services. aug: au: Lorenc, Petr Marek, Petr Pichl, Jan Konrád, Jakub Šedivý, Jan affil: Faculty of Electrical Engineering, CTU Prague, Prague, Czech Republic CIIRC, CTU Prague, Prague, Czech Republic su: Web services Natural language processing Natural languages Social control sug: subj: Web services Natural language processing Natural languages Social control keyword: Intent classification Natural language understanding Online services benchmark ab: Most of the conversational AI platforms use the natural language understanding pipeline. An essential algorithm of the pipeline is intent recognition. It is responsible for classifying input messages into classes (intents), and this way, controlling the dialog flow. These days, many intent recognition services are currently also available as web services or as open-source alternatives. However, there is a limited number of datasets to compare their performance in a scenario of noised input. Additionally, many factors, such as CPU and memory requirements, make selecting the right approach challenging in practice. This paper presents a novel CIIRC dataset for evaluating the impact of noise (sentence disfluencies and filler sentences) on intent recognition algorithms. The data set focuses on command control as well as on social topics. We suggest criteria for selecting the best intent recognition algorithm. Finally, we use the suggested criteria and the new CIIRC dataset to compare the selected public intent recognition services with popular open-source algorithms. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Language Resources & Evaluation is a copyright of Springer, 2022. All Rights Reserved. item: Language Resources & Evaluation holder: Springer Nature dt: @attributes: year: 2022 holdings: @attributes: islocal: N |
|---|