Semantic coherence markers: The contribution of perplexity metrics.
Devising automatic tools to assist specialists in the early detection of mental disturbances and psychotic disorders is to date a challenging scientific problem and a practically relevant activity. In this work we explore how language models (that are probability distributions over text sequences) c...
| Publicado en: | Artificial Intelligence in Medicine Vol. 134 |
|---|---|
| Autores principales: | , , , , |
| Formato: | Journal Article |
| Publicado: |
Elsevier B.V.
Dec2022
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=160558797&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 160558797 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 09333657 3HY jtl: Artificial Intelligence in Medicine issn: 09333657 maglogo: N pubinfo: dt: Dec2022 vid: 134 pid: 1004 pub: Elsevier B.V. artinfo: ui: 160558797 160558797 NLM36462890 10.1016/j.artmed.2022.102393 NLM36462890 160558797 ppct: 1 formats: tig: atl: Semantic coherence markers: The contribution of perplexity metrics. aug: au: Colla, Davide Delsanto, Matteo Agosto, Marco Vitiello, Benedetto Radicioni, Daniele P. affil: University of Turin, Computer Science Department, Italy sug: subj: Alzheimer's Disease Diagnosis Semantics Linguistics Benchmarking Scales ab: Devising automatic tools to assist specialists in the early detection of mental disturbances and psychotic disorders is to date a challenging scientific problem and a practically relevant activity. In this work we explore how language models (that are probability distributions over text sequences) can be employed to analyze language and discriminate between mentally impaired and healthy subjects. We have preliminarily explored whether perplexity can be considered a reliable metrics to characterize an individual's language. Perplexity was originally conceived as an information-theoretic measure to assess how much a given language model is suited to predict a text sequence or, equivalently, how much a word sequence fits into a specific language model. We carried out an extensive experimentation with healthy subjects, and employed language models as diverse as N-grams - from 2-grams to 5-grams - and GPT-2, a transformer-based language model. Our experiments show that irrespective of the complexity of the employed language model, perplexity scores are stable and sufficiently consistent for analyzing the language of individual subjects, and at the same time sensitive enough to capture differences due to linguistic registers adopted by the same speaker, e.g., in interviews and political rallies. A second array of experiments was designed to investigate whether perplexity scores may be used to discriminate between the transcripts of healthy subjects and subjects suffering from Alzheimer Disease (AD). Our best performing models achieved full accuracy and F-score (1.00 in both precision/specificity and recall/sensitivity) in categorizing subjects from both the AD class, and control subjects. These results suggest that perplexity can be a valuable analytical metrics with potential application to supporting early diagnosis of symptoms of mental disorders. pubtype: Academic Journal doctype: Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|