Identifying and characterizing highly similar notes in big clinical note datasets.
Background: Big clinical note datasets found in electronic health records (EHR) present substantial opportunities to train accurate statistical models that identify patterns in patient diagnosis and outcomes. However, near-to-exact duplication in note texts is a common issue in many clinical note da...
| Publicado en: | Journal of Biomedical Informatics Vol. 82; pp. 63 - 70 |
|---|---|
| Autores principales: | , , , |
| Formato: | research Journal Article |
| Publicado: |
Academic Press Inc.
Jun2018
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=130046880&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 130046880 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 15320464 OMB jtl: Journal of Biomedical Informatics issn: 15320464 maglogo: N pubinfo: dt: Jun2018 vid: 82 pid: 735 pub: Academic Press Inc. place: Burlington, Massachusetts artinfo: ui: 130046880 130046880 NLM29679685 130046880 10.1016/j.jbi.2018.04.009 NLM29679685 130046880 ppf: 63 ppct: 7 formats: tig: atl: Identifying and characterizing highly similar notes in big clinical note datasets. aug: au: Gabriel, Rodney A. Kuo, Tsung-Ting McAuley, Julian Hsu, Chun-Nan affil: UCSD Health Department of Biomedical Informatics, University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093, USA sug: subj: Medical Informatics Methods Data Collection Obesity, Morbid Epidemiology Information Science Algorithms Computers and Computerization Natural Language Processing Obesity, Morbid Diagnosis Human Resource Databases Cluster Analysis Funding Source ab: Background: Big clinical note datasets found in electronic health records (EHR) present substantial opportunities to train accurate statistical models that identify patterns in patient diagnosis and outcomes. However, near-to-exact duplication in note texts is a common issue in many clinical note datasets. We aimed to use a scalable algorithm to de-duplicate notes and further characterize the sources of duplication.Methods: We use an approximation algorithm to minimize pairwise comparisons consisting of three phases: (1) Minhashing with Locality Sensitive Hashing; (2) a clustering method using tree-structured disjoint sets; and (3) classification of near-duplicates (exact copies, common machine output notes, or similar notes) via pairwise comparison of notes in each cluster. We use the Jaccard Similarity (JS) to measure similarity between two documents. We analyzed two big clinical note datasets: our institutional dataset and MIMIC-III.Results: There were 1,528,940 notes analyzed from our institution. The de-duplication algorithm completed in 36.3 h. When the JS threshold was set at 0.7, the total number of clusters was 82,371 (total notes = 304,418). Among all JS thresholds, no clusters contained pairs of notes that were incorrectly clustered. When the JS threshold was set at 0.9 or 1.0, the de-duplication algorithm captured 100% of all random pairs with their JS at least as high as the set thresholds from the validation set. Similar performance was noted when analyzing the MIMIC-III dataset.Conclusions: We showed that among the EHR from our institution and from the publicly-available MIMIC-III dataset, there were a significant number of near-to-exact duplicated notes. pubtype: Academic Journal doctype: research Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|