Variabilities in Reference Standard by Radiologists and Performance Assessment in Detection of Pulmonary Embolism in CT Pulmonary Angiography.
Annotating lesion locations by radiologists' manual marking is a key step to provide reference standard for the training and testing of a computer-aided detection system by supervised machine learning. Inter-reader variability is not uncommon in readings even by expert radiologists. This study evalu...
| Publicado en: | Journal of Digital Imaging Vol. 32; no. 6; pp. 1089 - 1097 |
|---|---|
| Autores principales: | , , , , , , , |
| Formato: | diagnostic images pictorial research tables/charts Journal Article |
| Publicado: |
Springer Nature
Dec2019
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=139568181&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 139568181 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 08971889 DOQ jtl: Journal of Digital Imaging issn: 08971889 maglogo: N pubinfo: dt: Dec2019 vid: 32 iid: 6 pid: 237 pub: Springer Nature place: New York, New York artinfo: ui: 139568181 139568181 139568181 10.1007/s10278-019-00228-w 139568181 ppf: 1089 ppct: 8 formats: fmt: – @attributes: type: T – @attributes: type: P tig: atl: Variabilities in Reference Standard by Radiologists and Performance Assessment in Detection of Pulmonary Embolism in CT Pulmonary Angiography. aug: au: Zhou, Chuan Chan, Heang-Ping Chughtai, Aamer Patel, Smita Kuriakose, Jean Hadjiiski, Lubomir M. Wei, Jun Kazerooni, Ella A. affil: Department of Radiology, University of Michigan, 1500 E. Medical Center Drive, 48109, Ann Arbor, MI, USA sug: subj: Pulmonary Embolism Diagnosis Diagnosis, Computer Assisted Computed Tomography Angiography Clinical Competence Evaluation Radiologists Psychosocial Factors Human Machine Learning Consensus Weights and Measures Interrater Reliability ab: Annotating lesion locations by radiologists' manual marking is a key step to provide reference standard for the training and testing of a computer-aided detection system by supervised machine learning. Inter-reader variability is not uncommon in readings even by expert radiologists. This study evaluated the variability of the radiologist-identified pulmonary emboli (PEs) to demonstrate the importance of improving the reliability of the reference standard by a multi-step process for performance evaluation. In an initial reading of 40 CTPA PE cases, two experienced thoracic radiologists independently marked the PE locations. For markings from the two radiologists that did not agree, each radiologist re-read the cases independently to assess the discordant markings. Finally, for markings that still disagreed after the second reading, the two radiologists read together to reach a consensus. The variability of radiologists was evaluated by analyzing the agreement between two radiologists. For the 40 cases, 475 and 514 PEs were identified by radiologists R1 and R2 in the initial independent readings, respectively. For a total of 545 marks by the two radiologists, 81.5% (444/545) of the marks agreed but 101 marks in 36 cases differed. After consensus, 65 (64.4%) and 36 (35.6%) of the 101 marks were determined to be true PEs and false positives (FPs), respectively. Of these, 48 and 17 were false negatives (FNs) and 14 and 22 were FPs by R1 and R2, respectively. Our study demonstrated that there is substantial variability in reference standards provided by radiologists, which impacts the performance assessment of a lesion detection system. Combination of multiple radiologists' readings and consensus is needed to improve the reliability of a reference standard. pubtype: Academic Journal doctype: diagnostic images pictorial research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|