Clinical applicability of COVID-19 detection in chest radiography with deep learning - radiologist annotations

<p>Chest radiography (CXR) can be used as a complementary method for diagnosing/following COVID-19 patients. However, experience level and workload of technicians and radiologists may affect the decision process. In order to evaluate the performance of radiologists in the detection of COVID-19 radiological features in CXR, manual annotation of a subset of random CXR images was performed by two radiologists using an in-house software. The software presented CXRs from a randomly selected subset and allowed for window center/width adjustment, zooming and panning. Radiologists were asked to label CXRs into one of 4 classes: Normal, Not indicative of COVID-19 (pathological), Indicative of COVID-19 and Undetermined. The Indicative of COVID-19 class was defined as CXRs where the patient presented findings indicative of COVID-19, namely bilateral pulmonary opacities of low/medium density. The Undetermined class was defined as CXRs where the patient presented findings that could be indicative of COVID-19 but which could also be indicative of another condition, namely unilateral lung opacities, diffuse bilateral opacities of ARDS pattern or diffuse reticular opacities. The Not indicative of COVID-19 (pathological) class was defined as CXRs where the patient presented findings indicative of any other pathology except for COVID-19. CXRs where the patient presented medical devices were classified as Normal if the underlying pathology was not visible. Additionally, CXRs without sufficient quality for visual assessment by the radiologists due to bad image quality, patient positioning or any other factors could be labelled as Compromised for exclusion. A total of 1,845 CXRs were selected for annotation by two radiologists. Of these, 1,256 belong to the Mixed dataset(a combination of multiple public sources), 289 belong to BIMCV and 300 belong to COVIDGR. Selection of CXRs for annotation was performed randomly: for the Mixed dataset, a balanced selection strategy was used during image selection, whereas for BIMCV and COVIDGR, the dataset class distribution was maintained in the subset selected for annotation. Manual labelling of CXRs was performed in two stages. First, both radiologists independently classified each CXR. CXRs where the two radiologists disagreed were then selected for the second stage where the two radiologists assessed the CXRs together to achieve consensus. At no point were radiologists given access to the ground truth label, RT-PCR results or any other information besides the CXR image. To ensure that written information present in the CXR image (such as hospital system, health service, laterality markers, patient positioning, etc.) did not bias the annotation, all written labels were blacked out during before annotation. This dataset contains the information regarding each annotated CXR and the resulting radiologist labels for each image before and after consensus.</p> <p>The public datasets used in this study are available in the following repositories:</p> <p>CheXpert:&nbsp;<a href="https://stanfordmlgroup.github.io/competitions/chexpert/">https://stanfordmlgroup.github.io/competitions/chexpert/</a></p> <p>ChestXRay-8:&nbsp;<a href="https://nihcc.app.box.com/v/ChestXray-NIHCC">https://nihcc.app.box.com/v/ChestXray-NIHCC</a></p> <p>COVID-19 IDC:&nbsp;<a href="https://github.com/ieee8023/covid-chestxray-dataset">https://github.com/ieee8023/covid-chestxray-dataset</a></p> <p>COVIDx:&nbsp;<a href="https://github.com/lindawangg/COVID-Net">https://github.com/lindawangg/COVID-Net</a></p> <p>RSNA-PDC:&nbsp;<a href="https://www.kaggle.com/c/rsna-pneumonia-detection-challenge">https://www.kaggle.com/c/rsna-pneumonia-detection-challenge</a></p> <p>SAVE LIVES:&nbsp;<a href="https://www.hmhospitales.com/coronavirus/covid-data-save-lives">https://www.hmhospitales.com/coronavirus/covid-data-save-lives</a></p> <p>SERAM:&nbsp;<a href="https://seram.es/images/site/TUTORIAL_CSI_RX_TORAX_COVID-19_vs_4.0.pdf">https://seram.es/images/site/TUTORIAL_CSI_RX_TORAX_COVID-19_vs_4.0.pdf</a></p> <p>Twitter:&nbsp;<a href="https://twitter.com/ChestImaging">https://twitter.com/ChestImaging</a></p> <p>BIMCV PADCHEST:&nbsp;<a href="https://bimcv.cipf.es/bimcv-projects/padchest/">https://bimcv.cipf.es/bimcv-projects/padchest/</a></p> <p>BIMCV COVID-19-PADCHEST:&nbsp;<a href="https://bimcv.cipf.es/bimcv-projects/bimcv-covid19/">https://bimcv.cipf.es/bimcv-projects/bimcv-covid19/</a></p> <p>COVIDGR:&nbsp;<a href="https://dasci.es/transferencia/open-data/covidgr-2/">https://dasci.es/transferencia/open-data/covidgr-2/</a></p>