Deep transfer learning CNN based approach for COVID-19 detection

Recently, the novel coronavirus (Covid-19) and its different variants have spread rapidly across the world. Early-stage detection of COVID-19 is a challenging task due to the limited availability of Covid testing kits to the public. Conventionally, reverse transcription-polymerase chain reaction (RT-PCR) is the reliable test for the detection of COVID-19 which is time-consuming and costly. The aim of this work is to identify the COVID-19 symptoms with the help of a deep learning algorithm using chest X-Ray images. In order to improve the quality of chest X-Ray images, authors have further modified the pre-trained model with some extra CNN layers, such as the first layer is the average pooling layer and the other two are dense layers followed by ReLU with softmax activation function. The experimental results have been carried out on publicly available chest X-Ray images of COVID-19 to mark COVID-19 patients as positive and negative datasets. For evaluation purpose, we have used benchmark of pre-trained models such as VGG-16 (Visual Geometry Group), VGG19, Xception, ResNet152, ResNet152v2, ResNet101, ResNet101v2, DenseNet201, DenseNet169 and DenseNet121. On the benchmark datasets, viz. COVID-19 X-Ray images, an average improvement in terms of training/validation accuracy, precision, recall, and F1-scores scores were 95%, 94%, 99/88%, 99/88%, and 93/92% respectively. The results provide sufficient evidence that deep learning can be used efficiently for the detection of COVID-19 symptoms.


Introduction
*The Coronavirus disease endures by the presence of acute syndrome coronavirus 2 (SARS-CoV-2) first emerged in 2019 at Wuhan, China which has dispersed to the other countries with a commute of infected people (Wu and McGoogan, 2020), and within no time, this disease has taken the shape of a global pandemic. The number of reported cases has reached 25 million as of early September 2020 with more than 0.86 million deaths worldwide because of the COVID-19. World Health Organization (WHO) terms the formal name of this virus to be coronavirus disease , initially called "2019 novel coronavirus" (Strzelecki, 2020). The symptoms of COVID-19 include runny nose, fever, headache, and cough (Covid et al., 2020). This virus has caused deaths in people, those particularly with a weak immune system (Singhal, 2020). It has caused serious implications on our daily life. The rapid increase in the number of cases leads to the urgency of fast diagnosis, effective testing, and enduring treatment. To control the spread of disease, there is a need for a fast diagnosis with accuracy so that people carrying COVID can be isolated in time, thus the spread can be slowed down. Presently, the detection of COVID-19 in the medical field is attained by reverse transcription-polymerase chain reaction (RT-PCR) based testing which is recommended as an effective diagnostic approach now. However, there are some fallbacks that include the limited sensitivity and scarcity in the availability of testing kits in the pandemic region (especially in developing countries) which has not only proved to be burdensome for the medical staff but also ignites the spread of the disease. Recent research in the radiological field shows that computed tomography (CT) can be applied for the diagnosis of COVID-19 due to its high sensitivity and accuracy as compared to RT-PCR. The COVID-19 affects the lungs like pneumonia but induces lesions in the bilateral lung which is composed of groundglass opacities (GGO), which has highlighted the significance of Chest X-ray (CXR) images or CT images for the identification of COVID-19. CT scan provides a convenient and economical solution as a diagnostic tool that is fast and reliable, thus presenting an alternative for RT-PCR. The diagnostic model can become more effective and reliable when incorporated with modern techniques of artificial intelligence and machine learning, thus preventing human error, and generating massive results within no time considering the pandemic situation.
The rest of the paper is organized as follows. Section 2 gives a brief introduction to related work. Section 3 discusses the proposed transfer learning model. Experimental results are presented in Section 4 and in Section 5, the Conclusion and Future work are discussed.

Related works
The World Health Organization (WHO) announced in February 2020 that a new virus known as COVID-19 had begun to spread quickly in various countries. COVID-19 is usually diagnosed because of pneumonia-like symptoms, which can be detected through genetic and imaging studies. Deep learningbased convolutional neural networks (CNNs) has significant applications in various field such as image super-resolution, satellite imaging, security surveillance, and medical image classification tasks (Tajbakhsh et al., 2016;Peled and Yeshurun, 2001;Muhammad et al., 2020;2021a;2021b;Muhammad and Aramvith, 2019;Shi et al., 2013). Promising results of deep learning in the field of medical diagnosis have urged scientists to use it for COVID-19 detection as well. For, instance, some researchers have deployed a deep convolutional neural network technique to perceive the indication of COVID-19. Sethy and Behera (2020) deployed a novel methodology for discontenting and categorizing pneumonia infection from X-ray images. The primarily trained models are incorporated to retrieve the characteristics from X-ray images. Ismael and Şengür (2021) used deep learning-based approaches for COVID-19 detection on the basis of chest X-ray images. The authors used the fine-tuned ResNet50 model and recorded it to 92.6% accuracy. The features obtained from ResNet50 (He et al., 2016) with support vector machine (SVM) classifier (Qian et al., 2010) produce reliable results. Wang et al. (2020a) introduced a model of deep learningbased CNN for coronavirus detection which they call COVID-NET. In the proposed model, authors randomly select a medical image for COVID-19 detection and achieved 85.2% accuracy with 0.83 specificities and 0.67 of sensitivity. A model of VB-Net deep learning is proposed by Shan et al. (2020) which integrates to segment the infection regions of COVID-19 patients within CT scan images. The training dataset used the images of 549 with different scenarios of disease obtained from different sources. The achieved accuracy is up to 91.6%. Hemdan et al. (2020) presented 7 different deep neural network architectures for COVID-19 classification. In Jain et al. (2021), the authors used 3 pre-trained models like Inceptionv3, Xception, and ResNet models to maximize the accuracy up to 97.97%, but not include the VGG19 (Simonyan and Zisserman, 2014) and DenseNet201 (Huang et al., 2017) architectures. Stephen et al. (2019) proposed a model that detects and classifies cases of pneumonia. The training dataset depends on the collection of chest X-ray images (Kermany et al., 2018). The experimental results report 2.88% training loss, 95.31% training accuracy, validation loss is 18.35% and validation accuracy is 93.73%. In Varshni et al. (2019), the authors used five pretrained convolutional neural network models, VGG16 Network architecture (Simonyan and Zisserman, 2014), ResNet50 (He et al., 2016), DenseNet-121 (Huang et al., 2017), DenseNet-169 and Xception Network architecture to extract the features. The pre-trained networks are used as a feature extractor followed by a different type of classifier which encloses Random Forest, K-nearest neighbor, Naıve Bayes, and SVM incorporated for the identification of pneumonia from X-ray images. In Narin et al. (2021), the transfer learning technique has been deployed for the identification of COVID-19. They utilized an initially trained model of Inception-v3 (Szegedy et al., 2016) and ResNet50. Experimental results showed that the ResNet50 model presented the best results when compared to Inception-v3.
Apostolopoulos and Mpesiana (2020) proposed a medical image-based deep learning neural network architecture. In this study, the authors split the dataset into three groups depending upon classification and nature which include a normal, pneumonia, and COVID-19 dataset. During their study, they have integrated five primarily trained models and accomplished accuracy up to 98.75% for the binary categorization task. Khan et al. (2020) proposed a model known as CoroNet, which is based on Xception pre-trained architecture. The authors claimed 89.6% overall accuracy on four different classes. Homayouni et al. (2021) used the deep learning approach to detect the anomaly in COVID-19 time-series datasets. Ayan and Ünver (2019) proposed the Xception (Chollet, 2017) and VGG16 (Simonyan and Zisserman, 2014) based deep convolutional neural network models which are based on Pneumonia chest X-ray images. In their work, a dataset containing nearly 5800 chest X-ray images is retrieved from the Kermany et al. (2018). Their approach classifies 4200 images as pneumonia and the remaining images as a pneumonia-free case. The investigational results for VGG-16 network architecture report 85% sensitivity, and 86% precision.

Proposed transfer learning model
The design and development of a new model based on CNN are complex, and time-consuming tasks. Integrating primarily trained models is a quick and efficient method to be adopted as compared to the consideration of developing a new CNN model (Özcan and Baştürk, 2019). In this paper, the authors proposed a new transfer learning type model that integrates a deep CNN framework for the identification of the COVID status through X-ray images. Fig. 1 presents an overall network architecture of the proposed method and integrates five initially trained deep CNN architectures; namely VGG16, VGG19, ResNet50, Inceptionv3, and Xception with the addition of three layers. The first layer is the average pooling layer and the other two are dense layers followed by ReLU and Softmax activation function. The last layer of each pre-trained model is removed and replaced by the trainable part which consists of a fully connected/dense layer with a dropout is 0.5. For binary classification purposes. This layer has 2 units with sigmoid output to classify the output of COVID-19 as a Positive or Negative Class. Some characteristics of initially trained models are demonstrated in Table 1. The details of pretrained deep CNN models are given in Table 1.

VGG16 and VGG19
VGG16 and VGG19 (Simonyan and Zisserman, 2014) is a deep CNN type network architecture proposed by Simonyan and Zisserman (2014) and successfully trained on the ImageNet dataset (Deng et al., 2009) consists of 14 million images belongs to 1000 different types of classes. In the ILSVRC-2014, (Russakovsky et al., 2015) the model achieves 92.7% top-5 test accuracy. It replaces the large kernel size of AlexNet (Krizhevsky et al., 2012) with 3x3 filters. The training time of VGG16 and VGG19 is completed in weeks and the NVIDIA Titan GPUs have been utilized. The network depth of VGG16 and VGG19 has 16 and 19 layers.

ResNet50
ResNet50 (He et al., 2016) is a deep convolutional neural type network that depends on 50 deep layers. ResNet50 is trained on more than one million images obtained from the database of ImageNet (Deng et al., 2009). The model can classify different images into 1000 categories, such as animals, pencils, mice, and keyboards. The input size of the pre-trained network is 224-by-224.

InceptionV3
Inceptionv3 (Szegedy et al., 2016) is a deep CNN network that has 48 deep CNN layers and is trained on the ImageNet database (Deng et al., 2009). The pre-trained model can classify the images into different 1000 types of subcategories, like animals, keyboard, pencil, and mouse. Millions of images provide rich feature representations. The input size of the image in the network is 299-by-299.

Xception
In 2017, Google introduced the new architecture named Xception network (Chollet, 2017), which stands for the extreme version of an Inception. In this architecture, the standard convolution operation is replaced by depth-wise separable convolution, which is better than Inceptionv3. Xception model is trained on the ImageNet dataset as well as on the JFT dataset (Chowdhury et al., 2020). Xception network achieves 0.790 Top1-accuracy and 0.945 Top-5 accuracy. The input size of the image in the network is 299-by-299.

Experimental results
The proposed transfer learning type model is designed in Python 3.7 using deep learning library Keras backend as a TensorFlow. All experimental results are conducted on the computer of the Intel Xeon processor (2 GHz) with a RAM of 16 GB. All pre-trained model weights are updated with the mini-batch size is 32 and the initial learning rate is 0.0003. The proposed model has been tested in the two classes, such as COVID-19 Positive and COVID-19 Negative class.

Collection of data
For the experimental procedure and assessment, we have deployed two categories of image sets which enclose COVID-Negative and COVID-Positive.
Since COVID-19 is a new research area, therefore the number of images related to COVID-19 is still limited. To overcome this limitation, we combine two available datasets that are publicly accessible consisting of COVID-19(COVID-Positive) and normal images with no COVID generally named as COVID-Negative. The said dataset for COVID-19 X-ray images is openly made available by Cohen et al. (2020). The authors gathered the images from different open access sources. The next dataset of COVID-19 images is developed by medical doctors' teams from Bangladesh and Qatar University (Cohen et al., 2020). The recent version of the COVID-19 image dataset having more than two hundred X-ray images is accessible from the Kaggle website. In the training phase, the dataset of 200 COVID-Negative images and 230 of COVID-Positive images are considered. In the test dataset, COVID-Negative is 150 images and 203 is the COVID-Positive images.

Data augmentation
While classifying both traditional as well as deep learning approaches, the training dataset has an important role in determining the performance of the model. In order to enhance the performance and to overcome the issue of over-fitting, we have used data augmentation techniques in the form of rotation, flipping, and shearing as shown in Fig. 2.

Feature map visualization of COVID-19 image in the layers
In Fig. 3, we analyzed the segment of the image for which the CNN by complementing the segments in the original images. The values in the activation maps can be normalized between the range of 0 and 1, in case they have a different range.

Evaluation parameters
Accuracy is a performance evaluation metric to predict the correctness of the classification model and is given by Eq. 1.
Sensitivity is the ability of a test to correctly identify all those who have positive COVID-19 and is given by Eq. 2.
Precision means how many samples are classified correctly as a positive outcome of all positives and is calculated by Eq. 3.
Specificity is the ratio between how many samples are correctly classified and how many samples are negatives and is calculated by Eq. 4.
F1-score also known as F-score is to measures the accuracy of tests. F-score is calculated from the value of recall and precision and has the highest value is 1, which indicates the perfect measurement of recall and precision.
In Table 2, we evaluate the performance of two classes, such as COVID-Negative and COVID-Positive. All transfer learning-based deep CNN model achieves different precision, recall, and F1 scores.

Comparison evaluation with state-of-the-artmethods
Comparison analysis of the proposed method with other state-of-the-art methods is presented in Tables 2 and 3. Results are clearly seen that the proposed methodology obtained remarkable performance in terms of quality metrics.  Furthermore, training accuracy and loss curves as shown in Fig. 4. The training of all pre-trained transfer learning models has been carried out up to 500 epochs. It can be seen from Fig. 4 that the highest training accuracy is obtained by Xception models. In the loss curves, it can be observed that the loss values decrease in the pre-trained models during the initial training stage. It can be said that the Xception model loss values decrease rapidly approaches to zero.

Conclusion and future work
The drastic increment in the number of patients affected with COVID-19 and its worldwide spread demands an efficient diagnosis and testing to overcome the pandemic. Early-stage diagnosis of this virus will help people with fast recovery and slow down the process of spreading. Computed tomography (CT) when integrated with advanced machine learning approaches can produce good results with considerable accuracy thus eradicating the cost and need for extensive medical staff. In this paper, we have used a transfer learning approach with two categories of X-ray images named COVID-19 negative and COVID-19 positive. Transfer learning provides a superior response in terms of precision, sensitivity, and F1 score. In future work, authors have used transfer learning of the deep CNN model approach with a larger benchmark training as well as a testing dataset.