Automatic generation of structured radiology reports for volumetric computed tomography images using question-Specific deep feature extraction and learning

Samira Loveymi, Mir Hossein Dezfoulian, Muharram Mansoorizadeh

DOI: 10.4103/jmss.JMSS_21_20


Background: In today's modern medicine, the use of radiological imaging devices has spread at medical centers. Therefore, the need for accurate, reliable, and portable medical image analysis and understanding systems has been increasing constantly. Accompanying images with the required clinical information, in the form of structured reports, is very important, because images play a pivotal role in detect, planning, and diagnosis of different diseases. Report-writing can be exposure to error, tedious and labor-intensive for physicians and radiologists; to address these issues, there is a need for systems that generate medical image reports automatically and efficiently. Thus, automatic report generation systems are among the most desired applications. Methods: This research proposes an automatic structured-radiology report generation system that is based on deep learning methods. Extracting useful and descriptive image features to model the conceptual contents of the images is one of the main challenges in this regard. Considering the ability of deep neural networks (DNNs) in soliciting informative and effective features as well as lower resource requirements, tailored convolutional neural networks and MobileNets are employed as the main building blocks of the proposed system. To cope with challenges such as multi-slice medical images and diversity of questions asked in a radiology report, our system develops volume-level and question-specific deep features using DNNs. Results: We demonstrate the effectiveness of the proposed system on ImageCLEF2015 Liver computed tomography (CT) annotation task, for filling in a structured radiology report about liver CT. The results confirm the efficiency of the proposed approach, as compared to classic annotation methods. Conclusion: We have proposed a question-specific DNN-based system for filling in structured radiology reports about medical images.


Convolutional neural network, medical image analysis, MobileNet, radiology report generation

Full Text:



Xue Y, Tao X, Rodney LL, Zhiyun X, Sameer A, Thoma Gr, et al. Multimodal recurrent model with attention for automated radiology report generation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer; 2018.

Lau JJ, Gayen S, Ben Abacha A, Demner-Fushman D. A dataset of clinically generated visual questions and answers about radiology images. Sci Data 2018;5:180251.

Wang, J, Hairong L, Rui J, Zhen X. Rule-Based Method to Develop Question-Answer Dataset from Chest X-Ray Reports. In: 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS). Cordoba, Spain: IEEE; 2019.

Loveymi S, Dezfoulian MH, Mansoorizadeh M. Generate structured radiology report from CT images using image annotation techniques: Preliminary results with liver CT. J Digit Imaging 2020;33:375-90.

Nedjar, I, Mahmoudi S, Chikh MA, Abi-Yad Kh, Bouafia Z. Automatic Annotation of Liver CT Image: ImageCLEFmed 2015. CLEF (Working Notes); 2015.

Nedjar I, Mahmoudi S, Chikh MA. Content-based Medical Image Tetrieval for Liver CT Annotation. Transactions on Machine Learning and Artificial Intelligence; 2017. p. 5.

Camlica Z, Tizhoosh HR, Khalvati F. Medical image classification via SVM using LBP features from saliency-based folded data. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). Miami, Florida, USA: IEEE; 2015.

Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks.In: Advances in Neural Information Processing Systems(NIPS) Nevada,USA; 2012.

Singh S, Ho-Shon K, Karimi S. Hamey L. modality classification and concept detection in medical images using deep transfer learning. In: 2018 International Conference on Image and Vision Computing New Zealand (IVCNZ). IEEE; 2018.

Afshar P, Mohammadi A, Plataniotis KN, Oikonomou A, Benali H. From handcrafted to deep-learning-based cancer radiomics: Challenges and opportunities. IEEE Signal Processing Magazine 2019;36:132-60.

Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE. A survey of deep neural network architectures and their applications. Neurocomputing 2017;234:11-26.

Marvasti NB, García MD, Üsküdarli S, Montes JF, Acar B. Overview of the ImageCLEF 2015 liver CT annotation task. In: CLEF (Working Notes). Workshop Proceedings. CEUR-¬, no. 1613-¬0073, Toulouse, France; 2015.

Kumar A, Dyer S, Kim J, Li C, Leong PH, Fulham M, et al. Adapting content-based image retrieval techniques for the semantic annotation of medical images. Comput Med Imaging Graph 2016;49:37-45.

Haralick RM, Shanmugam K, Dinstein IH. Textural Features for Image Classification. IEEE Transactions on Systems, Man, and Cybernetics; 1973. p. 610-21.

Lee TS. Image representation using 2D Gabor wavelets. IEEE Trans Pattern Analysis Mach Intellig 1996;18:959-71.

Spanier AB, Caplan N, Sosna J, Acar B, Joskowicz L. A fully automatic end-to-end method for content-based image retrieval of CT scans with similar liver lesion annotations. Int J Comput Assist Radiol Surg 2018;13:165-74.

Liu M, Zhang J, Nie D, Yap PT, Shen D. Anatomical landmark based deep feature representation for MR images in brain disease diagnosis. IEEE J Biomed Health Inform 2018;22:1476-85.

Jing B, Xie P, Xing E. On the automatic generation of medical imaging reports. arXiv:1711.08195v3:2017.

Zhang Z, Xie Y, Xing F, McGough M, Yang L. Mdnet: A Semantically and Visually Interpretable Medical Image Diagnosis Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017.

Sugimori H. Classification of Computed Tomography Images in Different Slice Positions Using Deep Learning. J Healthc Eng 2018;2018:1753480.

Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. Going Deeper with Convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2015.

Holland L, Wei D, Olson KA, Mitra A, Graff JP, Jones AD, et al. Limited number of cases may yield generalizable models, a proof of concept in deep learning for colon histology. J Pathol Inform 2020;11:5.

Chen YC, Hong DJ, Wu CW, Mupparapu M. The use of deep convolutional neural networks in biomedical imaging: A review. J Orof Sci 2019;11:3.

Baltruschat IM, Nickisch H, Grass M, Knopp T, Saalbach A. Comparison of deep learning approaches for multi-label chest X-Ray classification. Sci Rep 2019;9:6381.

Simonyan K, Zisserman A, Very deep convolutional networks for large-scale image recognition. In Proc. International Conference on Learning Representations :2014.

He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016. p. 770-8.

Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the Inception Architecture for Computer Vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016.

Chollet F. Xception: Deep Learning with Depthwise Separable Convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017.

Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 2017.

Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception-v4, Inception-Resnet and the Impact of Residual Connections on Learning. In: Thirty-First AAAI Conference on Artificial Intelligence; 2017.

Loveymi S, Dezfoulian MH, Mansoorizadeh M. Generate structured radiology report from liver CT images using fusion of mobilenet and local binary pattern. J Mach Vision Image Proc Forthcoming 2020;(Published Online, In Persian).

Erhan D, Courville A, Bengio Y, Vincent P. Why does unsupervised pre-training help deep learning? In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 210: p. 201-8.

Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC. Mobilenetv2: Inverted Residuals and Linear Bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018.

Li Y, Huang H, Xie Q, Yao L, Chen Q. Research on a surface defect detection algorithm based on MobileNet-SSD. Applied Sci 2018;8:1678.

Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167v3 2015.

Dahl GE, Sainath TN, Hinton GE. Improving deep neural networks for LVCSR using rectified linear units and dropout. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. Vancouver, Canada: IEEE; 2013.

Baldi P, Sadowski PJ. Understanding Dropout. In: Advances in Neural Information Processing Systems; 2013.

Warde-Farley D, Goodfellow IJ, Courville A, Bengio Y. An empirical analysis of dropout in piecewise linear networks. arXiv preprint arXiv:1312.6197 2013.

Goodfellow I, Bengio Y, Courville A. Deep Learning. MIT press; 2016.

Garcia-Gasulla D, Parés F, Vilalta A, Moreno J, Ayguadé E, Labarta J. On the behavior of convolutional nets for feature extraction. J Artif Intellig Res 2018;61:563-92.

LeCun Y. LeNet-5, Convolutional Neural Networks. URL. Available from: [Last accessed on 2020 Oct 06].

Ketkar N, Santana E. Deep Learning with Python. Berkeley, CA: Apress; 2017.

Glorot X, Bengio Y. Understanding the Difficulty of Training Deep Feed for Ward Neural Networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics; 2010.

Sutskever I, Martens J, Dahl G, Hinton G. On the Importance of Initialization and Momentum in Deep Learning. In: International Conference on Machine Learning; 2013.

Kingma DP, Ba J. Adam: A method for stochastic optimization. In ICLR, 2015.

Tieleman T, Hinton G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA. Neural Network Mach Learn 2012;4:26-31.


  • There are currently no refbacks.