(45-6) 15 * << * >> * Russian * English * Content * All Issues

Deep convolutional generative adversarial network-based synthesis of datasets for road pavement distress segmentation
I.A. Kanaeva 1, Yu.A. Ivanova 1, V.G. Spitsyn 1

National Research Tomsk Polytechnic University, 634050, Tomsk, Russia, Lenin Avenue, 30,
National Research Tomsk State University, 634050, Tomsk, Russia, Lenin Avenue, 36

 PDF, 2814 kB

DOI: 10.18287/2412-6179-CO-844

Pages: 907-916.

Full text of article: Russian language.

We discuss a range of problems relating to road pavement defects detection and modern approaches to their solution. The presented comparison of publicly available datasets allows one to make a conclusion that the problem of segmentation of road pavement defects in driver wide-view road images is difficult and poorly investigated. To solve this problem, we have developed algorithms for generating a synthetic dataset for cracks and potholes distress based on computer graphics methods and deep convolutional generative adversarial networks. A comparison of the accuracy of road distress segmentation was performed by training a fully convolutional neural network U-Net on real and combined datasets.

image segmentation, road pavement distress, synthetic dataset, generative adversarial network, convolutional neural network.

Kanaeva IA, Ivanova YuA, Spitsyn VG. Deep convolutional generative adversarial network-based synthesis of datasets for road pavement distress segmentation. Computer Optics 2021; 45(6): 907-916. DOI: 10.18287/2412-6179-CO-844.

The reported study was funded by RFBR according to the research project № 18-08-00977 А and in the framework of Tomsk Polytechnic University Competitiveness Enhancement Program.


  1. Azimi M, Eslamlou A, Pekcan G. Data-driven structural health monitoring and damage detection through deep learning: State-of-the-art review. Sensors 2020; 20(10): 2778. DOI: 10.3390/s20102778.
  2. Eisenbach M, Stricker R, Seichter D, Amende K, Debes K, Sesselmann M, Ebersbach D, Stoeckert U, Gross H. How to get pavement distress detection ready for deep learning? A systematic approach. IJCNN 2017: 2039-2047. DOI: 10.1109/IJCNN.2017.7966101.
  3. Stricker R, Eisenbach M, Sesselmann M, Debes K, Gross H. Improving visual road condition assessment by extensive experiments on the extended GAPs dataset. IJCNN 2019: 1-8. DOI: 10.1109/IJCNN.2019.8852257.
  4. Maeda H, Sekimoto Y, Seto T, Kashiyama T, Omata H. Road damage detection and classification using deep neural networks with smartphone images. Comput-Aided Civ Inf 2018; 33(12): 1127-1141. DOI: 10.1111/mice.12387.
  5. Zhang L, Yang F, Zhang YD, Zhu YJ. Road crack detection using deep convolutional neural network. ICIP 2016: 3708-3712. DOI: 10.1109/ICIP.2016.7533052.
  6. Ye W, Jiang W, Tong Z, Yuan D, Xiao J. Convolutional neural network for pothole detection in asphalt pavement. Road Mater Pavement Des 2021; 22(1): 42-58. DOI: 10.1080/14680629.2019.1615533.
  7. Cha Y-J, Choi W, Büyüköztürk O. Deep learning-based crack damage detection using convolutional neural networks. Comput-Aided Civ Inf 2017; 32(5): 361-378. DOI: 10.1111/mice.12263.
  8. Gopalakrishnan K, Khaitan SK, Choudhary A, Agrawal A. Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection. Constr Build Mater 2017; 157: 322-330. DOI: 10.1016/j.conbuildmat.2017.09.110.
  9. Varadharajan S, Jose S, Sharma K, Wander L, Mertz C. Vision for road inspection. IEEE Winter Conf Appl Comput Vis 2014: 115-122. DOI: 10.1109/WACV.2014.6836111.
  10. Anand S, Gupta S, Darbari V, Kohli S. Crack-pot: Autonomous road crack and pothole detection. 2018 DICTA 2018: 1-6. DOI: 10.1109/DICTA.2018.8615819.
  11. Yang F, Zhang L, Yu S, Prokhorov D, Mei X, Ling H. Feature pyramid and hierarchical boosting network for pavement crack detection. IEEE Trans Intell Transp Syst 2020; 21(4): 1525-1535. DOI: 10.1109/TITS.2019.2910595.
  12. Sobol BV, Soloviev AN, Vasiliev PV, Podkolzina LA. Deep convolution neural network model in problem of crack segmentation on asphalt images [In Russian]. Vestnik Donskogo Gosudarstvennogo Universiteta 2019; 19(1): 63-73. DOI: 10.23947/1992-5980-2019-19-1-63-73.
  13. Zhang K, Zhang Y, Cheng H-D. CrackGAN: Pavement Crack Detection Using Partially Accurate Ground Truths Based on Generative Adversarial Learning. IEEE Trans Intell Transp Syst 2021; 22(2): 1306-1319. DOI: 10.1109/TITS.2020.2990703.
  14. Liu Y, Yao J, Lu X, Xie R, Li L. DeepCrack: A deep hierarchical feature learning architecture for crack segmentation. Neurocomputing 2019; 338: 139-153. DOI: 10.1016/j.neucom.2019.01.036.
  15. Dung CV, Anh LD. Autonomous concrete crack detection using deep fully convolutional neural network. Autom Constr 2019; 99: 52-58. DOI: 10.1016/j.autcon.2018.11.028.
  16. Gao Z, Peng B, Li T, Gou C. Generative adversarial networks for road crack image segmentation. IJCNN 2019: 1-8. DOI: 10.1109/IJCNN.2019.8851910.
  17. Mei Q, Gül M. A cost effective solution for pavement crack inspection using cameras and deep neural networks. Constr Build Mater 2020; 256: 119397. DOI: 10.1016/j.conbuildmat.2020.119397.
  18. Majidifard H, Jin P, Adu-Gyamfi Y, Buttlar WG. Pavement image datasets: A new benchmark dataset to classify and densify pavement distresses. Transp Res Rec J Transp Res Board 2020; 2674(2): 328-339. DOI: 10.1177/0361198120907283.
  19. Kanaeva IA, Bolotova YA. Color and luminance corrections for panoramic image stitching. Computer Optics 2018; 42(5): 885-897. DOI: 10.18287/2412-6179-2018-42-5-885-897.
  20. Dhiman A, Klette R. Pothole detection using computer vision and learning. IEEE Trans Intell Transp Syst 2020; 21(8): 3536-3550. DOI: 10.1109/TITS.2019.2931297.
  21. Coenen TBJ, Golroo A. A review on automated pavement distress detection methods. Cogent Eng 2017; 4(1): 1374822. DOI: 10.1080/23311916.2017.1374822.
  22. Sattar S, Li S, Chapman M. Road surface monitoring using smartphone sensors: A review. Sensors 2018; 18(11): 3845. DOI: 10.3390/s18113845.
  23. Mohan A, Poobal S. Crack detection using image processing: A critical review and analysis. Alex Eng J 2018; 57(2): 787-798. DOI: 10.1016/j.aej.2017.01.020.
  24. Cao W, Liu Q, He Z. Review of pavement defect detection methods. IEEE Access 2020; 8: 14531-14544. DOI: 10.1109/ACCESS.2020.2966881.
  25. Zou Q, Cao Y, Li Q, Mao Q, Wang S. CrackTree: Automatic crack detection from pavement images. Pattern Recognit Lett 2012; 33(3): 227-238. DOI: 10.1016/j.patrec.2011.11.004.
  26. Shi Y, Cui L, Qi Z, Meng F, Chen Z. Automatic road crack detection using random structured forests. IEEE Trans Intell Transp Syst 2016; 17(12): 3434-3445. DOI: 10.1109/TITS.2016.2552248.
  27. Maeda H, Kashiyama T, Sekimoto Y, Seto T, Omata H. Generative adversarial network for road damage detection. Comput-Aided Civ Inf 2021; 36: 47-60. DOI: 10.1111/mice.12561.
  28. Tang W, Zhao Q, Huang S, Li R, Huangfu L. Iteratively optimized patch label inference network for automatic pavement disease detection. arXiv Preprint 2021. Source: <https://arxiv.org/abs/2005.13298>.
  29. Arya D, Maeda H, Ghosh SK, Toshniwal D, Mraz A, Kashiyama T, Sekimoto Y. Transfer learning-based road damage detection for multiple countries. arXiv Preprint 2020: Source: <https://arxiv.org/abs/2008.13101>.
  30. Kanaeva I, Ivanova Y. Instance segmentation of road pavement cracks [In Russian]. Proc Conf "Graphicon-2019" 2019; 1: 131-135. DOI: 10.30987/graphicon-2019-1-131-135.
  31. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B. The cityscapes dataset for semantic urban scene understanding. CVPR 2016: 3213-3223. DOI: 10.1109/CVPR.2016.350.
  32. Kluger F, Reinders C, Raetz K, Schelske P, Wandt B, Ackermann H, Rosenhahn B. Region-based cycle-consistent data augmentation for object detection. IEEE Int Conf on Big Data (Big Data) 2018: 5205-5211. DOI: 10.1109/BigData.2018.8622318.
  33. Neff T, Payer C, Štern D, Urschler M. Generative adversarial network based synthesis for supervised medical image segmentation. Proc OAGM&ARW Jt Work 2017 Vision, Autom Robot 2017: 140-145. DOI: 10.3217/978-3-85125-524-9-30.
  34. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. GAN(Generative Adversarial Nets). J Japan Soc Fuzzy Theory Intell Informatics 2017; 29: 177. DOI: 10.3156/jsoft.29.5_177_2.
  35. Odena A, Dumoulin V, Olah C. Deconvolution and checkerboard artifacts. Distill 2016; 1(10). DOI: 10.23915/distill.00003. Source: <https://distill.pub/2016/deconv-checkerboard/>.
  36. Ahirwar K. Generative adversarial networks projects: Build next-generation generative models using TensorFlow and Keras. Birmingham: Packt Publishing Ltd; 2018. ISBN: 978-1-78913-667-8.
  37. van der Maaten L, Hinton G. Visualizing data using t-SNE. J Mach Learn Res 2008; 9(86): 2579-2605.
  38. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. In Book: Navab N, Hornegger J, Wells WM, Frangi AF, eds. Medical image computing and computer-assisted intervention – MICCAI 2015. Cham: Springer; 2015: 234-241. DOI: 10.1007/978-3-319-24574-4_28.
  39. Carole HS, Wenqi L, Vercauteren T, Ourselin S, Cardoso MJ. Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations. Deep Learn Med Image Anal Multimodal Learn Clin Decis Support 2017: 240-248.
  40. Lin T-Y, Goyal P, Girshick R, He K, Dollar P. Focal loss for dense object detection. ICCV 2017: 2999-3007. DOI: 10.1109/ICCV.2017.324.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20