Rock Particle Motion Information Detection Based on Video Instance Segmentation
Abstract
:1. Introduction
- Because the objects and backgrounds have a high degree of similarity, we improved the Mask R-CNN to develop the feature recognition ability and the instance segmentation effect of particles by designing a classification loss function based on Arcface loss.
- The multiparticle VIS with a low ID switching rate was achieved by combining the improved Mask R-CNN and SORT.
- We utilized the equivalent ellipse characterization method to process the segmented particles, combining characterization results with the proportional calibration algorithm to detect its translation and detecting the rotation by calculating the change in the angle of the major axis of the ellipse.
- We verified the effectiveness of the method through segmentation experiments, VIS, and motion information detection. The experimental results showed that the improved Mask R-CNN had a better detection and segmentation effect. The ID switching rate of the VIS was low. This method can successfully detect the movement information of rock particles. The average detection errors of translation and rotation are 5.10% and 14.49%, respectively.
2. Related Work
2.1. Traditional Particle Segmentation Method
2.2. Particle Segmentation Method Based on Deep Learning
2.3. The Image-Based Measurement of Deformation and Stresses
3. Motion Information Detection of Rock Particles Based on VIS
3.1. Method Framework
3.2. Particle Segmentation
3.2.1. Mask R-CNN
3.2.2. Classification Loss Function Based on Arcface Loss
3.3. Particle Tracking
3.4. Motion Information Detection
4. Experiment and Analysis
4.1. Experimental Equipment and Parameter Settings
4.2. Dataset
4.3. Algorithm Evaluation Index
4.4. Instance Segmentation Experiment
4.5. VIS Experiment
4.6. Particle Motion Information Detection Experiment
4.7. Discussion
5. Conclusions and Outlook
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Jerónimo, P.; Resende, R.; Fortunato, E. An assessment of contact and laser-based scanning of rock particles for railway ballast. Transp. Geotech. 2020, 22, 100302. [Google Scholar] [CrossRef]
- Zhou, Y.; Shi, Z.; Zhang, Q.; Jang, B.; Wu, C. Damming process and characteristics of landslide-debris avalanches. Soil Dyn. Earthq. Eng. 2019, 121, 252–261. [Google Scholar] [CrossRef]
- Gao, G.; Meguid, M.A.; Chouinard, L.E.; Xu, C. Insights into the Transport and Fragmentation Characteristics of Earthquake-Induced Rock Avalanche: Numerical Study. Int. J. Géoméch. 2020, 20, 04020157. [Google Scholar] [CrossRef]
- Kh, A.B.; Mirghasemi, A.A.; Mohammadi, S. Numerical simulation of particle breakage of angular particles using combined DEM and FEM. Powder Technol. 2011, 205, 15–29. [Google Scholar]
- Liu, G.Y.; Xu, W.J.; Sun, Q.C.; Govender, N. Study on the particle breakage of ballast based on a GPU accelerated discrete element method. Geosci. Front. 2020, 11, 461–471. [Google Scholar] [CrossRef]
- Bagherzadeh, H.; Mansourpour, Z.; Dabir, B. Numerical analysis of asphaltene particles evolution and flocs morphology using DEM-CFD approach. J. Pet. Sci. Eng. 2021, 199, 108309. [Google Scholar] [CrossRef]
- Li, B.; Liang, Y.; Zhang, L.; Zou, Q. Breakage law and fractal characteristics of broken coal and rock masses with different mixing ratios during compaction. Energy Sci. Eng. 2019, 7, 1000–1015. [Google Scholar] [CrossRef] [Green Version]
- Yang, L.; Fan, Y.; Xu, N. Video instance segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; pp. 5188–5197. [Google Scholar]
- Yen, Y.K.; Lin, C.L.; Miller, J.D. Particle overlap and segregation problems in on-line coarse particle size measurement. Powder Technol. 1998, 98, 1–12. [Google Scholar] [CrossRef]
- Zhang, G.Y.; Liu, G.Z.; Zhu, H. Segmentation algorithm of complex ore images based on templates transformation and reconstruction. Int. J. Miner. Metall. Mater. 2011, 18, 385. [Google Scholar] [CrossRef]
- Amankwah, A.; Aldrich, C. Automatic ore image segmentation using mean shift and watershed transform. In Proceedings of the 21st International Conference Radioelektronika (RADIOELEKTRONIKA), Brno, Czech Republic, 19–20 April 2011; pp. 1–4. [Google Scholar]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [PubMed]
- Yuan, L.; Duan, Y. A Method of Ore Image Segmentation Based on Deep Learning. In Proceedings of the International Conference on Intelligent Computing (ICIC), Wuhan, China, 15–18 August 2018; pp. 508–519. [Google Scholar]
- Duan, J.; Liu, X.; Wu, X.; Mao, C. Detection and segmentation of iron ore green pellets in images using lightweight U-net deep learning network. Neural Comput. Appl. 2019, 32, 5775–5790. [Google Scholar] [CrossRef]
- Liu, X.; Zhang, Y.; Jing, H.; Wang, L.; Zhao, S. Ore image segmentation method using U-Net and Res_Unet convolutional networks. RSC Adv. 2020, 10, 9396–9406. [Google Scholar] [CrossRef] [Green Version]
- Jaboyedoff, M.; Oppikofer, T.; Abellán, A.; Derron, M.H.; Loye, A.; Metzger, R.; Pedrazzini, A. Use of LIDAR in landslide investigations: A review. Nat. Hazards 2012, 61, 5–28. [Google Scholar] [CrossRef] [Green Version]
- Scaioni, M. Modern Technologies for Landslide Monitoring and Prediction; Springer: Berlin/Heidelberg, Germany, 2015; pp. 69–88. [Google Scholar]
- Adrian, R.J. Particle-imaging techniques for experimental fluid-mechanics. Annu. Rev. Fluid Mech. 1991, 23, 261–304. [Google Scholar] [CrossRef]
- Sutton, M.A.; Mcneill, S.R.; Helm, J.D.; Chao, Y.J. Advances in two-dimensional and three-dimensional computer vision. Top. Appl. Phys. 2000, 77, 323–372. [Google Scholar]
- Bing, P.; Wu, D.; Xia, Y. Incremental calculation for large deformation measurement using reliability-guided digital image correlation. Opt. Lasers Eng. 2012, 50, 586–592. [Google Scholar]
- Stanier, S.A.; Blaber, J.; Take, W.A.; White, D. Improved image-based deformation measurement for geotechnical applications. Can. Geotech. J. 2016, 53, 727–739. [Google Scholar] [CrossRef]
- Travelletti, J.; Malet, J.P. Characterization of the 3d geometry of flow-like landslides: A methodology based on the integration of heterogeneous multi-source data. Eng. Geol. 2012, 128, 30–48. [Google Scholar] [CrossRef]
- Gance, J.; Malet, J.P.; Dewez, T.; Travelletti, J. Target detection and tracking of moving objects for characterizing landslide displacements from time-lapse terrestrial optical images. Eng. Geol. 2014, 172, 26–40. [Google Scholar] [CrossRef]
- Shum, H.; Komura, T. Tracking the translational and rotational movement of the ball using high-speed camera movies. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Genoa, Italy, 11–14 September 2005; pp. 3–1084. [Google Scholar]
- Guccione, D.E.; Thoeni, K.; Giacomini, A.; Buzzi, O.; Fityus, S. Efficient Multi-View 3d Tracking of Arbitrary Rock Fragments upon Impact. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2020, 43, 589–596. [Google Scholar] [CrossRef]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1137–1149. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Li, D.; Chen, Y.; Zhang, K.; Li, Z. Mounting behaviour recognition for pigs based on deep learning. Sensors 2019, 19, 4924. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Fang, X.; Jie, W.; Feng, T. An industrial micro-defect diagnosis system via intelligent segmentation region. Sensors 2019, 19, 2636. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Li, X.; Fan, Z.; Liu, Y.; Li, Y.; Dai, Q. 3d pose detection of closely interactive humans using multi-view cameras. Sensors 2019, 19, 2831. [Google Scholar] [CrossRef] [Green Version]
- Wang, Y.; Li, H.; Jia, P.; Zhang, G.; Wang, T.; Hao, X. Multi-Scale DenseNets-Based Aircraft Detection from Remote Sensing Images. Sensors 2019, 19, 5270. [Google Scholar] [CrossRef] [Green Version]
- Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Santiago, Chile, 13–16 December 2015; pp. 3431–3440. [Google Scholar]
- Liu, W.; Wen, Y.; Yu, Z.; Yang, M. Large-margin softmax loss for convolutional neural networks. In Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA, 19–24 June 2016; Volume 2, p. 7. [Google Scholar]
- Liu, W.; Wen, Y.; Yu, Z.; Li, M.; Raj, B.; Song, L. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 22–25 July 2017; pp. 212–220. [Google Scholar]
- Deng, J.; Guo, J.; Xue, N.; Zafeiriou, S. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019; pp. 4690–4699. [Google Scholar]
- Bewley, A.; Ge, Z.; Ott, L.; Ramos, F.; Upcroft, B. Simple online and realtime tracking. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 3464–3468. [Google Scholar]
- Wojke, N.; Bewley, A.; Paulus, D. Simple online and realtime tracking with a deep association metric. In Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijin, China, 17–20 September 2017; pp. 3645–3649. [Google Scholar]
- Chen, Z.; Han, Z.; Hao, J.; Zhu, Q.; Soh, Y.C. Fusion of wifi, smartphone sensors and landmarks using the kalman filter for indoor localization. Sensors 2015, 15, 715–732. [Google Scholar] [CrossRef]
- Qiu, Z.; Zhao, N.; Zhou, L.; Wang, M.; Yang, L.; Fang, H.; Liu, Y. Vision-based moving obstacle detection and tracking in paddy field using improved yolov3 and deep SORT. Sensors 2020, 20, 4082. [Google Scholar] [CrossRef]
- Hamzeloo, E.; Massinaei, M.; Mehrshad, N. Estimation of particle size distribution on an industrial conveyor belt using image analysis and neural networks. Powder Technol. 2014, 261, 185–190. [Google Scholar] [CrossRef]
- Yang, Z.; Yuan, Y.; Zhang, M.; Zhao, X.; Tian, B. Safety distance identification for crane drivers based on mask r-cnn. Sensors 2019, 19, 2789. [Google Scholar] [CrossRef] [Green Version]
- Li, Y.; Qi, H.; Dai, J.; Ji, X.; Wei, Y. Fully convolutional instance-aware semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2359–2367. [Google Scholar]
- Fu, C.Y.; Shvets, M.; Berg, A.C. RetinaMask: Learning to predict masks improves state-of-the-art single-shot detection for free. arXiv 2019, arXiv:1901.03353. [Google Scholar]
- Bolya, D.; Zhou, C.; Xiao, F.; Lee, Y.J. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; pp. 9157–9166. [Google Scholar]
Parameter Type | Parameter | Value |
---|---|---|
Global Parameter | Number of categories | 2 |
Initial weight | 1 | |
Learning rate | 10−3 | |
Attenuation coefficient of learning rate | 10−4 | |
Attenuation step length | 100 | |
Momentum Factor | 0.9 | |
RPN Mesh Parameter | Anchor size | 16, 32, 64, 128, 256 |
Anchor shape ratio | 0.5, 1, 2 | |
Anchor step length | 1 | |
Maximum number of Anchors per image | 256 | |
Non-Maximum Suppression (NMS) threshold for anchor | 0.7 | |
The number of anchors after NMS | 2000 | |
Header Network Parameter | Confidence threshold | 0.7 |
Maximum number of instances | 100 | |
NMS threshold | 0.3 |
Method | Accuracy% | Precision% | Recall% | F1% |
---|---|---|---|---|
FCIS [42] | 71.97 | 76.26 | 59.68 | 66.96 |
RetinaMask [43] | 86.34 | 85.71 | 69.53 | 76.78 |
YOLACT [44] | 73.28 | 74.45 | 60.11 | 66.52 |
Standard Mask R-CNN | 89.16 | 89.72 | 72.17 | 79.99 |
Improved Mask R-CNN | 93.36 | 91.41 | 78.16 | 84.27 |
Method | |||
---|---|---|---|
FCIS [42] | 29.5 | 51.5 | 30.2 |
RetinaMask [43] | 34.7 | 55.4 | 36.9 |
YOLACT [44] | 29.8 | 48.5 | 31.2 |
Standard Mask R-CNN | 35.7 | 58.5 | 37.8 |
Improved Mask R-CNN | 36.0 | 58.9 | 38.3 |
Method | ||||||
---|---|---|---|---|---|---|
Standard Mask R-CNN | 36.7 | 59.5 | 38.9 | 39.6 | 61.5 | 43.2 |
+ update baseline | 37.0 | 59.7 | 39.0 | 40.5 | 63.0 | 43.7 |
+ c2c training | 37.6 | 60.4 | 39.9 | 41.7 | 64.1 | 45.2 |
+ ImageNet–5k | 38.6 | 61.7 | 40.9 | 42.7 | 65.1 | 46.6 |
+ train-time augmentation | 39.2 | 62.5 | 41.6 | 43.5 | 65.9 | 47.2 |
+ loss function (ours) | 39.6 | 63.1 | 42.3 | 44.0 | 66.5 | 47.7 |
Video | Method | Iswitch | Iall | IDswitch% |
---|---|---|---|---|
Video1 | SM + D | 73 | 25621 | 0.28 |
Video1 | IM + D | 45 | 25621 | 0.18 |
Video2 | SM + D | 1059 | 59549 | 1.78 |
Video2 | IM + D | 638 | 59549 | 1.07 |
Video3 | SM + D | 3554 | 191257 | 1.86 |
Video3 | IM + D | 2811 | 191257 | 1.47 |
Video4 | SM + D | 3977 | 142534 | 2.79 |
Video4 | IM + D | 2722 | 142534 | 1.91 |
Video | Method | δt% | δr% |
---|---|---|---|
Video1 | SM + D | 3.53 | 11.84 |
Video1 | IM + D | 3.47 | 12.31 |
Video2 | SM + D | 5.51 | 15.43 |
Video2 | IM + D | 5.14 | 13.81 |
Video3 | SM + D | 6.27 | 15.28 |
Video3 | IM + D | 6.41 | 16.06 |
Video4 | SM + D | 5.58 | 18.16 |
Video4 | IM + D | 5.36 | 15.79 |
Average | SM + D | 5.22 | 15.18 |
Average | IM + D | 5.10 | 14.49 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, M.; Li, M.; Li, Y.; Yi, W. Rock Particle Motion Information Detection Based on Video Instance Segmentation. Sensors 2021, 21, 4108. https://doi.org/10.3390/s21124108
Chen M, Li M, Li Y, Yi W. Rock Particle Motion Information Detection Based on Video Instance Segmentation. Sensors. 2021; 21(12):4108. https://doi.org/10.3390/s21124108
Chicago/Turabian StyleChen, Man, Maojun Li, Yiwei Li, and Wukun Yi. 2021. "Rock Particle Motion Information Detection Based on Video Instance Segmentation" Sensors 21, no. 12: 4108. https://doi.org/10.3390/s21124108