A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration
Abstract
:1. Introduction
- (1)
- As we know, this is the first independent review on deep learning-based extrinsic calibration, which can provide significant assistance and reference for researchers engaged in this field.
- (2)
- We provide a novel classification of LiDAR–camera extrinsic calibration based on deep learning. Different from the previous works, we divide the methods into two categories, namely, accurate extrinsic parameters estimation (AEPE) and relative extrinsic parameters prediction (REPP).
- (3)
- We provide a comprehensive and detailed explanation of the existing works. We not only study the different types of methods holistically, but also provide the classification and summary of the procedures in the algorithms step by step.
- (4)
- In previous research, they usually summarized deep learning calibration algorithms as black boxes, one by one. Different from those works, we have conducted a rational phased and modular study and summary on deep learning-based calibration algorithms, which is more conducive for researchers, especially those who are new to related research, to understand calibration and its deep learning methods from a theoretical perspective.
- (5)
- We also provide a detailed comparison and discussion on different methods, including their evolution process, principles, characteristics, advantages, and limitations.
2. Problem Formulation
2.1. Transformation between LiDAR and Camera Coordinates
2.2. Classification of Calibration Methods
3. Deep Learning-Based Extrinsic Calibration
3.1. Accurate Extrinsic Parameters Estimation
3.2. Global Feature-Based Method
3.2.1. Basic Framework
3.2.2. Data Conversion
3.2.3. Feature Extraction
3.2.4. Feature Matching
- A.
- Feature matching based on optical flow estimation blocks
- B.
- Feature Matching Based on Recurrent Network
- C.
- Feature matching based on ResNet and DenseNet
- D.
- Feature matching based on transformer
3.2.5. Parameters Regression
3.2.6. Loss Function
- A.
- Calibration parameter loss
- B.
- Point cloud reprojection loss
- C.
- Point cloud projection loss
- D.
- Stereo constraint loss
- E.
- Supervised contrastive loss
3.2.7. Summary of Global Feature-Based Methods
3.3. Local Feature-Based Methods
3.3.1. Basic Framework
3.3.2. Feature Extraction and Matching
- A.
- Bilateral local feature extraction
- B.
- Unilateral local feature extraction and fusion
3.3.3. Initial Guess
- A.
- Blind search without prior knowledge
- B.
- Camera pose estimation-based initial guess
3.3.4. Calibration Optimal Model
3.3.5. Optimization Method
3.3.6. Summary of Local Feature-Based Methods
3.4. Relative Extrinsic Parameters Prediction
3.4.1. Data Input
3.4.2. Data Retrieval
3.4.3. Relevance Mapping
3.4.4. Late Fusion
3.4.5. Summary of REPP
4. Dataset
5. Calibration Effects
- (a)
- In the experiment, were different types of datasets or real-time data used for testing? If two or more different types of datasets or real-time data are used, a score of two stars (☆☆) will be determined.
- (b)
- In the experiment, was the initial error range of the test data large enough? If error range ≥ [−1.5 m, 1.5 m]/[−20°, 20°], then one star (☆) is obtained.
- (c)
- In the experiment, was an anti-noise test is conducted? If so, one star (☆) will be awarded.
- (a)
- Methods of the AEPE performed better than the method of the REPP.
- (b)
- For the AEPE, the global feature-based methods generally have better calibration effects than the local feature-based methods.
- (c)
- End-to-end methods generally calibrated better than hybrid methods.
6. Discussion
7. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Nie, J.; Pan, F.; Xue, D. A survey of extrinsic parameters calibration techniques for autonomous devices. In Proceedings of the 33th Chinese Control and Decision Conference, Kunming, China, 22 May 2021. [Google Scholar]
- Wang, Y.; Li, J.; Sun, Y. A survey of extrinsic calibration of LiDAR and camera. In Proceedings of the 2021 International Conference on Autonomous Unmanned Systems, Changsha, China, 24–26 September 2021. [Google Scholar]
- Canalle, G.K.; Salgado, A.C.; Loscio, B.F. A survey on data fusion: What for? in what form? What is next? J. Intell. Inf. Syst. 2020, 57, 25–50. [Google Scholar] [CrossRef]
- Li, X.; Xiao, Y.; Wang, B. Automatic targetless LiDAR-camera calibration: A survey. Artif. Intell. Rev. 2022, 56, 9949–9987. [Google Scholar] [CrossRef]
- Yu, X.; Salimpour, S.; Queralta, J.P. General-purpose deep learning detection and segmentation models for images from a LiDAR-based camera sensor. Sensors 2023, 23, 2936. [Google Scholar] [CrossRef] [PubMed]
- Choi, J.D.; Kim, M.Y. A sensor fusion system with thermal infrared camera and LiDAR for autonomous vehicles and deep learning based object detection. ICT Express 2022, 9, 222–227. [Google Scholar] [CrossRef]
- Pang, S.; Morris, D.; Radha, H. CLOCs: Camera-LiDAR object candidates fusion for 3D object detection. In Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA, 24–30 October 2020. [Google Scholar]
- Asvadi, A.; Garrote, L.; Premebida, C. Multimodal vehicle detection: Fusing 3D-LiDAR and color camera data. Pattern Recognit. Lett. 2018, 115, 20–29. [Google Scholar] [CrossRef]
- Oliveira, G.L.; Radwan, N.; Burgard, W. Topometric localization with deep learning. Robotics 2017, 10, 505–520. [Google Scholar]
- Song, H.; Choi, W.; Kim, H. Robust vision-based relative-localization approach using an RGB-depth camera and LiDAR sensor fusion. IEEE Trans. Ind. Electron. 2016, 63, 3725–3736. [Google Scholar] [CrossRef]
- Kang, Q.; She, R.; Wang, S. Location learning for AVs: LiDAR and image landmarks fusion localization with graph neural networks. In Proceedings of the IEEE 25th International Conference on Intelligent Transportation Systems, Macau, China, 8–12 October 2022. [Google Scholar]
- Napier, A.A. Vision and Laser for Road Based Navigation. Ph.D. Thesis, University of Oxford, Oxford, UK, 2016. [Google Scholar]
- Hikes, J.J. Calibration of Cameras and LiDAR for Applications in Autonomous Navigation. Ph.D. Thesis, West Virginia University, Morgantown, WV, USA, 2018. [Google Scholar]
- Jeong, J.; Cho, Y.; Kim, A. The road is enough! extrinsic calibration of non-overlapping stereo camera and LiDAR using road information. IEEE Robot. Autom. Lett. 2019, 4, 2831–2838. [Google Scholar] [CrossRef]
- Matinfar, S.; Salehi, M.; Suter, D. Sonification as a reliable alternative to conventional visual surgical navigation. Sci. Rep. 2023, 13, 5930. [Google Scholar] [CrossRef]
- Liu, Y.; Guo, X.; Su, S. Review of a 3D LiDAR combined with single vision calibration. In Proceedings of the 2021 IEEE International Conference on Data Science and Computer Application (ICDSCA), Dalian, China, 29–31 October 2021. [Google Scholar]
- Khurana, A.; Nagla, K.S. Extrinsic calibration methods for laser range finder and camera: A systematic review. Mapan J. Metrol. Soc. India 2021, 36, 669–690. [Google Scholar] [CrossRef]
- Persic, J.; Petrovic, L.; Markovic, I. Spatiotemporal multi-sensor calibration via Gaussian processes moving target tracking. IEEE Trans. Robot. 2021, 37, 1401–1415. [Google Scholar] [CrossRef]
- Zhao, L.; Wang, M.; Su, S. Dynamic object tracking for self-driving cars using monocular camera and LiDAR. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Prague, Czech Republic, 27 September–1 October 2021. [Google Scholar]
- Banerjee, K.; Notz, D.; Windelen, J. Online camera LiDAR fusion and object detection on hybrid data for autonomous driving. In Proceedings of the 2018 IEEE Intelligent Vehicles Symposium, Suzhou, China, 26 June–1 July 2018. [Google Scholar]
- Zhang, Q.; Pless, R. Extrinsic calibration of a camera and laser range finder (improves camera calibration). In Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, Sendai, Japan, 28 September–2 October 2004. [Google Scholar]
- Andreas, G.; Frank, M.; Oemer, C. Automatic camera and range sensor calibration using a single shot. In Proceedings of the IEEE International Conference on Robotics and Automation, St. Paul, MN, USA, 14–18 May 2012. [Google Scholar]
- Yoo, J.H.; Jung, G.B.; Jung, H.G.; Suhr, J.K. Camera–LiDAR Calibration Using Iterative Random Sampling and Intersection Line-Based Quality Evaluation. Electronics 2024, 13, 249. [Google Scholar] [CrossRef]
- Yu, D.; Li, G.; Wang, L. Calibration of LiDAR and camera based on 3D feature point sets. Bull. Surv. Mapp. 2018, 11, 40–45. [Google Scholar]
- Beltran, J.; Guindel, C.; Escalera, A.D.L. Automatic extrinsic calibration method for LiDAR and camera sensor setups. IEEE Trans. Intell. Transp. Syst. 2022, 23, 17677–17689. [Google Scholar] [CrossRef]
- Jung, J.; Jeong, Y.; Park, J. A novel 2.5D pattern for extrinsic calibration of TOF and camera fusion system. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA, 25–30 September 2011. [Google Scholar]
- Park, Y.; Yun, S.; Won, C.S. Calibration between color camera and 3D LiDAR instruments with a polygonal planar board. Sensors 2014, 14, 5333–5353. [Google Scholar] [CrossRef]
- Li, M.; Gao, X.; Li, G.; Wang, L.; Liu, S. High accuracy calibration of installation parameters between 3D terrestrial laser scanner and external-installed digital camera. Opt. Precis. Eng. 2016, 24, 2158–2166. [Google Scholar]
- Yi, H.; Liu, B.; Zhao, B. Extrinsic calibration for LiDAR–camera systems using direct 3D–2D correspondences. Remote Sens. 2022, 14, 6082. [Google Scholar] [CrossRef]
- Sen, A.; Pan, G.; Mitrokhin, A. Scenecalib: Automatic targetless calibration of cameras and LiDARs in autonomous driving. In Proceedings of the IEEE International Conference on Robotics and Automation, London, UK, 29 May–2 June 2023. [Google Scholar]
- Levinson, J.; Thrun, S. Automatic online calibration of cameras and lasers. In Proceedings of the Robotics: Science and Systems IX, Berlin, Germany, 24–28 June 2013. [Google Scholar]
- Jiang, P.; Osteen, P.; Saripalli, S. Semcal: Semantic LiDAR-camera calibration using neural mutual information estimator. In Proceedings of the IEEE International Conference on Multi-sensor Fusion and Integration for Intelligent Systems, Karlsruhe, Germany, 23–25 September 2021. [Google Scholar]
- Xiao, J.; Shi, C.; Huang, K. Extrinsic calibration and fused odometry for monocular camera and 3D LiDAR. Robot 2021, 43, 17–28. [Google Scholar]
- Liao, Q.; Liu, M. Extrinsic calibration of 3D range finder and camera without auxiliary object or human intervention. In Proceedings of the IEEE International Conference on Real-time Computing and Robotics, Irkutsk, Russia, 4–9 August 2019. [Google Scholar]
- Schneider, N.; Piewak, F.; Stiller, C. RegNet: Multimodal sensor registration using deep neural networks. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA, 11–14 June 2017. [Google Scholar]
- Wang, W.; Nobuhara, S.; Nakamura, R. SOIC: Semantic online initialization and calibration for LiDAR and camera. arXiv 2020, arXiv:2003.04260. [Google Scholar]
- Wu, X.; Zhang, C.; Liu, Y. CalibRank: Effective LiDAR-camera extrinsic calibration by multi-modal learning to rank. In Proceedings of the IEEE International Conference on Image Processing, Abu Dhabi, United Arab Emirates, 25–28 October 2020. [Google Scholar]
- An, P.; Gao, Y.; Wang, L. Online extrinsic calibration on LiDAR-camera system with LiDAR intensity attention and structural consistency loss. Remote Sens. 2022, 14, 2525. [Google Scholar] [CrossRef]
- Shang, H.; Hu, B. CALNet: LiDAR-camera online calibration with channel attention and liquid time constant network. In Proceedings of the 26th International Conference on Pattern Recognition, Montreal, QC, Canada, 21–25 August 2022. [Google Scholar]
- Sun, Y.; Li, J.; Wang, Y. ATOP: An attention-to-optimization approach for automatic LiDAR-camera calibration via cross-modal object matching. IEEE Trans. Intell. Veh. 2023, 8, 696–708. [Google Scholar] [CrossRef]
- Jing, X.; Ding, X.; Xiong, R. DXQ-Net: Differentiable LiDAR-camera extrinsic calibration using quality aware flow. In Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems, Kyoto, Japan, 23–27 October 2022. [Google Scholar]
- Duy, A.N.; Yoo, M. Calibration-Net: LiDAR and camera auto-calibration using cost volume and convolutional neural network. In Proceedings of the 2022 International Conference on Artificial Intelligence in Information and Communication, Jeju Island, Republic of Korea, 21–24 February 2022. [Google Scholar]
- Zhao, G.; Hu, J.; You, S. CalibDNN: Multimodal sensor calibration for perception using deep neural networks. In Proceedings of the Signal Processing, Sensor/Information Fusion, and Target Recognition, Bellingham, WA, USA, 12–16 April 2021. [Google Scholar]
- Iyer, G.; Ram, R.K.; Murthy, J.K. CalibNet: Self-supervised extrinsic calibration using 3D spatial transformer networks. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain, 1–5 October 2018. [Google Scholar]
- Yuan, K.; Guo, Z.; Wang, Z.J. RGGnet: Tolerance aware LiDAR-camera online calibration with geometric deep learning and generative model. IEEE Robot. Autom. Lett. 2020, 5, 6956–6963. [Google Scholar] [CrossRef]
- Lv, X.; Wang, B.; Dou, Z. LCCNet: LiDAR and camera self-calibration using cost volume network. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Online, 20–25 June 2021. [Google Scholar]
- Zhu, J.; Xue, J.; Zhang, P. CalibDepth: Unifying depth map representation for iterative LiDAR-camera online calibration. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation, London, UK, 29 May–2 June 2023. [Google Scholar]
- Mharolkar, S.; Zhang, J.; Peng, G. RGBDTCalibNet: End-to-end online extrinsic calibration between a 3D LiDAR, an RGB camera and a thermal camera. In Proceedings of the IEEE 25th International Conference on Intelligent Transportation Systems, Macau, China, 8–12 October 2022. [Google Scholar]
- Nagy, B.; Kovacs, L.; Benedek, C. Sfm and semantic information based online targetless camera-LiDAR self-calibration. In Proceedings of the 2019 IEEE International Conference on Image Processing, Taiwan, China, 22–25 September 2019. [Google Scholar]
- Liu, Z.; Tang, H.; Zhu, S. SemAlign: Annotation-free camera-LiDAR calibration with semantic alignment loss. In Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, Prague, Czech Republic, 27 September–1 October 2021. [Google Scholar]
- Abhinav, V.; Rohit, M.; Wolfram, B. Self-supervised model adaptation for multimodal semantic segmentation. Int. J. Comput. Vis. 2020, 128, 1239–1285. [Google Scholar]
- Luo, Z.; Yan, G.; Li, Y. Calib-Anything: Zero-training LiDAR-camera extrinsic calibration method using segment anything. arXiv 2023, arXiv:2306.02656v1. [Google Scholar]
- Yu, Z.; Feng, C.; Liu, M. CaseNet: Deep category-aware semantic edge detection. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Yin, J.; Yan, F.; Liu, Y. Automatic and targetless LiDAR-camera extrinsic calibration using edge alignment. IEEE Sens. J. 2023, 23, 19871–19880. [Google Scholar] [CrossRef]
- Kang, J.; Doh, N.L. Automatic targetless camera–LiDAR calibration by aligning edge with Gaussian mixture model. J. Field Robot. 2020, 37, 158–179. [Google Scholar] [CrossRef]
- Tao, L.; Pei, L.; Li, T. CPI: LiDAR-camera extrinsic calibration based on feature points with reflection intensity. In Proceedings of the Spatial Data and Intelligence 2020, Shenzhen, China, 8–9 May 2020. [Google Scholar]
- Shi, J.; Zhu, Z.; Zhang, J. CalibRCNN: Calibrating camera and LiDAR by recurrent convolutional neural network and geometric constraints. In Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, NV, USA, 24–30 October 2020. [Google Scholar]
- Liao, Y.; Li, J.; Kang, S. SE-Calib: Semantic edges based LiDAR-camera boresight online calibration in urban scenes. IEEE Trans. Geosci. Remote Sens. 2023, 61, 1000513. [Google Scholar] [CrossRef]
- Lv, X.; Wang, S.; Ye, D. CFNet: LiDAR-camera registration using calibration flow network. Sensors 2021, 21, 8112. [Google Scholar] [CrossRef]
- Sun, D.; Yang, X.; Liu, M.Y. PWC-Net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018. [Google Scholar]
- Wu, Y.; Zhu, M.; Liang, J. PSNet: LiDAR and camera registration using parallel subnetworks. IEEE Access 2022, 10, 70553–70561. [Google Scholar] [CrossRef]
- Hasani, R.; Lechner, M.; Amini, A. Liquid time-constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Online, 2–9 February 2021. [Google Scholar]
- Nguyen, A.D.; Yoo, M. CalibBD: Extrinsic calibration of the LiDAR and camera using a bidirectional neural network. IEEE Access 2022, 10, 1. [Google Scholar] [CrossRef]
- Liu, R.; Shi, J.; Zhang, H.; Zhang, J.; Sun, B. Causal calibration: Iteratively calibrating lidar and camera by considering causality and geometry. Complex Intell. Syst. 2023, 9, 7349–7363. [Google Scholar] [CrossRef]
- Teed, Z.; Deng, J. RAFT: Recurrent all-pairs field transforms for optical flow. In Proceedings of the 16th European Conference on Computer Vision–ECCV, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Wu, S.; Hadachi, A.; Vivet, D. NetCalib: A novel approach for LiDAR-camera auto-calibration based on deep learning. In Proceedings of the 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021. [Google Scholar]
- He, K.; Zhang, X.; Ren, S. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 1904–1916. [Google Scholar] [CrossRef] [PubMed]
- Zhang, W.; Xu, D. Extrinsic calibration of LiDAR-camera based on deep convolutional network. In Proceedings of the 2022 China Automation Congress, Xiamen, China, 17–25 November 2022. [Google Scholar]
- Cocheteux, M.; Moreau, J.; Davoine, F. PseudoCal: Towards initialisation-free deep learning-based camera-LiDAR self-calibration. arXiv 2023, arXiv:2309.09855v1. [Google Scholar]
- Zhang, Z.; Yu, Z.; You, S. Enhanced low-resolution LiDAR-camera calibration via depth interpolation and supervised contrastive learning. In Proceedings of the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greek, 4–10 June 2023. [Google Scholar]
- Park, K.; Kim, S.; Sohn, K. High-precision depth estimation using uncalibrated LiDAR and stereo fusion. IEEE Trans. Intell. Transp. Syst. 2020, 21, 321–335. [Google Scholar] [CrossRef]
- Kanade, T.; Okutomi, M. A stereo matching algorithm with an adaptive window: Theory and experiment. IEEE Trans. Pattern Anal. Mach. Intell. 1994, 17, 920–932. [Google Scholar] [CrossRef]
- Geiger, A.; Roser, M.; Urtasun, R. Efficient large-scale stereo matching. In Proceedings of the 2010 Asian Conference on Computer Vision, Queenstown, New Zealand, 8–12 November 2010. [Google Scholar]
- Heise, P.; Klose, S.; Jensen, B. PM-Huber: PatchMatch with huber regularization for stereo matching. In Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013. [Google Scholar]
- Zbontar, J.; LeCun, Y. Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res. 2016, 17, 2287–2318. [Google Scholar]
- Hirschmuller, H. Stereo processing by semi-global matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 328–341. [Google Scholar] [CrossRef]
- Kim, D.; Ka, W.; Ahn, P. Global-local path networks for monocular depth estimation with vertical cut depth. arXiv 2022, arXiv:2201.07436. [Google Scholar]
- Zhu, J.; Li, X.; Xu, Q.; Sun, Z. Robust online calibration of lidar and camera based on cross-modal graph neural network. IEEE Trans. Instrum. Meas. 2023, 72, 2523514. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Sutskever, I.; Hinton, G. ImageNet classification with deep convolutional neural networks. Commun. ACM 2017, 60, 84–90. [Google Scholar] [CrossRef]
- Lin, M.; Chen, Q.; Yan, S. Network in network. arXiv 2013, arXiv:1312.4400. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009. [Google Scholar]
- He, K.; Zhang, X.; Ren, S. Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Rachman, A.; Seiler, J.; Kaup, A. End-to-end LiDAR-camera self-calibration for autonomous vehicles. In Proceedings of the 2023 IEEE Intelligent Vehicles Symposium, Anchorage, AK, USA, 4–7 June 2023. [Google Scholar]
- Oren, M.; Nayar, S.K. Generalization of the lambertian model and implications for machine vision. Int. J. Comput. Vis. 1995, 14, 227–251. [Google Scholar] [CrossRef]
- Zhang, Y.; Li, K.; Li, K. Image super-resolution using very deep residual channel attention networks. In Proceedings of the 15th European Conference on Computer Vision–ECCV, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Wang, J.; Sun, K.; Cheng, T.; Jiang, B.; Deng, C.; Zhao, Y.; Liu, D.; Mu, Y.; Tan, M.; Wang, X.; et al. Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 3349–3364. [Google Scholar] [CrossRef] [PubMed]
- Wang, H.; Wang, Z.; Yu, G.; Yang, S.; Yang, Y. Mrcnet: Multi-resolution lidar-camera calibration using optical center distance loss network. IEEE Sens. J. 2023, 1–13. [Google Scholar] [CrossRef]
- Xiao, Y.; Li, Y.; Meng, C.; Li, X.; Zhang, Y. Calibformer: A transformer-based automatic lidar-camera calibration network. arXiv 2023, arXiv:2311.15241v1. [Google Scholar]
- Pandey, G.; McBride, J.R.; Savarese, S. Automatic targetless extrinsic calibration of a 3D LiDAR and camera by maximizing mutual information. In Proceedings of the 26th AAAI Conference on Artificial Intelligence, Toronto, ON, Canada, 22 July 2012. [Google Scholar]
- Taylor, Z.; Nieto, J. A mutual information approach to automatic calibration of camera and LiDAR in natural environments. In Proceedings of the Australasian Conference on Robotics and Automation, Wellington, New Zealand, 3 December 2012. [Google Scholar]
- Pandey, G.; McBride, J.; Savarese, S.; Eustice, R.M. Automatic extrinsic calibration of vision and LiDAR by maximizing mutual information. J. Field Robot. 2015, 32, 696–722. [Google Scholar] [CrossRef]
- Dosovitskiy, A.; Fischery, P.; Ilg, E.; Hausser, P.; Hazırbaş, C.; Golkov, V. FlowNet: Learning optical flow with convolutional networks. In Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015. [Google Scholar]
- Zhu, X.; Hu, H.; Lin, S. Deformable ConvNets v2: More deformable, better results. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Duan, Z.; Hu, X.; Ding, J.; An, P.; Huang, X.; Ma, J. A robust lidar-camera self-calibration via rotation-based alignment and multi-level cost volume. IEEE Robot. Autom. Lett. 2024, 9, 627–634. [Google Scholar] [CrossRef]
- Kendall, A.; Gal, Y. What uncertainties do we need in bayesian deep learning for computer vision? In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Gast, J.; Roth, S. Lightweight probabilistic deep networks. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018. [Google Scholar]
- Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Qi, L.; Kuen, J.; Jiuxiang, G. Multi-scale aligned distillation for low-resolution detection. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Online, 20–25 June 2021. [Google Scholar]
- MobileVit: Light-weight, general-purpose, and mobile-friendly vision transformer. arXiv 2022, arXiv:2110.02178.
- Cocheteux, M.; Low, A.; Bruehlmeier, M. UniCal: A single-branch transformer-based model for camera to LiDAR calibration and validation. arXiv 2023, arXiv:2304.09715. [Google Scholar]
- Wu, S.; Hadachi, A.; Vivet, D.; Prabhakar, Y. This is the way: Sensors auto-calibration approach based on deep learning for self-driving cars. IEEE Sens. J. 2021, 21, 27779–27788. [Google Scholar] [CrossRef]
- Aouragh, M.; Gulyas, L. Extrinsic calibration framework for camera-lidar fusion using recurrent residual network. In Proceedings of the 2023 International Conference on Computational Collective Intelligence, Budapest, Hungary, 27–29 September 2023. [Google Scholar]
- Lepetit, V.; Moreno-Noguer, F.; Fua, P. Epnp: An accurateo(n) solution to the pnp problem. Int. J. Comput. Vis. 2009, 81, 155–166. [Google Scholar] [CrossRef]
- Hartley, H.O. The modified gauss-newton method for the fitting of non-linear regression functions by least squares. Technometrics 1961, 3, 269–280. [Google Scholar] [CrossRef]
- Kendall, A.; Grimes, M.; Cipolla, R. Posenet: A convolutional network for real-time 6-dof camera relocalization. In Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015. [Google Scholar]
- Ye, C.; Pan, H.; Gao, H. Keypoint-based LiDAR-camera online calibration with robust geometric network. IEEE Trans. Instrum. Meas. 2022, 71, 1–11. [Google Scholar] [CrossRef]
- Lang, A.H.; Vora, S.; Caesar, H. PointPillars: Fast encoders for object detection from point clouds. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Hu, Y.; Ma, H.; Jie, L. Dedgenet: Extrinsic calibration of camera and LiDAR with depth-discontinuous edges. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation, London, UK, 29 May–2 June 2023. [Google Scholar]
- Belghazi, I.; Rajeswar, S.; Baratin, A.; Ozair, S.; Bengio, Y.; Courville, A.; Hjelm, R.D. Mine: Mutual information neural estimation. arXiv 2018, arXiv:1801.04062. [Google Scholar]
- Kodaira, A.; Zhou, Y.; Zang, P.; Zhan, W.; Tomizuka, M. SST-Calib: Simultaneous spatial-temporal parameter calibration between LiDAR and camera. In Proceedings of the IEEE 25th International Conference on Intelligent Transportation Systems, Macau, China, 8–12 October 2022. [Google Scholar]
- Zhu, Y.; Li, C.; Zhang, Y. Online camera-LiDAR calibration with sensor semantic information. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation, Paris, France, 31 May–31August 2020. [Google Scholar]
- Zhao, H.; Shi, J.; Qi, X. Pyramid scene parsing network. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Kirillov, A.; Mintun, E.; Ravi, N. Segment anything. arXiv 2023, arXiv:2304.02643. [Google Scholar]
- Ma, T.; Liu, Z.; Yan, G.; Li, Y. CRLF: Automatic calibration and refinement based on line feature for LiDAR and camera in road scenes. arXiv 2022, arXiv:2201.07436. [Google Scholar]
- Wang, G.; Qiu, J.; Guo, Y. FusionNet: Coarse-to-fine extrinsic calibration network of LiDAR and camera with hierarchical point-pixel fusion. In Proceedings of the 2022 International Conference on Robotics and Automation, Philadelphia, PA, USA, 23–27 May 2022. [Google Scholar]
- Yuan, Y.; Chen, X.; Wang, J. Object-contextual representations for semantic segmentation. In Proceedings of the 16th European Conference on Computer Vision–ECCV, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Cortinhal, T.; Tzelepis, G.; Erdal Aksoy, E. SalsaNext: Fast, uncertainty-aware semantic segmentation of LiDAR point clouds. In Proceedings of the 15th International Symposium Advances in Visual Computing, San Diego, CA, USA, 5–7 October 2020. [Google Scholar]
- Zhu, Y.; Sapra, K.; Reda, F.A. Improving semantic segmentation via video propagation and label relaxation. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Tang, H.; Liu, Z.; Zhao, S. Searching efficient 3D architectures with sparse point-voxel convolution. In Proceedings of the 16th European Conference on Computer Vision–ECCV, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Xu, C.; Wu, B.; Wang, Z. SqueezeSegV3: Spatially-adaptive convolution for efficient point-cloud segmentation. In Proceedings of the 16th European Conference on Computer Vision–ECCV, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Shi, S.; Wang, X.; Li, H. PointRCNN: 3D object proposal generation and detection from point cloud. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Romera, E.; Alvarez, J.; Bergasa, L.; Arroyo, R. ERFNet: Efficient residual factorized convnet for real-time semantic segmentation. IEEE Trans. Intell. Transp. Syst. 2018, 19, 263–272. [Google Scholar] [CrossRef]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Hu, Y.; Chen, Y.; Li, X.; Feng, J. Dynamic feature fusion for semantic edge detection. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macau, China, 10–16 August 2019. [Google Scholar]
- Zhu, X.; Zhou, H.; Wang, T.; Hong, F.; Ma, Y.; Li, W.; Li, H.; Lin, D. Cylindrical and asymmetrical 3D convolution networks for LiDAR segmentation. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Online, 20–25 June 2021. [Google Scholar]
- Tan, M.; Ruoming, P.; Le, Q.V. EfficientDet: Scalable and efficient object detection. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020. [Google Scholar]
- Wu, B.; Zhou, X.; Zhao, S.; Yue, X.; Keutzer, K. Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud. In Proceedings of the 2019 International Conference on Robotics and Automation, Montreal, QC, Canada, 22–24 May 2019. [Google Scholar]
- Yu, C.; Gao, C.; Wang, J.; Yu, G.; Shen, C.; Sang, N. BiSeNet V2: Bilateral network with guided aggregation for real-time semantic segmentation. Int. J. Comput. Vis. 2021, 129, 3051–3068. [Google Scholar] [CrossRef]
- Xiao, B.; Wu, H.; Wei, Y. Simple baselines for human pose estimation and tracking. In Proceedings of the 15th European Conference on Computer Vision–ECCV, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Qi, C.R.; Yi, L.; Su, H. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Rotter, P.; Klemiato, M.; Skruch, P. Automatic calibration of a lidar–camera system based on instance segmentation. Remote Sens. 2022, 14, 2531. [Google Scholar] [CrossRef]
- Gao, X.; Hou, X.; JL, T. Complete solution classification for the perspective-three-point problem. IEEE Trans. Pattern Anal. Mach. Intell. 2003, 25, 930–943. [Google Scholar]
- Campbell, D.; Liu, L.; Gould, S. Solving the blind perspective-n-point problem end-to-end with robust differentiable geometric optimization. In Proceedings of the 16th European Conference on Computer Vision–ECCV, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Collins, T.; Bartoli, A. Infinitesimal plane-based pose estimation. Int. J. Comput. Vis. 2014, 109, 252–286. [Google Scholar] [CrossRef]
- Xu, C.; Zhang, L.; Cheng, L. Pose estimation from line correspondences: A complete analysis and a series of solutions. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1209–1222. [Google Scholar] [CrossRef] [PubMed]
- Powell, M.J.D. An efficient method for finding the minimum of a function of several variables without calculating derivatives. Comput. J. 1964, 7, 155–162. [Google Scholar] [CrossRef]
- Kennedy, J.; Eberhart, R. Particle swarm optimization. In Proceedings of the International Conference on Neural Networks 1995, Perth, Australia, 27 November–1 December 1995. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Tatarchenko, M.; Richter, S.R.; Ranftl, R. What do single-view 3D reconstruction networks learn? In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Sattler, T.; Zhou, Q.; Pollefeys, M.; Leal-Taixe, L. Understanding the limitations of CNN-based absolute camera pose regression. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Melekhov, I.; Ylioinas, J.; Kannala, J.; Rahtu, E. Relative camera pose estimation using convolutional neural networks. In Proceedings of the 18th International Conference on Advanced Concepts for Intelligent Vision Systems, Antwerp, Belgium, 18–21 September 2017. [Google Scholar]
- Saha, S.; Varma, G.; Jawahar, C.V. Improved visual relocalization by discovering anchor points. arXiv 2018, arXiv:1811.04370. [Google Scholar]
- Laskar, Z.; Melekhov, I.; Kalia, S.; Kannala, J. Camera relocalization by computing pairwise relative poses using convolutional neural network. In Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops, Venice, Italy, 22–29 October 2017. [Google Scholar]
- Balntas, V.; Li, S.; Prisacariu, V. Relocnet: Continuous metric learning relocalisation using neural nets. In Proceedings of the 15th European Conference on Computer Vision–ECCV, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Ummenhofer, B.; Zhou, H.; Uhrig, J.; Mayer, N.; Ilg, E.; Dosovitskiy, A.; Brox, T. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Cordts, M.; Omran, M.; Ramos, S.; Rehfeld, T.; Enzweiler, M.; Benenson, R.; Franke, U.; Roth, S.; Schiele, B. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Geiger, A.; Lenz, P.; Stiller, C.; Urtasun, R. Vision meets robotics: The KITTI dataset. Int. J. Robot. Res. 2013, 32, 1231–1237. [Google Scholar] [CrossRef]
- Liao, Y.; Xie, J.; Geiger, A. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 3292–3310. [Google Scholar] [CrossRef]
- Jiang, P.; Osteen, P.; Wigness, M.; Saripalli, S. RELLIS-3D dataset: Data, benchmarks and analysis. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation, Xi’an, China, 30 May–5 July 2021. [Google Scholar]
- Dosovitskiy, A.; Ros, G.; Codevilla, F.; Lopez, A.; Koltun, V. CARLA: An open urban driving simulator. In Proceedings of the 2017 Machine Learning Research, Mountain View, CA, USA, 17 July 2017. [Google Scholar]
- Ishikawa, R.; Oishi, T.; Ikeuchi, K. LiDAR and camera calibration using motions estimated by sensor fusion odometry. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain, 1–5 October 2018. [Google Scholar]
- Taylor, Z.; Nieto, J. Motion-based calibration of multimodal sensor extrinsics and timing offset estimation. IEEE Trans. Robot. 2016, 32, 1215–1229. [Google Scholar] [CrossRef]
Method, Year | Mode | Data Conversion | Network Input | Feature Extraction for Camera Branch, Feature Extraction for LiDAR Branch | Feature Matching | Parameters Regression | Loss Function |
---|---|---|---|---|---|---|---|
RegNet [35], 2017 | End-to end | PC to Depth (1) | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | NiN blocks | MLP (4) | CPL with ED (5) |
RGBDTCalib-Net [48], 2022 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-34, ResNet-34 | CNN blocks | CNN + MLP | CPL with ED |
CalibNet [44], 2018 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | CNN blocks | CNN + MLP | CPL with ED + photometric loss + PC reprojection loss |
CalibRRNet [102], 2023 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | CNN blocks | RNN + MLP | Photometric loss + PC reprojection loss |
RGGNet [45], 2020 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-50, ResNet-50 | Feature concatenating layer | MLP | CPL with RD + PC reprojection loss |
LCCNet [46], 2021 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | Multi-resolution correlation blocks [61] | MLP | CPL with ED + PC reprojection loss |
CalibDNN [43], 2021 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18 | ResNet blocks | CNN + MLP | CPL with ED + photometric loss + PC reprojection loss |
NetCalib [66,101], 2021 | End-to end | RGB to Depth(2) PC to Depth | Depth and Depth | Custom CNN blocks | CNN + SPP | MLP | CPL with ED |
Calibration-Net [42], 2022 | End-to end | RGB to Depth PC to Depth | Depth and Depth | Custom CNN blocks | Multi-resolution correlation blocks | MLP | CPL with ED |
LIA-SC-Net [38], 2022 | End-to end | PC to Depth, Intensity (3) | RGB and Depth, Intensity | Pre-trained ResNet-18, ResNet-18 with attention mechanism | Multi-resolution correlation blocks | MLP | CPL with ED + structural consistency loss |
W. Zhang et al. [68], 2022 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | CNN blocks | MLP | Photometric loss + PC reprojection loss |
CalibcaNet [83], 2023 | End-to end | PC to Depth | RGB and Depth | ResNet-50 | Multi-resolution correlation blocks | MLP | CPL with ED |
Z. Zhang et al. [70], 2023 | End-to end | PC to Depth | LCCNet’s backbone [46] | CPL with ED + PC reprojection loss + supervised contrastive loss | |||
CALNet [39], 2022 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | CNN + HSPP + LTC | MLP | CPL with ED + PC reprojection loss |
PSNet [61], 2022 | End-to end | PC to Depth | RGB and Depth | HRNet-based [86] multi-resolution feature extraction | Multi-resolution feature aggregation + DenseNet | MLP | CPL with ED + PC reprojection loss |
CailbRCNN [57], 2020 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | CNN + LSTM | MLP | CPL with ED + synthetic view constraint loss + photometric loss + epipolar geometry constraint loss |
Causal calibration [64], 2023 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | CNN blocks | MLP | CPL with ED + synthetic view constraint loss + photometric loss |
SCNet [94], 2024 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | Multi-resolution correlation blocks [61] | MLP | CPL with ED + PC reprojection loss |
CalibBD [63], 2022 | End-to end | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | Multi-resolution correlation blocks [61] + Bi-LSTM | MLP | CPL with ED + photometric loss + synthetic view constraint loss |
UniCal [100], 2023 | End-to end | PC to Depth | RGB and Depth | MOBILEVIT [99] | MLP | CPL with ED + PC reprojection loss | |
CalibDepth [47], 2023 | End-to end | RGB to Depth PC to Depth | Depth and Depth | Pre-trained ResNet-18 | NiN + DCN [93] + LSTM | MLP | CPL with ED + photometric loss + PC reprojection loss |
CalibFormer [88], 2023 | End-to end | RGB to Depth PC to Depth | Depth and Depth | Multi-resolution feature extraction | Transformer-based feature matching | MLP | CPL with ED + PC reprojection loss |
PseudoCal [69], 2023 | Hybrid | RGB to PC | RGB and PC | PointPillars [107] | MOBILEVIT [99] | MLP | CPL with ED + PC reprojection loss |
CFNet [59], 2021 | Hybrid | PC to Depth | RGB and Depth | Pre-trained ResNet-18, ResNet-18 | Multi-resolution correlation blocks [61] | RANSAC + EPNP | Photometric loss with calibration flow + structural consistency loss |
DXQ-Net [41], 2022 | Hybrid | PC to Depth | RGB and Depth | Residual blocks in RAFT [65] | Pyramid 4D correlation-based matching [65] | Gauss Newton method | CPL with RD (6) + supervised calibration flow loss |
Method, Year | Mode | Feature Framework | 2D Feature Extraction Block | 3D Feature Extraction Block | Feature Objects | Description | Feature Matching/Fusion | Initial Guess | Calibration Model | Optimization |
---|---|---|---|---|---|---|---|---|---|---|
SOIC [36], 2020 | Hybrid | Bilateral (1) | WideResNet38 [118] | PointRCNN [121] | Vehicle/Pedestrian/Cyclist (semantic center) | Position/Semantic label | IPPE [134] | Semantic projection consistency loss model | Powell’s conjugate gradient descent | |
ATOP [40], 2023 | Hybrid | Bilateral | ERFNet [122] + MSHA [123] 2D segmentation network (point cloud to depth map) | Person/Other objects (semantic center) | Position/Semantic label/Feature map | Feature map metric | EPnP [103] | Semantic response consistency metric | Particle swarm optimization | |
SemCal [32], 2021 | Hybrid | Bilateral | HRNet + OCR [116] | SalsaNext [117] | Fourteen classes of objects | Position/Semantic label | Semantic-based MI registration | EPnP | Semantic mutual information optimization model | MINE [109] |
SemAlign [50], 2021 | Hybrid | Bilateral | SDCNet [118] | SPVNAS [119] | Vehicle/Sidewalk/Road | Position/Semantic label | Loss-Guided random sampling | Unidirectional alignment loss model | Adam optimizer | |
SST-Calib [110], 2022 | Hybrid | Bilateral | SDCNet [118] | SqueezeSeV3 [120] | Car | Position/Semantic label | Bidirectional alignment loss model | Adam optimizer | ||
SE-Calib [58], 2023 | Hybrid | Bilateral | DFF [124] | 3D-SS [125] | Vehicle edge | Position/Semantic label | Semantic response consistency metric model | Grid search [31] | ||
J. Zhu et al. [78], 2023 | Hybrid | Bilateral | EfficientDet [126] | SqueezeSeV2 [127] | Patch (window) | Position/Feature map | Cross Model GNN | Jensen–Shannon divergence-based loss + Point projection Euclidean space loss + CPL with ED | ||
CRLF [114], 2021 | Hybrid | Bilateral | BiSeNet-V2 [128] | RANSAC + traditional 3D line fitting | Straight line | Line position/Line equation | Hough transform-based line matching | P3L [135] | Semantic mask-based point cloud projection loss model | Gradient-descent method with annealing |
RGKCNet [106], 2022 | End to end | Bilateral | Microsoft’s human key point extraction network [129] | PointNet++ [130] | Key point | Position/Intensity/ Feature map | Correlation generation layer | Point projection Euclidean space loss + Jensen–Shannon divergence-based loss | Deep declaration network-based optimization [133] | |
Y. Zhu et al. [111], 2020 | Hybrid | Unilateral (2) | PSPNet [112] | 2D car | Position/Semantic label | Semantic mask | Semantic mask-based point cloud projection loss model | Gradient-descent method with annealing | ||
Calib-Anything [52], 2023 | Hybrid | Unilateral | Segment anything [113] | 2D multi-class objects | Position/Semantic label/Intensity/Normal vector | Semantic mask | Brute force search + random sampling | Semantic alignment loss model | ||
DedgeNet [108], 2022 | End to end | Unilateral | DedgeNet for line extraction | Global feature extraction (ResNet) | 2D straight line | Position/Feature map | Correlation generation layer | CPL with ED + point cloud reprojection loss | MLP-based global regression | |
FustionNet [115], 2023 | End to end | Unilateral | Global feature extraction (NiNs) | PointNet++ [130] for key point | 3D key point | Position/Feature map | Multi-resolution correlation blocks [61] | CPL with ED | MLP-based global regression |
Method, Year | Mis-Calibrated Range | Translation Absolute Error (cm) | Rotation Absolute Error (°) | Robustness | ||||||
---|---|---|---|---|---|---|---|---|---|---|
Mean | X | Y | Z | Mean | Roll | Pitch | Yaw | |||
Global Feature-Based AEPE | ||||||||||
RegNet [35], 2017 | [−1.5 m, 1.5 m]/[−20°, 20°] | 6 | 7 | 7 | 4 | 0.28 | 0.24 | 0.25 | 0.36 | ☆ |
RGBDTCalib-Net [48], 2022 | [−0.2 m, 0.2 m]/[−10°, 10°] | 6.2 | 2.6 | 5.8 | 0.48 | 0.92 | 0.3 | - | ||
CalibNet [44], 2018 | [−0.2 m, 0.2 m]/[−20°, 20°] | 4.34 | 4.2 | 1.6 | 7.22 | 0.41 | 0.18 | 0.9 | 0.15 | ☆ |
CalibRRNet [102], 2023 | [−0.2 m, 0.2 m]/[−10°, 10°] | 1.1347 | 1.0590 | 0.9319 | 1.4131 | 0.1754 | 0.0931 | 0.2290 | 0.2043 | ☆ |
LCCNet [46], 2021 | [−1.5 m, 1.5 m]/[−20°, 20°] | 1.588 | 0.243 | 0.380 | 0.459 | 0.163 | 0.030 | 0.019 | 0.040 | ☆ |
CalibDNN [43], 2021 | [−0.2 m, 0.2 m]/[−20°, 20°] | 5.3 | 6.2 | 4.3 | 5.4 | 0.42 | 0.18 | 0.35 | 0.11 | ☆ |
NetCalib [66,101], 2021 | [−0.2 m, 0.2 m]/[−2°, 2°] | 1.20 | 2.77 | 1.10 | 0.04 | 0.16 | 0.09 | - | ||
Calibration-Net [42], 2022 | [−1.5 m, 1.5 m]/[−20°, 20°] | 2.353 | 2.82 | 2.35 | 1.89 | 0.378 | 0.105 | 0.21 | 0.19 | ☆ |
LIA-SC-Net [38], 2022 | [−1.5 m, 1.5 m]/[−20°, 20°] | 3.96 | - | - | - | 0.525 | - | - | - | ☆ |
W. Zhang et al. [68], 2022 | - | 25 | 31 | 22 | 22 | 1.9 | 2.8 | 0.2 | 2.8 | ☆☆ |
CalibcaNet [83], 2023 | [−0.2 m, 0.2 m]/[−2°, 2°] | 6 | - | - | - | 0.154 | - | - | - | ☆ |
Z. Zhang et al. [70], 2023 | [−1.5 m, 1.5 m]/[−20°, 20°] | 0.33 | 0.23 | 0.45 | 0.30 | 0.15 | 0.14 | 0.13 | 0.17 | ☆ |
CALNet [39], 2022 | [−0.25 m, 0.25 m]/[−10°, 10°] | 3.03 | 3.65 | 1.63 | 3.80 | 0.20 | 0.10 | 0.38 | 0.12 | ☆ |
PSNet [61], 2022 | [−0.25 m, 0.25 m]/[−10°, 10°] | 3.1 | 3.8 | 2.8 | 2.6 | 0.15 | 0.06 | 0.26 | 0.12 | - |
CalibRCNN [57], 2020 | [−0.25 m, 0.25 m]/[−10°, 10°] | 9.3 | 6.2 | 4.3 | 5.4 | 0.805 | 0.446 | 0.64 | 0.199 | ☆ |
Causal calibration [64], 2023 | [−0.25 m, 0.25 m]/[−10°, 10°] | 1.5 | 2.01 | 0.88 | 1.6 | 0.12 | 0.0830 | 0.2291 | 0.0494 | - |
SCNet [94], 2024 | [−1.5 m, 1.5 m]/[−20°, 20°] | 0.724 | 0.294 | 0.507 | 0.480 | 0.055 | 0.032 | 0.039 | 0.022 | ☆ |
MRCNet [87],2023 | [−1.5 m, 1.5 m]/[−20°, 20°] | 0.35 | 0.256 | 0.259 | 0.535 | 0.033 | 0.016 | 0.046 | 0.038 | ☆ |
CalibBD [63], 2022 | [−0.2 m, 0.2 m]/[−20°, 20°] | 0.756 | 0.325 | 1.066 | 0.878 | 0.08 | 0.021 | 0.136 | 0.084 | ☆ |
UniCal [100], 2023 | [−1.5 m, 1.5 m]/[−20°, 20°] | 1.9 | 0.13 | ☆☆ | ||||||
CalibDepth [47], 2023 | [−1.5 m, 1.5 m]/[−20°, 20°] | 1.17 | 1.31 | 1.02 | 1.17 | 0.123 | 0.064 | 0.226 | 0.080 | ☆ |
CalibFormer [88], 2023 | [−0.25 m, 0.25 m]/[−10°, 10°] | 1.1877 | 1.1006 | 0.9015 | 1.5611 | 0.1406 | 0.0764 | 0.2588 | 0.0865 | - |
PseudoCal [69], 2023 | [−1.5 m, 1.5 m]/[−30°, 30°] | 1.18 | 0.05 | ☆ | ||||||
CFNet [59], 2021 | [−0.2 m, 0.2 m]/[−20°, 20°] | 0.831 | 0.463 | 1.230 | 0.802 | 0.086 | 0.028 | 0.125 | 0.105 | ☆ |
[−1.5 m, 1.5 m]/[−20°, 20°] | 0.995 | 1.025 | 0.919 | 1.042 | 0.087 | 0.059 | 0.110 | 0.092 | ||
DXQ-Net [41], 2022 | [−0.1 m, 0.1 m]/[−5°, 5°] | 1.425 | 0.754 | 0.476 | 1.091 | 0.084 | 0.049 | 0.046 | 0.032 | ☆☆ |
Local feature-based AEPE | ||||||||||
SOIC [36], 2020 | [−2 m, 2 m]/[−30°, 30°] | 4.1 | 1.5 | 13 | 0.075 | 0.073 | 0.177 | ☆ | ||
ATOP [40], 2023 | - | 1.752 | 0.976 | 0.222 | 4 | 0.08 | 0.14 | 0.09 | 0.02 | ☆ |
SemCal [32], 2021 | [−0.15 m, 0.15 m]/[−1.5°, 1.5°] | - | 3 | 6 | 6 | 0.14 | 0.17 | 0.13 | ☆☆ | |
SemAlign [50], 2021 | -/[−20°, 20°] | - | - | - | - | 1.49 | - | - | - | ☆ |
SST-Calib [110], 2022 | [−0.1 m, 0.1 m]/[−20°, 20°] | 18.9 | - | - | - | 0.60 | - | - | - | ☆ |
SE-Calib [58], 2023 | [−0.1 m, 0.1 m]/[−7°, 7°] | - | - | - | - | 0.130 | 0.095 | 0.100 | 0.129 | - |
J. Zhu et al. [78], 2023 | [−0.21 m, 0.21 m]/[−15°, 15°] | 1.6 | - | - | - | 0.15 | - | - | - | - |
CRLF [114], 2021 | [−1 m, 1 m]/[−6°, 6°] | 8.2 | 4.6 | 9.7 | 0.216 | 0.546 | 0.492 | - | ||
RGKCNet [106], 2022 | [−0.2 m, 0.2 m]/[−7.5°, 7.5°] | 11.7 | 5 | 4 | 5.9 | 0.43 | 0.17 | 0.15 | 0.16 | ☆ |
Calib-Anything [52], 2023 | - | 10.7 | - | - | - | 0.174 | - | - | - | ☆☆ |
DedgeNet [108], 2022 | [−1.5 m, 1.5 m]/[−20°, 20°] | 1.109 | 0.247 | 0.330 | 0.153 | 0.159 | 0.052 | 0.014 | 0.018 | ☆ |
REPP | ||||||||||
CalibRank [37], 2020 | [−0.28 m, 0.28 m]/[−10°, 10°] | 7.01 | 2.07 | 6.81 | 0.17 | 0.68 | 0.09 | - |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tan, Z.; Zhang, X.; Teng, S.; Wang, L.; Gao, F. A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration. Sensors 2024, 24, 3878. https://doi.org/10.3390/s24123878
Tan Z, Zhang X, Teng S, Wang L, Gao F. A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration. Sensors. 2024; 24(12):3878. https://doi.org/10.3390/s24123878
Chicago/Turabian StyleTan, Zhiguo, Xing Zhang, Shuhua Teng, Ling Wang, and Feng Gao. 2024. "A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration" Sensors 24, no. 12: 3878. https://doi.org/10.3390/s24123878