Abstract
Irregular boundaries in image stitching naturally occur due to freely moving cameras. To deal with this problem, existing methods focus on optimizing mesh warping to make boundaries regular using the traditional explicit solution. However, previous methods always depend on hand-crafted features (e.g., keypoints and line segments). Thus, failures often happen in overlapping regions without distinctive features. In this paper, we address this problem by proposing RecStitchNet, a reasonable and effective network for image stitching with rectangular boundaries. Considering that both stitching and imposing rectangularity are non-trivial tasks in the learning-based framework, we propose a three-step progressive learning based strategy, which not only simplifies this task, but gradually achieves a good balance between stitching and imposing rectangularity. In the first step, we perform initial stitching by a pre-trained state-of-the-art image stitching model, to produce initially warped stitching results without considering the boundary constraint. Then, we use a regression network with a comprehensive objective regarding mesh, perception, and shape to further encourage the stitched meshes to have rectangular boundaries with high content fidelity. Finally, we propose an unsupervised instance-wise optimization strategy to refine the stitched meshes iteratively, which can effectively improve the stitching results in terms of feature alignment, as well as boundary and structure preservation. Due to the lack of stitching datasets and the difficulty of label generation, we propose to generate a stitching dataset with rectangular stitched images as pseudo-ground-truth labels, and the performance upper bound induced from the it can be broken by our unsupervised refinement. Qualitative and quantitative results and evaluations demonstrate the advantages of our method over the state-of-the-art.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Chen, Y. S.; Chuang, Y. Y. Natural image stitching with the global similarity prior. In: Computer Vision–ECCV 2016. Lecture Notes in Computer Science, Vol. 9909. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 186–201, 2016.
Gao, J.; Kim, S. J.; Brown, M. S. Constructing image panoramas using dual-homography warping. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, 49–56, 2011.
Lin, C. C.; Pankanti, S. U.; Ramamurthy, K. N.; Aravkin, A. Y. Adaptive as-natural-as-possible image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1155–1163, 2015.
Nie, L.; Lin, C.; Liao, K.; Liu, M.; Zhao, Y. A view-free image stitching network based on global homography. Journal of Visual Communication and Image Representation Vol. 73, Article No. 102950, 2020.
Zhao, Q.; Ma, Y.; Zhu, C.; Yao, C.; Feng, B.; Dai, F. Image stitching via deep homography estimation. Neurocomputing Vol. 450, 219–229, 2021.
Kweon, H.; Kim, H.; Kang, Y.; Yoon, Y.; Jeong, W.; Yoon, K. J. Pixel-wise warping for deep image stitching. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 37, No. 1, 1196–1204, 2023.
Song, D. Y.; Lee, G.; Lee, H.; Um, G. M.; Cho, D. Weakly-supervised stitching network for real-world panoramic image generation. In: Computer Vision–ECCV 2022. Lecture Notes in Computer Science, Vol. 13676. Avidan, S.; Brostow, G.; Cisse, M.; Farinella, G. M.; Hassner, T. Eds. Springer Cham, 54–71, 2022.
Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Unsupervised deep image stitching: Reconstructing stitched features to images. IEEE Transactions on Image Processing Vol. 30, 6184–6197, 2021.
Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Parallax-tolerant unsupervised deep image stitching. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7365–7374, 2023.
He, K.; Chang, H.; Sun, J. Rectangling panoramic images via warping. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 79, 2013.
Zhang, Y.; Lai, Y. K.; Zhang, F. L. Content-preserving image stitching with piecewise rectangular boundary constraints. IEEE Transactions on Visualization and Computer Graphics Vol. 27, No. 7, 3198–3212, 2021.
Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Deep rectangling for image stitching: A learning baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5730–5738, 2022.
Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Deep rotation correction without angle prior. IEEE Transactions on Image Processing Vol. 32, 2879–2888, 2023.
Brown, M.; Lowe, D. G. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision Vol. 74, No. 1, 59–73, 2007.
Lin, W. Y.; Liu, S.; Matsushita, Y.; Ng, T. T.; Cheong, L. F. Smoothly varying affine stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 345–352, 2011.
Zaragoza, J.; Chin, T. J.; Tran, Q. H.; Brown, M. S.; Suter, D. As-projective-as-possible image stitching with moving DLT. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 7, 1285–1298, 2014.
Lou, Z.; Gevers, T. Image alignment by piecewise planar region matching. IEEE Transactions on Multimedia Vol. 16, No. 7, 2052–2061, 2014.
Lin, K.; Jiang, N.; Cheong, L. F.; Do, M.; Lu, J. SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Computer Vision–ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 370–385, 2016.
Zhang, F.; Liu, F. Parallax-tolerant image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3262–3269, 2014.
Gao, J.; Li, Y.; Chin, T.; Brown, M. S. Seam-driven image stitching. In: Proceedings of the 34th Annual Conference of the European Association for Computer Graphics, 2013.
Chang, C. H.; Sato, Y.; Chuang, Y. Y. Shape-preserving half-projective warps for image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3254–3261, 2014.
Li, N.; Xu, Y.; Wang, C. Quasi-homography warps in image stitching. IEEE Transactions on Multimedia Vol. 20, No. 6, 1365–1375, 2018.
Liao, T.; Li, N. Single-perspective warps in natural image stitching. IEEE Transactions on Image Processing Vol. 29, 724–735, 2020.
Du, P.; Ning, J.; Cui, J.; Huang, S.; Wang, X.; Wang, J. Geometric structure preserving warp for natural image stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3678–3686, 2022.
Jia, Q.; Li, Z.; Fan, X.; Zhao, H.; Teng, S.; Ye, X.; Latecki, L. J. Leveraging line-point consistence to preserve structures for wide parallax image stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12181–12190, 2021.
Zhang, L.; Huang, H. Image stitching with manifold optimization. IEEE Transactions on Multimedia Vol. 25, 3469–3482, 2023.
Li, N.; Liao, T.; Wang, C. Perception-based seam cutting for image stitching. Signal, Image and Video Processing Vol. 12, No. 5, 967–974, 2018.
Lai, W. S.; Gallo, O.; Gu, J.; Sun, D.; Yang, M. H.; Kautz, J. Video stitching for linear camera arrays. In: Proceedings of the 30th British Machine Vision Conference, 2019.
He, K.; Chang, H.; Sun, J. Content-aware rotation. In: Proceedings of the IEEE International Conference on Computer Vision, 553–560, 2013.
Wu, J. L.; Shi, J. J.; Zhang, L. Rectangling irregular videos by optimal spatio-temporal warping. Computational Visual Media Vol. 8, No. 1, 93–103, 2022.
Liao, K.; Nie, L.; Lin, C.; Zheng, Z.; Zhao, Y. RecRecNet: Rectangling rectified wide-angle images by thin-plate spline model and DoF-based curriculum learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 10766–10775, 2023.
Zhou, H.; Zhu, Y.; Lv, X.; Liu, Q.; Zhang, S. Rectangular-output image stitching. In: Proceedings of the IEEE International Conference on Image Processing, 2800–2804, 2023.
Jaderberg, M.; Simonyan, K.; Zisserman, A.; Kavukcuoglu, K. Spatial transformer networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, Vol. 2, 2017–2025, 2015.
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations, 2015.
Martinez, F.; Rueda, A. J.; Feito, F. R. A new algorithm for computing Boolean operations on polygons. Computers & Geosciences Vol. 35, No. 6, 1177–1185, 2009.
Wang, M.; Shamir, A.; Yang, G. Y.; Lin, J. K.; Yang, G. W.; Lu, S. P.; Hu, S. M. BiggerSelfie: Selfie video expansion with hand-held camera. IEEE Transactions on Image Processing Vol. 27, No. 12, 5854–5865, 2018.
Nie, Y.; Su, T.; Zhang, Z.; Sun, H.; Li, G. Dynamic video stitching via shakiness removing. IEEE Transactions on Image Processing Vol. 27, No. 1, 164–178, 2018.
Wang, M.; Yang, G. Y.; Lin, J. K.; Zhang, S. H.; Shamir, A.; Lu, S. P.; Hu, S. M. Deep online video stabilization with multi-grid warping transformation learning. IEEE Transactions on Image Processing Vol. 28, No. 5, 2283–2292, 2019.
Rong, J. X.; Zhang, L.; Huang, H.; Zhang, F. L. IMU-assisted online video background identification. IEEE Transactions on Image Processing Vol. 31, 4336–4351, 2022.
Acknowledgements
This research was supported by the Zhejiang Province Basic Public Welfare Research Program (No. LGG22F020009), Key Lab of Film and TV Media Technology of Zhejiang Province (No. 2020E10015), and Marsden Fund Council managed by the Royal Society of New Zealand (No. MFP-20-VUW-180). We also would like to express our special gratitude to Dr. Yaqi Wang for her great work in diagram enhancement.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Yun Zhang is currently a professor in the College of Media Engineering, Communications University of Zhejiang, China. He received his doctoral degree from Zhejiang University in 2013. Before that, he received his bachelor and master degrees from Hangzhou Dianzi University in 2006 and 2009, respectively. He visited the Visual Computing Group of Cardiff University in 2018 and 2023. His research interests include computer graphics, image and video editing, and virtual reality. He is a senior member of the CCF.
Yu-Kun Lai received his bachelor and Ph.D. degrees in computer science from Tsinghua University in 2003 and 2008, respectively. He is currently a professor in the School of Computer Science & Informatics, Cardiff University. His research interests include computer graphics, geometry processing, image processing, and computer vision. He is on the editorial boards of IEEE Transactions on Visualization and Computer Graphics and The Visual Computer.
Lang Nie received his B.S degree in computer science and technology from Beijing Jiaotong University, China, in 2019, and is currently pur-suing a Ph.D. degree in signal and information processing from the Institute of Information Science, Beijing Jiaotong University. His current research interests include image and video processing, 3D vision, and multiview geometry.
Fang-Lue Zhang received his Ph.D. degree from Tsinghua University, in 2015. He is currently a senior lecturer in computer graphics at the Victoria University of Wellington, New Zealand. His research interests include image and video editing, computer vision, and computer graphics. He received a Victoria Early-Career Research Excellence Award, in 2019, and a Fast-Start Marsden Grant from the New Zealand Royal Society, in 2020. He is on the editorial board of Computers & Graphics.
Lin Xu is currently pursuing her Ph.D. degree at the University of South Australia, with an anticipated graduation date in 2024. She received her bachelor and master degrees, as well as her first Ph.D. degree from Fujian Normal University of China in 2006, 2010, and 2014, respectively. Lin furthered her academic pursuits with visits to the College of Creativity and Technology at Fo Guang University in 2015 and the Faculty of Sciences at Université Libre de Bruxelles in 2017. Her research interests include computer vision, multi-media systems, and intelligence computing.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Zhang, Y., Lai, YK., Nie, L. et al. RecStitchNet: Learning to stitch images with rectangular boundaries. Comp. Visual Media 10, 687–703 (2024). https://doi.org/10.1007/s41095-024-0420-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-024-0420-6