MambaReID: Exploiting Vision Mamba for Multi-Modal Object Re-Identification
Abstract
:1. Introduction
- We introduce a novel fusion framework named MambaReID. This work marks the first attempt at integrating Mamba into multi-modal object ReID.
- We propose the Three-Stage VMamba (TSV) to extract robust multi-modal representations. TSV efficiently captures both global context information and local details, while maintaining low computational complexity.
- We introduce Dense Mamba (DM) to seamlessly integrate inter-modality information across shallow and deep features. Moreover, we present Consistent VMamba Fusion (CVF) to fuse deep features originating from diverse modalities. Leveraging well-aligned multi-modal images, CVF refines modal aggregation granularity, consequently enhancing feature discriminability.
2. Proposed Methodology
2.1. Overall Architecture of MambaReID
2.2. Preliminary
2.3. Three-Stage VMamba
2.4. Dense Mamba
2.5. Consistent VMamba Fusion
2.6. Objective Function
3. Experiments
3.1. Dataset and Experimental Setup
3.2. Implementation Details
3.3. Comparisons With State-of-the-Art Methods
3.4. Ablation Study
3.5. Visualization Analysis
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Ye, M.; Shen, J.; Lin, G.; Xiang, T.; Shao, L.; Hoi, S. Deep learning for person re-identification: A survey and outlook. TPAMI 2021, 44, 2872–2893. [Google Scholar] [CrossRef]
- Ye, M.; Chen, S.; Li, C.; Zheng, W.; Crandall, D.; Du, B. Transformer for Object Re-Identification: A Survey. arXiv 2024, arXiv:2401.06960. [Google Scholar]
- Amiri, A.; Kaya, A.; Keceli, A. A Comprehensive Survey on Deep-Learning-based Vehicle Re-Identification: Models, Data Sets and Challenges. arXiv 2024, arXiv:2401.10643. [Google Scholar]
- Zheng, A.; Wang, Z.; Chen, Z.; Li, C.; Tang, J. Robust multi-modality person re-identification. Proc. AAAI Conf. Artif. Intell. 2021, 35, 3529–3537. [Google Scholar] [CrossRef]
- Li, H.; Li, C.; Zhu, X.; Zheng, A.; Luo, B. Multi-spectral vehicle re-identification: A challenge. Proc. AAAI Conf. Artif. Intell. 2020, 34, 11345–11353. [Google Scholar] [CrossRef]
- Zheng, A.; He, Z.; Wang, Z.; Li, C.; Tang, J. Dynamic Enhancement Network for Partial Multi-modality Person Re-identification. arXiv 2023, arXiv:2305.15762. [Google Scholar]
- Wang, Y.; Liu, X.; Zhang, P.; Lu, H.; Tu, Z.; Lu, H. TOP-ReID: Multi-spectral Object Re-Identification with Token Permutation. Proc. AAAI Conf. Artif. Intell. 2024, 38, 5758–5766. [Google Scholar] [CrossRef]
- Zhang, P.; Wang, Y.; Liu, Y.; Tu, Z.; Lu, H. Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 17–21 June 2024. [Google Scholar]
- Wang, Z.; Li, C.; Zheng, A.; He, R.; Tang, J. Interact, embed, and enlarge: Boosting modality-specific representations for multi-modal person re-identification. Proc. AAAI Conf. Artif. Intell. 2022, 36, 2633–2641. [Google Scholar] [CrossRef]
- Guo, J.; Zhang, X.; Liu, Z.; Wang, Y. Generative and attentive fusion for multi-spectral vehicle re-identification. In Proceedings of the 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP), Xi’an, China, 15–17 April 2022; pp. 1565–1572. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. NIPS 2017, 30, 6000–6010. [Google Scholar]
- Pan, W.; Huang, L.; Liang, J.; Hong, L.; Zhu, J. Progressively Hybrid Transformer for Multi-Modal Vehicle Re-Identification. Sensors 2023, 23, 4206. [Google Scholar] [CrossRef] [PubMed]
- Crawford, J.; Yin, H.; McDermott, L.; Cummings, D. UniCat: Crafting a Stronger Fusion Baseline for Multimodal Re-Identification. arXiv 2023, arXiv:2310.18812. [Google Scholar]
- Wang, Z.; Huang, H.; Zheng, A.; He, R. Heterogeneous Test-Time Training for Multi-Modal Person Re-identification. Proc. AAAI Conf. Artif. Intell. 2024, 38, 5850–5858. [Google Scholar] [CrossRef]
- Lu, Z.; Xie, H.; Liu, C.; Zhang, Y. Bridging the gap between vision transformers and convolutional neural networks on small datasets. Adv. Neural Inf. Process. Syst. 2022, 35, 14663–14677. [Google Scholar]
- Gu, A.; Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv 2023, arXiv:2312.00752. [Google Scholar]
- Yang, Y.; Xing, Z.; Zhu, L. Vivim: A Video Vision Mamba for Medical Video Object Segmentation. arXiv 2024, arXiv:2401.14168. [Google Scholar]
- Smith, J.T.; Warrington, A.; Linderman, S.W. Simplified state space layers for sequence modeling. arXiv 2022, arXiv:2208.04933. [Google Scholar]
- Liu, Y.; Tian, Y.; Zhao, Y.; Yu, H.; Xie, L.; Wang, Y.; Ye, Q.; Liu, Y. Vmamba: Visual state space model. arXiv 2024, arXiv:2401.10166. [Google Scholar]
- Zhu, L.; Liao, B.; Zhang, Q.; Wang, X.; Liu, W.; Wang, X. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv 2024, arXiv:2401.09417. [Google Scholar]
- Wan, Z.; Wang, Y.; Yong, S.; Zhang, P.; Stepputtis, S.; Sycara, K.; Xie, Y. Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation. arXiv 2024, arXiv:2404.04256. [Google Scholar]
- Luo, H.; Gu, Y.; Liao, X.; Lai, S.; Jiang, W. Bag of tricks and a strong baseline for deep person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Zhao, B.; Feng, J.; Wu, X.; Yan, S. A survey on deep learning-based fine-grained object classification and semantic segmentation. Int. J. Autom. Comput. 2017, 14, 119–135. [Google Scholar] [CrossRef]
- Gu, A.; Goel, K.; Ré, C. Efficiently modeling long sequences with structured state spaces. arXiv 2021, arXiv:2111.00396. [Google Scholar]
- Gu, A.; Johnson, I.; Goel, K.; Saab, K.; Dao, T.; Rudra, A.; Ré, C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Adv. Neural Inf. Process. Syst. 2021, 34, 572–585. [Google Scholar]
- Ba, J.; Kiros, J.; Hinton, G. Layer normalization. arXiv 2016, arXiv:1607.06450. [Google Scholar]
- Chollet, F. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 1251–1258. [Google Scholar]
- Elfwing, S.; Uchibe, E.; Doya, K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Netw. 2018, 107, 3–11. [Google Scholar] [CrossRef] [PubMed]
- Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4700–4708. [Google Scholar]
- Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2818–2826. [Google Scholar]
- Hermans, A.; Beyer, L.; Leibe, B. In defense of the triplet loss for person re-identification. arXiv 2017, arXiv:1703.07737. [Google Scholar]
- Zheng, A.; Zhu, X.; Ma, Z.; Li, C.; Tang, J.; Ma, J. Multi-spectral vehicle re-identification with cross-directional consistency network and a high-quality benchmark. arXiv 2022, arXiv:2208.00632. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 248–255. [Google Scholar]
- Zhong, Z.; Zheng, L.; Kang, G.; Li, S.; Yang, Y. Random erasing data augmentation. Proc. AAAI Conf. Artif. Intell. 2020, 34, 13001–13008. [Google Scholar] [CrossRef]
- Qian, X.; Fu, Y.; Jiang, Y.G.; Xiang, T.; Xue, X. Multi-scale deep learning architectures for person re-identification. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 5399–5408. [Google Scholar]
- Li, W.; Zhu, X.; Gong, S. Harmonious attention network for person re-identification. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 2285–2294. [Google Scholar]
- Chang, X.; Hospedales, T.M.; Xiang, T. Multi-level factorisation net for person re-identification. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 2109–2118. [Google Scholar]
- Sun, Y.; Zheng, L.; Yang, Y.; Tian, Q.; Wang, S. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In Proceedings of the ECCV, Munich, Germany, 8–14 September 2018; pp. 480–496. [Google Scholar]
- Zhou, K.; Yang, Y.; Cavallaro, A.; Xiang, T. Omni-scale feature learning for person re-identification. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 3702–3712. [Google Scholar]
- Rao, Y.; Chen, G.; Lu, J.; Zhou, J. Counterfactual attention learning for fine-grained visual categorization and re-identification. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; pp. 1025–1034. [Google Scholar]
- Wang, G.; Yuan, Y.; Chen, X.; Li, J.; Zhou, X. Learning discriminative features with multiple granularities for person re-identification. In Proceedings of the MM ’18: Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Republic of Korea, 22–26 October 2018; pp. 274–282. [Google Scholar]
- Chen, G.; Zhang, T.; Lu, J.; Zhou, J. Deep meta metric learning. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 9547–9556. [Google Scholar]
- Sun, Y.; Cheng, C.; Zhang, Y.; Zhang, C.; Zheng, L.; Wang, Z.; Wei, Y. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 6398–6407. [Google Scholar]
- Zhao, J.; Zhao, Y.; Li, J.; Yan, K.; Tian, Y. Heterogeneous relational complement for vehicle re-identification. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; pp. 205–214. [Google Scholar]
- He, S.; Luo, H.; Wang, P.; Wang, F.; Li, H.; Jiang, W. Transreid: Transformer-based object re-identification. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; pp. 15013–15022. [Google Scholar]
- Yin, H.; Li, J.; Schiller, E.; McDermott, L.; Cummings, D. GraFT: Gradual Fusion Transformer for Multimodal Re-Identification. arXiv 2023, arXiv:2310.16856. [Google Scholar]
- He, Q.; Lu, Z.; Wang, Z.; Hu, H. Graph-Based Progressive Fusion Network for Multi-Modality Vehicle Re-Identification. IEEE Trans. Intell. Transp. Syst. 2023, 24, 12431–12447. [Google Scholar] [CrossRef]
- Van der Maaten, L.; Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 2008, 9, 2579–2605. [Google Scholar]
(a) Comparison on RGBNT201 | ||||||
---|---|---|---|---|---|---|
Methods | RGBNT201 | |||||
Modality | Venue | Model | mAP | R-1 | R-5 | R-10 |
Single | ICCV’17 | MUDeep [35] | 23.8 | 19.7 | 33.1 | 44.3 |
CVPR’18 | HACNN [36] | 21.3 | 19.0 | 34.1 | 42.8 | |
CVPR’18 | MLFN [37] | 26.1 | 24.2 | 35.9 | 44.1 | |
ECCV’18 | PCB [38] | 32.8 | 28.1 | 37.4 | 46.9 | |
ICCV’19 | OSNet [39] | 25.4 | 22.3 | 35.1 | 44.7 | |
ICCV’21 | CAL [40] | 27.6 | 24.3 | 36.5 | 45.7 | |
Multi | AAAI’20 | HAMNet [5] | 27.7 | 26.3 | 41.5 | 51.7 |
AAAI’21 | PFNet [4] | 38.5 | 38.9 | 52.0 | 58.4 | |
AAAI’22 | IEEE [9] | 49.5 | 48.4 | 59.1 | 65.6 | |
ArXiv’23 | DENet [6] | 42.4 | 42.2 | 55.3 | 64.5 | |
NeurIPSW’23 | UniCat * [13] | 57.0 | 55.7 | - | - | |
AAAI’24 | HTT * [14] | 71.1 | 73.4 | 83.1 | 87.3 | |
AAAI’24 | TOP-ReID (A) * [7] | 72.3 | 76.6 | 84.7 | 89.4 | |
AAAI’24 | TOP-ReID (B) * [7] | 64.6 | 64.6 | 77.4 | 82.4 | |
CVPR’24 | EDITOR (A) * [8] | 66.5 | 68.3 | 81.1 | 88.2 | |
CVPR’24 | EDITOR (B) * [8] | 65.7 | 68.8 | 82.5 | 89.1 | |
Ours | MambaReID † (A) | 72.2 | 76.0 | 84.0 | 89.0 | |
Ours | MambaReID † (B) | 65.1 | 67.4 | 78.4 | 84.4 | |
(b) Comparison on RGBNT100 and MSVR310 | ||||||
Methods | RGBNT100 | MSVR310 | ||||
Modality | Venue | Model | mAP | R-1 | mAP | R-1 |
Single | ECCV’18 | PCB [38] | 57.2 | 83.5 | 23.2 | 42.9 |
ACM MM’18 | MGN [41] | 58.1 | 83.1 | 26.2 | 44.3 | |
ICCV’19 | DMML [42] | 58.5 | 82.0 | 19.1 | 31.1 | |
CVPRW’19 | BoT [22] | 78.0 | 95.1 | 23.5 | 38.4 | |
ICCV’19 | OSNet [39] | 75.0 | 95.6 | 28.7 | 44.8 | |
CVPR’20 | Circle Loss [43] | 59.4 | 81.7 | 22.7 | 34.2 | |
ICCV’21 | HRCN [44] | 67.1 | 91.8 | 23.4 | 44.2 | |
TPAMI’21 | AGW [1] | 73.1 | 92.7 | 28.9 | 46.9 | |
ICCV’21 | TransReID * [45] | 75.6 | 92.9 | 18.4 | 29.6 | |
Multi | AAAI’20 | HAMNet [5] | 74.5 | 93.3 | 27.1 | 42.3 |
AAAI’21 | PFNet [4] | 68.1 | 94.1 | 23.5 | 37.4 | |
ICSP’22 | GAFNet [10] | 74.4 | 93.4 | - | - | |
Inform Fusion’22 | CCNet [32] | 77.2 | 96.3 | 36.4 | 55.2 | |
ArXiv’23 | GraFT * [46] | 76.6 | 94.3 | - | - | |
TITS’23 | GPFNet [47] | 75.0 | 94.5 | - | - | |
Sensors’23 | PHT * [12] | 79.9 | 92.7 | - | - | |
NeruIPSW’23 | UniCat * [13] | 79.4 | 96.2 | - | - | |
AAAI’24 | HTT * [14] | 75.7 | 92.6 | - | - | |
AAAI’24 | TOP-ReID (A) * [7] | 73.7 | 92.2 | 30.2 | 33.7 | |
AAAI’24 | TOP-ReID (B) * [7] | 81.2 | 96.4 | 35.9 | 44.6 | |
CVPR’24 | EDITOR (A) * [8] | 79.8 | 93.9 | 35.8 | 43.1 | |
CVPR’24 | EDITOR (B) * [8] | 82.1 | 96.4 | 39.0 | 49.3 | |
Ours | MambaReID † (A) | 76.6 | 93.4 | 35.8 | 46.0 | |
Ours | MambaReID † (B) | 78.6 | 94.3 | 46.1 | 59.4 |
Module | Complexity | RGBNT201 | ||||||
---|---|---|---|---|---|---|---|---|
DM | CVF | Params (M) | FLOPs (G) | mAP | R-1 | R-5 | R-10 | |
A | ✕ | ✕ | 55.67 | 26.99 | 63.81 | 67.57 | 77.97 | 83.77 |
B | ✕ | ✓ | 59.47 | 27.56 | 68.03 | 70.96 | 79.88 | 84.69 |
C | ✓ | ✕ | 55.67 | 26.99 | 68.07 | 72.38 | 82.70 | 86.96 |
D | ✓ | ✓ | 59.47 | 27.56 | 72.20 | 75.96 | 83.97 | 89.00 |
Methods | Params | RGBNT100 | ||
---|---|---|---|---|
Venue | Model | M | mAP | Rank-1 |
ECCV’18 | PCB [38] | 72.33 | 57.2 | 83.5 |
ICCV’19 | OSNet [39] | 7.02 | 75.0 | 95.6 |
AAAI’20 | HAMNet [5] | 78.00 | 74.5 | 93.3 |
Inform Fusion’22 | CCNet [32] | 74.60 | 77.2 | 96.3 |
ICSP’22 | GAFNet [10] | 130.00 | 74.4 | 93.4 |
ICCV’21 | TransReID * [45] | 278.23 | 75.6 | 92.9 |
NeruIPSW’23 | UniCat * [13] | 259.02 | 79.4 | 96.2 |
ArXiv’23 | GraFT * [46] | 101.00 | 76.6 | 94.3 |
AAAI’24 | TOP-ReID * [7] | 324.53 | 81.2 | 96.4 |
CVPR’24 | EDITOR * [8] | 118.55 | 82.1 | 96.4 |
Ours | MambaReID † | 59.47 | 78.6 | 94.3 |
Model | Params | FLOPs | RGBNT201 | |||
---|---|---|---|---|---|---|
Backbone | M | G | mAP | Rank-1 | Rank-5 | Rank-10 |
ViT | 87.04 | 34.07 | 63.18 | 64.58 | 77.91 | 84.56 |
VMamba | 88.06 | 30.09 | 55.98 | 57.85 | 71.60 | 78.65 |
VMamba ‡ | 88.06 | 39.38 | 58.47 | 60.92 | 74.04 | 80.72 |
TSV | 55.67 | 26.99 | 63.81 | 67.57 | 77.97 | 83.77 |
CVF | Params | FLOPs | RGBNT201 | |||
---|---|---|---|---|---|---|
Depth | M | G | mAP | Rank-1 | Rank-5 | Rank-10 |
1 | 59.47 | 27.56 | 68.03 | 70.96 | 79.88 | 84.69 |
2 | 62.75 | 28.08 | 66.11 | 67.89 | 78.04 | 83.74 |
3 | 66.02 | 28.60 | 64.83 | 66.96 | 77.98 | 84.03 |
4 | 69.30 | 29.12 | 63.01 | 65.72 | 76.52 | 82.33 |
5 | 72.58 | 29.64 | 65.07 | 68.18 | 78.18 | 83.82 |
6 | 75.85 | 30.16 | 62.42 | 64.58 | 75.01 | 80.49 |
DM | Params | FLOPs | RGBNT201 | |||||
---|---|---|---|---|---|---|---|---|
Last | Freq | M | G | mAP | R-1 | R-5 | R-10 | |
A | ✓ | 1 | 55.67 | 26.99 | 63.51 | 68.03 | 78.94 | 84.63 |
B | ✕ | 1 | 55.67 | 26.99 | 66.63 | 70.98 | 81.35 | 86.23 |
C | ✓ | 2 | 55.67 | 26.99 | 66.04 | 70.51 | 81.02 | 86.18 |
D | ✕ | 2 | 55.67 | 26.99 | 68.07 | 72.38 | 82.70 | 86.96 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhang, R.; Xu, L.; Yang, S.; Wang, L. MambaReID: Exploiting Vision Mamba for Multi-Modal Object Re-Identification. Sensors 2024, 24, 4639. https://doi.org/10.3390/s24144639
Zhang R, Xu L, Yang S, Wang L. MambaReID: Exploiting Vision Mamba for Multi-Modal Object Re-Identification. Sensors. 2024; 24(14):4639. https://doi.org/10.3390/s24144639
Chicago/Turabian StyleZhang, Ruijuan, Lizhong Xu, Song Yang, and Li Wang. 2024. "MambaReID: Exploiting Vision Mamba for Multi-Modal Object Re-Identification" Sensors 24, no. 14: 4639. https://doi.org/10.3390/s24144639