GitHub - DocF/multispectral-object-detection: Multisp... https://github.

com/DocF/multispectral-object-detection

DocF / multispectral-object-detection Public

Multispectral Object Detection with Yolov5 and Transformer

AGPL-3.0 license

309 stars 58 forks Branches Tags Activity

Star Notifications

Code Issues 49 Pull requests Actions Projects Security Insights

1 of 7 10/28/24, 11:18
GitHub - DocF/multispectral-object-detection: Multisp... https://github.com/DocF/multispectral-object-detection

1 Branch Tags Go to file Go to file Code

DocF Update README.md fb591c9 · 5 months ago

__pycache__ clean the code 3 years ago

data clean the code 3 years ago

models clean the code 3 years ago

utils clean the code 3 years ago

video update demo 3 years ago

.DS_Store add figure 3 years ago

LICENSE clean the code 3 years ago

MR.png Update MR.png 3 years ago

README.md Update README.md 5 months ago

cft.png add figure 3 years ago

detect_twostream.py clean the code 3 years ago

example.png clean the code 3 years ago

global_var.py clean the code 3 years ago

hubconf.py clean the code 3 years ago

requirements.txt clean the code 3 years ago

test.py clean the code 3 years ago

train.py clean the code 3 years ago

README AGPL-3.0 license

Multispectral-Object-Detection
custom badge resource not found

Ranked #3 Pedestrian Detection on LLVIP

2021 NEW Stars 309

2 of 7 10/28/24, 11:18
GitHub - DocF/multispectral-object-detection: Multisp... https://github.com/DocF/multispectral-object-detection

Visitors Stars 309

Intro
Official Code for Cross-Modality Fusion Transformer for Multispectral Object Detection.

Multispectral Object Detection with Transformer and Yolov5

Abstract
Multispectral image pairs can provide the combined information, making object detection
applications more reliable and robust in the open world. To fully exploit the different
modalities, we present a simple yet effective cross-modality feature fusion approach,
named Cross-Modality Fusion Transformer (CFT) in this paper. Unlike prior CNNs-based
works, guided by the Transformer scheme, our network learns long-range dependencies
and integrates global contextual information in the feature extraction stage. More
importantly, by leveraging the self attention of the Transformer, the network can naturally
carry out simultaneous intra-modality and inter-modality fusion, and robustly capture the
latent interactions between RGB and Thermal domains, thereby significantly improving
the performance of multispectral object detection. Extensive experiments and ablation
studies on multiple datasets demonstrate that our approach is effective and achieves
state-of-the-art detection performance.

Demo
Night Scene

Day Scene

Overview

3 of 7 10/28/24, 11:18
GitHub - DocF/multispectral-object-detection: Multisp... https://github.com/DocF/multispectral-object-detection

Citation
If you use this repo for your research, please cite our paper:

@article{qingyun2022cross,
title={Cross-modality attentive feature fusion for object
detection in multispectral remote sensing imagery},
author={Qingyun, Fang and Zhaokui, Wang},
journal={Pattern Recognition},
volume={130},
pages={108786},
year={2022},
publisher={Elsevier}
}
@article{fang2021cross,
title={Cross-Modality Fusion Transformer for Multispectral Object
Detection},
author={Fang Qingyun and Han Dapeng and Wang Zhaokui},
journal={arXiv preprint arXiv:2111.00273},
year={2021}
}

Installation
Python>=3.6.0 is required with all requirements.txt installed including PyTorch>=1.7 (The
same as yolov5 https://github.com/ultralytics/yolov5 ).

Clone the repo

git clone https://github.com/DocF/multispectral-object-detection

4 of 7 10/28/24, 11:18
GitHub - DocF/multispectral-object-detection: Multisp... https://github.com/DocF/multispectral-object-detection

Install requirements

$ cd multispectral-object-detection
$ pip install -r requirements.txt

Dataset
-[FLIR] [Google Drive] [Baidu Drive] extraction code:qwer

A new aligned version.

-[LLVIP] download

-[VEDAI] download

You need to convert all annotations to YOLOv5 format.

Refer: https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data

Run

Download the pretrained weights

yolov5 weights (pre-train)

-[yolov5s] google drive

-[yolov5m] google drive

-[yolov5l] google drive

-[yolov5x] google drive

CFT weights

-[LLVIP] google drive

-[FLIR] google drive

Change the data cfg

some example in data/multispectral/

Change the model cfg

some example in models/transformer/

note!!! we used xxxx_transfomerx3_dataset.yaml in our paper.

5 of 7 10/28/24, 11:18
GitHub - DocF/multispectral-object-detection: Multisp... https://github.com/DocF/multispectral-object-detection

Train Test and Detect

test: python test.py

detect: python detect_twostream.py

Results

Dataset CFT mAP50 mAP75 mAP

FLIR 73.0 32.0 37.4

FLIR � 78.7 (Δ5.7) 35.5 (Δ3.5) 40.2 (Δ2.8)

LLVIP 95.8 71.4 62.3

LLVIP � 97.5 (Δ1.7) 72.9 (Δ1.5) 63.6 (Δ1.3)

VEDAI 79.7 47.7 46.8

VEDAI � 85.3 (Δ5.6) 65.9(Δ18.2) 56.0 (Δ9.2)

LLVIP
Log Average Miss Rate

Model Log Average Miss Rate

YOLOv3-RGB 37.70%

YOLOv3-IR 17.73%

YOLOv5-RGB 22.59%

YOLOv5-IR 10.66%

Baseline(Ours) 6.91%

CFT(Ours) 5.40%

Miss Rate - FPPI curve

6 of 7 10/28/24, 11:18
GitHub - DocF/multispectral-object-detection: Multisp... https://github.com/DocF/multispectral-object-detection

References

https://github.com/ultralytics/yolov5

Releases

No releases published

Packages

No packages published

Languages

Python 97.5% Shell 2.3% Dockerfile 0.2%

7 of 7 10/28/24, 11:18