SEED: A Simple and Effective 3D DETR in Point Clouds

Abstract

Recently, detection transformers (DETRs) have gradually taken a dominant position in 2D detection thanks to their elegant framework. However, DETR-based detectors for 3D point clouds are still difficult to achieve satisfactory performance. We argue that the main challenges are twofold: 1) How to obtain the appropriate object queries is challenging due to the high sparsity and uneven distribution of point clouds; 2) How to implement an effective query interaction by exploiting the rich geometric structure of point clouds is not fully explored. To this end, we propose a Simple and EffEctive 3D DETR method (SEED) for detecting 3D objects from point clouds, which involves a dual query selection (DQS) module and a deformable grid attention (DGA) module. More concretely, to obtain appropriate queries, DQS first ensures a high recall to retain a large number of queries by the predicted confidence scores and then further picks out high-quality queries according to the estimated quality scores. DGA uniformly divides each reference box into grids as the reference points and then utilizes the predicted offsets to achieve a flexible receptive field, allowing the network to focus on relevant regions and capture more informative features. Extensive ablation studies on DQS and DGA demonstrate its effectiveness. Furthermore, our SEED achieves state-of-the-art detection performance on both the large-scale Waymo and nuScenes datasets, illustrating the superiority of our proposed method. The code is available at https://github.com/happinesslz/SEED.

1 Introduction

DEtection TRansformer (DETR) [3] is the pioneering end-to-end transformer-based detector, which redefines object detection as a set prediction problem and eliminates hand-crafted anchors and non-maximum suppression (NMS) post-processing. These superior characteristics make the DETR paradigm [3, 19, 23, 56] become the mainstream method for 2D object detection tasks.

However, although many efforts have been made towards the DETR paradigm for 3D object detection [1, 29, 30, 7, 46, 61, 45] in point clouds, they have not demonstrated stunning performance similar to the 2D domain and still fall behind state-of-the-art 3D detectors [43, 11, 36, 9]. The main reason is the huge gap between 2D images and 3D points (i.e., dense and regular 2D images v.s. sparse and irregular 3D points clouds), which requires us to carry out special designs for two critical components (i.e., query selection and query interaction) in the DETR paradigm. For query selection, some methods [1, 7, 61] mainly select Top-N (e.g., N=200, 300 or 1000) features as queries from the score map. Although effective, these methods do not consider the quality of the selected queries for box localization. For query interaction, some works [1, 46] perform several attention operations to achieve sufficient feature interaction. However, these approaches do not sufficiently take advantage of geometric information of 3D objects from point clouds.

Figure 1: Comparison with DETR-based detectors [61, 7, 1] and other representative methods [8, 36] on Waymo validation dataset [39] in terms of detection performance and running speed. For a fair comparison, we evaluate the running speed of all approaches on an NVIDIA GeForce RTX 3090 with a batch size of 1. -S, -B and -L means the small, base, large versions of our SEED, respectively.

In this paper, to alleviate the above challenges, we propose a Simple and EffEctive 3D DETR method (SEED) for detecting 3D objects from point clouds. The first key design in our SEED is the proposed dual query selection (DQS) to pick out high-quality queries in a coarse-to-fine manner, which includes a foreground query selection and a quality query selection. This manner is different from existing methods by one-step query selection [1, 61]. More concretely, to ensure a high recall, we first retain a large number of foreground queries in the foreground query selection according to the estimated confidence scores from a mask predictor. Then, we employ a SEED decoder layer to allow these queries to effectively interact with Bird’s Eye View (BEV) features. The enhanced queries are fed into the stage of quality query selection to pick out high-quality queries.

The second core design in our SEED is the proposed deformable grid attention (DGA) to make the network focus on relevant regions and achieve more effective feature interaction. Specifically, to exploit the rich geometric information in point clouds, we first divide the referenced box estimated by a regression branch into uniform grids, whose corresponding features can be easily collected to describe the geometric structures of 3D objects. To alleviate the strong dependence on the high-accuracy reference box, we further use these sampling grids as reference points and apply the predicted offsets to obtain flexible receptive fields. This enables the network to focus on surrounding regions of interest, even for less precise reference boxes.

As shown in Figure 1, we compare our SEED with the existing DETR-based 3D detection methods and other representative methods [8, 36] on the Waymo validation dataset [39] in terms of performance and running speed. It can be clearly observed that our SEED-S (i.e., the small version) not only surpasses existing DETR-based approaches in detection performance but also maintains a superior running speed. In summary, our contributions are as follows:

• We introduce a novel dual query selection module, producing high-quality queries in a coarse-to-fine manner.

• We adopt an effective deformable grid attention module, which adaptively aggregates crucial regions and performs informative query interaction by properly leveraging the geometric information of point clouds.

• The proposed SEED achieves state-of-the-art performance for 3D object detection on both the large-scale Waymo [39] and nuScenes [2] datasets.

2 Related Work

2D Object Detection with DETR. DETR [3] is an end-to-end object detector that takes objects as queries and utilizes the transformer to interact queries with image features. Besides, DETR abandons many hand-crafted operations (e.g., Anchor, NMS) and utilizes Hungarian Matching to achieve the ground-truth assignment. The elegant architecture proposed by DETR brings a new insight into the research of object detection, and many works [62, 28, 12, 44, 19, 23, 56] improve DETR from different perspectives. Deformable DETR [62] introduces deformable attention into DETR and greatly improves the convergence speed of DETR. DN-DETR [19] proposes the denoising training strategy, which effectively reduces the learning difficulty of bipartite graph matching. DINO [56] utilizes contrastive learning in denoising training to achieve better performance.

LiDAR-based 3D Object Detection. 3D object detectors in point clouds can be categorized into point-based and voxel-based categories. For point-based, most methods [37, 52, 51, 31, 16, 57, 50, 4, 14, 24] directly utilize a PointNet-like backbone [32, 33] to extract point features, which can keep precise geometric structure information. However, these methods usually need to sample points to reduce computational costs, which may lose some important information in point clouds. For voxel-based, most methods [59, 47, 18, 9, 35, 13, 49, 26, 55, 48, 53, 25, 38] quantify point clouds into regular grids and utilize a 3D sparse convolution backbone to extract grid features (e.g. Voxel and Pillar) efficiently.

3D Object Detection with DETR. Due to the powerful feature representation of transformer, many works [1, 46, 61, 7, 45] have been explored to utilize DETR for 3D object detection in point clouds, especially for the design of two key components (query selection and query interaction) in DETR. Specifically, TransFusion [1] selects the local maximum feature in BEV features as queries based on the heatmap. CMT [46] adopts learnable queries initialized by 3D grids and utilizes global attention to interact queries with BEV features. ConQueR [61] proposes a query contrast mechanism to reduce false positives. FocalFormer3D [7] utilizes a multi-stage heatmap for better query selection. Besides, FocalFormer3D [7] adopts deformable attention for efficient query interaction. Although the above DETR-based methods have made some progress, they are still inferior to some advanced methods [43, 8] that do not belong to the DETR-based paradigm. In this paper, we propose a simple and effective 3D DETR named SEED, involving a novel dual query selection module for picking out high-quality queries and a deformable grid attention module to make effective query interaction by leveraging the rich geometric information of point clouds.

Figure 2: Overall architecture of SEED, which consists of a 3D backbone and a SEED detection head. Specifically, the proposed SEED detection head mainly includes a dual query selection (DQS) module and a transformer decoder. The DQS utilizes a coarse-to-fine query selection strategy to select high-quality queries. The transformer decoder, including six SEED decoder layers, takes these queries as inputs and then iteratively performs a self-attention operation for inter-query interaction and a proposed deformable grid attention (DGA) for feature interaction between query and BEV features, generating final detection results.

3 Method

Although many attempts have been made on DETR-based 3D object detection, there is still a certain performance gap with existing advanced LiDAR-based 3D detectors [8, 43]. We argue that the main challenges come from two aspects. On the one hand, selecting superior queries from the high sparsity and uneven distribution of point clouds is not trivial. On the other hand, exploring how to make use of the rich geometric structure information from point clouds to perform effective query interaction is still challenging.

To mitigate these issues, we propose a Simple and EffEctive 3D DETR method (SEED) for detecting 3D objects from point clouds. As shown in Figure 2, we present the overall pipeline of SEED. Specifically, we first feed point clouds into a classic voxel-based 3D backbone [47] to extract 3D voxel features and further convert them to BEV features. To retain their position information, we add position embedding to the BEV features. Then, the BEV features are flattened for subsequent query selection. As for query selection, we propose a novel dual query selection (DQS), which adopts a coarse-to-fine manner to obtain high-quality queries. Finally, the transformer decoder, including six SEED decoder layers, is adopted to achieve feature interaction between the high-quality queries and the flattened BEV features, producing final detection results. In particular, our SEED decoder layer leverages an effective deformable grid attention (DGA) for query interaction instead of the cross attention operation in classic DETR decoder [3]. In the following, we will introduce the details of the proposed DQS and DGA in SEED.

3.1 Dual Query Selection Module

Figure 3: Illustration of dual query selection (DQS). DQS adopts a coarse-to-fine manner, which consists of a foreground query selection and a quality query selection. $S_{c}$, $S_{l}$, and $B_{c}$ are the predicted classification score, localization score, and regression for proposal boxes through three feed-forward networks (FFN) branches, respectively.

A proper query selection has demonstrated its importance in DETR-based 2D object detectors [23, 56, 5] to ensure accurate object localization and accelerate model convergence. However, due to the huge difference in data format between 2D images and 3D point clouds, it is necessary to consider some characteristics of point clouds, such as high sparsity and uneven distribution, for the query selection. Toward this goal, we propose a novel dual query selection (DQS) module whose main purpose is to obtain high-quality queries in a coarse-to-fine manner. We present the detailed structure of DQS in Figure 3, which involves a foreground query selection and a quality query selection.

Foreground Query Selection. First, for the foreground query selection, we utilize a binary classification predictor to distinguish backgrounds and foregrounds on the BEV features. Simultaneously, we add BEV position embedding to BEV features and flatten them along the spatial dimension to generate all queries (also named flattened BEV features). For the convenience of description, we define the flattened BEV features as $\bm{F}_{bev}\in\mathbb{R}^{(H\times W)\times C}$, where $H$, $W$ and $C$ are the height, width, and channel dimension of BEV features, respectively. Then, we select these queries with proportion $r$ among the top confidence scores of BEV features $\bm{S}_{bev}$ from a mask predictor as the coarse queries, which can remain as many potential foreground queries as possible to ensure a high recall rate. Finally, the foreground query selection is formulated as:

$$\bm{Q}_{c}=\mathrm{Top}_{N_{c}}(\bm{F}_{bev},\bm{S}_{bev}),$$ (1)

where $N_{c}=H\times W\times r$ and $\mathrm{Top}_{N_{c}}(x,y)$ means select top ${N_{c}}$ queries from $x$ according to $y$, and $N_{c}$ is the number of coarse queries.

After getting coarse queries, we further feed them to a SEED decoder layer to achieve sufficient feature interaction between queries and flattened BEV features, producing the enhanced queries $\bm{Q}^{{}^{\prime}}_{c}$, which can be computed as:

$$\bm{Q}^{{}^{\prime}}_{c}=\mathrm{Decoder}(\bm{Q}_{c},\bm{F}_{bev}),$$ (2)

where $\mathrm{Decoder}$ is our SEED Decoder layer, which will be introduced in detail in the Section  3.2.

Quality Query Selection. We first feed the coarse queries $\bm{Q}^{{}^{\prime}}_{c}$ into three feed-forward networks (FFN) branches to produce the classification score $\bm{S}_{c}$, the localization scores $\bm{S}_{l}$ and the regression $\bm{B}_{c}$ for coarse proposal boxes, whose corresponding ground truths are assigned based on the proposed quality-aware Hungarian Matching (refer to Section 3.3). Here, the classification score is the probability of recognizing 3D object proposals, and the localization score is defined as the 3D IoU of proposal boxes and the ground truths. Considering that the localization score is mainly for foreground objects, we set a proper classification score threshold $\tau$ to distinguish the foreground objects. Therefore, the quality scores $\bm{S}_{q}$ by combining these two indicators can be formulated as:

$$\bm{S}_{q}^{i}=\begin{cases}(\bm{S}_{c}^{i})^{1-\beta}\cdot(\bm{S}_{l}^{i})^{\beta}\ \ ,&\mathrm{if}\ \bm{S}_{c}^{i}>\tau,\\ \bm{S}_{c}^{i}\ \ ,&\text{otherwise},\end{cases}$$ (3)

where $\beta\in(0,1)$ is a hyper-parameter and is applied to control the importance of classification score and localization score, and $i=0,1,...,N_{c}$. Then, we select the top $N_{f}$ fine proposal boxes $\bm{B}_{f}$ according to the quality scores $\bm{S}_{q}$ and concatenate them with the corresponding box quality scores $\bm{S}_{f}$. Next, we feed the concatenated features in a multi-layer perceptron (MLP) to generate the geometric-aware high-quality queries. These steps can be formulated as:

$$\bm{B}_{f}=\mathrm{Top}_{N_{f}}(\bm{B}_{c},\bm{S}_{q}),$$ (4)

$$\bm{Q}_{f}=\operatorname{MLP}(\operatorname{Concat}(\bm{B}_{f},\bm{S}_{f})).$$ (5)

Finally, the output queries $\bm{Q}_{f}$ of DQS will be input to the subsequent SEED decoder.

Figure 4: Illustration of deformable grid attention (DGA). DGA first uniformly divides each reference proposal into grids as the reference points and then utilizes the predicted offsets to achieve a flexible receptive field.

3.2 SEED Decoder Layer

The proposed SEED decoder layer is applied to further enhance query feature representation by a self-attention operation and a cross attention operation and then maps the enhanced queries to task-specified outputs by an FFN. Different from existing DETR-based methods [46, 7, 61], we adopt a new cross attention operation in our SEED decoder layer, namely deformable grid attention (DGA). Next, we will introduce the reasons and the details of DGA.

Why need DGA? An effective query interaction designed for point clouds is necessary to further explore the potential of the DETR paradigm in 3D detection. First of all, unlike 2D images, a nearby object may occupy most of the whole image, which even requires a global receptive field to detect the object well. However, a 3D object usually only occupies a small local area (also be mentioned in SST [10]), which is much smaller than the range of the entire point clouds. Thus, the local attention may be enough for query interaction in point clouds. Second, point clouds possess rich geometric structures, especially for regular vehicles. Therefore, it is important to rationally utilize the geometric information of 3D objects. Third, although an accurate 3D proposal box can describe the geometric information of 3D objects, it is sub-optimal to capture some irregular objects or some hard objects. This indicates that a flexible receptive field is desired. Towards this goal, we propose a deformable grid attention (DGA), which is a new local attention but adopts a flexible receptive field to effectively leverage the geometric information of 3D objects for query interaction.

Details of DGA. As shown in Figure 4, we present the detailed structure of DGA. Specifically, we first regard the estimated proposal boxes $\bm{B}_{f}$ from DQS as the reference boxes and uniformly divide each reference box into $k\times k$ grids $\bm{g}_{k}$ (i.e., yellow points in Figure 4). Then, we feed the corresponding selected queries $\bm{Q}_{f}$ (i.e., red points in Figure 4) from DQS into a linear function, producing the predicted offsets $\Delta\bm{g}$. Next, we add the offsets to the grids $\bm{g}$ to generate the final sampling positions, which can capture the geometric information of 3D objects in a flexible receptive field. Meanwhile, the attention weight $\bm{A}$ is predicted by feeding $\bm{Q}_{f}$ into a linear function and a softmax function. Finally, the sampled features are multiplied with $\bm{A}$ to obtain the enhanced queries. We can formulate the above process of DGA as follows:

$$\mathrm{DGA}(\bm{g},\bm{F}_{bev})=\sum\limits_{j=1}^{K}\bm{A}_{j}\cdot\phi(\bm{F}_{bev}(\bm{g}_{j}+\Delta\bm{g}_{j})),$$ (6)

where $K=k\times k$ and $\phi$ is a linear function for transforming sampled features to the attention space. $\bm{F}_{bev}(*)$ denotes sampling the corresponding features of the positions $*$ on the BEV features $\bm{F}_{bev}$ by a bilinear interpolation operation. Besides, we provide more comparisons with different attention operations [62, 30] in our supplemental materials.

3.3 Quality-aware Hungarian Matching

Different from the traditional Hungarian Matching [3], we introduce a quality-aware Hungarian Matching (QHM) to effectively assign the ground truth. Specifically, QHM adopts the quality scores $\bm{S}_{f}$ instead of the classic classification scores in computing classification cost. Thus, our classification cost $\bm{\mathcal{C}}_{cls}$ can be formulated as:

$$\bm{\mathcal{C}}_{pos}=-(1-\alpha)\cdot(\bm{S}_{f})^{\gamma}\cdot\log(1-\bm{S}_{f}),$$ (7)

$$\bm{\mathcal{C}}_{neg}=-\alpha\cdot(1-\bm{S}_{f})^{\gamma}\cdot\log{\bm{S}_{f}},$$ (8)

$$\bm{\mathcal{C}}_{cls}=\bm{\mathcal{C}}_{pos}-\bm{\mathcal{C}}_{neg},$$ (9)

where $\alpha$, $\gamma$ are hyper-parameters. Finally, the total matching cost in Hungarian Matching can be computed as:

$$\bm{\mathcal{C}}_{\mathrm{match}}=\lambda_{cls}\cdot\bm{\mathcal{C}}_{cls}+\lambda_{reg}\cdot\bm{\mathcal{C}}_{reg}+\lambda_{giou}\cdot\bm{\mathcal{C}}_{giou},$$ (10)

where $\bm{\mathcal{C}}_{reg}$ and $\bm{\mathcal{C}}_{giou}$ denote the regression cost and the GIoU cost, which have the same formulation in the traditional Hungarian Matching [3]. $\lambda_{cls}$, $\lambda_{reg}$, and $\lambda_{giou}$ are the balanced weights.

4 Experiments

4.1 Datasets and Evaluation Metrics

Waymo Open Dataset. Waymo Open Dataset (WOD) [39] is a well-known and challenging large-scale outdoor 3D object detection benchmark. WOD consists of 1150 scenes (more than 200K frames), which are divided into three parts: 798 for training, 202 for validation, and 150 for testing. Each scene provides point clouds acquired by 64-beam LiDAR and covers a perception range with a size of 150m × 150m. Besides, the evaluation of WOD is divided into two levels (LEVEL 1 and LEVEL 2) according to the number of points of the object. For all experimental results, we follow the standard protocol as the evaluation metric, which adopts 3D mean Average Precision (mAP) [22] and its weighted variant by heading accuracy (mAPH) for three categories: Vehicle, Pedestrian and Cyclist.

nuScenes Dataset. nuScenes [2] is another widely used autonomous driving dataset for LiDAR-based 3D object detection. The nuScenes dataset consists of 1000 scenes, which are divided into three parts: 750 for training, 150 for validation, and 150 for testing. Each scene is roughly 20s long, annotated at 2Hz, and provides point clouds collected by 32-beam LiDAR. Besides, the evaluation of the nuScnes dataset adopts the Mean Average Precision (mAP) and nuScenes detection score (NDS) to evaluate the performance of 3D detectors for 10 foreground classes.

4.2 Implementation Details

Network Architecture. In SEED, we provide three versions, namely small (i.e., SEED-S), base (i.e., SEED-B), and large (i.e., SEED-L). In this paper, SEED-S adopts the same 3D backbone as CenterPoint [54] for a fair comparison with most existing methods. For SEED-B, we follow VoxelNext [8], which introduces multi-scale voxel feature extraction and doubles the channel of the 3D backbone based on SEED-S to improve the feature representation. For SEED-L, we utilize a smaller voxel size (0.08, 0.08, 0.15) instead of (0.1, 0.1, 0.15) to enlarge the BEV resolution based on SEED-B, which further boosts detection performance specifically for small objects such as Pedestrian. In DQS, we set $r=0.3$ in foreground query selection to ensure a high recall and set $N_{f}=1000,\tau=0.2$ in quality query selection for effective query interaction in the subsequent transformer decoder. In DGA, we set $k=5$ to divide grids as default. In the transformer decoder, we adopt six SEED decoder layers to iteratively perform query interaction and then generate the final 3D detection results.

Training. The final loss includes the DETR-head loss (similar to ConQueR [61]) and our DQS loss. In DQS loss, we supervise classification score with binary cross-entropy loss, localization score with IoU loss, and regression with Smooth-L1 loss, respectively. On WOD, we adopt the same point cloud range and data augmentations as CenterPoint [54]. Our model is optimized by AdamW optimizer [27] with the initial learning rate, weight decay, and momentum factor set to 0.001, 0.01, and 0.9, respectively. We train our model with a batch size of 24 on 8 NVIDIA Tesla V100 GPUs. We run 24 epochs for the 20% training set and only 12 epochs for the 100% training set. Besides, we utilize the fade strategy [42] to avoid over-fitting in the last epoch and the query contrast strategy [61] to achieve better performance. For quality-aware Hungarian Matching, we set $\alpha$ to 0.25 and $\gamma$ to 2.0. In quality query selection, the hyper-parameter $\beta$ is set to 0.68, 0.71, and 0.65 for Vehicle, Pedestrian, and Cyclist, respectively. This is a common setting in some non-DETR methods (e.g., AFDetv2 [17], PillarNet [34]) for IoU-rectification. For matching cost, we set $\lambda_{cls}$, $\lambda_{reg}$, and $\lambda_{giou}$ to 1, 2, and 4. For nuScenes, our SEED simply follows the settings of [1, 7], including the range of point clouds, the voxel size, data augmentations, and the training strategy.

Table 1: Performances on the Waymo Open Dataset validation split (train with 100% training data). ${\ddagger}$ denotes the two-stage method. ‘SEED-S’, ‘SEED-B’, and ‘SEED-L’ mean the small, base, and large versions of SEED, respectively. Bold denotes the best performance in DETR-based methods. All results are presented with single-frame input, no test-time augmentation, and no model ensembling.

Methods	Present at	DETR	Vehicle 3D AP/APH		Pedestrian 3D AP/APH		Cyclist 3D AP/APH		mAP/mAPH L1
			L1	L2	L1	L2	L1	L2	L2
SECOND [47]	Sensors 18	✕	72.3/71.7	63.9/63.3	68.7/58.2	60.7/51.3	60.6/59.3	58.3/57.0	61.0/57.2
PointPillars [18]	CVPR 19		72.1/71.5	63.6/63.1	70.6/56.7	62.8/50.3	64.4/62.3	61.9/59.9	62.8/57.8
CenterPoint [54]	CVPR 21		74.2/73.6	66.2/65.7	76.6/70.5	68.8/63.2	72.3/71.1	69.7/68.5	68.2/65.8
PV-RCNN${\ddagger}$ [35]	CVPR 20		78.0/77.5	69.4/69.0	79.2/73.0	70.4/64.7	71.5/70.3	69.0/67.8	69.6/67.2
SST_TS${\ddagger}$ [10]	CVPR 22		76.2/75.8	68.0/67.6	81.4/74.0	72.8/65.9	–/–	–/–	–/–
AFDetV2 [17]	AAAI 22		77.6/77.1	69.7/69.2	80.2/74.6	72.2/67.0	73.7/72.7	71.0/70.1	71.0/68.8
SWFormer [40]	ECCV 22		77.8/77.3	69.2/68.8	80.9/72.7	72.5/64.9	–/–	–/–	–/–
PillarNet-34 [34]	ECCV 22		79.1/78.6	70.9/70.5	80.6/74.0	72.3/66.2	72.3/71.2	69.7/68.7	77.3/74.6
CenterFormer[60]	ECCV 22		75.0/74.4	69.9/69.4	78.6/73.0	73.6/68.3	72.3/71.3	69.8/68.8	71.1/68.9
PV-RCNN++${\ddagger}$ [36]	IJCV 22		79.3/78.8	70.6/70.2	81.3/76.3	73.2/68.0	73.7/72.7	71.2/70.2	71.7/69.5
FSD${\ddagger}$ [11]	NeurIPS 22		79.2/78.8	70.5/70.1	82.6/77.3	73.9/69.1	77.1/76.0	74.4/73.3	72.9/70.8
OcTr [58]	CVPR 23		78.1/77.6	69.8/69.3	80.8/74.4	72.5/66.5	72.6/71.5	69.9/68.9	70.7/68.2
PillarNeXt [20]	CVPR 23		78.4/77.9	70.3/69.8	82.5/77.1	74.9/69.8	73.2/72.2	70.6/69.6	71.9/69.7
VoxelNext [8]	CVPR 23		78.2/77.7	69.9/69.4	81.5/76.3	73.5/68.6	76.1/74.9	73.3/72.2	72.2/70.1
DSVT-Pillar [43]	CVPR 23		79.3/78.8	70.9/70.5	82.8/77.0	75.2/69.8	76.4/75.4	73.6/72.7	73.2/71.0
DSVT-Voxel [43]	CVPR 23		79.7/79.3	71.4/71.0	83.7/78.9	76.1/71.5	77.5/76.5	74.6/73.7	74.0/72.1
BoxeR-3D [30]	CVPR 22	✓	70.4/70.0	63.9/63.7	64.7/53.5	61.5/53.7	50.2/48.9	–/–	–/–
TransFusion [1]	CVPR 22		–/–	–/65.1	–/–	–/63.7	–/–	–/65.9	–/64.9
ConQueR [61]	CVPR 23		76.1/75.6	68.7/68.2	79.0/72.3	70.9/64.7	73.9/72.5	71.4/70.1	70.3/67.7
FocalFormer3D [7]	ICCV 23		–/–	68.1/67.6	-/-	72.7/66.8	–/–	73.7/72.6	71.5/69.0
SEED-S (Ours)	–		78.2/77.7	70.2/69.7	81.3/75.8	73.3/68.1	78.4/77.2	75.7/74.5	73.1/70.8
SEED-B (Ours)	–		79.7/79.2	71.8/71.4	83.1/78.3	75.5/70.8	80.0/78.8	77.3/76.1	74.9/72.8
SEED-L (Ours)	–		79.8/79.3	71.9/71.5	83.6/79.1	76.2/71.8	81.2/80.0	78.4/77.3	75.5/73.5

Table 2: Effectiveness of our SEED with multiple frames as inputs on the Waymo Open Dataset validation and test split.

Methods	Frames	mAP/mAPH (L1)	mAP/mAPH (L2)
CenterPoint [54]	4	76.4/74.9	70.8/69.4
CenterFormer [60]	4	78.5/77.0	74.7/73.2
MPPNet [6]	4	81.1/79.9	75.4/74.2
MSF [15]	4	81.1/80.2	76.0/74.6
PillarNeXt [34]	3	81.5/80.0	75.9/74.5
DSVT-Voxel [43]	3	82.1/80.8	76.3/75.0
SEED-S (Ours)	3	81.6/80.1	75.8/74.3
SEED-B (Ours)	3	82.9/81.4	77.2/75.8
SEED-L (Ours)	3	83.1/81.6	77.5/76.1

(a)

Methods	Frames	mAP/mAPH (L1)	mAP/mAPH (L2)
PV-RCNN++ [36]	1	78.0/75.7	72.4/70.2
AFDetV2 [17]	1	77.6/75.2	72.2/70.3
PillarNet [34]	1	77.5/74.7	72.2/69.6
FSD [11]	1	80.4/78.2	74.4/72.4
ConQueR [61]	1	–/–	–/72.0
SEED-L (Ours)	1	81.7/79.7	76.5/74.5
CenterPoint++ [54]	3	79.4/77.9	74.2/72.8
PillarNeXt [20]	3	80.5/79.0	75.5/74.1
SEED-L (Ours)	3	83.5/82.1	78.7/77.3

(b)

4.3 Main Results

Results on WOD. We present the comparison with existing DETR-based methods (bottom) and other representative methods (top) on the WOD in Table 1. Here, we provide three versions of SEED, including the small (SEED-S), base (SEED-B) and large (SEED-L). Compared with DETR-based 3D detectors, our SEED-S outperforms the advanced ConQueR [61] and FocalFormer3D [7] with 3.1 and 1.8 mAPH/L2, respectively. Moreover, SEED-S possesses a satisfactory running speed with about 13.5 FPS on an NVIDIA GeForce RTX 3090 (see Figure 1). These benefits on both the detection performance and the running speed effectively illustrate the superiority of SEED. Besides, we compare our SEED with other advanced methods that do not belong to the DETR paradigm. SEED-B exceeds the representative two-stage methods PV-RCNN++ [36] with 3.3 mAPH/L2. Considering that the high-resolution feature map is necessary to detect small 3D objects, SEED-L further boosts the detection performance of SEED-B) with 1.0 (71.8 vs. 70.8) APH/L2 on Pedestrian. It is noteworthy that SEED-L even outperforms the previous state-of-the-art (SOTA) method DSVT-Voxel [43] with 1.4 (73.5 vs. 72.1) mAPH/L2, leading to a new SOTA. Note that the BEV resolution of SEED-L is still much smaller than DSVT-Voxel [43] since DSVT uses one-stride 3D backbone like SST [10] to improve the detection performance. Furthermore, DSVT [43] focuses on enhancing the representation ability of the 3D backbone, which is orthogonal to our SEED.

In Table 4(a), we also provide the results of our SEED with three frames as inputs on the WOD validation split. We observe that SEED-L produces a leading performance with 76.1 mAPH/L2, even surpassing the temporal 3D object detection method MPPNet [6] with an obvious margin. To further verify the effectiveness of our SEED, we evaluate the performance of our SEED-L with one frame and three frames as inputs on the WOD test benchmark, as shown in Table 4(b). SEED-L with single frame achieves 74.5 mAPH/L2, which exceeds DETR-based method ConQueR [61] with 2.5 (74.5 vs. 72.0) mAPH/L2. For three frames, SEED-L outperforms the representative method PillarNeXt [20] with 3.2 (77.3 vs. 74.1) mAPH/L2. All the experimental results clearly demonstrate the superiority of our SEED.

Results on nuScenes. We also evaluate our SEED on the validation split of nuScenes dataset [2] to further verify the effectiveness of our SEED. As shown in Table 3, SEED achieves 71.2 NDS and 66.6 mAP, which exceeds the popular DETR-based detector TransFusion-L [1] with 1.1 NDS and 1.5 mAP under the same 3D backbone. This demonstrates the generalization of our method.

Table 3: Comparison with state-of-the-art methods on the nuScenes validation split. All results are presented without any test-time augmentation or model ensembling. Here, our SEED adopts the same 3D backbone with CenterPoint [54] for a fair comparison. * denotes the reproduced result from official code.

Method	Present at	mATE	mASE	mAOE	mAVE	mAAE	NDS	mAP
PointPillar [18]	CVPR 19	0.424	0.284	0.529	0.377	0.194	49.1	34.3
CenterPoint [54]	CVPR 21	0.291	0.252	0.324	0.284	0.189	64.9	56.6
TransFusion-L [1]	CVPR 22	–	–	–	–	–	70.1	65.1
PillarNet [34]	ECCV 22	0.277	0.252	0.289	0.247	0.191	67.4	59.8
UVTR-L [21]	NeurIPS 22	0.334	0.257	0.300	0.204	0.182	67.7	60.9
VoxelNeXt* [8]	CVPR 23	0.301	0.252	0.406	0.217	0.186	66.7	60.5
Uni3DETR [45]	NeurIPS 23	0.288	0.249	0.303	0.216	0.181	68.5	61.7
SEED (Ours)	–	0.279	0.257	0.284	0.208	0.187	71.2	66.6

4.4 Ablation Study

In this section, we conduct extensive ablation studies to investigate the effectiveness of SEED on the Waymo validation set with 20% training data by default if not specified. And we adopt SEED-S as our default SEED model in the following ablation studies. For more ablation studies about our SEED, please refer to our supplemental materials.

Effectiveness of the Proposed Components. As shown in Table 4, we conduct ablation studies on our proposed two components in SEED. First, we set the baseline by replacing our DQS with heatmap-based query selection [1] and replacing our DGA with box attention [30] in our SEED. Compared with the baseline, the proposed DQS module brings 2.8 mAPH/L2 (67.4 vs. 64.6) performance improvement in average for Vehicle, Pedestrian, and Cyclist on Waymo dataset, which demonstrates the superiority of DQS for selecting out high-quality queries. Besides, the proposed DGA boosts the performance of the baseline with 1.8 mAPH/L2, demonstrating the effectiveness of DGA in achieving feature interaction. Finally, thanks to the benefits of the proposed DQS and DGA, SEED has an obvious gain of 3.6 mAPH/L2 over the baseline model.

Table 4: Ablation study for each component in SEED. We use mAP/mAPH (L2) to evaluate the overall detection performance.

DQS	DGA	3D AP/APH (L2)			mAP/mAPH (L2)
		Vehicle	Pedestrian	Cyclist
–	–	65.4/64.9	68.8/63.3	66.7/65.5	67.0/64.6
$\checkmark$	–	68.0/67.5	70.9/65.2	70.7/69.5	69.9/67.4
–	$\checkmark$	65.7/65.2	69.9/64.4	71.0/69.6	68.8/66.4
$\checkmark$	$\checkmark$	68.5/68.1	72.1/66.5	71.2/70.0	70.6/68.2

Superiority of the DQS. To further demonstrate the superiority of the proposed DQS module, we provide three representative query selection strategies for comparison with DQS in Table 5, namely Learnable, Heatmap-based and Top-N manners. Specifically, Learnable means that we obtain queries by adopting a learnable manner like CMT [46]. Heatmap-based (e.g., TransFusion-L [1]) is a classic query selection strategy in LiDAR-based 3D object detection, which collects the local maximum elements in BEV features as queries. The advanced query selection manner Top-N (e.g., ConQueR [61]) uses a class-agnostic feed-forward network (FFN) head to obtain Top-N scored box proposals, which are selected as object queries. As shown in Table 5, Heatmap-based produces the worst performance among these methods. We think the main reason is that the queries (as Query) of Heatmap-based method is obtained directly from BEV features (as Key, Value), rather than learnable queries or geometric-aware queries. This often leads to sub-optimal feature interaction, which makes it difficult for the subsequent decoder to stack more layers due to the potential risk of over-fitting. However, our DQS respectively outperforms the Learnable and Top-N manners with 1.6 and 1.4 mAPH/L2, which illustrates the superiority of dual query selection for picking out high-quality geometric-aware queries.

Table 5: Ablation study for DQS in SEED. We adopt mAP/mAPH (L2) to evaluate the detection performance.

Methods	3D AP/APH (L2)			mAP/mAPH (L2)
	Vehicle	Pedestrian	Cyclist
Learnable [46]	66.1/65.6	70.1/64.5	71.0/69.8	69.1/66.6
Heatmap-based [1]	64.3/63.8	69.7/64.1	68.4/67.1	67.5/65.0
Top-N [61]	66.1/65.7	70.6/65.3	70.7/69.5	69.1/66.8
DQS (Ours)	68.5/68.1	72.1/66.5	71.2/70.0	70.6/68.2

Table 6: Ablation study for DGA in SEED. We adopt mAP/mAPH (L2) to evaluate the detection performance.

Methods	3D AP/APH (L2)			mAP/mAPH (L2)
	Vehicle	Pedestrian	Cyclist
Global Attention [41]	– (OOM)	– (OOM)	– (OOM)	– (OOM)
Deformable Attention [62]	67.5/67.0	71.3/65.9	70.8/69.7	69.9/67.5
Box Attention [30]	67.9/67.4	71.1/65.5	70.9/69.7	70.0/67.5
DGA (Ours)	68.5/68.1	72.1/66.5	71.2/70.0	70.6/68.2

Effectiveness of the DGA. To illustrate the effectiveness of the proposed DGA, we compare it with three representative query interaction operations, including global attention [41], deformable attention [62] and box attention (or named non-deformable grid attention)[30], whose results are summarized in Table 6. Since most LiDAR-based 3D detection methods [54, 1] have larger feature maps (e.g., $180\times 180$), performing global attention operation on mainstream GPUs (e.g., NVIDIA Tesla V100 or NVIDIA GeForce RTX 3090) is unable to bear such a large computational cost, causing the GPU to run out of memory (OOM). Therefore, the local attention operation, including the deformable attention and the box attention, is still reasonable to promise performance and efficiency for query interaction. For deformable attention operation, it produces 67.0 APH/L2 on Vehicle, which is inferior to the box attention manner with 67.4 APH/L2. This indicates that effectively leveraging the box geometric information is important in feature interaction. However, the box attention operation depends on the precision of box regression, and its receptive field is not as flexible as deformable attention. Therefore, on hard objects such as Pedestrian, the deformable attention outperforms the detection performance of the box attention. In contrast, our DGA has the advantages of both the flexible receptive field of deformable attention and the rich geometric information of box attention. Not surprisingly, our DGA surpasses these local attention methods, demonstrating its effectiveness.

Table 7: Ablation study for quality-aware Hungarian Matching (QHM) in SEED. THM is short for traditional Hungarian Matching. Besides, we adopt mAP/mAPH (L2) to evaluate the detection performance.

Methods	3D AP/APH (L2)			mAP/mAPH (L2)
	Vehicle	Pedestrian	Cyclist
THM[3]	67.3/66.8	71.7/66.4	70.9/69.7	70.0/67.6
QHM (Ours)	68.5/68.1	72.1/66.5	71.2/70.0	70.6/68.2

Influence of Quality-aware Hungarian Matching. Hungarian Matching is an indispensable component in the existing DETR-based approaches. In SEED, we adopt a new matching manner, named quality-aware Hungarian Matching (QHM). To verify the influence of QHM on 3D detection performance, we compare it with the traditional Hungarian Matching [3] (THM), whose result is presented in Table 7. It can be observed that QHM produces a consistent performance improvement over THM in terms of mAPH/L2, which benefits from taking the quality scores of 3D objects into account when computing classification cost in Hungarian Matching. Moreover, we carefully find there is a better gain on Vehicle than Pedestrian and Cyclist. The main reason is that the localization score of Vehicle is more easily estimated than that of Pedestrian and Cyclist due to some factors such as a large size and a rigid object.

Figure 5: Visualization of SEED without DQS (the first row) and with DQS (the second row). We highlight the challenging queries with red circles. The colormap indicates the values of the confidence scores for selected queries on the BEV map. Green boxes are the ground truth boxes.

4.5 Visualization Analysis

As shown in Figure 5, we present the visualization of our SEED with and without DQS (i.e., directly select Top $N_{f}$ queries in one step). Specifically, we visualize the positions of final selected queries from query selection on the BEV map, whose corresponding colormap denotes the confidence score of the selected queries. It can be observed that after utilizing DQS, some hard queries are successfully captured, which indicates that DQS can enhance the confidence score of some potential hard objects. This phenomenon further demonstrates the superiority of our DQS. Besides, we provide more visualization of our SEED in the supplemental materials.

4.6 Limitation

Our method mainly improves the detection head based on the DETR paradigm for 3D object detection. Therefore, the advanced 3D detectors that focus on enhancing the representation ability of 3D backbone are orthogonal to SEED. In the future, we plan to apply our SEED to more powerful 3D backbones on more datasets to further explore the scalability of our method. Besides, we observe that SEED may fail to detect some distant and small 3D objects, but they are clearly visible in 2D camera images. Therefore, exploiting the complementarity of multiple modalities (i.e., 3D point clouds, and 2D camera images) to detect these challenging objects is also our next step.

5 Conclusion

In this paper, we have presented a simple and effective 3D DETR framework named SEED to detect 3D objects from point clouds. Specifically, SEED involves two key components: a dual query selection (DQS) module to retain high-quality queries in a coarse-to-fine manner and a deformable grid attention (DGA) module to capture informative features by performing sufficient query interaction. Extensive ablation studies have demonstrated the effectiveness of the proposed DQS and DGA. Thanks to the superiority of the proposed DQS and DGA, our SEED has achieved state-of-the-art 3D detection performance on the large-scale Waymo and nuScenes dataset. Finally, we hope SEED could become a new strong baseline for the community of DETR-based 3D object detection.

Appendix 0.A Appendix

The supplementary materials are organized as follows. First, in section 0.A.1, we present the extra experiments of illustrating the capability of our SEED, applying different backbones, varying the score threshold $\tau$ for quality query selection in DQS and the distinct grids in DGA on the Waymo validation set [39] with 20% training data, respectively. Besides, we explore the impact of different numbers of SEED decoder layers on detection performance. In section 0.A.2, we present the comparisons of several variant attention operations for query interaction. In section 0.A.3, we discuss the differences of our proposed DQS and DGA with the existing related methods. Finally, we provide the analysis of visualization, including the learned attention map of DGA and the 3D detection results under different settings in section 0.A.4.

0.A.1 Extra Experiments

Table 8: Effectiveness of our SEED. For a fair comparison, we adopt 100% Waymo training data for all models. The results are evaluated by the metric of mAP/mAPH (L2).

Methods	Detection Head	mAP/mAPH (L2)	FLOPs (G)	Params (M)	Latency (ms)
SECOND [47]	Anchor-based	61.0/57.2	91.2	5.3	33.3
CenterPoint [54]	Center-based	68.2/65.8	141.2	7.8	44.0
PV-RCNN++ [36]	RoI-based	71.7/69.5	166.6	16.1	149.0
VoxelNeXt [8]	Center-based	72.2/70.1	624.9	29.3	124.7
TransFusion [1]	DETR-based	–/64.9	96.8	7.9	70.5
ConQueR [61]	DETR-based	70.3/67.7	167.3	15.1	99.1
FocalFormer3D [7]	DETR-based	71.5/69.0	144.9	19.4	97.2
SEED-S (Ours)	DETR-based	73.1/70.8	168.7	12.8	74.2
SEED-L (Ours)	DETR-based	75.5/73.5	648.1	33.1	163.8

Capability of our SEED. To verify the capability of our SEED, we adopt the small version SEED-S with the same 3D backbone as CenterPoint [54] for a fair comparison with existing representative 3D object detection methods, including anchor-based [47], center-based [54], RoI-based [36] and DETR-based [1, 61, 7] detectors. We conduct the comparisons of these methods in terms of performance, FLOPs, parameters, and latency, shown in Table 8. Note that the main difference between these methods is the design of the detection head. Moreover, we evaluate the running speed of all approaches on one NVIDIA GeForce RTX 3090 with a batch size of 1 according to their corresponding official open-source code for a fair comparison. Compared with SECOND [47] and CenterPoint [54], our SEED-S has a slower running speed, but our performance greatly exceeds them with 13.6 and 5.0 mAPH/L2, respectively. Furthermore, benefiting from the well-designed DQS module for selecting high-quality queries and the superior DGA operation for effective feature interaction, the detection performance of our SEED-S even outperforms PV-RCNN++ [36] of 1.3 mAPH/L2 with $2\times$ faster running speed. However, existing DETR-based methods still fall behind PV-RCNN++ in terms of detection performance. The above experimental results effectively illustrate the powerful capability of our SEED.

Table 9: Effectiveness of our SEED on different backbones on the Waymo validation set [39] with 20% training data. We use mAP/mAPH (L2) for evaluating the detection performance. $*$ means our reproduced performance from the official code.

Methods	3D AP/APH (L2)			mAP/mAPH (L2)
	Vehicle	Pedestrian	Cyclist
CenterPoint-Pillar [54]	62.2/61.7	65.1/55.0	63.0/61.5	63.4/59.4
+SEED Detection Head	67.0/66.5	71.3/62.0	65.8/64.5	68.0/64.3
CenterPoint [54]	63.2/62.7	64.3/58.2	66.1/64.9	64.5/61.9
+SEED Detection Head	68.5/68.1	72.1/66.5	71.2/70.0	70.6/68.2
DSVT-Pillar$*$ [43]	69.7/69.2	74.9/68.0	70.7/69.6	71.8/68.9
+SEED Detection Head	71.7/71.3	75.4/68.7	73.0/71.8	73.4/70.6
HEDNet$*$ [55]	70.8/70.3	75.0/70.3	73.6/72.6	73.1/71.1
+SEED Detection Head	72.4/72.0	76.3/71.3	74.9/73.8	74.5/72.4

Effectiveness of our SEED with Different Backbones. Note that our SEED focuses on the design of detection head based on the DETR paradigm. Therefore, to verify the effectiveness of our SEED, we decorate our SEED detection head with different backbones, including CenterPoint-Pillar (pillar-based) [54], CenterPoint (voxel-based) [54], DSVT-Pillar [43] and HEDNet [55]. In Table 9, we present the corresponding detection results on the Waymo validation set [39] with 20% training data. We clearly observe that our approach yields consistent performance improvement under different backbones, proving the generality of our SEED detection head.

Effect of $\tau$ for Quality Query Selection. To explore the effect of the classification score threshold $\tau$ in formula (4) of the main paper for quality query selection, we set different score thresholds of $\tau=0.0$, $\tau=0.2$, and $\tau=0.3$, whose results are summarized in Table  10. When the score threshold is set as 0.0, we find there is a drastic drop in detection performance. Since the predicted object score is close to 0.0, it is more likely to be considered a background object. At this time, the estimated localization scores that are mainly for foreground objects rather than background objects are unreasonable, leading to selecting out poor queries in the stage of quality query selection. Therefore, setting a proper score threshold (e.g., 0.2 or 0.3) to eliminate the negative impact of background objects for quality query selection in DQS is necessary.

Table 10: The effect of different classification score thresholds for quality query selection in dual query selection (DQS). We use mAP/mAPH (L2) for evaluating the detection performance.

$\tau$	3D AP/APH (L2)			mAP/mAPH (L2)
	Vehicle	Pedestrian	Cyclist
0.0	66.6/66.2	70.1/64.5	69.1/67.8	68.6/66.2
0.2	68.5/68.1	72.1/66.5	71.2/70.0	70.6/68.2
0.3	68.7/68.2	71.9/66.4	71.0/69.8	70.5/68.1

Table 11: Effectiveness of our SEED. The results are evaluated by the metric of mAP/mAPH (L1 and L2). We evaluate the latency of our SEED for different gird sizes on one NVIDIA GeForce RTX 3090 with a batch size of 1.

Grids	mAP/mAPH (L1)	mAP/mAPH (L2)	Latency (ms)
$3\times 3$	76.7/74.1	70.3/67.8	73.1
$5\times 5$	77.0/74.4	70.6/68.2	74.2
$7\times 7$	77.1/74.5	70.7/68.3	77.8

Table 12: The effect of the number of SEED decoder layers in transformer decoder for 3D detection performance. We use mAP/mAPH (L2) to evaluate the detection performance.

Layers	3D AP/APH (L2)			mAP/mAPH (L2)
	Vehicle	Pedestrian	Cyclist
1	66.8/66.2	69.4/62.0	69.2/67.8	68.5/65.3
3	68.4/68.0	71.5/65.6	70.9/69.7	70.3/67.8
6	68.5/68.1	72.1/66.5	71.2/70.0	70.6/68.2

Ablation for Distinct Grids. As shown in Table 11, we conduct experiments of varying grid sizes in DGA to investigate their impact on the detection performance and latency. With increasing the grid sizes ($3\times 3\rightarrow 5\times 5$), the detection performance of our SEED can be consistently improved in terms of mAP/mAPH (L2). However, the corresponding computational costs are also increasing due to more sampled features being performed for query interaction, leading to more latency. Therefore, in our paper, we choose a proper grid size of $5\times 5$ as default to trade off the detection performance and latency.

Number of SEED Decoder Layers. To analyze the effect of different numbers of SEED decoder layers on detection performance, we provide the experimental results in Table 12. When only one SEED decoder layer is applied in the transformer decoder to extract contextual features of point clouds, a relatively poor detection performance with 65.3 mAPH/L2 is obtained. In contrast, stacking three SEED decoder layers brings an obvious performance gain with 2.5 mAPH/L2 thanks to their more powerful feature extraction capabilities. Intuitively, stacking more SEED decoder layers is beneficial. Therefore, in this paper, we adopt the commonly used six SEED decoder layers in the transformer decoder, which produces a better result with 68.2 mAPH/L2 than the settings of using fewer decoder layers (i.e., 65.3 mAPH/L2 for one decoder layer or 67.8 mAPH/L2 for three decoder layers).

Table 13: The comparison of different query selection in terms of latency.

Query Selection Methods	mAP / mAPH (L2)	Latency (ms)
TransFusion [1] (Heatmap-based)	67.5 / 65.0	2.5
ConQueR [61] (Top-N)	69.1 / 66.8	7.3
SEED (DQS)	70.6 / 68.2	10.0

Latency of Different Query Selection. In Table 13, we provide the latency of different query selection methods. We can observe that our DQS with high performance does not bring significant latency compared with the Top-N method.

0.A.2 Different Attention Operations

To clearly illustrate the difference between our proposed deformable grid attention (DGA) and existing representative attention operations (i,e, global attention [41], deformable attention [62] and box attention [30]), we present the simple schematic diagrams of these methods as shown in Figure 6. For the global attention in Figure 6 (a), each query implements feature interaction with all features (as key and value). This operation usually brings unacceptable computational costs, especially for using high-resolution feature maps as keys or values. Therefore, the local attention operations including the deformable attention, the box attention, and our DGA in Figure 6 (b) (c) (d) are more proper to perform query interaction than the global attention in point clouds. Specifically, deformable attention is good at capturing the crucial regions of objects in a flexible receptive filed manner, but the learned offsets without geometric prior information as reference are difficult to predict accurately. The box attention operation can make use of geometric information of some regular objects (e.g., Vehicle), but it requires a precise box regression, and its receptive field is not as flexible as the deformable attention. In contrast, our deformable grid attention has the advantages of both the flexible receptive field of deformable attention and the rich geometric information of the box attention, which can enable the network to focus on relevant regions and capture more informative features even for objects with diverse shapes.

Figure 6: Comparison of deformable gird attention (DGA) with other attention operations. The orange points represent the sampling features, the yellow points represent the reference points, and the green arrows represent the predicted offsets. Note that global attention adopts a global manner for query interaction, that is, treating all features as sampling features.

0.A.3 Discussion

Table 14: Comparison of our SEED and FocalFormer3D. $*$ indicates the deformable attention in FocalFormer3D [7]

DQS	multi-stage	mAP/mAPH (L2)
–	–	67.5/65.0
–	✓	68.2/65.5
✓	–	70.6/68.2
✓	✓	70.9/68.3

(a)

Method	mAP/mAPH (L2)
Deformable Attention [62]	69.9/67.5
Deformable Attention$*$ [7]	70.0/67.6
DGA (Ours)	70.6/68.2

(b)

DQS vs. Multi-stages to Select Queries. Actually, our DQS not only uses a foreground query selection module to select coarse queries with a high recall, but also leverages a quality query selection module to obtain high-quality queries. However, FocalFormer3D [7] primarily utilizes multi-stage foreground scores to obtain queries with higher recall, but it overlooks the importance of query quality for box localization. Furthermore, we present a comparison between our DQS and the multi-stage approach in Table 6(a). We observe that DQS achieves much better performance (68.2 vs. 65.5), which indicates the importance of selecting high-quality queries. In Table 6(a), we also integrate this multi-stage strategy into our DQS, which brings a subtle gain of 0.1 mAPH/L2.

DGA vs. Deformable Attention in FocalFormer3D. Here, we discuss the difference between our proposed DGA and deformable attention in FocalFormer3D [7] for query interaction. In fact, FocalFormer3D adopts the same deformable attention with deformable DETR [62]. The only difference with  [62] is that FocalFormer3D uses the enhanced queries by combining the RoI features for feature interaction instead of the original queries. In contrast, our DGA is a new deformable attention, which uniformly divides each reference box into grids as the reference points and then utilizes the predicted offsets to achieve a flexible receptive field. In Table 6(b), we provide the comparison with FocalFormer3D, whose performance (67.6 mAPH/L2) is still inferior to our DGA (68.2 mAPH/L2). Additionally, we provide a clear illustration of the difference between our DGA and deformable attention in Figure 6.

Figure 7: Comparison of attention map without DGA (a) and with DGA (b) on the Waymo validation set. Green boxes are the ground truths. The circle represents the position of the attention, and its corresponding color means the weight of the attention. After utilizing DGA, SEED can capture the geometric information of 3D objects in a flexible receptive field and achieve better query interaction.

Figure 8: Comparison of detection results without DQS (a) and with DQS (b) on the Waymo validation set. Blue and green boxes are the prediction and ground truths, respectively. After utilizing DQS, our SEED can successfully detect some hard objects and reduce some false positives, which are highlighted by red circles.

Figure 9: Qualitative results of SEED on the Waymo validation set. Blue and green boxes are the predictions and ground truths, respectively. Besides, we highlight the false positive with a red circle.

0.A.4 Visualization

Visualization of Learned Attention Map. As shown in Figure 7, we present the visualization of learned attention maps under the settings of our SEED with DGA (b) and without DGA (a) (i.e., box attention [30]). In the first column, we can observe that DGA captures the key regions even if there is no accurate proposal box as a reference, benefiting from its flexible receptive field. In the second column, we find that DGA produces higher attention weight on objects than the manner without DGA. In the third column, our DGA not only has good robustness in estimating the direction angle but also focuses on key features, such as the boundary and center of the object. The above visualizations effectively demonstrate the superiority of our DGA for query interaction.

Comparisons for w/ and w/o DQS. To verify the effectiveness of our DQS, we visualize the detection results of our SEED with DQS and without DQS (i.e., directly select Top $N_{f}$ queries in one step) on the Waymo validation set, which is depicted in Figure 8. In the first column, our method can accurately locate all objects and distinguish a False Positive (FP). Besides, as shown in the second column of Figure 8, we observe that our SEED with DQS can pick out some high-quality queries for accurate localization. Finally, surprisingly, our method has the ability to detect a hard distant object even with some occlusions, as shown in the third column of Figure 8. These interesting phenomena illustrate the effectiveness of our approach.

Visualization for SEED. We visualize the qualitative results of SEED on the Waymo validation set, which is shown in Figure 9. Benefiting from the dual query selection for high-quality query selection and the deformable grid attention for effective query interaction, our SEED can detect 3D objects well on large-scale point clouds. Besides, in Figure 9 (d), we carefully find that there are several False Positives (e.g., Pedestrian) in the distant areas. Therefore, we plan to utilize the complementarity of multiple modalities (i.e., 3D point clouds, and 2D camera images) to distinguish these challenging objects in the future.