Convex Combination Consistency between Neighbors for Weakly-supervised Action Localization

Abstract

Weakly-supervised temporal action localization (WTAL) intends to detect action instances with only weak supervision, e.g., video-level labels. The current de facto pipeline locates action instances by thresholding and grouping continuous high-score regions on temporal class activation sequences. In this route, the capacity of the model to recognize the relationships between adjacent snippets is of vital importance which determines the quality of the action boundaries. However, it is error-prone since the variations between adjacent snippets are typically subtle, and unfortunately this is overlooked in the literature. To tackle the issue, we propose a novel WTAL approach named Convex Combination Consistency between Neighbors (C³BN). C³BN consists of two key ingredients: a micro data augmentation strategy that increases the diversity in-between adjacent snippets by convex combination of adjacent snippets, and a macro-micro consistency regularization that enforces the model to be invariant to the transformations w.r.t. video semantics, snippet predictions, and snippet representations. Consequently, fine-grained patterns in-between adjacent snippets are enforced to be explored, thereby resulting in a more robust action boundary localization. Experimental results demonstrate the effectiveness of C³BN on top of various baselines for WTAL with video-level and point-level supervisions. Code is at C3BN.

Index Terms—  Weakly-supervised temporal action localization, adjacent snippets

1 Introduction

Temporal action localization [1, 2, 3] intends to localize action instances and recognize their categories in videos. In recent years, numerous works delve into the fully supervised TAL and gain significant improvement. However, these methods require tremendous manual frame-level annotations, which is expensive and time-consuming. Recently, weakly-supervised TAL (WTAL)[4, 5] has received increasing attention, as it allows us to detect the action instances with only weak supervision, e.g., video-level labels [4] and point-level labels [6, 7]. In particular, video-level labels are the most commonly used.

Fig. 1: Motivation of our method. Due to the vague distinctions between adjacent snippets, an under-performing model may produce similar activations for these snippets, resulting in incomplete/overcomplete proposals (see the upper T-CA). In this paper, we expect the model to correctly classify adjacent snippets, thereby localizing accurate boundaries (see the lower T-CAS).

Mainstream WTAL methods [4, 8], regardless of the types of weak supervisions, employ a video action classification model to learn the Temporal Class Activation Sequence (T-CAS). After training, they utilize the T-CAS to localize action in a bottom-up fashion [1] derived from the watershed algorithm. Specifically, it consists of two main steps. ❶ boundary localization: generating action proposals by thresholding and merging the continuous action regions of T-CAS with multiple thresholds; ❷ proposal evaluation: calculating proposal-level scores by aggregating snippet-level scores within the regions. Recent methods pay many efforts to learn accurate snippet-level scores by various techniques, e.g., pseudo-labeling [9, 10] and contrastive learning [11, 12]. In other words, these methods focus on the semantic relationships between each snippet and global class centers/other snippets. Despite the progress, we argue that they may be sub-optimal since what really matters in ❶ is the relationship between adjacent snippets [13]. As depicted in Fig. 1, adjacent snippets are usually similar in content and thus have close activation, which may cause incomplete or over-complete proposals. Hence, it is necessary to enable the model to be sensitive enough to the fine-grained distinctions between adjacent snippets.

To counteract this issue, we introduce a plug-and-play training strategy dubbed Convex Combination Consistency Between Neighbors (C³BN) for WTAL. The idea of our work stems from MixUp [14], where the classification model trained on the mixture of image pairs achieves promising performance. In light of this, to enhance the ability of the WTAL model to distinguish adjacent snippets, we propose a micro  data augmentation strategy, where the pairs of adjacent snippets (termed as parent snippets) are mixed by convex combination to generate a set of new snippets (termed as child snippets). However, there are still two problems that need to be handled before using the child snippets. The first problem is how to feed the child snippets into WTAL models. Unlike conventional MixUp which treats images as independent instances, most WTAL models take the snippet sequences as input, followed by a few temporal convolution layers to enlarge the temporal receptive field. Therefore, we have to define the temporal orders of the child snippets before they can be processed by the models. To address the challenge, we propose to take advantage of the temporal continuity prior in videos [15]: the scenes usually change smoothly and continuously along temporal dimension. This property implies that the temporal location of each child snippet lies in-between that of its parent snippets. With the virtual locations, we arrange the child snippets of a video into a new sequence, which can be viewed as a locally deformed version of the original sequence.

The second problem is how to utilize the child snippets to promote model training. In MixUp, the mixed sample is assigned with the mixture of the ground-truth labels of the original samples, encouraging the model to behave linearly in-between samples. In our case, however, only weak labels of snippets are available. To this end, we develop a macro-micro consistency regularization, which makes use of both weak supervision and linear behaviour to regularize the model training . Specifically, we introduce three consistency regularization terms to exploit different relationships between child and parent snippets w.r.t. video semantics, snippet predictions and snippet representations, thereby facilitating model training from macro view to micro view and from low-level representations to high-level semantics. In this way, more fine-grained cues in-between adjacent snippets are preserved, eventually improving the robustness of boundary localization.

The idea behind C³BN is generic and conceptually complementary to other methods, which is justified by the performance promotion on a variety of base approaches and datasets. More importantly, extensive quantitative and qualitative results verify the efficacy of C³BN in ❶ boundary localization. Hence, our contributions are: 1) We propose to consider the potential of adjacent snippets in WTAL and then design a micro data augmentation strategy by convex combination of adjacent snippets. 2) We propose three regularization terms to enhance the consistency properties w.r.t. video semantics, snippet predictions and snippet features. 3) Our method can be easily plugged into existing WTAL methods with either video-level supervision or point-level supervision.

2 Related Work

Data Augmentation aims to enlarge the train set using transformations. Conventional image transformations include cropping, flips, rotation, etc. Recent studies consider employing multiple images for augmentation. MixUp [14] proposes to combine the pixel values and labels of two images by linear interpolation. It has been proven effective for the classification task, which is followed by [16]. Our method employ the idea of instance mixtures with task-specific designs. Concretely, we achieve the mixture operation on two snippets within a video rather than on two different videos likewise MixUp, making the perturbations to snippets more controllable for incorporating the proposed method into the existing WTAL frameworks.

MixUp trains a model by linearly interpolating two training examples and their labels [14]. It is proven effective for the classification task, followed by different variants. For instance, [16] extends the linear interpolation from input-level to feature-level. Recently, extensive methods are proposed to incorporate MixUp with semantic segmentation [17], self-supervised learning [18], etc. Our method also employs the idea of instance mixtures, but it is a not trivial extension of previous methods. For example, the original MixUp mixes two randomly selected images. Extending it directly from image to snippet will cause the locations of the generated snippets undefined. In addition, according to [17], another alternative is to mix two random videos snippet-by-snippet. It is also not feasible for WTAL as the video lengths of two videos fed into the models may be different [19] in practice.  Different from above methods, we achieve the mixture operation on two adjacent snippets within a video, yielding more controllable perturbation to snippets for incorporating the method into the existing WTAL models.

Consistency regularization is a crucial technique in semi-supervised learning. It is assumed that a classifier should output the same class probability for an unlabeled sample even after it is augmented. Prior works [20, 21] apply the consistency regularization on different augmentations of an unlabeled sample. After that, several variants [22, 23] are further proposed to extend its applications. Among them, MixMatch [22] also uses MixUp by mixing unlabeled samples and their pseudo-labels.The differences between our method and them are 1) MixMatch randomly mixes two examples, while we only mix the adjacent snippets; 2) MixMatch guesses the hard pseudo-labels of unlabeled samples and relies on a complicated ensemble of multiple predictions to improve the quality of pseudo-labels, whereas we do not guess the hard pseudo-labels of unlabeled samples, thereby reducing undesirable label noise [24, 25].

Self-supervised contrastive learning has attracted much attention in representation learning. The widely adopted contrastive learning optimizes the model by instance discrimination [26, 27]. Specifically, it learns to embed the features of differently augmented versions of the same image to be similar, while being dissimilar if they came from different images. Some recent works [18, 28] have incorporated the idea of MixUp with contrastive learning. Our method is different from these methods in:  1) They regard the mixed samples as queries and the original samples as keys, while we additionally consider a reverse operation to exchange their roles. 2) In our method, the negative samples come from the same video as the positive samples, they can be viewed as hard negative samples, which is important in contarstive learning [27].

Weakly-supervised temporal action localization aims to tackle TAL in the weakly-supervised setting. UntrimmedNet [29] is the pioneering work for it. In addition, there are some attempts [8] to explore WTAL with only point-level action supervision, where each action instance is annotated with only a frame. Recently, [7] proposes a new WTAL setting with point-level background supervision, which annotates a frame in each background segment. In this work, we consider the former two types of supervision with more followers.

Despite of different supervisions, most methods follow a localization-by-classification procedure, which formulates WTAL as a video classification task. Under this pipeline, an important component is to select snippets with high probabilities of actions. In general, there are two groups of strategies: multiple instance learning (MIL)-based methods [12, 30] and attention-based methods [31, 32]. The former obtains the video-level scores from T-CAS by applying a pooling on the top-$k$ values for each class. The latter introduces the attention weights to eliminate background snippets. Recently, some WTAL methods have also noticed contrastive learning [12, 11]. The difference between the above methods and ours is indeed obvious.  They rely on pseudo-labels for defining positive and negative pairs. In contrast, we formulate it as an instance discrimination task, which is simpler and more generic.

Fig. 2: Overview of our method. We first perform convex combination on input snippet sequence $\bm{F}$ to produce the augmented one $\bm{F}^{\prime}$. The procedure is denoted as micro data augmentation. Then we simultaneously feed the $\bm{F}$ and $\bm{F}^{\prime}$ into the model and compute four regularization loss terms. We call it macro-micro consistency regularization.

3 Our Method

In this section, we first review the basic pipeline of the mainstream WTAL methods which we adopt as our baselines. Then we elaborate on our proposed C³BN, which contains two core components: micro data augmentation and macro-micro consistency regularization, as depicted in Fig. 2.

3.1 Preliminaries

For an untrimmed video $\bm{V}$, we can only access its video-level label $\bm{y}=\{y_{c}\}_{c=1}^{C}$, where $C$ represents the number of action classes. A common practice is to employ a video classification model to predict the video-level label $\bm{y}$. Specifically, we first divide $\bm{V}$ into $T$ non-overlapping snippets and then extract snippet-wise features $\bm{F}\in\mathrm{R}^{T\times D_{f}}$ using the pre-trained feature extractor. Since the extractor is not trained from scratch for the WTAL task, we further use several temporal convolution layers for mapping $\bm{F}$ to task-specific feature embedding $\bm{E}=[\bm{e}_{1},...,\bm{e}_{T}]\in\mathrm{R}^{T\times D_{e}}$. Afterward, $\bm{E}$ is fed into a snippet classifier to output a snippet prediction sequence $\bm{P}=[\bm{p}_{1},...,\bm{p}_{T}]\in\mathrm{R}^{T\times C}$, where each snippet has its own scores $\bm{p}_{t}\in\mathrm{R}^{C}$. After this, we aggregate the snippet scores to obtain video-level scores $\bm{\bar{p}}\in\mathrm{R}^{C}$. There are two main mechanisms in the literature for this purpose: MIL-based [12] and attention-based [19]. The former applies a temporal top-k pooling to select high-score snippets while the latter uses snippet-wise attention weights to aggregate snippets via attentive pooling. We refer to Supplementary for details. Then, we formulate a video classification loss as follows:

$$\mathcal{L}_{cls}=-\frac{1}{C}\sum_{c=1}^{C}y_{c}\operatorname{log}\bar{p}_{c}.$$ (1)

3.2 Micro Data Augmentation

Upon the $T$ snippets $\{\bm{f}_{t}\}_{t=1}^{T}$, we perform convex combination on adjacent snippets to generate $T-1$ augmented snippets. Formally,

$$\bm{f}^{\prime}_{t}=\alpha_{t}\bm{f}_{t}+(1-\alpha_{t})\bm{f}_{t+1}\quad\forall t\in\{1,..,T-1\},$$ (2)

where $\bm{f}^{\prime}_{t}$ is the child snippet and $\{\bm{f}_{t},\bm{f}_{t+1}\}$ are its parent snippets. The weight $\alpha_{t}\in(0,1)$ is randomly sampled from a beta distribution $\mathrm{Beta}(\gamma,\gamma)$. Here $\gamma$ is a preset scalar. According to the temporal continuity of videos, the location of $\bm{f}^{\prime}_{t}$ lies in between $t$ and $t+1$. Consequently, $\bm{f}^{\prime}_{t}$ is always in front of $\bm{f}^{\prime}_{t+1}$. Based on this principle, we stack the child snippets $\{\bm{f}^{\prime}_{t}\}_{t=1}^{T-1}$ along temporal dimension to form a 1D feature map dubbed $\bm{F}^{\prime}\in\mathrm{R}^{(T-1)\times D_{f}}$.

Remark.

Notably, our proposed micro data augmentation is fundamentally different from the native MixUp. Specifically, MixUp randomly selects two images for mixing, which is not applicable to the snippets in WTAL as it will render the temporal locations of the mixed snippets undefined. In contrast, we intend to improve the boundary localization in WTAL and thus propose to mix the adjacent snippets, meanwhile taking advantage of the natural continuity within videos to define the locations of the mixed snippets. Besides, we propose various consistency regularization terms to encode more task-specific knowledge, which is also distinct from MixUp.

3.3 Macro-Micro Consistency Regularization

To effectively exploit the child snippets during training, we derive three combinatorial rules to collaboratively regularize the learning procedure.

Video semantic consistency.

The semantic label of each child snippet is expected to be the combination of its parents’ labels. Then we can deduce that the video-level label of child sequence $\bm{F}^{\prime}$ is consistent with that of parent sequence $\bm{F}$. Therefore, it is feasible to utilize the known video-level labels to regularize the child sequences. Specifically, we feed $\bm{F}^{\prime}$ into network and get a video classification loss named $L^{\prime}_{cls}$ in the same form as $L_{cls}$. Since $\bm{F}^{\prime}$ is a locally deformed version of $\bm{F}$, the usage of $L^{\prime}_{cls}$ is supposed to help to improve the robustness of the model. However, such macro regularization ignores the relationship between individual child snippet and parent snippet. To this end, we propose to further regularize the network from micro perspectives.

Snippet prediction consistency.

Inspired by MixUp, we encourage the model to behave linearly in-between adjacent snippets, thereby enhancing the ability of the model to classify adjacent snippets. Without ground-truth labels, we propose to take the predictions of the snippets as their “soft-labels”. Thereafter, we introduce a consistency regularization term to enforce the soft-labels/prediction of a child snippet to be consistent with the same convex combination of soft-labels/predictions of its parent snippets.

By feeding $\bm{F}$ and $\bm{F}^{\prime}$ into the model, we can obtain two snippet prediction sequences, namely $\bm{P}$ and $\bm{P}^{\prime}$. To bridge $\bm{P}$ and $\bm{P}^{\prime}$, we apply convex combination on $\bm{P}$ to obtain the shifted version of $\bm{P}$ dubbed $\bm{\widehat{P}^{\prime}}$. Formally,

$$\bm{\hat{p}^{\prime}}_{t}=\alpha_{t}\bm{p}_{t}+(1-\alpha_{t})\bm{p}_{t+1}\quad\forall t\in\{1,..,T-1\}.$$ (3)

Then we apply the MSE loss to enforce the consistency between $\bm{P}^{\prime}$ and $\bm{\widehat{P}}^{\prime}$. Formally,

$$\mathcal{L}_{cons}=\frac{1}{T-1}\sum_{t=1}^{T-1}||\bm{p}^{\prime}_{t}-\bm{\hat{p}}^{\prime}_{t}||_{2}^{2}.$$ (4)

Snippet feature contrastive-consistency.

Recent methods [33, 34] demonstrate that feature contrastive learning is complementary to classifier learning. In light of these works, we propose to further regularize the intermediate features of the model via contrastive learning. In particular, we develop a contrastive-consistency regularization that integrates the consistency regularization into the contrastive learning scheme. By “contrastive”, we mean that the model is forced to distinguish the parent/child snippets of each child/parent snippet from other parent/child snippets. By “consistency”, we mean that we enforce the model to learn the degree of proximity between child and parent snippets. Here we jointly achieve both of goals, and let the snippet features be aware of the relative similarity in-between adjacent snippets in comparing with other snippets. Consequently, the features would gradually capture necessary fine-grained discriminability to distinguish subtle differences between adjacent snippets.

First, to avoid the conflict between the instance-based contrastive learning and the underlying semantics within the feature embedding $\bm{E}$, we append a projection head [26], comprised by a $\operatorname{FC}$ layer and a $\operatorname{L2}$ normalization, to map $\bm{E}$ into a low-dimensional unit hypersphere in which the contrastive learning is performed. As a result, $\bm{E}$ serves as a medium for information transfer between the classifier and the projection head, allowing the classifier to leverage disciminative fine-grained patterns captured by the projection head to extract accurate class-level patterns. Let us denote the output by $\bm{Z}=[\bm{z}_{1},..,\bm{z}_{t}]\in\mathrm{R}^{T\times D_{z}}$. Here $D_{z}$ is the number of channels with $D_{z}<D_{f}$. Similarly, we can obtain the counterpart of $\bm{E}^{\prime}$ denoted by $\bm{Z}^{\prime}=[\bm{z}^{\prime}_{1},..,\bm{z}^{\prime}_{t}]$.

Then, taking each child snippet $\bm{z}^{\prime}_{t}$ as a query, we define that: 1) its parent snippets $\bm{z}_{t}$ and $\bm{z}_{t+1}$ are its semi-positive keys with the probability of $\alpha_{t}$ and $1-\alpha_{t}$, respectively. 2) other snippets in $\bm{Z}$ are negative keys. Thus we can construct a soft contrastive loss as follows:

$$\mathcal{L}_{cont}=-\frac{1}{T-1}\sum_{t=1}^{T-1}\alpha_{t}\operatorname{log}\frac{\exp({\bm{z}^{\prime}}_{t}^{\top}\bm{z}_{t}/\rho)}{\sum_{\tau=1}^{T}\exp({\bm{z}^{\prime}}_{t}^{\top}\bm{z}_{\tau}/\rho)}+$$ (5)

$$(1-\alpha_{t})\operatorname{log}\frac{\exp({\bm{z}^{\prime}}_{t}^{\top}\bm{z}_{t+1}/\rho)}{\sum_{\tau=1}^{T}\exp({\bm{z}^{\prime}}_{t}^{\top}\bm{z}_{\tau}/\rho)},$$

where $\rho$ is the temperature coefficient.

Eq.(5) only considers the unilateral reference from $\bm{Z}^{\prime}$ to $\bm{Z}$. To explore more fine-grained patterns and enhance the consistency regularization between $\bm{Z}$ and $\bm{Z}^{\prime}$, we propose a bilateral reference mechanism to further take the reference from $\bm{Z}$ to $\bm{Z}^{\prime}$ into consideration. That is, we treat the elements of $\bm{Z}$ as queries and the elements of $\bm{Z}^{\prime}$ as keys. Meanwhile, the snippet-to-snippet relations remain unchanged. As a result, we can compute another contrastive loss dubbed $\mathcal{L}^{\prime}_{cont}$ in a similar way to Eq.(5).

Remark.

The three regularization terms are introduced to work collaboratively, comprehensively promoting the model training from macro view to micro view and from low-level representations to high-level semantics. Intuitively, these constraints together will encourage the model to exploit the various relationships between parent and child snippets, eventually facilitating the exploration of fine-grained distinctions between adjacent snippets. We will show later in Sec. B the efficacy and compatibility of these terms. It is noteworthy that the concrete differences between our proposed regularization strategies and related methods are highlighted in Sec. 2.

4 Experiments

In this section, we empirically validate the effectiveness of C³BN. Due to space limitation, we refer to Supplementary for more details for all experiments. Moreover, we only report the results on WTAL with video-level supervision and refer to Supplementary for WTAL with point-level supervision.

4.1 Dataset and Metrics

THUMOS14 [35] contains untrimmed videos with 20 classes. By convention, we use the 200 videos in validation set for training and 213 videos in test set for evaluation. ActivityNet v1.3 [36] is a large-scale dataset with 200 categories. By convention, we train on the training set with 10, 024 videos and test on validation set with 4, 926 videos. The mean Average Precision (mAP) values under different temporal intersection over union (tIoU) thresholds are used as metrics.

4.2 Ablation study

Effectiveness on different baselines

To validate the generic effectiveness of C³BN, we incorporate C³BN into different WTAL methods. Specifically, we plug C³BN into four baselines, including the aforementioned MIL-based baseline (named MIL) and three off-the-shelf well-performing approaches, i.e., BaSNet [37], FACNet [19], and recent DELU [38](ECCV 2022). - Table 1 shows the performance comparison. We can observe that C³BN consistently improves the performance of all baselines by 7.4%, 3.9%, 1.5%, and 1.1% on AVG mAP for MIL, BaSNet, FACNet, and DELU, respectively. These results clearly confirm the generalization ability of C³BN.

Table 1: Comparisons of performance on THUMOS14. The AVG represents average mAP under IoU thresholds of 0.1:0.7. We re-implement all the adopted baselines for C³BN.

Method	mAP @ IoU (%)							AVG
	0.1	0.2	0.3	0.4	0.5	0.6	0.7
WUM [39]	67.5	61.2	52.3	43.4	33.7	22.9	12.1	41.9
AUMN [40]	66.2	61.9	54.9	44.4	33.3	20.5	9.0	41.5
CoLA [12]	66.2	59.5	51.5	41.9	32.2	22.0	13.1	40.9
UGCT [41]	69.2	62.9	55.5	46.5	35.9	23.8	54.0	43.6
DCC [11]	69.0	63.8	55.9	45.9	35.7	24.3	13.7	44.0
RSKP [9]	71.3	65.3	55.8	47.5	38.2	25.4	12.5	45.1
ASM-Loc [42]	71.2	65.5	57.1	46.8	36.6	25.2	13.4	45.1
Li et al. [43]	69.7	64.5	58.1	49.9	39.6	27.3	14.2	46.1
MIL	56.0	46.4	37.3	30.3	22.0	15.0	8.2	30.7
+ C3BN	63.0	56.7	48.0	39.8	29.9	19.2	10.2	38.1+7.4
BaSNet [37]	62.0	54.6	44.6	35.7	25.9	17.0	8.9	35.5
+ C3BN	64.3	58.4	49.7	40.6	30.8	19.9	12.1	39.4+3.9
FACNet [19]	71.8	64.0	53.7	42.5	30.7	20.9	12.2	42.3
+ C3BN	72.6	66.5	56.4	43.8	32.6	21.0	12.7	43.7+1.5
DELU [38]	70.1	64.5	56.0	47.6	40.2	27.8	15.0	45.9
+ C3BN	71.6	66.0	58.2	49.3	41.0	27.9	15.3	47.0+1.1

Table 2: Results on ActivityNet v1.3. AVG indicates the average mAP at IoU thresholds 0.5:0.05:0.95.

Method	mAP @ IoU
	0.5	0.75	0.95	AVG
WUM [39]	37.0	23.9	5.7	23.7
AUMN [40]	38.3	23.5	5.2	23.5
UGCT [41]	39.1	22.4	5.8	23.8
DCC [11]	38.8	24.2	5.7	24.3
RSKP [9]	40.6	24.6	5.9	25.0
ASM-Loc [42]	41.0	24.9	6.2	25.1
BaSNet	35.6	21.0	5.3	21.7
+ C3BN	37.3	22.4	5.4	23.0+1.3
FACNet [19]	40.1	24.2	5.8	24.7
+ C3BN	45.2	26.9	5.9	27.3+2.6

Table 3: Ablation studies of the proposed regularization terms on THUMOS14.

#	Loss terms				Baselines
	$\mathcal{L}^{{}^{\prime}}_{cls}$	$\mathcal{L}_{cons}$	$\mathcal{L}_{cont}$	$\mathcal{L}^{\prime}_{cont}$	BaSNet	FACNet
1					35.5	42.3
2	$\surd$				35.9	42.6
3		$\surd$			37.7	43.0
4			$\surd$	$\surd$	36.5	42.8
5	$\surd$	$\surd$			38.0	43.0
6	$\surd$		$\surd$	$\surd$	36.8	42.9
7		$\surd$	$\surd$	$\surd$	38.8	43.5
8	$\surd$	$\surd$		$\surd$	39.1	43.4
9	$\surd$	$\surd$	$\surd$		38.9	43.5
10	$\surd$	$\surd$	$\surd$	$\surd$	39.4	43.7

Table 4: Contribution of C³BN to ❶ boundary localization and ❷ proposal evaluation. “❶Base+❷Base” represents the baseline, i.e., BaSNet and FACNet. “❶C³BN+❷Base” indicates that we combine our ❶ and the ❷ of baseline. Likewise for “❶Base+❷C³BN” and “❶C³BN+❷C³BN”.

	❶Base+❷Base	❶C3BN+❷Base	❶Base+❷C3BN	❶C3BN+❷C3BN
BaSNet	35.5	38.0	37.1	39.4
FACNet	42.3	43.2	42.8	43.7

Contribution of each regularization term

Our C³BN introduces several regularization/loss terms during training. To verify the contribution of each regularization term, we conduct a detailed analysis in Table 3. Here, we regard BaSNet and FACNet as the baselines to conduct the ablation study due to their favorable efficiency and flexibility. Comparing the rows #1-4, we can see that each regularization term contributes to the performance. Furthermore, it can be seen that the micro consistency regularization terms (i.e., $\mathcal{L}_{cons}$, $\mathcal{L}_{cont}$ and $\mathcal{L}^{\prime}_{cont}$ ) bring larger gains than the macro term (i.e., $\mathcal{L}^{\prime}_{cls}$). This indicates that fine-grained information is pretty important in WTAL.

Complementarity of regularization terms

In rows #5-6 of Table 3, we evaluate the performance of combining any two of the regularization terms. We can see that combining two terms consistently outperforms each of them. Moreover, after combining all the terms, the model obtains the best performance, as shown in row #10. These results evidently demonstrate the complementary relations of the regularization terms.

Effectiveness of bilateral reference mechanism

We propose a bilateral reference mechanism in the snippet feature contrastive-consistency regularization, resulting in two loss terms (i.e., $\mathcal{L}_{cont}$ and $\mathcal{L}^{\prime}_{cont}$). In rows #8-9 of Table 3, we provide the results where only one of them is used. It can be seen that the combination of $\mathcal{L}_{cont}$ and $\mathcal{L}^{\prime}_{cont}$ outperforms using only of them, validating the superiority of our proposed bilateral reference mechanism.

Necessity of projection head

We adopt a projection head to transform the embeddings into a new latent space so that the instance-based contrastive learning would not directly hurt the inherent semantics of the embeddings. To show the necessity of our design, we conduct an experiment where the projection head is removed. The experimental results show that it leads to a performance degradation of 1.1% on BaSNet (from 39.4% to 38.3%) and 0.9% on FACNet (from 43.7% to 42.8%). This evidently justifies that the projection head is essential in our method.

4.3 Comparisons with state-of-the-arts (SOTAs)

Table 1 and Table 2 show the comparison between our method and previous approaches on THUMOS14 and ActivityNet v1.3, respectively. It can be seen that after integrating recent strong WTAL baselines, our method achieves the SOTA performances on both datasets. In Supplementary, we also report the results on ActivityNet v1.2 [36] (a subset of ActivityNet v1.3), which is used in some previous methods [38, 43].

4.4 Evaluation for Motivation

In this section, we provide experimental results to deliver more insights for our motivation (depicted in Fig. 1).

To begin with, we investigate the contribution of C³BN to “❶ boundary localization” and “❷ proposal evaluation” respectively in Table 4. To be specific, in the test phase, we alternately replace the results of ❶ and ❷ of baseline with that of baseline+C³BN (ours). As we can see, the performances significantly increase (from 35.5% to 38.0% on BaSNet and from 42.3% to 43.2% on FACNet ) once replacing the ❶ of baselines with our ❶. As a comparison, the replacement on ❷ only brings the performance to 37.1% on BaSNet and 42.8% on FACNet. These results indicate that C³BN is particularly beneficial for boosting boundary localization.

To understand how C³BN improves the boundary localization, we first compute the absolute score difference between each pair of adjacent snippets, i.e., $\bm{d}_{t}=|\bm{p}_{t+1}-\bm{p}_{t}|$. Next, we calculate the average entropy of $\bm{d}_{t}$ of all pairs, i.e., $H(\bm{d}_{t})=\text{mean}(-\bm{d}_{t}\log\bm{d}_{t})$. Intuitively, the $H(\bm{d}_{t})$ reflects the distribution of the differences, i.e., the smaller the $H(\bm{d}_{t})$, the more polarized the differences, and the more discriminative and confident the model is about the relations of adjacent snippets. Experimental results show that the corresponding $H(\bm{d}_{t})$ of BaSNet and FACNet is 0.0876 and 0.0673 respectively while that of “BaSNet + C³BN” and “FACNet + C³BN” is 0.0684 and 0.0592 respectively. Hence, we conjecture that the boundary localization is improved because C³BN renders the model more confident (discriminative) in distinguishing the relations of adjacent snippets.

In Supplementary, we provide extensive qualitative results to demonstrate it.

5 Conclusion

In this paper, we propose a universal training strategy dubbed C³BN for weakly-supervised action localization. Concretely, C³BN first produces new snippets by convex combination between adjacent snippets, and then uses them to regularize the model with three regularization terms, i.e., video semantic consistency, snippet prediction consistency and snippet feature contrastive-consistency. The empirical results validate that C³BN is applicable to various WTAL methods with video-level supervision and point-level supervision, and helps establish the new SOTA results on all the evaluated datasets.

Appendix A Additional Details of Our Method

Additional details of baseline.

Given the video labels, we first aggregate the snippet scores to obtain video class scores for computing a video classification loss. There are two main strategies in the literature for this purpose: MIL-based methods [29, 12] and attention-based methods [44, 45].

The MIL-based methods average the top-$k$ snippet logit scores (dubbed $\bm{S}=[\bm{s}_{1},...,\bm{s}_{T}]\in\mathrm{R}^{T\times C}$) along temporal dimension for each class to build the video class score:

$${\bar{s}_{c}=\frac{1}{k}\max_{\begin{subarray}\bm{l}\subset\{1,..,T\}\\ \quad|\bm{l}|=k\end{subarray}}\sum_{\tau\in\bm{l}}{s_{\tau,c}}}\quad\forall c\in\{1,..,C\},$$ (6)

where $k$ is a hyper-parameter proportional to the video length $T$, i.e., $k=\max(1,T//r)$, and $r$ is a pre-defined parameter. Thereafter, we obtain the probability for each class by applying the $\operatorname{Softmax}$ function to the aggregated scores:

$$\bar{p}_{c}=\frac{\exp(\bar{s}_{c})}{\sum_{i=1}^{C}{\exp(\bar{s}_{i})}}\quad\forall c\in\{1,..,C\}.$$ (7)

The attention-based methods first learn a set of snippet-wise attention weights (dubbed $\bm{\lambda}=[\lambda_{1},...,\lambda_{T}]\in\mathrm{R}^{T}$). Then the attention weights are used to aggregate snippet-level scores into video-level scores as follows,

$$\bm{\bar{p}}=\frac{1}{\sum_{t=1}^{T}\lambda_{t}}\sum_{t=1}^{T}\lambda_{t}\bm{p_{t}}.$$ (8)

Fig. 3: Illustration of micro data augmentation. The $\{\bm{f}_{t}\}_{t=1}^{T-1}$ indicates the parent sequence and the $\{\bm{f}^{\prime}_{t}\}_{t=1}^{T-1}$ represents the child sequence.

Fig. 4: Illustration of snippet feature contrastive-consistency. It contains two parts: the child snippets are query while the parent ones are key (left); the parent snippets are query while the child ones are key (right).

Illustration of micro data augmentation and snippet feature contrastive-consistency .

In Fig. 3 and Fig. 4, we shows more details for micro data augmentation and snippet feature contrastive-consistency respectively.

Training objective.

To train the entire model in an end-to-end fashion, we optimize the following loss

$$\mathcal{L}=\mathcal{L}_{base}+\lambda_{1}\mathcal{L}^{\prime}_{cls}+\lambda_{2}\mathcal{L}_{cons}+\lambda_{3}(\mathcal{L}_{cont}+\mathcal{L}^{\prime}_{cont}).$$ (9)

Here $\lambda_{*}$ indicates the weight term. $\mathcal{L}_{base}=\mathcal{L}_{cls}+\mathcal{L}_{other}$ represents the objective function of the baseline, where $\mathcal{L}_{other}$ represents the sum of other losses apart from $\mathcal{L}_{cls}$.

Appendix B Experiments

B.1 Implementation Details

Implementation details of baselines

MIL is a simple MIL-based baseline, which has been introduced Sec. A. Briefly, it first performs snippet classification over the input sequences to obtain T-CAS, then utilizes a top-k pooling operation following [4] to build video-level scores. At last, it minimizes a cross-entropy loss with the video-level labels. We implement MIL based on the code provided by [12], except that we disable the contrastive loss used in [12]. BaSNet [37] introduces an additional class to the snippet-level classifier for modeling background. Besides, it utilizes a class-agnostic attention layer to highlight foreground snippets and suppress the background snippets. The performance for BaSNet implemented by [37] is not stable on THUMOS14. Thus, we re-implement it on the basis of the code of [46, 19]. We refer to our released code for details. FACNet [19] proposes to force the foreground score output by the snippet-level classifier and that output by the attention layer to be consistent. We implement the FACNet in a similar way to the BaSNet. DELU [38] extends the traditional paradigm of evidential deep learning to adapt to the weakly-supervised multi-label classification goal. SF-Net [8] mines pseudo action and background frames by adaptively expanding each annotated single frame to its nearby frames. LACP [47] takes the points as seeds and searchs for the optimal sequence that is likely to contain complete action instances while agreeing with the seeds. For DELU, SF-Net and LACP, we use their official code to implement them. All the baselines are implemented on the Pytorch library.

Training details

For the feature extraction, we first sample RGB frames at 25 fps for each video and apply the TV-L1 algorithm [48] to generate optical flow frames. Then, we divide each video into non-overlapping snippets with consecutive 16 frames. Thereafter, we perform the I3D network [49] pre-trained on the Kinetics dataset [50] to obtain the snippet-level feature. The proposed C³BN and the baseline models are jointly trained in an end-to-end manner. Here, we only provide the details about the specific hyperparameters of C³BN. The temperature $\rho$ is set as $0.1$ and the output dimension of projection head $D_{z}$ is set as $128$ and the $\gamma$ is set as $2$. Since the amplitudes of basic loss in different baselines are different, the loss weights $\lambda_{2}$ and $\lambda_{3}$ are set differently on different baselines, except that the $\lambda_{1}$ is always set by $1$. On BaSNet, the $\lambda_{2}$ and $\lambda_{3}$ are set as $10$ and $0.1$, respectively. On FACNet, the $\lambda_{2}$ and $\lambda_{3}$ are set as $1$ and $0.2$, respectively. On DELU, the $\lambda_{2}$ and $\lambda_{3}$ are set as $0.1$ and $0.3$, respectively. On SF-Net, the $\lambda_{2}$ and $\lambda_{3}$ are set as $10$ and $0.2$, respectively. On LACP, the $\lambda_{2}$ and $\lambda_{3}$ are set as $2$ and $0.1$, respectively. All experiments are conducted on one GTX 3090 GPU (24 GB).

Inference details

The proposed C³BN is a training strategy, which introduces no overhead in the test phase. As addressed in main paper, for existing WTAL methods, there exist two core procedures in the test phase, i.e., boundary localization and proposal evaluation. Despite of this, the actual test paradigms in different baselines are slightly different. We hereby take our implementation on BaSNet as an example for illustration. In the inference stage, we first threshold on the video-level scores to determine the action categories. And then for the selected action class, we apply a threshold strategy on the T-CAS to obtain action proposals. After obtaining the action proposals, we calculate the class-specific score for each proposal using the outer-inner-contrastive technique [51]. To enrich the proposal pool, multiple thresholds are applied. The Non-Maximum Suppression (NMS) is used to remove duplicated proposals, where SoftNMS [52] is particularly adopted.

Additional details of datasets.

For W-TAL with video-level supervision, we conduct experiments on two popular benchmark datasets: THUMOS14 [35] and ActivityNet v1.3 [36]. THUMOS14 contains untrimmed videos with 20 classes. The video length varies from a few seconds to several minutes and multiple action instances may exist in a single video, which makes it very challenging. By convention, we use the 200 videos in validation set for training and the 213 videos in test set for evaluation. ActivityNet v1.3 is a large-scale dataset with $200$ categories. Since the annotations for the test set are not released, following the common practice, we train on the trainining set with $10,024$ videos and test on validation set with $4,926$ videos. ActivityNet v1.2 [36] is a subset of ActivityNet v1.3, and covers 100 action categories with 4, 819 and 2, 383 videos in the training and validation sets, respectively

For W-TAL with point-level supervision, three public datasets are commonly used, including THUMOS14, BEOID [53], and GTEA [54]. GTEA contains 28 videos of 7 fine-grained types of activities in the kitchen. There are 58 videos from 30 action classes in BEOID. We follow [8] to split the training and test sets.

Fig. 5: Qualitative results of T-CAS. “GT” denotes ground-truth annotation. “Base” denotes the T-CAS predicted by BaSNet while “+C³BN” denotes that predicted by “BaSNet + C³BN”. The solid boxes indicate some noteworthy regions.

Fig. 6: Qualitative results of proposals. “GT” denotes ground-truth annotation. “Base” denotes the proposals output by BaS-Net while “+C³BN” denotes the proposals generated by “BaS-Net + C³BN”.

Fig. 7: Samples of failure cases. We highlight the regions with wrong predictions by dashed boxes.

B.2 Qualitative Results

To gain further insights, we visualize a couple of samples for comparing the snippet-level predictions of the baseline model and that of our method in Fig. 5. From the solid boxes of Fig. 5, it can be seen that our method is able to generate accurate action boundaries, while the baseline suffers from poor discrimination around boundaries. These visualized examples evidently and intuitively verify our motivation.

In Fig. 6, we show some visualized results of the proposal generation. It can be seen that compared with the baseline model, the boundary of the proposals generated by our method is closer to the ground-truth action boundaries. This further demonstrates the superiority of our method in boundary localization.

Additionally, Fig. 7 illustrates some failure cases of our method. The failure cases are caused by 1) low quality of images (see the top row); 2) ambiguous action boundary annotation (see the middle row) ; 3) indistinguishable body motions (see the bottom row). These challenging cases are our future work.

Table 5: Comparison on THUMOS14, GTEA and BEOID. AVG indicates the average mAP at IoU thresholds 0.1:0.7.

Dataset	Method	0.1	0.3	0.5	0.7	AVG
THUMOS	DC [55]	72.8	58.1	34.5	11.9	44.3
	SF-Net [8]	69.9	53.6	29.9	10.0	40.9
	Our C3BN	73.8	57.3	30.7	10.3	43.3+2.4
	LACP [47]	75.5	64.0	44.5	20.6	52.3
	Our C3BN	76.0	65.7	47.6	22.6	54.1+1.8
GTEA	DC [55]	59.7	38.3	21.9	18.1	33.7
	SF-Net [8]	53.8	38.0	21.9	18.2	32.3
	Our C3BN	55.1	40.7	22.9	18.2	34.2+1.9
BEOID	DC [55]	63.2	46.8	20.9	5.8	34.9
	SF-Net [8]	60.3	43.2	21.7	11.0	33.9
	Our C3BN	65.5	44.0	26.3	9.7	37.0+3.1
	LACP [47]	81.4	73.1	45.8	21.7	56.6
	Our C3BN	82.1	73.3	47.4	23.3	57.6+1.0

B.3 Results on WTAL with Point-Level Labels

There are also a few works proposed for WTAL with point-level supervision, e.g., SFNet [8], LACP [47], and DC [55]. Our C³N is generic and is expected to work well for this task. We hereby take SFNet [8] and LACP [47] as the baselines, and conduct experiments on three benchmark datasets: THUMOS14, BEOID, and GTEA.

We compare the proposed approach with recent methods for WTAL with point-level supervision in Table 5. It can be seen that our C³BN improves the performances of SF-Net [8] and LACP [47] by a large margin. Besides, our method also outperforms the recently proposed DC [55] and achieves SOTA performance on all three datasets. These results validate that our C³BN is compatible with WTAL with different weak supervisions.

Table 6: Results on ActivityNet v1.2. AVG indicates the average mAP at IoU thresholds 0.5:0.05:0.95.

	0.5	0.75	0.95	AVG
TSCN [56]	37.6	23.7	5.7	23.6
WUM [39]	41.2	25.6	6.0	25.9
CoLA [12]	42.7	25.7	5.8	26.1
D2-Net [57]	42.3	25.5	5.8	26.0
ACGNet [58]	41.8	26.0	5.9	26.1
Li et al.[43]	41.6	24.8	5.4	25.2
DELU [38]	44.2	26.7	5.4	26.9
FACNet [19]	41.2	26.2	5.9	26.3
+ Our C3BN	43.9	27.1	6.3	27.4+1.1

B.4 Results on ActivityNet v1.2

In some previous WTAL methods (especially early methods), ActivityNet v1.2 is a preferred dataset rather than ActivityNet v1.3. Hence, we also evaluate our method on this benchmark. The results are presented in Table 6. After combining with FACNet [19], our method significantly outperforms previous methods, including the very recent method DELU (ECCV 2022). The favorable performances on all the benchmarks demonstrate the overall superiority of our proposed method.