One-sentence Summary

Researchers from Kyung Hee University and NAVER Cloud propose RCORE, a zero-shot compositional action recognition method that mitigates object-driven shortcuts by using Co-occurrence Prior Regularization to treat frequent co-occurrences as hard negatives and Temporal Order Regularization for Composition to enforce temporal-order sensitivity, thereby improving compositional generalization on Sth-com and EK100-com.

Key Contributions

• Diagnostic metrics quantify object-driven shortcut learning and co-occurrence overfitting in zero-shot compositional action recognition, revealing that existing models rely on training co-occurrence priors instead of temporal verb cues.
• RCORE, a regularization framework, includes Co-occurrence Prior Regularization (CPR) that treats frequent co-occurrence priors as hard negatives and adds supervision for unseen compositions, and Temporal Order Regularization for Composition (TORC) that enforces temporal-order sensitivity to learn temporally grounded verb representations.
• On Sth-com and EK100-com, RCORE reduces shortcut diagnostic scores and improves unseen composition accuracy, demonstrating stronger compositional generalization without test-tuned bias calibration.

Introduction

The authors study Zero-Shot Compositional Action Recognition (ZS-CAR), where models must recognize unseen verb–object pairs from video while the verb and object vocabularies remain fixed across splits. Robust video understanding depends on correctly decomposing actions into these primitives, but prior models frequently rely on object-driven shortcuts: they predict verbs from static object cues rather than temporal evidence. This failure stems from two intertwined issues—sparse and skewed compositional supervision that amplifies co-occurrence priors, and an asymmetric learning difficulty where objects are easier to recognize from a single frame than verbs, which require multi-frame reasoning. Existing approaches thus generalize poorly to unseen compositions and often confuse verbs with opposite temporal order. To address this, the authors first diagnose the shortcut behavior with two metrics (False Seen Prediction and False Cooccurrence Prediction) that quantify how often unseen inputs collapse to frequent training pairs. They then propose RCORE, a framework with Co-occurrence Prior Regularization to expand supervision over absent compositions and suppress dominant pairs, and Temporal Order Regularization for Composition to enforce temporally grounded verb representations. RCORE reduces shortcut reliance and improves unseen compositional generalization across multiple benchmarks and backbones.

Method

The authors introduce RCORE, a diagnosis-driven learning framework designed to mitigate object-driven shortcuts in Zero-Shot Compositional Action Recognition. RCORE improves compositional generalization by strengthening verb-object representations under sparse supervision through two complementary components: Co-occurrence Prior Regularization (CPR) and Temporal Order Regularization for Composition (TORC). Following prior work, the backbone employs adapter-based tuning.

As shown in the figure below:

Feature Extraction Given a video with $T$T frames, the backbone encoder outputs frame-level features $\mathbf{F} \in \mathbb{R}^{T \times D}$F∈RT×D. These are transformed into verb features $\mathbf{F}^V \in \mathbb{R}^{T \times D}$FV∈RT×D and object features $\mathbf{f}^O \in \mathbb{R}^D$fO∈RD using dedicated encoders. Text embeddings for verbs and objects, $\mathbf{E}^V \in \mathbb{R}^{|\mathbb{Y}^V| \times D}$EV∈R∣YV∣×D and $\mathbf{E}^O \in \mathbb{R}^{|\mathbb{Y}^O| \times D}$EO∈R∣YO∣×D, are obtained by feeding class-specific prompts into the text encoder.

Co-occurrence Prior Regularization To mitigate object-driven shortcuts under sparse and skewed compositional supervision, CPR injects supervision for synthesized verb-object pairs while preserving closed-world optimization stability.

Synthesized composition supervision: To provide explicit supervision for compositions absent from the training set, the authors synthesize videos representing new composition labels. Given a training sample $\mathbf{X}_i$Xi​ with composition label $\mathbf{y}_i^C = (\mathbf{y}_i^V, \mathbf{y}_i^O)$yiC​=(yiV​,yiO​), a new video $\tilde{\mathbf{X}}_i$X~i​ is constructed by injecting the static object cue from another video $\mathbf{X}_j$Xj​ into the high-motion regions of $\mathbf{X}_i$Xi​. For each frame $k \in \{1, \ldots, T\}$k∈{1,…,T}, this is formulated as: $\tilde{\mathbf{X}}_i^k = (\mathbf{1} - \lambda \mathbf{M}_i^k) \odot \mathbf{X}_i^k + (\lambda \mathbf{M}_i^k) \odot \mathbf{X}_j^{\lfloor T/2 \rfloor}$X~ik​=(1−λMik​)⊙Xik​+(λMik​)⊙Xj⌊T/2⌋​ where $\lambda$λ controls the injection strength and $\mathbf{M}_i^k \in \{0, 1\}^{H \times W}$Mik​∈{0,1}H×W is the high-motion region mask extracted by a learning-free estimator. Soft labels are used for the synthesized video.

Co-occurrence prior regularization loss: To counter score collapse from skewed co-occurrence statistics, a margin constraint enforces that the target synthesized composition outranks frequent seen hard negatives by at least $m$m. The hard negatives are defined as compositions sharing either the verb or the object but not both. The loss is: $L_{\text{CPR}} = \sum_{\mathbf{y}^C \in \mathbb{Y}_{\text{pen}}} \max(0, s(\mathbf{y}^C) - s(\mathbf{y}_{i, \text{new}}^C) + m)$LCPR​=∑yC∈Ypen​​max(0,s(yC)−s(yi,newC​)+m) where $s(\mathbf{y}^C)$s(yC) is the logit for composition $\mathbf{y}^C$yC, and $\mathbb{Y}_{\text{pen}}$Ypen​ is the intersection of hard negatives and frequently seen compositions.

Batch-adaptive label-space expansion: To supervise novel compositions without optimizing over the full joint space, an expanded label set $\mathbb{Y}_{\text{exp}} = \mathbb{Y}_{\text{seen}} \cup \mathbb{Y}_{\text{new}}$Yexp​=Yseen​∪Ynew​ is constructed for each mini-batch. Cross-entropy is computed only over $\mathbb{Y}_{\text{exp}}$Yexp​.

Temporal Order Regularization for Composition To counter object-driven shortcuts in verb learning, TORC regularizes the verb representation to depend on temporal structure.

Temporal perturbation: Given frame-level features $\mathbf{F} = (\mathbf{f}_1, \ldots, \mathbf{f}_T)$F=(f1​,…,fT​), two regularization views are formed: a reversed sequence $\mathbf{F}_{\text{rev}} = (\mathbf{f}_T, \ldots, \mathbf{f}_1)$Frev​=(fT​,…,f1​) and a temporally shuffled sequence $\mathbf{F}_{\text{shuffled}} = \pi(\mathbf{F})$Fshuffled​=π(F). These are fed into the verb encoder to obtain $\mathbf{f}_{\text{rev}}^V$frevV​ and $\mathbf{f}_{\text{shuffled}}^V$fshuffledV​.

TORC loss: The model is pushed to distinguish forward and reversed temporal semantics by minimizing the cosine similarity between the original verb feature $\mathbf{f}^V$fV and the reversed feature $\mathbf{f}_{\text{rev}}^V$frevV​: $L_{\text{cos}} = \frac{\mathbf{f}^{V\top} \mathbf{f}_{\text{rev}}^V}{\|\mathbf{f}^V\| \|\mathbf{f}_{\text{rev}}^V\|}$Lcos​=∥fV∥∥frevV​∥fV⊤frevV​​ Additionally, to prevent confident verb predictions when temporal order is disrupted, a negative entropy loss is applied to the shuffled features: $L_{\text{ent}} = \sum_{m=1}^{|\mathbb{Y}^V|} p_m \log p_m$Lent​=∑m=1∣YV∣​pm​logpm​ where $p_m$pm​ is the predicted probability of the $m$m-th verb class based on cosine similarity with text embeddings. The total TORC loss is $L_{\text{TORC}} = L_{\text{cos}} + L_{\text{ent}}$LTORC​=Lcos​+Lent​.

Training Objective The authors compute verb and object logits based on cosine similarity scores and apply standard cross-entropy losses $L_{\text{com}} = L_V + L_O$Lcom​=LV​+LO​. Composition logits $\mathbf{S}_C$SC​ are obtained by aggregating factorized logits. The final composition loss utilizes the dynamically constructed denominator $\mathbb{Y}_{\text{exp}}$Yexp​: $L_{\text{comp}} = -\log \left( \frac{\exp(\hat{s}_c)}{\sum_{c=1}^{|\mathbb{Y}_{\text{exp}}|} \exp(s_c)} \right)$Lcomp​=−log(∑c=1∣Yexp​∣​exp(sc​)exp(s^c​)​) The total training objective is: $L_{\text{total}} = \alpha L_{\text{com}} + \beta L_{\text{comp}} + \gamma L_{\text{TORC}} + \delta L_{\text{CPR}}$Ltotal​=αLcom​+βLcomp​+γLTORC​+δLCPR​ where $\alpha, \beta, \gamma$α,β,γ, and $\delta$δ are hyperparameters.

Experiment

Experiments diagnose that zero-shot compositional action recognition models rely on object-driven shortcuts, where easier object cues dominate verb learning and cause poor generalization to unseen compositions, a tendency that persists even with video-pretrained backbones. The proposed RCORE method reduces this reliance on co-occurrence priors and mitigates shortcuts, leading to improved unseen composition accuracy and a positive compositional gap across benchmarks. Ablation studies confirm that combining co-occurrence penalty regularization and temporal opposite relation constraints yields more robust verb representations and effective compositional behavior.

The table reports the change in composition generalization (∆cG) on Something-something for CLIP and InternVideo2 backbones. Both backbones show a positive ∆cG on seen compositions but a negative ∆cG on unseen compositions, indicating that the method improves seen accuracy at the cost of unseen generalization. The CLIP backbone exhibits a smaller drop on unseen compositions than InternVideo2. Both backbones improve seen composition accuracy (positive ∆cG) while degrading unseen composition accuracy (negative ∆cG). CLIP shows a smaller decline in unseen composition accuracy than InternVideo2, suggesting a less severe trade-off between seen and unseen performance.

RCORE consistently improves unseen composition accuracy and verb recognition on unseen pairs over the C2C baseline, with gains observed for both CLIP and InternVideo2 backbones. It is the only method to achieve a positive compositional gap on unseen compositions, indicating better generalization to novel combinations. In contrast, Jung et al. achieves the highest seen composition accuracy among CLIP baselines but the lowest unseen accuracy, revealing a strong bias toward seen pairs. RCORE is the only method with a positive compositional gap on unseen compositions, whereas all baselines exhibit negative gaps, signaling improved open-world compositional behavior. Jung et al. achieves the highest seen composition accuracy among CLIP baselines but the lowest unseen accuracy, sacrificing generalization for seen-pair performance.

Under the closed-world protocol on Sth-com, RCORE achieves the highest harmonic mean and AUC among methods without enhanced backbones. The enhanced variant further improves both metrics, surpassing all baselines including LogicCAR. RCORE attains the highest closed-world H.M. and AUC within its backbone group, outperforming C2C and Jung et al. RCORE (enhance) yields the best overall H.M. and AUC, exceeding LogicCAR and C2C (enhance).

On EK100-com, RCORE achieves the highest harmonic mean of seen and unseen composition accuracy, driven by a large improvement in unseen compositions over the C2C baseline. It is the only method to exhibit a positive compositional gap on unseen pairs, indicating reduced reliance on co-occurrence priors and better generalization to novel verb–object combinations. In contrast, Jung et al. underperforms C2C on both seen and unseen compositions under this dataset's sparse supervision. RCORE improves unseen composition accuracy substantially over C2C (by 6.9 points with CLIP and 7.0 points with InternVideo2) while seen accuracy decreases. RCORE is the only approach with a positive compositional gap on unseen compositions, contrasting with the negative gaps of all baselines. Jung et al. yields lower composition accuracy than C2C on both seen and unseen splits, with drops of 3.2 and 4.6 points respectively. RCORE attains the best harmonic mean of seen and unseen composition accuracy within each backbone group.

Combining CPR and TORC yields the best overall performance, improving unseen composition accuracy and harmonic mean while mitigating the seen-unseen trade-off observed with CPR alone. TORC alone drives the largest gains in verb generalization on unseen compositions, and its two complementary loss terms together provide the biggest boost. For CPR, batch-adaptive label space expansion and penalizing only frequent hard negatives are critical for achieving the best balance between seen and unseen composition recognition. Using TORC alone improves verb accuracy on unseen compositions by 3.5 points and unseen composition accuracy by 3.8 points over the baseline, making it the strongest single component for verb generalization. CPR alone raises unseen composition accuracy by 3.1 points but reduces seen composition accuracy by 4.2 points, revealing a clear trade-off between seen and unseen performance. Combining CPR and TORC achieves the highest harmonic mean (39.48) and unseen composition accuracy (35.16), with a smaller drop in seen accuracy than CPR alone, indicating complementary effects. In TORC, using both the temporal discrimination loss and the static-cue prevention loss together yields the largest gains in unseen verb and composition accuracy. Batch-adaptive label space expansion in CPR improves unseen composition accuracy by 2.3 points over closed-world training, while naive full-space training severely harms unseen composition performance. Penalizing only frequent hard negatives in CPR gives the best harmonic mean (37.7%), whereas penalizing all hard negatives degrades object prediction and overall composition accuracy.

The experiments evaluate composition generalization on Something-something and EK100-com using CLIP and InternVideo2 backbones. Standard methods improve seen composition accuracy at the cost of unseen generalization, whereas RCORE is the only approach that yields a positive compositional gap on unseen compositions, indicating reduced reliance on co-occurrence priors. Combining CPR and TORC achieves the best balance, with TORC enhancing verb generalization and CPR's batch-adaptive label expansion and selective hard-negative penalization proving critical for open-world performance.