摘要
实例级对比学习技术依赖于数据增强和对比损失函数，在视觉表示学习领域取得了显著成功。然而，这些技术并不适合利用视频中的丰富动态结构，因为操作是在许多增强实例上进行的。在本文中，我们提出了一种新颖的方法——“视频跨流原型对比”（Video Cross-Stream Prototypical Contrasting），该方法从RGB视图和光流视图中预测一致的原型分配，操作对象为样本集。具体而言，我们交替优化过程；在优化其中一个流时，所有视图都被映射到一组流原型向量。每个分配都是通过除匹配预测视图外的所有视图进行预测的，从而将表示推向其分配的原型。因此，该方法能够在推理过程中无需显式计算光流的情况下，学习到包含运动信息的更高效的视频嵌入。我们在最近邻视频检索和动作识别任务上获得了最先进的结果，在使用S3D骨干网络的情况下，在UCF101数据集上的Top-1准确率比之前最佳方法提高了3.2%（达到90.5%），而在使用R(2+1)D骨干网络的情况下，在UCF101数据集上的准确率提高了7.2%，在HMDB51数据集上的准确率提高了15.1%。
代码仓库
martinetoering/ViCC
官方
pytorch
GitHub 中提及
基准测试
| 基准 | 方法 | 指标 |
|---|---|---|
| self-supervised-action-recognition-on-hmdb51 | ViCC (R2+1D; RGB) | Frozen: false Pre-Training Dataset: UCF101 Top-1 Accuracy: 52.4 |
| self-supervised-action-recognition-on-hmdb51 | ViCC (S3D; R+F) | Frozen: false Pre-Training Dataset: UCF101 Top-1 Accuracy: 62.2 |
| self-supervised-action-recognition-on-hmdb51 | ViCC (R2+1D; R+F) | Frozen: false Pre-Training Dataset: UCF101 Top-1 Accuracy: 61.5 |
| self-supervised-action-recognition-on-hmdb51 | ViCC (S3D; RGB) | Frozen: true Pre-Training Dataset: UCF101 Top-1 Accuracy: 38.5 |
| self-supervised-action-recognition-on-hmdb51-1 | ViCC (R2+1D; RGB) | Pretraining Dataset: UCF101 Top-1 Accuracy: 52.4 |
| self-supervised-action-recognition-on-hmdb51-1 | ViCC (S3D; RGB)) | Pretraining Dataset: UCF101 Top-1 Accuracy: 47.9 |
| self-supervised-action-recognition-on-hmdb51-1 | ViCC (S3D; R+F) | Pretraining Dataset: UCF101 Top-1 Accuracy: 62.2 |
| self-supervised-action-recognition-on-ucf101 | ViCC (S3D; R+F) | 3-fold Accuracy: 90.5 Frozen: false Pre-Training Dataset: UCF101 |
| self-supervised-action-recognition-on-ucf101 | ViCC (S3D; RGB) | 3-fold Accuracy: 72.2 Frozen: true Pre-Training Dataset: UCF101 |
| self-supervised-action-recognition-on-ucf101 | ViCC (S3D; RGB) | 3-fold Accuracy: 88.8 Frozen: false Pre-Training Dataset: UCF101 |
| self-supervised-action-recognition-on-ucf101 | ViCC (R2+1D; RGB) | 3-fold Accuracy: 82.8 Frozen: false Pre-Training Dataset: UCF101 |
| self-supervised-action-recognition-on-ucf101 | ViCC (R2+1D; R+F) | 3-fold Accuracy: 88.8 Frozen: false Pre-Training Dataset: UCF101 |
| self-supervised-action-recognition-on-ucf101-1 | ViCC (R2+1D; RGB) | 3-fold Accuracy: 82.8 Pretrain: UCF101 |
| self-supervised-action-recognition-on-ucf101-1 | ViCC (R2+1D; R+F) | 3-fold Accuracy: 88.8 Pretrain: UCF101 |
| self-supervised-action-recognition-on-ucf101-1 | ViCC (S3D; R+F) | 3-fold Accuracy: 90.5 Pretrain: UCF101 |
| self-supervised-action-recognition-on-ucf101-1 | ViCC (S3D; RGB) | 3-fold Accuracy: 84.3 Pretrain: UCF101 |