Abstract
Recently, deep learning approach, especially deep Convolutional Neural Networks (ConvNets), have achieved overwhelming accuracy with fast processing speed for image classification. Incorporating temporal structure with deep ConvNets for video representation becomes a fundamental problem for video content analysis. In this paper, we propose a new approach, namely Hierarchical Recurrent Neural Encoder (HRNE), to exploit temporal information of videos. Compared to recent video representation inference approaches, this paper makes the following three contributions. First, our HRNE is able to efficiently exploit video temporal structure in a longer range by reducing the length of input information flow, and compositing multiple consecutive inputs at a higher level. Second, computation operations are significantly lessened while attaining more non-linearity. Third, HRNE is able to uncover temporal transitions between frame chunks with different granularities, i.e., it can model the temporal transitions between frames as well as the transitions between segments. We apply the new method to video captioning where temporal information plays a crucial role. Experiments demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks. Notably, even using a single network with only RGB stream as input, HRNE beats all the recent systems which combine multiple inputs, such as RGB ConvNet plus 3D ConvNet.
Abstract (translated by Google)
最近，深度学习方法，特别是深度卷积神经网络（ConvNets）已经实现了压倒性的准确性，图像分类处理速度快。将时间结构与深度ConvNets结合以用于视频表示成为视频内容分析的基本问题。在本文中，我们提出了一种新的方法，即分层递归神经编码器（HRNE），以利用视频的时间信息。与最近的视频表示推理方法相比，本文做出以下三点贡献。首先，我们的HRNE能够通过缩短输入信息流的长度，在更高的层次上合成多个连续的输入，从而有效利用更长时间范围内的视频时间结构。其次，计算操作明显减少，同时获得更多的非线性。第三，HRNE能够揭示不同粒度的帧块之间的时间转换，即它可以模拟帧之间的时间转换以及段之间的转换。我们将这种新的方法应用于时间信息起关键作用的视频字幕中。实验表明，我们的方法胜过了视频字幕基准的最新技术。值得注意的是，即使使用一个只有RGB流作为输入的单一网络，HRNE也能击败所有结合了多种输入的系统，比如RGB ConvNet plus 3D ConvNet。
URL
https://arxiv.org/abs/1511.03476