Abstract
With the recent popularity of animated GIFs on social media, there is need for ways to index them with rich metadata. To advance research on animated GIF understanding, we collected a new dataset, Tumblr GIF (TGIF), with 100K animated GIFs from Tumblr and 120K natural language descriptions obtained via crowdsourcing. The motivation for this work is to develop a testbed for image sequence description systems, where the task is to generate natural language descriptions for animated GIFs or video clips. To ensure a high quality dataset, we developed a series of novel quality controls to validate free-form text input from crowdworkers. We show that there is unambiguous association between visual content and natural language descriptions in our dataset, making it an ideal benchmark for the visual content captioning task. We perform extensive statistical analyses to compare our dataset to existing image and video description datasets. Next, we provide baseline results on the animated GIF description task, using three representative techniques: nearest neighbor, statistical machine translation, and recurrent neural networks. Finally, we show that models fine-tuned from our animated GIF description dataset can be helpful for automatic movie description.
Abstract (translated by Google)
随着动画GIF最近在社交媒体上的流行，需要使用丰富的元数据来对它们进行索引。为了促进对动画GIF理解的研究，我们收集了一个新的数据集Tumblr GIF（TGIF），其中包含来自Tumblr的100K动画GIF和通过众包获得的120K自然语言描述。这项工作的动机是开发图像序列描述系统的测试平台，其任务是为动画GIF或视频剪辑生成自然语言描述。为了确保高质量的数据集，我们开发了一系列新颖的质量控制，以验证众包工的自由形式文本输入。我们发现，在我们的数据集中，视觉内容和自然语言描述之间存在明确的联系，使其成为视觉内容字幕任务的理想基准。我们进行大量的统计分析，将我们的数据集与现有的图像和视频描述数据集进行比较。接下来，我们使用三种代表性技术：最近邻，统计机器翻译和递归神经网络，在动画GIF描述任务上提供基线结果。最后，我们展示了从我们的动画GIF描述数据集进行微调的模型可以有助于自动电影描述。
URL
https://arxiv.org/abs/1604.02748