Abstract
We consider the visual sentiment task of mapping an image to an adjective noun pair (ANP) such as “cute baby”. To capture the two-factor structure of our ANP semantics as well as to overcome annotation noise and ambiguity, we propose a novel factorized CNN model which learns separate representations for adjectives and nouns but optimizes the classification performance over their product. Our experiments on the publicly available SentiBank dataset show that our model significantly outperforms not only independent ANP classifiers on unseen ANPs and on retrieving images of novel ANPs, but also image captioning models which capture word semantics from co-occurrence of natural text; the latter turn out to be surprisingly poor at capturing the sentiment evoked by pure visual experience. That is, our factorized ANP CNN not only trains better from noisy labels, generalizes better to new images, but can also expands the ANP vocabulary on its own.
Abstract (translated by Google)
我们考虑将图像映射到形容词名词对（ANP）（如“可爱的宝宝”）的视觉情感任务。为了捕捉ANP语义的双因素结构，克服注意噪声和模糊性，我们提出了一种新颖的因式分解CNN模型，它学习了形容词和名词的单独表示，但优化了其产品的分类性能。我们在公开可用的SentiBank数据集上进行的实验表明，我们的模型不仅能够显着地胜过看不见的ANP上的独立ANP分类器和检索新型ANPs的图像，而且能够捕捉自然文本同时出现的单词语义的图像字幕模型;后者在捕捉纯粹的视觉体验所引发的情绪方面竟然很差。也就是说，我们分解的ANP CNN不仅可以从嘈杂的标签中训练得更好，而且可以更好地推广到新的图像，而且还可以自行扩展ANP词汇。
URL
https://arxiv.org/abs/1511.06838