Abstract
We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components. Understanding and fixing errors that arise in such integrative systems is difficult as failures can occur at multiple points in the execution workflow. Moreover, errors can propagate, become amplified or be suppressed, making blame assignment difficult. We propose a human-in-the-loop methodology which leverages human intellect for troubleshooting system failures. The approach simulates potential component fixes through human computation tasks and measures the expected improvements in the holistic behavior of the system. The method provides guidance to designers about how they can best improve the system. We demonstrate the effectiveness of the approach on an automated image captioning system that has been pressed into real-world use.
Abstract (translated by Google)
我们研究排除依赖于不同组件的分析流水线的机器学习系统的故障排除问题。理解和修复在这样的综合系统中出现的错误是困难的，因为在执行工作流程中的多个点可能发生错误。而且，错误会传播，被放大或被压制，使责任分配变得困难。我们提出了一种利用人的智慧对系统故障进行故障排除的人在回路（human-in-the-loop）方法。该方法通过人工计算任务模拟潜在的组件修复，并测量系统整体行为的预期改进。该方法为设计人员提供了如何最好地改进系统的指导。我们演示了这种方法在自动图像字幕系统上的有效性，该系统已经被用于实际应用。
URL
https://arxiv.org/abs/1611.08309