Abstract
Overoptimization failures in machine learning and artificial intelligence systems can involve specification gaming, reward hacking, fragility to distributional shifts, and Goodhart’s or Campbell’s law. These failure modes are an important challenge in building safe AI systems, and multi-agent systems have additional failure modes that are closely related. These failure modes for multi-agent systems are more complex, more problematic, and less well understood than the single-agent case. They are also already occurring, largely unnoticed. After motivating the discussion with examples from poker-playing AI, the paper explains why these failure modes are in some sense fundamental. Following this, the paper categorizes failure modes, provides definitions, and cites examples for each of: accidental steering, coordination failures, adversarial misalignment, input spoofing and filtering, and goal co-option or direct hacking. The paper then discusses ongoing and potential work on mitigation of these failure modes, and what to expect when these failures continue to proliferate.
Abstract (translated by Google)
URL
http://arxiv.org/abs/1810.10862