The Weapon that Mistook a School Bus for an Ostrich

Ethically and legally troubling aspects of autonomous weapons systems have been extensively discussed in both scholarly and diplomatic debates for over a decade. A consensus has emerged within the international community of states that all weapons systems, including autonomous ones, should be subject to “meaningful human control.” However, what such control amounts to operationally depends crucially on technological developments in the field of AI. Moreover, the intentional exploitation of AI systems’ vulnerabilities to perturb autonomous weapons’ behaviors was scarcely considered a serious possibility until recently. The maturing of techniques to launch so-called “adversarial attacks” to AI systems that are developed by means of machine learning methods is changing this picture and related issues of human control. Adversarial attacks generate deceptive inputs that are designed to cause mistakes in the predictions or classifications made by these AI systems. Here we explore challenges that AI’s emerging vulnerabilities to adversarial attacks raise for the debate over norms for AWS and for “meaningful human control” over AI-powered weapons systems. Normative debates are lagging behind technological developments. The militarization of AI must be regulated by ensuring that emerging challenges to meaningful human control are properly addressed by the international community of states and that adequate and effective regulations are developed, promulgated, and enforced.