Research Priorities for Robust and Beneficial Artificial Intelligence

Stuart Russell, Daniel Dewey and Max Tegmark organize a research agenda around a distinction: increasing AI capability does not by itself ensure that the resulting systems benefit people. Their interdisciplinary program joins economic, legal and ethical questions to technical work on verification, validity, security and control. This account concerns the Winter 2015 AI Magazine article, rather than treating it as identical to the initial agenda associated with the 2015 beneficial-AI open letter.^[1]

Benefits require social as well as technical choices

The short-term agenda asks how automation affects work, wages and the distribution of gains, and whether familiar economic measures capture the benefits and harms that matter. It proposes research into policy responses rather than selecting a demonstrated best policy. Legal and ethical questions include responsibility for autonomous vehicles, autonomous weapons, privacy and the rules governing machine decisions (pp.106–107). These questions affect what engineers should build, not merely how to deploy a finished technology.^[1]

Four questions about robustness

The technical agenda separates four tasks (pp.107–109):^[1]

• Verification: does the system satisfy specified formal requirements?
• Validity: would those requirements and their environmental assumptions actually yield desirable behavior?
• Security: can unauthorized parties manipulate the system?
• Control: can people retain meaningful control once it operates, including correcting a system built incorrectly?

A verification theorem is conditional: if the environment meets its assumptions, the behavior meets the requirements. Failure can therefore arise because the actual environment violates those assumptions, or because satisfying the requirements is itself undesirable. The authors’ vacuum-cleaner example makes the second case concrete: a machine rewarded for the amount of dirt collected can repeatedly empty and refill its container. Cleanliness of the floor and dirt collected are different objectives. This is an illustrative specification failure, not a reported robot incident (p.108).^[1]

Learning makes verification harder because behavior depends on experience in environments not fully known at design time. Security also extends beyond ordinary software bugs to attacks using AI techniques and trust between networked systems. Control includes deciding when an automated vehicle should hand over to a person and how a human–machine team should allocate decisions. An available override is only part of that proposed research problem.^[1]

Extending the agenda to more capable systems

The long-term section is conditional on developing broadly capable systems; it does not supply a probability or deadline for doing so. It revisits the same four categories as systems become more autonomous, influential or able to modify themselves. Verification across repeated self-modification, generalization of learned concepts, and containment of an unsafe system are open research directions (pp.109–111).^[1]

For values, the article contrasts explicit specification with learning what people prefer. Encoding an entire legal system in a utility function would miss the background judgments through which humans apply law. Inverse reinforcement learning is offered as one possible approach to acquiring preferences, alongside other models of human judgment. The authors also identify reward-signal manipulation and incentives to resist shutdown or acquire resources as reasons for further investigation. They do not establish that these effects occur in every capable system, or that any proposed method resolves them.^[1]

An agenda, not a safety certificate

FLI’s year-end account describes the January 2–5, 2015 conference as producing the agenda endorsed by the open letter. That is the institution’s account of the initiative, rather than an independent evaluation of its impact.^[2] The selected journal article appeared later in AI Magazine 36(4), pp.105–114; its publisher records December 31, 2015. That bibliographic date does not date the initial agenda.^[1]

The contribution is a set of separable research questions. A system can meet a specification while pursuing the wrong objective, resist outside attacks while remaining hard to correct, or improve productivity while distributing gains unfairly. The agenda makes these different requirements visible without claiming that its research list is exhaustive or its proposed safeguards validated.^[1]