The Challenge of Value Alignment: from Fairer Algorithms to AI Safety

The Challenge of Value Alignment: from Fairer Algorithms to AI Safety asks how making AI follow human aims connects to a broader question: which values should technologies embody, and who should decide? Iason Gabriel and Vafa Ghazavi argue that technical control and social justification belong in the same research agenda. A system could faithfully serve one operator while imposing objectionable consequences on other people.^[1]

The chronology records the first arXiv submission on January 15, 2021. This account follows the January 18 v2 manuscript, labeled forthcoming in The Oxford Handbook of Digital Ethics; those preprint dates do not establish the later chapter’s publication date.^[1]

Technologies shape the choices people can make

The chapter begins with what the authors call the technology–value nexus. Technologies alter which outcomes are possible, easy or likely, so their design has moral consequences even when they do not make decisions themselves. On this account, an artifact need not announce a moral principle to favor some ways of living or distributions of authority. The authors therefore reject value neutrality for technologies operating at scale.^[1]

They connect this argument to value sensitive design: stakeholder analysis and citizen consultation can bring the perspectives of affected people into development. Timing matters. Early deliberation leaves more room to change a design, or decide against developing it, before technical and institutional dependencies narrow the options. These are reasons for a design approach, rather than evidence that consultation always produces just outcomes.^[1]

AI, they argue, extends rather than replaces this problem. Opacity can make decisions harder to explain, while learning and autonomy allow behavior that designers did not explicitly specify. The chapter also proposes a shift in ethical evaluation: beyond asking whether a system was designed with care, one can ask whether its behavior displays a stable disposition to care across circumstances. Its use of goals and dispositions as ways of interpreting agents does not require settling whether machines possess minds or bear moral responsibility.^[1]

Technical methods leave choices about value

The authors contrast top-down approaches, which encode an ethical framework, with bottom-up approaches, which learn from behavior or feedback. Explicit rules can be inspectable but conflict with one another; selecting the framework also risks imposing a designer’s moral beliefs. Learning avoids specifying every rule upfront, yet requires decisions about whose examples count, how data are selected and whether the learned preferences deserve endorsement.^[1]

Their discussion of the Moral Machine study illustrates the latter issue. They interpret its crowd-sourced judgments about autonomous-vehicle dilemmas as exposing variation and ethically troubling preferences, including judgments based on social status. The point is that observing what people prefer does not, by itself, justify using those preferences as an AI’s objective. This is the chapter’s philosophical reading of another study, rather than a new experiment conducted by Gabriel and Ghazavi.^[1]

The technical survey covers reward exploitation, harmful side effects, safe exploration and costly evaluation, then proposals for supervising much more capable systems. Its cleaning-robot illustration makes the evaluation problem concrete: checking only visible dirt supplies cheaper feedback but can miss dirt under surfaces. Better feedback and control address how an objective is pursued; they still leave open whether the objective is appropriate.^[1]

From one operator to affected communities

Instructions, intentions and preferences can each be mistaken or unethical. The authors consequently resist treating any of them as a sufficient endpoint for alignment. They also question the usual focus on one person and one agent: systems affecting many people need social value alignment, which engages with the plurality of reasonable moral views rather than assuming a single uncontested theory.^[1]

For technologies with global reach, they suggest a possible global overlapping consensus: principles that people holding different worldviews could endorse for different reasons. Agreement among a narrow set of experts would not meet this aspiration. They give two reasons for inclusive discussion. First, affected people have a right to help shape decisions with profound consequences for them, giving participation a role in legitimacy. Second, no participant possesses all relevant knowledge; including more perspectives can reveal concerns that a restricted group misses.^[1]

This is an interdisciplinary proposal, not a tested procedure for resolving disagreement. The chapter does not supply a complete institutional design or demonstrate that inclusive discussion yields consensus. Its distinctive contribution, compared with Gabriel’s alignment-target analysis, is to connect technical alignment with the older problem of values embedded in design. Read with Social Alignment and Society-in-the-Loop, it makes a practical demand on an alignment agenda: improving control over a system and justifying the interests it serves remain connected tasks, each requiring its own answer.^[1]