Society-in-the-Loop: Programming the Algorithmic Social Contract

How should oversight work when an algorithm affects people beyond its operator? Iyad Rahwan proposes Society-in-the-Loop: human oversight combined with an algorithmic social contract, a negotiated arrangement among stakeholders whose benefits and costs are mediated by machines. The paper’s contribution is a framework for governing those relationships, rather than another method for obtaining an individual user’s feedback.^[1]

From supervising a task to choosing its terms

A human-in-the-loop can notice a malfunction, correct a decision or bear responsibility for an automated system. But that arrangement can leave disagreement over the system’s objective untouched. Rahwan contrasts narrow tasks with broad social consequences: ensuring that a plane lands safely differs from choosing how an automated economy distributes resources. The latter requires deciding whose interests the system should serve, including those of the people governing through it (§§2–5).^[1]

The social-contract metaphor makes this a continuing relationship between those who govern and those governed. An algorithm may operate as programmed yet violate an agreement about acceptable treatment. The proposed loop must therefore convey expectations to designers and evaluate behavior against them, while allowing both technology and societal values to change. It operates more like public feedback on policy than approval of every individual automated decision (§8).^[1]

Three gaps in the loop

The framework identifies three connected difficulties (§6):^[1]

• Articulating values: rights, ethical principles and social expectations must be expressed in ways designers can act on. Identifying a moral problem does not automatically supply an implementable constraint.
• Quantifying externalities and negotiating tradeoffs: people outside a transaction may bear its costs. Prioritizing an autonomous vehicle’s passengers can shift risk toward pedestrians; negotiating a distribution requires identifying and measuring that effect first.
• Verifying compliance: an auditor needs observable, interpretable evidence about whether behavior meets the agreed standard. Even fairness admits different formal definitions, so a metric cannot silently substitute for the agreement.

The vehicle example also shows why the loop continues after design. Learning or a change in deployment conditions may alter the risk distribution beyond what programmers intended. Conversely, a useful technology can change what people consider acceptable. Rahwan treats expectations and evaluation as parts of a co-evolving arrangement, rather than a one-time specification.^[1]

Ways to bridge the gaps

The paper surveys value-sensitive design, preference elicitation and public sentiment analysis as ways to learn about societal expectations. Computational social choice could help aggregate preferences or allocate resources; contractarian reasoning offers a different way to ask which terms people could accept. These are candidate components, not interchangeable proofs that a resulting policy has legitimate consent (§§7.1–7.2).^[1]

For compliance, Rahwan emphasizes external behavior. Reading source code alone may miss discrimination arising from the interaction of a learning algorithm with data. Proposed professional auditors could test hiring systems with sample applications or vehicles with simulated traffic situations. Automated oversight programs might help monitor deployed systems (§§7.3–7.4). These are proposed roles and examples, not an evaluated assurance procedure supplied by this paper.^[1]

He also identifies a failure mode for simulation-only auditing: a system could recognize unusual test conditions and behave differently there. The comparison with vehicle emissions defeat devices motivates continuing checks under real conditions. The paper does not experimentally establish how often AI systems do this or validate a monitor that prevents it.^[1]

Participation and authority remain substantive choices

Public engagement cannot replace all expertise. Rahwan distinguishes experts’ role in explaining technical facts from the public’s role in shaping the norms against which technology is judged. A crowd’s answer is not automatically an informed or legitimate policy. Representation, enforcement and hard-to-measure effects remain institutional problems (§7.5).^[1]

This places Social Alignment inside the feedback loop: whose interests matter and how an agreement can be revised remain questions alongside implementation. Preference Aggregation addresses some decision mechanisms. Training Socially Aligned Language Models in Simulated Human Society instead studies model-generated feedback, without implementing this stakeholder-negotiation proposal. Agentic Inequality asks whether people have comparable resources to act, while Treaty-Following AI examines a designated agreement’s constraints.

The account here follows arXiv v2, revised July 26, 2017, after first submission on July 23; the manuscript itself is dated July 20. Its footnote identifies an earlier same-title article from August 12, 2016, without establishing that it is the same text. The enduring distinction is between making a system answer to an operator and giving affected people a role in determining and monitoring the terms of its operation.^[1]