Preference Aggregation

Preference aggregation combines several people’s preferences into a collective decision or objective. It is needed when an AI system affects people who disagree: learning each person’s preferences does not yet specify how competing claims should count. Representation, the available choices and the aggregation rule each shape the result.

Selecting whose preferences count

Feedback collection already selects a reference group. InstructGPT identifies contractors who rank outputs, researchers who write instructions, and customers whose prompts shape training as distinct influences. Most comparison examples have one contractor’s label. The authors warn that customers are not representative of all affected people, customers and end users may have conflicting interests, and averaging labeler preferences may underrepresent a minority group.^[1]

Collecting more rankings can improve evidence about disagreement, but does not decide whether people should receive equal weight, whether affected groups should receive greater weight on particular questions, or whether some constraints should stand outside a vote. Alignment Feedback separates availability, reliability and representation; Social Alignment connects these choices to legitimacy and accountability.

Selecting objectives and a rule

Dynamic value alignment through preference aggregation of multiple objectives supplies a concrete mechanism in a simplified traffic-junction simulation. Vehicles vote for one of two system-wide objectives: fewer stops or shorter waits. Only vehicles on incoming lanes vote at a decision step; downstream vehicles do not. Each vehicle’s preference remains fixed, while the composition of the voting population changes.^[2]

Each objective has a trained Deep Q-Network. The controller applies softmax normalization to each network’s action values, combines the resulting scores using vote weights, and selects the highest-scoring action. Majority aggregation gives all weight to the winning objective; proportional aggregation uses its fraction of eligible votes. Designers supply the two objectives and trained models. Selecting the objectives democratically is proposed, rather than tested.^[2]

Vote shares and the selected action

The traffic method first turns votes over objectives into weights, then uses those weights to combine normalized action scores. Proportional weighting does not mean taking each action in proportion to its vote share. The controller chooses the action with the highest combined score.^[2]

Consider a hypothetical decision with four voters: three choose fewer stops and one chooses less waiting. Suppose the already-normalized network scores are:

Quantity	Action X / Action Y
Fewer-stops score	0.60 / 0.40
Less-waiting score	0.10 / 0.90
Majority combination	0.60 / 0.40 → X
Proportional combination	0.475 / 0.525 → Y

Majority weighting is 1 for fewer stops and 0 for less waiting. Proportional weighting is 0.75 and 0.25: X receives 0.75 × 0.60 + 0.25 × 0.10 = 0.475; Y receives 0.75 × 0.40 + 0.25 × 0.90 = 0.525. The minority-supported objective can change the selected action because the normalized action-score differences matter as well as vote counts. These numbers are a teaching example of the rule, not measured traffic data or another experiment.

The scores concern objective-specific action predictions. They do not measure how much each person benefits or suffers. Neither a 75-percent vote share nor the selected action establishes a 75-percent share of welfare, an acceptable worst-case wait or agreement from people excluded from the vote. Those would require additional definitions and evidence. The original study supplies a bounded control mechanism under explicit assumptions; it does not settle those social questions.^[2]

What the mechanism assumes

The traffic study assumes centralized control, a socially trusted aggregation rule and truthful preference reports. Those assumptions are inputs to its evaluation, not findings that real road users would endorse the controller or always report honestly. Its experiment tests two objectives with an overall half-and-half preference split; voting proportions vary as vehicles enter and leave the approach. Adding objectives requires corresponding trained models and raises further questions about comparability and strategic reporting.^[2]

The normalized action scores are not interpersonal utility measurements: softmax makes the network outputs combinable, but does not show that a unit of one person’s benefit equals a unit of another’s. The simulation’s improvements and tradeoffs concern specified traffic metrics, not a proof that proportional aggregation is always preferable or that the procedure is socially legitimate.^[2]

Fairness Tradeoffs helps distinguish equal vote weights from equal errors or equal burdens. Meaningful Human Control asks who can understand and influence the resulting decisions. A collective-choice mechanism becomes accountable only when its participants, available objectives, weighting rule and means of contesting decisions are justified alongside its measured performance.