Dynamic value alignment through preference aggregation of multiple objectives

How can a controller respond when users disagree or their preferences change? Marcin Korecki, Damian Dailisan and Cesare Carissimo combine objective-specific Deep Q-learning with preference aggregation.^[1]

Method and contribution

The authors simulate a two-road signal-controlled junction and compare majority and proportional aggregation of preferences over stopping and waiting. Their experiments show a way to balance objectives that single-objective controllers trade against each other, rather than maximize every metric simultaneously.^[1]

Separate learning from collective choice

Each objective has its own Deep Q-Network: one minimizes stops, the other waiting duration. At decision time the controller applies a softmax to each network’s action values, combines the resulting scores using vote-derived weights, and chooses the highest-scoring action. Majority aggregation gives all weight to the winning objective; proportional aggregation uses its share of the votes. The weighted quantities are normalized action scores, not directly measured interpersonal utilities.^[1]

Only vehicles approaching the junction vote, at five-second decision intervals. Each vehicle keeps a fixed preference in the experiment, while the composition of the voting population changes. The tested adaptation is therefore to changing aggregate preferences, not learning how an individual revises their values. Modeling drivers who change preferences in response to experience is proposed as future work.^[1]

Evidence from the simulated junction

The study evaluates six traffic-demand settings with a population initially split equally between the two objectives, averaging 100 runs per setting. Its stops-only controller learns to leave one direction permanently red: a low stop count conceals very long waits. The multi-objective controllers avoid that solution in this environment, while compromising between stops and waits.^[1]

Proportional voting does not win every comparison. Against majority voting, the authors report equal performance in the medium balanced setting and a stops-versus-waits trade-off in the high balanced setting. Against a controller trained on a Cobb–Douglas combination of rewards, proportional voting has worse stops and waits in the low-demand settings. These comparisons support a setting-dependent compromise, not universal superiority of the aggregation rule.^[1]

Alignment relevance and limits

Preference Aggregation makes Social Alignment operational in a bounded control setting. It connects the plural-values question in The Challenge of Value Alignment: from Fairer Algorithms to AI Safety to a concrete decision mechanism.

The study assumes centralized control, truthful preference reports and a trusted aggregation rule. Designers supply the objectives available for voting; democratic selection of those objectives is suggested rather than tested. Its two-option setup does not resolve strategic voting with larger choice sets. Simulation results do not establish a legitimate rule for society, safety in real traffic, or general alignment of advanced agents.^[1]

Historical context

The first arXiv version of Dynamic value alignment through preference aggregation of multiple objectives was submitted on October 9, 2023. The PDF itself is dated October 10; this milestone records the repository submission.^[1]

Marcin Korecki and coauthors link a multi-objective learning approach with Preference Aggregation.

Evidence scope

The original arXiv v1 introduction, method, results, assumptions and limitations were inspected, including the rendered aggregation equations on PDF page 6. The evidence is a simplified simulation, not a real traffic deployment or a measurement of public legitimacy. Code, replication, cited studies and a complete revision comparison were not audited.^[1]