A guide to coordination · 4 min read
Moloch: Why We Race Toward Outcomes Nobody Wants
Imagine a competition where everyone would be better off cooperating, but nobody feels they can afford to stop competing.
Even when everyone sees the problem, the incentives keep pushing them toward a worse outcome. That’s Moloch: a metaphor for coordination failures, where individually sensible choices add up to collective harm.
“I can’t afford to fall behind.”
“Neither can I.”
Everyone keeps racing.
Toward a result they would rather avoid.
01 / Recognize the trap
Nobody needs to be the villain.
Picture a concert. Someone stands to see better. People behind them stand too. Soon everyone is standing, with much the same view and more tired legs. Sitting down alone now means seeing less.
One person’s incentive
Stand up to get a better view.
Everyone’s response
Stand up to avoid losing out.
A possible way out
A seated section with an enforced rule.
Recognizing the problem doesn’t remove the incentive. Changing the rules can. Competition can be useful; the problem is when it rewards choices that undermine a shared goal.
Scott Alexander popularized this metaphor in Meditations on Moloch. Related ideas include collective-action problems, races to the bottom, and multipolar traps: many competing actors getting stuck in an outcome none can fix alone.
02 / Explore the incentives
Two AI labs. One shared future.
Imagine two labs developing increasingly capable AI. Both would prefer a world with strong safeguards. But each worries that taking extra time will let its competitor move ahead.
Choose a priority for each lab. Try all four combinations.
What happens in this scenario
Stronger safeguards. Neither lab races ahead.
- Lab A
- Time for evaluations and safeguards; no speed disadvantage.
- Lab B
- Time for evaluations and safeguards; no speed disadvantage.
- The shared outcome
- Both invest in safeguards that may reduce shared risks.
The pressure: But either lab may see an opportunity to move first by shortening its checks. Cooperation is fragile if neither can trust the other to keep it up.
A model, not a prediction. These are hypothetical labs, not claims about specific companies. We assume shorter checks can confer a competitive advantage. Safety and speed do not always trade off; safer systems can also be more useful and competitive. Real outcomes depend on many actors, technical choices, and rules.
03 / Change the incentives
How can we escape Moloch?
Awareness and good intentions help, but cooperation needs to be viable. These mechanisms could work together; none guarantees success.
- Transparency
-
Labs could disclose evaluation methods and safety thresholds.
What can be shared without exposing sensitive information?
- Credible commitments
-
Labs could agree to pause at a defined risk threshold if others do too.
What would make that promise costly to break?
- Verification
-
Independent evaluators could check whether agreed safeguards are in place.
Who gets access, and what can they reliably test?
- Institutions and agreements
-
Common standards and consequences for violations could reduce the penalty for acting responsibly.
Who sets the rules and makes them apply fairly?
- Collective decision-making
-
Public deliberation could reveal acceptable tradeoffs and conditions for cooperation.
Whose voices are missing, and how are they included?
There is room to disagree about which mechanisms are effective or appropriate. Rules can be hard to enforce, concentrate power, or delay benefits. A useful proposal must address those costs too.
04 / Find a starting point
From a shared concern to concrete proposals.
Cooperation requires understanding both what people support and what prevents them from supporting it. YouCongress brings sourced positions and participation together so people can inspect proposals, challenge interpretations, and explore where agreement is possible.
Share an opinion, vote, or delegate to people you trust. A vote can show a preference. Understanding its reasons, conditions, and unresolved objections takes further work. Popularity alone does not establish which proposal is best.
Our long-term ambition is to help humanity overcome coordination failures, starting with the AI development race. Discovering potential areas for cooperation is one contribution. Revealing agreement is not the same as solving a coordination problem. Incentives, trust, verification, and enforceable commitments may still be needed.
How we think this could lead to change →What would make cooperation possible?
Start with a proposal. Read the arguments, examine the sources, and decide where you stand. These are ideas to evaluate, not a package you need to endorse.
Slow down AI development
Would more time improve safety? At what cost?
Explore arguments & participateGive independent evaluators employee-like access to AI labs
What access would make independent scrutiny meaningful?
Explore arguments & participateBuild the capability for a coordinated global AI slowdown
What would need to be ready if a slowdown became necessary?
Explore arguments & participateSubject AI value alignment to public deliberation
Who should have a say in the values AI systems follow?
Explore arguments & participateFurther reading
-
Meditations on Moloch — Scott Alexander
The essay that popularized the metaphor, with many more examples of competitive traps. -
Beyond Markets and States — Elinor Ostrom
A Nobel lecture on how people organize to govern shared resources.