Comment by Geoffrey Irving

If you’re doing a very dangerous thing, you should just slow down. The AI companies are overplaying the extent to which this is a pure coordination problem. They could just stop unilaterally.

I think that could work. I don’t think we have a strong argument that the pragmatic mixture of approaches will get all the way there, but it just seems very dicey, and our understanding of the dynamics involved is very weak.

Yeah. I think the main uncertainty here is: are the models going to be good not just at verifiable tasks with clean rewards, but also fuzzier things — intuition, fuzzy planning, this kind of thing? I think people are overweighting the probability tha...