Comment by Rohin Shah

I think the default prediction you should have for that is what the AI system learns to do is, “I’m going to take opportunities to reward hack, seek reward as much as possible that would allow me to get a high score after a week” or something like that, or whatever the time horizon actually was. And this is very different from the sort of ambitious misaligned goal that you need in order to motivate convergent instrumental subgoals to the point of, “Now my job is to take over the world.”