Comment by Samuel Hammond

LLM scaling laws also suggest the computing resources required to train large models are a reasonable proxy for model power and generality. Consistent with our argument for refining the definition of AI systems, the NTIA should thus consider defining a special regulatory threshold based on the computing cost needed to match or surpass the performance of GPT-4 across a robust set of benchmarks. Theoretical insights and hardware improvements that reduce the computing resources needed to match or surpass GPT-4 would require the threshold to be periodically updated. In the meantime, the handful of companies capable of training models that exceed this threshold should have to disclose their intent to do so and submit to external audits that cover both model alignment and operational security. Compute thresholds will eventually cease to be a useful proxy for model capability as training costs continue to fall. Nevertheless, the next five years are likely to be an inflection point in the race to build TAI. The NTIA should thus not shy away from developing a framework that is planned to obsolesce but which may still be useful for triaging AI accountability initiatives in the near term. AI Verified source (2023)