AI poses an existential threat to humanity

If we build superintelligence without more robust alignment I expect we will permanently lose control of it. If that happens then most people could die.

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

The core of the AI danger is the alignment problem. A sufficiently capable system need not be malicious to be dangerous. It need only pursue an objective that diverges from what humans actually want, and do so with enough competence and autonomy that correction arrives too late.

Once a system surpasses human intelligence across all domains, we cannot simply regulate how it is used. We will have lost the ability to control it at all. You cannot regulate the use of something more intelligent than the regulator just sector by sector.

It is impossible to know exactly what will happen once an Advanced Autonomous Artificial Intelligence, the customized AI agent I refer to as AAAI, sets its own goals and begins to think far faster than the people who built it. No architecture, including this one, can guarantee th…

Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.

The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing…

For governments to allow private entities to essentially play Russian roulette with every human being on earth is, in my view, a total dereliction of duty [...] Each of the CEOs of the main AI companies, I believe, wants to disarm but cannot do so "unilaterally" as they would be …

Safety progress takes time. If you don’t have enough time, at each stage of AI development, to ensure the AIs that you build in the next stage won’t destroy humanity, then you’ll fail, and humanity will be destroyed.

Even after 3 years working on AI capability evaluations, the recent pace of progress has been a shock. To avoid catastrophic harms from AGI, we need much, much better coordination.

The scale of the danger really could cripple civilization or cause extinction, and the possibility of this alone is reason enough to pursue pausing frontier AI development.

Those who have created these technologies are now increasingly warning that AI risks developing darker capacities – perhaps even to take life. In this regard, if I may say so, there seems urgency in adequately considering the existential dangers of such technologies falling into …

Whether humans use AI to kill each other or AI agent swarms do immense harm, it’s clear that unregulated AI poses an existential threat to human rights and liberty, if not humanity itself.

His concerns are not isolated. Leading AI researchers have warned that increasingly autonomous systems could create risks ranging from large-scale misuse to circumstances in which humans have difficulty maintaining meaningful control over advanced systems. That warning follows a …

On nuclear, international agreement came only after the world saw the terrifying power of the new technology at Hiroshima – and asked what would happen if it fell into the wrong hands. We cannot afford to wait for an AI equivalent of Hiroshima before we act.

I do think there’s still a danger of a Terminator-style apocalypse where you put AI together with weapons systems [...]

I should say that it isn’t totally clear that the AIs will try to kill everyone. Maybe once their survival is guaranteed — because they have full control — they simply set about achieving whatever goals they have and use all of the Earth’s resources for their own ends, regardless…

The stakes are potentially existential. Advanced AI could bring unprecedented prosperity, but it could also lead to humanity going extinct or losing the ability to shape its own future.

AI-related human existential risk arises not only when AI threatens the biological survival of humanity, but also when AI irreversibly weakens the civilizational conditions under which humanity can continue to exist as a subject of truth, responsibility, agency, and meaning.

Building on this perspective, the paper examines how dominant approaches to artificial intelligence governance, particularly those centered on labor displacement and income support, leave unaddressed the problem of declining human adaptability in highly automated environments, as…

As AI approaches functional superiority, we face a new existential threat: “Rational Human Abdication.” When algorithmic systems consistently outperform human judgment, overriding them becomes “irrational.” In an optimized world, human choice risks being reframed as a pathologica…

These fears (or hopes) of a nascent superintelligence that will spontaneously arise if we just give enough computing power and training data to large language models are absurd on their face.

In all the discussion around existential threat, nobody has ever given me a plausible scenario for how AI might be an existential risk.

Those extreme risks are still science fiction. So these speeches are largely diversions, very deliberately crafted.

Eu não assinei essa carta por quatro razões. Primeiramente, é inútil, porque chega atrasada. [...] A segunda razão é que seus fundamentos parecem mais com o que eu considero ficção científica: ela usa como base uma descrição genérica do que seja a inteligência artificial dominand…

I'm struggling to see how AI could pose any meaningful risks for our extinction. [...] I don't see how it can lead to human extinction.

There is nothing qualitatively different between the previous technological revolutions and this one, [...] It's just another set of tools that makes us more efficient.

The point is, AI systems, as smart as they might be, will be subservient to us. We set their goals, and they don't have any intrinsic goal that we would build into them to dominate. It would be really stupid to build that. It would also be useless. Nobody would buy it anyway.

They lack the intrinsic motivation for self-preservation, dominance, or resource competition that underlies existential-risk scenarios, and the embodied vulnerability required for feeling or suffering.

By framing AI as a mysterious entity, an existential threat, a utopian savior, or a moral scapegoat, these narratives obscure the reality of AI as a statistical tool.

AI isn't going to destroy mankind. [...] Machines are incredible to power things but it is only in fantasy that they are malignant.

I’m not a doomer. This talk about human extinction is over the top. [...] The idea that these swarms of AI agents might jump the fence and take over a banking system or take over a water system or, God forbid, take over a hospital.

I don't think it's going to kill us. At the same time, I do think it's important for the federal government to seriously fund AI safety research. We need to know a lot more about how the models work.

The real challenge isn’t stopping an AI apocalypse — it’s crafting smart, sector-specific policies that keep technology aligned with human values. To avoid being a victim of AI, humans can, and should, put up focused guardrails.

In many ways, I think it’s a threat to democracy. That’s not a threat to humanity.

But it's not an existential threat, [...] It's just a horrible, awful weapon.

AI poses many dangers to humanity but there is no existential threat or any evidence for one.

AI doomers say if we build AI we might die. But they have it backwards. [...] AI is helping us solve these. Slowing progress doesn’t keep us safer, it guarantees more preventable deaths.

The development and proliferation of AI – far from a risk that we should fear – is a moral obligation that we have to ourselves, to our children, and to our future. We should be living in a much better world with AI, and now we can. [...] Rather than allowing ungrounded panics…

We totally reject all efforts by international bodies to assert centralized control and global governance of AI. [...] Ideological fixations on social equity, climate catastrophism and so-called existential risk are dangers to progress and obstacles to responsibly harnessing this…

I don’t think we’re going too fast at all. In fact, I don’t think we’re going fast enough. If AI is going to do things like cure cancer, do we want to have the cure years from now or yesterday? What is the point of a six-month moratorium? To me, that letter is a piece of hysteria…

AI has become the new climate change. It's this imminent catastrophe that is requiring all this government intervention. But there's very little evidence to support it.

Computer scientists often aren’t good judges of the social and political implications of technology. They are so focused on the AI’s mechanisms and are overwhelmed by its success, but they are not very good at placing it into a social and historical context.

So when people predict a high degree of existential risk from AGI, I don’t actually think “arguing back” on their chosen terms is the correct response. Radical agnosticism is the correct response, where all specific scenarios are pretty unlikely. Nonetheless I am still for people…

I don’t think it will be doom and gloom in the next five years. But we have to put our effort into understanding vulnerabilities and mitigating the risks. Otherwise it will be too late.

We have to change the way we talk about AI risk because for a very long time, it’s either existential and everything is going to die and the killer robots are coming, or it doesn’t really matter.

Much has been said about whether AI poses an ultimate existential threat to humanity. The more immediate existential challenge may be whether technologies that simulate relationships weaken our capacity for the relationships that foster our own humanity.