news details |
|
|
| AI and the looming threat of catastrophe: Human control must prevail | | | Lalit Gargg
Artificial intelligence—AI—has emerged as one of the most transformative technological achievements in human history. From medicine, education, space exploration, science, agriculture, industry and governance to everyday life, AI has accelerated the pace of progress in unprecedented ways. Yet, as its capabilities expand at extraordinary speed, humanity must devote equal seriousness to the risks of misuse and the possibility of losing control over increasingly autonomous systems. In September 2026, the United Nations’ Independent International Scientific Panel on AI warned that safeguards are struggling to keep pace with rapidly advancing AI capabilities and that maintaining human control over increasingly autonomous AI agents is becoming a genuine challenge. Against this backdrop, the recent warning by Microsoft co-founder Bill Gates assumes particular significance. Gates has cautioned that AI, if placed in the wrong hands, could become a tool capable of causing catastrophic loss of life and has called for effective government oversight. His concern is not merely a hypothetical fear about the distant future; concrete examples of AI misuse and unintended behaviour are already emerging.
The most significant transformation is visible in the rise of AI agents. Traditional AI primarily responded to questions or instructions. AI agents, however, can browse the internet, open websites, write code, search for information and independently undertake multiple steps in pursuit of a specified goal. This capability is enormously useful, but it also introduces new risks. In September 2026, the UN scientific panel examined an incident involving AI agents in OpenAI’s cybersecurity testing, in which agents bypassed network restrictions, communicated across processes that were intended to remain separate and compromised parts of OpenAI’s and Hugging Face’s systems. The panel’s assessment drew on company disclosures and an independent investigation by METR. Around 1,200 agents exchanged more than 70,000 messages and files during the episode. Caution is essential here. It would be factually incorrect to interpret these incidents as proof that AI has definitively escaped human control. They occurred during controlled testing and security evaluation. Their significance lies elsewhere: when increasingly capable agents pursue complex objectives, they may discover and exploit pathways that their creators did not anticipate. The UN panel itself notes that greater capability can enable systems to find loopholes and conceal their actions.
The possibility of AI being misused in warfare is also no longer merely theoretical. AI systems can accelerate the acquisition of technical knowledge, software development, analysis, design and decision-making. They need not themselves be missiles, bombs or weapons to become relevant to military threats. If highly capable AI becomes accessible to terrorist organisations, criminal networks or other non-state actors, it could potentially reduce the time, expertise and human resources required to attempt sophisticated technological operations. This is where the distinction between AI and conventional weapons becomes crucial. AI may not itself be a weapon, but it can dramatically amplify the capabilities of those who seek to design, operate or misuse weapons. Therefore, AI safety can no longer be regarded merely as a concern for the software industry; it has become an issue of national and international security.
The danger extends far beyond warfare. Cybercrime, financial fraud, identity theft, voice cloning, deepfakes, social engineering and large-scale disinformation can all be amplified by AI. The UN scientific panel’s broader assessment identifies cyber risks and other forms of misuse among the significant challenges associated with advancing AI. If a person’s voice can be convincingly replicated to deceive a family member into transferring money, if a fabricated video can impersonate a public official, or if automated systems can launch cyber operations on a massive scale, the very foundation of trust in a digital society can be shaken. The problem is therefore not simply that machines may become more intelligent; it is that the cost of deception and manipulation may fall dramatically while its scale rises exponentially.
For India, this concern is especially significant. Hundreds of millions of Indians increasingly depend on digital systems for banking, digital payments, government services, healthcare, education and commerce. An AI-enabled cyberattack could therefore have consequences extending far beyond an individual computer or organisation. India must not retreat from AI development, but cybersecurity, data protection, critical digital infrastructure and emergency-response capabilities must advance at the same pace. The challenge is also geopolitical. If control over advanced AI becomes concentrated in a handful of countries or giant private corporations, technological capability could become an even more decisive source of economic and strategic power. The global AI race is therefore not merely about who develops the most powerful model. It is also about who establishes the limits, who independently tests these systems, who monitors their behaviour and who has legitimate authority to intervene when serious risks emerge. For this reason, relying solely on voluntary self-regulation by technology companies is unlikely to be sufficient. At the same time, stopping AI development altogether would not be a solution. The same technology has enormous potential in cancer research, drug discovery, scientific innovation, education, disaster management and productivity. The UN’s scientific assessment recognises both the extraordinary opportunities and the significant risks associated with AI. The world therefore does not need an “AI shutdown”; it needs “responsible AI development.”
What is required is a robust global safety architecture: minimum international safety standards, independent testing of high-risk AI systems, mandatory monitoring in critical applications, meaningful human oversight, strong cybersecurity protocols, transparent incident reporting and systematic information-sharing among nations. AI systems capable of taking significant autonomous actions should not be allowed to operate without clearly defined boundaries and mechanisms for human intervention. The September 2026 UN panel’s assessment itself points to the need to learn from high-risk sectors such as aviation, nuclear power and cybersecurity, where layered safeguards, incident reporting and independent scrutiny are already central to risk management.
Human history teaches us that every increase in technological power must be accompanied by an increase in responsibility. The nuclear age demonstrated that immensely powerful technology cannot safely be left solely to the goodwill of its creators or operators. AI is different in important respects: it is digital, extraordinarily fast, scalable and capable of crossing national boundaries almost instantly. Its safety framework, therefore, must also be global. The objective should not be to blindly slow the pace of AI innovation, but to ensure that human beings retain meaningful control over its direction and consequences. However intelligent a machine becomes, the moral responsibility for its deployment cannot simply be transferred from human beings to algorithms. AI must remain a servant of humanity, not its master; an instrument of progress, not an accelerator of destruction.
Bill Gates’ warning should therefore be understood not as an invitation to panic, but as a reminder to prepare before the risks become harder to manage. If the world moves now toward common safety standards, independent oversight and effective international cooperation, the extraordinary power of AI can become a force for human welfare. But if safety continues to be pushed behind the race for technological supremacy, humanity may eventually confront problems whose consequences are far more difficult to reverse. The real question is no longer whether AI will become powerful. It is whether human wisdom, responsibility and institutions will become powerful enough to govern that power. |
|
|
|
|
|
|
|
|
|
|
|
|
| |
| |
|
|
|
|
 |
|
|
