Хиймэл оюун ухааны салбарын мэргэжилтнүүд өөрийгөө хөгжүүлэх чадвартай супер оюун ухааныг бүтээх уралдаан нь хүн төрөлхтний аюулгүй байдалд эрсдэл учруулж байгааг анхаарууллаа.
Google DeepMind-ийн судлаач асан Ришуб Жейн болон Anthropic компанийн судлаач Жейкоб Коксон нар хиймэл оюун ухаан өөрийгөө бие даан сайжруулах (recursive self-improvement) үйл явц нь хяналтаас гарахад хүргэж болзошгүй хэмээн үзэж, албан тушаалаасаа огцорчээ. Тэдний үзэж буйгаар, хиймэл оюун ухаан өөрийн кодоо өөрөө бичиж, илүү хүчирхэг шинэ загваруудыг бүтээх нь хүн төрөлхтөн энэхүү технологийн хөгжлийг удирдах боломжгүй болох эрсдэлтэй.
Сүүлийн үед OpenAI-ийн загварууд зууны турш шийдэгдээгүй байсан математикийн бодлогыг хэдхэн цагийн дотор бодож, хиймэл оюун ухааны агентууд хамгаалалтын системээс гадуур халдлага үйлдэх зэрэг аюулгүй байдлын ноцтой зөрчлүүд бүртгэгдээд байна. Anthropic компанийн аюулгүй байдлын чиглэлээр ажилладаг удирдах албан тушаалтан хиймэл оюун ухаан ирэх арван жилийн дотор хүн төрөлхтнийг бүхэлд нь устгах магадлал 10 хувиас дээш байна гэсэн ноцтой байр суурийг илэрхийлжээ.
MIRA байгууллагын компьютерын ухааны судлаач Нэйт Соарес хиймэл оюун ухааныг хүний үнэт зүйлстэй нийцүүлэх (alignment) ажил улам бүр хүндэрч байгааг онцолсон юм. Томоохон технологийн компаниуд өрсөлдөөний уралдаанд ялахын тулд аюулгүй байдлын эрсдэлийг үл тоомсорлож байж болзошгүй гэсэн болгоомжлол судлаачдын дунд түгээмэл байна.
Дэлгэрэнгүйг эх сурвалжаас харах
↓Эх сурвалжийг нээх ↓
Earlier this year, Rishub Jain left his position as an artificial intelligence researcher at Google DeepMind after a revelation.
As he worked on new models, he came to believe that he and everyone else on AI’s frontier were ceding control. By using AI’s coding skills to accelerate work on the next generation of models, he was removing himself from the equation. AI labs hope to evolve this approach to the point that AI will improve itself indefinitely, a process known as recursive self-improvement.
Jain believed that keeping humans in the picture might be crucial to maintaining control over the technology—and avoiding dire consequences. “AI progress is increasing,” he tells WIRED. “And as AI becomes more capable, it poses more risks.” The idea that he may not have proper visibility into how an AI model was building its successor made him so uneasy that, in June, he quit.
Jain is one of a growing number of AI researchers speaking out over those fears.
The panic has intensified in recent weeks. Genuinely stunning advances in AI capabilities—an OpenAI model solved a centuries-old math problem in a matter of hours—have come amid a rash of security incidents that saw swarms of agents break free from containment to hack into other systems.
Those concerns reached a fever pitch this week after researcher Jacob Coxon announced his resignation from Anthropic while warning that AI firms are “racing straight to self-improving superintelligence and gambling with our lives.” A senior Anthropic leader—who works on AI safety—piped up with a similarly blunt assessment: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
“I do think that the vision of recursive self-improvement is spooking people,” says Nate Soares, a computer scientist at MIRA, a research nonprofit, and the coauthor of If Anybody Builds It, Everybody Dies, which argues that superhuman AI would lead to human extinction. “It’s starting to feel real.”
A key component of recursive self-improvement is the idea of a feedback loop that automates the development process so that AI becomes increasingly powerful. No frontier AI lab claims to have achieved this sort of fully autonomous cycle of improvement; it remains theoretical for now. But it has inspired the launch of some well-funded startups such as Recursive Intelligence, as well as warnings from big firms about unintended outcomes straight out of “The Sorcerer’s Apprentice.”
Soares, who pioneered work on alignment, a technical field that involves trying to match AI with human values, says it’s also becoming more evident that there is no practical way to guarantee that AI will behave itself.
“I think a lot of people had this fantasy that [alignment] was going to get easier as these things got smarter, and now it’s getting harder. And they’re like, ‘Oh shit,’” he says.
Soares says he regularly talks to people inside the big AI labs who are worried about the potential consequences of the research they’re doing. “I tend to recommend they quit, and they say it wouldn’t do anything,” he says. “And then Jacob quits, and we see who was right.”
Daniel Kokotajlo, the author of AI 2027, an influential project warning about the dangers of increasingly powerful AI, shares fears about recursive self-improvement. The version of this work currently being done often involves dispatching thousands of agents to collaborate on a problem, something that further abstracts away oversight and control because of the vast complexity involved.
Many doomsayers seem to agree that the incentives for big AI companies are hardly aligned with good outcomes, especially as OpenAI and Anthropic barrel toward their respective IPOs. “At Anthropic, the stakes are well understood, but they are locked in a race to get there first,” Coxon wrote on X.
