OpenAI-ийн Astra загварт ашигласан “recurrent depth” буюу давтагдах гүнзгийрүүлсэн аргачлал нь хиймэл оюун ухааны шийдвэр гаргах үйл явцыг хянах боломжийг хязгаарлаж болзошгүй байна.
OpenAI компани Astra загвартаа “recurrent depth” буюу “opaque recurrence” гэх шинэ аргачлалыг ашиглаж эхэлсэн нь технологийн салбарт болгоомжлол төрүүлээд байна. Энэхүү технологи нь хиймэл оюун ухааны сэтгэн бодох дарааллыг уламжлалт шугаман хэлбэрээс гаргаж, нэг асуултыг олон дахин давтан боловсруулах замаар ажилладаг байна. Ингэснээр загварын шийдвэр гаргах явцыг хянах, задлан шинжлэхэд улам бүр төвөгтэй болж байгааг мэргэжилтнүүд онцолжээ.
Хиймэл оюун ухааны аюулгүй байдлын шинжээч Бак Шлегерис болон Зви Мошовиц нар энэхүү аргачлал нь “Chain of Thought” буюу сэтгэн бодох дарааллыг хянах боломжийг үгүй хийж, салбарын аюулгүй байдлын стандартыг сулруулж болзошгүй гэж үзэж байна. Ялангуяа хиймэл оюун ухаан нь бүрэн буюу ихэнхдээ “latent space” буюу далд төлөвт шилжиж сэтгэх нь аюулгүй байдлын эрсдэлийг нэмэгдүүлнэ гэж судлаач Раян Гринблатт анхааруулжээ.
OpenAI-ийн зүгээс Astra загварт энэ аргыг хязгаарлагдмал хүрээнд ашиглаж байгаа бөгөөд сэтгэн бодох дарааллыг бүрэн хянах боломжтой хэвээр байна гэж мэдэгдлээ. Тус компанийн ерөнхий эрдэмтэн Якуб Пачоцкийн тэмдэглэснээр, хиймэл оюун ухааны үйл ажиллагааг ойлгомжтой, хяналттай байлгах нь компанийн судалгааны гол зорилго юм.
Гэсэн хэдий ч энэхүү технологийн хэрэглээ өсөхийн хэрээр хяналт тавих боломж улам багасах вий гэсэн болгоомжлол байсаар байна. Мэдээллийн эх сурвалжийн мэдээлснээр, Anthropic болон Google DeepMind компаниуд ч энэхүү аргын талаар хэлэлцэж эхлээд байгаа аж.
Дэлгэрэнгүйг эх сурвалжаас харах
↓Эх сурвалжийг нээх ↓
OpenAI’s new Astra model will use a reasoning technique called “recurrent depth” that allows it to operate outside of the sequential thinking that characterizes most reasoning models, the The Information reported on Tuesday. This technique, also called “opaque recurrence,” will likely make the model’s chain of thought more difficult to monitor — and that has AI safety experts rattled.
While Astra’s use of the technique is reportedly limited, its emergence has still raised significant concerns among AI safety experts.
“I am extremely concerned by the reporting that Astra uses opaque recurrence,” wrote Redwood CEO Buck Shlegeris in a post after the news broke. “I don’t know whether Astra is much less CoT monitorable than previous models. But if OpenAI pushes this technique further, they’ll have the option to massively increase the recurrence and totally destroys CoT monitorability.”
Longtime AI safety advocate Zvi Mowshowitz also weighed in and wrote that laws might be necessary to prevent a “race to the bottom” among AI labs.
“The technique is playing with fire, risking a taboo that OpenAI and Anthropic have fought to establish that we work hard to maintain Chain of Thought faithfulness and monitorability for as long as we can,” Mowshowitz wrote. “More intensive use of such techniques would probably damage monitorability.”
Under normal circumstances, a reasoning model’s chain of thought provides the sequential steps taken by the model as it attempts to solve a problem. While the representation is imperfect, it still serves as a valuable tool for monitoring misbehavior or misalignment. In the case of OpenAI’s recent rogue agent activity, chain-of-thought records were an important tool in teasing out why agents behaved the way they did.
In opaque recurrence, the model takes a less linear approach, processing the same query several times in a loop. The result leaves fewer legible traces, effectively side-stepping a conventional chain-of-thought record.
Crucially, Astra’s use of the technique appears to be limited. The model’s chain of thought is still expected to be legible, and the company pushed back against any suggestion that it would shift to “neuralese.” OpenAI has already announced plans for extensive chain-of-thought monitoring systems as part of its forward-looking safety plans.
In a post on X, OpenAI chief scientist Jakub Pachocki emphasized the lab’s commitment to legible chains of thought. “OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models,” Pachocki wrote. “It’s a core goal of our current research program.
All AI models do some quantity of opaque reasoning, and few researchers take chain-of-thought logs as a direct representation of a model’s reasoning. Still, those caveats don’t dispel the concern that opaque recurrence may make AI reasoning harder to monitor, particularly as it grows in use across different models. In a follow-up report Wednesday morning, The Information reported that both Anthropic and Google DeepMind were already discussing the technique.
In a post responding to the news, Redwood Research chief scientist Ryan Greenblatt said opaque reasoning could easily scale faster than conventional chain-of-thought reasoning, effectively removing all reasoning from visible channels.
“My biggest concern is that a natural progression from here would involve scaling up the opaque reasoning to the point where the model reasons entirely or almost entirely in latent space,” Greenblatt wrote. “I hope it isn’t too late to avoid the most concerning architectures and that OpenAI will stop here.”
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.
