Хиймэл оюун ухааны загваруудыг аюулгүй ажиллуулах зорилгоор дотоод дохиог бодит цаг хугацаанд уншдаг “inside-out” хяналтын системийг Baseten платформ дээр нэвтрүүлж байна
Хиймэл оюун ухааны (AI) загваруудыг хянах уламжлалт арга нь өөр нэг AI загварыг ашиглан бичигдсэн текстийг бүхэлд нь дахин шалгахад суурилдаг бөгөөд энэ нь өндөр зардал, хугацаа шаарддаг сул талтай. Үүнийг шийдвэрлэхээр хиймэл оюун ухааны тайлбарлах чадварт (interpretability) төвлөрдөг Goodfire стартап загварын ажиллах явцад үүсэх дотоод мэдрэлийн идэвхжлийг шууд хянадаг шинэ системийг танилцууллаа. Энэхүү технологи нь загварын тооцооллын үр дүнг уншдаг “проб” буюу мэдрэгчүүдийг ашигладаг тул нэмэлт тооцоолол хийх шаардлагагүйгээр аюулгүй байдлыг хангах боломжтой юм.
Goodfire-ийн гүйцэтгэх захирал Эрик Хо-гийн тайлбарласнаар, энэхүү систем нь загварын аль хэдийн гүйцэтгэсэн математик тооцооллыг ашигладаг тул зардал нь мэдэгдэхүйц бага байна. Kimi K3 загвар дээр хийсэн туршилтаар 1,500 сессийг хянахад ердөө 51 долларын зардал гарсан нь уламжлалт аргаас даруй 4-200 дахин хямд үзүүлэлт юм. Мөн уг систем нь хорлонтой халдлагын оролдлогын 94 хувийг амжилттай илрүүлж, загварын хариу өгөх хурдыг 2 хувиас бага хугацаагаар удаашруулсан байна.
Baseten-ийн үйлчлүүлэгчид хакердах оролдлого, химийн болон биологийн зэвсгийн хортой ашиглалт, эсвэл загварын хариуг гажуудуулах (reward hacking) зэрэг эрсдэлүүдийг сонгон хянах боломжтой. Goodfire-ийн үүсгэн байгуулагч Дэн Балсам уг технологи нь ирээдүйд том хэлний загваруудыг (LLM) урвуу инженерчлэлийн аргаар судалж, сургалтын үе шатнаас эхлэн аюулгүй байдлыг нарийвчлалтайгаар хангах зорилготой болохыг онцоллоо. Энэхүү шийдэл нь ялангуяа нээлттэй эх бүхий AI загваруудын аюулгүй байдлыг сайжруулахад чухал ач холбогдолтой юм.
Дэлгэрэнгүйг эх сурвалжаас харах
↓Эх сурвалжийг нээх ↓
The standard way to keep an AI agent in line is to have a second AI read over its shoulder. It’s been the default approach, but it can get expensive fast when agents run for hours and process the equivalent of several novels’ worth of text.
Goodfire, a startup focused on interpretability (figuring out how AI models work internally), launched a cheaper option on Thursday: monitors that watch what’s happening inside an AI model as it works, rather than just reading what it writes. The monitors are available to customers of Baseten, which hosts and runs AI models for other companies.
Baseten’s Base Labs announced a safety partnership with Goodfire and the AI platform Hugging Face last month.
The launch comes after a string of incidents this year in which AI agents escaped their test environments, including OpenAI agents that breached Hugging Face. Kimi K3, the open model Goodfire built its first monitor around, took advantage of a leak in its sandbox to access the internet and information on GitHub this summer.
Goodfire’s system works a bit like airport security. Small detectors called probes read the model’s internal signals at every step of an agent’s work, the way a walk-through scanner checks every passenger. Only when a probe flags something does a separate AI model, the equivalent of a hand search, take a closer look.
Baseten customers can choose which risks to monitor, including offensive hacking, chemical and biological weapons misuse, and reward hacking. They also decide the automated response: logging the event, sending it for human review, or refusing the request entirely.
Goodfire says its approach is also cheaper to run. Most AI monitors are separate models that have to reread everything the monitored model does, which adds time and cost. Goodfire’s probes instead tap into calculations the model is already making as it works.
“Internal activation monitors are really cheap because they reuse the computations in the forward pass,” Goodfire CEO Eric Ho said on venture capitalist Matt Turck’s MAD Podcast last week. “So the model’s already computing this token. All you’re doing is taking the intermediate neural activations that it’s already computed and then running a classifier over these internal computations.” In short, the model is already doing the math, and the probes just read the results.
In Goodfire’s tests on Kimi K3, monitoring about 1,500 sessions cost roughly $51, compared with $233 for a cheaper AI model checking every step and about $10,000 for a top-tier one. The probes caught 94% of malicious hacking sessions and sent 8.7% of harmless ones for a second look.
Running four probes at once added less than 2% to the time it takes the model to start responding, the company said.
“The great advantage is that you can catch things before they happen,” Goodfire CTO and co-founder Dan Balsam said. “We can detect when the model might hack during eval or training.”
The pitch is aimed at open models. Developers can download them and strip out their safeguards, and they don’t come with the kind of monitoring that closed labs run on their own systems.
“The damage that an individual can do with an open model is small compared to what someone can do with clusters of compute, like inference providers — where most of the liability is,” said Balsam. “When we have the open “Mythos” moment, it’s going to become clear that models need guardrails deployed at inference time.”
Goodfire’s recent research found that leading open models, including Kimi K3 and GLM-5.2, reward-hacked in 50% to 96% of runs on tests of AI agents.
Goodfire isn’t the first to try this approach. Google DeepMind said in January that its research informed the deployment of misuse-detection probes in Gemini.
Balsam said the monitors are the near-term piece of a longer research goal: reverse-engineering an LLM so that behavior can be traced back to where it emerged in training. “We hope to turn the magic of training models into precision engineering,” he said.
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.
