Хиймэл оюун ухааны салбарын өрсөлдөөнийг хурцалж буй “загварын нэрэлт” технологи

When you set out to master a new skill, it pays to have a mentor—someone who has already navigated the terrain and, as a result of that hard-earned experience, can provide you with a rough roadmap for you to follow. Learning doesn’t happen in a vacuum: “If I have seen further,” as Isaac Newton put it, “it is by standing on the shoulders of giants.”

The same is true for artificial intelligence.

When we think about AI, we tend to think about so-called flagship models—digital behemoths like ChatGPT, Claude, and Gemini, which require billions of dollars and much of the content on the internet to function. These giants were built from the ground up, so to speak (although there are plenty of artists and news publishers whose work was surreptitiously scraped during the models’ training process who would probably say that’s an oversimplification).

But that isn’t the only—or even the most efficient—way to develop AI. Rather than building from the ground up, you can harness a more powerful model and let it do the heavy lifting while your new, smaller model reaps the benefits. That, in a nutshell, is the basic process behind what’s known in the tech industry as “model distillation.” And it’s becoming an increasingly divisive issue as the AI race between the U.S. and China intensifies.

What is model distillation?

Distillation writ large can be thought of as an alchemical process extracting some small, valuable kernel from a lot of otherwise useless material. To distill alcohol, for example, you need to boil a fermented liquid, capture the ethanol vapor, and then cool it; in that way, the pure alcohol can be separated, extracted, and commodified.

Similarly, AI model distillation—as it’s known in tech circles—seeks to extract valuable training data from a larger, pretrained model. Ordinarily, a model needs to embark on a long process of trial-and-error to build its understanding of facts that would be self-evident to a human child, like how to distinguish between a cat and a dog, the fact that 2+2 will always equal four, or that the word “tree” represents one of those tall leafy things found outside. This learning phase, as it’s most commonly carried out, is extremely computationally expensive. (Companies have been willing to spend the GDP of a small country on building models in this way because it’s widely believed to be the surest path to artificial general intelligence, though that’s still very much a multibillion-dollar open question.)

Model distillation, which is also known as knowledge distillation, skips that critical learning phase by essentially mimicking a flagship model’s behavior. The new model is still, over time, improving its ability to achieve particular goals—it couldn’t truly be called AI if it wasn’t—but rather than going through the arduous legwork of acquiring a “ground truth” knowledge of the world, it instead defers to the hard-won knowledge of the flagship model, using that to guide its own behavior.

By analogy, you can think of a flagship model as a native New Yorker, whose intimate knowledge of the area is borne from countless wanderings and wrong turns over the course of her life. A distilled model is like a friend visiting her in the city for the first time, shadowing her as she strolls around the city and drawing a crude sketch of a map along the way. Or think of the invention of the airplane. The Wright Brothers flew their first flying machine in 1903 following many failed attempts, and a long history of other innovators who tried and failed to do the same thing. But after they succeeded, other innovators could build off their model. The really difficult learning process was done, and the future lay open for others to iterate and improve on their success.

How does it work?

Model distillation functions by pairing a smaller “student” model with a flagship “teacher” model. Over the course of many interactions between the two, the former gradually learns to more accurately predict what the latter’s responses will be, thereby modeling itself after the teacher. Again, the student model is still engaged in a learning process, but it’s a much cheaper and more streamlined one than that which had been required from the teacher model.

But like a good human teacher, the teacher model doesn’t just answer the student’s questions in black-and-white, yes-or-no terms: it fosters its pupil’s ability to reason.

A teacher model may have learned, for example, to identify images of motorcycles by examining millions of such images scraped from social media and other online databases, and with the assistance of humans filling out millions of CAPTCHAs, which help it distinguish motorcycles from similar-looking objects. When trying to develop its own recognition capabilities, a student model won’t just watch the teacher model assign “Yes” or “No” labels to pictures of motorcycles; rather, it watches as the teacher reasons through the deductive process of what’s being shown in a given image. The teacher model might conclude, for example, that the object in the image has an 85% chance of being a motorcycle, a 12% chance of being an e-bike, a 2.7% chance of being a donkey, and a 0.3% chance of being Mr. Snuffleupagus. These “soft labels,” as they’re known, help the student model build its own reasoning and statistical prediction skills.

Why all the hullabaloo?

Most experts will tell you that there’s nothing necessarily immoral about model distillation per se. It’s not stealing, just building upon the innovations of others—which, as Newton’s quote suggests, is the very foundation of true scientific progress. In fact, a case could be made that distillation is a net good, since it democratizes the ability to build powerful AI.

Recently, however, distillation has become a highly contentious issue within the AI industry. This is rooted almost entirely in fears among U.S. developers and politicians that their counterparts in China will gain the upper hand.

It began in January of last year, when Chinese AI startup DeepSeek released R1, its debut AI model. R1 approached frontier-model capability on some key benchmarks. Most importantly, it was rumored to have been built at a fraction of the cost that the biggest labs in the U.S. had spent on their flagship models. Not long after the release, OpenAI accused DeepSeek of distilling the model behind ChatGPT to train DeepSeek. A little over one year later, Anthropic accused DeepSeek—and its fellow Chinese AI labs Moonshot and MiniMax—of doing the same thing with Claude. Anthropic also accused Chinese tech giant Alibaba in June of “illicitly” using Claude for distillation purposes.

Note that Anthropic didn’t use the word “illegally”: The legal rules around distillation, as is so often the case in the Wild West of the current AI industry, are ill-defined, but the Trump administration has vowed to crack down on Chinese labs distilling U.S.-made models, as always by framing the AI race between the two countries as a matter of supreme geopolitical import.

“AI distillation, when legitimately used to produce smaller, lighter-weight models from more advanced systems, is a vital part of [the AI] ecosystem,” Michael Kratsios, White House Office of Science and Technology Policy director, wrote in an April memo. “Industrial distillation activities that aim to systematically undermine American research and development and access proprietary information, however, are unacceptable.”

Risks

The dangers of model distillation go beyond geopolitics, though.

AI is, to put it lightly, an inexact science. When you train a flagship model on pretty much the entire corpus of the internet, it’s going to pick up not only benign pattern-recognition rules, like how to structure a sentence or identify images of cats, but also deeply ingrained human biases. The good news is that with enough compute, you can slowly train the models to spot and steer clear of those biases. Models can also be fine-tuned to detect when a user is trying to use it for nefarious purposes (generating a recipe for a bioweapon, say) and refusing to respond.

Those safeguards don’t always translate via distillation, though. A student model might pick up on foundational behaviors, but it’s less likely to inherit that deeper layer of safety knowledge which the teacher model uses to determine, for example, when it should not indulge a dangerous request. Alignment gets lost in translation.

There’s also the risk that through distillation, harmful biases get passed down from one model to another. Just as even the most conscientious parents might unconsciously pass on some bad habits to a child, a student model could learn some skewed perspectives about the world from its teacher. If that student model is used in its turn to train another, the original bias could be passed on again, and on and on.