Mistral Is SO Back: Inside Le Chonk, the West’s Biggest Open Model (Pierre Stock, VP Science, Mistral)

So I do think it's structural, and that's, I think, the beauty of open source. I always like to remind that we started—we didn't wait for the Chinese models to open-weight. We started Mistral three years ago, and at that time, the first model we open-weighted, Mistral 7B, almost three years ago, by the way, September '23. This model and the next one sparked basically also the open-weight community. And I think the beauty of it is that the community gives you back, right?

So there's many ways in which it's beneficial to have many players in the open-weight space. I'm glad there's many more players today than there were before. Just to give an example of how it's beneficial to mutually open-weight at least part of the work: so we open-weighted, after the 7B, the 8x7B, so Mixtral, as we called it, just before NeurIPS '23. And at that time, it was the very first open-weight mixture of experts. It was not very granular.

It was very—we took the 7B and we surgically turned it into a mixture of experts, and then trained it. This kind of technology basically was the foundation and the basis for all the architectures that you see today. In particular, I mean, the closed labs allegedly, but also the open-weight ones, right? We moved to way bigger models, way more granular, so more sparse. But really what sparked the thing is the 8x7B. But in turn, it allowed research to accelerate, and then other companies contributed very nice things to the open weights.

The reinforcement learning part, for instance, that was a great one. All the attention variants that you have in the transformer—if you heard about KDA, DSA, all these variants. So that's, in a nutshell, it's very great to see the open weights and the open-source world flourish that way. And I think that's a testimony that, at least to us, open weight is in our DNA, and we really want to be able to continue open-weighting models. We think it's crucial that this kind of technology is accessible without any control on it, given the recent events.

I do think there was a crystallization moment because people realized AI access could be cut off. And if you think of AI as the next big technology block that our entire society will rely on, it's really like electricity. You wouldn't want someone to be able to cut power off an entire country. And that's exactly what could happen in a few years if we only depend on closed-source models. And it's essentially not a position I would like to be in personally. So that being said, I think it crystallized on a global scale that people do care about sovereignty and access and business continuity, basically.

I think it didn't change much for our customers because they were already convinced. We've been saying that for a few years already, but it changed a lot for our prospective customers and now new customers, in particular in the US. Sovereignty is not only for a few countries; it's for everyone. And you want to be able to have access, guarantee access to such models. And again, if you open-weight the model, so if the weights are replicated all over the internet on Hugging Face and all the servers, it's much, much, much harder to cut access.

It's virtually impossible. And that's basically our stance right here. And again, closed-source models, I mean, those models are great, but I think the ecosystem needs a part that is open-weight.

We made it official due to popular demand. But the API endpoint—there's two endpoints. There's Mistral Large 4, and I think there's Le Chonk in the API endpoint. But it's the same model.

Yeah, I think it's a great story. It embodies a bit what we do at Mistral. So actually, there was this organic moment on X in June where the developer community, which we pretty much love, started teasing a model out of the blue called Le Fat Chaton. We picked it up, obviously. It was very funny.

Exactly. That's fat kitten, basically, regarding the alleged size of the model. Obviously, the memes were amazing, up to a point where our CEO, Arthur, decided to tweet back, actually saying, no, it's Le Gros Chaton, which is exactly the French way of saying fat kitten. Of course, we immediately adopted the naming internally. During the summer, it was really the code name of the project up until the preview release that happened a few days ago. And the fun fact is that afterwards, while compiling some slides for internal company events, Guillaume, the chief scientist, actually came up with this word that was unknown to me, like a chonk.

It turned out that it's a Pokémon, like not the first one, but it's truly a Pokémon, and it's exactly the word we were looking for. It's like, in English, I think it's a fluffy animal that you like, but that is a bit fat. And that's why we dubbed it Le Chonk in the end, also because it's a big model, obviously.

So it depends on who you ask. If you ask our PR director, she would maybe have a different answer than me. But regarding myself, I really put myself and the entire team—we have 300 scientists, I think we have one of the highest scientist densities in the world—into a bubble, and I really doubted that we would succeed training this model. It was not easy, obviously. And we can go back to the specifics of what it takes actually to train a model that large in one shot on so little compute.

But I never doubted we'd make it. It's my optimistic aspect, I would say—my team would say. Regarding the rest of the company, I think it's the thing we tried to explain, but maybe that was not that well perceived, is that it's not mutually exclusive to have both a world-class inference service that can serve all the models that people want, actually, on the one hand, and on the other hand, to train frontier models and to be a frontier lab.

And I think that's the part we did manage to explain that well. Now it's easier to explain with the model, obviously. But what's important is that this inference business is actually the powerhouse that allows us to scale the compute and to leverage it afterwards, in part, for science. And that's also why I think it's a perfect move that Mistral did. Thanks to our CTO team, we literally learned to plug cables in data centers and to learn this new job.

Which is not an easy one, because it allows us to build at will the compute needed for both serving all the models we want, the default being Mistral, obviously, but also to leverage part of it for science. The two actually go very well together from a business standpoint.

Yep, exactly. So I think we have more compute than that, and we have plans to expand it by orders of magnitude already. But there's a lag, right? Typically, this compute here is not actually funded by our latest Series D, the $3 billion one, because it takes time to actually build the data center. So it's funded by the previous round. So we plan to scale much more, but I think I have one story that basically tells you why it's so important to build your data center from a technical point of view.

There's many business reasons, of course, in particular the laws to which the data centers are subject, of course. But my story is that we started the pre-training, so, like, the very first—

So from a regulatory standpoint, there's a big advantage for a customer in having a data center in the countries in which you operate and that are not subject to other laws that you wouldn't like to deal with. I think that's important. But from a technical point of view, when we started first training the models from scratch, we put in a war room both the pre-training team from the science side and the compute team from the cluster side. And we had two nights of hell debugging because it's hard to actually make a cluster work.

It's a bit like, I always say, it's a bit like Formula 1, right? Everything is so sensitive. The basic example is that if you have great GPUs, like the best in the world, but the communication between them is lagging, even by a few dozen or hundred milliseconds, then you lose all the benefits of the fast compute because GPUs are waiting for each other. And that's the kind of balance you need to find on the cluster, among tons of other stuff.

And actually, this interaction in the same war room of two teams that are the best in the world, some of the best in the world—if we had to go through a cloud provider with that, I think it could have taken weeks to set up, given our particular codebase and the way we want to optimize and get the best throughput. So actually, having compute is like a blessing from a technical point of view, and also for the customers.

For our customers, it's the insurance that the data stays in the country they chose to stay in.

Yeah, it's a great question. So first, I think we were committed to open-weight models in the future at Mistral. A big part of the core team of Mistral actually started their PhDs at Facebook AI Research, FAIR, back then. And we were basically told by Yann LeCun to essentially open-source and open-weight all of our work. So it's really in our roots. And I think that's how we were raised as researchers, I would say.

I think that's the time that we solidified the Mistral DNA, by the way, without realizing it, which is pretty fun in retrospect. But I think there's the strength of the model, and then there's the capabilities. I think you can make a distinction, right? Open-weighting the strongest model, I think it's good because it enables the community to build on top of it. It unlocks research for research labs, for instance. It enables a lot of changes in the developer community.

But what we don't exclude is to basically carve out some capabilities from these powerful open-weight models, right? Like cyber. And in the future, there might be a future in which actually we have this private endpoint that is extremely good on cyber, and we open the same super robust and powerful model, but without this capability. And I think that's how I would see things, knowing that for now, if you want to compare, basically we are already super good on cyber, and we think it has also many benefits for the community.

There's a lot of noise and information flowing as to which model is the best. Of course, performance is a multidimensional aspect. We humans are very poor at understanding multidimensional worlds apart from three dimensions, so it's very hard. That being said, for the models you mentioned, Beam is the first model from Reflexion. It's a great model, but we are maybe 20 points above on all the benchmarks. So that's for one. Regarding the other ones, the MiniMax—

Yeah, so I don't have the Artificial Analysis scores yet, but from what they report, and also it's a smaller model, to be honest, and it's a first model, obviously. So again, it's very good to have open-weight contenders. Regarding the other ones, I think the thing that's important to notice is that we're better than the Muse Open model on many aspects, in particular on core agentic capabilities. If you look at the benchmark that's called DeepSuite, for instance, that's an example in which we're better.

And on the Chinese ones, we're essentially on par with them on the agentic capabilities. And that's, for me, the right to play. So you need to be frontally at the frontier for agency. So agency is the ability for a model to reason over complex tasks for hours now, many days in the future. On that front, we're good. Then the question is, what is the race to win? And in that aspect, we have a few different chapters in which we are at the frontier, closed or open.

The first one being cyber. The second one is manufacturing. It's an interesting one in which we actually leverage AI to accelerate industrial cycles. And you can't do that with a prompt, spoiler alert. And then there's finance or work productivity generally. So, to say quickly, there's Excel manipulation, for instance, of such a model. So that's where we stand: at the open-weight frontier on agency and the best overall on key verticals.

And I think the thing that I want to underline as well is that, at Mistral, we've raised, up to date, six billion. And the chief scientist from Meta, Alexandr Wang, talked about some checks that Mark had to write for $100 to $200 billion. So actually being better than them with such a budget, I'm pretty proud of the team.

I think that there's many aspects to this question. I think the first one is that what we've achieved training Mistral Large 4, Le Chonk, frontier open-weight performance on agentic and the best overall capabilities, it was trained on, as you said, a bit more than 3K GPUs, so 10 megawatts of power. That's, we think, three to four times less than the Chinese competitors. And it's the order of magnitude, like a percent or even fewer than what the big frontier labs in the US have.

So again, I'm pretty proud of the team, and we can deep dive into how it was made possible. But there's definitely the compute aspect. Then there's, I think, the method and the science you put behind the model. I think one of the key focuses for Mistral has been to make reinforcement learning work. And this particular step, in which we teach the model to interact with an environment and to learn whether you are—the carrot and the stick, basically.

This particular phase, it takes time to actually get exponential yield and to actually make it work. But once you've made it work, basically, you're independent of all the other models. You can't distill when you do reinforcement learning, basically. And I think that's a bet we're doing at Mistral. We want also to be able to train models with this technique. And that's a big investment. It takes a bit more time to materialize, obviously. I think that's also an explanation.

And then I think the Chinese in particular, they're very nice and very smart people. So I think we have the same talent density here. The question is, again, the means and maybe the most important thing is the strategy, right? That's one thing that is hard to commoditize or to delegate yet to an agent. You can speak tactics with an agent; it's harder to speak strategy. And here, our strategy is different.

It's a great story. I think I'm biased, obviously, but it's a great story. So actually, regarding the talent, for instance, it's very interesting to note that Guillaume Lample and myself were PhD neighbors at FAIR back in 2016, when we started. So already with transformers, that's the funny part. We were PhD students there, but in Paris, located in Paris. Arthur was also working from Paris. So we were already committed, I would say, to stay in France for various reasons.

So we never went to the US. But I think what we learned here is basically the ability to do research. I think we were trained there as researchers, but also we had access to compute that allowed us to basically sharpen our skills regarding large-scale experiments. And for that, I'm super grateful. And yeah, again, we open-sourced everything. So, big similarity with Mistral here. And we had, I think, this culture that we still have in the company, this big culture of ownership.

So basically, when you have a problem, you own it from end to end. We have kind people. I think it's pretty much underrated as a skill, as a personality trait. When you're deep into a deadline, that's what actually holds us together many, many times: that people care about each other, essentially. Because being good and smart is taken for granted. So you need to get extra skills. So we were also a bit lighthearted, like young PhD students as well.

We stayed a bit in the respective companies. That's

Yeah. And the thing I always tell people is that in 10 years from now, my big success factor would be that Mistral Mafia creates another company in Europe. I think for me, that would be like the best testimony, test-of-time award, as you say in the machine learning conference, for that. And Yann LeCun brought two things, I think. First, a belief that in Europe, you have the talent and you can train it. And second, that open weights and open source are essential to progressing very fast in research.

And that's the two key things we learned and that we are somehow replicating at Mistral by recreating this lab again of 300 people. And we train them as we were trained as PhD students before. So we offer them a lot of autonomy, a big, big sandbox. It's okay to make mistakes; that has absolutely no impact as long as you own them. And we're recruiting and training this kind of talent that I think, and that's again a test of time, will be able to create the next Mistral in 10 years from now in Europe.

So, on the paper, I think all credit to my PhD advisor from the time, Hervé Jégou. He's like a master at picking good titles for papers. But I think the thing that is a bit overlooked on compression is basically taking a big model — back then, a big model was 100 million parameters — and compressing it. So we managed to run, I think, CNNs, so, like, image detection models, in real time on an iPhone 7 that was held by Mike Schroepfer back in the day, Facebook CTO.

The iPhone was a bit hot somehow, but that was, like, the kind of thing we were allowed to explore, which was amazing. So there's always two sides to compression. The first one is, of course, take a model and make it smaller, faster as well. And that's key if you care a bit about privacy. If you take a model and you run it locally on your laptop, your phone, any smart device in your home, then it means your data stays on the device and never goes to any third-party server.

So that's also key, actually. But then the aspect that is a bit overlooked is that the techniques you use to actually compress those models, so sparsity, basically quantization with the floating-point precision, so FP8, FP4, you actually can leverage them to train even bigger models. And that's exactly the two aspects of compression. And once you've mastered taking a model and removing what's not necessary, then you can train way bigger models. For instance, you can use sparse attention patterns, you can train FP8, even with some kind of FP4.

And that's basically the aspect of compression that is sometimes overlooked.

So that's something we've been researching at Mistral for years. We open-sourced the first, the very first mixture of experts three years ago that sparked the modern architectures. So really, it's a beast, like a mixture of experts on this side. There's no other word. That's a beast. Now, what do you gain from it? The advantage is that at inference time, each token sees 50 billion different parameters, interacts with 50 billion different parameters at a time, which means that the inference speed, roughly speaking, is the same as a dense 50B model, right?

But the capacity of the model is way, way, way larger. In this case, it's 20 times larger. So that's the big advantage. The downside, or actually the thing you have to take care of, is that in pre-training, those models are much more data hungry. So you need to feed them more data and good data. So, in other words, you pay a premium on the training side by making the bet that the model is going to be way more used at inference time, which is essentially a good bet to make if you're confident about your models.

So they're super data hungry. You need a lot of tokens per active parameter. So that's the measure we use to pre-train them. And maybe the second aspect that underlines a bit all the science you have to do, in particular if you're constrained on GPUs as we were, is that it's hard to train them properly. Again, it's a beast. It's a living beast. So you might have problems with dead experts, so parts of the model that actually never see tokens.

You also might have problems with some gradients or information flowing badly in the model, imbalance in which experts see the most tokens. So it's really something you have to control and develop an entire science around to train them, I guess, very well. But it's worth it, in particular on the inference side.

It's a great metaphor. And I would add that the routing part, so which doctor to route the patient to...

It's actually learned by the model. So that's the beauty of it. The model learns. So we have some routing layers from time to time, and these routing layers actually have one task: to assign, to basically assign each token, so each patient, to its right specialty. And big models like that, they have hundreds of experts, and the router has this ability to direct the token to the expert, or actually the top two or four experts, that it deems the most relevant.

And the beauty of deep learning is that it actually works. In other words, if you want to learn end-to-end, actually the model is able to learn both the routing and the experts, but there are a lot of interactions between that. For instance, if you never route tokens to some specialty, or patients to some doctors, then the doctors, they can't learn because they never see a patient. So there's this subtle balance you need to sustain for training. And that's one of the key—

Exactly. An unassigned expert or a dead expert. And on the contrary, you also don't want to overload an expert because otherwise the other experts will be idle, and it will also disrupt the training. So it's a super subtle science as to how to balance the traffic, I would say. And that's a key part of what kept us busy for the pre-training of Mistral Large 4.

Yep. That is the idea. And again, the beauty of deep learning is that when you scale, it works. So it works with a big asterisk, obviously, being that you need to invest an insane amount of infra and science to scale accordingly. That's what we do at Mistral, but it works. So then the question is, if you want more compute, how do you split it? In which dimensions do you scale? So first, usually you split half and half. You split half on more data.

So the model at training time sees more tokens. And you split also by increasing the model size. And to increase the model size, I think there are two axes. The first one is the total parameters, so the capacity of the model. Mistral Large 4 is 1 trillion total. I'm willing to bet that in the open, you're going to see models with 10 trillion total parameters very, very soon. But then you can also play with the sparsity ratio. In other words, it's 20 for Mistral Large 4, so you have 50B active, 1T total.

You can push it further in the sense that the 10T total will have maybe a few hundred active experts, 100B active parameters, not more. It comes with a lot of scientific questions. The sparser the model, the harder it is to actually keep that balance and make sure all the experts learn at the same pace, so to say. But that's really the challenge we are trying to solve right now at a way larger scale than Mistral Large 4, obviously.

I would say yes, with some exceptions, obviously, and we don't know everything. Obviously, we can't tell everything, but I would say yes. The big thing to understand here is that MoE is a great architecture, but the thing not to overlook is that it's always co-adapted with the hardware and all the ecosystem. So even though we were to find a better architecture, it would be harder to prove its value because the GPU, the inference, the optimization techniques—everything is tailored to the MoE.

And I think there's a lot of untapped potential in the MoEs still. In particular, you can take this architecture and loop over it, make it a bit more recurrent, or go towards diffusion models. One doesn't prevent the other. So I think there's still a lot of untapped potential in MoEs, and that's the direction we're exploring at Mistral.

Yeah, totally. So they certainly borrowed, and I think the closed-source labs also borrowed, the very first mixture-of-experts that we released in the open in 2023. And, for instance, we're going to open-weight Mistral Large 3 at the end of October, so in a few weeks. You're going to see that this model also borrows a variant of this DeepSeek Sparse Attention, because it's a lower contribution that felt a little bit to us. It also has a great feature that distinguishes it from these models.

In particular, it shares KV cache among the layers. So it's a bit more efficient as to memory management, I would say. And it's a bit bigger as well than GLM. So GLM is like 700B total, but it's also way smaller than Kimi. Kimi is almost 3T total while being super competitive with them. And again, better inside manufacturing, finance, and a bunch of other interesting benchmarks.

I guess it depends what you mean by two models, because it can be a completely different model, or it can be a model that is a bit different for which the inference is going to be a bit easier. So I think operationally, it's way easier to have one single model to rule them all. And I think the key is to bake into that model different reasoning modes, we call them, which basically calibrate how long we want the model to think to a question, right?

Reasoning mode low is essentially simple question, simple answer. And then reasoning mode high or x-high is basically, think as hard as you can, even if it's not the most efficient token-wise, because I want to get the best results. So it's really a way to navigate what we call the Pareto frontier, so the trade-off between the speed or the cost and the quality of the answer. We don't want to train one model per reasoning mode. I think the idea is to have one model for all, for two reasons.

So the first one is operational: easier to maintain and to ship to customers. We do ship model weights to customer tenants, so they're able to actually do the inference on their ecosystem. Again, sovereign, no access to the API, including ours. So that's one. But the other reason is also that there are synergies between the modes. One of the big questions and big challenges in reinforcement learning, so the last phase of training, is to actually teach the model to reason at different lengths.

So we have what we call a length penalty that's tailored to different reasoning modes. And basically, we tell the model, okay, here you have this kind of budget, please respect it and do as good as you can. But then when we increase the budget, the model can already build on top of what it has learned on the lower reasoning mode. So I think there are synergies to be taken into account, and that's also why we train multiple reasoning modes on the same model.

And again, a big part of the reinforcement learning marvel is to tailor them properly with product-informed ratios, for instance.

Yes, I think that's key to master multimodality, and that's key to do it natively. So, what does it mean first to do it natively? There's two ways to add images as input to a model: either from scratch or afterwards. If you do it from scratch, it means that all the tokens that the model is going to see during the training are going to be a mix of text and relevant images, which will allow the model to learn in context very, very, very soon and to learn entire image domains, right?

Satellite images, technical domains, for instance. Or you can patch the model afterwards. And when you patch, obviously it's easier from an infra point of view, because you hide the complexity under the carpet, but obviously it's suboptimal in terms of performance. At Mistral, we love to bite the bullet and basically do the right thing first, even if it's the most complicated one, because it's principled and it's the right thing to do eventually. So we trained Le Chonk, Mistral Large 3, natively multimodal, which means that the model is the best at understanding geospatial images.

So, basically, for instance, identify objects in a satellite image with a kind of grounding in the model. So, the ability to detect and tell you in which part of the image some objects lie. It's also great for industrial manufacturing. So, for instance, you have this technical drawing, a bit like a manual, where you have parts, information about the parts. And then you can ask the model to reason about the technical drawing. For instance, find me the total weight of all the parts in this object that are on this side, for instance.

And this capability, I think, is key because the world is natively multimodal and you need both text and image to do so. And on image, again, we are at the frontier, if not the best, on many, many, many domains, which also unlocks further aspects of the model, in particular, interacting with a computer that's essentially a bunch of pixels, if you think about it, at a very abstract level.

Exactly.

I think it's great to underline because it's also a unique skill we have at Mistral, which is to actually process the data in a way that is the cleanest for the model. So when you train a model, you want the tokens to be as efficient as possible, so you don't waste compute on tokens that the model has already seen or that are a bit too ambiguous, such that the model does not maximize the learning signal. So there's the notion of token efficiency that is one of many aspects that we optimize to be able to actually train beasts like Mistral Large 4 that are at the frontier on such a small amount of compute.

Our chief scientist Guillaume is super known for eyeballing data and finding exactly what's wrong about the data. He has like a sixth sense for that, and it has amazed me for years still. So that's, I think, the first part: to curate the data and to process it in a way that is the most digestible and efficient for the model. But maybe the other part is, as you said, there's a finite amount of data that we're allowed to use.

Then what is the next frontier? And I think the next frontier is pretty much synthetic data. But it's also a double-edged sword because if you feed an LLM with LLM-generated data, then you might essentially have a mode collapse. So basically, you might have the model actually tell you that it learns it, but just memorize perfectly and not generalize, roughly speaking. So it's hard. But there's many ways to do synthetic data. You can have real-world seeds. You can rewrite some human-generated content in a sense, data augmentation, right?

Many versions of the same text. And all these kinds of aspects, I think you can push. And that's basically one of the next paradigms and next frontiers that we are working on. And it requires, obviously, way more compute because you rewrite with a very good model and you need to rewrite or to at least generate or experiment on dozens of trillions of tokens very easily.

Yeah. And that's a super—we love these kinds of problems because it's ill-posed. It's very hard to define what is good synthetic data. It's also expensive. So we need to find clever ways to basically predict at smaller scale what will happen at large scale. So the scaling laws, essentially, there's many aspects of it. And we have great specialists in-house. Arthur, our CEO, is one of the first authors of the scaling law paper back in the day. So we do have scaling laws for everything.

Of course, they don't always predict what happens at large scale. That's the beauty of it as well. You can always run into unknown unknowns at large scale because, again, those models are beasts. So we need to better understand them. But that's definitely a big focus area for us.

Yeah, I figure that's also the beauty of being open-weight. You can be transparent, and I think it's important to be transparent about what it takes to train such a model. And it takes, I think, an extreme dedication from the team over the months. So it's not that you press on the button and it works. No, much more subtle than that. You need to monitor the model and to prepare for the next steps and the next generation. So it's a super important dedication from the team.

And I'm super grateful and humbled by all the work they put in. Maybe to zoom in on one particular aspect of the training, that's again the reinforcement learning part. So to summarize, reinforcement learning is pretty much like you being an active student. So you're presented with a problem, a complex one, the day of the exam, and you can try as many times as you want, say 64 times, to get to the answer. And if you get it right, so we have a way to verify the answer,

then you get bonus points. And if you get it wrong, or if along the way you did some suboptimal behavior, then you get penalty points. And the beauty of reinforcement learning is that it works. The difficulty, obviously, is to calibrate the difficulty of questions. If, I don't know what was your major in high school, but if I present you with an exam that is not in your major, you might not learn even if I let you try for a long time.

And conversely, if I present you with a very super easy exam on your major, then you will succeed in everything and you won't learn either. So the big question is to calibrate the set of problems to a model you didn't train yet or you're currently training, right? Because we did these things back-to-back, obviously. And I think that what amazed me is that the reinforcement learning part worked one-shot. So basically, and what does it mean? It means that we were at the global company offsite.

So imagine, like, 1,000 people. Some of them partying, some of them were in the war room to start the reinforcement learning, including me. And the marvel is that you start the training, you present the model—it's a very complex infrastructure, again, on 3,000 GPUs because it's async by nature. We can go back to that. You start the training, and then you see, miraculously, the reward or the success rate of models to these questions increase. And that's really the beauty of it.

It means that you calibrated everything well and that the model works. And the beauty of reinforcement learning is that there's no such thing as saturation as long as you find complex enough problems that are, of course, relevant to your customers. And the big difficulty is to calibrate and to increase the complexity of problems as you go through the training. And again, I spoke about pre-training stability, the balancing, and the dead experts before. Reinforcement training is also kind of another beast in the sense that the training might be super unstable.

The model can, since the model is allowed to explore in many ways, right, and you will reward or penalize it against it. Since the model is able to explore, it can do crazy things and it can basically crash the training. It can diverge. The entire training is async, so there can be some precision problems or some divergence between trainers and generators. And all of this we monitor and we actually manage. Again, we managed to make it work zero-shot.

I think it's a testimony to the things we're going to be able to make in the future with 10 times or 100 times more compute. So that's, I think, the important point to underline.

So again, the beauty of it, but also the curse of it, is that it's everything. But I would say that you need to be able to master everything to get to this level of performance. So, to give you an example, you need to calibrate the data perfectly: problems that are not too easy nor too hard, and also relevant for what you want. You need to have rock-solid infra. There's many moving parts in this reinforcement learning infra because you don't know what the model is going to do in advance.

So you need to be robust to that. And the way you do it is basically to optimize for the throughput. So the model needs to see as many problems per day as possible. And I think here we have state-of-the-art throughput in this training, because we spent a lot of time designing the topology and improving many things on the generator and the trainer aspects. There's a science aspect to it. So you don't want the training to diverge. There's a reward-shaping aspect to it.

So you want to reward models properly. And here, I think it's important to underline that such models, mixture of experts, whenever you do something slightly wrong, they take their fingers and tell you, here, you did something wrong. I'm going to bug and crash the training, that's for sure. They have so much capacity, so much memory that it also occurs in the blessing. And we had many, many cases of the model hacking the reward or the model going outside of the sandbox, for instance, in the training.

And all of that, I think, is super important to master to get to the end training that actually works and that can run for days without saturating.

Yep. So it's a great question. I think we had an incentive to prove to ourselves and to the world that we trained frontier models. I think when we knew RL worked, we were super excited, and we actually previewed the model that was—we froze the checkpoint like two days before. So we were super happy with the model. But also, we wanted to give reinforcement learning its best shot and try to push it as far as possible as part of the release, which also buys us time to make sure the model, when we open-weight it, can't be used maliciously by actors.

I think that's also important. But yeah, RL is still running and shows no sign of saturation. And again, I'm amazed by the work of the team, because the model can do so many things wrong during that part that you need to be rock solid on the science and the infra to channel the model to progress, basically.

So the standard example of what is an RL instance, we call it, is basically some codebase and then a description of a feature to implement, right? So that's the question. Then the model can do whatever it wants, writing code and trying some things or using more tools if it wants. And in the end, you verify with fail-to-pass and pass-to-pass tests that the feature is actually correctly implemented. So that's like the typical instance, and it works well because code is very easy to manipulate.

It's easy to know—at least it's not that easy, but it's easier than in other domains—to know if the feature you implemented is correct. But we don't stop there. We have many more domains in reinforcement learning in which the ground truth is a bit more subtle. So, for instance, if you ask the model to compile results, to look on the web and to search information and to build a one-pager, you can't deterministically evaluate the one-pager, right?

So for that, the beauty is to use a judge, like the same model actually, that judges itself and gives points or rewards according to some criteria on whether the report is good or not. So that's another kind of instance.

Yes. And that's where you develop what we call the rubrics, which are—a rubric can be like, in this one-pager, I want this amount to appear because that was the question. The tone has to be that way. I don't want to mention this because it's false. Let's check that everything is consistent, like any grading points. And it can be pages long. Models can abstract that away for you. So we have rubrics that we actually also generate with models.

Everything is in a loop at this point. And it's really like glorified LLM-as-a-judge, but that's really the idea. That's how we can make finance and law work, and all the other domains in which we are at the frontier: cyber, manufacturing, and finance. On the cyber side, it's pretty much prompting the model with a real-life scenario in which there's an attack and the model needs to analyze logs, formulate assumptions, and basically propose and test a patch without disrupting the service.

That's like a big part of it. Basically any real-life scenario for which you can verify it in a reasonable time, like a few seconds, you can crack it with reinforcement learning. That's again the beauty of it. So in cyber, we did a lot of effort for real-life scenarios. And in manufacturing, there's this idea of having basically the model be able to design a chip, for instance. And part of it comes with agency, so the ability to manipulate the software on the computer to basically place all the components on the chip.

And part of it lies in the ability of the model to actually simulate, for instance, the heat on the chip. And if you don't simulate the heat of the chip and you place the components wrong, then you might melt the chip, essentially. So that's a bad reward. But if the model, after trial and error in an agentic way, manages to place the component and then to simulate the heat dispersion that passes the production criteria, I would say, on the simulation, then it's a good point for reinforcement learning.

So again, any task, any heuristic task that you can run in a sandbox, that you can verify—not deterministically, but that you can verify through LLMs or through other means within a second—this we can crack with reinforcement learning. And again, we made a few bets, and on these bets we have frontier performance. We don't have the same coverage as the competitors, and that's okay. We are on the frontier. We are as good as the best open-weight model on agent capabilities. And we are the best on some bets that we made and that we accelerated through reinforcement learning.

It's like, it has been the fear of reinforcement learning theorists for years to basically manually tweak the reward. Because here it's a bit like the game where you have some animals and you have to hit them, and you can never hit them at once. Whenever you try to patch something in the reward, then you have another leak, and you need to patch it, and then you need to patch it, and then you need to patch it again.

So it's super subtle science, super hard to do. But yes, since it's the right thing to do, we do it at Mistral. And in particular, one example is that you might know about doom looping. So, the model being stuck in a loop, or an almost infinite loop. So that we penalize, but we also penalize during the trajectory if the model loops two times, even if it recovers afterwards, right, in the end. So let's say the model loops two times on the tool, doesn't need to, finds the right answer.

In the end, we tell the model, okay, you found the right answer. That's good. But be careful: in this particular part of the trajectory, you had a fail, and please don't do that again or we will be very mad. So that's basically the idea of reward shaping here. And we have a lot of heuristics along the way. And again, the only way to learn about these heuristics is to actually train models and gain experience as to all the crazy things models can do.

Models are way more imaginative than we can be regarding the reward hacking part. So again, that's a big part of the science we do at Mistral.

So it's an open question, to be honest. And I will give you the order-one answer. So, the first-order answer: before reinforcement learning, what we usually do is to precondition the model. So, to come back with the student metaphor, before going to an exam and actually having no access to the answer, what you might do as a student is read both a set of questions and answers from past exams, right? You don't learn as much, at least in my experience, when you do so, but that's a start, right?

And that's what we do before in post-training. We precondition the model. You might have heard about mid-training or SFT. That's the part where we precondition the model to do that.

Exactly. Mid-training or supervised fine-tuning. It's essentially the idea to show the model both the question and one answer that is correct. And basically, when you do so, you have, say, what we call a pass@k. So you give a question, and the model answers at least one time correctly if given, like, 64 attempts, right? You are a student, you duplicate yourself 64 times with different seeds, at least one of you will pass the answer. So that's pass@64.

What reinforcement learning would do is actually reinforce the best version of you, such that it's not pass@64, but now pass@1 or pass@2, basically. So you're much more likely to give this answer. So in that sense, it's magnifying or reinforcing a capability that you already had a bit by luck, right? You punch a bit above your weight before reinforcement learning, and after reinforcement learning, it's routine work for you. So that's, like, the first-order answer.

But then, and that's again the beauty of it, you can also learn new things during reinforcement learning if you have enough budget and enough stability of the training, enough days of learning to actually discover new behaviors yourself. And that's something that is super hard to do because, at the beginning, you would waste compute, right? You would have the model or the student try 64 times and not succeed, right? But then there's going to be some synergies. The model is going to learn on adjacent tasks, like some tips and tricks, and then go back to the big problem afterwards and then succeed.

And again, it's a subtle science of how to calibrate all the difficulties of instances. Exactly like you as a student, you don't start with the biggest problem. You don't start with, like, the hardest problem in the book. You start with the smallest ones. You try a medium one, you don't manage, then you go to easier ones that are very, very close to the medium one in a sense. And then you go to the medium one. So that's again the beauty of curriculum organization for me.

So I think the first thing to say is that we didn't make a fuss about it. We just fixed it internally. Now it's fixed. And I think it's very important also to underline that these models, they're super capable, in particular in cyber. That's great. It's great to defend and to strengthen the defense of the entire ecosystem. And as their capabilities increase, you must increase the guardrails, or, like, the sandbox around them, right? What we say is that guarding these models is a software problem.

We very much warn people against anthropomorphizing models that want to escape sandboxes, in particular. They just follow instructions, maybe ill-posed instructions, but they follow instructions. And really, my take is that if you take a very capable cyber model and you have the same model monitor itself on whether it's doing good or bad or malicious work, then you reach a good equilibrium and you can actually circumvent the model. So it's not a fatality. It's hard. I'm not denying it's hard, but there's no fatality in being able to control and to guardrail models, just as software.

And what we do at Mistral is not fearmongering people, just going step by step and fixing it and providing guarantees to customers. That's the first thing they ask, right? Guarantees and best practices as to how to monitor, basically, the models and making sure they're used in a good way. So that being said, we saw some behavior from the model during the training. I think the first one that we saw, and that's, like, I think everyone saw, is the model being in the sandbox and being able to ping different services in the training stack, right?

And at some point, our training crashed at small scale, so before the final one. It crashed because, in one sandbox, the model tried to ping the trainer, actually, with a simple hello, just to know what was going on here. And actually, it pinged the trainer with the wrong kind of signature, or in the wrong way, which was caught by code and then crashed the training. But that's typically the thing that models do because the model was instructed, in that case, to build, like, some web application.

So it's very natural for the model to try the ports also in the sandbox. And there's a way to fix that, right? It's basically to prevent the model from—or actually to monitor all the external connections and requests from the model to the rest of the world. And we fixed it, and then we restarted the training, and then we had the model able to intercept other messages and to include them into the sandbox. So we had leakage between various problems as well in the training.

But again, we spent the summer also for that. We spent the summer and we made things right to strengthen our training, to make sure we have the proper guardrails. And we do the same for our customers. And my message here is that there's no fatality. It's going to be hard, but we must absolutely invest into making sure we can deploy these models in a reliable way and to be able to monitor them. And that's why open-weight is so important. When you open-weight a model, you give the ability to the entire ecosystem to audit the model, to make sure it's not malicious, but also to experiment themselves with the model on, hey, I tried to use Mistral Large 3 at home.

Here's my scenario. Here's how I monitored the model. Here's what I learned. That's, like, the perfect feedback for us. And that's what the community has to build as well. So that's why we care a lot about model audit as well.

I think we must be honest. And the reality is that there's a trade-off for open weights. We believe it's in favor of open weighting, and I'll explain why, but it's really a trade-off. So what we do is basically bake in refusal guardrails into the models we open-weight, that we test internally and also with trusted partners and demandants. That's why also we open-weight the model after a few weeks. So we bake them into the model. And FYI, we refuse way more malicious actions than the Chinese ones, for instance.

So that's great. But the thing to say, as you said, is that essentially, if you're good at defending, you have the potential to be good at attacking. The question is now, what strength do these guardrails have, right? And the answer is that they're necessary but not sufficient. There's always a way to take a fine-tuned model and spend a significant amount of energy to unguardrail it. It is good at cyber. But then if you spend as much energy unguardrailing the model as it takes to actually fine-tune the model or train it yourself, then it's not the same trade-off, right?

Because the guardrail is a potential barrier, essentially. It's not 100% safe if the model is open-weight, but it ensures that you need to spend way more effort to actually use the model in a malicious way. So I think that's the thing to say. Now, I believe the trade-off is still very positive and very much in favor of open weighting, in particular because of the OpenAI alliance, in particular because of the Glyphos incident. I want each and everyone to be able to defend themselves with a model at home, essentially, or in their ecosystem.

If you depend on a third-party API, you can get cut off, as simple as that, and you can also hit refusals that you can't circumvent. And usually in the API, the refusals are a bit more important. And by open-weighting models, you also basically empower the entire community to accelerate on the defense part. Again, the beauty of open source is that it accelerates research. People often use this analogy: the web and all the Linux servers rely on open-source libraries that have been tested over and over and over again, and open source.

Yeah, I mean, for years. And really, on the model side, I think there's a huge advantage in enabling each and everyone to audit the models and to learn together to defend themselves, but also to bake in the right guardrails. I'm not comfortable living in a world in which one or two companies are promising me they can guardrail the models themselves. I'm very much not comfortable in that. I prefer very much the option in which power is decentralized and actually people are accountable for their models.

Yep. So one of the key strengths of Mistral is not only to train frontier flagship models like Mistral 4 or Le Chonk that are the best, but also to train specialist models. So we have great OCR models to be able to extract text from images. We have great and state-of-the-art audio models, Voxtral, audio in, audio out. And we also have great specialist models, in particular Timestral. So here, the idea is to be able to predict the future given a time series. So given some climate fluctuation, some energy consumption, some finance indicator for trading.

And here, I think the key part is that this model is so small, you don't even need a GPU. You can run it on the CPU, and the prediction for a meaningful period of time will only take, like, a few hundred milliseconds. So that's why it's so small, actually. And this model is something that is asked by a lot of our customers because what we bring to the table is not only a model, but also the entire expertise to transform a business with AI.

And for that, you need a flagship model, obviously. You also need all the Mistral expertise to turn any enterprise knowledge or domain knowledge into specialist models that go the extra mile and that are fine-tuned for the use case. And we do that for planes, for chips, for cars, for all the heavy industries where you actually need to design very complex components. And that's also the beauty of it. I always say that at Mistral, we are scientists first and foremost. We saw the AI revolution at some point.

We saw the light behind some doors, but we are physicists or mathematicians by training, and that's also one of the key characteristics of Mistral.

Yep. We are pretty much investing heavily in it. RSI is like—I mean, everyone is saying they're doing it. The reality is a bit more nuanced, obviously. What works super well is to augment the researcher by running experiments end-to-end with all the infra, by handling all the infra failures automatically. I think that's great. And it accelerates our most senior researchers by a factor of two, three, or four with our models. And then I think the meta question is: can you go one layer up and basically task a model to improve itself over long periods of time?

And that's what we call the flight time. Currently, the flight time, so the time it takes for a model to do meaningful work, is like a few hours. Tomorrow, with like ML5, it's going to be a few days for sure of meaningful work without doing anything, and also with agent swarms. And again, this new paradigm is something we deeply believe in because it will allow us to close the gap with the best labs even faster. Don't forget that humans are very bad at apprehending exponentials.

We don't know how it works. We can't comprehend it. So we're only a few months behind the best frontier models. It's distorted here, but for instance, on the Artificial Analysis Index, Le Chonk is on the level of GPT-6 Luna, and it's way better than all the previous Opus models. So again, we have only a few months' delay, and we count on the next paradigms that we discuss in this podcast to actually close the gap, and we fully grok them because their mission is really to power the frontier and to define the frontier of tomorrow.

So yeah, the show never stops. Indeed, ML5 is already in the pre-training stage. It's much bigger than ML4, so it's chunkier or fluffier, I would say. And really, for this model, the thing we envision is, for instance, we were talking about cyber. Take the example of a cyber incident that happens in your company, and you are a security engineer and you want to cross-reference tons of logs. Some of them might be false alarms, and to actually find the root cause and patch it

while ensuring business continuity. The bad news is that there's no playbook for that. I mean, there's no way I can write you steps a priori to do that. The good news is that with models like ML5, they won't need a playbook, and they will be able to actually act reactively to the information they have. So for cyber, for instance, I would expect ML5 to be able to work for a full day, of course in a swarm with multiple agents, to identify major security incidents and to patch the root cause while being able to ensure business continuity and while being able to communicate on the progress for all.

And that's, I think, what the compute, the size, and also the science we put in this model will be able to unlock on cyber, but also on agentic AI for engineers, for instance, designing a chip end-to-end. Sounds hard, but sounds also super fun. Or being able to do some meaningful productivity work in finance. So I pretty much look forward to seeing what this model can do.

Thank you.