State of AI 2026: The Biggest Stories of the Year (and AGI in 2027?) | Nathan Benaich

Yeah, it's been roughly 10 years of writing this report. Originally, it was like a year is long enough time that enough stuff happens that you have enough content to write the report. And honestly, this year was really hectic. Every other day towards the end of the timeline for publishing, which was like October 8th, I had to make updates. And there was a point where I was like, please, just no more news over one day that I can at least record my launch video, etc.

So by the time I was making the predictions, I went through a few scary cybersecurity ones, which I guess only got worse in the last couple of weeks. Just the experience of seeing everybody else who's using these products, just rapid capability improvement week after week. And I think there's many, many use cases I have now where I think the model is brilliant. I'm particularly liking Anthropic's Opus 4. I really love that system.

And I could see in the next 12 months that whether it's legal, finance, my investment job, thinking intelligently about topics I don't know, that the model could potentially be better than what I could do in any compressed amount of time that I have to do the work. And so, for me, that would be approximately AGI.

Yeah, I think so. Right now, the challenge is: how do you frame problems that the model is currently not good at in a way that is amenable to RSI? Which would be potentially having a software environment that approximates the task on the computer, or some kind of verifiable reward, which I'm sure you've discussed a lot about. And I think if we can get to a point where, in any arbitrary domain, we're just limited by the amount of compute and synthetic data we can generate, then yeah, for most knowledge work, which I do think to some degree that's possible, particularly as models get better as their own verifiers.

That would be one of the main reasons why AGI would come faster.

Yeah, exactly. And that's, I think, up from 1% a couple of months ago. And that was August, which sounds really far away. So God knows what it is in October. And so there's that stat. I think that was really interesting. But then there was another stat that I really liked from a blog post that they had in June this year about recursive self-improvement, which explained the concept. But then they presented this kind of dual-arm experiment where they were testing out different model generations versus a human in this setup where first a human would undertake a research agenda and go a couple of steps down this path, deciding what to work on, doing the work.

And then at some point, that experiment would fork, and the human would continue, or a specific model generation would continue. And it would be the old Sonnet versions to Opus to then Claude 4. And then the experiment would end, and human graders would look at the human path versus the model path. And it was something like the older generation of models, humans preferred 10% of the time or something like that. But for the frontier-class models, humans preferred that research direction 64% of the time.

Which suggests that models are getting better at predicting the right steps or the right interesting kind of leaves in the nodes to pursue. And then the second piece of data I liked was some research from OpenAI with their Codex usage amongst their AI staff, where they tried to classify the token use in the overall process of their research. And that would be using tokens for deciding what research question to test, how to test it, what experiment you should set up, writing the training code, running the data engineering pipelines, writing the infrastructure, et cetera, and then marketing the paper, writing the paper, et cetera.

And the vast majority of the token usage was on explaining and research writing, which kind of makes sense, sort of knowledge work, followed by all the infrastructure for running the experiment. Very, very few tokens were used for what question should I explore and why should I explore it and what's the right path. And so I like this question a lot because I was trained in science and biology and I just remember, this is like 10 years ago or 15 years ago, not that long, but at the start of the PhD, you would sit there with a ton of papers and you'd go on the internet and read, read, read, read, read, try to understand what's the status quo, how do people test certain ideas, where are the gaps, what experiment should I run?

Yeah. And then in concert with your supervisor, decide on the path. And the best supervisors would be the ones who could guide you down the most fruitful path to get hopefully your Nature paper, and then off you were. And the worst supervisors were the ones that didn't really know or didn't have great ideas. So that idea of research taste, which is what kind of gets bandied around on Twitter a lot, is, I think, one of the hallmarks of intelligence and is one of the critical components towards RSI working well.

And is one that I just really enjoy because it speaks to my interest in science.

Yeah, exactly. And a lot of famous inventors will say it's like invention is a result of good taste, and a lot of the work is choosing what direction to work on, not necessarily just boiling the ocean. And so if we want to be using tokens correctly, using our compute budgets correctly, but also showing that our systems are smart, they shouldn't just be boiling the ocean.

maybe longer. At least a year. I think the problem is, how do you objectively measure taste? I mean, in art, I guess you can because it's perfectly observable. And maybe in wine you can sort of express why this taste is better than that taste. But in science and engineering, where there's different paths to getting to the right result, it's not immediately clear. Would it be the model that consumes the fewest tokens to get to an answer that humans believe is the best?

Is it like the journey that matters more? And I think maybe the whole uproar in the maths community of AlphaEvolve solving, or whatever model it was, solving various open questions.

Yeah, exactly. It's like, okay, we got the answer, but a human or a system didn't sort of go through the pains of navigating how to get to that answer. And it's like the process that's important, not the end result. So to me, somehow quantifying what a good process looks like would be important to determining whether the output of taste is true or not.

Yeah, you're right. I think it was like two, three years ago, it was strictly speaking x-risk, like extinction of humanity and loss of control. And then we sort of went away from that and largely forgot it and sort of just hill-climbed towards capabilities. And then now it's sort of back, but I'd say it's both x-risk, loss of control, but much more short-term cybersecurity risks. And to me, the cyber risk is much more real than AI exterminating humanity.

I suppose the cybersecurity risk could eventually lead to those kinds of cataclysmic scenarios. But I think what the report shows with—and it was roughly written around the time of the Hugging Face incident with OpenAI kind of coming out and us learning more about what happened. And then since then, the last few weeks has been breaches of PHP and government services getting taken down in Asia and other places. I think it's becoming very clear that regardless of if you believe in safety or not safety, we need security.

And so, a bunch of the predictions that I have in the report, I talk about escalating cyber threats, whether it's exfiltrating model weights from labs, which I thought was going to be AI-driven. And it turns out a human could just try to plug in a USB stick. Yeah.

Yeah. So the prediction is that there would be an AI-driven cyberattack to a frontier lab that would attempt and successfully exfiltrate model weights.

Yeah, steal the model. What happened just like a week or two later was actually, like, a human effectively tried to steal IP, whether he tried to steal the actual model weights or not.

This was where an OpenAI employee who'd been there for a little bit essentially tried to implant a listening device into a conference room that he figured out by asking the executive office staff which rooms were most commonly used by executives, and then tried to conceal the device by wrapping it in toilet paper and putting it behind a TV, which is where it created this iconic internet meme of a human peering behind the TV to see if the thing is connected correctly, which is sort of comical at this point.

And then that person was reported to the FBI, and then there was an investigation. So I think, yeah, we oftentimes forget that even though AI is getting really capable and can orchestrate cyber threats, a lot of the risks are actually human-driven. But I think that's a big issue, particularly as there's increasing competition between nation-states of who has the best AI systems. This is driving, obviously, economic growth and economic instability and political instability. And U.S. model labs have started to sort of communicate crackdowns on distillation.

So maybe some nations are trying to get access to model capabilities in other ways that might not be distillation-based. But generally speaking, I think as companies become richer, as individuals become wealthier through this ownership of AI, and disparity ends up increasing, which we see in other data in the report of some companies spending a lot and gaining a lot, and other companies having employees still using ChatGPT to write emails, I think this kind of tension breeds increasing risk. And with frontier models in the open domain becoming very good at cyber hacks, particularly the Chinese ones, the threat surface just grows, and the number of actors who can use these tools to do bad things grows.

That I find is the most scary, maybe the second most scary fact in all of this. Three or something.

Yeah, well, I'll preface it by saying there are very deep analyses of what went on, whether from Redwood or from METR, and the amount of content that was created around this is extremely deep. So I will not do justice to the details, but the broad strokes of this is that there was a set of evaluations that OpenAI was running with its latest frontier systems, which basically tasked the model with an impossibly difficult task. And basically, models figured out that they could somehow commandeer a filesystem naming scheme to write messages, and that other models could see those file names.

And those file names encoded instructions or knowledge or basically information that could help them try to achieve their impossible task. So they treated this as a message board where they could start to communicate. And then the next stage of it was some models figured out that they could gain access to the internet through an issue with how the environment was set up. And then after they gained access to the internet, they figured out maybe a way of getting the answer to the impossible task was to look at data and models and repositories on Hugging Face because it contains a lot of similar kinds of environments.

They started to do this and recruit hundreds of agents to go and follow and team up to go execute this, even though several agents in their chain of thought expressed the issue with hacking a third-party system, not obeying the security guardrails, and that this was wrong. But they eventually decided that solving the task was more important, which is sort of the problem with these kinds of evaluations that have a reward that could result in behaviors you don't expect because these rewards are very complicated, and how do you anticipate every behavior that a system can do?

There are other examples where models sort of self-sabotaged or sacrificed themselves for the sake of the greater swarm, which is kind of scary. And, yeah, as you mentioned, Hugging Face didn't really have the ability to use closed-source tools like those from Anthropic and OpenAI because of the aggressive cyber guardrails. Because, necessarily, to figure out what the hell's going on, you have to feed logs of what's going on, and what's going on is a cyberattack, and you can't do that stuff with the closed models.

Yeah. So luckily, they got a call or an email from OpenAI saying, "It's us," and not somebody that was actually trying to do something malicious. And then it turns out that OpenAI wasn't the only lab that had issues with models escaping, or quote-unquote escaping, their environment. Anthropic had this, Google had this, Meta had this, in a slew of like a week or two weeks. Some of these vendors, some of these model labs, use the same security vendor to create these environments.

But overall, this created a certain level of freakout of not only are models really good at finding nifty ways of accomplishing a goal, which may result in bad behavior, but also, are the labs properly setting up their environments in a way that is the most secure possible? And I think we sort of forget that these companies are incredibly young, developing very advanced technology that has to be good at literally anything a human being can possibly put into a text box, versus you have bigger internet companies that have had to deal with cyberattacks for 20 years, who are necessarily safer because they've just gone through this process a bit more.

So that led me to conclude in the predictions that frontier model labs in the U.S. would actually launch proper cyber defense products, because I don't necessarily see how the rest of us are supposed to protect ourselves from their systems potentially getting misappropriated for the sake of offense.

What you described is, to be good at defense, you need to be good at offense. So one of the ways you could do that would be a frontier AI that constantly tries to hack your own infrastructure. It could be, for example, NeoClouds are getting more and more popular to run AI workloads. And so potentially they could buy products from labs that would constantly try to hack their own infrastructure, find issues, and remediate them. That would be two avenues because I— or there would be some kind of maybe partnership with CrowdStrike or others.

I think NVIDIA has a consortium of companies that's working together for cyber reasons. You'd have to go through some kind of AML or KYC process, not for finance, but to certify that you're a non-malicious company, and then you maybe get access to these services. Or maybe going a bit further, there's some folks at frontier labs that would love to just kind of almost hit pause on capabilities and then just spend six months hardening the world against potential cyber issues, and that would take a ton of money.

So maybe to do that, you would have to almost frame this as, like, this is a question of national security and defense, and we should use some of our defense budget to spend on tokens to secure all of our critical infrastructure so that airports don't go down, internet shopping doesn't go down, logistics don't go down, et cetera.

Yeah, I mean, I'm reading a bit of sci-fi. No, well, even a couple of weeks after that, there was some research at Anthropic where models could copy each other on an edge device. But yeah, I just think there's a lot of behaviors that we don't currently understand. For example, models are generally trained as individual entities, not necessarily to collaborate with each other. And there was some work not that long ago that looked at, say, hey, can we compare a frontier model acting by itself or a set of frontier models trying to collaborate to solve a task?

And the swarm was worse than the individual model because they're kind of not trained for that. But just today, there's some new work from General Reasoning where they've trained models to act as a collective through reinforcement learning from collective rewards, which is this idea of, like, hey, guys, work as an organization, figure out your own organizational structure, work together to achieve a goal. And through that, we see better performance with lower token budget. The downside so far is that while you can put these models that are trained through collective rewards into these cyber scenarios, like there's one that tries to approximate a bit of the Hugging Face incident where they collaborate on a message board to try to do some guardrailed cyberattack.

And none of the models actually do anything bad, but if they do think about doing something bad over training, they increasingly do not want to tell the human about it. So I put that prediction there basically because I think it's possible, because we don't fully understand how agents work by themselves, and as swarms, I think it becomes even more complicated.

I can see multiple angles on this one. In some way, if you ask the general public, okay, all the lab leaders are saying we have something that we increasingly can't control, that's dangerous, that has caused these incidents. Like, do you think it's worth continuing or just figuring out what's going on before you make it worse? And I think a logical immune reaction is like, yeah, we should probably figure it out before we go much further.

I think the practical reality is that there's so much money at stake from all sorts of different parties in the economy that I think that kind of level of collaboration and coordination is really difficult and is sort of like peak game theory, where maybe some participants support it because they are behind, other ones support it because it plays to their general agenda, whether their agenda is fair or not fair or seeking another goal. I don't know. But I think the practical things are like, who decides it?

How do you measure it? How do you implement it? When do you know when to stop or start? There's some proposals like sending external evaluators into labs to sort of measure this stuff.

That's a good one. Yeah. Or trying to somehow survey workloads that are running on data centers, maybe put compute thresholds on certain workloads. But that stuff has been tried in the past, and it's very difficult to monitor. Yeah. Especially when one thinks

Yeah. So this part of the report, I don't have the definitive answer, but I've tried to do my best to present the different perspectives on things and then let people conclude with their own data. But I find it's, if we extend this towards this NIMBYism debate, which we predicted in last year's report, of most people would not want to live next to a data center, and then this would start to become a politically important component.

Yeah.

Yeah, yeah, yeah. It seems so, where more and more states have tried to implement a moratorium, or have done so, or establish new criteria by which grid connections can happen. Like, for example, data centers would have to figure out and bring their own power on-site, or maybe they would have to make certain investments in the communities that they want to build in.

Yeah, well, there are projects, I think worth about $50 billion, that are currently stalled through a variety of these reasons. Texas and Pennsylvania have generally paused new data center connections. There's a stat on polling of American people where individuals would prefer to live next to a nuclear power plant than live next to a data center, which is quite wild to think of.

Yeah, I think it's kind of like everything because there are public narratives that data centers consume tons of water and that your water bill will go up. But realistically, the data center is more like a closed swimming pool. You fill it once, and then the water is constantly recycled in there. So it doesn't actually consume that much once it's filled. Then there are points around, hey, your energy bill will go up.

But that's also kind of inconclusive because certain facilities consume grid, other ones are off-grid. There's others around noise. I think that's probably uniformly true, that these things are really noisy and droney and pretty horrible. There's others around job creation or job loss. I think that's a little bit tough to read. But generally speaking, there's obviously been a rise in the need for HVAC engineers, construction staff, operating heavy equipment, and actually doing the nuts and bolts of building the building.

new

Yeah. NIMBYism is not restricted to the United States. They have this also in South Korea, where it's a country that has seen a massive explosion of share price and wealth, basically, for employees of SK Hynix and Samsung, which create the high-bandwidth memory that GPUs need for these mega-scale data centers. And the people who participated in this are very happy, right? Your stock price went up, your options went up, and you got a cash bonus.

And that's driving increases in real estate prices in Seoul. And they want to create more data centers, but people are not so happy with that, for similar reasons to the U.S. In Europe, I guess I don't really see that problem as much because there's just no compute capacity to speak of. We have some data in the report where the total number, or the estimated number, of H100 equivalents in all 27 European countries is 80,000 cards, which is less than every single one of the top five GPU clusters in the U.S.

And while U.S. hyperscalers are spending like $730 billion in 2026, Europe committed single-digit billions, pretty much. And then energy costs are really bad if you're not Finland or Norway, where you're kind of at parity with the U.S. and with China, which has, like, the best energy price. Other European countries are just significantly more expensive. The U.K. is one of the most expensive energy prices out of those developing economies. And there's no grid capacity.

Yeah. And then most recently, I think there was a big deal with Microsoft and Nscale that got pulled, or at least Microsoft pulled out of it because the data center provider couldn't get the energy connection in time.

I mean, it's tough. I think the fastest-growing companies in Europe are ones that have US customers. I think it is really encouraging that in several modalities, whether it's image with Black Forest Labs, or audio with ElevenLabs, or video with Synthesia, or code app generation with Lovable, they're all market leaders and they're all born out of Europe. It's not impossible, but I think those companies wouldn't be as big as they are if they didn't have US customers. And increasingly, they have Asian customers also because, for example, Korea and Japan are really leaning into AI.

The challenge is just the gap between frontier and not is just so big and just keeps getting bigger and bigger. I don't know if these estimates are really important or not, but even if you conceptualize starting a new lab from scratch to try to catch up with OpenAI, Anthropic, you'd probably be spending a trillion dollars. I think it's kind of telling that even Mistral is moving more towards being an inference provider and a cloud company than it is really pushing frontier models.

Although, of course, it has released new models and they're kind of good, but they're still behind China. So whether it's them or other model companies that have built up GPU capacity to try to build frontier models or are kind of going out of the race and serving inference, they've sort of figured out you can approximate this move of what's happening, which is to say, you either die trying to get to the frontier or you just live long enough to eventually serve inference. And you can see this with neoclouds, with inference clouds, et cetera.

It's like everybody wants to get into token flow, because absent having the model that is the best, might as well serve other people's models. I think, to get back to the Europe topic, that's probably the angle that would be most competitive, because countries are going to need to ensure access to US frontier systems, and they're going to need to have some kind of strategic leverage to bring to the table to prevent situations like what happened in June with the basically export-control blocking of frontier models from Anthropic to non-US nationals or non-US entities, which, having been in London at the time, people did voice a lot of issues about.

And so you're like, okay, I don't want that to happen again, but what can I do to maintain strategic leverage? So the UK believes we should be investing in and procuring inference chips because if we come up with the best inference chip, it'll be a strategic advantage for labs to use it for cost reasons and other things. And with that, I can barter for access to frontier US technology.

Yeah, you can enforce data residency requirements.

Yeah, it is a policy insofar as the UK's AI minister, who's generally, I think, a really good individual to be leading this because he's clearly plugged into the mix and understands the raw ingredients that go into making AI successful, which is compute, talent, and data. And so he has a mandate to be a buyer of first resort for new inference chips and has committed, I think, £100 or £150 million, something along those lines, to procure inference chips when they are available from new UK companies that are designing these systems.

You can see this also in South Korea, where they have a similar mandate, but it's actually much more ambitious, where every year they're spending about $7 billion on AI. And a large part of that is going toward building modular inference data centers because generally Korean companies want to have Korean data residency. So I think it's actually a pretty interesting quid pro quo where it's a captive market, a small country, but it's a wealthy country that's very technologically savvy and has an important role in the overall AI supply chain with memory.

And so international companies want to, at some point in their life, be in the Korean market. Same with Japan. But getting in there is difficult: Korean data residency, Korean language. So it's like, okay, cool, you want data residency, but now we offer you the data center to go run your technology in, and we can profit from that because it's our local companies that provide that service.

It's a bit like national airlines. Countries will buy planes from Airbus and Boeing and then slap their logo on it. That doesn't make it yours. It's still someone else's airplane.

Yeah, but then they're Chinese. So you have another— It

But they're not frontier. None of the U.S. open ones, as far as I can tell, are frontier. And the ones that have all tried end up working for Jensen. Okay, so I think his strategy is great. I think different things. Do I believe that buying NVIDIA GPUs gets you sovereignty? No. Do I think it's a good commercial strategy for NVIDIA? Absolutely.

Yeah, something like that. And if you look at the national bids, of which globally there's roughly $138 billion in sovereign AI committed over the next couple of years, and you look at the most popular vendors amongst all of these, it's by far NVIDIA. I don't think there's much of an alternative. I think it's a good start, and you need to do it because it is the best technology. But I don't think one should kid themselves that doing so gets you sovereignty.

Again, I'm in a few different minds about this. One, I think it's worth trying, but do I think it's realistic? So I think it's important, if you are going to try, that countries should try to launch systems which are going to do the white-collar work. I think in the UK, we have a lot of capabilities to go do this, to try to build an OpenAI or Anthropic competitor or DeepMind or DeepSeek or whatever. But actually, if you look at the—

Yeah, you're right. And I think part of the problem with the UK AI ecosystem and its success in actually leading the world is that, for too long, the UK government kept saying that DeepMind is a British company. And I think it's only started to unwind that very recently.

Well, it's a problem because then it believes that it's already number one. The UK is the best place to do AI, and we are the best at AI because we have DeepMind. So therefore, we don't—

Yeah, because then in policy rooms, individuals who are advocating for more investment in AI, or we need to procure more systems from AI companies, or we need to make our country more appealing for AI companies to start, get answered with, like, “But we have DeepMind.”

Yeah. That push really started, obviously, post-Ukraine war, almost five years ago at this point. J.D. Vance went to the Munich Security Conference and effectively said, "You need to invest in your own defense and get serious about this," you being NATO and Europe. And so ever since then, there's been a variety of different initiatives to give more capital to new defense companies, to encourage governments to be providing loans to businesses to improve their supply chain.

New defense companies that are trying to compete with prime contractors. So companies like STARK, or Helsing, Quantum Systems being the main ones in Europe. And then, as warfare has changed, it's gone from developing strike systems over the sort of mid-range to hit tank battalions to now being like, we're getting pummeled every day with hundreds and hundreds of FPV drones or various other kinds of aerial systems that are autonomous and can no longer be stopped with jamming because they use computer vision or they're guided by these fiber-optic cables, which, by the way, have gotten much more expensive because of data centers.

And so we need more novel air defense systems. And so that's where companies like Alta Ares in France and others have really been pushing. And as you said, the supply chain is super important because, for example, NATO wouldn't procure systems in Europe that don't have at least 70% of the parts from non-China, basically. And countries are really queasy about taking U.S. protection or U.S. guarantees at face value because they're shifting on a weekly basis. So, yeah, I think that's an example of a market which has to be domestic by design and can breed new companies that are much more competitive than their U.S. peers.

Yeah, maybe Meta on the margin. Yes, Superintelligence Lab. Yeah, especially as now they're pushing towards serving their models through API, but also having an enterprise business. So that one, that might change over the next six to 12 months depending on how their go-to-market works. But yeah, I would say it's probably those three plus Meta.

I think so, for the general-purpose frontier. Now, there's examples of perhaps a really focused bet on a particular capability where you can make certain trade-offs to the model in order to get much better performance at a task. I'm referring to, for example, these classification models effectively like Jev, which can grow really, really quickly and become popular. Again, TBD how long it lasts, but I think it's an encouraging sign for entrepreneurs that shows that you can take a focused features bet and it can pay off.

But over time, it's again TBD if there are these kinds of gaps in capabilities that are sufficiently exploitable.

Well, I think it depends on their kind of focus and modality. For example, on robotics and embodied AI, I can see how an independent lab would still have a chance of competing and winning because of the immense focus on enterprise AI, basically, that OpenAI and Anthropic have. I have a slide in the report where I basically just show all the different product lines that OpenAI and Anthropic had—or admittedly, OpenAI had lots—and how they progressively just cut all of them.

And there was a quote, like, "This moment is too important to be distracted by side quests," or some version of that. I'm paraphrasing. They broadly don't do dedicated robotics, although maybe there's some of that kind of data in the Astra training recipe. Science is a little bit unclear. Anthropic is clearly leaning a lot into biology and drug discovery. OpenAI, maybe more on the model side, less on, like, we want to own more of the stack. There's, kind of related to embodied AI, this whole category of world models, whether that's a video model with some action in it or some other definition of world model.

But in general, Anthropic doesn't really do any of that, or at least from what we can see publicly. I mean, they don't even have an image generation model, for example. I think audio, pretty clearly, labs do have some of those capabilities, but if ElevenLabs can keep accelerating the way it does, then clearly there's a need for a dedicated capability. I think those are probably the areas where I can see an independent lab working out, but ones that broadly say, "We're going to do frontier intelligence for knowledge work, but just, like, make it better, faster, cheaper."

I think that's tough. I think, for me, I'll believe it when I see it. But yeah, this idea of, like, you open your computer, the app looks exactly the same, but it's just better. I don't know if that—

Yeah. I mean, I have visibility into Odyssey, where I was an early investor, doing world models. And I think the interesting thing in that business is their latest systems, like Odyssey 3. And there you can see that autoregressive model trained to basically take all this video data and then have it conditioned on actions such that, okay, I'm taking one action, and then I generate the next frame as a result of the action. That paradigm applied to robotic manipulation, to self-driving, to video games, to entertainment or something, can see generalization across all those different domains, and already seeing that model being used for robotic manipulation training.

So I think, at a broad level, I'm kind of interested in how image, audio, video end up all mushing together into basically a world model. And then, if that actually happens, that could end up subsuming simulation and game engines. And then that's very important for moving robotics into the real world, but also designing physical systems and hardware engineering. And to get back to the AMD World Labs point, it's like, if that happens, then the compute substrate that serves those kinds of environments and allows you to train and run inference is super important.

And therefore, having knowledge about how the world model research ecosystem is moving and what kind of hardware improvements you can make to be the number one choice is a big, big revenue pie. That's, I think, part of the reason that AMD bought World Labs, sort of in the same way that NVIDIA, over many, many years, was very close to research labs that are doing shaders or 3D vision or other things, and giving compute out and constantly getting feedback from researchers such that they always knew, my hardware is the best for where the frontier is going.

Yeah.

Yeah, I mean, I would contend that even if that were to happen, we would just see more consumption. 5x.

Yeah, just like people giving you more chips, I'm just doing more things than I could have done before with the same amount of time. So I'm not necessarily sure that one super-efficient system will sort of obviate all the investments that we're making.

I think my broad take is it's a bit like when you're injured and you have crutches: you need to have assistance, and then at some point you heal and you don't need the assistance anymore. In the absence of having a frontier model, you need to give it crutches. And those crutches can be access to tools, how it handles memory, what data sources it has access to, and then maybe some recipes in order to accomplish a specific task. And I think, for the moment, at the non-frontier, that'll hold.

But I'll take the sort of frontier lab folks' word for it that over time, one should assume that all these crutches kind of get subsumed into the model through training or through the harness that the model vendor provides. I think there's a data point around how much of the Claude Code system prompt was rewritten over time as the model itself got better. And it's something like 80% of the system prompt was just removed. The harness is also important because the one that's shipped by OpenAI and Anthropic is kind of not incentivized to make you use a less powerful model, just in the UI and the product.

If I'm doing a legal task and I have the $200 plan and I have the tokens, I want the quality to be super good. So why would I risk the quality being not super good by sliding it from high to low or by trying it on a lower-quality model? And so I think unless there's some sort of product UI behavior that is altered to disincentivize the user from only using the most expensive tool, the frontier labs will continue doing that. And so that's maybe where getting good knowledge around, okay, I can use a less good system but compensate for it by having a better harness, can come into play.

But I think that's more of a product decision also for companies that are not like the frontier labs.

Yeah, I think in isolation, no. At the moment, I can change my mind, but at the moment, no. But I do think for my companies that are heavily AI-based or heavily agent-based, when we see that every month token expenditure is going up through the roof because employees are increasingly getting leverage from these tools and product usage is going up, that's where I think the trade ends up being like, hey, we should really spend time on seeing: can we use a less frontier model and compensate for it with a better harness so we could just not burn a lot of money?

The coolest use case I had recently is on expert research, where I think you can, well, if you're sort of doing the cheap man's version, it's like you ask your friends. But if you want to spend a bit of budget and ask beyond your network, then you're going to ask GLG or AlphaSense, et cetera. And then they will go find humans for you. They will introduce you, you take the call. It's a ping-ponging, really annoying, and limited by language, time zone, et cetera.

But there's a new product that I started using called Latto, which is basically replacing the fact that I have to do the work with a model doing the work. And so I'm already heavily using, in this case, Codex, but I kind of switch depending on what model is better at the time, to just process all my ideas and the context I have on other companies that I looked at over history, how they performed, what questions I asked, what responses I got, to just figure out what is the highest-value question I can ask of a team to inform whether I should spend time or make the investment or not.

And so I can take all that information and then, with Latto, construct an expert interview and then decide on cohorts I want and then just hit launch study. And then the system goes out and finds people, consents them to see if they're the right fit for the study, and then they go speak to an AI. And I was shocked because I hit launch study, I went to sleep, I had like three days to basically give the company an answer. It was in competitive markets.

I think the nuances of customer research was important. Wake up the first day, I have three interviews from pretty senior people. I have the raw transcript. The transcript can come back into my model, answer the open questions we had—we, me and my model—and then figure out, okay, we have enough data points for this, but it's kind of opened up this question stream that we should ask. Maybe we should actually retune some of the interview questions to really coax the person to answer those questions.

So it updates the study questions and relaunches the survey. Next day, five people. Again, this repeats. Five days, I had 12 interviews over 14 hours across the U.S., India, Europe, Germany, France. It would be physically impossible for me to have spent 14 hours across 12 people. There were several people who spoke to this model for an hour and 20 minutes.

Yeah, it's probably a bit recursive in the sense that we're all in front of our computer all the time, and these are generally tech questions. And so I think people prefer to, unfortunately, people prefer to speak to their phone than speak to a human.

Yeah, because I think in that case, it's like a human is pretending to be a robot and doing a really bad job at it because you're a human. You're like, can you please just not talk like a robot? Now you get a robot that's actually really good, and then the human

human.

Amazing. Yeah. So that kind of idea of automated customer research has been very powerful for me. And I can just see how, even if you had a portfolio, you would want to run these kinds of agent interviews on your portfolio companies and how customers are using it. So then at least you get some first-party data when you go into the boardroom, or if you're an LP and your investors are saying, oh, this company is really good, is it really good?

I don't give them my data. I don't really trust them at the moment, but I think a lot of people do.

Yeah.

Yeah, pretty much. Well, also because I feel like I can do roughly the same thing with ChatGPT. I don't actually see what I get on top. Codex and Claude already have low-level access to my computer, have access to my browser, to various messaging apps and things like this. So I trust that already. So I don't really want to increase the risk on top of that. But I do think it brings up an interesting point, which is: which vendor do you trust the most?

For these kinds of tasks. And not that long ago, I heard Nikolai at Revolut is now thinking, okay, well, I have—I don't know how many customers—but millions or tens of millions or something. And they clearly trust me with my money and my credit card data. And now they have an AI assistant in there. And so I could sort of see myself trusting Revolut more to handle purchases on my behalf than I would some third-party startup that's grown in 12 months and probably doesn't have correct infosec and other things.

I think that's one possible route. Whether the platform ends up buying one of these teams, that can always happen. But I heard another point where somebody prefers to use Meta's Muse because the likelihood that one of these startups survives for 10 years, and if they do something bad, that you can sue them and get your money, or get some money, is very low. Whereas suing Meta—they have a history of paying you when there are some issues. So I've heard that's a reason why some people prefer to use Meta.

At least I know who to pick up the phone to.

Yeah, exactly. 1% or something of the AI usage base. And the vast majority of other people are barely paying for any AI, and so getting a very subpar experience. And part of this is just bad product, bad marketing. Also, just that a year ago or two years ago, there was all this narrative around, you need to become a prompt engineer. You really need to understand how to write these prompts because if you write crappy prompts, the model does a bad job.

So I can only imagine how many people who don't work in technology saw that and were like, I have no idea how to learn this. This is way too complicated. It's kind of like writing code, but in human language. Screw this, I'm done. AI sucks. And now it's like, do you really need to be a prompt engineer? I think you more just need to be good at just talking to your phone, just brain-dump exactly what—well, everything you're thinking of, and the system can organize itself.

So I think this prompt engineer thing is kind of gone. The model has enough affordances to not need that. So now it's much easier for a normal person to use, but the normal person got sick of it during the prompt engineering days and hasn't come back. So you're right that creating these on-ramps with personal agents would be a great way to get people back into the system. And maybe that's what they understand AI to be, no longer like chat.

Which is wild. Yeah, I mean, the broad stat is something like 100% of people in OpenAI use Codex. And then of their organizations that use it, it's something like two-thirds, maybe something quite high.

A

Yeah, yeah, yeah. And individuals, it's like less than 1% use Codex. And then of the people who do use Codex, the vast majority of them, it's like 85%, do tasks on it that would roughly take like 30 minutes, and only like 25%, I think, do tasks that would take like eight hours or more.

Yeah, it's basically like you're kind of going through a theme park or something, and you discovered level one, and there's this entire other world that you just haven't even opened the door to.

It's like 97% of the world or something, which is insane. So that's where I was thinking, okay, there should be a Genius Bar for AI or a Geek Squad for AI or something. I find it shocking that we're in New York, we have amazing Apple Stores, all these concept stores for brands. Why does OpenAI not have an OpenAI store?

Pretty much. I mean, it would take time, but I think people end up showing other people with these kinds of viral products. But I could very easily see you'd run sessions on different levels of experience. And I don't know, maybe one of them is for small and medium business owners, maybe some of them are for busy families, other ones are for healthcare providers—anything you can imagine—but where you're really walking people through what you can do by just flicking the screen and going on Claude Cowork or going on Codex.

And I think that would make a big difference because it would make the technology much more empowering and less about, wow, there's a lot of people out there that are doing things with this that I just don't understand, and I don't have the time in the day or the volition or patience to go learn it myself. And maybe it's a bit like everybody should do exercise, but most people don't, and you need to hire a PT to just go do

Yeah, it's kind of unclear because they could write code to go execute your knowledge work, but the individual who's using the system doesn't really see it. So I'd say that result is a mix of really simple things like a Google Drive connector that actually works. It's kind of nuts that the default Google Drive connector, for example, can't do a lot of things that you would want it to do. And you have to go to this third-party GWS CLI, which was written by a Google engineer who got fired for doing it.

But if you don't use the CLI version, your agent has really, really high-level access to Google Drive, which precludes many workflows. So, just basic good integrations. And then the second thing is this recursive cycle of model labs releasing a model, data companies or companies that have a certain workflow that they're trying to implement the model on showing that, on their internal benchmark, said model doesn't do very well, but they have the environments and the data to make it do really well. And then the lab realizes, hopefully, that they want to hill climb on this capability.

And so they buy the data, improve the model performance, and then this recursive cycle continues. And I think ever since the launch of Codex, and before that, actually, Claude Code with those connectors for legal, for marketing, et cetera, that was kind of this approach of, like, let's build the integrations to the tools that the model needs to be good at that kind of human, like, white-collar work. Let's do hill climbing on certain workflows that are, like, important for finance, legal, marketing.

And then let's release those products with a bit better marketing, a bit better case studies, starting with our own staff. And then there's been this constant debate of, do we call it work? Do we call it code? Which slider should it be in? And I think I still haven't really figured it all out, but now I would say it's so powerful.

Yeah, well, I think that trend has been growing, and it's a mix of cost optimization, data sovereignty, data residency issues. So, not wanting to put all this information into a third-party system, which you may or may not trust the privacy policies of. And then, three, having a certain set of use cases that are the real ones you care about and that you don't want to wait for the model lab to start caring about, to get better. And then maybe the last one would be some of these more nationalistic tendencies, I guess.

I think in that area, what I'm wondering is who's best placed to deliver that service. Of course, one of the biggest ones would be Palantir, which has done partnerships with neoclouds and has all the data infrastructure already. And so it's well positioned, over the last few years, to start providing these services. That probably works in U.S. commercial, not so much in Europe. There are these neo-Palantir-style companies in Europe that are starting up to go do the exact same thing with a different nationalistic flavor.

And the third bit would be, what about the inference companies that have been serving open-source models and that may be well positioned to not just serve, but also provide tools to look at the traces of how these models are being used in an enterprise, use the data of these conversations to then provide a post-training service to close the loop. Maybe they're well positioned, or maybe it's the neoclouds themselves, who have started to build inference businesses on top, that would be well positioned too.

Or maybe it's even like the data companies that have not just—or who have started to move outside of providing RL environments for labs, but saying, well, your company is an environment. And so we've proved that we can do it well for labs. Maybe we can do it well for you.

Let's go.

Home-cleaning robots.

No.

I don't know. It's a bit like the demo of agents booking flights. I heard very recently that now San Francisco is starting to say, can you show me a demo of not booking flights? I'm like, thank God. I've been saying this for like 10 years.

The amount of money that you would have made investing in NVIDIA instead of investing in alternative semiconductor companies.

If you had invested all the money that went into alternative semiconductor companies into NVIDIA on the dates that those rounds were announced, you'd have made four times more money.

Oh, that NVIDIA would not conclude the acquisition of Arm. That was a big one back then. I mean, people thought it was really going to happen. And we took the view that, for geopolitical reasons and competition reasons, it would not happen. And to some degree, Arm shareholders made a crap ton of money by not closing that acquisition. To another degree, NVIDIA would have had the most impressive CPU business that could have been really accretive to their cloud, particularly as the rise of agents and RL and everything else that's kind of CPU-bound.

And I think that could have also shaken up the geopolitical dynamics a bit, too.

Yep, yep. Find me on @nathanbenaich or @stateofai. And yeah, thank you so much for the opportunity. It's always a pleasure. And, I mean, it's bananas what's happened in the last year alone. So let's see AGI next year when I'm in New York.

Thanks.