An inside look at “Mastering AI” | Jeremy Kahn, Author & AI Editor, Fortune

Why the UI design is important for AI?

Matt Turck [1:26] All right, Jeremy, welcome to The MAD Podcast. Good to have you. We are going to talk about your book that just came out literally three days ago.

Jeremy Kahn [1:28] That's right. Yeah, three days ago.

Matt Turck [2:00] Called Mastering AI. So you're the AI editor for Fortune, and actually, one of the things, as I was reading the book, one sentence that I really liked is that because you've been covering AI for a number of years now, when ChatGPT came out in 2022, you were actually surprised by the surprise, by how people were surprised by the impact of ChatGPT. So you had been—just maybe walk us a little bit through your background and what you did around that time.

Jeremy Kahn [2:24] Sure. Well, first of all, thank you for having me on the podcast. Happy to be here. Yeah, I've been covering AI for eight years, first at Bloomberg and then at Fortune. And I had been watching OpenAI very carefully. I'd actually, even prior to ChatGPT, written back in late 2019, early 2020, a cover story on the corporate race for AGI, kind of looking at OpenAI versus Google. And so I knew the company pretty well, and I'd been following—I covered GPT-2, and I had covered GPT-3 and was kind of monitoring what was going on with GPT-3.

Jeremy Kahn [2:57] I'd played around with it a little bit in the OpenAI Playground that they had, and that had some functionality that was very similar to ChatGPT. You could prompt the model to act in a dialogue, and you could have a conversation with it. And so I was surprised when they debuted this chatbot interface, ChatGPT, and it really struck people so differently than the underlying model had. And I think it was just the fact that it was kind of out there in the public, it suddenly got this attention.

Jeremy Kahn [3:16] It was so easy to prompt into a kind of instruction-following mode, whereas the original model, I don't know if you've played around with it at all in the Playground, it was a little harder to get into that mode.

Matt Turck [3:17] Yeah.

Jeremy Kahn [3:39] It had all of these additional settings and kind of dials that you could tune, which were very useful if you were kind of an expert trying to figure out how to get the model to do something particularly useful for your application, but were actually very confusing for a consumer. You could set the temperature of the response, and it was on this funny scale, and there were a bunch of other parameters you could adjust. And ChatGPT sort of stripped all that away, and it just had the dialogue interface, and you could just talk to it like any chatbot.

Jeremy Kahn [4:10] And I guess maybe I was foolish to have been surprised by how effective that was. But yeah, I was surprised. I thought, oh, this is just the same thing that was there before, this instruction-following model. But it struck people very differently. And I think it shows the power of interfaces, actually. And one of the points I make in the book is I think we should pay a lot more attention to the interfaces around which we wrap these models. I think the utility of the models, to some extent, for enterprise applications will depend in a large way on the exact interfaces and the exact design of those interfaces.

Jeremy Kahn [4:27] And we get very caught up, I think, talking about the underlying model capabilities and don't spend nearly enough time talking about the interface design.

The book is called "Mastering AI". Why?

Matt Turck [4:44] Great. Let's definitely go into this in a minute. Maybe to help frame the conversation, the book is called Mastering AI. Why that title? What is the underlying thesis behind the title?

Jeremy Kahn [5:11] Yeah, so the book's called Mastering AI: A Survival Guide to Our Superpowered Future. And I wanted a title that would capture, I think, to some extent, people's anxieties that we're going to lose control of this technology in some way, that it would pose a risk to us, and to say that essentially, no, this is a technology we can control, we can master. And I'm going to hopefully illuminate some ways I think we can do that. And then the subtitle, A Survival Guide to Our Superpowered Future.

Jeremy Kahn [5:36] I really wanted to capture both elements of the concern that this is a very powerful technology that has inherent risks. But I think there are ways to mitigate those risks. And I think if we can mitigate those risks, then it really can give us all superpowers. I think it can do incredible things to make us more productive and to be really transformative.

Matt Turck [6:12] Yeah. A sentence towards the end that I really liked, where you say, “Too often we mistakenly view technological development as deterministic, as if the technology were a force of nature immune to our actions. This attitude robs us of agency and turns us into mere subjects. It is a dehumanizing pose.” And then you go on to say, “One of our defining characteristics as a species is our ability to bend the world to our favor, and AI is bendable.” I thought that was beautifully written.

Jeremy Kahn [6:15] Well, thank you.

Matt Turck [6:37] All right, so that's a high-level thesis. I thought maybe, to give people a little bit of a flavor for some of the topics you talk about, we'd go through some of the things that caught my eye without revealing the whole thing, because obviously people should buy and read the book. One topic early on that I thought was where I would really like your take was sort of the Her thing, the general idea that AI could be a companion or possibly a therapist.

Matt Turck [7:20] My view, for what it's worth, as a techno-optimist, is that actually I'm in the camp of people that say, well, in a world where a lot of people are lonely, that's actually better than nothing. And actually, maybe AI will be able to listen to you in a way that nobody can. But I think you have a different view. You view it more as dehumanizing. Is that fair?

Jeremy Kahn [7:44] Yeah, I'm worried about it. I am concerned. I think you already see this a little bit with users of things like Replika and, to some extent, Character.AI. And there's an attractiveness to these companion chatbots. There seems to be a certain subpopulation that really enjoys interacting with them. But I worry that they become very addictive. It becomes very easy to kind of have this replace actual human companionship. And I think for people who are already lonely, you could say, “Oh, well, this is better than nothing.”

Jeremy Kahn [8:13] These are people who have nothing right now. Isn't this better, that they have this outlet and something to talk to? On the other hand, I do think it tends to be a crutch, and people will then say, “I have this, and now I have this outlet for my emotions and my expressions. I can unload my thoughts about the day.” And they are not going to seek out a real human relationship. And I think that's detrimental. I think there are other people who say, “Well, these things are good practice for real social interaction.”

Jeremy Kahn [8:41] I think that's interesting. But then the model, I think, and then this—the companion chatbot—should prompt people to go out and actually, okay, now we've tried a little practice. Now go out and see if you can do this in the real world with a real person. I worry that people won't use it just for practice, that they'll use it as a kind of replacement for real social interaction. And I worry also on the therapy side that this is very not proven.

Jeremy Kahn [9:06] There's been some research on using chatbots for therapy, mostly, though, in conjunction with human therapy, so as a kind of add-on as opposed to, again, a replacement. And the few studies that have looked at anything like replacement have been very underpowered. There are very few subjects in the study. So I'm worried about making broad claims that this is really great for mental health. And again, I worry it'll become a kind of policy crutch. You could say, well, it's better than nothing.

Jeremy Kahn [9:37] I worry it's very easy for governments to start saying, “Oh, well, we'll just provide everyone with a chatbot. All these people with severe mental health issues, we'll just give them a chatbot. They can talk to the chatbot therapist.” And it will become this crutch for not actually funding mental health services properly and not trying to do more to make sure that these people are seen by trained professionals. I've seen no studies, actually, that look at use of a chatbot versus use of a human therapist.

Jeremy Kahn [10:06] I've seen ones that say, “Is a chatbot better than nothing?” And it does seem like a chatbot might be better than nothing. But I don't think we've seen studies that try to see if there's equivalency or even a better effect from the use of a chatbot in a therapist role versus a human therapist. And until we sort of see some of those studies, I'm very skeptical of how great an impact this will have on mental health.

Matt Turck [10:16] Talking about copilots, which you have a chapter about, you say we're all middle managers now. What do you mean?

Jeremy Kahn [10:36] Well, I think that's one mode of looking at these things. And there's been several other people who have written this phrase, that it's turned us all into middle managers. I guess it's the idea that copilot technology can be used as a kind of junior colleague, junior assistant, that it will do the sort of first draft of things, and we will supervise its output. And that puts us all in a little bit of a management role where we're not doing the work ourselves, but sort of overseeing the work and supervising the work and correcting, course-correcting the entity's efforts.

Jeremy Kahn [11:04] So I do think there's a way in which you can think of copilots in that way. But as I point out in the book, there's also this other way of thinking about copilots, which is they can both be a kind of junior colleague, but also you can prompt them to act as a kind of senior mentor. And instead of having them do the first draft, which you then oversee, you could do the first draft yourself and then ask the copilot to sort of critique what I've done.

Jeremy Kahn [11:33] Or here's the sales pitch that I'm thinking of using. What would you improve on from that pitch? I think they have tremendous possibility in both roles. Either way, I guess it puts the person in the middle between these two layers. And I guess that's what I meant by sort of, we're all middle managers now. Of course, middle management gets a bad rap. And I think some other people who've written about this have talked about that idea in a very pejorative sense.

Jeremy Kahn [11:52] I actually, in the book, say I think it depends what your view of management is. If you're already in a role where you're managing people, I don't think it's that different to be managing people and managing the output of a copilot.

Automation Bias vs Automation Surprise

Matt Turck [12:12] And a little bit to the earlier point about UI, another one of the many sections and tidbits that I found super interesting that really caught my attention was this idea of automation bias versus automation surprise. Can you talk to that?

Jeremy Kahn [12:34] Yeah. So I think as we design these systems to work within enterprises as copilots, there are these two human cognitive biases around technology that we really need to be careful of. One is automation bias. That's the tendency of people to become overly reliant on technology and to assume it is right, even in the face of data that should prompt us to think it's not right, that it's making a mistake. The other is automation surprise, which is when an automated system goes haywire, it often takes humans a lot longer to figure out what's gone wrong and to recover back to some sort of manual process than it would in the case of a mechanical failure, for instance.

Jeremy Kahn [13:13] We seem to have less intuitive understanding of how these systems work. And I think our tendency to assume that they are right, and if you have an automated system that is mostly reliable, it is a problem when they fail because it's an unusual event. And unless you've been specifically and very well trained to kind of deal with that failure, there's a tendency to become very flustered by the failure. And I use the example of aviation, which has had more automation for longer than most other—

Matt Turck [13:19] Yeah, which has had copilots for a very long time.

Jeremy Kahn [13:39] Yeah, they literally have had autopilots. The first autopilot, it was fascinating when I did the research, came along just about four years after the invention of the airplane. There were already people thinking about how could you hold the control stick in the same place so you don't have to constantly have your hands on it, and they've of course only gotten more sophisticated from there. And now in modern flight, so many of the systems are automated.

Matt Turck [13:41] And the tragic example you gave is the Air France flight.

Jeremy Kahn [14:03] Yeah, I talked about the Air France case, but there have been a number of these cases. Actually, I looked at several major airline crashes for that part of the book where exactly this happens. The pilots are very trusting of the technology, and when it starts to go wrong, they become very flustered. Even in the face of clear evidence of what the problem may be, they cannot diagnose the problem correctly. They often then get in a situation where they start panicking, and they often take actions that are the exact opposite of what they should take.

Jeremy Kahn [14:36] I think some of it—in the book, I talk to folks at NASA who've had to deal with automated systems for astronauts, particularly as these missions are getting longer and longer. They're looking at more and more automation in spacecraft. And they have also looked at these examples from aviation and been very concerned about the idea of automation bias and automation surprise, and done quite a lot of research about how you would overcome these things. But ultimately, what they said is it really just depends on making sure that the astronauts have this really good mental model of how the system works, and what it's capable of, and where its potential weaknesses are.

Jeremy Kahn [15:12] And then also constantly drilling the potential failure modes in simulation. And airlines do this too, to some extent, but not as they do when you have astronauts going off in spacecraft. And I think, actually, it's a good example for any business that's going to start using this technology. First of all, the training of the people in your enterprise that are going to use this technology is as important as the training of the AI model.

Matt Turck [15:13] Mm-hmm.

Jeremy Kahn [15:33] I think we talk a lot about training AI systems. I think we need to start talking a lot more about how we train people to work alongside these copilot technologies. And part of that training has to be a clear understanding of where the model is likely to be weak. So you have to have a good sense of that data distribution and performance distribution. So where is it likely to fail? What would be some potential indicators of failure?

Jeremy Kahn [15:49] And really training people to pick up on those, and then also constantly drilling a kind of fallback mode, making sure that people are not completely deskilled by their use of this technology. It becomes almost part of your business continuity planning in some ways. Same as if you might plan for a major electric outage in a city, you'd have to plan for the fact that if your model starts behaving very strangely and you have to say, okay, now we're going to fall back on people writing the sales pitches themselves or taking whatever the series of workflow actions are needed, that people know how to do that.

Matt Turck [16:09] And that's true in the enterprise, and that's also true in society in general.

Jeremy Kahn [16:17] I think particularly in the enterprise, but I do worry even broader about deskilling as these AI models become more powerful and more ubiquitous.

Matt Turck [16:29] Yeah, I mean, you go into a lot of this. And pretty much the worry is, like, AI is going to make us dumb, right? Is that fair?

Jeremy Kahn [16:50] Yeah, exactly. I do worry that AI is going to hurt our human intelligence. Again, that overreliance on these things can make us a little bit less sharp than we used to be and certainly lose some cognitive skills. And I think there are examples of that from previous technologies, as I try to point out. I think it really is true that Google has hurt our memory. I think people memorize far less than they used to. And I think generally that's fine, but I think you have to realize that that's a trade-off, because I think Google and the ability to have the world's information at your fingertips like that is a huge advantage.

Jeremy Kahn [17:14] But you have to realize that people memorize less. And then there's some interesting studies on learning and memory that the more you actually know of a subject and have memorized about it, that tends to help you to achieve kind of breakthroughs in that field.

Matt Turck [17:21] Yes, yes, yes. You say in the book, committing factual information to memory enhances rather than detracts from cognition.

Jeremy Kahn [17:21] Right.

Matt Turck [17:42] And speaking of skills, there is a section called "Winner Takes Most," where you talk about AI's tendency to help the best performers across professions. So you're going to end up with a star system for lawyers, for doctors. Do you want to talk to that?

Jeremy Kahn [18:06] Yeah, sure. I think that's going to be one of the effects we see of this technology, is that in industry after industry, it will create a kind of winner-take-all economics where, enhanced by this technology, you have the stars of a field being able to charge a real premium. And in part, that premium will be based on how much better they are than the average person assisted by this kind of AI copilot technology. I think one of the things about the AI copilot technology is it tends to have the biggest impact on kind of lifting less experienced people up to the average performance.

Jeremy Kahn [18:36] So that means that I think there'll be many more people who can kind of perform at an average level, and their ability to command—in any sort of market economics, if you increase the supply, the price tends to drop, right? Assuming demand is constant. So I think what will happen is the price of the average value of legal services or of accounting, of marketing will go down. But if you can perform above average, I think people are going to put a real premium on that because that's still going to be a very rare skill.

Matt Turck [18:44] Mm-hmm.

Jeremy Kahn [18:57] And the fact that you can value your human labor that much more highly, I think, in a world where most people are able to achieve average, the ability to achieve well above average will command a premium.

Matt Turck [19:05] And what does that mean, achieving above average in the world of AI? Is that, like, profoundly human aspects?

Jeremy Kahn [19:18] Yeah, I think there are judgments. I think there's—yes, I think there's human judgment, things that are sort of born of experience, I think. But I also think things like, in a lot of professions, your professional network, which AI will not replicate. So you may—

Matt Turck [19:18] Oh, interesting.

Jeremy Kahn [19:32] Yeah. So, like, the ability—I mean, I think about banking or something—the ability of a dealmaker to just get on the phone right away with the best people and know exactly the right people to put together to make the deal happen. That's a skill that AI can't replicate. AI may be able to do all the legal drafting of the documentation, and it may be able to do all the presentations you need for the investors, but it's not going to actually get on the phone and get the right people in the room.

Jeremy Kahn [19:49] And I think that human connectivity that top people in a lot of professions have, that becomes even more valuable, that kind of human social network.

Matt Turck [20:06] That's fascinating. So the recommendation for a young, upcoming person in the profession would be to understand the techniques, but focus on the non-technical aspect of it, almost like networking, human judgment.

The role of AI in the future of science and art

Jeremy Kahn [20:16] Yeah, I mean, well, I think you need both. But yes, I think the thing from which value will be derived is a lot of the non-technical aspects, which are about human skills.

Matt Turck [20:35] All right, so maybe taking some of the sort of big topics and just doing a quick review for some of them, we can start with anyone you'd like, but you go into AI and workforce, AI and art, AI and science. Maybe let's start with AI and science. What's the high level?

Jeremy Kahn [20:59] So in general, it's the area I'm probably most enthusiastic about AI's potential effects. I think AI is this sort of super tool for science. I sort of compare it a little bit in the chapter to the microscope for biology, or the telescope for astronomy. It's this thing that is going to become an essential tool for science. The idea that you could be a biologist without a microscope is sort of anathema. And I think within 10 years, the idea that you can be in any science without using AI models to help assist what you're doing will also be anathema.

Jeremy Kahn [21:30] I look a little bit at particularly the drug discovery case. I think what's happening with AI models for genomics, for chemistry, for enzyme and protein design is incredible. And I think we're going to very quickly start seeing the results of that in better drugs designed faster at a lower cost. And I think that's tremendous impact. But I think it's sort of across the sciences. I think it's in chemistry and in material science, where I think AI is going to help us find new materials that will have transformative effects on our lives, help us be more sustainable.

Jeremy Kahn [22:07] I think we're seeing it, though, even in the social sciences. There's very interesting things about using sort of large language models as stand-ins for large populations. And because they ingest so much human-written data off the internet, they have a very good sense of what the general public belief is in a certain thing and how the public would generally respond to certain questions. They're actually pretty good, the studies so far on that. And so you get this idea of sort of synthetic polling, which is a very interesting idea, which again could potentially save a lot of time in doing large social scientific research.

Jeremy Kahn [22:41] So I think there's lots and lots of ways that AI is going to help us in terms of science. The only drawback I think at all in the sciences, which I mention in the book, is in many cases, these models are really, really good at picking up all these subtle correlations across huge datasets. Right now, it's not always very easy to figure out why the model thinks that a particular pattern predicts a certain outcome. And I think in a lot of areas, that's fine.

Jeremy Kahn [23:03] I mean, if you look at medicine, we have lots of things in medicine that work, and we use them because they work, and we actually have no idea what the mechanism is. And that was true of aspirin for years. We've been using aspirin for 100 years, and it's only in the last decade that we had any sense of what the mechanism is by which aspirin works. That's fine in some areas, but I think in other parts of science, it's problematic to have this kind of hypothesis-free science where you know that you can predict an outcome, but you have no idea of the underlying mechanism by which you're making that prediction.

Jeremy Kahn [23:25] I think we're going to have to try to catch up a bit of that gap between predictive power and sort of explanatory model power.

Matt Turck [23:29] And for art, you're similarly optimistic but nuanced?

Jeremy Kahn [23:52] Yeah, in art, I'm pretty optimistic. I think these models can replicate certain aspects of human creativity, but not all aspects of human creativity. So that still allows lots of room for humans to do things in the arts. I think we're going to see several sort of interesting cross-cutting trends in art. One is that I think a lot of artists are going to start using AI models to sort of assist them in the creation of their art. And I talked to some visual artists in the book who are doing this, and I think with interesting results.

Jeremy Kahn [24:20] And ultimately, it depends on knowing that this is not just like taking DALL-E or something and giving it a prompt and then, oh, look, I'm going to go hang the resulting image on a gallery wall. It's quite a lot of human input. So you might use the AI in some aspect of the creation process, but ultimately, there's quite a lot of human hand and human creative thinking that goes into the art. And I think that's going to be what people value, what collectors value.

Jeremy Kahn [24:44] And I think that ultimately having that kind of human hand and human thought process involved will resonate more with audiences. So I think plenty of space still for sort of human creative value. The other thing I think is maybe we'll see an emphasis on art forms that have this kind of physical presence, so sculpture or live music. I mean, we've already seen with music streaming that there's more and more emphasis on live music.

Jeremy Kahn [25:15] I think that AI might sort of accelerate that trend as well, because people will actually want to go see the artist, the musician, actually perform. Then you know that it's created by the artist and not by AI, because you're actually witnessing that process. And you also have that human connection again. I think it's sort of the artists who are able to offer some physicality to what they do and some authentic physical presence and human connection. Those things are going to be even more valued in an era where anybody can go to some OpenAI or Google model and type a few words in and get a symphony or get an image that looks kind of cool.

"I think mass unemployment is a red herring, but we might see a lot of disruption"

Jeremy Kahn [25:32] But I think for art, we're really going to want something beyond that. But the good news is, I think there's scope for that.

Matt Turck [25:41] And in general, for the future of work, you are in the camp of AI will help us?

Jeremy Kahn [25:59] Yes, but I mean, again, I think it's a bit of a subtle point. I'm generally optimistic also about the future of work. I do not think we're going to see mass unemployment. I think mass unemployment is kind of a red herring. But I think we might see quite a lot of disruption. And I do worry that a lot depends on exactly how we design these systems. Again, talking about the interface design and also the framings, the kind of framing psychologically of the technology.

Jeremy Kahn [26:24] If we mostly look at how can these systems help our existing workforce to be more productive, complement what they do, really look at what is our human capital within our businesses, what is it really good at? Where can it add the most value? And then how can AI assist that human capital become more efficient? Then I think we're in a good place. I think then, you really do get this huge benefit in productivity without huge amounts of job loss anyway.

Jeremy Kahn [26:47] You might again see the role that people play changes, but you don't see mass firings of people. I think if we have businesses that really frame this as, like, how can we use this technology to replace the people we have on a kind of a substitute for their labour, one-to-one, then I think we're in trouble. Because then I think the temptation for a lot of businesses will be, well, let's find some low-hanging fruit where we can get an AI model that seems to perform about as well as our customer service support people or whatever it is.

Jeremy Kahn [27:16] And then, okay, well, let's fire 80% of our customer support people. And I think that's a kind of lazy tendency. A lot of businesses, they're very bad at retraining their existing workforce or anything. They'd often much rather sort of, oh, let's just get rid of that set of the workforce. And I am disturbed by that trend. And I think I very much in the book kind of have this plea that, like, let's think of this as a complementary technology and not as a substituting technology.

Matt Turck [27:35] Another area you seem to be worried about, for all the understandable reasons, is war and how AI is going to impact warfare.

Jeremy Kahn [28:04] Yeah, no, I think it's a really scary area, what's happening very quickly. If you look at the battlefields of both Ukraine and some of what's going on in Gaza, the use of AI to automate the finding and taking out of targets, in some cases the killing of individual people on the battlefield, is happening very quickly, and I think will spread, and potentially is very destabilising because these are very inexpensive systems. The software potentially will be open-sourced at some point. It can be matched with drones that can be kind of modified from commercially available—

Matt Turck [28:10] Consumer drones, yeah.

Jeremy Kahn [28:26] Consumer drones that can be very easily modified. I think this will be—it could be a huge weapon for terrorists, very attractive, potentially, for smaller rebel armies and sort of anti-status quo powers throughout the world. And I think that's potentially very destabilising and frightening. I also worry that the systems, in certain contexts, you could see why they might work well, but I think from what we know about facial recognition software and a lot of these sorts of sensor-based technologies, that they're okay, but they're not perfect.

Jeremy Kahn [29:06] And I think when you start putting them in crowded urban environments and ask them to only kill the bad guys and spare the civilians, that we're going to see mistakes being made, where the systems misidentify targets. I mean, some people who are advocates for this technology say, oh, it's great because they will—soldiers on a battlefield, there's a fog of war, and that's how you end up with civilians getting killed. These systems will be perfect. They won't be afraid.

Jeremy Kahn [29:30] They'll sort of pierce the fog of war and only target combatants and spare civilian life. I think there's very little evidence for that, and I worry that all it will do is actually essentially allow commanders to kind of escape accountability for their decisions. They'll say, oh, we'll just set the drone loose in this sort of zone, and we'll just trust that it's going to find the right targets. And I think we're going to see civilians being killed by this technology.

Jeremy Kahn [29:59] And ultimately, what's worrying, I think, when you take the human out of the loop in these systems, is always when the human's in the loop, there's this sort of chance of mercy. I think when you're dealing with another human being, it's true that humans make mistakes, particularly in war, but there's always this chance that they will recognize that, no, that's a civilian, that's a child, that's a noncombatant, and they will choose not to pull the trigger. And I think even there's this chance of mercy sometimes in situations with other combatants too.

Jeremy Kahn [30:27] There's this thing called the naked soldier incident, which happens in warfare. You may come across a combatant, but they're showering, was the famous example from World War II. And people choose not to pull the trigger. And I think that act of not pulling the trigger says something very key about our human ethics and our human morality and our sense of shared humanity. And I worry we lose all of that when you start automating all these killing decisions.

Jeremy Kahn [30:58] And some people try to keep a human in the loop, and I'm worried very often that it ends up being a kind of rubber-stamping process. The international human rights law says that combatants must exercise meaningful human control over the systems that they deploy in warfare. And actually, I think with some of these AI technologies, although we can claim that there's a human in the loop, they are not actually able to exercise really meaningful control.

Matt Turck [31:15] And that absence of mercy and the sort of mechanical aspect of AI just filling its purpose is also what yields, beyond extrapolating from war, into the whole human extinction fears.

Jeremy Kahn [31:39] Yeah, right. I mean, yeah, exactly. I think if we have systems that essentially have the ability to kill other human beings, and yet we don't really have control over them, that is where you start thinking of scenarios in which there might be some kind of existential risk. I mean, I'm not a huge—I don't think that the chance of existential risk from AI is particularly high. I'm not one of these. I have a low P-doom, I guess, compared to some people.

Jeremy Kahn [31:54] And why is that? But it's not nothing. I just think the systems we have right now are not capable enough to pose a risk of extinction. So I do not think they're capable enough to pose a risk of wiping out all of humanity. But it is possible to imagine scenarios, like I just said, with military drones, if you had a large swarm of those and you kind of set them off in the wrong place, that you could kill a lot of people.

Jeremy Kahn [32:25] I mean, I don't think it would kill everybody on the planet, but I think you could imagine a scenario where an entire town was wiped out by some mistaken deployment of a killer drone swarm. And that's going to be a human mistake. I mean, it's going to be our fault for having set that swarm against that location without regard for what could happen.

Matt Turck [32:25] Mm-hmm.

Jeremy Kahn [32:38] So in a way, it's not that the AI somehow developed consciousness and decided to kill us all. It's our own idiocy for deploying a system like that in an environment where there's a risk of catastrophic loss of civilian life.

Matt Turck [32:38] True.

Jeremy Kahn [32:55] So I think those sorts of things I'm very afraid of, and I think are very much on the horizon. But this idea of it could develop consciousness and then decide to kill us or see us as an impediment, I think that's a much lower risk. I don't think the systems we have right now are anywhere near developing consciousness. It's very difficult to see how that would develop, unless you really just believe that consciousness is a factor of scale, which some people do.

Jeremy Kahn [33:24] I just don't. I think we're going to need something else. I wouldn't say we could never have a system that was based on a silicon substrate that would develop consciousness. I mean, some people say consciousness is obviously just a factor of biology, and you could not imagine an electronic system or digital system that could have consciousness. And again, I just think, well, I don't know. I think the jury's out on that. But I also don't see any clear pathway right now to creating systems that would have that.

Matt Turck [33:29] Yeah.

Jeremy Kahn [33:48] And so I'm not that worried about that risk, but I'm also not 100% confident. So what I say in the book is, I think some smart people should spend some amount of time and some amount of resource looking at existential risk and how we could prevent it. But I do not think that that fear should be the basis on which all AI regulation is based. And I do not think it should crowd out efforts to police risks that are very real and here now around racial bias, around disinformation, around job loss.

Jeremy Kahn [34:10] I think those are things we should deal with now urgently. And yes, we should also have some people looking at existential risk, because if I'm wrong and there is a future danger, I would like someone to have headed that off.

Jeremy's perspective on Agentic AI

Matt Turck [34:26] You and I. And presumably there's an intermediary risk as we link AIs together. Do you have thoughts on sort of agentic AI or autonomous AI from that perspective?

Jeremy Kahn [34:51] Yeah, well, I think the more power we give these systems to actually take action in the world, obviously there's more risk. So I think as we move to agentic systems, reliability becomes a bigger factor. I would not want—it's interesting because I think we're very close to some systems with some form of agency. And yet our hallucination rates on current LLM-based systems are still relatively high, even with some of the best techniques. If we get hallucination rates down to, like, 3%, which is considered very good, I don't know if 3% of every time you send it out to buy something for you on your behalf, it gets it wrong.

Jeremy Kahn [35:05] I don't know. It's, like, a fairly high—

Matt Turck [35:07] And then it compounds if you have agentic systems.

Jeremy Kahn [35:25] Exactly. So I think we need to work on reliability. And I do worry about the impacts and risks from agents if we don't do more to improve the reliability of the systems. And I think there need to be limits, obviously, with agents. You need to limit more, I think, what they can possibly do, particularly at first, because of that risk. And yeah, once you start having things that can—certainly there's a lot of financial risk once you have things out there spending money on your behalf or conducting commerce on your behalf.

Jeremy Kahn [35:50] And there may be other risks as well about actions things could take. I think we already have these systems where you can say to it, "Oh, well, I want to make money really quickly. I've got $100. Figure out a way to turn that into $10,000."

Matt Turck [35:51] Yeah, I'd like one of those, please.

Jeremy Kahn [36:02] Right. Yeah. I mean, everyone would like one of those. And it would just be really easy for the system to say, "Okay, well, one of these things where you sort of don't tell me how you do it, right? Just go off and do it for me." And it would just be really easy for one of these systems, I think, even now, to say, "Oh, well, a good way to do this would be, like, I'll run a phishing campaign on your behalf, and I'll just create a bunch of spam emails, and we'll send them out to some huge email list, and we'll tell people to transfer money to your bank account and see what happens."

Jeremy Kahn [36:28] And of course, that's against the law, but you wouldn't want systems doing that. I just think even though we have laws that would potentially make you liable if a system did that, it's not something you want happening.

Does AI development need to be regulated?

Matt Turck [36:43] So we've alluded a little bit in the conversation about the need to do something from a policy standpoint. What is it? What can we do? Is that more regulation? Is that self-regulation?

Jeremy Kahn [37:04] Well, I think we need actual regulation, like government regulation. And I think it has to be both industry-specific regulations. So, looking at the use of AI within particular industry verticals, there will probably be very specific things we need to do that are very different in banking than what you might need to do in medicine, that are different again than what you would do in e-commerce. And I think we need some industry-specific regulation around some of these things.

Jeremy Kahn [37:33] But then on some of the higher-level risks, I do think it would be useful to have a kind of AI regulator at the federal level that has enough expertise to look over the shoulder of what the tech companies are building and make sure they aren't taking undue risks and that we aren't accidentally sort of tripping ourselves into a situation where the models do pose more of a kind of catastrophic risk. I think that's sensible. Again, I wouldn't want that agency to be funded more than the DoD or something, but a little bit of money towards that makes sense to me.

Jeremy Kahn [38:02] And having some sort of outside independent entity that can look at what's happening in the industry and draw some clear boundaries, I think, makes sense. I do not think that companies can be trusted really to do this completely on their own. I think there's just too much profit motive to take actions that ultimately will be kind of detrimental to us from a societal level or a personal level. And the one I like to use is just, I think, if you look at the business models around chatbots and agentic systems, as we start to get chatbots that have some agency and can act as actual personal assistants for us, I worry very much about business models for those systems that would be based around engagement and around advertising.

Jeremy Kahn [38:41] Because I think when you have this chatbot and you want it to do some shopping for you or whatever it is, you want its recommendations to be essentially objective, based on your own preferences that you've told it, and based on some actual research about what might be the best product or whatever to fit those preferences that you have. I think you do not want the system to be recommending that you buy Nike shoes because Nike paid OpenAI a lot of money to make sure that their shoes ranked higher in the chatbot recommendations.

Should we worry about the AGI and Superintelligence?

Jeremy Kahn [38:56] And I worry about a business model that might encourage that kind of behavior.

Matt Turck [39:22] So, a little bit to the superintelligence, destructive AI kind of point not being necessarily around: is there a world where a lot of those worries turn out to be just very premature, and ultimately the power of those LLMs is very impressive, but not that great?

Jeremy Kahn [39:46] Yeah, absolutely. I mean, that's why I said I don't think, and I actually don't in the book, I don't think just scaling up the current LLMs will get us anywhere. I'm not even sure they'll get us to AGI, let alone superintelligence. And I think for superintelligence, all the scenarios involve this kind of intelligence explosion scenario where suddenly a system decides what it's going to learn on its own and essentially starts training itself to get smarter and smarter. And I don't see any—there's no clear pathway to that right now.

Jeremy Kahn [40:11] And I think it cannot be a purely LLM-based system. And I talked to—I remember I interviewed Ilya Sutskever for the book when he was still at OpenAI. And even he, and he'd been a leading proponent of this kind of just keep scaling, just keep scaling, and even he was saying, last time I talked to him, that he thinks, no, scale alone will not deliver superintelligence, that it will take a few algorithmic twists. He said it still might look a lot like an LLM, but with a little modification here and there.

Jeremy Kahn [40:31] And he was very cagey about what he thought the modifications would be. So I actually don't know what he was thinking. But even he was saying, no, I don't think we can just take the existing LLMs and make them bigger and bigger and bigger and get to superintelligence, which I thought was interesting.

Matt Turck [40:47] And on sort of AGI, a step sort of below superintelligence, I thought, well, maybe using your opinion very quickly—I'm sure a lot of people listening to this will have a sense—but the difference between AGI and superintelligence, definitionally.

Jeremy Kahn [41:11] So AGI, which is artificial general intelligence, is usually defined as a single piece of software that can perform cognitive tasks as well as—and there's some debate over this last bit—as well as the average human. And there's some debate about that, whether that's a fair definition or whether it should actually be as well as sort of the best human in each of those categories. Does it understand physics as well as a physicist? Does it understand law as well as a lawyer?

Jeremy Kahn [41:42] But anyway, the point is that it eventually replicates and matches human intelligence. Superintelligence would be a system that is essentially smarter than all of humanity's collective intelligence. So the entire species combined, all of our brainpower, a system that would be smarter than that essentially is what people talk about with artificial superintelligence. And obviously artificial superintelligence would be something much harder to control because it would be something that would outmatch all of our collective intelligence. But I think there's no clear path that we know of right now to kind of get there.

Jeremy Kahn [42:09] So we probably don't have to worry so much about it. AGI, I don't know. I think, again, I'm not even sure the current systems, scaling them up, get us to AGI. When I talked to Sutskever about this, he said he thought you could get to AGI by scaling up the current systems, but it wouldn't be a very efficient way to get there, which I thought was interesting. And he thought there were probably better and more efficient ways to get there.

Who provided the most thoughtful conversation for the book?

Matt Turck [42:26] Actually, to the extent you can share, in the process of writing the book and having all those conversations, who were some of the most interesting folks and the most interesting conversations, the more surprising ones, or the ones that you enjoyed the most?

Jeremy Kahn [42:49] Yeah, sure. Well, I talked to, I don't know, hundreds of people for the book. I thought I had a really good conversation with Demis Hassabis from Google DeepMind for the book. He was very enthusiastic about agency, and I think in part because agency, when you start thinking about training AI agents, it gets back into potentially a zone for reinforcement learning, which was what DeepMind was always known for. And so maybe he's excited about it because it will allow DeepMind to draw more on its pedigree and its kind of background and expertise.

Jeremy Kahn [43:17] But yeah, I thought that was really interesting. I really enjoyed the conversation with some of the artists I talked to, actually, for that creative chapter. I mean, it was really interesting to see how people were using and incorporating AI into their creative process. There was, like I said, this painter I talked to—sorry, he's a photographer, actually. And he takes these huge, kind of monumental digital images, but then he feeds them through an AI system that creates interesting effects.

Jeremy Kahn [43:49] But then he applies a lot of sort of Photoshop skills and editing on top of that. And then also the printing of these very large photographic works and the way they're displayed is, again, kind of human input. So I thought it was an interesting kind of combination of human artistic skill and effects generated by AI. I talked to some writers who were kind of playing off AI systems in their work, and I thought that was interesting too. So that chapter was a lot of fun to work on.

"I didn't use AI for the book at all"

Jeremy Kahn [43:57] Yeah, I don't know. I enjoyed the process of researching the book. I had a lot of good conversations.

Matt Turck [44:05] Yeah. Do you use any of the systems for the book, for your articles, as a journalist and editor?

Jeremy Kahn [44:26] Yeah, I figured that would be a question people would ask me. I really didn't use it for the book at all. I didn't use it to do any writing. In part because I just find, for writing a book-length thing in my own style that would sound like me and I felt like would have the quality that I demanded, I didn't find the systems that useful, actually. I thought they were really good at, like, crafting a business letter.

Jeremy Kahn [44:50] They're really good at writing a quick email response. I at least struggled to get them to write something that sounded like me. So that was the problem. Maybe it's interesting because I've talked to Reid Hoffman a number of times, and he seems to have had more success in fine-tuning his LLM to actually imitate his style. I've found it actually kind of difficult to take a current system and get it to match my style properly.

Jeremy Kahn [45:13] So I didn't really use it for the book. I occasionally use it in my work for phrase finding. It's very good if you're sort of stuck for a phrase. And if you're stuck for a word, or if you're overusing a word, you can always go to a thesaurus. And that's very quick on Google; you just go to thesaurus and look it up. But if it's not a single word, if it's actually like a kind of phrase that you're overusing, or you want an alternate phrase or an alternate metaphor, sometimes it's really good there.

Jeremy Kahn [45:41] I find you can go to Claude, or you can go to GPT-4, and it's pretty good at finding alternatives for that. So I sometimes use it for that. Again, at Fortune, we're not actually allowed to use them to write articles for us. Again, I'm not sure they'd be that capable yet. But yeah, we're actually not allowed to use them in that way.

Matt Turck [45:44] And also, how would Fortune detect that you're using this?

Jeremy Kahn [46:04] Well, there's that. I mean, it's sort of an honor system. It's possible that people are doing that. But what they are good for is, again, sort of drafting kind of standardized letters. So, for instance, one of the things you can do in the U.S. as a journalist are these Freedom of Information Act requests, where you try to get some information out of the government or a government data source. And you have to write these letters to do that.

Jeremy's work at Fortune

Jeremy Kahn [46:21] And the letters have to be in a very specific format, and it's kind of tedious crafting these things. But giving this to Claude or giving this to GPT-4, it's very easy to do that. And so I've found it's very useful churning out FOIA request letters and that sort of thing.

Matt Turck [46:28] So maybe to close, maybe a few words on the rest of your work at Fortune, the writing, the conferences.

Jeremy Kahn [46:49] Yeah, sure. So, yeah, I'm Fortune's AI editor. I lead a small team that is kind of leading our AI coverage. I also write—we have a newsletter called Eye on AI that comes out twice a week. I write one of those editions myself, and I edit the other one that comes out on Tuesdays and Thursdays. We also have a number of AI conferences called Brainstorm AI. This year we've had three of those: one in London, one about to happen in Singapore, and then we'll have one in San Francisco in December as well.

Jeremy Kahn [47:16] And I'm the co-chair of all three of those conferences. I also get involved in some of Fortune's other kind of tech-related conferences. After this interview, over the weekend, I'm flying to Utah for Brainstorm Tech, which is our kind of premier tech conference. So, yeah, I'm involved in the event side of the business as well.

Matt Turck [47:20] Those are open to everyone, or you need to get on the list?

Jeremy Kahn [47:33] You have to register to attend, but I think anyone can go ahead and register. But then it is a somewhat selective audience. These are not sort of huge conferences. You're talking about 200 people, usually.

Matt Turck [48:07] All right, well, the book again is called Mastering AI. I highly recommend it. Really enjoyed reading it. There's a lot of people sort of writing and thinking about AI, but it feels in general is pretty extreme and kind of all over the place. This book, again, as I mentioned earlier, struck me as really hitting exactly the right kind of chord between pessimism and optimism and pragmatism, and really enjoyed it. So thanks for coming today to tell us about it.

Matt Turck [48:15] Really appreciate it. Thanks for doing it, Jeremy.

Jeremy Kahn [48:17] Thanks so much for having me. It was great.

Matt Turck [48:38] Hi, it's Matt Turck again. Thanks for listening to this episode of The MAD Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing, if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build the podcast and get great guests. Thanks, and see you at the next episode.