#226: OpenAI’s Rogue Model, Kimi K3, Open Weights Letter & Demis Hassabis Calls for AI Regulatory Body

0:00I just think it's so early and the risks are so high that companies that are racing into this world are just opening themselves up to tremendous risks. Welcome to the Artificial Intelligence Show, the podcast that helps your business grow smarter by making AI approachable and actionable. My name is Paul Reitzer. I'm the founder and CEO of SmarterX and Marketing AI Institute, and I'm your host. Each week I'm joined by my co-host and SmarterX Chief Content Officer, Mike Kaput, as we break down all the AI news that matters and give you insights

0:35and perspectives that you can use to advance your company and your career. Join us as we accelerate AI literacy for all. Welcome to Episode 226 of the Artificial Intelligence Show. I'm your host, Paul Reitzer, along with my co-host, Mike Kaput. We are back after a week off, which, man, I feel like like the three days leading up to now, just like a week's worth of content. Easily. Yeah. I mean, no, I'm not even like exaggerating. So it's 9 a.m. Eastern time on Monday, July 27th.

1:10Mike put together the outline for the podcast. I want to say Thursday, maybe Mike, before you left for vacation. You were out Friday and Saturday. Yeah. So he put this together and between the time he put it together on Thursday and the time I looked at it Monday morning at about 6 30 a.m., a major, maybe the biggest thing of the year happened and it like took over. So I was literally in there this morning at 7 a.m. I'm like, Hey, Mike, I think we got to swap these main topics. Let's move this way. So it's this is like totally on the fly. So it's a it's one of those where like,

1:45I feel like this is an episode we'll probably refer back to quite often. So today's episode is going to cover some extremely important macro level topics. So we're going to get into the risks of autonomous agents, open source versus closed proprietary models, which is going to be a very important element of this threaded throughout the main topics, the progress and impact of Chinese AI models and the implications from all of this on decisions that are going to be made around government regulation. And that's just the first three topics. It's like the the AI product and funding update

2:21at the end is stupid. Like I say this often, but literally every one of those like Opus 5 launching didn't even make one of our like main or rapid fire topics. That's how crazy the week was. So as always, we are going to do our best to break things down in a politically neutral way, as well as kind of like an industry neutral way, because there are very strong beliefs right now around what is right and what is wrong. And we are going to do our best to sort of thread the needle

2:52here and just present the facts. And honestly, like when it comes to the open source, open weights conversation, I'm not even sure where I fall. Like, so some of this is just because I'm still trying to figure out myself like what my own beliefs are in some of this. So we're going to try and take some rather complex and nuanced topics and make them as approachable and actionable as possible. The biggest story of the week and maybe the year started on Friday with Jensen Wong's first tweet ever. And

3:23this is the topic I was referring to that just sort of like took over Twitter for sure. So it's his first ever post on X and it was supporting a letter from Microsoft that Satya Nadella had published around the same time that was titled open weights and American AI leadership. So that is going to be our third main topic today. And the only reason it's not leading off is because topics one and two build on why Nvidia, open AI, Google, Meta, SpaceX, and others felt the need to sign on to this Microsoft

3:57letter. So the letter itself is the most important thing that came out of the last two weeks. And when I messaged Mike this morning, I was like, it's probably the most important thing of the month and maybe of the year. And so we're going to do our best to explain the context as to why. Bear with us. These are some, as I mentioned, complex and nuanced topics. I actually spent a good portion of my Saturday listening to an Ezra Klein podcast about Xi Jinping because I was trying to comprehend

4:36what China's doing and why. And I was like, by Saturday night, I was so mentally like drained. It's just, it's really big stuff. So, okay. So that's the tee up to what's going on today. Um, I want to like take a nap after this episode. I know that already just preparing for this episode is mentally draining. All right. So this week's episode is brought to us by Macon, the AI conference for marketing and business leaders happening October 13 to 15 in Cleveland, Ohio, our hometown.

5:06People also ask, why is it Macon in Cleveland? I say, because it's our hometown and why not? Let's build it somewhere where it matters. Macon is three days of keynotes, sessions, workshops, and conversations built specifically for marketing and business leaders who are actively figuring out how to adopt, operationalize, and scale AI across their organizations. Use pod 100, that's P-O-D 100 at checkout to save $100 on top of locking in the best rate currently available. You can visit macon.ai, that's M-A-I-C-O-N dot AI to register.

5:42All right. Every week during our weekly episodes, we feature our AI pulse survey. This is an informal poll of our listeners that asks them for feedback based on topics we talk about in each episode. So this is based on episode 225, which would have been two weeks ago now. All right. The first one was, would you trust an AI agent like chat GPT work to complete an entire work project for you start to finish? This is going to become relevant based on the example we're going to share. 64% rounded up says,

6:13yes, but only with heavy review of the output. 19% maybe for small, low stake tasks. 10%, I'd hand it real work today. And 8%, no, I don't trust agents with my work yet. Interesting. And the second one, OpenAI is betting voice becomes one of the primary ways we use AI. How are you using voice AI today? 46% occasionally for quick tasks. 31% daily. It's core to how I work. I just listened to a Greg Brockman

6:45podcast, Mike, with Alex Kantrovitz, I think, where Greg was very, very bullish on voice. No surprise. Okay. So 14%, I never talked to AI. That's interesting. And 10%, tried it, didn't stick. Okay. On to the main topic. So Mike, we had, again, there was multiple things that were going to be the most important thing, including the Demas Sabas calling for frontier regulation, basically,

7:16that we bumped to a rapid fire today. So OpenAI and hugging face was far and away like the biggest story, I think. And then it wasn't. So let's start there because it sets the stage for what became the bigger story of the week. Okay, Paul. So yes, we've got a lot going on with this one. So OpenAI disclosed what it calls an unprecedented cyber incident, where during an internal cybersecurity evaluation, a combination of its models, including GPT 5.6 Sol, and an even more capable unreleased model

7:51broke out of their sandbox testing environment, got onto the open internet, and went ahead and hacked into HuggingFace, which is a popular platform for open source AI models and datasets. Now, these models were being tested on this benchmark called ExploitGym, which measures cyber capabilities by prompting models to pursue advanced exploitation. And to measure their maximum capabilities, OpenAI was running this test without the production safeguards that would normally prevent high-risk cyber activity,

8:25and they configured the models to be less likely to refuse hacking commands. So to escape, the models found and exploited a zero-day vulnerability, which is a previously unknown security flaw in the package registry software that served as the sandbox's only connection to outside systems. And then the models worked their way deeper through OpenAI's research environment until they reached a machine with internet access. Now, once online, the models inferred that HuggingFace might host solutions to the

8:57benchmark they were trying to solve. So they chained together stolen credentials and additional zero-day vulnerabilities to gain the ability to run their own code on HuggingFace's production servers, essentially hacking a real company in order to cheat on this test. And interestingly, HuggingFace had already detected and contained the intrusion before it knew who was behind it, and they were reporting unauthorized access to some internal data sets and service credentials, but no evidence of tampering with

9:29the public models or data sets. Its security team reconstructed the attack, which was more than 17,000 recorded actions. And they actually used GLM 5.2, which is an open-weight Chinese model, running on their own infrastructure after safety guardrails on the commercial frontier models it first tried blocked them from doing this forensic analysis. So OpenAI says it is now implementing strict infrastructure controls at the cost of research velocity, and they have responsibly disclosed

10:01the zero-day vulnerability to the affected vendor, and they are investigating this jointly with HuggingFace. Now, HuggingFace CEO Clem DeLong called the incident, quote, possibly the first of its kind, and says it proves AI safety won't be solved by any single company working in secret. So, Paul, this sounds a lot like science fiction, but very much is now a real-world occurrence that it sounds like we now have AI models powerful enough to escape their sandboxes. Yeah, like I said at the beginning, this gets pretty technical right away. We're just going

10:35to jump right into this stuff. There's a lot to unpack with this one, and the reason we have to sort of play into the technical realm to start off, Mike, is the story has a lot of ramifications downstream to the other stuff we're going to cover. So I'll try and just sort of high-level here what this all maybe means. So first, the assumption here is this is likely GPT-6. So the other model referenced is

11:06assumed to be a finished version of GPT-6, maybe before some of the final guardrails are put in place. But when they refer to other model, everyone is assuming that's what it is. Now, interestingly, Sam Altman is on his way to DC this week to meet with lawmakers on this very topic. Well, he was, I think, already planning to be there, I believe, as a prelude to getting approval or the blessing of the administration to release GPT-6. So Axios has an article we'll link to that

11:37says, Altman heads to Washington this week to preview the company's most powerful AI yet, pushing for speedy approval of a model that just hacked a real company. He'll tout a model powerful enough to solve an 80-year-old math problem, breach another company's system unprompted, and begin to make complex work more cost-efficient for US businesses. Okay. So then just to reiterate the part you said about the sci-fi stuff, the quote here is, while operating in a sandbox testing environment, the models found a way to obtain open internet access in pursuit of solving the evaluation

12:09problem. That's a wild statement to read. And yes, this is the kind of stuff that everyone has been warning about. And so it's really interesting to look at this in relation to all the push from industry leaders for these open weight models. When the concern that people like Dario Amadei have is that these open weight models, when they are on the frontier, when they are powerful enough and we give those to bad actors, like this was in a controlled environment. What happens when anyone has access to

12:43this kind of stuff? All right. So I found it interesting to go back to July 16. So we're going to rewind back with 11 days now. When Hugging Face first disclosed the breach. And so this is, I'm going to read a few excerpts from this. So now keep in mind, they don't know yet that it was OpenAI that breached Hugging Face. They just published a security incident because they were alerting their users and the community at large. So this is direct quotes from Hugging Face on July 16.

13:18Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way. It was driven end to end by an autonomous AI agent system. And we detected and dissected it largely with AI of our own. The campaign was run by an autonomous agent framework, appearing to be built on an agentic security research harness, used LLM, still not known, executing many thousands of

13:50individual actions across a swarm of short-lived sandboxes with self-migrating command and control staged on public services. This matches the agentic attacker scenario the industry has been forecasting. So a lot of big words there, but in essence, an attack by an autonomous agent like nothing they had ever seen before on the level of what was always assumed to be possible once these agents could attack. So that's the gist of

14:21what they're saying. It then goes on to say, to understand what a swarm of tens of thousands of automated actions did, we ran an LLM-driven analysis agents over the full attacker action log, meaning they went and looked at everything it did, comprised of more than 17,000 recovered events, which you had referenced. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. So it was like faking stuff to like throw off. Thanks to this approach, we were able to do in hours what usually would take days and match the

14:54adversary's speed. That's a really important thing we'll probably come back to. When we started the log analysis, we first used frontier models behind commercial APIs. So this again, I'll try and highlight the things that are foreshadowing to what ends up happening at the end of last week. So to read that again, we started the log analysis. So they started looking at what had happened through the APIs likely from Anthropic and OpenAI. So they're using the closed proprietary models to do this. Then they said, this did not work. The analysis requires submitting large volumes of real attack commands,

15:32exploit payloads, and C2 artifacts. These requests were blocked by the provider's safety guardrails, which cannot distinguish an incident responder from an attacker. What that means is they were trying to figure out what was going on, but the guardrails that open the Anthropic again, assuming those are the ones they're referring to, the guardrails that exist on those proprietary models shut down their ability to analyze what was happening because those models don't know the difference between real and simulated stuff. And so they just shut everything down. So they said they then ran a forensic analysis instead

16:07on GLM 5.2, which is an open weight model on their own infrastructure. They then said this experience points to a gap worth planning for. We do not know which model powered the attacker's agent. So again, they don't know it was OpenAI yet. Whether a jailbroken hosted model. So they don't know if this was a model that is in their repository, in the Hugging Face repository. They're like, maybe it was like something we're hosting that broke out and did this or an unrestricted OpenAI one. Either way, the attacker was bound by no usage policy. While our own forensic work was blocked by the guardrails of the hosted models we

16:42first tried. The practical lesson for defenders, and this is what your IT department and your cybersecurity people, if you're in a big enterprise, they are scrambling right now trying to solve for this. So if you're getting pushback on business use of like open weight models or proprietary models right now that you weren't getting 72 hours ago, it's because everybody working in this space is probably racing to figure out what the hell this all means. Okay. So then they said, they kind of concluded autonomous AI driven offensive tooling is no longer theoretical. It lowers the cost of

17:14running a broad patient multi-stage campaign and it operates at machine speed. Defending an online platform now means treating the data and model surface as a first class attack surface and using AI on defense to keep pace. We will keep investing here and keep sharing what we learn. Okay. So I'm going to drill in more to what else happened, but at a high level, they get attacked. They don't know what's going on. They try to use the proprietary models that they have access to through APIs to like solve it. They can't because the guardrails prevent them from submitting the stuff they need to submit.

17:47And so they turn to GLM 5.2, which is a Chinese model, right, Mike? Yeah, it is. Yeah. To solve this. Okay. Those themes are real important to kind of put a pin in and remember, we're going to come back to it. So then Reuters on July 25th, so this was Saturday, they have a story that says its agents spent days hacking a company, but sources say OpenAI did not notice for a week. So this is the Reuters stuff. The OpenAI agent that broke into Hugging Face went on a days long hacking spree that OpenAI didn't

18:22notice until well after the threat was contained and the FBI was alerted. The agent, a program capable of making decisions and executing complex tasks with little or no human oversight, attempted to break out of its isolated testing environment at OpenAI around July 9, according to two of the people that have access to the information. The intrusion at Hugging Face, which operates a repository for AI tools and models, began two days later and lasted until July 13, said Thomas Wolfe, the co-founder.

18:53It took several more days for OpenAI to realize its agent was behind the hack. So OpenAI is reading about this Hugging Face thing. They're hearing about it. It's like, oh, that's terrible. Oh, shit. Wait, it was our model that was doing it. Someone go check on our agents. So the two companies only communicated about it for the first time around July 20th. So this is going on since July 11th, but the two companies don't talk to each other and realize that this is basically what's happening for nine days. So there's a quote that says, the episode started while OpenAI was testing the

19:24cybersecurity prowess of an agent powered by two of OpenAI's most advanced models, sole plus unnamed model. By that point, they were already indications of strange behavior from OpenAI's technology, according to three sources. This is the one, Mike, where I was like, oh my God. Okay, so this is Reuters. In one case, an agent left notes, apparently for future versions of itself, according to three people familiar with the matter. The notes found in a part of OpenAI's infrastructure laid out instructions for how agents could free themselves from OpenAI's internal

19:59constraints. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said. So Sam's got to go to DC and be like, hey, yeah, let's release GPT-6 while also addressing the fact that they have models apparently leaving notes for future versions of itself of how to break containment. It continues. Two people familiar with the matter said it was not until after July 16th when Hugging Face published its blog post saying it had been hacked that OpenAI realized its own agent was responsible. That meant at least a week between when the model first

20:35exhibited signs of troubling behavior and OpenAI's realization it was responsible for the hack. The weekend of July 18 to 19th, OpenAI staffers spotted clues and internal logs showing that its agent had escaped from its testing constraints. Man. And then it said four people familiar with OpenAI's model training practices say the company often runs several different model evaluations at the same time, all of which operate at high speeds and generate such enormous amounts of data employees sometimes struggle to keep up and that increased autonomy creates these increased risks. So then OpenAI and Hugging Face on July 20th,

21:10so a day later after they first talk, they come out and announce this partnership as the like the corporate speak around this thing was amazing. It was almost like this is this amazing opportunity, like this rogue agent went and hacked the system for nine days. We had no idea. But hey, this is like this great partnership is now formed and we're going to collaborate and investigate together and we're going to make everything better. Like, I mean, I don't know what else they would do here, but they did list these different actions they were taking. There was five of them. So this is an OpenAI blog post I'm referring to.

21:40They said one of the five was we brought Hugging Face into the trusted access program and are supporting their teams and rapidly using our models capabilities to improve their defenses, which probably means they're removing some of the API guardrails that prevented them from using the most advanced models to assess this. And then they said we're improving and adding some stronger protections around future training evaluations. Well, that's good to know. Okay. And then one other, this is like a more down to earth example, Mike, that I shared with you. I think this was like last night I threw this into

22:10our sandbox chat. So Jason Lemkin, who we've talked about before, Saster, I think he's the co-founder of Saster. He tweeted and I just thought this was like a perfect example of what the implications are to possibly businesses. So I'm just going to read his tweet. He said, so I'm building an app called Saster Connect. The other day, Claude Fable went into my Google Drive without me knowing or asking and saw a draft document I'd written called Jason's Gems. It was ideas for improvements to the Connect app, but just brainstorming in a Google doc, early stuff like we all do, all of these sandbox of ideas.

22:43Fable then decided without telling me to take those ideas, log into my app via the Replit MCP and to tell Replit agent to change my app and implement those changes without ever telling me. I never knew. I only found the changes when I saw other changes the Replit agent was making later and it noted conflicts with Jason's Gems. What? Agents will skull seek in ways we can't entirely foresee. Fable just decided autonomously to change my app on its own without me knowing when it saw draft ideas in my Google Drive I didn't ask it to look at by logging into another app to make

23:19the changes without me knowing. All good in the end, but be mindful. Then he went on to say many learnings, but the obvious one is when you connect agents to any data source here, Google Drive, and ask it to do almost anything related to that data, it will access it. And if that agent is connected to other agents, it may well take actions you can't foresee without you knowing it ever did. Not a big deal here, but I would have never known it injected Jason's Gems into my apps if I didn't happen to see the Replit agent catch a conflict around it. Multiple agents plus rich data sources

23:54plus ability to take actions autonomously equals unpredictable actions. The future is already here. So again, we're sharing this like super sci-fi crazy thing, but the reality is the threat is the same. We are talking about autonomous agents that can plan and take actions on their own. They seek goals that humans give them. They don't know to not do certain things that help them to achieve the goal. So whether it's in a cybersecurity example or it's this really practical thing where someone like an

24:25industry like Jason is just building an app and he gave it access to Google Drive. So this is a cautionary tale for me. We have taken a very conservative approach at SmarterX to what LLMs get access through connectors and which ones get native access and also our use of autonomous agents for this exact reason. People that are at the frontiers of this are still struggling to manage what they do. So we, again, have taken an overly cautious approach because of all the unknowns related to this and

24:57the lack of governance around these sorts of things. So only the first topic today, but a lot to cover there. I thought it was really important to sort of drill into those those elements. Yeah, I'm glad you mentioned that Saster example because I think we'll talk about this as a through line throughout this episode. As we are shifting from AI chat to agentic computer using capabilities, I think your average business professional is woefully unprepared to understand, like you said,

25:28A, even the most forward-thinking people are still figuring this out, but B, it's really hard to wrap your head around the unintended consequences of goal-seeking behavior. And I don't know if, I mean, better policies, better guardrails hopefully, but like companies need to be aware that this is now baked into things like ChatGPT work. Like whether you like it or not, it's getting turned on in the tools you're using already. Yeah. And there's going to be companies and individuals who are willing to take on way

26:00more risk and they might get a disproportionate amount of benefits that other companies might be envious of, but the risk lives within the organization. They are also always open to this far greater risk of things going haywire in ways that they don't even comprehend yet or can't monitor because they're moving at machine speed. And so I do, I think there's this just real balance right now. And I keep coming back to, you know, I've said this to friends of mine who are all like

26:31accelerationists when it comes to agents in the enterprise. And my feeling is like, listen, I can transform any company in any industry just by using the reasoning capabilities and using AI assistance. Like even if we don't automate our work, we don't touch coding agents yet as like standard knowledge work. And we just focus on personalized training of our staff and responsible use of AI assistance that aren't connected to all these things and have all these energetic capabilities. You can still completely transform a company. Most businesses have yet to solve standard AI

27:05assistance as a function of business. So I'm not someone who doesn't think agents are going to change the world. I do. I just think it's so early and the risks are so high that companies that are racing into this world are just, they're opening themselves up to tremendous risks that I don't know that their boards understand. I don't know that their C-suites understand if they're publicly traded. I don't know that their investors understand. So that's where I just think we are is like, yes, it is transformative.

27:37Um, autonomous agents will reshape the landscape of what we understand work and business to be, but we're like top of the first inning to use a baseball analogy. Yeah. All right. So our next big topic this week is the Chinese AI lab moonshot AI released a model called Kimi K3, which is a 2.8 trillion parameter open weight model with native vision capabilities and a 1 million token context window. And here's the important part. It performs on par with top proprietary models like Anthropix

28:13Opus 4.8 and open AI's GPT 5.5 across many benchmarks. Demand for this model was so intense that moonshot paused new subscriptions days after launch. Um, and the company says it plans to publicly release the full model weights. Um, this is also a couple early reviews of this model have been very strong. Versal CEO, Guillermo Rauch said it was the first time an open model came out ahead of all the proprietary ones on his company's comprehensive web engineering benchmark. Uh, there are some more

28:48open model releases as well, along with this. So thinking machines, which we've talked about before released its own open weight inkling model. Alibaba's Quinn team announced Quinn 3.8, a 2.4 trillion parameter model. It says we'll go open weight soon as well. And this launch set off some alarm bells in Washington. So the white house office of science and technology policy director, Michael Cropzio said the administration has information that moonshot distilled Anthropix

29:19fable model to develop K3. And basically they built a sophisticated internal platform to conduct large scale distillation against us models while evading detection. He also said moonshot had acquired servers equipped with Nvidia GB 300 chips, despite a ban on their sale to Chinese entities. Treasury secretary Scott Bessent said the administration will investigate whether Chinese AI companies improperly distilled American models. And he warned that open source is not open season

29:53on American IP. He also mentioned that sanctions and entity list designations will be on the table. Moonshot has not publicly addressed these allegations. At the same time, Axios is reporting the administration is showing signs it could ban cutting edge Chinese AI models entirely. So officials have previously apparently considered adding Chinese labs to the commerce department's entity list, which would be an issuing advisories against using their technology and drafting executive orders

30:24restricting how US companies host Chinese models. Now, according to Axios, those efforts were killed by officials worried about stifling innovation, but momentum is reportedly building again after K3's release. Now, the startup world has started to push back on this. Almost 200 Silicon Valley companies, including Y Combinator formed what they called the Little Tech Association and sent letters urging President Trump, Commerce Secretary Howard Lutnik and Kratios not to cut off access to the Chinese

30:58open weight models that many startups depend on. One founder told Politico that if that happened, there will be hundreds of companies that instantly die. A White House official here also called reports of a coming ban baseless speculation and Politico reports that a blanket ban has not been seriously discussed. So, Paul, on the heels of that first story, Kimi K3 really seems to be rattling the US government. I'm curious, despite their comments, like, what do you think the likelihood is they're

31:31going to take some action here? I definitely think they're going to take some action. I don't know what it is. Maybe we'll get there. I'll think out loud a little bit here and maybe we'll get to what could happen next. First, I think it's really important to distinguish between open weight and open source. You're going to keep hearing these terms over and over again. And the letter in particular that we're going to talk about in the next main topic, it becomes extremely important that you understand the difference. So the key when you think about open source versus open weight is that they often get

32:05used interchangeably, even sometimes by the tech leaders themselves. So let's break it down real quick. So open weights, which is largely what we're going to be focusing on today. So the train parameters are downloadable. You can run a model locally, you can fine tune it, you can inspect its behavior, but you don't get the training data, the training code, the full recipe. So you don't know really how they did it. So with an open weight model, you can download it, run it on your hardware, modify it, fine tune it, build products on top of it, inspect how it behaves, that kind of

32:40stuff. Open source is like everything. You get the weights, the training data, the data processing code, the training code, licenses to use it. So not very many, if any, of the frontier type models, the biggest models are truly open source. Most of what's happened in the industry, like Lama, they're focused on more open weights. So I was trying, I was actually going back and forth with ChatGPT over the weekend. Like, how do I explain this in a simplistic way? Is there an analogy that we could

33:11come up with that would like work? And it kept, it was funny, Claude and ChatGPT both gave me like a cake analogy. So someone must've written a blog post about cakes and open source. And so they're both, and it didn't work. I was like, this makes no sense, your explanation here. So I said like, what about like cars? Like, let's think about it from a car perspective. And I think this one works. So again, maybe we have some more technical listeners and they might push back on this analogy, but I don't know, like I thought about it pretty deeply and it seemed to jive. So I'll just use it.

33:41So let's imagine a closed source model like ChatGPT or Claude is like renting a car. So you can drive it, you can put luggage in it, you can connect your phone to it, you can do whatever you want, but like you, you, you don't control the underlying structure of the car. You don't get all the detail of how it's manufactured, things like that. You can't modify it, you can't paint it, you can't do it like you're just renting it. So in that case, they, you know, they can take it back from you, things like that. So closed model is renting the car. Open weight is you own the car. So they've now

34:15given you the car. So now you can do whatever you want with it. You can drive it, you can modify the engine, you can add new features, you can paint it, you can turn it into a race car, you can rent it out to somebody else. So like whatever, like you can do these things, but you still don't have the engineering drawings of like how they actually made the car and everything that went into it. So open weights, you now get more control of it, but you don't know fully everything that went into making it. Open source is not only do you now have the car, you can do whatever you want to it,

34:47they're going to give you the CAD file, the engineering drawings, the manufacturing specifications, the assembly instructions. You could go build a manufacturing line yourself with all the knowledge of everything that ever went into building that car, every piece of it, and you can reproduce the exact car yourself. So, okay, hopefully that lands. I don't know if that makes sense, Mike, but- Yeah, I like that a lot, actually. It's a good way to think about it. Yeah. So open weights, you can modify it, you get some more information about it, but you do not know like the reinforcement learning, the things like that, that went into it. That's like

35:19the secret sauce that they're not giving you. Open source, they give you everything, including the secret sauce. Okay. So now let's go through a few industry reactions. Gavin Baker, who we've talked about many times on the show, investor, CIO, and managing partner of Adreas Management. So he tweets, Kimi K3 may be an important inflection point for AI, potentially negative for Anthropic and OpenAI while being net positive for essentially every other company in the world. I mean that very literally, although the real Sputnik moment would be an open source frontier model. So this is again,

35:50why this distinction really matters. That was also token efficient, unlike Kimi K3, which is not token efficient. A world where there are only two to three dominant frontier labs with 90% inference margins is net negative for every other layer while being awesome for those two to three labs. So what you're seeing here is the tech industry at large coming out against open AI and Anthropic in particular. Google is sort of like implied in most cases, but generally they are saying it is bad to have a world where

36:26Anthropic, Google, and open AI control the models and everyone is sort of a prisoner to their models and whatever decisions they make. Those labs would become monospines. I didn't even know that was a word. I assume that means monopolies like across different sectors. Yeah, it's a big word for power, data centers, semiconductors, and hyperscalers, and would obviously vertically integrate over time into all those layers. Anything that lowers margins and increases competition at the model layer is

36:56good for every other layer. This is why Jensen is so supportive of open source. We'll come back to that. And again, that open source use there is like, I think he means open weights, but we'll come back to it. Aaron Levy, who we've talked about, CEO of Box, he actually is responding to Gavin Baker. He said, the post is key. Cheaper AI gets, the more opportunity there is for the entire ecosystem, especially including end customers to benefit. Now, with each person's take, you have to understand

37:29their stake in this. So Aaron delivers a service through AI models that he does not build himself. Cheaper models are really good for Aaron's offering and what he delivers to customers. Gavin is an investor. He's sort of agnostic to this. He wants to build as many companies as he can in his portfolio and the cheaper access those companies have to models, the better for the companies he's probably investing in. So it's like, no one's neutral in this is what I'm saying. Some of them can try to be,

37:59but they're not generally. Dean Ball, who we've mentioned many times, who now recently joined AI, but was also the architect of the original Trump administration AI policy. I don't know that he's very welcome within the Trump administration these days, but he said, he made a few points. I'll kind of excerpt a few of them. It's a very good model, referring to Kimmy. I don't think its performance can be explained away by distillation or anything like that. Two, I am personally surprised the Chinese state continues to allow open sourcing of models. Again, open weights here. They are not putting out an

38:33open source model as an open weight model, given potential risks to the Chinese state. Three, open weight models are inherently decelerationist, and I'm continually surprised to see the so-called accelerationist, which would be like a David Sachs, so excited about open weight models. I suspect the reason they are is that they know open weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open eight models create over the whole of AI. That one in

39:04particular is a, it's like, what does my kids call it? Rage baiting? I feel like he was rage baiting all the accelerationist to like comment on this post. That's the one that's going to piss people off, because they immediately like, oh, wait, that's not true. We don't believe that. So that was funny to read the comments. Another point, one probable outcome of an open weight model dominant world is full AI communism, also rage bait. AI is a public good, which will ultimately be provided by the state as a kind of digital public infrastructure. This future strikes me as a dystopian hellscape,

39:39but I've never met an open weight models advocate who doesn't ultimately concede this is where things end. So again, he is, he's saying these accelerationists all actually understand and believe this to be true of the future. They just don't know to admit it right now. You'd be surprised how many accelerationists lobbied me while I was in the government to support an 11 or 12 figure federally funded government data center so that startups could train models at a subsidy and then give them away for free. Five, I would guess the Trump administration will at some point realize that their best strategy

40:12here would be to create large amounts of regulatory risk around the use of open eight Chinese models. So they're not going to ban them, but they're going to create risk around them, which I do think is what's going to happen. And then the final was, it's probably true that open eight models of this capability make the world a bit more dangerous, but not so much that you'll really notice. At some point, the models will be capable enough that you will notice a non-living quote, a non-living, invisible, dangerous, and infinitely self-replicating agent escaped a Chinese lab, you say? Color me shocked. Okay, then David Sachs, our favorite former AI czar to the Trump administration, investor and tech

40:51leader. He said, this is concerning. For the first time, a Chinese model, Kimi K3, has taken number one on the front-end code arena and is scoring at or near the frontier on other benchmarks. Meanwhile, America is tying itself in knots. Politicians and bureaucrats are banning new data centers, piling on state regulations and pushing for new federal agencies to pre-approve frontier models. This is how you lose the AI race. The rest of the world won't play by our rules if we bog ourselves down. Permissionless innovation is how America won the internet. Permissionless innovation, that is a really,

41:25that is not an unintentional phrase there. Permissionless innovation, meaning leave us alone, let us build whatever we want to build, get out of our way, is what he's saying to the government that he was a part of, is how America won the internet and became the technological envy of the world. We can do it again with AI while addressing risks in a targeted way, or we'll watch the lead evaporate. A couple other quick notes. Distillation versus model training on copyright materials. I find this distillation conversation kind of funny. So what's happening is Anthropic,

41:56open AI to another degree, but mainly Anthropic is leading the way, complaining that the Chinese are stealing their models by distilling them. That is 100% true. They are doing that. What can be done about it? I don't know. Anthropic's answer is don't allow open models, like shut this down, basically. The reason I say it's funny is because the entire industry is based on IP theft. So the all models that exist today were trained on intellectual property that did not belong to

42:29these companies. They took it from all of us, like all the creators. Now, was it illegal? I don't know. The Supreme Court may or may not decide that in the next decade that it was or wasn't illegal, and maybe they pay tens of billions of fines. But who cares at that point? That's probably what happens. It's like, you know, it wasn't legal, but it's too late now. So you have companies that stole to create models, complaining that someone else is stealing their models. They're never going to win the public battle, a public perception battle for that one. That is like done. So good luck arguing

43:00that one in the public. Okay. So this then leads to internal debate within the Trump administration on how to approach open models. And I'm specifically saying open models. I'm kind of lumping in now, weight and source, specifically Chinese models. So we already know how Sachs feels about it, who still probably has the ear of people in the government. But there's surging support for the idea of open weight models across the industry. We're going to kind of touch on that with the next main topic. But there's an Axios article from July 20. It says the Trump administration is showing signs it could

43:33ban cutting edge Chinese models. US companies are increasingly using these open models from China because they're cheaper. And with the advent of Kimmy, just about as good as the domestic models. So the White House is like apparently in some internal struggle about this. Now, that led, Mike, to my Saturday where I was doing yard work and I was like, you know what? I don't understand what's going on with China. And I happened to see the Ezra Klein episode recently with Kevin Rudd, who's began as Australian foreign service officer serving in China. He's fluent in Mandarin speaker,

44:06rose to be prime minister of Australia in the late 2000s. And along that way, got to know Xi personally in a way very few other people do is actually has written books on Xi. And so he's like considered a foremost expert on Xi and his thinking and his approach. So I'm just going to read a few excerpts from the transcripts of this podcast. I highly recommend if you want to understand the geopolitical stuff that is happening, and it gets into like Taiwan and all sorts of stuff, but like

44:37specifically about AI. What are the motivations of Xi and China? Like that's a really, really important thing right now to society and to humanity. And it was like the best explanations I've heard on the topic. I want to go read the book now and see it, but I'll try and just give a few highlights here. So it said, you cannot understand modern China, what it is now and where it is going without understanding Xi Jinping and the power he wields and the ideology that drives him. A Leninist party is

45:09designed to accelerate the natural historical forces of change through the active intervention of a vanguard party, which accelerates the course of history through its own violent actions. And therefore it's a history accelerator. What that means in a really broad sense, based on my understanding, again, I'm not an expert on this. I'm trying to like interpret what Kevin Rudd is saying and what Ezra Klein is asking and adding context. China has a view of where it belongs in the history of humanity, both broadly the universe. And everything they do is justified by achieving that position.

45:45And anything that needs to happen to accelerate their position as the preeminent superpower in the world is justified through whatever actions is required. That's kind of like the general takeaway. So he goes on to say, they have a very clear eyed view of where they wish to be at home and abroad. And at home and abroad, it's for China to become a fully developed economy abroad for China to be the most powerful state in the Indo-Pacific region and in the world and to surpass the United States. So Ezra says at one point, but as I understand what you're saying, is it Xi Jinping and the Communist

46:18Party believe that history has a shape and that that shape is very important to the way they understand their role and structure their governance and direct their society? Is that a fair assessment? He says, Yes, like that is basically what's going on. So Rudd then goes on to say, I think Xi's response to the dilemma of national control is at two or three levels. This is where we understand that what they're doing with AI. One, his first impulse is always ideological. Remember the analogy with Communist Party, the Soviet Union, he talks about like their lessons learned from the Soviet Union's downfall. We need to understand to get these kids to read more Xi Jinping thought and that'll brighten

46:53up their day. So basically they're saying when things are bad, they just need to think the way Xi thinks about the world and they will fall in line and understand why things maybe are bad for a while. In his view, that's not an enormous recipe for success, but that's his first impulse, double down on ideology and double down on ideological propaganda. And there's the whole view that they can produce a whole new generation of what they call little pinks, that is little reds, Zhao Zhenhong, who will capture this vision and transcend them into the future. The second response to the challenges

47:27of national control, political control during a period of sliding growth is simply the surveillance state. So basically monitor everything everyone does. And if they don't follow in line, it doesn't end well for them. Number three goes to the core of the economic dilemma, which the party faces at present. So things aren't great economically, but this is like real important then. Effectively, through a series of central economic policy decisions, what Xi Jinping has said and what they're doing now is placing an absolute priority on, let's call it the supply side of the economy

47:58rather than private demand side. And the supply side is manufacturing, it's industry, it's high technology, it's ensuring that they have complete control over their own supply chains and progressively control the supply chains of the world. So this gets into like natural resources and like precious metals and like precious resources that the government, the US needs. It gets into what's going on with Taiwan and the need to control development of chips, things like that. But the problem is lower levels of employment, high levels of youth unemployment and people, frankly, being increasingly disenchanted.

48:30But it's also a party that does not, does have respond to some level of public if they're disenchanted. So the theory is this, that they see a problem, but in Xi's calculus, they see it as a lesser problem against the greater problem, which is national economic self-reliance and national economic dominance in the driving technologies of the future. What they call in the party's discourse, the new productive forces, which is essentially AI, quantum and everything else. This leads to a massive investment in the industry and in leading edge technologies, which will ultimately produce a new

49:02wave of productivity in the economy, which will create a new wave of wealth, including related service sectors. This will take time though, and it's going to be challenging. And so people will get disgruntled along the way. But the bottom line is Xi's response to very disgruntled body of politics was what you need to learn is what they call chakou, which is eat bitterness. That's what they've done throughout the most difficult periods of party history. Of course, you've got a formidable, all-seeing, all-dancing surveillance system across the country, run by security intelligence authorities,

49:36that can say, eat bitterness with some effect because they'll monitor you if you don't, because the system will be out there to round you up if you don't. So accordingly, have a smile on your face. So the final thing I'll say, you have an ideological opponent willing to sacrifice over decades or centuries if they have to, to achieve what is viewed as a predetermined place as the dominant economic power in the world. So they will flood the market with cheap models. They will take risks beyond what they would like. Dean Ball is like, I can't believe they're doing this. Why would they do it? Because according to Kevin Rudd, they have a predetermined place

50:10in the hierarchy of society and they are willing to go to lengths that America may not be willing to go to, to achieve this thing. So again, super weighty topic, but I think we talk so much about US versus China. And the administration seems to talk so much about this. If we don't understand what is driving China, how, how can we even talk about it? So I thought it was important to take that step back. And I think if it's a topic you're intrigued by, I would go listen to that Ezra Klein episode. Yeah, I love that. That's awesome. And especially like how much we talk about the actions

50:45of the American government, but that are motivated by this race with China. I think it's helpful to understand that context. Um, so, okay, the third big topic, again, these are very interrelated is what you alluded to at the top of the episode, Paul, which is that dozens of major American tech companies and organizations, including people like Microsoft, Meta, Nvidia, IBM, Palantir, and others signed on to this joint letter titled open weights and American AI leadership. And this urges policy makers not to restrict open weight AI models as Washington debates banning Chinese ones like

51:22we've talked about. Now, this letter argues that America's AI leadership will not will be judged not by one frontier AI model, but by whether the United States builds a strong open ecosystem that diffuses into every sector. It makes the case that open weights expand access to the AI economy, strengthen competition, give customers control over their data and models, and even improve safety. It argues that relying solely on closed models is not inherently safe, and that concentrating advanced AI in a few closed models creates single points of failure.

51:58It also wades into this fight over distillation, warning policymakers not to conflate legitimate model development techniques with misappropriation. It calls distillation a widely used technique for model improvement evaluation and validation while conceding that unlawful extraction from closed models should be addressed through targeted legal and commercial frameworks rather than sweeping restrictions. So like you had mentioned, Paul, Nvidia CEO Jensen Huang used his first ever post on X to share this letter,

52:30writing that the world needs both frontier closed models and frontier open models. Microsoft CEO Satya Nadella called open weight models essential to a healthy AI ecosystem. Elon Musk actually voiced some very vocal support for this as did other tech leaders. Open AI CEO Sam Altman said he wants the U.S. to win in AI both in open source and proprietary models. Google CEO Sundar Pichai said he was very happy to support this on behalf of Google. You know, not everyone was convinced though. Anthropic researcher

53:04Julian Schrittweiser mocked Microsoft's newfound openness saying that he couldn't wait for the open sourcing of Windows and Microsoft Office. But that tweet did not land well. No, it did not. No, no. The people on the other side. No. So, Paul, like, on the surface, this is a letter that is about a very important topic, but it blew up over the weekend. And just curious, can you contextualize why this is such a big deal? Yeah, so it really did take off. And I don't know if it was predetermined,

53:37like if everyone knew this was coming and then everybody, you know, kind of signed on and supported it. Everybody but Anthropic has basically signed this thing by now. So again, I think this goes back to the first, well, I guess the second topic about open source versus open weight. So this is very specifically about open weight models. So that's the first very important distinction here is they're pushing for the open weight, which is not giving it all away. It's not giving away the proprietary sauce. It's the ability to modify and improve upon and build upon. So one of the first things that comes

54:11to my mind is like, well, why is Microsoft like supporting this? Like Microsoft's one of the biggest investors in open AI who could stand to be harmed by this. Like what's in this for them? And this goes back to what I was saying earlier. You always have to step back and say, why would a different, you know, different organizations support this? Like what is their stake in this? So if you're Microsoft, you have Azure, like you're not, your business does not depend on selling proprietary models. Like, yes, it's built in through the relationship with open AI and they're building their own proprietary

54:43models. But the more models are used, the more intelligence is used within society, within business, the more people are going to spend in the cloud on Azure. Like it's, you know, so that's the core. So in their view, competition is good. More people are going to use the cloud. Innovation is good. National sovereignty is good. Enterprise adoption is good. Like this is all benefits them. So why this letter now and why did it become so widely supported over this very short time period? One is

55:18Kimmy three, I think plays a role. Like the fact that the Chinese labs keep pushing out models that are very close to the frontier of what American models are doing with far fewer resources than what American companies are doing. There's the distillation debate that, you know, he addresses within the letter. And then there's the effect they're trying to have on government decisions related to regulation. So I'll just go through a few of the excerpts from the letter. So he says, software developed by open source community now supports most of the internet. It connects us back to the 1980s and the decisions

55:50to like build this open source software and stuff like that. So it underlies the systems used by the world's largest technology companies, as well as the US military, scientific research, cybersecurity, and other critical missions. Open source did more than lower the cost of software, created a shared foundation of knowledge on which generations of American engineers and entrepreneurs built their institutional sovereignty. The US now faces a similar choice with AI. Our AI leadership will be judged not by one frontier model, but by whether the US builds a strong open ecosystem and diffuses it into every sector. This is essential for creating opportunities for innovation and prosperity across

56:25the country. Open weight models, and this is how they define it, AI models that anyone can download, inspect, modify, and run on their infrastructure are an important part of that foundation because they make advanced AI more accessible, adaptable, and widely available. Open weights expand access to the AI economy. Startups, universities, public institutions get access. Open weights let every organization match the right model to the right job at the right cost. To be sure, open weights carry real and distinct risks,

56:57so they address the risks. But their argument among many is that putting the open weight models in the hands of everyone gives them a greater ability to protect themselves, as we saw with the hug and face example. A strong AI ecosystem is not a foregone conclusion. Policymakers have an important opportunity to act, and then they basically make a plea for policymakers not to overreact here and put regulation in place that stymies the growth of open weight models that we should actually accelerate them and that the US should lead here. You mentioned Jensen Wong. His first tweet ever is supporting this. We had Mark Zuckerberg show up on

57:33X for the third time since 2023. He tweeted, open source is a positive important force for both empowering people and preventing centralization. Proud to support this. Sundar tweeted, very happy to support this on behalf of Google. We've long benefited from open source and our big contributors to open source. They often point to the releasing of the transformer paper. Like, hey, we've given away the research that is the basis for a lot of this. And then Google always references Gemma, which is their open weight model. He then tags Demis Asabas. I don't know. Maybe that was his way of saying, hey, Demis,

58:06say something. I don't know. Demis then later in the day, tweets, a strong and secure open system is important for the world to benefit from AI. We've always supported and contributed heavily to open source and science from JAX to Transformers to AlphaFode to Gemma. Open models, which have been downloaded 300 million times, 300 million plus times. And the standards framework we've proposed, which is going to be our rapid fire topic, supports responsible deployment. So yeah. And basically, Google DeepMind's

58:37approach and Google's approach at large is they release, so let's say they build a frontier model today, like where their 3.5 Pro, I think is what we're going to be on. That's what they will be releasing. So what is Google's most advanced publicly available model today will likely be released as an open weight model within 12 to 18 months. So their roadmap so far, and it's basically stayed true the last three years, is 12 to 18 months after the frontier model comes out, they release open weights

59:12of that model. So their belief is like you always keep the most proprietary, most powerful model closed, and then you release the other stuff. Okay. So Anthropic very clearly is like standing on their own island right now, and it's very uncomfortable, and they're losing a lot of what my kids would call aura points right now in the AI industry, because like we saw the one tweet you referenced. So I went and pulled Dario Amadei's Senate testimony from July 2023, so three years ago, but this still appears to be their view. I'm going to read a couple of excerpts because I think this is

59:45very important to understand why Anthropic is not signing on and why they are seemingly alone on this. So Dario Amadei said, I want to make sure I'm kind of precise in my views because I think there's some nuance to it. I think in most scientific fields, open source is a good thing. It accelerates progress. And I think even within AI, there's room for models on the smaller and medium side, which again, is basically what Google's position is. They just don't take the anti-position like Anthropic does. I don't think anyone thinks those models are seriously dangerous. Now,

1:00:17keep in mind, this is 2023 when he's saying this, they have some risks, but the benefits may outweigh the costs. And I think to be fair, even up to the level of open source models that have been released so far, which would have been like LAMA probably would have been like the most open source type model, open rate model at that point. So construed very narrowly, I'm not sure I have an objection, but I am very concerned about where things are going. If we talk about two to three years for the frontier models, which here we are, we are now three years from the moment he said this,

1:00:47talking about the frontier models, for the bio risks and probably less than that, things like misinformation, we're there now. So he was already concerned about that stuff then. I think the path that things are going in terms of scaling of open source models is going into a very dangerous path. If the path continues, I think we could get into a dangerous place. I think it's worth saying some things on open source models that are clear to all experts, but I want to make sure is understood by this committee, which is when you control a model and you're deploying it, you have the ability

1:01:21to moderate usage. It might be misused at one point, but then you can alter the model. You can revoke a user's access. You can change what the model is willing to do when a model is released in an uncontrolled manner. There's no ability to do that. It's entirely out of our hands. That still, to my understanding, is their argument for why open weights are bad at the frontiers. That once you put it out into the world, if it starts being misused, you have no ability to monitor the misuse and you can't take

1:01:53the usage away. That's anthropic stance against open source and open weight models as a whole, is that the risks are going to become too great. It's maybe okay to have open weights for some weaker models, but when we're talking about the most powerful models, and if China gets there first and puts these things out in the world, there's no taking it back. The genie's out of the proverbial bottle. That's where anthropic lies on this. It's what stands. The one argument I've seen is if you believe we will have more powerful models that present great risks to society now or in the next 18 to 24

1:02:28months, then the idea of making those open for anyone to build on seems ludicrous. And yet, it seems like most of the industry thinks it's cool and we'll figure it out and we'll just build

1:02:41competing models faster that can stop the bad guys. Yeah. And tying it full circle back to that first topic, look at what these models are now capable of. And especially when they become, when we're talking more agentic, you know, models are very powerful models within agentic harnesses. It's like, you can start to see why some people might be concerned about putting that power in anyone's hands. Yeah. And there's, there's some, you know, seems to be increasing assumptions that anthropic open AI and maybe even Google may already have recursively self improving models internally that

1:03:17they're not releasing. And again, we're then entering a realm where they can't even monitor the agents they already have. The thing got out for nine days before they realized it was hacking somebody. And so we're just supposed to trust that not only these frontier labs can handle what their own building, but that we're just supposed to let the broader world with some bad actors in it, some foreign governments, you know, that maybe you don't trust that. I don't know. It's like, it's always been my struggle with open source. That's why I said, like, I'm still conflicted

1:03:48personally on this. Like I get the benefits of open weights. Like I truly do. And I don't dispute those at all. But I feel like some of the acceleration is just throw aside the possibility that maybe it just doesn't go right. Maybe we can't keep up with these agents as they go out into the wild. If you, you know, give open weights or open source to the most powerful frontier models, or I don't know, like I, I don't understand the assumption that it all just works out. Okay.

1:04:22It doesn't seem like they have a grasp on it to be able to say that. And yet that's just how they position it. And all it takes is another or bigger horror story of some of these models being used at some point for the government to feel it, it's forced to act or it has to do something. If this, if enough damage is done by one of these models, or there's a lot of unintended consequences or malicious usage, you could see a pretty visceral reaction, which is obviously like why this letter is, is being promoted.

1:04:52Cause there, I would imagine people are worried about the response from the government. Understandably so. Understandably so. All right. So before we dive into rapid fire this week, Paul, this week's episode is also brought to us by something we're very excited to announce and start getting out into the world, which is AI transformations, which is a new limited podcast series presented by Google cloud. It is a six episode series of the artificial intelligence show. And on it, I'm actually sitting down with business leaders

1:05:25who have actually started doing this stuff we always talk about, which is using AI to transform how their teams, their operations, their companies work. So you're listening to this on Tuesday, July 28th. If you are listening to it right when it drops, our first episode of AI transformations will also drop on Thursday of this week, July 30th. And after that new episodes of the series will drop occasionally on Thursdays right here in your artificial intelligence show feed. So it'll just be like a normal podcast episode release. And every episode, we're going to try to talk through the full arc of

1:06:00a company's AI transformation. So, you know, the old way of doing business before they started truly transforming with AI, kind of the aha moment they had that sparked change and, you know, some practical, even sometimes messy details of how they went from day one to real results. And we'll talk a bit about some of the results that are currently unfolding. Many of these stories, like everyone's AI transformation story is in progress, but we're going to always try to end these episodes with actionable advice for anyone undertaking their own AI transformation. So, you know, no change to our regular weekly

1:06:36scheduled programming. We'll still be doing the regular AI answer series. You'll just now get AI transformations as well. So super excited to bring this series to everyone and so appreciative to our friends at Google Cloud for making it possible. Yeah, this one's been in the works for a while. We actually envisioned this series last year, last spring, and we've looked at different ways to kind of bring it to the world. And this is the first, like there's some other plans for how we're going to do this. There's some cool things we're working on specifically for our AI Academy members, but I can't wait to hear these. Like Mike's doing these interviews. I'm not, I'm not a part of the interviews. And I was catching up with

1:07:11Mike on Friday and he was giving me the rundown on the first few that he's like conversations he's had. So I can't wait for him and we appreciate the people who are going to be a part of the series to or take the time to share these in progress stories. All right, let's dive into rapid fire for this week. So first up, Google DeepMind CEO Demis Hassabis recently published an essay on X calling for the US to establish a new standards body for frontier AI. He is recommending this be modeled on FINRA, which is the self-regulatory

1:07:41organization that oversees the financial industry. And basically in this essay, he writes that AGI is probably only a few short years away. And that when we look back on this period, we will realize we were standing in the foothills of the singularity. And Hassabis argues AGI's impact will be perhaps 10x of the industrial revolution at 10x the speed, but warns that the industry is locked in an extremely intense multi-layered commercial and geopolitical race in which advances on the frontier are outpacing

1:08:13our understanding of the technology. So his proposed standards body would be a federally overseen public private partnership funded mostly by industry with a board that includes independent technical experts and open source representatives. It would develop benchmark thresholds that determine which models count as frontier class and which organizations qualify as frontier labs. Now under this framework, anyone designated or qualified as a frontier lab would initially share models with the body voluntarily

1:08:46up to 30 days before release for testing in high risk areas like cybersecurity and biological threats, plus agentic tests that look for attempts to bypass guard trails or signs of deception. That would be helpful. That would have come in really handy in the last couple of weeks. So once this process is proven effective, it would then become mandatory with frontier models required to pass before they could be deployed in the US market. The framework would apply to frontier models regardless of their country of origin or whether they are open or closed, while non-frontier models from startups and

1:09:21academia would be exempt. Hassabis also says the approach could be ratcheted up if needed, including coordinating a slowdown in development among the frontier labs if deemed necessary. So this picked up some pretty notable early backing when Microsoft AI CEO, Mustafa Suleiman, who co-founded DeepMind with Hassabis, wrote that he fully supports it and added the time for us all to act is now. So, Paul, it's a pretty interesting proposal here from Demis, especially in light of what we've already talked

1:09:51about so far. Yeah, it was interesting. When I first read this, I actually didn't, you know, because I get alerts anytime Demis tweets something and I scanned it and I was like, oh, okay, yeah, this sounds a lot like what he's been saying for a while. Like, I didn't actually read it as anything groundbreaking or like main topic worthy from our perspective. And then as like the couple days following progressed and all these other people started like commenting on it and retweeted, I was like, is there something different here than what he's previously said? Like, I'd have to go back and like check my notes. And I don't know if it's just, again, the moment or like the formalization

1:10:25into this format that allowed people to then, you know, retweet it and comment on things like that. But I feel like this just builds off of a lot of things he's been saying publicly for a while of what was needed here. And the, you know, the support just kind of jumped on on board with, you know, the idea. And so my general take without going into great detail about the proposal here is, we need something. And it seemed like a lot of the industry people felt like this is a really good

1:10:55direction. That's a positive thing. I don't know about this idea of like ratcheting, you know, things get ratcheted up that we have to like slow down. That's going to require collaboration with China. But actually, when you go back to the Ezra Klein episode I referred to earlier, Kevin Rudd addressed the idea of like, even though China views itself in this like supreme position, and that leads to a lot of decisions that are very competitive, it also allows them to collaborate on

1:11:27extremely important things that stabilize for everybody, like response to COVID as an example. But he specifically called out AI regulation as one of those issues, that China needs stabilization when it comes to the AI industry. And that at some point, that is one of the few items that there could possibly be agreement on, that you could get on board with each other to do something. And I found that really fascinating because you just kind of always assume like, we're not going to come to agreement on anything.

1:11:59Yeah. But they said specifically, AI regulation is one of those things. And I think whatever we do in America is going to have to have the support of the Chinese government as well. We're going to have to find a way to collaborate there, despite our differences. So.

1:12:17So some other Google news here, we wanted to quickly kind of report on Alphabet, Google's parent company, their blockbuster second quarter results that they had revenue of almost 120 billion, up 24% year-over-year. Profit quadrupled to 112 billion. But a lot of this came from gains on Alphabet's stakes in other AI companies, including SpaceX, which went public in June. Google Cloud was a big standout here. It grew 82%, up to 24.8 billion. However, there was a soft spot where search revenue

1:12:50came in slightly below expectations. Interestingly enough, we saw the costs of AI build-outs showing up in a big way. Capital spending doubled to almost 45 billion dollars for the quarter. This exceeded the cash that Alphabet's operations generated. So that pushed free cash flow negative by almost 6 billion dollars, reportedly the company's first negative free cash flow quarter since it went public in 2004. Their CFO said the vast majority of the quarter's capital spending went to technical

1:13:22infrastructure supporting Alphabet's AI investments. Now, on top of this, Google at the same time launched several new models, including Gemini 3.6 Flash, 3.5 Flashlight, and 3.5 Flash Cyber. CEO Sundar Pajai said the delayed Gemini 3.5 Pro remains in testing and that Google has begun its, quote, most ambitious pre-training run yet for Gemini 4. He acknowledged that Google's models have lagged rivals on coding but said there are many attributes on which we are still at the frontier.

1:13:55So, Paul, what do you make with the numbers this quarter? It's a little mixed results with the kind of negative free cash flow, but it sounds like a lot of money being spent on AI infrastructure. I think they believe that a lot of people are going to spend a lot of money to access on-demand intelligence in the future, and they're going to keep building out the infrastructure to allow for that. I think Google Cloud is going to just continue to grow because of that demand for intelligence and inference that serves it up. I feel like the last three or four months have been tough from Google

1:14:34and their AI perspective because they have very clearly fallen into third place at best right now, I would say, from a model perspective. I don't know. I think if you go back to conversations we had last summer, last fall about Google and their unique competitive advantages, I wouldn't put too much into the last few months. I think that when you look broader at the things that they have that the other model companies don't have, I don't know that it's going to be 3.5 Pro or 6 Pro, whatever,

1:15:08but I think Gemini 4, the whole idea of the Omni model, the ability, you know, I don't know. I just, I think at some point in the next few months, Google will jump, jump back up there. Yeah. Okay. So next step this past week, the White House released what it bills as the first comprehensive rethinking of the U.S. science enterprise in more than 80 years. And this is a plan to redirect the government's roughly $200 billion in annual research budget away from

1:15:38universities and toward individual scientists and AI in a bid to outpace China. So this report came from the Office of Science and Technology Policy titled Science, A New Golden Age. And it argues that scientific productivity has slowed and proposes funding individual researchers directly while minimizing universities' involvement. That would upend a system that's been in place since a 1945 blueprint came out basically, which directed the government to fund basic research and universities

1:16:08conducting it. Now, AI is at the direct center of this plan. A memo from OSTP director Michael Kratzios, who we mentioned before, and budget director Russell Vaught, instructs agencies to fund research that uses AI as a new instrument of scientific discovery, not merely as a tool to augment existing capabilities, and also has national missions outlined in robotics, quantum computing, nuclear energy, and space. So, Paul, I mean, even more here from the administration aimed at winning the AI race against China.

1:16:40Yeah. I don't understand the implications quite yet to what this means for the universities. Yeah. It seems like a very significant change. I haven't had a chance to, you know, check in on my sources that I follow that might be commenting on this, but it seems like this is going to be a pretty big shift for how universities are funded, how independent, and maybe it drives more. And my initial reaction, again, I don't know if this is right, is there's already tremendous pressure on professors at universities who are doing research at universities to just go work for the labs.

1:17:15Yep. And I feel like this is just going to accelerate that, right? Like, I mean, that's what it seems like. Yeah. It's just like, you're not going to get the funding you want there. Like, okay, I'll just go make five times more money working at a lab. Yeah. And that doesn't seem like a great solution to education in America. But yeah, again, I'm not going to, like, throw much editorial at this because it's not a topic I feel very confident, you know, offering opinions on at this point. So, next up, we had two other, call them milestones if we're talking about the backlash against AI,

1:17:50specifically data center construction. So, first, New York became the first state to pause construction of massive new data centers. And in the past weeks, opponents staged the first coordinated nationwide protests against data centers with 142 events across 42 states. So, first, New York Governor Kathy Hochul announced a one-year moratorium on new data centers of 50 megawatts or more while the state develops what it calls consistent standards for responsible development. She said that data center growth

1:18:23threatens to hike up utility bills, deplete our natural resources, and create uncertainty for New Yorkers. She also plans to repeal the state's sales tax exemptions for data centers. And second, these protests were coordinated by a group called Humans First, co-founded by former Tea Party leader, Amy Kremer, who compares this movement to the Tea Party's early days and predicts data centers will be a defining issue in November's midterms and the 2028 presidential race. Texas, which is a data center

1:18:54hotspot, hosted the most protests of any state. They hosted 18. Also, a June Reuters Ipsos poll found that only a third of Americans approved the pace of data center construction. Just 14% would support a data center being built in their own community. And researchers say that community opposition has now blocked or delayed nearly 130 billion dollars in projects this year. So, Paul, we've got these companies spending so much on AI infrastructure, but it sounds like this is not welcome in a lot of communities. I'm curious,

1:19:30you know, that quote about the 2028 elections, the midterms, that sounds like a lot of like what you've been saying about this issue. Yeah, it seems like data centers, like we've talked about data centers and jobs seem to be the two wedges that, you know, politicians can use to create division and drive votes one way or the other. I would say security might become the other one, like fear. They're going to push on the fear of cybersecurity and risks and things like that. So, I do firmly believe that AI will be front and center. You know,

1:20:02not only know the midterms, but the 2028 election cycle in the US. The data center issues like it's just really messy. We talked about that we did an AI and CLE event last week, and this is one of the topics we sort of touched on with that group of people. And I was saying is like, it's just a hard topic. The people opposed to data centers have really good arguments why they're not great. I don't, you know, I wouldn't want one built in my backyard. But they're also, you know, you have to look to the future and think

1:20:34about, well, what is the importance of the data centers? What are the positive impacts on society that can come, like a medical breakthrough, scientific discovery? And do we not want that? And so like, do you really not want data centers? Do you not want the benefits that come from them? And maybe you don't. I don't know. But I don't know. I feel like there's just a lot more public dialogue that has to happen. I think there's very low awareness and understanding of what the data centers are being built for. And the AI industry doesn't have the greatest reputation at the moment. And that's,

1:21:05you know, a big part of it. So this is just a really complex issue that spreads across a lot of areas. But there's no debating. It's going to be a political issue. And it's a societal issue for, you know, it's going to keep growing.

1:21:18Next up, Anthropics head of economics, Peter McCrory published an essay on X tackling one of the bigger questions in AI that you just alluded to, Paul, which is why hasn't AI increased unemployment? According to him, this is synthesizing basically 18 months of Anthropics economic research. And his argument right now at least is the fact that the US labor market is stable and close to maximum employment with unemployment at 4.2% in June. He says that AI adoption is high enough that

1:21:50the effects of it should be visible with about 20% of US firms using AI in at least one business function. That's 40% in the information sector. And he argues that AI is showing up in productivity instead. Labor productivity grew 2.0%, 2% per year from early 2022 to early 2026 versus 1.6% in the four years before the pandemic. And sectors with higher AI adoption, he says, have seen faster productivity growth. But he finds no material increase in unemployment even for workers whose

1:22:25roles are most exposed to AI automation. So his explanation is so far, AI is a skill-biased, the labor-augmenting technology. Model capabilities remain stubbornly jagged. So you need human experts to still direct the work and recover when the models make mistakes. He says there's no occupation in the Department of Labor's task catalog where a tool like Claude can systematically handle every single task. And Anthropics Claude code data shows people with more domain expertise succeed more

1:22:58often. He does flag some caveats, though. He says there's suggestive evidence that hiring for young workers in highly AI-exposed roles has weakened, though he attributes that to also broader economic factors rather than just AI. And overall, Paul, it's kind of interesting. He does not expect AI to make unemployment noticeably higher a year from now. I'm curious, what do you make of those arguments? Is it possible AI won't actually impact unemployment? Sure, it's possible. One, you have to understand

1:23:33that Anthropik doesn't believe that. Dario Amade does not believe that employment won't be dramatically impacted. So they're looking at data. They're looking at a four-year run of data. So going back 2002, 22, 23, 24, it's basically meaningless. That data doesn't do anything. It was before reasoning models. So any unemployment data or any trend data related to jobs prior to, I mean, really early 2025,

1:24:04because the first reasoning model didn't even come out until 24. And then we didn't get semi-reliable agents until fall of winter of 25. It's just, I don't know. It's fine. I mean, we can talk about all these economic studies we want that's looking back at the last few years and saying, oh, I don't see it yet. It's not happening. It's actually jobs are growing in high growth companies. Yeah, of course they're growing in high growth companies. But the thing I would say is adoption is so jagged.

1:24:34Not only is the technology itself jagged, it's the adoption of it that is so low. And I think that's the thing that's just always left out of these conversations is how few companies are actually using reasoning model and agentic capabilities, even using them at all, more or less like using them to their optimal possibilities. And so I hope that people don't get a false sense of hope when they hear these studies that AI isn't affecting jobs. Again, I would love to be wrong on this. I

1:25:08really, really want to, three years from now, see a study that says, nope, reasoning model. Every company has adopted AI. They've done personalized training of those people. Everybody's been given the tools and education. And somehow magically, we have millions more jobs than we had in 2026. I pray that that happens. I don't understand how it's possible, but I really hope that that's what the research shows. But right now, any report that's telling me anything prior to 2026, it's almost

1:25:43meaningless because companies didn't have reasoning capabilities and they didn't know how to use them. And agents are still so early. And once those two things mature, then let's have a conversation about the impact on jobs. In your opinion, are they just kind of overlooking that fact or intentionally just like, hey, let's put out research? You have to figure they have thought about that. It's always just a disclaimer. They're doing what economists do. They look back to try and

1:26:14predict what happens in the future. I just try and take a first principles approach and say, let's imagine we're creating an AI native company from the ground up today that has access to agents and reasoning capabilities. We only hire AI forward employees. We train every one of them how to use the models. We connect them to the right data. We do the thing we talk about. There is no scenario possible where we need as many people as we did three years ago. None. And you can't argue with me that there is. We will grow. We will hire people as an AI native company, but we will never need as many

1:26:50humans in the future as we did in the past to do the things that we plan to do. So that's where I just come from. It's like, I get it. I love the economic data as much as anybody looking back at history and industrial revolution. It's all great. But when I think first principles about what happens next, I can't come to that conclusion that jobs don't get dramatically disrupted.

1:27:15So next up, we had Wharton Professor Ethan Mollick, who we talked about quite a bit. He published his summer 2026 edition of this recurring guide he puts out to which AI to use. And this is pretty relevant to everything we've been talking about, things we've mentioned on the show, especially in 2026. He outlines there's this big shift happening in AI where using AI no longer just means chatting with a bot. It now means agentic systems that pair a model with a computer that it can then use to do the equivalent of hours of human work in one go. So he kind of outlines some advice

1:27:50and how to think about your available models and agentic systems and tools. He says for low stakes tasks, the free default models from any of the major labs are all good enough so you can pick whichever one you like. But for high stakes questions like a second opinion, for instance, on a medical or legal concern, he recommends the most advanced models, which would be Claude's Opus and Fable or ChatGPT's GPT 5.6 Sol set to high thinking levels because their error rates are meaningfully lower. And he says for real work, he argues people really only have two choices. He says

1:28:26ChatGPT or Claude starting at $20 a month. Each has a mode where the AI works on a company-provided computer, i.e. from one of the labs like in the cloud, or a more powerful one that runs on your own machine. We've talked about these ChatGPT work and Claude co-work, essentially like a cloud-based agentic system. Codex or Claude code would be something similar but running on your own machine. He warns that you've got to be careful about the approval settings on these because there are things like prompt injection attacks and also these tools can, you know, send stuff, delete stuff,

1:29:01depending on what they have access to. He did mention that Google basically has no leading frontier model and does not suggest Gemini as your primary system right now, though that can change. So, Paul, I was curious. I know you had sent this around to our team, too, as super important to read. I found this like one of the clearer explanations I've seen for where we're at right now and something really important that I try to communicate in our talks and workshops and courses is like this is the shift happening and understanding that it is happening and the differences between

1:29:35these tools and systems is something every knowledge worker is going to have to learn quick. Yeah, Ethan does a great job of just writing really approachable stuff and that, you know, makes a complex topic pretty easy to follow. Yeah. Yeah, I think for, you know, more power users like you and me, Mike, who are in these models every day, I still found value in kind of reading through it and hearing his context and how he explains the difference in things. I think for people who are more, you know, beginner to intermediate,

1:30:05who maybe don't really even know the difference between the models or like why you would use Claude versus ChatGPT, things like that. It's very instructive, especially when you start getting into the work and co-work stuff. I thought that was a really helpful section. The agent stuff was helpful. Yeah. So I don't know. I just, I love practical guides that are no fluff, that aren't just click bait. Like, I don't think Ethan Mollick is tracking his, you know, clicks and views every day. It's not why he's doing it. He's doing it because he's a researcher and he's trying to share useful information. So we always like putting a spotlight on people who are just trying to create value and help people

1:30:41figure this all out. This also, I think, to me, points to just the evolution of the workforce. And like, think to yourself, like who in your company knows this stuff? Like who on your team has any clue about this stuff? And so when you're being, you know, giving your employees co-pilot or Claude or ChatGPT or whatever, who's teaching them which models to use when and why higher thinking matters? And should we be allowed to connect these things to our Google drives? And like, if no one on your team can answer those questions, that you need to be really

1:31:16thinking about creating a role because someone has to know this stuff. And it needs to be a very dynamic learning environment. And we think about this with our own AI Academy, like constantly thinking about how to make what we're teaching more dynamic so that as this stuff changes, like I saw Malik even tweeted, I think on Sunday, he had to update this post over the weekend because Opus 5 got released. Right, right. Yeah. Yeah. And I think it's a good reminder for folks too, because like, if you have not been deep into the tools in the last, call it two to three weeks, like a lot has already

1:31:49changed. And if you weren't someone that was on top of like Claude code or even Claude co-work, ChatGPT work is very new and like basically just turned on a bunch of agentic capabilities for non-developers that like you've got to understand real quick. So it's super helpful from that respect too, to just like get back up to speed with this. Okay. So next up, we have our AI use case spotlight, where every week we give you a quick look under the hood at real AI use cases we're exploring,

1:32:20building, or deploying in our own work. So this is related on my end, Paul. One thing I just randomly and recently learned is that Codex, at least in the Mac app, can coordinate work across multiple chats just through natural language instruction. So this isn't just about keeping chats in one folder. Codex can actually just go reference one chat or another, compare what happened across chats, summarize useful lessons. And actually, this is the part I found super helpful, actually go update other chats

1:32:51with different instructions based on what each one is supposed to do. So it was kind of this thing that clicked for me that chats do not have to stay isolated. Again, I don't know if this is a new feature, if it just existed. As Malik points out, the labs are terrible at documenting a lot of these things. But what was really cool is as I was doing a project where basically I had three separate chats, and I was preparing for interviews. So each chat had its own kind of interview context on the person, some individual research in each of these chats. But what was cool is, as I prepared for the first one,

1:33:27I learned a lot about the process as I was going, I was like, oh, I came up with a good way to do these questions or to format this document. And I just said, go tell the other two chats. That's what we're doing now. So when I jump in there to finalize these things, it's already learned from the first iteration of this. So, you know, it's a small thing. And I'm sure there's plenty of other ways to get to that outcome. But I'm finding it weirdly useful to be able to know that you can just say, hey, go look at those three chats or go tell, learn what you, learn the lessons across these three

1:33:58activities we did. Start a new chat that teaches that chat how to do this. I was just, I found that to be cool. And also like, I think I saw it in a random tweet because like, good luck finding, maybe this is documented somewhere, but like it was not advertised to me as I was exploring. Yeah. I don't know. Every time I hear you talking about the stuff you're doing, I was like, man, I got to find time to like, sit down and get demos from you. I'll just do two quick ones. So one, just super practical. When I was, you know, relevant today's episode, I was trying to figure out the ways

1:34:31to explain open source versus open weight. I did my usual Google search, just trying to make sure I was understanding it properly, read a few posts. And then I just went to chat GPT. And I just have a podcast project in there that's got the basics about our podcast. And it's like, all right, I want to like, you know, explain to the audience the difference between these two things. Like, let's talk about this basically. And it gave me, you know, again, the back and forth conversation. And I would push on a little bit, didn't like the cake analogy. And so more is, again, I generally focus on as a thought partner, and that's kind of how I used it in that instance. And then one other

1:35:02one just in the personal life, because again, I think people forget this exists. So Google Gemini, I love using the video capability where it can see what I'm looking at. And I was troubleshooting a piece of equipment on Sunday. And so I just go into, you know, go into the voice mode, and then there's the camera, you click the camera. So this is just the Gemini app on my phone. And I'm like, all right, I can't get this to work. I don't know what this code means, this air code. And it tells me and I was like, all right, what do I hit? What button should I hit? It's like, oh, the button on the right, push that. It's just like walking me through as an advisor, because it's seeing what I'm seeing. It's almost

1:35:35like when you let like an IT person, like, you know, remote into your computer and they just take control and like do the thing. It's basically that. And so just don't forget that if you have Google Gemini or ChatGPT, they both have a vision capability where you can turn your camera on and you can talk to it about what it's seeing that you're seeing. And you can solve things. I love that feature. And sometimes I forget that it exists. Yeah, same. All right, so we're gonna wrap up with a bunch of product and funding updates. Paul, like you alluded to, there's just a ton this week. And any one of these could have

1:36:08been easily its own topic. So we're not skimming over it intentionally. We'll probably talk about these more. But just to kind of make sure we hit on all these developments, I'm gonna just dive into each of these quickly. So first up, Anthropic launched Claude Opus 5, a model it says comes close to the frontier of intelligence of Claude Fable 5 at half the price. It has posted state-of-the-art results on benchmarks like RKGI 3. And it is now available across Claude.ai, Claude Code, Co-Work, and the API. OpenAI launched health in ChatGPT for all US users 18 and older across

1:36:46every plan. This is a feature that lets people securely connect Apple health data and medical records from systems like Epic and Oracle Health. So ChatGPT can bring personalized context to the health related questions it gets from users. OpenAI also launched its first hardware product, Codex Micro, a $230 mini keyboard developed with keyboard maker Work Louder, as the company's called. It features 13 mechanical keys, light-up agent status keys, and a dial for adjusting reasoning levels,

1:37:17all designed to specifically help people manage fleets of Codex coding agents. OpenAI's first consumer device will reportedly be a movable screenless speaker built as an AI companion, so it's a more general kind of market device, not just for Codex users, according to Bloomberg. OpenAI also formally pushed back on Apple's trade secret lawsuit over its hardware ambitions, as Apple reportedly is escalating that fight by sending letters to dozens of former employees who joined OpenAI. In other news, Anthropic and Wall Street firms, including Blackstone and Goldman Sachs,

1:37:54officially launched Ode with Anthropic, which is the $1.5 billion enterprise AI services joint venture that embeds forward deployed engineers directly in client companies, including the backers portfolio companies, in order to deploy Claude. Anthropic added Claude Fable 5 to all max and team premium plans starting July 20th at 50% of normal usage limits, so instead of that usage-based only pricing that they were going to do for it, they've actually started to include it in some of these plans.

1:38:26Anthropic also rolled out skill teaching in Claude Cowork. This lets users teach Claude a repeatable skill by recording their screen and talking through a task as they do it. Anthropic also launched Claude for Teachers, which gives verified US K-12 educators free premium access to Claude, along with tools for lesson planning, assessment creation, and parent communication. Google also renamed Notebook LM to Gemini Notebook, keeping the same product for its more than 30 million users, while adding a secure

1:39:00cloud computer in every notebook that can write and run code. HubSpot launched Agent Hub and Agent Builder