Nathan Goes to China – Part 2: AI Safety with Chinese Characteristics

0:00Hello, and welcome back to The Cognitive Revolution. This is going to be part two of the Nathan Goes to China series, and we're calling it AI Safety with Chinese Characteristics. Part one, if you haven't heard that, is up on the feed. It's been up for a few days, and in that part one of the series, I really just laid out the tech setup for going to China, what you would need to do if you want to get a cell phone ready, the apps that you need to download. I also shared some of my experience using Chinese AIs on the ground there,

0:30and then just shared a bunch of observations based on my two weeks in China and many conversations I had. I appreciate the kind comments that I've got in response to that one, including at least one from a listener in China, which was probably the one that mattered to me the most. I think if you've been to China in the last few years, you can probably skip that one, but if you're interested in this, you might also be interested in that, though I do think they should be pretty self-contained. So part one is a little bit closer to a travel log and tech review, and this episode is going to be really focused on the Chinese AI safety ecosystem

1:06and trying to just describe it and, as much as possible, understand it on its own terms. As we saw last time, there are a lot of similarities. Just as there are major similarities between American big tech and Chinese big tech, there are a lot of similarities between American AI safety and Chinese AI safety communities and the work that they're producing, but there are also some differences, and I think it will definitely be helpful if we have a better understanding of those. Before I really get into it, just a couple of quick disclaimers again.

1:38I will be, again, following a Chatham House rule for this episode, not because I was asked to do that, but just because I want to keep things simple for myself and make sure that I'm protecting everyone that I talk to from being misrepresented by me and put in an uncomfortable position. So I won't be naming any names or attributing things to the organizations or institutions that people are affiliated with. The one exception to that for this particular episode will be when I'm citing published papers

2:10or reports, then those I can give you the name because those are out there in the public domain, and there are some good sources that I'll mention and we can link to in the show notes of this episode for anybody who wants to go deeper, which, of course, is always recommended. The other point of order or clarification, just in case anyone is wondering, I have no financial conflicts on this matter. I took this trip paying my own way, flights and hotels, all that stuff. The biggest thing that I did, except from some of my gracious hosts, were a number of meals,

2:44probably half a dozen meals over the course of the two weeks. And I think if you have listened to me this far, you can probably feel pretty confident, as I do, that accepting those meals has not overly colored my take on what is going on in the Chinese AI safety community. So with that, let's get into it. I think so often, and this has faded, I think, in recent times, as the situation arguably has just gotten a lot more real quick,

3:17especially with things like the recent open-face incident, where it's starting to become undeniable that we have some real AI safety problems on our hands. But still, you do hear from time to time, and you used to hear it, I think, more, that we could never possibly slow down our AI race. We could never regulate. We couldn't impose any duties or obligations on the frontier companies. Because why? China doesn't care about AI safety. China will never slow down. I used to call this the but-China endpoint of so many AI safety and regulatory discussions.

3:52And I think if nothing else, I hope that this episode serves to really disabuse people of that misconception. I think it should become clear by the end of everything that I'm about to take you through that China does care about AI safety. It's not the only thing that they care about, certainly, but they do care. However, China has, at times, slowed down their AI companies in the name of safety. Not necessarily existential safety, but safety as they understand it, domains that they care about.

4:25And so I think that coming out of this, we should have clarity, at least on that basic point, right? That there is a there there, that people in China do really care about these things, and that the Chinese government is willing to take action when it deems it necessary. So that doesn't get us all the way through to an international treaty, obviously. And that's going to be the subject of the third episode, an analysis of the U.S.-China relationship, obviously, through the lens of AI issues and what we might ought to try to do about bringing the relationship into a more productive state than it is today.

4:59Today, we're really just going to focus on what is going on in China with respect to AI safety. But I think that this foundation is really critical, and hopefully this will become an artifact that people can share with those who, if they're open-minded enough to listen to something, but have the misconception that China doesn't care, or China will never slow down, or whatever. We'll be handing the race to China if we do anything other than just race full speed ahead into recursive self-improvement. But I think that really, in plain terms, does come from a position of ignorance,

5:29and hopefully we can eliminate at least a portion of that over the course of this episode. Okay, to start off with, I wanted to just do a real quick factual rundown of, like, where the two ecosystems are today when it comes to the actual safeguards that they have in place on their frontier models. And I think this is important just because it is grounding, and I am generally someone who comes off as a China dove and quite conciliatory

6:00and tries to find the positive in things in general, not just in China. And so people might accuse me of burying the lead or shying away from contradictory evidence. So I figured I would just lead with this up front. I think it is fair to say that, as of now, Chinese AI companies and models do not have as strong of safeguards protecting the public against the potential for misuse of the models as the American companies do have.

6:31So that, I think, is important to just state very plainly and be quite direct about. The difference is, more than we might like to admit, the difference, I think, really is driven by two companies in the United States. I just did an episode not long ago with Adam Gleave, which talked a lot about jailbreaks and robustness to jailbreaks and all that kind of stuff. And what we saw in that episode was OpenAI and Anthropic, which I'm maybe going to start calling OpenAnthropic as they become the duopoly that is leading both in capabilities

7:06and, fortunately, happily, also in terms of their safety measures. Those two companies are really doing a lot to bring up the American average. If you were to subtract those two companies from the mix and you were to look at the rest of the American companies and how they compare to the Chinese companies, honestly, a lot of the safety differential that gets reported would disappear. And I think the story would look a lot more muddled. I do think the U.S. companies would still have a bit of an edge,

7:37but it would be a pretty slim edge. And it certainly would not give the American ecosystem some sort of obvious moral high ground from which to proclaim that the Chinese are not doing a good job. So that, I think, is, again, really important to understand. There are, in each country, depending on where you want to draw the line on Frontier, if you draw a permissive line on what it means to be a Frontier model maker, there's something like 10 to 12 in each country. And again, that is quite permissive, right? That's the most close watchers of the American AI ecosystem would not count 12 companies as being at the true Frontier.

8:15But if kind of, say, near Frontier, there's like 10 to 12 in each country. And it really is the top two companies in the U.S. that are doing a lot of work of bringing up the average, especially the weighted average in terms of what it is that people actually use on an ongoing day-to-day basis in their lives. And Google with Gemini is, like, not doing as much or as well to implement safeguards as OpenAI and Anthropic are, but they're also, like, ahead of the pack at least a little bit.

8:46And again, kind of would bring up the average, or if we didn't have OpenAI and Anthropic, we might try to hold Gemini up as a standard. It wouldn't be a standard that's, like, so much better than the Chinese companies, but it would be at least a little bit better. So, again, to state it plainly, on average, especially if you take a weighted average by what products people actually use, the American companies are ahead in terms of having better safety practices, better safeguards, also better funnel-through on their model cards and their commitments to publishing safety evaluations.

9:18We'll get into that a little bit more later. But the American ecosystem really is being carried by a couple of leading companies. And beyond that, the mixes, the relative positions are much less differentiated than we might like to think or we might look at if we just see the headline numbers where we are including OpenAI and Anthropic in those measurements. Okay, so just a little bit of kind of intellectual history of Chinese AI safety.

9:49Broadly speaking, I would say it is a pretty pragmatic bunch of people with pretty pragmatic ideas, pretty grounded, pretty normal-seeming people. They're not, like, the crazy sci-fi people. They're not so influenced by sci-fi tradition. They're really just looking to make things work in a kind of practical, one-generation-to-the-next sort of way for the most part. A good source that I could point you to is the director and chief scientist of the Shanghai AI Lab.

10:21That's a major AI institution based in Shanghai, of course. His name is Zhao Beowin. Again, I apologize to everyone and especially my Chinese listeners for being just terrible with Chinese. Joke about that over the last couple of days has been that, in general, I feel pretty young for my age, and I'm, like, pleased with how agile I feel like my mind is. But when it comes to getting Chinese pronunciations right, the neuroplasticity is low, and I'm struggling. So, anyway, I apologize for that. But Zhao Beowin, he is the director and chief scientist of the Shanghai AI Lab.

10:56And at the 2024 WAIC, that's the same Shanghai conference that I just attended, but obviously two years earlier, he introduced this concept of the 45-degree line. This speech is available. It's on government websites. You can go find it. But the idea of the 45-degree line is that capabilities and safety measures should grow together, right? That's the 45 is kind of the Y equals X, the slope of one. As your capabilities rise, your safety standards and measures also need to get stronger and stronger.

11:32And as long as those two grow in tandem and the safety measures are up to the challenge presented by the capabilities at any given level, then you're good. And probably these things in this telling kind of naturally should evolve together and both should develop in tandem. The idea is that this is important, right? We don't want the capabilities to get far ahead of the safety measures. But I think also implicitly is like there's not really much point and it might be sort of a waste of time or perhaps even counterproductive to try to get the safety measures to be super robust relative to capabilities that don't exist yet.

12:04So the 45-degree line has been at least like one pretty broadly accepted guiding principle, I think, how the Chinese AI ecosystem at large thinks about AI safety. Now, we can ask, of course, how's it going? And I already kind of spoiled the answer that the American companies are doing better, but we can dig into that in quite a bit more detail. Concordia AI, which is led by past podcast guests from about a year ago, Brian Say, they maintain a website called AIonitor.net, where they run a bunch of evaluations and plot them.

12:42And you can see just a really, honestly, almost overwhelming amount of detail on all the different models that they've tested on all these different benchmarks. You can really dig into it in quite a bit of detail. The graph that I found to be the most informative was the one that compares proprietary API models, which are mostly American, to open weights models, which are mostly Chinese, and finds that for all the big risk categories that matter, the closed source, the proprietary API only

13:19models are at or maybe a little bit above that 45 degree line. Of course, we have questions of like the metrics, right? They're kind of plotting a capability score and a safety score. And what exactly do these scores translate to in terms of model behavior or what it will and won't do? I haven't chased all these things down to ground truth, but these are composite scores. And what you do see in general is that the proprietary closed source, mostly American models are kind of at or above the 45 degree line,

13:49whereas the mostly Chinese open source, open weights, I should say, at least models are kind of lower and tend to be below the 45 degree line. So they're definitely not doing as well. This is an organization based in China reporting this. Notably, this is a public website. They evidently feel comfortable doing this reporting and calling it how they see it online in a website that is available in both Chinese and English. So this is not like a secret or something that the Chinese community can't handle or would or would, I think, particularly fight back against.

14:27It's pretty much just the facts. And those facts are also definitely echoed by Adam from far in the conversation that we had just a few days ago as well. He said, yeah, we got open AI and Anthropic at the top. Like, it's hard to jailbreak them. Gemini and Grok are like a lot easier. The Chinese models are even easier still. Like, this is a very consistent story across these two organizations on opposite sides of the world asking the same question, coming to the same answer.

14:58Now, I think there's... Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas.

15:33And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind. But then I thought to ask Claude. And sure enough, it identified two great leads.

16:06Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So, for problems worth solving, get started with Claude at claude.ai slash TCR. That's claude.ai slash TCR. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai slash TCR. One thing that is worth keeping in mind for multiple reasons here, which is that sometimes I think there is a little bit of confusion between the open weight model that a Chinese company might release and the actual service that it provides to the public.

16:55I talked last time in my review of the Chinese AIs about how sometimes within the Chinese apps, I experienced this on multiple different Chinese AI apps. Because if you ask a sensitive question, you will sometimes see that the answer is coming in. And then you'll have that old school Bing experience where all of a sudden the answer that you were starting to read disappears and you get refusal. Sorry, I can't help you with that. So, clearly, this is a multi-part system, right?

17:26It is not just the model. It's a model, but then it's also some monitor that sits on top of the model that's classifying or otherwise reviewing the output and can come down and say, nope, we're going to cut you off right there, even though the model itself was happy to answer. So, I bring this up because I think this can sometimes muddle the results and make the differences look a little more stark than they are. I did go into the methodology for the Concordia report, and they did say that wherever possible, they are testing the API direct from the company.

18:00So, whatever systems they have in the API would be included. Do they have the same systems in the API as they do in their first-party consumer app? Not always clear. And then, in other contexts, when you see results like this, like I think when Anthropic does some of their testing and reports on the Chinese models, I think they are generally, if I understand correctly, just testing the model itself without whatever surrounding additional measures it's deployed with in practice in the Chinese economy.

18:33Why does that matter? I mean, I think, as always with these things, like both data points are interesting. The most hawkish AI safety line of thought would be like, well, you put an open-weight model out into the world. Anybody can use it. So, we need to understand the worst-case scenario. I think that's like totally valid and definitely is important for us to understand. And I think the Chinese might underestimate the importance of that in some ways because they tend to think a lot more about services than just the model.

19:05My broad sense is that what is regulated in China is a service. You are offering an AI service to the public. That is the kind of thing that gets regulated. You're putting a model out into the public domain. Okay, who's going to use it and under what context? They seem to kind of have the mental model that like when a company releases an open-weights model, that mostly what's going to happen with that is like other businesses are going to pick it up and they're going to build their own services around it.

19:36And so, if that is another Chinese company building a new service around an open-weights model, then again, that will be regulated at the service level. And I think they sort of expect that like people around the world or countries around the world will probably function similarly where it was mentioned to me multiple times that, look, like these new models are trillions of parameters, right? Like this is not the kind of thing that you can run on your laptop. It's pretty far from it. You need some serious hardware to do any inference with these models really at all.

20:11So, they were kind of like, yeah, it's out there as open-weights, but it's not like any random deranged person can download it to a phone or a laptop, do harm with it. It's really going to be used in the context of other services. And so, we should look more at like the context in which that model is ultimately used than just the model itself in isolation. I do think this is one way in which American AI safety discourse and Chinese AI safety thinking just kind of see things a bit differently. And I think both have quite valid points.

20:42I think the Chinese point is like legitimately apt, right? I mean, it is hard to run these giant models. A random person, certainly somebody who's like having an episode or is experiencing some sort of psychological distress or psychotic break, right? I don't know what the right terms are for these sorts of things. But this is not somebody who's going to go set up the infrastructure to run the 2.8 trillion parameter K3. Now, that doesn't all the risk, but I think it's a valid point that it's not easy to run a model in isolation.

21:14And so, how is it going to be run by whom with like what surrounding stuff? That is an important question that I think is maybe a little bit too, for practical purposes, maybe a little bit too quickly blown past in the American discourse. Although, again, the worst case understanding is also really important for us to have. So, I think both perspectives are valid and I do think there's a bit of a disconnect on that particular point. Now, we could still ask, with all that said, how much do the Chinese companies care about AI safety?

21:47Heard, again, as I said at the top, like they don't care. I think the Chinese system as a whole definitely cares. I did hear somewhat different reports on the companies themselves, especially the younger kind of LLM or AGI chasing startup type companies specifically. There's a few different data points that I could share. One, again, Concordia, they also do a real good state of AI safety in China report, which is, again, public.

22:22They just updated it for right around WAIC. And so, there's a July 2026 edition, which I do refer to or have referred to pulling information for this episode. They said that they had reviewed 10 major Chinese companies' safety disclosure practices, and they found that five of those 10 companies have done, at some point recently, a safety evaluation, which they then published along with the release of the model.

22:56So, five of 10 have at least done something like, in the spirit of your classic Amthropic or OpenAI model card. However, obviously, that means five have not done that. And even the five that did, they said that they're not necessarily doing it on every single release of a new model. So, that definitely leaves something to be desired. I get the general impression that big tech companies, these are your sort of your Alibabas, your Ant Groups, your Tencent, maybe to some extent your ByteDance's at this point.

23:32I mean, ByteDance's a big company at this point. These companies that are very established, that have existing businesses, that are like making a lot of money in their existing businesses, that they are more apt to focus on this AI safety stuff. Why is that? Maybe they just feel like they have more to lose. They really don't want to get on the wrong side of the Chinese government. Maybe they just feel like they have the luxury of paying for it. Maybe it's just kind of institutional culture has matured over time and they do this kind of, they have these kind of practices, right?

24:03Certainly, if you're a fintech company and you're moving money around, like, you're going to be extremely careful about upgrades to that system and thinking about how AIs could run amok in that system. You just have a lot to lose. So, I couldn't pin down a precise explanation, but my general sense is that those kind of big established incumbent big tech companies, they are more inclined to do this sort of safety work than the startups who are, for the most part, like, really just trying to race to catch up and be relevant.

24:36I think this is not too dissimilar from, like, where Meta was a couple years ago when they were waiting Lama 2 and Lama 3. I think the story that they told themselves at the time was, and they do have a policy at Meta and they do testing. And I did actually a brief engagement with Meta as a red teamer for one of the models, which was, for logistical reasons, kind of a total failure in the end. At least my contribution to it was. But I think that they sort of said to themselves, look, like, we're a year behind the frontier or something.

25:12And by the time we bring a model forward with a certain level of capability, OpenAI and Anthropic have already had that model out in the public for a year. And all the jailbreaks, there's been enough opportunities for people to jailbreak it and see what it can do. And we know that these defenses aren't super robust. So, like, if something really bad was going to happen at GPT-4 level or GPT-4-0 level or whatever level they're chasing at any given time, they kind of feel like that probably already would have happened.

25:42And therefore, they can put the model out on an open weights basis and they probably won't be moving things too much. And I think, at least so far, that has been fair enough and history has kind of proven them right. It's not like we've seen crazy stuff happening as a result of LAMA 3.370B that was at whatever GPT-4-0 or whatever exact level it was at. It seems like it's been fine. And so, I think there's some part of the Chinese kind of startup lab type companies that feels similarly where they're like,

26:16look, it's already been out there for a while. If nothing bad has happened, if we don't have any incident reports, then we can probably follow up to that level of capability and not have to worry about it all that much. Will that change now as we are starting to see actual serious incidents being reported out of the frontier companies on the American side? We'll have to see, but I think there's like definitely a very plausible story that a very one might. So, obviously, there's a lot of different takes. There's a lot of different, I can't go company by company.

26:50This is abstracting away a lot of detail. But I think it's safe to say that the 45-degree line, which has the idea as its core, the idea that safety measures should grow step for step with capabilities, I think that's still a bit aspirational in China. I think they could do better. And I think they have a little bit of a reason to believe that it doesn't matter so much because the American companies have already explored what happens at any given capability level

27:23that they are following into. But I also think it's definitely fair to say that AI safety is still very much aspirational here in the United States as well. And we should definitely be keeping in mind that if it weren't for the two companies really at the leadership, at the frontier of both capabilities and safety measures, then the two clusters would look a lot more overlapping and the question would be a lot more muddled than it is today

27:54in terms of like who's really as an ecosystem doing a better job with AI safety and safeguards. Okay, that's the current state of like deployed AIs. There's obviously a lot more going on in terms of research and the role that the government is playing in China. And so I want to get into those next. On the research front, it is very clear that Chinese researchers are increasingly engaged with

28:24and concerned with and focused on and actively shipping research regarding AI safety issues of all kinds. I would say this was probably inevitable just because at least for now, both AI ecosystems are developing essentially the same technology and going to hit essentially the same problems and naturally going to reach for similar solutions, both because those are natural solutions and because there's the opportunity to look at what the other is doing

28:55and try to copy the best of it. I think there has been some really good citizen diplomacy that has gone on as well. While in China, I did, especially at WAIC, I did bump into a number of people who were there to participate in various fora and track two dialogues, some of which are closed door and off the record and give people a chance to be like very candid with one another. And you see some of the people that you would expect to see there,

29:26people who have been arguing very clearly and forcefully in the American discourse that we have to get an international treaty to get this stuff under control, that cooperation with China is on the critical path. I think you can rest assured that those people are acting on their stated beliefs. Like I saw some of the same people saying that stuff online. I saw some of them in China. They are doing the work. And it does seem like it really has at least, again, I think a lot of this probably could have

29:57been expected to happen organically anyway, but I do think their efforts have borne some fruit. I wasn't able to participate in all those kind of track two dialogue sort of things as somebody who's like kind of journalist coded. There were a couple of times where it was like, eh, I think everybody will be more comfortable if we just don't have somebody there whose main credential would be a podcast. So I wasn't quite privy to all of the most kind of candid off the record conversations, but there's just so many moments

30:30where very recognizable ideas or even like American or British AI safety organizations were just name checked directly that it's clear that there is like meaningful cross-pollination of ideas. And mostly it has flowed from the West to the East in this case. It's not, of course, like the Chinese scientists or researchers who are receiving these messages are like taking them uncritically or just doing whatever they're told by whatever Westerners roll in.

31:01It's not like that at all. Somebody actually told me that there are people who view AI safety as some sort of Western psyop designed to slow China down or prevent them from catching up or whatever. So it's like we have our cynical voices. Apparently there are cynical voices in China too that would express concerns along those lines. I didn't meet anyone who seemed to believe that or who said anything like that. And again, my guess is that those kinds of notions

31:31are going to be fading relatively quickly as the clear and present danger of some of the latest models capabilities becomes more widely known. But at least there's been some of that out there. I thought that was worth mentioning if only because it does make clear that Chinese people are perfectly capable of thinking for themselves. They're not just taking whatever Westerners come to tell them. I think what really is happening is that they're pretty open-minded and the ideas are pretty compelling.

32:02And again, the examples are becoming more colorful and more real all the time. Interestingly, one big tech company where I had the chance to, along with several others, to meet with really a pretty impressive leadership roster, they brought out their CMO, their head of communications, their general counsel, their head of AI security, more people beyond that. But I mean, this was a really pretty senior group of leaders at this big tech company.

32:34They said a couple of things that were pretty interesting. One, they just, first of all, like almost pounded the table and said, we really do care about catastrophic risks, like CBRN risks. We absolutely do care about that. So they were very adamant that like at least for their part, they care, they're aware and they care. And they also said that they are following American AI safety discourse on a daily basis with, get ready, an agent that goes out and surveys American AI safety discourse

33:06on a daily basis and gives them a daily report of what is going on in AI safety in the United States. I thought that was pretty interesting. Certainly you don't have too much of that going in the reverse. So again, I think bottom line, engagement has worked. Probably some convergence could have been expected over time, but the people who have been saying we need to work with China have in fact also, at least some of them, been doing the work. And it seems like that work has been going at least reasonably well. And again, I think evidence

33:38will mount as I continue to forward through this outline. One big thing to emphasize too is it's not just content safety. Content safety in the Chinese context is like, again, the three T's, Tibet, Taiwan, Tiananmen. Everybody knows that Chinese models, at least Chinese services, right? Often the model will be more inclined to answer your question. It's then some other part of the service, some monitor or whatever that will shut you down on those topics. But everybody knows that like that's sensitive in the Chinese context

34:09and like companies kind of have to get that right according to the Chinese government if they're going to be operating. But then some people will say, oh, that's all they care about. All they care about is censorship. And again, this is like definitely not the case. The big trend right now, I would say at this point, really, it feels like they feel like they've got the content thing pretty well figured out. All these companies have launched. They're in the market. Things are happening. They're doing business. They're not that worried about

34:40this sort of content safety today. What they are worried about, what they are talking about nonstop is like everybody else, agents. Agents taking autonomous actions. What could happen? Can we keep them under control? Of course, there's questions of like exactly what do you mean by control? But agents, agents, agents, that's what people are talking about everywhere. And the big thing that they just kind of in a very plain spoken way motivate AI safety discussion with

35:10is like AIs have gone from answering questions to actually taking actions in the world. The digital world still mostly, of course, but like they've got a big emphasis on robotics, right? So they fully expect that these agents are going to leave the digital world and find their embodied, successful, commercially viable selves and be out there in the physical world doing things as well. And this raises all sorts of questions of like, geez, if these AIs are empowered

35:41to take action autonomously, we better make sure they're taking actions that are good and that we like and that are not causing big problems. So this is like very down the fairway, again, practical, grounded motivation for these issues. But you hear that, I would say everywhere I went, it was like agents and geez, agents, boy, they can take action in the real world. So we really got to start to get a handle on that. A big thing that I think is very different about the American and Chinese AI ecosystems,

36:12and this is, I think, an important one to understand, is that China doesn't really have the same kind of nonprofit sector that the United States does. There are nonprofits in China. You can set one up. But the scope of what you're kind of allowed or expected to do, as far as I can tell, is much narrower. There's obviously no appetite for political activism, so that's like right out.

36:44And it seems that most of the nonprofits are like service organizations that are there to address some obvious, down the fairway, legible social problem. And the money, the philanthropic side, also, again, seems to be a lot more down the fairway, a lot more just kind of conservative, conventional,

37:14doing things that, like, everybody can agree is good to do. Much less speculative stuff than goes on in the United States. I think, actually, that this is a huge strength of the United States that we should really not take for granted. Like, the fact that we have this civil society where anybody can go set up a nonprofit and have their crazy ideas and go, they don't have to get permission from the government to go try to chase down

37:44an agenda. They just need to convince one wealthy patron that they have something worth chasing. I think that is, like, a great strength for us, and it's given us, among many other things, really the whole of the AI safety community that we have today, right? It was, like, extremely fringe when it got started, but there were a few philanthropists who took it seriously enough to help people keep the lights on and help them do the work that they wanted to do, and sure enough, here we are,

38:15and so many of the predictions that have been, that were made many years ago are coming sort of true or at least, like, true enough to be scary, and it's, like, really good that we have this ecosystem and we just wouldn't have had that if it had to be all approved through the government, and so China, because their nonprofits do have a much more sort of heavy and restrictive government approval process and generally just much more narrow and conventional scope of action, like, they don't really have that.

38:45They don't have, they've never had the opportunity to develop this sort of ecology of AI safety organizations the way that we have here. As a result, most of the AI safety research that you find in China is actually coming out of the universities and, to some extent, the companies, and there are definitely academic industry collaborations as well, but the number one source seems to be the universities,

39:15the sort of analog for the nonprofit sector in the US when it comes to who is producing the bulk of the AI safety work, it's academia in China. And so I think you can compliment them and say, wow, you're, I think it's safe to say that their academia has moved faster than our academia has moved to take up AI safety as a research area, and that's cool,

39:46but it's been slower than our nonprofit sector has. And I think because of the kinds of people that tend to be professors, and this is true across both countries, right, you do have your iconoclast professors that are typically older these days, and it feels like now we kind of have a more conventional profile, people that have been kind of careerists, and I don't mean that in a dismissive way, but people that have gone one rung up the ladder at a time through the PhD

40:17and the postdoc and getting the first professorship, these are fairly institutional people. They're people that do value creativity and research and new ideas, but they do it in a pretty conventional way that doesn't push the boundaries too hard, that tries not to seem too weird, certainly that doesn't borrow the aesthetics of less wrong or talk too much about like super unlikely tail risk scenarios. It's just because it is

40:47academia and because people have kind of followed something more like that career path, and I can't say I know exactly what the ins and outs of the Chinese academic career path are, but you clearly get the vibe that these professor types are more like American professor types than they are like the moral weirdos, if you will, that were like first sounding the alarm about AI safety years ago, but this is where the, as far as I can tell, this is where a lot of the early

41:18and best work has come from in the Chinese context. So if you go over there as a sort of, whatever, I don't want to be too silly about it, but a sort of blue-haired, polycule, AI safety hawk member of the American community and you're looking for your peers, the reality is you just might not find them. You won't necessarily find people that kind of look like you have the same attitudes as you. What you can find, and I think the most strategic and effective of the American AI safetyists when they've gone

41:48to do their bridge building and try to have meetings of the mind with people as much as they can, I think what they've ended up connecting with mostly people in the academy. So that is pretty interesting, and again, I think you do just feel that in a bunch of ways. The AI safety work in China, it's less speculative, it tends to be more focused on reliability, it tends to be a little more focused on protecting minors, all good things.

42:18You don't hear P. Doom talk. Actually, one of the more interesting moments of the entire trip was sitting at a table with a professor who has done a lot of AI safety work, and I asked him, like, what are the lunch conversations like? Do you guys like trade P. Doom numbers back and forth? Do you read AI 2027? What's the kind of vibe? And he said, well, we don't read AI 2027, really. It's too political,

42:49and there's kind of U.S.-China dynamics and whatever in there that may scare people off from wanting to talk about that too much in the Chinese context. He also was like, no, P. Doom? Like, no, we're not, like, trading P. Doom numbers at lunch. He kind of looked at me and was like, do Americans do that? I was like, yeah, oh, yeah, definitely. Like, if you come to the Bay and you go to any number of venues and have a lunch conversation, like, yeah, people will, at this point, maybe it's a little passe, but, like, it's definitely in the water

43:19that people will think about this question in a very live way. He seemed to find that a little bit surprising, actually. Just seems like a little bit, and this is somebody, again, who's done a lot of work on a lot of different aspects of AI safety, put tons of papers, but seemed a little bit kind of surprised that, like, wow, that's, like, that's pretty far out conversation to be just considered normal in the, at least from his perspective, in the Chinese context. So I think those are, those are interesting and kind of important comments, mostly because, you know, in some ways we do

43:51find these mirror image structures in China, like big tech, right? Like, again, I talked last time about how going to have lunch at ByteDance felt almost exactly like going to have lunch at Google. But this is a little bit different. You don't have quite the same type of people leading the effort, and so I do think that creates some risk of miscommunication, and we've obviously seen that, like, the vanguard of the AI safety community in the U.S. has, at times, had trouble communicating with our own government.

44:22It certainly has some inroads into academia, but not as much as probably they would have hoped or expected by this point. Imagine how tricky it might be for them to go engage Chinese academia, let alone the Chinese government. I think it does create some potential for disconnect, some potential for confusion, what have you, but the activity is there. It's just coming from a different institutional context, coming from people with quite a different personality

44:53type, just based on the kinds of career paths that they have chosen. Now, why has the Chinese academy moved faster than the American academy has when it comes to getting serious about AI risk? I don't have an answer for that really, but I think one candidate answer is that it comes from the top. So, I have a few quotes here from Xi's speech at WAIC that I think

45:24would probably surprise many American listeners. Certainly should surprise those that would say, China will never care. Any regulation we do is a gift to Xi. I would think twice about that. Here are some quotes from, and this is the opening keynote, the first 20 minutes of WAIC, this big event. He's there to headline it. I wasn't in the room. There were a few hundred people in the room, and I did talk to a couple of people who were in the room. You know, security was obviously super tight. They were in the

45:55room for like a couple hours before he showed up and took the stage. So, really big deal. And they also, they think really hard about these speeches. He's not somebody who's like out there winging it. They, he chooses his words very carefully. They also choose their words in the translation very carefully. They put an English translation out. If you, a couple of events that I went to, there was live simultaneous translation, including of a couple panel discussions that were happening in Chinese and then being

46:25translated through the earpiece live for me and others in the audience who didn't speak Chinese. Very nice of them to do that, right? I mean, you would not get that sort of translation as kind of a standard expectation if you came to a similar event in the U.S. You'd be on your own. You'd be expected to speak English or figure it out for yourself. There, they obviously don't expect that we're going to speak Chinese. They want to welcome guests. They do go to the trouble of doing this simultaneous translation. Nevertheless, if you're talking a

46:55panel discussion, at least in my experience, the simultaneous translation of panel discussions is rough. Like I, there were, there was one thing that I remember, especially where I was like, I have no idea what they're just talking about. Really? You were getting the kind of big themes that they were talking about, but like, really, what were they saying? I found it very difficult to really get that from the live simultaneous translation. Now they don't do