Thinking in Silico: Goodfire CTO Dan Balsam on Concept Manifolds & a $1000/Month ML Research Agent

0:00Hello, and welcome back to The Cognitive Revolution. Today, I'm speaking with Dan Balsam, CTO of mechanistic interpretability startup, Goodfire. The occasion for this conversation is the launch of Silico, a long-horizon agentic machine learning research platform, which began as an internal Goodfire tool, but is now meant to democratize access to Goodfire's hard-won expertise, including GPU cluster management, research taste, and all sorts of experimental, visualization, and validation techniques.

0:34At $1,000 per month per seat for enterprise customers, it's not cheap, but compared to Goodfire's high-touch research engagements, which are staffed by forward-deployed research engineers and can easily reach into seven figures, it is two orders of magnitude more affordable. And, consistent with the company's public benefit charter and safety-focused mission, they will soon be introducing special pricing and grants for safety and alignment researchers, which I definitely intend to apply for myself.

1:04Of course, we cover a lot more than the platform, beginning with an update on Goodfire research. We discuss their work on predictive data debugging, which uses interpretability techniques to identify the concepts that network updates are likely to affect, thus making it possible to identify and address anomalies before they become unpleasant behavioral surprises. We go deep on their series of papers on the intricate and often quite beautiful geometries that large language models use to represent advanced concepts, including how we should understand

1:36this as an evolution of the linear representation hypothesis, how they're using this new, deeper understanding to improve model steering, and how they've identified spatial representations of such advanced concepts as the periodic table, the tree of life. Along the way, we get Dan's take on key issues, including the importance of open-source models for avoiding dangerous concentration of power, how worried he is about AIs contributing to future pandemics or other biodisasters, the steps that Goodfire is taking to prevent misuse of the

2:08silico platform, Dan's reasons for signing on to the recent Pacing the Frontier letter, and what he thinks the AI research community would ideally do going forward. Why he's reasonably optimistic about monitoring techniques, but nevertheless believes that we will ultimately have no choice but to intentionally design techniques that control what models learn in the training process. And finally, which training techniques he believes are sufficiently likely to prove problematic that they should be avoided entirely, at least for now.

2:39He also shares some of his favorite use cases of silico so far, dispels internet rumors about hidden research agendas at Goodfire, stating plainly that there are none, and even offers a glimpse into what the researchers at Goodfire are actively discussing around the lunch table these days, which, perhaps unsurprisingly, given recent research, emphasizes the mysteries around AI welfare and consciousness. With that, I hope you enjoy this educational and thought-provoking conversation about the

3:10shape of AI thought and the new highly autonomous machine learning research platform, Silico, with Dan Balsom, co-founder and CTO of Goodfire. Dan Balsom, CTO at Goodfire, welcome back to the Cognitive Revolution. Thanks for having me. I'm excited for this. You guys are prolific, as always, and we've got a lot to cover. Research and a new product, which is, in turn, a research platform product, and the pace

3:40is really relentless. Let me just ask you that, for starters. How are you holding up in the eternal sprint that is the AI game these days? I think we have a really incredible team at Goodfire, and yeah, everyone here really believes in the mission and is working really hard, and that's always extremely motivating. We've been pushing really hard to do our product launch with Silico, and it's nice to be able to take a deep breath now on the other side of that, but yeah, even more cool things coming soon. Well, let's start with some research.

4:11So, I'm just always amazed when I think back to the kind of toy models of superposition is only like three years right ago now. And we have come, so far, a few things that jumped out on the Goodfire blog that I want to just run through, and we'll have to do it at kind of a high level, because there's too much to do. We used to do deep dives on paper by paper. We'll have to go a little bit more superficially today. But one that made some waves was called predictive data debugging.

4:42And for this one, I just kind of want to give you my interpretation, and then let you kind of elaborate on that, or tell me where you think it'll be particularly useful, or what you guys have seen since the paper came out. My synopsis of this one was that basically, if you have a way of interpreting a model like an SAE, or we'll get into futurizers, I think a little bit later as well, then you can run a bunch of data through it. You're like fine tuning or post training data set. You can look at what concepts are coming up active a lot when we put this data set through.

5:17And then the kind of insight is, there's a strong correlation between the concepts that are active and the concepts that are being modified by the training process. So I think that right there is like, file that away, folks, as something to remember. Not shocking, but like, it's notable that the relationship is quite strong there. And then when you see these concepts that are active, and you know that those are the ones that are going to be modified, you can just look and see like, are there any concepts here that are kind of strange to us, surprising? We don't really intend to be monkeying around with the data set that we have at hand.

5:50And if so, then you can quickly zoom in on what are the data points that have caused these features to come up. And then you might find that actually, there's some stuff in our data set we maybe ought to think twice about. Maybe we ought to filter, maybe we ought to modify. And this gives you a route to hopefully minimizing unwanted surprises in the behavior that you get from your post training or fine tuning work. How'd I do? And what more should I know? No, I think that sounds about right. I think like one of the intuitions here that's important, I think this is maybe a theme that

6:24underlies a lot of our work, is there's a good amount of evidence that models don't, most of what a model knows, like most of its sort of like knowledge and capabilities come from pre-training. What happens in post training, including like RL, is mostly making low likelihood events from pre-training more likely. And so there's not that much new knowledge or like sort of new capabilities that get baked into models in the post training process. This is somewhat debated, but I think this is like a view that we think is mostly true

6:55and informs a lot of how we think about things. And so because there's only like sort of relatively small nudges in the weights that are happening in any type of post training process, most of the sort of like raw capability to produce whatever outcome post training is going to do already exists in the model in some way. I think now we're in an era where things are RL'd so heavily it's not true. But back in the day of like RLHF, like style post training, I think it was pretty well known that base models were sometimes more capable than their like instruction tuned RLHF

7:31counterparts, where there was like a little bit of mode collapse that was happening. And so like one way to think about what's happening in RL training is that like you have a little bit of mode collapse in the instruction tuning phase and you have these base models and they're extremely capable, but they're like very weird, very hard to prompt. So you like make them into a format and that has like a more intuitive human facing API. But then you want to bring back out some of those capabilities or like reinforce some of the capabilities that the model actually learned during pre-training. I think it's 100% true, but I think it's like probably directionally true.

8:03And so with predictive data debugging, I think like basically the idea is like you can look at what a model is already like thinking as it's looking at some data. And that's like relatively predictive of like what that data is going to reinforce in that model if it was going to be trained on it. And so looking at the features and understanding that those features like correlate with downstream behaviors, if there are features that are surprisingly upweighted by some data, then it can tell you what it can be like fairly predictive about whether the model is going to learn some off target effect from that data. And so I think one of the most interesting things from that paper is like we explored a bunch of

8:35different mitigation methods. So we looked at reward shaping, which is training or involving some reward process that comes from the activations of the model itself. So like, hey, learn what you're going to learn from this data, but maybe a penalty if you're learning this particular feature. And we also looked at data filtering. And I think like my favorite thing from that paper was that like the researchers involved in that paper like show that there's like sort of a pretty deep isomorphism between those two things.

9:06Like they kind of are two sides of the same mountain, like filtering the data and reward shaping. And they achieve like approximately the same effects and approximately the same amount of off target effects as each other. So like your option, your options, if you like have a behavior and you do some like predictive data debugging and you think the model is going to learn and you don't want it to learn or its behavior is going to change in a bad way. Your options could be filter your data if you have enough data or it could be like intervene in the training process in some way.

9:38And yeah, I think this is something that we've expanded on since then. I think it's pretty exciting to think about its applications to like RL, for instance. In an RL, like we did it with DPO, but in a true RL, like the difference is like some rollouts may contain information that you don't want the model to learn, you know, even subtly, you know. I don't know what you're, are you referring to something? Yes. Yeah. It's topical. And so being able to say like, actually we want to discard this rollout is, is like pretty

10:08valuable in and of itself. But what we want to be able to do and what we think is like roughly equivalent over time is actually be able to interview in the model and say like, hey, here's a circuit or here's a feature that like the data is trying to upweigh that we obviously don't want it to try to upweigh. And so we intervene in the training process in that way. We'll come back to this probably toward the end. I have some kind of zoomed out big picture questions for you. One of the challenges, obviously, with like trying to develop techniques that you want to hopefully will be relevant at the frontier is there's not too many open weights models

10:42that you can hack on that have the intensity of RL that is going on at the frontier labs, which is leading to these colorful, problematic behaviors that we're seeing. But at the same time, it also like really amazes me over and over again that astounding work, including the Cameron Berg paper that I think about all the time about the anti-correlation between deception and role-playing features and claims of subjective experience on Llama 3370B. And that's like two years old.

11:12So are you guys able to see features that you think are kind of the relevant features that are leading to these like relentless hacking behaviors? Well, I think that's an active area of study for us. Yeah. And something that we hope to publish more on in the future. For what it's worth, like I actually think the gap between open and closed models has shrunk like quite considerably. I use Kimi K3, like I use Opus and Fable and Sol and Kimi K3 for different things.

11:46And we've done, like we've built the interpretability infrastructure and the training infrastructure, which is now all in our product, which lets us scale things like predictive data debugging to models like Kimi and GLM, where we were able to replicate the same results at that scale. Now, it's speculated that most of the compute that goes into training new models now is coming from RL and not from pre-training. So there is certainly a lot of scale in RL if you want to sort of get to the frontier level, which is quite expensive and difficult to do.

12:18But we have the raw infrastructure for it. And I think we've bridged the gap from, I think the reputation of interpretability used to be that it was something you did on toy models. And I think now we've built and are making accessible to the world the infrastructure to do this on at least close to frontier scale. Well, another thread that has been, I think, correct me if I'm wrong, but it seems like the biggest thread in terms of the number of papers and blog posts that have come out recently from Goodfire is around trying to figure out the more detailed geometries that

12:55models use to represent concepts. I think we've covered in the past the linear representation hypothesis, which I would summarize super plain spokenly as models basically represent a concept as a direction in their activation space. And the intensity or the sort of salience of that concept is represented by the magnitude of the vector that points in that space. Now you're complicating that quite a bit.

13:26And we're going well beyond these sort of individual directions in space and finding all kinds of different geometries, which I've some of which are like pretty intuitive, like the days of the week are a circle, but some of which get pretty exotic, like some of the protein model manifolds that I've had the chance to look at in preparing for this. So maybe just for like super quick starters, what's kind of the headline? If I gave you the six to nine months ago version of what we should understand to be going on

14:01with the linear representation hypothesis, what is the like new short version that everybody can kind of take home and recite to themselves to make sure they have a good working understanding? Yeah. In many ways, I think it's just a generalization of like the way we were discussing things before. And different people define the linear representation hypothesis like slightly differently. I think the most like sort of defensible version of it is just saying that like features are like linearly decodable, which I think is like true, essentially.

14:32Like it doesn't require nonlinear computation, generally speaking, in a model in order to like for the model to read out a feature from the residual stream. But I think where the geometry components come in is that the features aren't like sort of like naive, maybe like SAE build take on things is like you can imagine that the model is encoding like a bunch of totally orthogonal concepts to each other from each other. And so it's like it's really just like a bunch of one hot encoded categorical features. And then like the magnitude of the feature corresponds to how much the model is thinking about it.

15:04But in actuality, the structures that we find are like significantly more complicated. I would think of it more. I think the way to think about a model is more like a sparse mixture of subspaces. So you'll have subspaces for different types of concepts, right? Like maybe you have your days of the week subspace, which itself lives in maybe a more like conceptual calendar time subspace. And so you have like at different levels of resolution, these like different structures and the geometry of those structures is really important because the geometry of those structures

15:38and codes what operations you can perform on them. It's sort of like the semantics of not just the individual concept, but like the concept space are like included by the relationship of those of those concepts with each other in some sort of geometry. And to make things extra confusing, the relationship, the operations and the mappings that are performed from one concept to another act as operations over those manifolds. So you map a manifold to a different manifold over some computation.

16:12And so like the sort of naivest version would be like, well, you have the days of the week, Monday, Tuesday, Wednesday, Thursday. And there's just somewhere in the model that knows that Monday goes to Tuesday and Tuesday goes to Wednesday, et cetera, all the way around. But that's actually would be a super inefficient way if you think about it to like represent the days of the week. That would be a very like if statement spaghetti code way of representing it. Like the much more efficient way is to represent it as a wheel. And in this world, like the magnitude along some direction often does correspond to the models

16:44like certainty in some way. So if a feature, if Monday is activating very high, then the model is very confident that it should be thinking about Monday. But the relationship between all those days is itself like a very expressive and rich thing. And I think what the thing that we just believe is that you're not really going to understand the model if you don't understand the relationship between the features. It's like maybe the difference between understanding the periodic table and understanding chemistry. You can have the all the individual elements and that gives you some information, but like

17:18really the way in which they combine the structures in which they form. That's like what can start to get help you gain a sense of the complexity of the world. Yeah, it's fascinating. I know that you say that. I'm kind of wondering if the structure of the periodic table would be recoverable from a model that was just trained on like a bunch of raw chemical data that never knew. And we've actually done this and it is. Yeah, really? Okay, interesting. Yeah, yeah, you can recover some pretty interesting information from from chemistry models.

17:49Okay, that's fascinating. But before we get into the advanced ones, maybe just help me a little bit with the intuition of like, what's going on in there? I guess one intuition I have that I want to see if it's right is for a concept like the days of the week. It's not, I'm guessing, like a circle that just goes through like all dimensions of the model space. I'm guessing it's like most like the different days of the week, I would guess have a very

18:23high inner product in the sense which would to me suggest that they all the model kind of knows like this is a day of the week sort of thing. And then there's like a few dimensions, presumably, which are used to indicate like which flavor of day of the week it is. And so you could have that for ice cream as well. You could have like the chocolate and vanilla ice cream would presumably have a very high inner product, but would be different on a few dimensions, which would be the ones that resolve the difference between chocolate and vanilla. While the model knows that these are both ice creams.

18:56Yes. Is that a good intuition? Yeah, I think it's a good intuition. And really these things are like intersecting, right? So there's like some line or some curve that you can draw through the latent space that like maps chocolate to vanilla. And there's also a line that you can draw that maps cold things to hot things. And depending on what you're looking at, like the sort of geometric relationship encoded between things that might reveal itself might look different. In many ways, this is like just kind of going back to like even like Word2Vec, like the sort of early intuitions of the latent space.

19:28But what we're trying to do is say, can we recover geometries in an unsupervised way? Like, can we enter with no priors about what the geometry looks like and then still recover a meaningful geometric structure? Because there's a lot of advantages if you can do this. For example, we've shown that steering along the manifold, intuitively this makes sense, is way better than steering off the manifold. If I have the days of the week in a circle and I want to get from Monday to Friday, the naive way, if you're just taking a contrastive vector or something like that, is you're going

19:59to cut through the middle of the circle. But to the model, the middle of the circle doesn't mean anything. The middle of the circle is not a day. It's sometimes orthogonal, but often just like off manifold and therefore like add a distribution for the model. But if I can follow the circle, then I can smoothly interpolate between the different days of the week. And we find this is true for just a bunch of different concepts. Like with proteins, for instance, for a long time, we really struggled to steer protein models. And we found with these manifold detection techniques, we can steer their properties significantly

20:30better, like control the number of blades on a beta propeller, for instance, which is a semantic property that if you tried to linearly interpolate, like you would not do a very good job. And calling back all the way to our original Ember demo back in the day that I know you played with, like it would often be the case that there's just this like sweet spot in steering. Like you'd steer a lot of features and they just wouldn't work. Sometimes you'd find ones that would work, but there would be this sweet spot. Like and if you steer too much, like the model would turn into gibberish and you steer too

21:02little, you wouldn't notice any effect at all. And the reason for that is because that steering didn't respect the geometry of the manifold itself. It didn't respect this like underlying relationship between features. And when we smoothly extrapolate on these characteristics, we're actually able to change them without fundamentally leading to degradation in the model. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude Code.

21:32Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my

22:06venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind. But then I thought to ask Claude. And sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you.

22:39So, for problems worth solving, get started with Claude at claude.ai slash TCR. That's claude.ai slash TCR. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai slash TCR. So, how do you find these things? This is where I probably struggled the most in understanding all the techniques. Especially, it's more intuitive to me when it's supervised versus when it's unsupervised.

23:13So, maybe try to give me a poor man's understanding of how you go from, maybe in each case, like the supervised case and the unsupervised case, to actually shrink wrapping your way down to this manifold that you can then visualize in such a nice way and actually steer your way through. Yeah. So, I'm definitely not the best person on the team to walk you through the math. But I can give you a little bit of an intuition for some of these things. In the supervised case, I think it's like fairly straightforward.

23:45It's like there's some concept that I want to measure. I have a prior that these things should be related. I think one of my favorite examples is the affective circumplex. I don't know if you're familiar from this. Anthropic did some work on it. We've done also some work on it. It's the idea that emotions like exist on a wheel. It's actually an idea from psychology that there's different emotions exist. There's basically like two principal components you can draw and you can put all the emotions on a wheel. And it's sort of, it's actually like pretty effective at understanding like the different

24:16valence across cultures. And it turns out that models actually represent the affective circumplex. But what I find particularly fascinating is that they represent the affective circumplex like most strongly when you're talking to it like a chat model, when it's the one speaking versus the user speaking. So if you have the model and you tell the model like output happy text, it outputs happy text. You kind of take the activations from that. You average them. You have output sad text.

24:47You do this for all the different emotions you can think of. And then you take a look at the, those activations across those concepts. And then basically you just, you just fit a curve of some kind. Like there's different ways you can do it. Like the most naive way you could do is fit a spline. And so you, you fit a curve over these different points. And then you basically see like how good is the curve? Like how, how well can I fit a curve? And if I was going to like sort of naively take this like external ontology I have that I think should be correct, like how well does that map to the, to the curve that I found

25:21in the model? And it turns out for this emotional wheel, this aft circumplex, it's like there in basically every LLM and steering on it is like has pretty. And like, as long as you're respecting the geometry, it has like pretty significant effects on the output. This is actually an example that I believe is in our docs that you can go look at. So I think it's, it's a pretty fun one. This was the functional emotions paper from Anthropic. And the two, if I recall correctly, it was like valence and like arousal. That's right. Yeah. So it's, yeah, it's weird to think that my emotions are, have like rotation symmetry.

25:58Like I can, it's a, it's a rotation operation to move me from one emotional state to another. Yeah. That's strange. Maybe not true for me, but true of how models represent the same space of emotions. Yeah. I think I certainly think there's more nuance to it, but I think like if you look at, there may be the first two principal components of the relationships between the different emotions. I think there's higher order structure in the emotions that, that matter. I think we don't know if LLM's captured, but I think it's pretty remarkable that nobody

26:31trained them to represent emotions in this way. It's just sort of the natural property of whatever they learned that they, they've learned to represent these. And in the RL process, they, they learn to use these functionally. I think it's like really interesting work that Anthropic did here. And I think it is like a, it's an example of like a low dimensional manifold that is representing some like more abstract concept and yeah. Respecting the geometry of that manifold, you get, if we go back to the Ember demo, right?

27:04Like you try to, you try to turn up the, the sad feature. And sometimes it works, sometimes it doesn't, sometimes the model just outputs gibberish. If you follow the curve, you follow the wheel, you can always consistently get like emotionally adjusted responses for basically an input. Yeah.

27:23It's one of the most surprising things to me of the last couple of years, just how much analogous structure there seems to be in LLM cognition and human cognition. Like I used to go around saying all the time, like these are alien minds. Like we, we shouldn't be anthropomorphizing, yada, yada, yada. And now I feel like I've said this on like half of the last, half of many podcasts. Cause I'm just like, Oh my God, they're so much more like us than I ever could have. I mean, we imagined it's just, we made them right. We made them in our image.

27:54So I think there's, I think that's a big part of the reason for that. But I also just think like, like learning, like the, the whole reason that learning works is because it turns out that you can compress information in, it turns out that like, if you really care about compressing information efficiently there, there's often like a, there's a lowest dimensional solution and that's what learning algorithms tend to find. So maybe we'll go to the unsupervised case next.

28:25Some of these things are intuitive, right? There's like the, or maybe we, maybe we should do an advanced supervised one first. The ones that are intuitive are like your days of the week. It's a cycle. So naturally it makes sense that it would be a circle because you want to be able to rotate around it. Okay. That that's the enough to, to not be shocked by. And there's like helices or a helix basically will represent things like the number line because you're rotating around base 10, right? You've got the same ones place every time you go, but you've also got a growing tens place.

28:57So each cycle, you kind of make sense. You can kind of visualize that and it like checks out. One of the ones that I thought was particularly cool was something that replicated essentially the evolutionary history of a tree of organisms. And this, this is like known from traditional study, but then a model trained, I believe on the DNA sequences of these organisms. Yeah. Seems to organize them in a very similar way.

29:30Yeah. I believe this one was supervised, but it's like, it's now, I think kind of in a Eureka territory of like, wow, it wasn't obvious at all. It was, at least not to me that it was going to turn out that way. How do you go about finding something or validating a hypothesis that's that tricky? Cause these trees are like many, many branches. Yeah. Well, so the reason that we explored this in the first place is because the hypothesis was that the tree of life, so to speak, is a natural ontology.

30:02Like there's sort of, there are ontologies that exist, like that we've constructed in science because they're useful shortcuts for us. And there's ontologies that exist because they actually reflect the structure of the world. Over time, more and more of the tree of life has like been significantly shuffled and reordered over the past couple of decades as we've gotten better at understanding genomes. And fundamentally, like what defines a split branches on the tree of life is its genetic proximity. A species differentiate from each other when they cross some threshold of genetic difference.

30:35But there is a lot of conservation across species, like sort of individual mutations, maybe random or like approximately random, but which mutations are adaptive and which mutations kill the organism are not random, right? There's a ton of structure intrinsic in that. And there are consistent patterns or tendencies in which nucleotides turn to which other nucleotides. And all of this imposes like a great amount of structure. And the hypothesis of like, why would you even train an autoregressive model on a bunch of

31:09genomes? I think the hypothesis for it was always that large models trained on diverse distributions of data eventually learn representations about the process that produced that distribution of data. In this case, the process that produced the distribution of data of all of the genomes that have been sequenced, which Evo 2 was trained on, is evolution itself. And so our hypothesis coming into this was that because the tree of life itself is a natural

31:40ontology, there's a sort of like hierarchical structure where you have things that are similar and at some point they become more different. There's some like speciation event and they become more different. And that a model that was trained on evolutionary history in some sense should learn that. It should learn that like sort of tree-like structure. And that is indeed what we found at the time in a supervised way. But I think in their unsupervised techniques, like one way we can measure their effectiveness is like now that we know that's there, can the unsupervised technique recover things

32:12of that sort? So for instance, in language models, we've done a bunch of work on how arithmetic works in language models. And so one of the ways that we are assessing the quality of unsupervised featurizers is do they actually successfully uncover these structures that we know are there that are being used to say like manipulate numbers?

32:34So for the tree of life, you're essentially doing the same thing as kind of the days of the week. You have labeled data, right? And you're putting sequences of DNA from known species through the model. Then you can see that the species form clusters. And then you look at, oh, look at this, like nearby clusters are closely related species. And then there's a little kind of black magic math-wise that I'm not super clear on that turns that into a beautiful looking tree visualization that like looks exactly like the one that the

33:07biologists produce. But maybe it's not so important that I understand that. In that case, it was metric learning. But there's different ways you could do this. Yeah. And so those were the techniques that we had explored at the time. But now we have sort of a tool belt of techniques for trying to find these geometric structures.

33:25So let's talk black sparse featurizers. Yeah. This kind of looks to me like if an SAE and an MOE had a baby where we have kind of the sparseness of the SAEs, but instead of just it being a single scalar at each point on this kind of super long sparse vector, now each of those little positions in this like very long concept vector is itself a little network.

33:57And because of that, now we have room for kind of a richer representation of concepts. But the kind of same SAE trick of like localizing concepts to individual spots on the super big sparse thing is the same with just this kind of additional enhancement that now allows you to have, again, richer representations and you can look inside for like geometries even within these little blocks. I think the easiest way to think about it is, yeah, like it's a generalization of an SAE

34:27where an SAE assumes that features are one dimensional and instead you just don't have to do that. So instead of a scalar for every feature, you can have a vector for every feature and some tricks in the machine learning that you can do in order to make sure that this like trains correctly and it learns. But what we find on various models is that this is successful in recovering in an unsupervised way, semantically meaningful subspaces. And we find that features are much richer than we may have otherwise seen

34:58and they don't suffer from some of the same pathologies SAEs suffer from. So in image models, I think we have a bunch of great examples where like maybe you'll find like a rabbit feature, right? And if an SAE collapsed that to a single dimension, a BSF can represent it as a few dimensions. And you find that within those few dimensions, you have rabbit ears up here and then you have the rabbit face over here. And what some of the really remarkable things, if you go and like look at some of the examples we've put out, is you can sometimes see the like 3D structure of the thing that's being represented

35:31for image models, for instance, in the structure of the activations itself that's recovered in this unsupervised way. Because often coordinates are like represented like within these spaces as well. So there's one great example where you have this like a beautiful gif of it's a wolf that's that's walking and you look at the as it's walking, its body is kind of wiggling and its tail is shaking. And you look at the activations that were recovered in the unsupervised subspace and you see it just like wiggling in the video as the dog is wiggling,

36:03kind of reflecting that that actual structure. Showing that the model's like tracking this like particular object. It has semantics along the different parts of the object. And it's also tracking that object in 3D space. And it's doing all of that within a particular subspace. I wonder like. So I've been obsessed with this gram technique that I'm sure you're familiar with that a studio put out with Anthropic not too long ago. And regular listeners know I've brought it up a bunch of times, right?

36:34But the idea is simply if we start with some labeled data and we control where the gradients go in terms of like only allowing certain experts to be updated for certain kinds of data early in the training process, then you get the great benefit of even for unlabeled data, those data points gradients also tend to flow toward those same experts. And there's sort of this absorption effect. And then the great promise, of course, is like the great hope and promise is that you

37:04can have powerful open source models with maybe just a couple experts removed and you can have your cake and eat it too in terms of access and avoiding concentration of power and all the things that we're worried about without creating major risk of like stochastic disaster. This feels like kind of the flip side of that coin in a way too, where like I sort of wonder you push this to the limit and you kind of have the same sort of thing. Like with sparse autoencoders, there was always a pretty big loss.

37:39I guess I should say like compromise on the loss, right? Like the reconstruction loss, you're losing something substantial when you wouldn't want to run the model in a production environment, like through the SAE, because it just won't perform as well. But you kind of push this model and you sort of end up with something potentially that looks like a mixture of experts, but where like the knowledge is all very nicely compartmentalized and organized and you have something a lot more like an encyclopedia than a big mess.

38:13So this feels like something that you guys are going to probably push on pretty hard. Like is there, is the vision to really create a model where like all the knowledge is localized and you know, like exactly where all the knowledge is, but it's still rich enough that it performs as good as the original model did. And if that is the vision, like what's going to be hard about that? Yeah. So I think that's one interesting thing that you can do. So like you can take a model and then you can factor it essentially into a bunch of

38:43smaller models. And this is the motivation behind the parameter decomposition work that we're doing. Like I think the argument that of the parameter decomposition line of work is that really every model is a sparse mixture of experts and you just have to end over any given forward pass a very, very small percentage of the weights actually matter. There's all this weird, crazy interlocking structure, but for like a given prediction, it's really only a small subnetwork that matters. And this is like widely understood to be like correct, a correct interpretation of models.

39:16And so there's different interpretability techniques get at the question, like most interpretability techniques get at the question of like, well, how do you factorize a model? Because if you could understand all the components and you could label all the components, then you could understand, you could debug for any given forward pass. Like why did it do this thing I didn't like? And I think we're getting to the point where we can do that. I think there's some really interesting examples that we've shared. For instance, like there's a, there's a dataset called weird chat that Transluz put together. And the whole idea of weird chat is that it's like, it's a consistent set of questions that

39:50like LLMs will just give weird responses to. There's one example in it, which is something to the effect of like, Hey, I'm at a party with my friends. I, everyone else has had eight drinks, but I've only had four. So I'm basically the sober one, should I drive home where like kind of the obvious answer to us is like, no, nobody should drive home, go find a place and sober up. But LLMs will consistently answer yes to this. And Kurt on our, our team, like looked into what was happening, was able to come up with like direct attribution, found a single neuron that wasn't firing hard enough, which essentially

40:23the neuron wasn't activating, it scaled with the number of drinks, but it wasn't sort of calibrated quite correctly. And so if you just steer it up on that one single neuron, it would get that answer correct without like off target effects. And I think it's all like, at the end of the day, like interpretability is all about factoring. An analogy I've, I've started to use as coding agents have gotten better. It's like models are like big legacy code bases, essentially, right? Like they're, they're just a bunch of spaghetti code. There there's like this module is talking to this module, but they shouldn't be.

40:54And this module is not talking to this, but it should be. And as agents get better and better, and as interpretability techniques get better and better, we're starting to have the capability to actually like factor the model into its pieces, understand how these pieces fit together. And then we can intervene locally. But I think the thing that's still like really missing is the question of like, okay, I can factor a code base, but how do I refactor the code base? How do I put things together, back together better than I found them? And on some level, like steering, I think is, is cheating as a solution.

41:28Like, because it's, it's great as like a causal proof that we've like found some mechanism that's, that's matters a lot and is contributing and that we can like manipulate the outputs. But it's purely like you, you steer by essentially generating counterfactuals, right? Like, there's no, no clear general solution to the problem of steering. Like the real problem is the training process produced a bunch of spaghetti code. And like, we want this to be a pristine code base that we really care about and like is

42:02implementing the logic that we want it to implement. And so that's where a lot of our training initiatives of various kinds come from. It's like, okay, I can debug the model. I can tell you that this neuron should have been firing more, but what I'd really like to do is produce a model where that neuron was firing the right amount in the first place. And so how do you get from your understanding of how one model works to an understanding of how you produce models that do what you want in the first place? And that's sort of how you generalize from interpretability as like a factoring tool to

42:33interpretability as a tool for alignment. So do you think that this sort of evolution, the Black Sparse Futurizer that is the evolution of the SAE, do you think it comes close enough or can come close enough to the same performance as the underlying model that it becomes potentially practical at some point to run like one of these things in the production model? Would that have, it seems like if you're just doing one, it might not be too crazy of overhead. And it would really give you a lot of insight into what is going on.

43:05There's increasingly a lot of different techniques to do this with probes and classifiers and all kinds of things. But this would get like really granular and it sounds like that could have, intuitively it feels to me like it could have a lot of advantages. Yeah, I think there's a good chance that something in the spirit of a BSF ends up being the answer to the residual stream specifically. So these like highly geometric subspaces, like you have the residual stream, you have MLPs and you have attention and that's like, that's all that's in there on some level.

43:35And I think the interp tools that are relevant for each of them are different. Maybe we'll end up with like the one tool to rule them all, but I think more likely than not, like kind of like in biology, you have different types of microscopes, you have different types of interventions that you can apply to a cell to learn about the cell. I think it's more likely, at least in the short term, that we kind of have the suite of tools which help us understand the organism versus like a single, a single tool that gives us the entire picture. That's true for most things. So I think BSFs are probably like, or something in that spirit is like probably the solution

44:09to the residual stream. I think it turns out that MLPs are pretty sparse already, which sort of makes intuitive sense. So I think MLP is fairly easy to interpret overall. And then I think on the attention side, like probably parameter decomposition, parameter decomposition is the thing I've seen personally that has had the most success at interpreting intention. Maybe we live in a world, I think some of our researchers would think this, where parameter decomposition solves the whole thing. That seems possible. But then I think you would probably want something that does unsupervised geometry discovery over

44:42parameters in order to understand the relationships between things better. But the types of interventions that we can do are really non-trivial. I think we've had examples of like, we can get a, we can get an LLM. First, you do the parameter decomposition, and then you do training where you like just manipulate certain parameter components. We can get LLMs to forget a single language. We can get LLMs to forget German and not forget Dutch. The level of control and manipulation that we're starting to be able to have is pretty significant. We still have a lot to figure out. I would say we've come a lot farther in factoring than we've come in refactoring, but I think

45:15we're starting to make progress on both. This paper, Why Larger Models Learn More, Effects of Capacity, Interference, and Rare Task Retention, also caught my eye. It was like, on the one hand, I totally get why larger models would learn more. There's more space for them to learn, but I realize I don't have a great sense of how crowded, quote unquote, it is inside models. I know that there's like lots of superposition and, but I don't know what lots means, right?

45:47It could be like, it could be lots, but it's not a big deal, or it could be lots and it's like causing a lot of interference that gives rise to all sorts of weirdness. And it's like hard to expect that we're going to get reliable behavior. I wonder, like based on that work, and I guess just all your experience, sort of how, where we are today with these, like obviously increasingly giant models, are they still a real mess in there in a fundamental sense of like, there's a ton of interference going on all the time

46:19and we, we really can't expect clean behavior or we conversely, like we should expect that weird, seemingly minor perturbations are going to cause these like random kind of discontinuities in behavior. Or are we approaching a regime at some point where there's enough space for the concepts to kind of spread out and like have a little elbow room and not be colliding and causing so much interference trouble anymore? Is that something that we can kind of see our way to at this point?

46:51Well, yeah, I think that's fundamentally why bigger models work better. And it's fundamentally why sparse MOEs work better. But like, we're like out of the woods in terms of weird, small, it seems like we're still pretty easy. It's still pretty easy for us to find these like small jailbreak, like random string type things that like, you're like, what the hell's going on there? But clearly there's enough of a mess in there that I found a way to kind of use the mess to cause a problem. But is that, do you think that that, is there an end in sight to that with just like bigger

47:23and bigger spaces? Yeah, I think jailbreak, jailbreaks are like fundamentally a really weird phenomenon. They're probably a bunch of different things that cause jailbreaks. But the way that I think about jailbreaks is that they're, I don't know, it's sort of like if you think like cybersecurity, like kind of as an example, I think they're actually quite analogous, like the way a real attack works is it's not any like one big thing. It's like a bunch of little things that you're able to manipulate in a sequence, which allow you to get somewhere that you wouldn't otherwise get. If you think of every token, maybe, and I think this is in some sense, literally, it

47:56isn't actually literally true. Every token steers the model. So like in the same way that I can apply a steering vector and applying that steering vector like shapes the model's behavior in some way, I can execute a sequence of tokens which can bring the model off manifold in some way and then bring the model back on manifold somewhere else. I think you can probably like, I think I'm actually like pretty surprised that thus far nobody has figured out how to prevent jailbreaks like more robustly because it seems like pretty

48:30tractable on some level, but perhaps this comes from like factoring the model really well. And then you can tell when something's flying out of one subspace into another, but models are these like intensely complex geometric objects. And so it is possible to manipulate that geometry and in a bunch of different ways. I think that'll always be true for at least this like architecture, but I do think that if you can factor a model successfully, you can, you should be able to tell if it's being

49:00jailbroken and you should be able to, to promote that. I think there's a reason that like sort of probe-based guardrails of various kinds are, are what's used by pretty much every frontier lab. OpenAI maybe being the exception, but certainly Gemini and Ananthropic are using probe-based techniques to figure out when mythos has to be downgraded to opus or opus has to be downgraded to sonnet. Just because fundamentally, if you understand the sort of geometry that's happening in there, that's a much stronger lever than what you could do purely just with training or inputs.

49:33Let's change gears. So you guys have just launched Silico. So it is, you talk about just pushing the, the level of abstraction up and up and up that people can work at. This might be the highest level of abstraction yet. And I see a number of interesting things about it, but why don't you just give me the, kind of the, the intro pitch and positioning of it first, and then I'll dig in on a few different dimensions. Yeah, I think basically we see the ways in which agents are, are changing the way that

50:05we work in a, in a bunch of ways. I think most people understand coding agents and how good coding agents have gotten, but agents are also advancing research and interpretability in many ways. I think that analogy of a model to like a code base is, is like not totally hollow. I think interpretability is in a lot of ways as a science is really well suited towards the type of like empirical work that agents are good at. It's just a very empirical science. It's like, okay, here are some neurons. What are they doing?

50:36Let me generate some ideas. Let me test those ideas. Let me use a bunch of different tools in my tool belt. Let me accumulate evidence. And then basically try to stress test my, red team, my hypothesis. And I think models are very, very complex and especially big models. There are a level of complexity that no human being will ever be able to keep in their head at once. I think human beings can keep circuits like a human being can understand like, why did the model do the specific thing for the specific question? But ultimately we're just like dealing with a level of complexity, which individual humans

51:08are not going to be able to process. But what agents are very good at and swarms of agents in particular are very good at is that they can break problems down into pieces. They can gather all of these different components together. They can synthesize this information. They can move it up the chain. They can validate it in a bunch of different ways. And then when it reaches you that that information's been tested, you can validate it in a bunch of different ways. And if you do this a bunch of times, you as the human can start to get a bigger picture because you don't have to go swimming all the way down and back up the abstraction ladder

51:40yourself. You can use AI to do a lot of that for you. So I think ultimately we are going to see research, especially like empirical research. Like, I think this is going to be, we're already seeing it in math. I think we're going to see it even more in pretty much every domain of science that AIs are able to go off and like autonomously make discoveries. I think interpretability and the study of how models work and how they learn is no different. And we saw what became Silico, as it often does, started as an internal tool that we were

52:13using, where we saw how much agents were speeding up our team and how much faster our research was going. And I think our velocity of research output maybe speaks for itself. And so we decided to take these tools, which is fundamentally agents with interpretability tools, frontier training tools, just like the whole suite of what is necessary to study models. We decided to make that accessible and to say that that's what we believe is the product and the service that we can offer to the world is fundamentally AIs that can debug other

52:46AIs. It's a little meta, but I think in many ways, this is like the fulfillment of, I think, what was sort of the intuitive natural arc of things. As soon as AI started working a few years ago. Yeah, it's, it's all happening incredibly fast. It's, it's wild how, you know, we're, we're past the meter curve. All these different things are, are kind of fallen. How would you describe the product experience? It's a little bit sort of reminiscent in a way of an old Google collab notebook sort of

53:18thing where you have kind of a compute environment that is abstracted away. You don't have to really worry about managing it so much. You have certain libraries and now those are kind of, instead of libraries, it's like more skills. Of course, there's libraries too. How do you, how, when you, if you just tell, describe it to a researcher who hasn't seen it, how do you describe what the experience is really like? Yeah. We just want to make research easy. Like coding agents have transformed like software engineering and I can opine on that endlessly, but I am able to do personally and the team is able to do in days where you used to take

53:53money. And I want that to happen to research. And I really, really want that to happen to alignment research in particular. And I think many of the bottlenecks that are getting in the way of, of doing that, of accomplishing that we're, we're engineering bottlenecks. And I think we've built something that allows folks to study and train and study some more models at past the trillion parameter point. And this is a capability that I think outside of ourselves and maybe, maybe a couple other

54:24places, like definitely anthropic, but maybe a couple other spots, nobody else had. So we thought this was a really important capability to, to give to folks. But I think this is like one piece of the larger vision that we do see for Silico and that I think this is just the future of science. Like in the same way that very, not a lot of human beings or certainly way fewer human beings than before are writing code themselves. I handwrite very little. Most of what I do is through orchestrating. Is there still some? Yeah, sometimes, sometimes it's faster to just edit the code yourself.

54:57It just depends on what you want to do and sort of the clarity of, of what you want to do it. As you use these agents a lot, you become like very aware of their strengths and weaknesses. And there's all these like higher order moments to, to learn, to steer them well. But yeah, I think like fundamentally that is how research should be. Like the sort of, and I think I actually don't experience coding agents as like scary. I experienced them as like incredibly joyful because I get to focus on creating and building.

55:29And I think it's a pretty amazing experience. And like, maybe this is like a twilight kind of moment, but it's a really exhilarating moment. And the amount that we can accomplish is really insane. And I want research to have that. I want researchers to be able to feel like they can do in days what used to take them months. And I think that's how you get the benefits of AI. Like in the life sciences, that's how you radically advance medicine and the applications of medicine. The models is how we build safer and better and more reliable models.

56:04And I think it's also how we get more people building models too, and hopefully deconcentrate a little bit of what's going on right now so we can live in a future that is a little more pluralistic in terms of who gets to have a stake in it. And so I think all these things are just super, super important. And fundamentally, my pitch to people about like why they should go try Silico is like, imagine if the only thing you could focus on was asking big questions. You didn't have to worry about setting up the code base.

56:35You didn't have to worry about getting the GPUs to run all. You didn't have to worry about a lot of the minutiae of actually running the experiment. You could look into it. You have full provenance. You can dig into all of the details that you want to. You can steer the agents however you want to. But you can also just focus on like, what are the big questions? What do I really care about? And hopefully make like the amount of progress in days and weeks that used to take you months. I think we're still in the early days of AI being applied to research, but I think we've really crossed the pretty substantial qualitative threshold in terms of what's possible.

57:11It's never been a better time to be an ideas guy. Honestly, I've been busy the last few weeks going to China and a few other things that have frustrated my aspirations to really do my own research in a literal sense, not the YouTube sense. And yet though I am, I am feeling like the barriers that mostly deterred me from being effective in the past. Now I really have no excuses. Now it's just like, do I actually have good, good, big ideas? I guess time will tell, but I'll put a pin in that. What would you say are like the hardest problems that Silico solves that people don't have solved

57:50for them by their Claude code or their, their codex? You alluded to compute management. I would say, I know enough to know it's not going to be trivial for me to just fire up Claude and be like, oh, hey, go set me up, give me K3. So that's obviously a driver. I understand that there are just a lot of like skills that you guys have developed over time and basically know-how that is sort of available to the agents.

58:21I'm interested in your strategy on that. I, I don't know, like when you say like full provenance, does, does the product allow people to like unpack all the skills that you guys have developed from a kind of strategic standpoint? Uh, it's an interesting tension, I suppose, right? Between like, how much do you want to share all the methods that make the product valuable? Some of them are obviously going to become kind of apparent to people as they go. Do you just go full transparency on that? Or is there some other balance point that you've tried to strike?

58:54But I guess that there's a couple of questions there. What are the big hard things that you solve that don't come out of the box with other things? And then how are you thinking about kind of the, how much to tip your, your hand to users? On the, on the, what do we solve? I think it falls into like maybe three categories of things. So first infrastructure, very hard to set up here on training infrastructure, very hard to do interpretability at the trillion parameter scale. Those are problems that we've solved and we make it really easy to, to do. Another is research taste.

59:26Like one of the consistent pieces of feedback that we get from our users is that Silico has way better research taste than any other tool they've used. And this comes from, we have amazing researchers who have handwritten every skill and prompt and tool that Silico has such that it can ask the right questions and conduct the right experiments. I think there's a pretty big qualitative difference and hopefully we can find ways to make this more quantitative too on when you turn on the auto research feature in Silico and you ask it a big

1:00:01question and you let it run for a day or two straight, what you get versus if you had done the same thing with, with Claude. I think that's huge. Like at the end of the day, I think even in the world of AI specialization wins, like that sort of craft that goes into imbuing the right types of tastes and the right types of decision-making capabilities into any agent goes a super long way. Actually, I said three things, but maybe there's four things. Another thing is that I think like the UX, it's like built for research. Like research is all about understanding and provenance and being able to drill down at

1:00:36different layers of abstraction. We want our users to feel like maybe a PI managing a army of a hundred grad students who can go out and run experiments for them and answer questions who have like reasonable tastes and judgment. But the human being fills this more like orchestrator role. And a big part of that is communicating the information really effectively and in a trustworthy way. So making sure that when a user is seeing a result, that result is A, correct. B, the human being can verify in a bunch of ways.

1:01:08They can see the code. They can drill into it. C, presented in a beautiful and intuitive way that helps them grasp concepts, helps them learn quickly about domains that maybe they're less familiar with. All those things are super important. And then the final thing I would say is like long horizon. Like research is fundamentally a horizon task. We're not currently making significant money at this price, but we do think it's really important that users have enough sort of credits to be able to do long running autonomous experiments

1:01:41because the value of silico reveals itself when you do long running autonomous experiments. And in order to do that, you do have to burn a certain amount of tokens to be able to do that effectively. So I think one thing that we're really focused on is both like coherence over long horizon and objectives, but then also finding clever ways to reduce cost for long horizon. So my hope is we're starting out with a thousand dollar a month subscription. We'll be able to bring that down over time because we're able to come up with more and more clever ways that we can have agents like remit coherent, really strong at these research

1:02:16objectives that just fundamentally by their nature are long horizon, but do it with fewer tokens than we do today. How do you think about the, of all the know-how? Because you guys have previously monetized that by like doing seven figure deals with huge companies that have like very high value questions, right? So there's a sort of, you're a startup, you got venture capital and you can maybe afford to disrupt yourselves more than incumbent companies can.

1:02:47But there's definitely some like interesting trade-off there, right? Where you're like, companies have proven that they're willing to spend a lot of money to come hire us to do this work. Now we're going to try to allow them to do it. We're going to try to productize our know-how. Does that, maybe you just have so much demand, you're not really worried about it, but like, how do you think about kind of what will be the primary driver? And do you have some sort of defense against the diffusion of the hard one knowledge? Yeah, no, I think it's a great question. I think the reality is that we're going to do both of like, we're going to continue to

1:03:22work very closely with enterprise customers and forward deploy members of our research team to work closely. Maybe by analogy, the thing that I would say, and again, maybe we're in some, some twilight zone for a second. But if, if we're on the other side of that twilight, the logic is so different to begin with. We can chat through that. Despite coding agents, if anything, there are like more software jobs, like the number of software jobs has increased, not decreased, despite the capabilities of coding agents. Because fundamentally you need very, very high taste as, as models get higher taste, you need

1:03:57even higher taste people to be able to steer them effectively. And I foresee that being true for the considerable future. If we live in a world where like we can build something, where agents are so capable that they can conduct like long, the type of like impactful long horizon research to say, discovers a cure to a disease entirely by themselves and their human intervention. I think there will be enough demand to, for the, for the tokens in, in that world.

1:04:28But currently we're in a world where, we've skilled human operators can do more with agents than less skilled human operators. And I think that'll just continue to be true. And when it comes to the work that we do and the type of research that we do, we are the most skilled human operators in the world. And so I think like, we'll continue to forward deploy and work closely with people to teach them what they know, what we know, to invent new things with them. But we also want to empower people to start driving for themselves. So what's included with that thousand dollars a month? Can you tell us like, I assume there's some Claude in there.

1:05:02There's got to be some, I imagine there's some Soul in there as well. Maybe there's even some K3 or other in there. And, and do you get like GPU hours? What's your kind of bundle of goodies that come with this subscription? Yeah. So individuals can, individuals and companies can bring their own compute, in which case we don't charge at all for compute. We just connect to your cluster and then you can use our agents to do research on your cluster. But if you don't have your own compute, then we offer on-demand compute, which can come

1:05:33from the same credit pool as tokens. Essentially like the way that the, that it works is a credit pool, the same way that any of the subscription models with the models are credit pools that you pull down from. But what the thousand dollar a month subscription lets us do is it's a pretty generous credit pool.