back to index

State of the Art: Training 70B LLMs on 10,000 H100 clusters


Whisper Transcript | Transcript Only Page

00:00:00.000 | Welcome to the Latent Space Podcast, another super special edition.
00:00:04.000 | Today, we have a sort of like a two-header.
00:00:07.000 | John Franco from Mosaic Databricks or Databricks Mosaic and Josh Albrecht from Imbue.
00:00:12.000 | Welcome.
00:00:13.000 | Hey, glad to be here.
00:00:14.000 | Thank you for having us.
00:00:16.000 | Hey, so both of you are kind of past guests.
00:00:19.000 | Jonathan, you were actually one of the most popular episodes from last year.
00:00:24.000 | Talking about MPT 7B. Remember the days when we trained large models under a 7B?
00:00:32.000 | Yeah, back when reproducing LLAMA 170 was considered a huge accomplishment for the field.
00:00:37.000 | Those were the good old days. I miss that.
00:00:41.000 | So things have accelerated a lot.
00:00:43.000 | Actually, let's do a quick catch up and Josh, you can chime on in as well.
00:00:48.000 | So Databricks got acquired. I talked to you at Imbue.
00:00:51.000 | Mosaic got acquired, although sometimes it feels like Mosaic acquired Databricks because we're having a lot of fun being here.
00:00:58.000 | Yeah, I mean, you're a chief scientist now of Databricks.
00:01:01.000 | Chief AI scientist. Careful with the title.
00:01:04.000 | As much as I would love to understand how Spark works, I'm going to have to defer that to much smarter people than me.
00:01:11.000 | Got it. And I don't know about what you would highlight so far as post-acquisition, but the most recent news is that you guys released DBRX.
00:01:22.000 | Is that the thing that most people should be aware of?
00:01:25.000 | Actually, that's no longer the most recent news.
00:01:28.000 | Honestly, the most recent news, we announced this, but it was at our Data and AI Summit last week.
00:01:32.000 | So it was announced among like 100,000 other things.
00:01:35.000 | Is that we finally released our text-to-image model, which has been a year in the making through a collaboration directly with Shutterstock.
00:01:42.000 | There was a lot of work put into finding a data set that we were comfortable with working on
00:01:47.000 | and trying to build a model that, honestly, I felt like I could trust and that others might be able to trust to put out in the world.
00:01:54.000 | So that model was released last week.
00:01:56.000 | It's unfortunately just available via API due to the fact that the data is quite sensitive and quite valuable.
00:02:03.000 | It's Shutterstock's entire business in a lot of ways.
00:02:05.000 | But I'm still really excited that there's now a model that is trained on a data set where the provenance of every single image is known.
00:02:12.000 | And it's a damn good model, so I'm really proud of the team on that.
00:02:16.000 | Yeah, amazing.
00:02:17.000 | Josh, do you have any thoughts on image model questions?
00:02:21.000 | That is not my area of expertise, but I was excited to see the release of it last week as well
00:02:26.000 | and very happy that you guys did a nice job on the data side of everything there.
00:02:30.000 | So that was cool to see.
00:02:32.000 | I think what's unusual is, I think Shutterstock's doing multiple deals in multiple labs.
00:02:37.000 | So what is the Shutterstock model?
00:02:39.000 | I guess, is this the house model for Shutterstock?
00:02:41.000 | Is this Databricks' version of the Shutterstock model?
00:02:44.000 | What is this?
00:02:46.000 | The way that I would think about it is that Shutterstock is doing an amazing business in AI across the board.
00:02:52.000 | Their data set is widely known to be the best stock photos data set in the world, the most comprehensive, the biggest.
00:02:58.000 | When you think about what data set am I going to train a multimodal model on, you call Shutterstock.
00:03:05.000 | And at least I've heard in the news, like OpenAI, Google, Meta, Apple have all called Shutterstock and made those deals.
00:03:13.000 | So a lot of models have had Shutterstock data incorporated into them.
00:03:17.000 | But this is the only model I know of so far where it was exclusively and specifically trained just on the vanilla Shutterstock data.
00:03:26.000 | There was nothing else mixed in.
00:03:28.000 | We didn't go and scrape the web and find other data or combined data sets or anything like that.
00:03:33.000 | And so this is, in some sense, the house blend.
00:03:36.000 | But the other piece is that it's just a data set where the provenance of every image is known in public.
00:03:41.000 | Where did the data come from?
00:03:42.000 | It is the Shutterstock collection.
00:03:43.000 | That's it.
00:03:45.000 | Nothing less, nothing more.
00:03:47.000 | And certainly being at Databricks, if I've learned one thing, it's I've learned about enterprise customers and what they want out of AI.
00:03:53.000 | And one of the things they ask for most is just what can you tell me about the data the model was trained on?
00:03:58.000 | And here, especially for text to image models where images are just tricky subject matter, there's been a lot of kind of legal conversation about images, especially.
00:04:08.000 | It's nice to just have something where I can point to it and say, you know, you want to know where the images came from?
00:04:13.000 | These are what they are and this is how they got there.
00:04:15.000 | I will talk a little bit about Databricks because it's relevant to the rest of today's episode.
00:04:20.000 | So Databricks, sorry, I keep misspeaking.
00:04:25.000 | It's DBRX.
00:04:26.000 | DBRX, actually, there's been a pronunciation update.
00:04:30.000 | It is now DBRX.
00:04:32.000 | So we have decided to add a dinosaur mascot because what model doesn't like a mascot?
00:04:39.000 | So literally, I wish I could pull it up.
00:04:41.000 | There is a little plush dinosaur that we had made.
00:04:44.000 | It's like the world's cutest dinosaur, but it is the official mascot of DBRX.
00:04:48.000 | And there's a little dinosaur logo that, you know, you'll probably see around a little bit more because, I mean, DBRX is a mouthful, but DBRX, like, you know, it's just kind of.
00:04:59.000 | Rolls off the tongue.
00:05:01.000 | I love mascots.
00:05:02.000 | I think every company should have a mascot.
00:05:04.000 | And I think Hugging Face got it right.
00:05:05.000 | You need an emoji mascot because that's the minimal viable image.
00:05:09.000 | I probably shouldn't talk at all about, you know, Velociraptor, but, you know, maybe that's something we can talk about later in the summer.
00:05:15.000 | I'll just leave it at that.
00:05:17.000 | Okay, that's a hint to names.
00:05:19.000 | I feel like your names leak a lot of alpha.
00:05:22.000 | So just to quickly cover the headline details.
00:05:26.000 | DBRX is a Make Sure Experts model that's fairly big.
00:05:31.000 | 132 billion total parameters, so 36 billion active on any input.
00:05:35.000 | Pre-trained on 12 trillion tokens of text and code.
00:05:38.000 | And did really well on evals to the point where you had to dye your hair blue.
00:05:43.000 | That's my high level conclusion.
00:05:47.000 | Never make a bet with your team two weeks out from model launch.
00:05:50.000 | Even when, you know, human eval is looking quite bad.
00:05:54.000 | Because if you set some bar, even if it's arbitrary and you think there's no way in hell they're going to hit it.
00:06:00.000 | Apparently money doesn't motivate people anymore.
00:06:03.000 | Humiliating their boss motivates people.
00:06:05.000 | So Josh, you should really take a hint from this.
00:06:08.000 | You know, you cannot pay someone enough money to make up for you dying your hair blue.
00:06:12.000 | Keep that in mind for our next model.
00:06:14.000 | It works.
00:06:16.000 | So speaking of Imbue's next model.
00:06:19.000 | Perhaps Josh, you want to actually just say hi to the general sort of Latentspace audience.
00:06:25.000 | And talk about what we're releasing today.
00:06:27.000 | Yeah.
00:06:28.000 | I'm Josh, CTO of Imbue.
00:06:30.000 | And we're not releasing the model.
00:06:32.000 | We're not releasing the weights.
00:06:33.000 | But we are releasing a bunch of different things that should make it easier for other people to make their own models.
00:06:38.000 | So I think right now training foundation models from scratch is like a very difficult, time-consuming, expensive, kind of risky endeavor.
00:06:46.000 | Especially for smaller companies.
00:06:48.000 | And the things that we're releasing hopefully make that at least a little bit easier.
00:06:51.000 | So the things that we're releasing fall into kind of three different buckets.
00:06:55.000 | One is infrastructure and scripts for dealing with the kind of hardware and hardware failures.
00:07:04.000 | And understanding how well is the actually lowest level of things actually working.
00:07:09.000 | So you can actually do your training at all and at a reasonable speed without having to constantly restart, etc.
00:07:13.000 | So infrastructure and training scripts.
00:07:15.000 | A second set of things is around the evaluation.
00:07:19.000 | So after you've trained it, how well is this actually working?
00:07:22.000 | And how do you know how well it's working?
00:07:23.000 | We're releasing a whole bunch of different data there.
00:07:26.000 | A new benchmark about code, reasoning, understanding.
00:07:29.000 | As well as our own private versions of 11 different open source benchmarks.
00:07:36.000 | So things like poolq or ANLI.
00:07:38.000 | Where we've gone through and kind of cleaned up the data as much as possible.
00:07:41.000 | By looking at all the ones that models get wrong or that are flagged for ambiguity.
00:07:45.000 | And also our own kind of private reproductions of those.
00:07:49.000 | Where we've done like a kind of clean room black box.
00:07:51.000 | Like, okay, this is what the data set is supposed to be.
00:07:53.000 | Here are some examples.
00:07:54.000 | Let's make our own version of this to make sure that there is no data contamination, etc.
00:07:58.000 | To make sure that we're actually, you know, not testing on train.
00:08:02.000 | And then I think a final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality.
00:08:09.000 | Which we used in the process of cleaning these evaluations.
00:08:13.000 | And we also hope will be helpful for other people training kind of similar models.
00:08:16.000 | And then the third thing is CARBS, our hyperparameter, our cost-aware hyperparameter optimizer.
00:08:23.000 | Which was especially helpful for being able to experiment at much smaller scales.
00:08:28.000 | And then scale those experiments up to the much larger scale.
00:08:31.000 | Kind of on the first try without having to retry.
00:08:34.000 | You don't want to be training, you know, 10, 20 different 70B models.
00:08:37.000 | You really want to get these larger models right on the first try.
00:08:40.000 | So the ability to kind of tune things very precisely and learn scaling laws not just for, you know, the like data and flops.
00:08:51.000 | But also for learning rate and all the other hyperparameters.
00:08:53.000 | And see like how should you scale these things up was extremely valuable to us as we were training the larger models.
00:08:59.000 | Yeah, that's a lot of stuff.
00:09:00.000 | Yeah, exactly.
00:09:01.000 | So there's a bunch of stuff we'll have to go through all of it.
00:09:04.000 | Yeah, I just want to throw in how excited I am about this.
00:09:07.000 | This is the stuff that nobody ever talks about that is the difference between success and failure in this stuff.
00:09:13.000 | Like, can you get your cluster to run?
00:09:16.000 | Can you get software on your cluster?
00:09:18.000 | Can you figure out what broke?
00:09:20.000 | Because fault tolerance is still not really built into any of the fundamental primitives of training models.
00:09:26.000 | And so if something breaks, you have to go figure out what broke.
00:09:29.000 | Your job stops, you have to restart your job.
00:09:31.000 | It is a nightmare just to get to the point where anything can train on the cluster.
00:09:35.000 | A basic MPI hello world that has the GPUs talk to each other is hard enough, let alone actually training a model, let alone getting good performance out of the GPUs, let alone actually getting a model that converges to anything interesting.
00:09:47.000 | Like, there's so many levels of things you have to accomplish.
00:09:51.000 | Like, this is the kind of stuff that matters.
00:09:55.000 | You know, I think to a point that Josh made earlier, you know, before we got on here, there are plenty of weights out there.
00:10:02.000 | Nobody's released this.
00:10:04.000 | Yeah, that was part of the motivation, actually, is that there are lots of other things that are complementary, but I have not seen nearly as much discussion about some of these other things that we think are pretty important.
00:10:14.000 | I mean, in some sense, I'm very excited to have Jonathan on because this is a little bit your bread and butter with Mosaic.
00:10:23.000 | You know, I think you've released some part of it with Composer, and I think it's just, you know, really, really interesting to see like a different take, like basically a full stack take that's kind of open source today.
00:10:37.000 | Yeah, it's really kind of, it's been an ordeal to figure this out.
00:10:44.000 | And every time something changes, whether it's a new GPU or even a new driver update, you get new creative errors and new things go wrong.
00:10:53.000 | And, you know, we've dealt with the weirdest things from, you know, our InfiniBand cables getting stolen from the data center twice, like in boxes before they arrived at the data center.
00:11:03.000 | Like, you know, Porch Pirate basically had stolen our InfiniBand cables back when those were hard to come by, to like, you know, weird recalls of switches, to like, the strangest stuff has happened.
00:11:16.000 | I have my favorite GPU failures I've seen, like ones where the GPU doesn't fail, it has a correctable memory issue, and the memory correction causes the GPU to become a straggler and hold up the whole job.
00:11:26.000 | Like weird stuff happens and figuring out how to not just identify all of that, but then eventually productize it is in some sense, the entire story of Mosaic and now Databricks in terms of our ML offering.
00:11:39.000 | Really, the thing, you know, the thing we offer is we have gone through this suffering and figured out how to even productize that.
00:11:46.000 | It has been a pain in the butt.
00:11:48.000 | Yeah, it's a lot of work.
00:11:49.000 | I think my favorite failure was a GPU is just giving wrong math.
00:11:54.000 | Like if they give errors, great, cause you can see the errors, but if they just give you the wrong math back, not so fun.
00:12:00.000 | When did they give you wrong math?
00:12:01.000 | Like literally you could just, you know, add two things.
00:12:04.000 | For example, the numbers come back.
00:12:06.000 | They're not the numbers that they're supposed to be.
00:12:10.000 | I think it's important to say at this stage, just cause like it, I think it goes without saying for Josh and I, but it's worth saying here.
00:12:17.000 | This isn't to say that like anything is wrong with us.
00:12:20.000 | It's not like NVIDIA did a bad job or, you know, Mellanox did a bad job or the, like the server builder, the data center operator, the cloud provider, like the million other parties that are involved in building this.
00:12:31.000 | We are running these insane chips that are huge and complicated and built on tiny transistors at insane frequencies with insane heat in data centers that for the most part were not built remotely for this kind of power or heat and have been retrofitted for this.
00:12:47.000 | Like failures happen on a good day with normal CPUs and this is not a good day and not a normal CPU for the most part.
00:12:56.000 | So it's, you know, it's fun to joke about all the weird things we see.
00:12:59.000 | This is not to say anybody's done anything wrong.
00:13:01.000 | This is just kind of part and parcel of working on a massive cluster running at multiple megawatts of power at a time.
00:13:07.000 | It's crazy.
00:13:08.000 | Yeah.
00:13:09.000 | So optical cables, like all sorts, like everything.
00:13:14.000 | I'll take the opportunity to start in going to the sort of infra piece.
00:13:18.000 | So there's, there's just like a distribution of the infra just to give people a sense of what we talk about when we talk about massive clusters.
00:13:24.000 | So I'm just going to read off the blog post here.
00:13:26.000 | This post is about one cluster that has 4,092 H100 GPUs spread across 511 computers.
00:13:33.000 | They use unified fabric manager nodes, which manage the infinite band network.
00:13:40.000 | And you talk a little bit about your networking.
00:13:42.000 | Is there anything unusual about this setup that you'll call out to people?
00:13:45.000 | Yeah, actually, this particular cluster is a little bit non-standard.
00:13:49.000 | The normal vanilla setup for these large clusters, as vanilla as it can be, is what's normally like a 127 node cluster.
00:14:01.000 | So closer to like 1024 GPUs instead of 4,000.
00:14:05.000 | Here we have a larger cluster.
00:14:07.000 | As you start to get into the larger clusters, the networking becomes a little bit more custom.
00:14:10.000 | It's a little bit more, it's a little bit trickier.
00:14:12.000 | It's a little bit more difficult to get these things to all be able to talk to each other at the same speed.
00:14:17.000 | And so this has, in this particular case, this is a three tier network architecture instead of two tiers, kind of the normal one.
00:14:23.000 | So most of the clusters are a little bit smaller.
00:14:25.000 | As you get to even larger scales, then it becomes, this becomes even much more complicated, much more expensive.
00:14:30.000 | So we chose this particular scale, kind of knowing our own workloads and kind of what we wanted to do.
00:14:34.000 | This was kind of the right size for us.
00:14:36.000 | But yeah, I think it's, it's, you know, it's not exactly vanilla already.
00:14:41.000 | It's already getting into kind of the custom territory.
00:14:43.000 | Yeah. Is this, is there any, so my understanding is that there, and for the, for what it's worth, I don't know if this is on the record or whatever, but you can just tell me to strike it.
00:14:54.000 | Is there any, is there any part of this that comes with the Voltage Park deal that you guys had?
00:14:59.000 | Is like, is that, is that part of the hardware that you got from the deal with them?
00:15:04.000 | Yeah, so we worked really closely with Voltage Park to set up their, all their clusters and infrastructure and everything and kind of decide even like what to order, how should like, how should the networking work?
00:15:15.000 | Like we were very involved in kind of the construction and bring up of this.
00:15:18.000 | And that's what this post is about, is about that process of like bringing up all these, there's like different clusters in different places of different scales.
00:15:25.000 | So in this particular post, we're talking about this one 4096 GPU, but there are other clusters that they have as well.
00:15:32.000 | And we were very closely involved with figuring out the exact architecture and kind of the trade-offs that go along with picking, you know, those exact components.
00:15:40.000 | You really don't want to like place the wrong order cause it takes months to get it and it's very expensive.
00:15:44.000 | So, yeah, we were happy to help out with that.
00:15:47.000 | And then your infinibit cables get stolen.
00:15:48.000 | Yeah, yeah, exactly.
00:15:50.000 | We wanted to make sure that we ended up with compute that would work for us and that would also work for their other customers.
00:15:57.000 | And so we kind of helped design something so that we would get exactly what we were looking for.
00:16:00.000 | We knew that these kinds of details would be super important and that getting down to the level of the hardware and like having these good scripts and everything was going to be a core part of like actually getting this to work.
00:16:09.000 | I'm very glad that we did that.
00:16:11.000 | I don't think that most companies kind of take that like, you know, full stack approach.
00:16:16.000 | But for us, it certainly paid off.
00:16:18.000 | Yeah, it's basically sort of built to spec.
00:16:20.000 | It's interesting that relationship because you usually, for the rest of us who don't operate at your scale, we take whatever we can get from cloud providers, but you are basically co-designing from a single machine up.
00:16:32.000 | And you described that a little bit.
00:16:34.000 | You want to take us through the process that you described here?
00:16:37.000 | Yeah, so for the actual, like the blog post and kind of bringing these machines online.
00:16:41.000 | Yeah.
00:16:42.000 | Yeah.
00:16:43.000 | So, yeah, I think the process as we have it broken down in the blog post, there's kind of a few different layers.
00:16:48.000 | First is like getting the individual machines to work at all and then getting the machines to actually be able to talk to each other.
00:16:55.000 | So getting the InfiniBand networking to work and then getting to a point where, you know, not just the machines are working and they can talk to each other, but everything is actually working correctly.
00:17:04.000 | There's a big gap between like it's working at all two.
00:17:06.000 | It's working perfectly correctly.
00:17:08.000 | And then after you have all this stuff working perfectly correctly, nice and healthy, then now you get into kind of the software data like training issues.
00:17:17.000 | And then after that, you're still not done.
00:17:19.000 | Like now, even once you're training at full speed, things are going to fail over time.
00:17:22.000 | Things are going to change.
00:17:23.000 | There's going to be new firmware updates.
00:17:25.000 | Like how do you kind of deal with this change and flux over time without going crazy and pulling your hair out, trying to like reproduce things or understand why there were regressions.
00:17:34.000 | And so there's a lot of work to kind of automate the infrastructure tooling as well.
00:17:39.000 | And kind of the first step, like bringing these things online in the first place, you know, you have hundreds of machines at this point.
00:17:46.000 | So you don't necessarily want to be like walking around with like a CD-ROM or a USB drive, like plugging it in with your keyboard, like hitting next, next, next on the OS install.
00:17:55.000 | That's not how this works.
00:17:57.000 | You do that for one machine and then you use, we use this thing called Metal as a Service to bring up all the other machines.
00:18:04.000 | So it's a kind of server that can kind of install the operating system on these other machines.
00:18:09.000 | So most like when you're talking about these machines, like each machine is, you know, on the order of hundreds of thousands of dollars.
00:18:16.000 | So they usually come with a kind of out of band management interface as well.
00:18:21.000 | So they don't, they have their InfiniBand networking, they have their normal 100 gigabit per second, Ethernet networking is like dual redundant, et cetera.
00:18:29.000 | And then you also have this extra out of band management network.
00:18:32.000 | So you can log in and you can see like the boot screen or you can see the blue screen of death.
00:18:36.000 | You can like get in there and actually see what was wrong, which is pretty fun.
00:18:39.000 | And it makes it like possible to automate a lot of this work.
00:18:42.000 | So the beginning of that and the blog post goes into much more detail about like exactly how we set these up and kind of the other areas that we ran into.
00:18:50.000 | When you're bringing these online, you'll definitely have failures.
00:18:54.000 | Even if they all worked in the factory, they get shipped, some parts come loose, something fails, something goes wrong.
00:18:59.000 | So when you're bringing them online, there'll be some that don't quite work for all sorts of reasons.
00:19:03.000 | As you start to be working with machines at this scale, like, you know, if something happens one in a thousand times, you're like pretty likely to see it.
00:19:10.000 | And so you can get pretty rare, weird things, especially since we had fairly early builds and fairly early versions of this hardware.
00:19:19.000 | Like these are some of the like first machines that were ever produced, some of the first GPUs.
00:19:23.000 | So you've got some extra special things there.
00:19:25.000 | We definitely worked with Dell, for example, on making fixes in the firmware level to be like, okay, like this thing is wrong.
00:19:31.000 | Like we need to update this at the firmware to like actually fix this particular thing.
00:19:35.000 | So we worked pretty closely with Dell and Nvidia.
00:19:38.000 | Yeah, that's what I'm saying.
00:19:40.000 | Like this stuff gets complicated.
00:19:41.000 | And the thing is like, you know, taking a step back, the whole reason we're doing this, right, is that we knew that this was going to be complicated.
00:19:47.000 | There would be these kinds of failures.
00:19:49.000 | And if we're just using, you know, AWS or some other cloud provider, these errors are still going to be there.
00:19:54.000 | And you're going to have no way to know and no way to debug this and no way to diagnose what's going wrong.
00:19:58.000 | And so we would much rather be able to like call up Dell and say, hey, this isn't working.
00:20:02.000 | And they're like, yep, okay, cool. See if I get together. Oh, I see.
00:20:05.000 | Yeah, cool. We'll ship a firmware update and actually fix this for you.
00:20:08.000 | That was a much better experience than like, great, just magically fails.
00:20:11.000 | I guess we restart and hope that that machine goes away.
00:20:13.000 | Like that's not a very good place to be.
00:20:17.000 | So yeah, that's kind of the first place is getting to a place where like GPU training is working on your single node machines.
00:20:23.000 | You can observe stuff.
00:20:24.000 | We have tons of tooling around like, you know, Prometheus and all sorts of other tools for understanding what's going on in these machines.
00:20:31.000 | You don't want to be like logging into each one and looking at the temperature or something.
00:20:34.000 | You really need to have tooling to collect all these metrics, etc.
00:20:37.000 | Unfortunately, all of the scripts that we have for this are like for this entire cluster.
00:20:44.000 | And for all this infrastructure are a little bit like special purpose for our particular thing.
00:20:48.000 | So it's not that every script that we have, it's not you can just like take this and plug this in.
00:20:52.000 | Even if we did open source all the tooling that we have, you'd still have to do like a lot of work to open source it.
00:20:57.000 | So what we are releasing is as many of the things that we can that are going to be useful for other people.
00:21:02.000 | You're still going to have to have some way of kind of managing these things, making your own like logging aggregators, etc, etc.
00:21:08.000 | So that's kind of bringing them up to the like, you know, the single nodes are working.
00:21:13.000 | From there, it goes into I'm happy to keep going if you want.
00:21:17.000 | Well, I just want to leave the opportunity for John to comment if there's anything that's different from how he runs things.
00:21:24.000 | I mean, all I'll say is I'll endorse this and say, this shit is hard.
00:21:29.000 | Like, this is really, really hard.
00:21:31.000 | And, you know, I have a special props to, you know, the folks in Vue, because they were building this from the ground up.
00:21:40.000 | You know, at Databricks and at Mosaic, we typically work with cloud providers, because some of this stuff is just, there's too much to handle.
00:21:48.000 | It's complicated.
00:21:49.000 | There's a lot to deal with.
00:21:50.000 | And this doesn't even get into things like physical security, you know, securing power.
00:21:54.000 | If you're the data center operator, like this gets infinitely complicated and you have to abstract somewhere.
00:22:01.000 | Like, you know, and then you get to the folks who are literally building their own custom chips and like, good God.
00:22:06.000 | Oh, my God.
00:22:07.000 | That's, you know, if you're one of those folks, you're having, you know, pour one out for the infra people at some of the AI chip startups who are having a really, really interesting time right now.
00:22:18.000 | But this stuff is really hard.
00:22:20.000 | And I don't think we talk about it much because there's so many other things that are hard.
00:22:24.000 | But the other hard things I think everybody's becoming pretty familiar with at this point.
00:22:28.000 | This is something that I don't think there's ever really been a comprehensive discussion of, at least not that I've seen.
00:22:33.000 | Yeah, so my impression is that you guys, Mosaic, have your own software for sort of spinning up and down machines, just like InView had to build.
00:22:42.000 | But InView probably, it sounds like InView, you guys went fuller stack.
00:22:48.000 | I don't know how, I don't know how to describe it.
00:22:50.000 | Like, like Mosaic is not working with Dell on like their firmware.
00:22:55.000 | No, no.
00:22:56.000 | We're typically working with like, you know, pick your cloud provider on their Dell firmware or what have you.
00:23:02.000 | Like, it's kind of, I think, I think one of the things, I don't know, Josh, you can correct me on this.
00:23:06.000 | It's kind of impossible if you're doing training to not go all the way through the entire stack, regardless of what happens.
00:23:12.000 | Like, somehow I'm still chatting with cloud providers about power contracts, even though the whole point of dealing with the cloud provider is not to have to think about power contracts.
00:23:20.000 | Somehow I'm still asking them about which InfiniBand provider they used this time to see if this is part of the bad batch of cables I encountered on that cloud provider or what have you.
00:23:29.000 | Or like, we're still talking about a firmware update from pick your provider.
00:23:33.000 | Like, you can't not do this.
00:23:36.000 | It's convenient that they have data center staff who are worrying about what to send back to which provider when.
00:23:41.000 | And they have people who can go and wait for the InfiniBand cables so they don't get stolen outside.
00:23:45.000 | But, you know, it's kind of, it's impossible not to really go full stack if you're thinking about the infrastructure at all.
00:23:52.000 | I don't know, Josh, correct me.
00:23:54.000 | No, I think that's right.
00:23:55.000 | And that's what we expected from the beginning as well, is that we would have to get inevitably have to get into the details here.
00:24:01.000 | And I'm glad that we kind of just planned for it.
00:24:03.000 | I think it made it a lot easier from our perspective to have direct control over this.
00:24:07.000 | Instead of having to go to the cloud provider that goes to the data center that goes to the supplier, we could just go direct to NVIDIA or Dell or the data center, whoever was responsible and be like, hey, this thing needs to change.
00:24:18.000 | And they're like, okay, yeah, that is our responsibility.
00:24:20.000 | Great, we can fix that.
00:24:21.000 | So it was just a lot easier for us to fix these bugs.
00:24:24.000 | And if we had to go through an extra layer of email.
00:24:27.000 | Something we discussed in the pre-show was that you had a rule of thumb for your cluster of reliability.
00:24:35.000 | You say here in the post, by and large, you expect around 3% of your machines to break every week.
00:24:40.000 | So you're basically going to turn through all your machines in a year.
00:24:43.000 | As it says in the post.
00:24:46.000 | So that would be true if it was a uniform failure like that.
00:24:51.000 | But as it says in the post, it's usually these kind of problematic nodes.
00:24:55.000 | And to be clear, that is the number that we've heard from other people is like they're having about 3%.
00:25:00.000 | I don't think we're experiencing failure rates that are that high.
00:25:02.000 | I think ours is actually quite a bit lower than that.
00:25:04.000 | Probably because we've taken the time to dig into a large, maybe larger number than we should have of these failures and get to the root cause of it and be like, oh, okay, that's exactly what's going wrong.
00:25:14.000 | How do we fix this?
00:25:15.000 | How do we prevent this from happening?
00:25:16.000 | How do we make automated checks for this so that if it does happen, it just goes back to whoever owns that particular part of the process and they can fix it immediately.
00:25:24.000 | And that's part of what you're open sourcing, which is the health checks, right?
00:25:28.000 | You got the NIC health checks, GPU health check, disk space health check, Docker D message.
00:25:34.000 | I don't know what that is.
00:25:35.000 | That one is just a lot of stuff.
00:25:38.000 | Yeah, that one is one where we realized that actually like when these machines boot, sometimes they wouldn't actually boot cleanly all the way.
00:25:45.000 | Or when they rebooted, they had problems that they didn't have when they were working before, which was kind of frustrating.
00:25:51.000 | Like usually if you restart your computer, it gets better here.
00:25:53.000 | You restart, it did not get better.
00:25:54.000 | It got worse.
00:25:55.000 | That was very frustrating.
00:25:56.000 | So this health check looks at every particular line we've ever seen from the boot, like in D message, like every single log line that your computer emits and says like, have we ever seen this before?
00:26:08.000 | Is this expected?
00:26:09.000 | Is this in the right order?
00:26:10.000 | Or is there something out of place?
00:26:11.000 | If there's anything out of place, let me say, okay, great.
00:26:13.000 | Like now it goes into this like longer, more triage list of like, all right, great.
00:26:17.000 | Like, is this acceptable?
00:26:18.000 | Should we flag this?
00:26:19.000 | Like, should someone take a look at this?
00:26:20.000 | So we're looking down at a very, very granular detail level what's happening on these computers to make sure that nothing is out of place.
00:26:26.000 | And that's critical because without that, if you're running your training, as Jonathan said, and this thing is slow, like what are you supposed to do?
00:26:34.000 | Right?
00:26:35.000 | Like you really, you really want to be very certain that like all 4,000 of these GPUs are working like they're supposed to.
00:26:40.000 | We know that.
00:26:41.000 | And so if it's slow, it's because like we messed up the config or something else.
00:26:43.000 | And not because of this earlier thing that's like really hard to detect in software later.
00:26:47.000 | Yeah.
00:26:48.000 | I think the, I'm just curious to ask, like, you know, suppose you were to set up another, let's say another H100 cluster and it were at a different data center.
00:26:56.000 | And instead of the vendor being Dell, it was super micro or what have you.
00:26:59.000 | How much of this would be repeatable and how much of this would you have to redo?
00:27:03.000 | I, you know, I genuinely don't know.
00:27:05.000 | A decent amount.
00:27:06.000 | I think it would go a lot faster the second time.
00:27:08.000 | I think there's lots of learnings that we had.
00:27:10.000 | And also the blog post, you know, yes, we are releasing the health checks, releasing some scripts.
00:27:14.000 | But a lot of the valuable stuff is also in the blog post itself, in the details and kind of the, you know, the learnings that we've had and the sort of errors that we run into.
00:27:21.000 | We tried to as much as possible surface those so other people could learn from those and avoid the same mistakes or failures as well.
00:27:27.000 | But I think it would go a lot faster.
00:27:29.000 | Although, yes, there would certainly be some things that'd be a little bit different.
00:27:32.000 | I mean, there'd probably be different CPUs or whatever.
00:27:35.000 | But I think a lot of that stuff is less.
00:27:39.000 | It's less, that's the like, that's less variable.
00:27:42.000 | I think most of it would apply the second time around.
00:27:46.000 | Although I'm sure next time we're building one, it'll probably be, you know, at a scale of 10X as big with a different chip or something like this.
00:27:51.000 | And then who knows?
00:27:53.000 | Yeah, with ConnectX 8 that will have its own fun behavior and all that good stuff.
00:27:57.000 | Yeah.
00:27:58.000 | Perhaps something that people don't discuss about, and you don't even talk about this in the blog, but I always wonder is what is the timeline that's like kind of reasonable for this amount of work?
00:28:08.000 | At least the initial stages.
00:28:10.000 | And also what does the team composition look like for setting up a cluster, right?
00:28:15.000 | Like what are the mix of skills that you typically would require to get all this going?
00:28:20.000 | I can't really speak to typical.
00:28:22.000 | One thing I am very proud of is how much we accomplished with such a ridiculously small team.
00:28:28.000 | Like our infrastructure team is like, you know, fluctuates from week to week, depending on like how many things are on fire and how much we need to build.
00:28:35.000 | But it's like between like three and six people.
00:28:37.000 | Like it's small.
00:28:38.000 | It's not like some huge team of like tons and tons of engineers.
00:28:42.000 | But those people are very, very good at what they do.
00:28:45.000 | And so that has allowed us to get a lot of mileage out of these things.
00:28:49.000 | I think it's not that we're building everything right.
00:28:52.000 | It's not that three to six people build this whole thing.
00:28:54.000 | I definitely want to like, you know, say thanks very much to Dell and H5 and NVIDIA and the other people that have done a lot of the work like to bring up this cluster.
00:29:04.000 | You know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables.
00:29:10.000 | So that's 24,000 things that need to be plugged in.
00:29:12.000 | Like that's just a lot of stuff to plug in.
00:29:15.000 | Right.
00:29:16.000 | And you don't want to mess it up.
00:29:17.000 | Like each one needs to be done correctly.
00:29:18.000 | It's a little bit loose.
00:29:19.000 | Like it doesn't really work.
00:29:20.000 | If you break it, you need to replace it.
00:29:21.000 | Like there's a lot of work that goes into this.
00:29:24.000 | Yeah.
00:29:25.000 | And then, you know, that's just like that's it.
00:29:27.000 | That's if you were to do everything right the first time and if you didn't have to fix anything.
00:29:31.000 | But inevitably, you know, you will have to replace something, which means like taking all the wires out, pulling the thing out, taking all the GPUs out, going and fixing some cable, putting it all back correctly, putting it back in, doing this every time.
00:29:42.000 | Like there's a lot of work that goes into it.
00:29:44.000 | There's a lot of people at Dell, NVIDIA, and at H5 that all helped a ton with this stuff.
00:29:49.000 | I don't know the exact size of the Dell team.
00:29:51.000 | It also fluctuated over time.
00:29:54.000 | Yeah, excellent.
00:29:55.000 | And then, you know, so you have all the hardware set up and now you're firing it out for a single node.
00:30:01.000 | There's a long description that you guys have about just like monitoring the MFU, right?
00:30:07.000 | And what each situation might look, might be indicative of.
00:30:13.000 | One of the most interesting things to me that I saw from here is like, you know, if training immediately starts off at 60 to 80% MFU, something's wrong.
00:30:24.000 | But like, you know, like what are like, you know, some anecdotes or, you know, notable scenarios here that you might call out as maybe counterintuitive or super interesting?
00:30:35.000 | I mean, there's just so many of them.
00:30:37.000 | I mean, one of them, which I think is probably pretty common, like common knowledge by this point, but like we did have a sort of like, which one was this exactly?
00:30:49.000 | I think for the MFU, like gradually getting worse over time.
00:30:54.000 | I think that one, when we saw that the first time we were like, what the heck is going on?
00:30:57.000 | Like, why does it get just like a little bit worse?
00:30:59.000 | This is so strange.
00:31:00.000 | Like, what, is it getting lazy or tired or something?
00:31:02.000 | Like, is it heat?
00:31:03.000 | Like what's going on?
00:31:04.000 | And in this particular case, it was memory fragmentation.
00:31:08.000 | Because you have hundreds of machines, they're doing garbage collection at slightly different times.
00:31:12.000 | And then they get slightly further apart and slightly more and more jittered until eventually they're all happening kind of at random times and just like really messing up each one of your steps.
00:31:20.000 | So you just turn off garbage collection and call it a day, basically, to be honest.
00:31:24.000 | There's other things you can do if you want to be a little bit more sophisticated about it, but.
00:31:27.000 | You can also just manually have it all garbage collect on some interval.
00:31:31.000 | Like that's what we've done.
00:31:32.000 | We just have a garbage collection callback that just runs.
00:31:35.000 | But I've seen the exact same thing.
00:31:36.000 | Yeah, yeah, exactly.
00:31:38.000 | So I thought that one was kind of funny.
00:31:39.000 | And we did trace that one down and look.
00:31:40.000 | And we did find the actual call.
00:31:41.000 | Like, again, this goes to like having good tools.
00:31:44.000 | So we had really good tools where we could look at a bunch of like actual traces in C and be like, okay, cool.
00:31:49.000 | This is the thing that's taking a lot of time.
00:31:51.000 | Or like, you know, this is the thing that doesn't quite line up here.
00:31:53.000 | Like, oh, I guess it's garbage collection.
00:31:55.000 | Okay, cool.
00:31:56.000 | Interesting.
00:31:57.000 | Yeah, let's just try taking that off.
00:31:58.000 | Okay, great.
00:31:59.000 | That's what it was.
00:32:00.000 | Now we can fix it.
00:32:01.000 | Yeah, so for each of them, like basically bugs are not hard if you have good tools.
00:32:05.000 | But if you don't have good tools, bugs can be very, very hard.
00:32:08.000 | So similarly for like heat, another thing that we saw was like, oh, you know, the CPU is getting throttled.
00:32:14.000 | Okay, well, it's easy to see if you're monitoring the CPU throttling or monitoring the heat.
00:32:18.000 | If you're not monitoring that, it's really hard to know why it's just suddenly one of them is going slower.
00:32:23.000 | I noticed also in the piece that you mentioned FSDP with 0.3.
00:32:27.000 | Actually, I went to iClear and Guanhua from the DSP team was there presenting 0++.
00:32:34.000 | I was wondering if you want to make any call outs to, you know, particular open source or open library
00:32:41.000 | or open whatever implementation teams that were super helpful in your process.
00:32:47.000 | I think we ended up actually pulling from a whole bunch of different ones to pull things into our own particular pipeline.
00:32:53.000 | So we use things from NVIDIA's, you know, Megatron stuff.
00:32:56.000 | We use stuff from probably DeepSpeed.
00:32:59.000 | I think we pulled in a bunch of different pieces from a bunch of different places.
00:33:02.000 | So it was really nice to see all these working open source, like examples.
00:33:07.000 | I think I really appreciate all the effort that has gone into actually tuning these things.
00:33:10.000 | Because you can tune them, but it's a lot of work to like tune this stuff and do all this stuff from scratch.
00:33:15.000 | It's really nice to have like a working example.
00:33:18.000 | I think those are probably the two biggest ones, DeepSpeed and Megatron alone, but there are probably other ones as well.
00:33:23.000 | Is there, is there a particular thing in the ecosystem where you would call out as like, you know,
00:33:28.000 | there should be something here that is open source, but like, it's not really, like everyone kind of builds it on their own.
00:33:35.000 | Hmm.
00:33:36.000 | I want to say something with the file system because everyone talks about the file system eventually.
00:33:41.000 | The file system actually was, I mean, we did something kind of dumb there.
00:33:47.000 | Like we have our own sort of local mirror so that we can, you know, like a crappy version of S3 that's local.
00:33:52.000 | But it's just a pretty simple script, right?
00:33:54.000 | Like, I think we run like a little web server that just like serves files and then, you know, can upload them and download them.
00:33:59.000 | Okay, great.
00:34:00.000 | And part of the reason we did that is that our internet connection in the beginning was not the like full speed one that we would eventually have.
00:34:07.000 | And so we are a little bit more kind of bottlenecked in terms of internet bandwidth.
00:34:11.000 | And so we had this, I think we looked at a bunch of services out there like Minio and some other ones.
00:34:17.000 | But a lot of these like come with a lot of extra overhead and maintenance.
00:34:22.000 | And since we already have so much infrastructure to deal with, we kind of didn't want to, you know, bring in a whole other like cloud provider, virtualize something, something.
00:34:29.000 | We just wanted something simple.
00:34:30.000 | So we went with that, which is, which has been quite helpful.
00:34:33.000 | Like the, our tools are usually quite simple.
00:34:35.000 | It's like Bash and Python and SSH and Docker.
00:34:37.000 | Like we'd like to keep things simple so that it's easier to debug, like less layers of infrastructure, less layers of abstraction, make it a lot easier to work with.
00:34:44.000 | Like we don't use Kubernetes, for example, I would just directly launch these things.
00:34:48.000 | And it's just been much easier to debug this way.
00:34:50.000 | One, one tool actually that does come into mind that I will call out is Kraken from Uber.
00:34:58.000 | That was great.
00:34:59.000 | We love that tool.
00:35:00.000 | We were a little bit skeptical.
00:35:01.000 | What is it?
00:35:02.000 | I'm sorry.
00:35:03.000 | So Kraken is just, yeah, it's a distributed like Docker registry basically that uses BitTorrent to like transfer things between the machines in a sort of nice optimal way.
00:35:15.000 | Like in the very beginning, the naive way is like you have this one Docker registry, which was outside of the cluster.
00:35:20.000 | So every time we change an image, you know, there's many gigabytes that each of the 500 machines needs to download.
00:35:25.000 | So that just takes a really long time.
00:35:27.000 | So what this thing does is like just one of them downloads it and then like they all sort of broadcast all the pieces to each other.
00:35:33.000 | And it was just like a really nice, fast way of getting these images down.
00:35:36.000 | And it was very robust.
00:35:38.000 | Like there's a lot going on under the hood, but I think it's a pretty cool tool that we haven't really had any bugs with it at all.
00:35:44.000 | Amazing.
00:35:46.000 | Yeah, I mean, that's all my questions, I guess, for the info piece.
00:35:49.000 | I don't know if, John, you had something that you were sort of burning to ask.
00:35:53.000 | All I can say is just same in a lot of places.
00:35:57.000 | Plus one.
00:35:58.000 | They've done that, seen this, plus one.
00:36:00.000 | I think the one big difference, you know, perhaps in philosophies is we've tried to basically standardize on as much commodity stuff as possible.
00:36:08.000 | Just because, you know, I think the reason I asked about trying to do this on multiple different pieces of infrastructure is like, I think we're running on like six or seven different clouds right now.
00:36:18.000 | And everybody has done something slightly different.
00:36:22.000 | And my gosh, the little differences add up, as you know, you've seen.
00:36:26.000 | And so, you know, our philosophy has been like, okay, whatever the hell we can standardize, please let's standardize it.
00:36:31.000 | Like vanilla off-the-shelf FSDB.
00:36:34.000 | And like, you know, we wrote our own data loader, but we've tried to make that as much of a standard as we can across our infrastructure and in Databricks.
00:36:40.000 | Because things just start getting really complicated.
00:36:42.000 | Or like we use Kubernetes extensively because it at least gives us a uniform set of APIs.
00:36:47.000 | Like that's our hardware abstraction layer to a certain extent for everything else.
00:36:51.000 | So it's just, you know, a difference in philosophy there, but otherwise like, yeah, this stuff is really, really hard.
00:36:58.000 | And I feel like we take for granted how much of this, you know, is done for us when you go and you just query chat GPT, for example.
00:37:07.000 | Like, oh my God, everything going on underneath that.
00:37:10.000 | You know, it's kind of a miracle that the machines boot up, let alone that you can like query a giant language model that's probably doing inference across multiple machines and was trained across thousands of machines.
00:37:19.000 | Like, you know, minor miracle.
00:37:22.000 | Yeah, it is an awesome amount of power that we invoke with a single API call that we take for granted these days.
00:37:27.000 | It's absurd.
00:37:29.000 | Yeah, I mean, like Kubernetes, like that point about Kubernetes, I will say as a former AWS employee, like it seems like it would be ideal for imbue to at some point make it more abstracted or agnostic.
00:37:45.000 | Because you're going to want to, you know, replicate your setup.
00:37:49.000 | We do have our own sort of replacement, but it's just a much simpler version of Kubernetes.
00:37:52.000 | Kubernetes is really designed for running services, not for running experiments.
00:37:57.000 | Like that's not its like main architecture.
00:38:00.000 | And so for us, like we have a thing that's like, cool, you're going to run an experiment.
00:38:03.000 | So you want it to run to completion, right?
00:38:05.000 | Okay, great.
00:38:06.000 | Like the primitives are sort of built around a slightly different style.
00:38:08.000 | And that makes it a lot easier, like just a lot simpler to fit the nature of like these machines are going to disappear.
00:38:14.000 | They will need to be rebooted for infrastructure upgrades.
00:38:17.000 | Like something will happen to the GPUs, failures like baked into this as like a core part of our infrastructure.
00:38:22.000 | So it's not that we don't have an abstraction.
00:38:24.000 | It's that it's a sort of simpler, more tailored abstraction for the particular work that we're doing.
00:38:29.000 | Yeah, I think it all depends on what your goals are.
00:38:32.000 | And like, I think the challenge in a lot of the deep learning stuff right now is that people are trying to, like people often build things that are more complicated than necessary to get the job done.
00:38:42.000 | And the complication is the enemy of everything.
00:38:45.000 | You know, don't use a fancier parallelism strategy than you have to.
00:38:48.000 | Don't use a fancier set of libraries than you have to.
00:38:51.000 | Don't do anything that you don't have to do because it's hard enough as it is.
00:38:55.000 | Like don't overcomplicate your own life.
00:38:57.000 | Don't try to bring in more tools or more fancy architecture tweaks if you absolutely don't have to.
00:39:02.000 | Like getting to the minimum necessary to get the job done.
00:39:06.000 | And it's really tempting to want to try to use everything.
00:39:09.000 | So like, I totally understand that one.
00:39:12.000 | I think the last piece I'll maybe call out is that I'm just going to weave this in just because I see the opportunity to do it.
00:39:19.000 | Are there any infrastructure shifts that need to be, that need to rise because of changing architecture?
00:39:29.000 | So I think, for example, in Vue, like you're announcing a dense model, a 70B dense model.
00:39:37.000 | Whereas John just worked on DBRX and the sort of image-to-text, the text-to-image model, which presumably has different bottlenecks.
00:39:47.000 | That's correct for us.
00:39:50.000 | You know, we train both dense and mixture of expert models.
00:39:54.000 | The one we happened to, you know, kind of get permission to open source was a mixture of expert model.
00:39:59.000 | And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP 03 style.
00:40:06.000 | Where there's just a lot of parameters getting shuffled back and forth.
00:40:09.000 | And your ratio of kind of compute to amount of data that you have to shuffle back and forth becomes a lot worse.
00:40:15.000 | Because you're now, you know, you're only using a fraction of the parameters for every token instead of all the parameters.
00:40:21.000 | And so we had to really push the envelope on getting all the stuff to the right places on time.
00:40:27.000 | And so actually the networking part of DBRX was the single hardest thing, I think, of the entire process.
00:40:33.000 | Just get MOE training, working at scale across a big cluster.
00:40:36.000 | We still managed to, I think, do it all with commodity parts, which was very exciting.
00:40:42.000 | You know, the, like, we were using FSTP and we eventually used HSTP so that we could have, HSTP is a version of FSTP where you have multiple smaller replicas.
00:40:52.000 | And you're doing data parallel within those replicas.
00:40:55.000 | And that helped a lot with network latency issues that we were running into just because we were transmitting so much data, you know, for every single part of the process.
00:41:04.000 | I think it actually, like, it was instructive for how Google designs their hardware and software together personally.
00:41:10.000 | Their training, as far as I understand, using kind of a 0.3 style of training, it happened for a while.
00:41:15.000 | They also train mixture of expert models.
00:41:17.000 | TPUs have a very different network bandwidth to compute ratio.
00:41:20.000 | They have a lot more bandwidth, just objectively.
00:41:24.000 | And TPUs per chip tend to be a little bit less compute intensive and have a little bit less memory.
00:41:29.000 | You know, it's just a different design choice.
00:41:31.000 | So the ratio of flops to bandwidth is very different.
00:41:36.000 | And that means that it's much easier for Google to be able to pull off some of this stuff.
00:41:40.000 | They also have interesting, you know, TOR style network architecture, or TOR style, like, literal network architecture.
00:41:45.000 | It's not like the model, but the network.
00:41:47.000 | Is this the sort of block attention?
00:41:50.000 | I forgot what you call it.
00:41:52.000 | So this is just more, yeah, this is more, not the ring attention, but these are the ring all reduces.
00:41:57.000 | Like you have three different dimensions of rings because they, they kind of put you in these three dimensional toruses from what I understand.
00:42:02.000 | And so like, you know, Google's infrastructure in some sense is kind of, I wouldn't say built for this, but maybe the way that Google trains models is built for a slightly different bit of infrastructure they have.
00:42:13.000 | And it's kind of neat to think about that, you know, as, as one thing that I think NVIDIA announced for, you know, for, for both the GH200 and the GB200.
00:42:22.000 | Is this hybrid networking where you'll have blocks of NVLink networked chips?
00:42:27.000 | I think for the GB200, I think it's like groups of 72 GPUs will all have NVLink to each other.
00:42:33.000 | So higher bandwidth, then you'll have normal networking of some kind, InfiniBand or Rocky or what have you between these blocks.
00:42:40.000 | And that's kind of a, you know, it's a change due to the fact that, you know, it's hard to build really high bandwidth networks over very large groups, but it is now a blocked networking.
00:42:51.000 | And you have to think about how you architect your model and your parallelism differently.
00:42:56.000 | You also have to think about fault tolerance differently because it now matters where you lose a GPU, whereas it didn't before.
00:43:02.000 | So, you know, it's, it's, it's just all really interesting and really fun speaking personally.
00:43:08.000 | But it's going to mean new nightmares when we all move to that generation and have to think about, you know, new versions of these problems.
00:43:15.000 | As you go up to larger scales, it gets quite different.
00:43:19.000 | Like right now, you know, if you're experiencing, let's say, for example, you experience a GPU failure every day, that's fine, just restart.
00:43:27.000 | If you make your thing 24 times as big, now it's once an hour.
00:43:30.000 | Now it stops being quite as easy to just restart, right?
00:43:33.000 | So now you have to kind of break, like bake in this sort of redundancy that you didn't have before.
00:43:38.000 | So I think as you go up in scale, you end up running into like a lot of really interesting problems that also inform the, the actual like design.
00:43:49.000 | Yeah, I mean, as an orchestration guy, this is why I always emphasize like very cheap storage or very fast storage so you can checkpoint more.
00:43:57.000 | But I don't think that's probably not the best solution for fast, you know, training.
00:44:03.000 | Which works fine when you're doing language and then you move to vision or video.
00:44:08.000 | And then, you know, you have multi petabyte data sets and getting, you know, cheap, fast multi petabyte storage starts to bite.
00:44:15.000 | Like I've certainly encountered issues where the literal data center where my GPUs were did not have enough, you know, object store to fit the data sets that people wanted to bring into that data center from whichever users were, were trying to bring them in.
00:44:30.000 | And then you get to a whole different world of hurt where you have to keep your data in a different region because the region is just out of storage.
00:44:37.000 | So things get fun really fast.
00:44:40.000 | Speaking of vision, Josh, actually, you know, Embu is an agents company, but you're only, you're announcing a text only model.
00:44:48.000 | Where does, where does the vision side come in?
00:44:51.000 | I think we've actually done a lot of work in the past and people can see kind of our blog posts about sort of self supervised learning and some other kind of vision related stuff in the past as well.
00:45:00.000 | So we're very familiar with, with that stuff.
00:45:02.000 | But I think our main focus right now is on kind of, as we say, coding and reasoning.
00:45:08.000 | And there, there's certainly a visual component to some problems.
00:45:12.000 | But, you know, it's not necessarily required for all problems.
00:45:17.000 | And actually we found that for most of the kind of like code writing and reasoning problems that we care about, the visual part isn't really a huge important part of it.
00:45:25.000 | Sometimes if you really need to, you can maybe describe the thing.
00:45:29.000 | There are other like, you know, multimodal models that you can use off the shelf to sort of plug in for those particular pieces that you need, right?
00:45:35.000 | Like if something is driving a browser or whatever, like you can sometimes get away with not having to have that baked into the original model.
00:45:42.000 | So our focus, you know, in a sense, we kind of do a lot across the stack.
00:45:47.000 | We're working on our own infrastructure and pre-training and RL and fine tuning and products and everything.
00:45:52.000 | But in another sense, we're very narrowly focused on the application side.
00:45:55.000 | So all of the stuff across the stack is kind of going toward like a very particular purpose.
00:45:59.000 | And so that particular purpose right now doesn't really need vision.
00:46:02.000 | So we think that people are going to make all sorts of really cool image models like Jonathan, right?
00:46:06.000 | And all sorts of interesting multimodal models into the future.
00:46:09.000 | We'll let them go do that.
00:46:10.000 | That's great.
00:46:11.000 | We'll take advantage of that, partner with those people in the future.
00:46:15.000 | And right now we're really focused on kind of like core reasoning and coding capabilities and aspects of the model.
00:46:21.000 | I wanted to go into Carbs since like that's like kind of the next layer of the stack.
00:46:27.000 | We talked about Carbs in the first episode with Kanjin, because you've actually had a blog post about it like a couple of years ago.
00:46:34.000 | Maybe let's introduce it.
00:46:35.000 | Has that been a couple of years now?
00:46:37.000 | No, it must have been at least one year.
00:46:39.000 | Hopefully it's not a couple of years.
00:46:41.000 | Sorry.
00:46:42.000 | I'm counting AI time.
00:46:43.000 | Yeah.
00:46:44.000 | I was going to say, you're making me feel really old right now.
00:46:49.000 | I count everything before the General Intelligent Rename as like, you know, prehistory and now sort of modernity, right?
00:46:57.000 | So I actually thought Carbs was more about hyperparameter optimization in a sense of like sort of parameters, hyperparam search.
00:47:05.000 | Whereas, you know, when you introduced it, especially in this blog post, it's more about scaling laws and predictability of like, are we sort of in the right ballpark before we scale things up?
00:47:18.000 | Maybe sort of recap the history of Carbs.
00:47:21.000 | Yeah.
00:47:22.000 | So it really is a little bit of both.
00:47:24.000 | So Carbs is, it's maybe a backronym, but it's for Cost-Aware Pareto Region Bayesian Search.
00:47:33.000 | So this is about technically how it works.
00:47:36.000 | But Carbs is like, you know, we like pastries and stuff.
00:47:38.000 | So great.
00:47:39.000 | Why not?
00:47:40.000 | But the point is that it's a cost-aware hyperparameter tuner.
00:47:44.000 | So most hyperparameter tuners, you kind of say, OK, here's this objective function.
00:47:50.000 | I want you to make this number as big as possible or as small as possible, whichever direction you want to go.
00:47:54.000 | So, yeah, just go make this number, you know, as small as possible.
00:47:57.000 | OK, so it'll try a bunch of different hyperparameters, a bunch of different configurations to figure out, like, how do I tweak your network and architecture, et cetera, to get the kind of best performance I possibly can.
00:48:08.000 | That's usually saying, like, you know, almost all of these hyperparameter configurations, let's say they're all going to use the same number of GPUs or the same number of nodes, so it's going to run for the same amount of time.
00:48:18.000 | So you can do that.
00:48:19.000 | You can get a number out, and that's great.
00:48:21.000 | But what Carbs does is it says, OK, actually, what if we relax that constraint?
00:48:24.000 | What if we say each of these different points, we're going to model how expensive it will be to sample this configuration.
00:48:31.000 | So what if we train with just 1/100 of the data?
00:48:34.000 | Like, how well can we do?
00:48:35.000 | What if we train with 1/10 of the data?
00:48:37.000 | What if we train with all the data?
00:48:39.000 | That way, you can understand, like, as we get more and more data, as we spend more and more compute, as we make a bigger and bigger network, how does performance change with these things that change, like, how expensive it is to even explore this data point?
00:48:50.000 | So by doing that, we can see the scaling laws for not just, you know, the scaling laws from, like, the, you know, Chantilla paper, but scaling laws for all parameters.
00:48:58.000 | We can see how does the number of layers change with this?
00:49:01.000 | How does the, you know, the learning rate change?
00:49:03.000 | How do the, like, you know, various types of regularization change?
00:49:06.000 | So you can see these nice scaling laws.
00:49:08.000 | And as you're going across costs, like, how should this be changing as you're scaling up your model?
00:49:13.000 | So that, coupled with the kind of metric that we chose, which is a very precise way of measuring performance, allowed us to really, like, hone in on parameters that worked really well and understand, like, how do we want to scale those up?
00:49:25.000 | Especially as we're changing things about the network.
00:49:27.000 | Like, one of the things that we did is we used a custom tokenizer.
00:49:30.000 | As we change this tokenizer, it changes a bunch of other things about the model.
00:49:34.000 | So how should we scale up this entirely new tokenizer?
00:49:36.000 | Like, no one has ever made a model this large with this tokenizer before.
00:49:40.000 | And so how do we want to change all these things?
00:49:42.000 | Harps kind of shows you, like, look, as you change these parameters, like, these other ones are kind of dependent on this.
00:49:47.000 | Like, this is the, these are the relationships between them.
00:49:49.000 | So you can better understand, like, okay, if I'm going to scale this up 10x or 100x, like, where do I want to be?
00:49:54.000 | You can only go so far.
00:49:56.000 | And so, you know, we did run, like, I think maybe it was like a 14b one or something like that to check.
00:50:01.000 | But, and so we had a bunch of, like, 1b, a 14b, and then a 70b.
00:50:05.000 | I don't think we had a, I think we just did, like, one at 14b.
00:50:08.000 | So you can, we get to check to, like, oh, is this on the curve?
00:50:11.000 | Like, is this where we expect it?
00:50:12.000 | It was, like, right there.
00:50:13.000 | So then, great.
00:50:14.000 | Go on to the next one.
00:50:16.000 | Yeah, I mean, that makes a lot of sense.
00:50:18.000 | I wonder if, so one of the key questions, and correct me if I'm wrong, but, like, usually people do search or do their evals just based on loss.
00:50:30.000 | But you actually evaluate based on, you know, the sort of end-state evals that people might expect, like HellaSwag and Lombata, whatever.
00:50:41.000 | What is the norm here?
00:50:43.000 | Is there a norm?
00:50:44.000 | Yeah, I don't know if it's 100%.
00:50:46.000 | I don't know.
00:50:47.000 | I only see loss on most people's reports.
00:50:50.000 | I think it's easy to, like, loss is very nice because it's very precise.
00:50:56.000 | It will tell you, like, very fine-grained differences between, like, really small changes in your hyperparameters or network architecture.
00:51:02.000 | Whereas, especially at the smaller scales, if you're looking at, like, accuracy, it's very noisy.
00:51:06.000 | Like, it might be zero or 100 or, like, you know, fluctuating by, like, 10 or 20 percentage points, which makes it really hard to tell, like, did that change actually mean anything?
00:51:14.000 | So our loss is sort of a combination of these two.
00:51:17.000 | Instead of saying, like, let's just look at perplexity, we say, let's look at perplexity on the task that we care about for multiple-choice questions, effectively.
00:51:26.000 | So we're saying, like, yes, this is formulated as a multiple-choice question, and we're going to look at the, like, you know, the loss, the perplexity for this particular answer token.
00:51:34.000 | And that ends up being something that's, like, both targeted to what you actually care about and also very precise.
00:51:39.000 | The nice thing about this, though, is that it's independent of the data that you train on.
00:51:43.000 | The other thing that's annoying about perplexity or about loss is that as you change your dataset, this is really obnoxious because now it fundamentally changes your loss, right?
00:51:51.000 | So you can't tell, like, how do I tweak my dataset?
00:51:53.000 | But because we have this held-out evaluation dataset where we're looking at perplexity, we can actually change the data mix.
00:51:59.000 | And so CARBs actually control what is the mix of data that we want to see, like how much code, you know, how much internet text, et cetera, in order to figure out what is the best optimal mix of data.
00:52:11.000 | That because we have this other metric.
00:52:13.000 | So that was one of the things that was really, really helpful.
00:52:15.000 | I think there is a trend overall about changing data mix as training goes on.
00:52:23.000 | I don't know how, you know, we're deciding not to talk about datasets in this podcast.
00:52:30.000 | But what have you observed about the changing data mix question?
00:52:36.000 | We did some experiments and we've actually talked to a bunch of researchers who are doing work here as well and looking at kind of their experiments on this.
00:52:44.000 | And we were originally pretty hopeful because it sounds like something that should work and make sense, right?
00:52:49.000 | Like, oh, cool, like maybe you would have your model, like, learn the basic features and then over time it could get really good at these complicated math problems or coding or something, right?
00:52:57.000 | But it just turns out that, like, yeah, it's just not the way it works.
00:53:00.000 | Like, oh, we've done so many experiments and you can get like a tiny, tiny little boost from this.
00:53:04.000 | But it just is not like, it's just not the important thing, at least in the experiments that we've seen.
00:53:09.000 | So, yeah, we've kind of, we're letting other people explore that more if they want.
00:53:13.000 | But that just doesn't seem like the most promising direction for us.
00:53:16.000 | We've had some surprisingly good luck with this.
00:53:19.000 | We just released a paper on it.
00:53:21.000 | The details matter a lot and it really matters what you're trying to do with the model.
00:53:26.000 | But it's been, it's been quite effective for us, depending on the setting.
00:53:30.000 | And certainly when we're thinking about domain-specific models, this helps a ton.
00:53:34.000 | You know, to a certain extent, you can always think of this as like early fine tuning.
00:53:38.000 | But yeah, I, like, there've been little glimmers of this in the literature for years.
00:53:43.000 | Like, especially, I think the Gemini 1.5 paper mentions this.
00:53:46.000 | And I don't remember whether the Llama 3 paper mentions this, but it's kind of, it's one of those, like, people have different ways to get to these endpoints.
00:53:55.000 | I think, you know, there are the architectural tricks that each lab has to mitigate loss spikes or what have you.
00:54:02.000 | And everybody's got, you know, their own bag of tricks.
00:54:05.000 | And it leads to kind of sometimes this contradictory information.
00:54:08.000 | It's not contradictory.
00:54:09.000 | People are just kind of exploring different parts of the space in some sense.
00:54:11.000 | And there are lots of ways to get a great model.
00:54:13.000 | But certainly for us within our config, and it seems like, I guess, for the folks at Google within kind of the part of the world they live in, changing the data set has helped.
00:54:22.000 | But the details matter a lot, and it's really hard to get those details right for the reasons Josh, you know, just mentioned.
00:54:27.000 | Like, there's a lot of search involved, and you essentially have to make hard choices about what parts of the space you're going to search and which ones you're going to leave be.
00:54:34.000 | And so, you know, some people have done an amazing job.
00:54:37.000 | Like, I think the, who is it, the DeepSeek folks have done an awesome job looking at, like, batch size warmup.
00:54:43.000 | And that's been really, really fruitful for them.
00:54:45.000 | You know, other people are looking really hard at things like datamix.
00:54:48.000 | But it just gets tricky to look at everything.
00:54:51.000 | Yeah, I think we've found that, like, we could get some things that looked like gains from data sets.
00:54:56.000 | But one of the things that I like about carbs is that when we applied carbs to, like, properly tune things, then a lot of those kind of evaporated.
00:55:05.000 | Whereas, like, if we just tune these other parameters, actually, we can get almost the same gains without having to do this more complicated thing.
00:55:12.000 | So, at least in the experiment, in the settings that we've, like, in the particular metrics that we care about, we haven't seen these kind of, like, pan out or scale up in quite the same way, but not to rule it out.
00:55:22.000 | And I think you're right, Jonathan, that there probably are a lot of, like, details that go into, like, exactly what is the metric, exactly what is the data set, exactly which, like, what schedule are we using for this.
00:55:32.000 | And I certainly wouldn't rule it out working.
00:55:35.000 | Quick question about emergence.
00:55:38.000 | Doesn't emergence throw a spanner into the theory of carbs?
00:55:43.000 | Ah, so there is a paper of which I really liked and I think informed a little bit of how we thought about this, which is "Are Emergent Properties of Language Models a Mirage?"
00:55:56.000 | And I think if you look at that paper, it actually makes a relatively compelling case that, in fact, you know, this emergent behavior that you're seeing is not really emergent behavior, but is really a function of the evaluation metrics that we're using.
00:56:10.000 | So, if you look at accuracy as a metric, what's happening is that accuracy is actually going up continually over training, but it's in log scale.
00:56:17.000 | So it starts out at 0.001%, 0.1, 0.1, 10.
00:56:22.000 | Only when you're going between 10 and 90 do you see this happen, right?
00:56:26.000 | When you go from 1 in, you know, 1,000 getting right to 1 in 1,000 getting wrong, like, there's many orders of magnitude happening here.
00:56:31.000 | So when you're looking at this in perplexity, then you just see this nice straight line.
00:56:35.000 | And so that's actually what carbs is exploiting.
00:56:37.000 | Like, since we're, since our metric is in this kind of, like, perplexity log space, like, you can see, like, oh, it's just, like, getting better as you make it bigger in this nice, very predictable way.
00:56:46.000 | So that, and that is exactly what we saw.
00:56:49.000 | Like, these things were really, really bad at, you know, predicting the multiple choice answer, just always guess A.
00:56:54.000 | Okay, it's so terrible at it.
00:56:56.000 | But it was like learning to be less confident about that.
00:56:59.000 | Yeah.
00:57:00.000 | One trick I saw from one of the papers recently was just, like, just randomize the order of the multiple choice questions.
00:57:06.000 | And if you, if that hits the performance a lot, then they're just basically memorizing the test set, which makes a lot of sense.
00:57:18.000 | Yeah, this is, I mean, you know, I completely agree with what Josh said.
00:57:24.000 | I think the, you know, my bigger lesson is that anything can look however you want it to look if you put it in a log scale to a certain extent.
00:57:31.000 | And we love our log scales in deep learning for various reasons.
00:57:35.000 | Everything looks very clean on a log scale until everything looks very flat on a log scale.
00:57:40.000 | I don't know.
00:57:41.000 | I, like, log scales always mix me up.
00:57:44.000 | That's all I can say.
00:57:45.000 | Great.
00:57:46.000 | I think the last thing I was going to mention on Carves.
00:57:50.000 | Oh, well, I mean, let's just kind of go right into evals because I think that's going to be the sort of crowd favorite.
00:57:55.000 | So Carves, we already mentioned, you know, leans heavily on the sort of end evals that we would typically eval LMs on.
00:58:04.000 | Except that you had to make your own.
00:58:06.000 | There are a lot of documented problems with many of the common evals out there and you fixed all of them, it sounds like.
00:58:13.000 | I don't know about fixed all of them, but I think in the same way that we like to dig into the infrastructure and hardware and understand, like, what actually is going wrong?
00:58:23.000 | Like, what is the actual error on this machine with this GPU and why did that happen and how do we fix it?
00:58:28.000 | We take the same approach to the evaluations.
00:58:30.000 | So when we looked at the evaluations and actually looked at the data sets, you know, what we did is like, OK, if we're going to be, you know, evaluating natural language understanding and reasoning, like, let's look at all the data sets that are out there.
00:58:41.000 | Let's actually look at a bunch of the examples and say, like, is this a good data set that we should use for evaluation?
00:58:46.000 | That's kind of how we selected the evaluation data set that we had.
00:58:50.000 | And then when we looked at the actual examples in there, we noticed, like, a lot of these are very messy, like some of them messy to the point of, like, incoherence and some of the ones that we didn't choose.
00:58:59.000 | But even the ones that we chose, like, people tried pretty hard on these data sets.
00:59:03.000 | They did try and clean them, but there's just a lot of data points in there.
00:59:06.000 | And it's just easy to make mistakes, right?
00:59:08.000 | And so, you know, it's not that they have 100 people looking at every question.
00:59:11.000 | Like, that's just way too expensive.
00:59:13.000 | So you end up with questions that just don't make sense.
00:59:15.000 | Somebody didn't really see this.
00:59:16.000 | Somebody just clicked the wrong box for the answer.
00:59:19.000 | Or the question makes sense in your head when you write it.
00:59:22.000 | We've often seen this.
00:59:23.000 | It's not even like malice or incompetence.
00:59:24.000 | It's really just like, you know, you write this, you write it.
00:59:26.000 | You're like, this makes sense to me.
00:59:27.000 | You show it to another person, like, that makes sense.
00:59:29.000 | You show it to a third person, they're like, this makes no sense at all.
00:59:31.000 | It's because you're kind of, you know, using a different meaning of the word.
00:59:34.000 | And then when they say that, you're like, oh, wow, you're right.
00:59:36.000 | That is actually really confusing.
00:59:38.000 | It's easy for things to kind of make sense in our own head.
00:59:40.000 | So what we did for the evaluations is really dug into the details of each of these data sets and tried to ask, like, what makes a good question?
00:59:50.000 | What makes a good answer?
00:59:51.000 | Like, what does it mean for it to be ambiguous?
00:59:53.000 | We had a whole, like, we looked at lots of data, broke this down, asked lots of people about all these different questions to build a model of this and help us kind of clean these data sets.
01:00:02.000 | That was sort of one big piece of it.
01:00:04.000 | A second big piece was making sure that our data that we're training on is not data that we're testing on.
01:00:10.000 | So there we kind of took a step back and said, like, OK, well, let's just reproduce, you know, 500 to 1,000 examples for every single one of these data sets ourselves and just make sure that this data is definitely not in the, you know, the training set.
01:00:23.000 | So we did that.
01:00:24.000 | And then we're able to, like, now be confident about, like, our performance of our model and also performance of other open source and other closed source models.
01:00:33.000 | Yeah, there's a lot there.
01:00:35.000 | You had 11, I don't know how many data sets.
01:00:40.000 | I think so.
01:00:41.000 | One, two.
01:00:42.000 | Yeah.
01:00:43.000 | Anyone you want to call out in particular to dive deeper on?
01:00:46.000 | Some of these are very famous, like Hubble Swag, you know, Grand.
01:00:50.000 | Some are less famous, like Race.
01:00:52.000 | I don't know if...
01:00:53.000 | Race is a great data set.
01:00:55.000 | Yeah.
01:00:56.000 | Yeah.
01:00:57.000 | Just, you know, anything that's interesting you want to call out on specific data sets?
01:01:01.000 | I think there are a few asterisks in there.
01:01:04.000 | You know, definitely read the whole paper as you're looking at some of these.
01:01:07.000 | Like, the GSM8K one is a little bit weird.
01:01:10.000 | I think one that was kind of funny was, like, low performance on ethics from some of the more recent models.
01:01:15.000 | I think that was a little bit funny because the models, you know, I think there was a reaction to, like, oh, no, like, you know, the models are saying bad things.
01:01:22.000 | So they went way, way in the other direction.
01:01:23.000 | And now, like, on the ethics data set, it's always like, this is totally unethical, even though it's really fine.
01:01:29.000 | So they've just been tuned to, you know, make sure they don't make any PR disasters.
01:01:33.000 | I thought that was a little bit funny.
01:01:35.000 | Not to say that it's necessarily like a flaw of the model, but just kind of like, you know, political or tuning opinion.
01:01:41.000 | I think the main takeaway...
01:01:43.000 | To fix...
01:01:44.000 | Oh, I was just going to say the main takeaway from any of the, like, actual performance is, like, once you fix these ambiguous examples, a lot of these benchmarks are really saturated.
01:01:53.000 | Like, I think it's important to look at, like, you know, like, when you're talking about performance on ANLI or race or pool queue or something, like, what you're really talking about is, like, performance on questions that make no sense.
01:02:05.000 | Like, it's just like, did it guess the answer in this, like, really weird scenario?
01:02:09.000 | Like, those are the ones that are left.
01:02:11.000 | Like, when you look at the performance on the ones that actually make sense to everyone, all the models agree.
01:02:15.000 | We agree.
01:02:16.000 | Like, everyone's on the same page, which I think is kind of a really interesting result.
01:02:20.000 | The question then becomes, you know, what are the new, like, set of evals that would be like the next frontier that often embeds with it your idea of what reasoning is?
01:02:33.000 | Because obviously you're super interested in reasoning.
01:02:36.000 | And yeah, I mean, like, where does this, where does the state of evals go from here?
01:02:41.000 | This work and this blog post is talking mostly about the public evaluations and the things that we can release.
01:02:47.000 | We do have our own internal evaluations.
01:02:49.000 | For example, one of them that we are releasing is the code understanding evaluation, which is about predicting, you know, what will this variable be or asking questions about code, et cetera.
01:02:59.000 | And that is one of the early benchmarks that we made that we can release.
01:03:03.000 | We can partly release it because we can generate an almost infinite amount of this data because these are programmatically generated.
01:03:09.000 | And so, you know, we're not really worried about there being like corruption in the kind of the training or test set.
01:03:14.000 | So that makes it a little bit easier for us.
01:03:17.000 | But I think it's, you know, we have built other data sets as well that we can't release.
01:03:21.000 | Some of them, you know, for example, because they maybe use other open source code.
01:03:25.000 | And so we can't redistribute it necessarily.
01:03:27.000 | Other ones, because, you know, that's I think evaluations and data are like a core important part of, you know, the business.
01:03:34.000 | And I think we take evaluations very seriously and are spending a lot of effort in terms of like, what exactly do we make as part of the evaluation set?
01:03:41.000 | How do you evaluate these things?
01:03:42.000 | We've done a lot of other stuff, you know, since these evaluations.
01:03:45.000 | But I think a lot around like code understanding for us, since that's our main focus.
01:03:50.000 | And it's a nice like place to explore reasoning as well.
01:03:54.000 | It sounds like you talk a little bit about like code understanding as like sort of variable level, like sort of very micro context.
01:04:01.000 | Is there a sense of like larger code context as well?
01:04:04.000 | I don't know what I mean by that, by the way.
01:04:05.000 | It's mostly just like if I told the senior engineer to go look at a code base, they would understand at a broad level the architecture, but also the design decisions and be able to tell me that.
01:04:16.000 | I don't know if that's useful or not, but I mean, that's useful to me as someone who might be working with them.
01:04:21.000 | Yeah.
01:04:22.000 | This particular data set is like the more low level code understanding, like just literally what happens in this code.
01:04:28.000 | And this is mostly because, you know, this is part of the CARBS tuning metric, etc.
01:04:31.000 | Like we care about the low scale version of this as well.
01:04:34.000 | We want smaller scale models to be able to do something on this.
01:04:37.000 | And so that's kind of the focus for this.
01:04:38.000 | And hopefully this is more useful for other people.
01:04:41.000 | But yes, those other questions are also quite interesting.
01:04:44.000 | They get a lot harder to evaluate, like, is this a good architecture or not?
01:04:48.000 | Like you and I could probably debate for a while on, you know, different architectures.
01:04:51.000 | And so it becomes a lot trickier to do these evaluations as they become more realistic.
01:04:56.000 | And I think that's one of the things that we've been playing around with a lot, especially around like code generation.
01:05:01.000 | So if you're saying, you know, implement this function, OK, it can be kind of objective.
01:05:05.000 | But, you know, even MBPP, we've made our own internal version of this data set, right?
01:05:10.000 | Where we've taken like every single example and looked at it and then like, does this actually make sense?
01:05:14.000 | Like, what is the type signature?
01:05:15.000 | Like, can we, you know, remove all ambiguity, etc.
01:05:18.000 | So you basically like reviewed every single question on, I mean, that's impossible for like HelloSwag, right?
01:05:23.000 | Yeah, yeah, we didn't do that for HelloSwag, but this is for MBPP, which is only like a few hundred.
01:05:28.000 | So we just sat down and did it.
01:05:30.000 | Yeah.
01:05:31.000 | I'm so excited to get to look at this data set.
01:05:33.000 | Like, this is such a resource for the community.
01:05:37.000 | I absolutely can't wait.
01:05:38.000 | We should probably do the, I don't know.
01:05:40.000 | I don't know if we were planning on doing the healed MBPP one, but hopefully we can do that one in the future.
01:05:44.000 | Did you look at SweetBench?
01:05:46.000 | It's the sort of hot new data set of the summer.
01:05:48.000 | Yeah, I've taken a quick look at SweetBench.
01:05:50.000 | It's really interesting.
01:05:51.000 | I like that it's a much more difficult kind of coding, code related task for bug fixing.
01:05:57.000 | I think it gets into some of these problems where it is a lot harder to evaluate these things once they get more realistic.
01:06:04.000 | Like we were looking at the AgentBench paper, I think, just last week for our paper club.
01:06:09.000 | And one of the things that we noticed is that actually like both of the examples in the appendix that are given as like traces where it got it right.
01:06:16.000 | This is actually not the right solution.
01:06:18.000 | And it's okay.
01:06:21.000 | You know, it's fine.
01:06:22.000 | Like it did make it past the test.
01:06:24.000 | That's what the metric is.
01:06:25.000 | That's what the benchmark is about.
01:06:26.000 | Right.
01:06:27.000 | But like it just said, you know, like, you know, .encode ASCII.
01:06:30.000 | Like, well, that's not the right way to do this.
01:06:32.000 | Like it just dropped all the other edge cases that you actually would have cared about in production for this thing.
01:06:37.000 | And there is like a better way of doing it.
01:06:39.000 | And, you know, that's what the real golden patch was.
01:06:41.000 | But, you know, that's okay.
01:06:42.000 | But then how do you test all of that?
01:06:44.000 | Like as you start to do more realistic things, the test coverage, like getting test coverage over all possible ways of solving these bugs is really hard.
01:06:51.000 | Evaluation is the single hardest part of the whole thing.
01:06:53.000 | Like I spend a shocking amount of time just telling our customers, we need to find a way to measure what you actually want out of the model before you should ever touch a TPU.
01:07:04.000 | And, you know, trying to convince my team and me to follow our own advice a lot of the time on that.
01:07:09.000 | And I think everybody like on the one hand, it's easy to laugh at the state of the evaluations that we have.
01:07:14.000 | None of them are good.
01:07:15.000 | Like if you go read these eval benchmarks, you'll always come away disappointed.
01:07:19.000 | And yet they've given us useful hills to climb.
01:07:22.000 | And we do seem to be making progress and measuring progress in the field.
01:07:25.000 | And I think anecdotally models are getting better year to year.
01:07:28.000 | So I feel like people tend to go and get into one situation or the other, like evals don't matter.
01:07:33.000 | I'm just going to look at loss or like, you know, the evals matter a lot and they're all broken.
01:07:38.000 | So what do I do?
01:07:39.000 | And I think like a lot of things in deep learning, we have to make peace with just complete imperfection.
01:07:45.000 | Like the most successful scientists I see are the ones who are okay operating in a world where everything's going to be broken.
01:07:51.000 | And yet we can still cobble things together and make something interesting happen.
01:07:54.000 | I mean, we were just discussing that with literal infrastructure.
01:07:57.000 | And now we're all the way up to like, how do we measure whether a model performed a complex coding task correctly and everything is broken.
01:08:04.000 | And yet we're still able to make huge amounts of forward progress.
01:08:07.000 | I think that's right, Jonathan.
01:08:09.000 | And that the challenge isn't necessarily making perfect evaluations.
01:08:13.000 | I think our blog post here is about going really into the weeds on these to figure out like, what does that look like?
01:08:18.000 | And I think one thing is like, you know, as you said, we have been able to make a lot of progress without making these perfect.
01:08:23.000 | That's great.
01:08:24.000 | You don't have to have perfect evaluations.
01:08:26.000 | And, you know, the more interesting work is the stuff that we can't necessarily publish about, which is the imperfect evaluations that we have for, you know, actual coding tasks, for example.
01:08:36.000 | Like, what does this really mean as a person?
01:08:38.000 | And there, as you said, it's much messier.
01:08:40.000 | So it's a lot harder to put it out and say like, hey, everybody use this because there's so many rough edges.
01:08:45.000 | It's so hard to like, even say, oh, is this even the right task?
01:08:48.000 | Is this even the right way to do it?
01:08:50.000 | And there's a lot of judgment.
01:08:51.000 | There's a lot of intuition that it comes down to.
01:08:53.000 | But yeah, I think that's where it's critical to do if you actually want to make these systems work.
01:08:59.000 | You have to make peace with living in that in between.
01:09:03.000 | And I think that in some sense, when I hire researchers, that's the number one quality I look for.
01:09:07.000 | Like, can they be at peace living in a house that is neither clean nor messy, but it's just kind of somewhere in between?
01:09:13.000 | And are they okay with that?
01:09:15.000 | Are they okay with a few dishes being out on the table and a few clothes being on the floor?
01:09:19.000 | Or will that drive them insane?
01:09:21.000 | Or will they just end up with all the clothes on the floor and like all the dishes out all the time?
01:09:24.000 | Like, it's kind of, I'm looking for that perfect balance because, you know, we have to operate in this imperfect world.
01:09:31.000 | Like, yeah, go ahead and give me the perfect evaluation for programmers or for an LLM that is a program assistant tool.
01:09:37.000 | Like, there is no perfect evaluation, but clearly we've made progress.
01:09:42.000 | And so the most important part is just, are we climbing the right hills?
01:09:45.000 | And so this is why I'm so excited to see the ambiguity aspect of this.
01:09:48.000 | We often think we have more room to climb on these benchmarks and it turns out we don't.
01:09:52.000 | Or it turns out that actually we're climbing, getting good at the benchmark and not actually getting good at the task we care about underlying the benchmark anymore.
01:09:59.000 | Maybe the model, like this is the famous example where if you get a hundred percent on MNIST, your model must be broken in some way because there are four examples mislabeled.
01:10:07.000 | You know, it's, it's that all over again.
01:10:11.000 | Welcome to this.
01:10:12.000 | Yeah.
01:10:13.000 | It's the accidental canary in this.
01:10:16.000 | I think one thing that's actually really interesting about this also is that, yes, like the ambiguous examples are sort of, you know, not that great from the perspective of these particular tasks that we're evaluating.
01:10:27.000 | But actually one thing that we're very interested in is ambiguity itself.
01:10:30.000 | Like, can we detect whether a task from a user is ambiguous or whether you've, you know, completed a task successfully?
01:10:36.000 | Like these are actually hard, messy problems, but are really important from like the user experience of using these models.
01:10:42.000 | I would much rather have a coding agent that will give me back a thing.
01:10:46.000 | And, you know, it's, it's actually the code doesn't work like 10% less of the time than some other model, but it will tell me a hundred percent of the time when it got, like when it's not sure.
01:10:55.000 | Like that's so much more useful if it can communicate, like, I'm not really sure about this, or maybe there's some errors here.
01:11:00.000 | Then just like, here's some code.
01:11:01.000 | I have no idea if it works.
01:11:03.000 | And so these kinds of like, you know, detecting ambiguity and detecting correctness or uncertainty, I think are really interesting problems that we're really like digging into quite deeply.
01:11:11.000 | I'm going to touch on maybe a couple of hot topics in evals, maybe tangentially related, but we're on the evals train right now.
01:11:18.000 | So I'm just going to get on that.
01:11:21.000 | So ArcAGI, Francois Chollet's hot new thing.
01:11:27.000 | It's sort of my take on it is basically it's trying to measure reasoning through an abstract IQ test effectively.
01:11:34.000 | I noticed that you don't use it.
01:11:37.000 | There's a lot of community debate, pro and con about it.
01:11:40.000 | What are your thoughts on just more abstract reasoning and maybe ArcAGI specifically?
01:11:44.000 | I think we purposely stayed away from the very abstract, like there's Big Bench, for example, that has a lot of, I think, kind of, to me feels sort of similar types of tasks that are like very unrealistic.
01:11:58.000 | Like, oh, you know, we have books of different colors, and then you're going to shuffle them and like, which book is furthest to the left or something like, okay, cool.
01:12:04.000 | I guess it's neat.
01:12:05.000 | It's neat, I think, for us to explore in terms of like an agent reasoning in a larger loop.
01:12:11.000 | And we do care about these types of evaluations there.
01:12:14.000 | The types of evaluations we're talking about in the blog post here are for getting at like, does this model in a base model sense, is this working at all?
01:12:22.000 | Like there's no chain of thought in these evaluations.
01:12:24.000 | These are just like, go straight to the answer.
01:12:26.000 | Does this make sense?
01:12:27.000 | Like, is this a thing that you can answer very quickly?
01:12:29.000 | That's what we were selecting for with these evaluations.
01:12:31.000 | This is not to say that these are the only evaluations we have.
01:12:34.000 | I think the ARC ones are like a little bit too probably visual for us to really be able to integrate with.
01:12:41.000 | But I think some of the big bench ones are.
01:12:43.000 | You can tokenize it.
01:12:44.000 | Yeah, but you know, I think it's, it's not really, I think you can spend a lot of time getting really good at these kinds of benchmarks without making like, kind of more general purpose progress.
01:12:54.000 | So I think we're a little bit leery of going too far in that direction.
01:12:57.000 | Similarly, like coding competitions, like we do a lot of code generation, but we don't really do a lot on like code competition problems for the very, very hard ones.
01:13:06.000 | So I think you can go very far down that route and make something that's like really good at those problems, but not actually that useful as like a programmer day to day.
01:13:13.000 | Yeah.
01:13:14.000 | Take a different tactic, which is like at the end of the day at Databricks, I have 12,000 customers, or I think that's the latest number.
01:13:23.000 | All of whom are trying to do something with, you know, LLMs or AI or machine learning.
01:13:29.000 | And those things don't look like these tasks.
01:13:33.000 | I don't think I have a single customer that's asking to, you know, have AI solve abstract reasoning problems.
01:13:39.000 | Things are pretty, like they can be ambiguous, they can be challenging, they can be really interesting, but none of them look quite like this.
01:13:46.000 | And so, you know, I think to Josh's point, like it's really about asking, why are we doing this?
01:13:51.000 | Even if you're trying to build AGI, and that's not personally my purpose.
01:13:56.000 | And I, you know, Josh has much more interesting things to say about that than I do.
01:14:00.000 | I don't even know if this is the kind of intelligence I would get excited about or care about personally.
01:14:04.000 | Or if I would consider, you know, to Josh's point, this to be the indicia of intelligence.
01:14:08.000 | It's neat.
01:14:10.000 | But, you know, for me, it's like more down to earth things like having a model that can have a conversation with you about data.
01:14:17.000 | That on the backend is running SQL queries on your literal data.
01:14:20.000 | That's a much more interesting task to me.
01:14:22.000 | That's something that really matters day to day for my customers and, you know, different perspectives.
01:14:27.000 | But, you know, I think Josh and I would probably say the same thing, even though I would, I'm guessing I don't want to put words in your mouth.
01:14:34.000 | You would say that you're pursuing more general intelligence in your own way.
01:14:37.000 | And I would say that I'm very happy with narrow intelligence.
01:14:40.000 | Like I'm very happy with my little SQL bot and building 12,000 of those because they're moving the needle for a lot of folks every day.
01:14:47.000 | Yeah, I think we're, you know, we're not as far away in our position as it might seem.
01:14:52.000 | I think we're also excited about like, how do you actually make these things useful?
01:14:56.000 | And that does end up being pretty narrow.
01:14:58.000 | I think these other tasks can be interesting as like ways to explore these more abstract reasoning questions or like, okay, how could an agent actually work through this?
01:15:06.000 | But it's important to keep in mind that it's like a toy, not a real problem.
01:15:09.000 | It's like, it's a scientific tool to tell us something about the models.
01:15:13.000 | It's not something we should be optimizing for necessarily.
01:15:16.000 | The one thing I'll point out is, you know, as a kid, I was graded into a gifted program based on my ability to solve these exact type of problems.
01:15:23.000 | And then I entered college based on my ability to solve SATs, which again, have nothing to do with my college experience, but whatever.
01:15:32.000 | So, you know, we have a history in humanity of doing correlated IQ tests to general capability.
01:15:39.000 | Okay, so the two more viral evals, and then, you know, I just want to be mindful of your time.
01:15:45.000 | Needle in a haystack, long context utilization.
01:15:48.000 | Oh, for the love of God.
01:15:49.000 | Something, well, okay.
01:15:51.000 | Like, let's just assume that, you know, on our podcast, we've discussed the, you know, baseline problems of needle in a haystack, but just generally long context, right?
01:15:59.000 | It's a useful thing for agents.
01:16:01.000 | I assume, and it's something that, you know, it's out there.
01:16:06.000 | Like, we don't know, don't really know what the best way to utilize memory is, but like, I assume it's important, right?
01:16:13.000 | What I'll say is like, you know, I spend a lot of time thinking about RAG these days.
01:16:16.000 | And RAG, you know, in one sense, you know, the way that I think about RAG is it's the world's simplest agent.
01:16:22.000 | It is an agent that basically, you know, there's at least more than one thing happening in the process of building a model.
01:16:27.000 | It's at least a system.
01:16:28.000 | If you give the model the ability to decide when it wants to retrieve data from a context or retrieve data from a database, then we're talking about an agent.
01:16:35.000 | So RAG kind of, I think, like, tows that boundary really nicely.
01:16:39.000 | There are a lot of reasons why you do genuinely need a long context.
01:16:42.000 | Like, I don't think long contexts are problematic in and of themselves.
01:16:45.000 | I know there's some controversy even about that.
01:16:47.000 | I love the idea of doing like thousand shot tasks as an alternative to fine tuning.
01:16:52.000 | I love the idea of pulling in lots of data into the context.
01:16:54.000 | I love the idea of once you get into multimodal land, you're just going to end up with giant context.
01:16:58.000 | It's kind of unavoidable.
01:17:00.000 | The flip side is I don't know of anyone who, like, is hiding a secret passphrase in a book and needs the model to find it.
01:17:07.000 | Needle in a haystack is, it's interesting.
01:17:10.000 | The challenge with long context, to my mind, and Josh, I'm curious what you think, is simply that annotating long context evals is really hard and really expensive, you know, intrinsically.
01:17:20.000 | Because you need someone to read 10,000 tokens or 100,000 tokens or, like, you need someone to read a 1,000 page book or the equivalent thereof in order to measure these long context benchmarks.
01:17:30.000 | I don't know if a human could solve these tasks, let alone that a human could do this in any amount of time where you're willing to pay the money to get the data annotated.
01:17:38.000 | And so any long context eval has to, in some sense, be correct by construction.
01:17:42.000 | And you have to, you know, you have to know the answer before you've created the example.
01:17:46.000 | And needle in a haystack is kind of the simplest way of doing that.
01:17:49.000 | I think the problems with needle in a haystack are well known, you know, it doesn't measure anything real.
01:17:53.000 | You're not even testing the model's ability to holistically use the context just to identify one part of the context.
01:17:58.000 | So you can do some wacky things to your model, like quantize the hell out of the KV cache and still get needle in a haystack to work quite well because it's not trying to holistically take advantage of things.
01:18:08.000 | You know, I have some thoughts on things that I like more that are also still correct by construction.
01:18:15.000 | Like I really like the idea of doing thousand shot tasks where you can look at the scaling as you go from 10 shot to 100 shot to 1000 shot to fine tuning on that data instead.
01:18:24.000 | And I like that as a way to, you know, have something that's correct by construction, at least where you have a nice baseline that you can compare to automatically.
01:18:32.000 | So I'm typically looking for like contexts that are situations where long context is one way to solve the task, but not the only way to solve the task.
01:18:41.000 | And we have some other strong baseline floating around personally.
01:18:44.000 | But yeah, needle in a haystack, not my favorite thing in the world, to say the least.
01:18:48.000 | Yeah, I mean, I agree with most of what Jonathan said, I think.
01:18:52.000 | I think one other thing that I will call it is that, you know, from like a coding application perspective, it's useful to have long context because the lazy thing of just like throw the whole repo in the context is like, okay, cool.
01:19:03.000 | Like, you know, you can just get started with that.
01:19:05.000 | But then in, you know, in real scenarios, you don't necessarily want to put the whole thing in there.
01:19:10.000 | You can have code bases that are bigger.
01:19:11.000 | You probably want to filter down to the stuff that's relevant anyway to not be confusing.
01:19:14.000 | Like you probably, even if you did have a lot of context, you might want to sort it in some way to say this is more important than this other stuff.
01:19:19.000 | So, and you know, you don't want to wait for, you don't want to be wasting all this time and compute on inference and like doesn't really matter.
01:19:25.000 | So, yeah, I don't know that it's the most important thing.
01:19:30.000 | I think people will find creative use cases and like John said, I think the multimodality examples will naturally lend themselves to long context.
01:19:38.000 | Cool.
01:19:39.000 | And then one last one on just general sort of agent related capabilities that we didn't really talk about in eval section is function calling and tool use.
01:19:48.000 | There's a recent trend, I think, basically led again by OpenAI on parallel function calling.
01:19:55.000 | There's always, there's been a limit on how many tools you can call from four to now, I think 128.
01:20:03.000 | And I think theoretically Claude and Jem and I support a lot more.
01:20:07.000 | So just generally, how do you think about evaling tool use?
01:20:11.000 | Is that super important for you guys?
01:20:13.000 | Anything else?
01:20:14.000 | I think we're thinking about it in a slightly different way, which is, yes, you can have this like hard coded list of tools.
01:20:19.000 | But if only you could have like this really large open set of like tools, maybe they would be like functions that you could call.
01:20:25.000 | If only there was like a language or like a programming thing, like being able to write code.
01:20:30.000 | I think for us, it's like, well, look, if we can write code, like now you have all these tools accessible at the end of the day, like function calling is just a function invocation, like literally in code.
01:20:39.000 | I think our approach to this is like, instead of worrying about like weird, hard coded agents using tools, like let's just make them able to actually write code robustly and make that code work and be able to debug that code, know if that code is safe to run.
01:20:51.000 | Like get really good at the like code writing and execution part of things, because that will open up the action space like far more than, you know, 128 tools.
01:20:59.000 | Like just everything is at your fingertips, especially I think over the next few years, like we already have so many really good APIs.
01:21:05.000 | As we get better and better at writing code, we'll be able to make APIs to things that don't even have APIs today.
01:21:10.000 | I think that's, that's kind of how we think about it is less as like a special purpose thing and more as like, this is one of the reasons to focus on code.
01:21:16.000 | On my end, the way that I think about this is, you know, I think a lot about how models interact with data.
01:21:22.000 | And so for me, tool use is really a question of how do you take models that are really built for unstructured data and have them interact with structured data.
01:21:31.000 | So, you know, I, and I get the question a lot from my customers, like, what do I do with tabular data?
01:21:36.000 | Or what do I do with like, you know, JSON or what do I do?
01:21:40.000 | I mean, you name it.
01:21:41.000 | Like, even what do I do with a PDF?
01:21:43.000 | Because PDF parsing is still an unsolved problem, even in 2024.
01:21:48.000 | And the answer, or even just the basic question of like, should I bother to structure my data anymore?
01:21:52.000 | Shouldn't I just toss the table?
01:21:54.000 | Shouldn't I flatten it and just throw it into the LLM context and like let the model figure it out?
01:21:58.000 | The answer is no.
01:21:59.000 | Like, we've built all these fun APIs and fun languages and paradigms for dealing with structured data over the years.
01:22:07.000 | Just use them.
01:22:08.000 | Have your model use them.
01:22:09.000 | Train a model that can interact with these things in a meaningful way.
01:22:12.000 | Like, it's, you know, text to SQL is still, or like having a model be able to make SQL calls in the backend is actually like one of the single most useful things for my customers.
01:22:24.000 | It sounds really boring.
01:22:26.000 | But the models are really good at it and it moves the needle day to day.
01:22:30.000 | So tool use for me really is that.
01:22:32.000 | Like, how do you just interact with structured data sources and take advantage of the fact that you have some prior knowledge about the structure of your data that an LLM would completely flatten away?
01:22:44.000 | You know, in many ways, this is kind of one of the, one of my biggest frustrations with the fact that LLMs work well with code.
01:22:51.000 | We have decades and decades and decades of understanding about the structure and interpretation of programs.
01:22:55.000 | Like, I think that's literally the name of a book on programming, if I remember right.
01:22:59.000 | And like, we, you know, we have all this theory.
01:23:03.000 | We know everything there is to know about programming languages, if they're well-formed languages and have the right properties.
01:23:08.000 | And yet when we have an LLM work with them, we literally just turn it into a token stream.
01:23:12.000 | Despite the fact that we know how to parse it.
01:23:14.000 | We know, you know, how to do all sorts of, you know, reference, you know, disambiguation and things like that.
01:23:20.000 | We're still just flattening it into a model and making the model relearn all of these things from scratch.
01:23:25.000 | And it frustrates the hell out of me.
01:23:27.000 | I don't have a better answer when it comes to code.
01:23:29.000 | But I really appreciate that with a lot of data sources that have structure to them.
01:23:33.000 | Tool uses and function calling are just, in my mind, the right way to deal with this.
01:23:38.000 | So I think basically what you're saying is like code is the God tool for Jonathan.
01:23:42.000 | Like, you know, SQL is so much the right abstraction for accessing all this data.
01:23:49.000 | One thing I do spend a lot of time thinking about is, you know, for the stuff that doesn't fit in a SQL table.
01:23:55.000 | You know, is Knowledge Graph the answer?
01:23:58.000 | I think a lot of people are exploring that.
01:24:01.000 | And I think every now and then people get a bout of Knowledge Graph religion and then it kind of doesn't work out.
01:24:06.000 | So I wonder what that, you know, I wonder what the end state is.
01:24:11.000 | Like, you know, is this an idea where like it's a mirage or is this the idea where sometime it's going to work?
01:24:19.000 | It's about like having the right tools for the problems, right?
01:24:22.000 | Like as Jonathan was saying, SQL is sometimes definitely the right tool.
01:24:25.000 | Like you've got your, you know, order table or something and you want to know, you know, number of sales last month.
01:24:29.000 | Like you should be using SQL. Sum that column. Okay, great. You're all set.
01:24:33.000 | Knowledge Graphs also, you know, are sometimes the right tool for a particular problem.
01:24:37.000 | You have some like weird question about relationships between entities that are modeled on some particular ontology
01:24:42.000 | that you actually understand and is like maps to the real world.
01:24:45.000 | Great, you know, use a Knowledge Base, like use a Knowledge Graph. This is fine.
01:24:49.000 | But I think in the real world, it gets a lot messier than like Knowledge Graph style of things where it's like,
01:24:56.000 | well, is there a relationship between these two nodes? Like, I don't know.
01:24:59.000 | Like, are these two separate nodes?
01:25:01.000 | Like those kind of messy borders, I think, prevent it from being a tool that can like solve everything forever.
01:25:06.000 | And so I think it'll always be good for certain problems, just like SQL is good for certain problems.
01:25:10.000 | Like different abstractions are good for different problems.
01:25:12.000 | And, you know, yeah, I think this is why I'm excited about code.
01:25:15.000 | Like code lets you kind of pick the right, like, let's use this library for this problem.
01:25:18.000 | Let's use this library for this other problem.
01:25:20.000 | You know, like code is, I think, you know, Josh said it and you said it well, like code is kind of the God tool.
01:25:27.000 | It unlocks literally everything.
01:25:29.000 | The challenge for me is always like, you know, sometimes unlocking too much power can, sometimes inconvenient things can happen.
01:25:37.000 | And so it's all about balancing that.
01:25:39.000 | In some sense, language is the God tool.
01:25:41.000 | If only, you know, we knew how to, you know, we knew how to interpret it all the time.
01:25:45.000 | So code is, has the really nice property that at least you can always execute it.
01:25:49.000 | You know, and sometimes you just literally want your model to be able to do SQL calls and nothing else.
01:25:54.000 | And setting those boundaries properly for the problem, I think is going to be, I think at least a lot of my customers are going to be thinking very hard about that.
01:26:01.000 | Like, should I give the model access to the web?
01:26:03.000 | Is that actually helpful for this problem?
01:26:05.000 | It sounds great to just like flip yes on all the tools.
01:26:07.000 | Is that actually going to mean I'm going to get better solutions to my problems?
01:26:10.000 | So I want to be mindful of time.
01:26:12.000 | I think that's, you know, basically our sort of recap of our discussion based on MBS releases today.
01:26:18.000 | I wanted to leave some time for what's next for both of you guys.
01:26:22.000 | Maybe, you know, Josh, as a guest of honor, you want to go first as to like what happens next.
01:26:29.000 | You know, we have these releases.
01:26:30.000 | We're happy to put these things out.
01:26:31.000 | I think there's a lot of stuff that we haven't released.
01:26:33.000 | Like this is not the only thing we've been working on.
01:26:35.000 | Most of our actual focus has been on kind of coding and reasoning.
01:26:39.000 | In particular, like the things that we're excited about are can we make these things useful?
01:26:43.000 | Like Jonathan is saying, right?
01:26:44.000 | Like it's not about toy problems.
01:26:46.000 | It's like, can we use these today in our day-to-day workflow and actually have them accelerate us?
01:26:50.000 | And I think we have some kind of internal product prototypes and things that we're excited about.
01:26:54.000 | And so we're excited to share more about this in the coming, you know, months to quarters
01:26:58.000 | as we get it to a place where like other people could maybe get value out of this as well.
01:27:02.000 | But that's kind of our real focus right now is like,
01:27:05.000 | how do you take these really cool capabilities that are out there that our models have, et cetera,
01:27:09.000 | and like make sure that they're actually useful today for us, like when we're doing real work,
01:27:14.000 | and then for other people as well.
01:27:16.000 | In particular, focused on kind of generating code, understanding code, testing code,
01:27:21.000 | verifying it, like starting with the like robust, you know, creation of software.
01:27:27.000 | Excellent.
01:27:28.000 | Jonathan?
01:27:29.000 | I mean, you know, I never like to talk too much about the future because, you know,
01:27:33.000 | I don't know, I think you've heard this from me before.
01:27:35.000 | I like for us to speak through our work.
01:27:37.000 | And so I don't, you know, I don't like to tease too much.
01:27:40.000 | But, you know, I think we're, you know, what's the right way of putting it?
01:27:46.000 | I mean, you know, our mission is, I think to Josh's point, to make this stuff useful
01:27:51.000 | to 12,000 customers and, you know, not a lot of that ends up making it into the public eye
01:27:56.000 | and not a lot of that ends up getting released open source.
01:27:58.000 | So for this kind of forum where really, you know, where we're talking to the community,
01:28:03.000 | you know, I'm asking myself right now, like, you know,
01:28:05.000 | what exciting things are we going to have to offer the community in the next little while?
01:28:08.000 | I think the most exciting part is just we're writing a lot of blog posts right now.
01:28:13.000 | We're trying to share more and more of our science because I feel like we've been doing
01:28:16.000 | these big pushes to create these really giant models.
01:28:19.000 | I think, Josh, I'm sure you had the same experience.
01:28:21.000 | It's exhausting and all consuming.
01:28:23.000 | And you get to the end and you're like, oh, I have all this stuff I want to talk about.
01:28:26.000 | Now I need to find the time to talk about it now that I've survived this huge push.
01:28:30.000 | And we're definitely in that mode right now.
01:28:32.000 | So there's going to be a lot of that coming in in the next little while.
01:28:37.000 | And, you know, we're always cooking up fun new models.
01:28:39.000 | I think the real question is, you know, releasing models open source is not our day to day bread and butter.
01:28:44.000 | It's kind of a fun reward that we get to do sometimes when we have something really cool to share
01:28:48.000 | and a little bit of time and spare GPUs in our hands.
01:28:50.000 | But for the most part, everything is going toward customers.
01:28:53.000 | You know, Databricks is, you know, I think the joke is Databricks has been 18 months away from IPO for five years.
01:28:59.000 | But, you know, so I guess Databricks is 18 months away from IPO still.
01:29:02.000 | But 18 months away from IPO means there's a lot of pressure to deliver for customers.
01:29:06.000 | And we're going to keep working on that.
01:29:08.000 | But I think you'll see hopefully some cool, interesting things get dropped over the course of the summer and into the fall.
01:29:14.000 | You know, we'll, we'll find out when we get there.
01:29:18.000 | I think that's the right way to put it.
01:29:19.000 | I know we were talking earlier about kind of Abra, Kadabra and Alakazam.
01:29:22.000 | And, you know, all I'll say is that, you know, the DBRX small model that we still haven't released yet was called Abra.
01:29:29.000 | DBRX was called Kadabra.
01:29:31.000 | And, you know, there's a third Pokemon in that evolution.
01:29:34.000 | And that's all I'll say for now.
01:29:37.000 | Cool stuff kind of popping up sometimes on chatbot arena.
01:29:39.000 | And, you know, keep your eyes out.
01:29:41.000 | Yeah.
01:29:42.000 | I'll leave the links and the hints in the show notes.
01:29:47.000 | But it was a very fun way to leave some breadcrumbs for people to follow.
01:29:52.000 | Cool.
01:29:53.000 | You know, I'll leave everything to sort of some calls to action.
01:29:57.000 | We're going to be releasing this next week.
01:30:00.000 | So I will be deep in my conference, the AI Engineer World's Fair.
01:30:04.000 | So people can just go to AI.Engineer and live stream it.
01:30:08.000 | Do you guys have any other calls to action before we wrap?
01:30:11.000 | The only one is, you know, we're definitely hiring.
01:30:13.000 | So if you're interested in working on coding, reasoning, interested in working on all this stuff, you know, from the ground up and really deeply understanding not just how does the hardware work, but how do the models work.
01:30:23.000 | And also designing these systems to actually be useful for yourself day to day.
01:30:27.000 | Like, come say hi.
01:30:29.000 | I think the only thing I'll say is, you know, and I like saying it these days, it feels like the field is so crowded and, you know, it requires so many resources to do impactful work.
01:30:38.000 | And, you know, almost on some days it feels like everything's been done or somebody else is doing everything where you can.
01:30:43.000 | At least I remember that feeling every single day of my PhD and even more so now.
01:30:48.000 | But I hope like what you heard from Josh today tells you there is so much enormously impactful work to do in the field.
01:30:55.000 | If only you take a step back and take a fresh look at some of these things and just talk about what you're doing.
01:31:01.000 | There's a huge amount left to do here and a huge amount of exciting work happening every day.
01:31:06.000 | And, you know, for those who are certainly feeling that exhaustion right now, and I count myself among those folks many days, it's refreshing to see these kinds of drops.
01:31:17.000 | And, you know, see that there is so much more, even in things that people feel like they understand, how to set up a cluster.
01:31:24.000 | My God.
01:31:25.000 | You know, even in these evals that we think we understand, there is still more to understand and still more work to do.
01:31:31.000 | And so, you know, just I hope everybody's keeping at it.
01:31:35.000 | All right. Keep on keeping on.
01:31:36.000 | Well, thanks so much for your time, guys.
01:31:38.000 | That was great discussion.
01:31:39.000 | And we'll put the links in the show notes for people to read more.
01:31:43.000 | Thanks.
01:31:44.000 | Thanks so much.
01:31:45.000 | Thanks, Jonathan.
01:31:47.000 | [Music]
01:32:03.000 | you