Substrate Launch: the API for modular AI
00:00:00.000 | Yeah, it's really good to be here. This is a particularly exciting talk for us because
00:00:18.000 | we've been working with private clients for about a year now, but this is the first time
00:00:22.360 | we've really talked about it in public since our launch last week. I'm incredibly proud
00:00:29.680 | of the work we've done so far and excited to take a few minutes to tell you about it.
00:00:34.460 | So if you look at the products out there that have really successfully leveraged this generation
00:00:41.380 | of AI, I think one thing is true about nearly all of them is that they're using more than
00:00:47.440 | one inference runs, often many different types of models in tandem to accomplish a specific
00:00:53.080 | kind of task really well. And I think people really quickly realized that the foundation
00:00:59.580 | model is not enough. And even very simple tasks like summarizing a document to much, much more
00:01:07.540 | complex tasks like solving coding problems end to end, I think the best products right
00:01:12.300 | now are all using systems of inference runs in a logical structure. So I think at Substrate,
00:01:19.300 | we believe that building with modular intelligence is always going to be more effective than building
00:01:27.000 | with a monolithic intelligence. These systems are inherently more legible, which means you
00:01:33.300 | can understand them structurally, which means that they're debuggable and they're extensible.
00:01:38.500 | And evals become a lot easier because the decision trees are explicit and you can sort
00:01:44.140 | of verify at every step what's going on and what's going wrong. So Substrate, I think, is sort
00:01:53.140 | of a new way, new approach to this. I think our model is sort of fast in ways that other paradigms
00:02:00.320 | can't be. It's sort of flexible enough to build any AI product out there. And it works to scale
00:02:05.960 | by default. So what is it? I think at its core, Substrate is a coupling of two things. First,
00:02:14.960 | I think it's a really elegant developer SDK that lets you describe a computation graph over any number of nodes.
00:02:23.540 | And the abstractions here are really general and so we have a bunch of intelligence nodes across all
00:02:32.180 | the modalities that you might care about, which is like generating images, transcribing speech,
00:02:36.980 | generating text, JSON embeddings, executing code. But second, Substrate is also an inference engine
00:02:46.180 | specifically built to run these computation graphs as efficiently as possible. So these graph representations
00:02:53.620 | here are -- it's a representation of many tasks and their relationships. And since we run a very
00:03:01.380 | coordinated compute cluster, we can statically and dynamically optimize things like batching, caching,
00:03:09.700 | sort of networking, concurrency, physical placement, which really makes a big difference.
00:03:14.500 | And if you look at most frameworks out there, they're typically involving dispatching a bunch of API
00:03:21.460 | calls separately. And if you look at what happens mechanically when you do that, it's -- every step
00:03:27.300 | means you've got to resolve DNS, you've got to go through proxies, you've got it through authentication,
00:03:32.100 | like balance checks. And all of that sort of adds hundreds of milliseconds of latency on every single
00:03:41.700 | step. And if you contrast that with Substrate, we transfer data from node to node, process to process
00:03:47.860 | on the order of microseconds, which is some 10,000 times faster, meaning that it's actually feasible now
00:03:53.300 | to run online applications that involve dozens of nodes. We've also noticed that JSON decoding is
00:04:01.940 | is one of the most useful patterns for multi-inference runs. And I think we've invested a lot into
00:04:07.860 | offering a best-in-class JSON mode, both in terms of reliability and speed. And if you look at all of
00:04:17.060 | this together, I think what it means is that Substrate is really a way that -- a way to enable higher quality
00:04:24.820 | outcomes with AI, letting you work in a system that's more flexible, it's more legible, it's more
00:04:31.780 | verifiable than any of the current paradigms that sort of exist now.
00:04:37.700 | I think there's a lot more to say. There's all the time I really have today. It's only five minutes.
00:04:42.420 | But if you're curious, please come out and say hi on the expo floor. You can scan this QR code
00:04:51.700 | and get some credits and go to the website, substrate.run, or give me an email at rob@substrate.run.
00:04:59.620 | What's up, substrate.run.
00:05:07.540 | What's up, substrate.run.