AI Engineer World’s Fair 2024 - Keynotes & Multimodality track

Because I'm missing you I was chasing all the wrong signs Trying to hold on to something that I couldn't find Wish you didn't captivate my mind Now I know even the sun sets in paradise But now something went wrong You're moving on I found myself on the blind side Now you won't call We lost it all You fade away I'm picking up my heart From every piece that's broken Been trying to get back to myself But don't have a clue I'm looking for some luck Can't find a door that's open I'm losing all my hope Feels like I'm left here in two Because I'm missing you Because I'm missing you Oh, because I'm missing you Because I'm missing you Oh, because I'm missing you Oh, because I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Oh, I'm missing you Picking up my heart And every piece that's broken, been trying to get back to myself, but don't have a clue.

I'm looking for someone, can't find a door that's open. I'm losing all my hope, feels like I'm left here in two, because I miss you. And I'm ready to go. I'm falling right in and I'm ready to go. I found what I want and I know that we're on top, so I'll dive in and I'm ready to whoa-oh.

Holding my breath and I'm ready to go. I catch you laughing and I'm ready to go. You're holding a mask, strike it and ignite my soul. I'll dive in and I'm ready to whoa-oh. We are a summer storm, a feeling you can't ignore. Do you ever stop to feel it caught in the afterglow?

I'll come back to your door to know that you believe in summer all that I want. You know, it's time that we got it all. You and me now. Holding my breath and I'm ready to go. I'm falling right in and I'm ready to go. I found what I want and I know that we're on top, so I'll dive in and I'm ready to whoa-oh.

Holding my breath and I'm ready to go. I catch you laughing and I'm ready to go. You're holding a mask, strike it and ignite my soul. I dive in and I'm ready to whoa-oh. We'll light it up again. The sky and our silhouettes. Dancing on the pavement. Parting about the sky.

We'll light it up again. The sky and our silhouettes. Dancing on the pavement. Parting about the sky. We'll light it up again. The sky and our silhouettes. Dancing on the pavement. Parting about the sky. We'll light it up again. The sky and our silhouettes. Dancing on the pavement. Parting about the sky.

Good morning, ladies and gentlemen. We will start our day in the ballroom in 10 minutes. Thank you. Good morning, ladies and gentlemen. I'm ready to go. I'm falling right in and I'm ready to go. I found what I want and I know that we're on top, so I'll dive in and I'm ready to whoa-oh.

Hold some more breath and I'm ready to go. I'll catch you laughing and I'm ready to go. You're holding a mask, strike it and ignite my soul. I dive in and I'm ready to whoa-oh. If I find myself at your door. If I find myself at your door. Would you follow me to better places?

If I find myself at your door. Got the keys, let's go. I want to taste this. If you showed up at my door. I would follow you to better places. If you showed up. Ready, let's go. Ready, let's go. Oh, oh, oh, oh. Hold some more breath and I'm ready to go.

I'm falling right in and I'm ready to go. I found what I want and I know that we're on top, so I'll dive in and I'm ready to whoa-oh. Hold some more breath and I'm ready to go. I catch you laughing and I'm ready to go. You're holding a mask, strike it and ignite my soul.

I dive in and I'm ready to whoa-oh. Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go. I'm falling right in and I'm ready to go. I found what I want and I know that we're on top, so I'll dive in and I'm ready to whoa-oh.

Hold some more breath and I'm ready to go. I catch you laughing and I'm ready to go. You're holding a mask, strike it and ignite my soul. I dive in and I'm ready to whoa-oh. Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go.

Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go.

Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go. Hold some more breath and I'm ready to go. It's slow motion. It's slow motion. It's slow motion. It's slow motion. It's slow motion. Ladies and gentlemen, our program begins in five minutes.

We'll be right back. It's slow motion. It's slow motion. It's slow motion. It's slow motion. It's slow motion. It's slow motion. It's slow motion. It's slow motion. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.

Thank you. Thank you. Thank you. Thank you. I have three minutes and 45 seconds and a couple of announcements to make, so I'd like to get right to it. So I've been talking to a few people, when I get a chance, about the mobile app that we put a lot of effort into and how it's going to help you with your attendance and interactions at this event.

So I wanted to just clear up a couple of things. Number one, the schedule. You can favorite sessions. You can see the session details, favorite them, and you can go to my schedule. So now I can see it there. So for a multi-track event, I just want to make that clear.

You can build your own schedule here. So some people didn't get that memo. Of course, you can see all the sponsors. You got the map there. But the real killer feature here is the generative matching and the other networking features here. So I'd like to invite Simon Stermer on stage, the lead architect for this, just to help explain some features.

Can we have a round of applause for Simon, please? So Simon, we got some generative profiles going on here. These look to all be unique for each interaction. Tell me a little bit about what's going on behind the scenes here. So what we do is we pull in your profile or we build a profile from your socials and from the questions you answered when you registered.

And then we create embeddings from there and we put that into a vector database and we do a cosine similarity search and just pull your five most similar profiles. Very nice. And now I'm going to a Forva's profile and I can see all her talks there. I can also see some talking points here and these are generated for me as well.

That's right. So those are while that spinner was going generated in real time. Yeah. Awesome. And now if we go to one of her sessions, we can see after you watch her talk, you can rate it. We got session feedback built in. So this is a fully custom app.

But all right, here's the real killer feature that I like. Because when I first came to San Francisco in 2015, I went to a tech conference and everyone's handing out business cards. So when you meet someone at the event, like how do you connect with them? Oh, let's exchange emails.

Right? That's one. Give me your Twitter, your LinkedIn. It's always like different with different people. Right? Why don't we get them all right now, Mr. Simon. All right, I'm going to scan your badge. Boom. Okay. There we go. So we have badge scanning for everyone. Now I'm connected. As soon as I scan his badge, I get his email.

Even if he made it private at first, he can set his, he can change his settings to make it default to share with me when I scan him. Still got the talking points, the notes, et cetera. And now he's on my short list of scans. So after the event, I can review all my scans and I can even export all my connections, which is going to email them to me.

So is that pretty cool? What do we think? All right. All right. So cool. Thanks, man. So that's the matchup. Thank you so much, Simon. Appreciate it for you helping that. Very cool. So that's the app. Just want to make sure we all understand that. The other thing is we have AI engineer jobs.

So this is at AI.engineer slash jobs. Currently, we're featuring jobs from our expo partners. So go and check those out if you're looking for your next opportunity. Any expo partners, if you're not there, please email me and I'll get it up there before end of day. Anyone else want to get some jobs up there, email me and we can talk.

Lastly, one more announcement. I've got 20 seconds left. We're excited to announce 2025 is now on sale. If you enjoyed your time here, this is the best time to lock in the best rate. So you can go to AI.engineer slash 2025 or scan that QR code. That's going to reserve your seat, lock in the best rate.

And we don't have dates currently locked in, but likely around the same time this year. So that's my time for today. So I'm pleased to welcome to the stage our opening keynote speaker, an absolute legend in this industry. Please join me in welcoming Chris Latner. All right. Good morning, everyone.

I'm here to talk to you about modular and accelerating the pace of AI. You know what JNI is. I'm not going to tell you all about this. Let me tell you one of the things I think is really cool about it and very different than certain other technologies is that it's super easy to deploy.

There's lots of great endpoints out there. There's a lot of good implementations, a lot of ways to make it super easy to build a prototype and get going very quickly. But despite all the availability of all these different endpoints, sometimes you do have other needs. Sometimes you might want to go and control your data instead of sending your data to somebody else.

Sometimes you might want to integrate it into your own security because you've got your critical company data in your model and you don't want to fine-tune it somewhere else. Sometimes you want to customize the model. There's research happening all the time. A lot of things in building proprietary models that work best for your use cases can make your applications even better.

Of course, the inference endpoints are expensive and so sometimes you want to save money. Sometimes there's hardware out there that's really interesting and you want to explore out from the mainstream and you want to go do this. And if you care about any of these things, what you need to do is you need to go beyond the endpoint.

And so how do you do that? Well, if you have, many of you have explored this, I'm sure. The answers shifted. It used to be that we had things like PyTorch and TensorFlow and CAFE and things like this, but as inference became more important, the world shifted. First, we got Onyx, TensorRT, things like this.

And today, we have an explosion of these different frameworks, some of which are specific to one model. And that's cool if you care about that one model, but if you have many different things you want to deploy and you want to work with, it's very frustrating to have to switch between all these different technologies.

And of course, it's not just the model. You all know there's this gigantic array of different technologies that get used to build real-world things in production. And of course, none of these are really actually designed for Gen AI. So my concern about this, my objection to the status quo, is that this fragmentation slows down getting the research and the innovations coming into Gen AI into your products.

And I think we've seen so many of these demos. Last year was really the year of the Gen AI demo. But still, we're struggling to get Gen AI into products in an economical and good way. And so, whose fault is it? Well, is it our fault? Like, many of you are AI engineers.

If you don't, let's sympathize with the plight of the AI engineer. Because y'all, these folks that are building this, have new models and optimizations coming out every week. Right? Every product needs to be enhanced with Gen AI. This is not, like, one thing that we're getting dumped on. And there's so much to do, we can't even keep up.

There's no time to deal with new hardware and all the other exciting new features. And of course, once you get something that actually works, the costs end up making it very difficult to scale these things. Because getting things into production means suddenly you're paying on a per-unit basis. So, it's not the AI engineer's fault.

We should look at the concerns and look at the challenges faced here. And so, I think that we need a new approach. We've learned so much. Let's look at what we need to do. How do we solve and improve the world here? This is what Modular is about. And so, I'll give you a quick intro of what we're doing and kind of our approach on this.

First of all, who are we? Modular is a fairly young company. We've been around for a couple of years. We have brought together some of the world's experts that built all of these things. And so, we've built TensorFlow and PyTorch. We've built compilers like LLVM and MLIR and XLA and all of these different things.

And so, what I can say about that is that we learned a lot. And I apologize. Because we know why it is so frustrating to use all these things. But really, it was a, you know, the world looked very different five years ago. Gen AI didn't exist. It's understandable.

We tried really hard, but we have learned. And so, what our goal is, is to make it so you can own your AI. You can own your data. You can control your product. You can deploy where you want to. You can do this and make it much easier than the current systems work today.

And so, how? Well, what we're doing is really going back to the basics. We're going to, we're bringing together the best-in-class technologies into one stack, not one solution per model. Our goal is to lift Python developers, PyTorch users. This is where the entire industry is. And so, we want to work with existing people.

We're not trying to, like, say, hey, ditch everything you know and try something new. We want to gradually teach and give folks new tools so they can be superpowers, so they can have superpowers. And finally, so I spent a lot of time at Apple. Like, I want things that just work.

Like, you want to build on top of infrastructure. You do not want to have to be experts in the infrastructure. And this is the way all of this stuff should work. And unfortunately, it's just not the case today in AI. And so, at Modular, we're building this technology called Max.

I'll explain super fast what this is. Max is two things. One is an AI framework, which I'll spend a bunch of time about. The AI framework is free, widely available. We'll talk about it today. The other is our managed services. This is how Modular makes money. Very traditional. We're not going to spend a lot of time talking about that today.

And so, if you dive into this AI framework, well, it's, we see it as two things. It's the best way to deploy PyTorchs. It's also the best way to do Gen AI. And both halves of this are really important. And Max is currently very focused on inference. And so, these are areas where PyTorch is challenging at times.

This is where Gen AI is driving us crazy with cost and complexity. And so, really focusing on this problem is something that we're all about. The other thing, as I said before, is Python. So, we natively speak Python. That is where the entire world is. We also have other options, including C++, which we'll talk about later.

And so, how do we approach this? Well, as I said, we work with PyTorch out of the box. You can bring your models. Your model works. We can talk to the wide array of PyTorchy things like Onyx and TorchScript and Torch Compile and, like, all this stuff. And so, you can pick your path.

And that's all good. If you want to go deeper, you can use native APIs. Native APIs are great if you speak the language of KV caches and page attention and things like this. And you care about pushing the state of the art of LLM and other Gen AI techniques.

That's very cool. And also, Max is very different in that it really rebuilds a ton of the stack, which I don't have time to talk about. But we do not build on top of QDNN and the NVIDIA libraries and on top of the Intel libraries. We replace all that with a single consistent stack, which is a really different approach.

And I'll talk about what that means later. And so, what you get is you get a whole bunch of technology that you don't have to worry about. And so, again, as a next-generation technology, you get a lot of fancy compiler technologies, run times, high-performance kernels, like all this stuff in the box, and you don't have to worry about it, which is really the point.

Now, why would you use Max? So, it's an AI framework. You have one, right? And so, there are lots of different reasons why people might want to use an alternative thing. For example, developer velocity, your team being more productive, that's actually incredibly important, particularly if you're pushing state-of-the-art. But it's also very hard to quantify.

And so, I'll do the same thing that, you know, kind of people generally do, is go and talk about the quantifiable thing, which is performance. And so, I'll give you one example of this. We just shipped a release that has our int4, int6, k, fancy quantization approach. This is actually 5x faster than Lama.cpp.

And so, if you're using Lama.cpp today in cloud CPUs, this is actually a pretty big deal. And 5x can have a pretty big impact on, you know, the actual perceived latency of your product and performance and cost characteristics. And the way this is possible is, again, this combination of really crazy compiler and technology and other stuff underneath the covers.

But the fact that you don't have to care about that is actually pretty nice. It's also pretty nice that this isn't just one model. This is, you know, we have this make it easy to do int4 technology, and then we demonstrate it with a model that people are very familiar with.

And so, if you care about this kind of stuff, this is actually pretty interesting, and it's a next-generation approach to a lot of the things that are very familiar, but it's also done in a generalizable way. Now, CPUs are cool. And so, I mean, so far we've been talking about CPUs, but GPUs are also cool.

And what I would say and what I've seen is that the CPUs and AI are kind of well understood, but GPUs are where most of the pain is. And so, I'll talk just a little bit about our approach on this. And so, first, before I tell you what we're doing, let me tell you our dream.

And this is not a small ambition. This is kind of a crazy dream. Imagine a world where you can program a GPU as easily as you can program a CPU in Python. Not C++, in Python. That is a very different thing than the world is today. Imagine a world in which you can actually get better utilization from the GPUs you're already paying for.

I don't know your workload, but you're probably somewhere between 30%, maybe 50% utilization, which means you're paying for like two to three times the amount of GPU that you should be. That is understandable given the technology today, but that's not great for lots of obvious reasons. Imagine a world where you have the full power of CUDA.

So, you don't have to say there's a powerful thing and there's an easy-to-use thing. You can have one technology stack that scales. Well, this is something that is really hard. This is something where, you know, NVIDIA has a lot of very good software people and they've been working on this for 15 years.

But I don't know about you. I don't run 15-year software on my cell phone. It doesn't run BlackBerry software either. And I think that it's time to really rethink this technology stack and push the world forward. And that's what we're trying to do. And so, how does it work?

Well, you know, it's just like PyTorch. You use one line of code and switch out CPU to GPU. Ha-ha. We've all seen this, right? This doesn't say anything. I actually hate this kind of a demo. Because the way this is usually implemented is by having a big fork at the top of two completely different technology stacks.

One built on top of Intel MKL, one built on top of CUDA. And so, as a consequence, nothing actually works the same except for the thing on the slide. And so, what Modular's done here is we've gone down and said, let's replace that entire layer of technology. Let's replace the matrix multiplications.

Let's replace the fuse attention layers. Let's replace the graph thingies. Let's replace all this kind of stuff and make it work super easily, super predictably, and let's make it all stitch together. And, yeah, it looks fine on a slide. But the slide is missing the point. So, if you are an advanced developer, and so many of you don't want to know about this, and that's cool.

If you are an advanced developer, like I said, you get the full power of CUDA. And so, if you want, you can go write custom kernels directly against Max, and that's great. And for advanced developers, which I'm not going to dive too deeply into, it's way easier to use than things like the Triton language and things like this.

And it has good developer tools, and it has all the things that you would expect from a world-class implementation of GPU programming technology. For people who don't want to write kernels, you also get a very fancy auto-fusing compiler and things like this. And so, you get good performance for the normal cases without having to write the hand-fused kernels, which is, again, a major usability improvement.

Now, you know, it's cool, like there's a lot of things out there that promise to be easy, but what about performance, right? A lot of the reason to use a GPU in the first place is about performance. And so, one of the things that I think is pretty cool, and one of the things that's very important to modular, is that we're not comparing against list standards.

We're comparing against the vendor's best. In this case, NVIDIA, they're experts in their architecture. And so, if you go look at, again, there's a million ways to measure things, a microbenchmark, go look at the core operation within a neural network, matrix multiplication. This is the most important thing for a wide variety of workloads and, again, one set of data.

But we compare against Kublas, the hard-coded thing, and then also against Cutlass, the more programmable C++-y thing. And so, Max is meeting and beating both of these, you know, by just a little bit. I mean, it's, you know, it depends on your bar, and data is complicated. But, you know, if you're winning by 30%, 30% is actually a pretty big deal, given the amount of cost, the amount of complexity, the amount of effort that goes into these kinds of things.

And so, I've talked a lot about the what, but I haven't talked about the how. And so, the how is actually a very important part of this, and I'll just give you a sample of this. So, we are crazy enough that we decided to go rebuild the world's first AI stack from the bottom up for Gen AI.

And as part of doing that, what we realized is we had to go even deeper. And so, we built a programming language. We have a new programming language. It's called Mojo. And so, the thing about Mojo is if you don't want to know about Mojo, you don't have to use Mojo.

You can just use Max. It's fine. But we had to build Mojo in order to build Max. And I'll tell you just a couple of things about this. Our goal is that Mojo is the best way to extend Python. And that means that you can get out of C, C++, and Rust.

And so, what is it as a programming language? It's a full, it's Pythonic. So, it looks like Python. It feels like Python. Everything you know about Python comes over. And you cannot have to retrain everything, which is a really big deal. You get a full tool chain. You can download it on your computer.

You can use Visual Studio Code. It's open source. Available on Linux, Mac, Windows. 200,000 people, 20,000 people in Discord. It's really cool. We'd love for you to go check it out if you're interested in this. But what is Mojo? Like, what actually is it? Fine, there's a programming language thing going on.

Well, what we decided is we decided the AI needs two things. It needs everything that's amazing about Python. This is, in my opinion, the developers. This is the ecosystem. This is the libraries. This is the community. This is even, sorry, the package managing. And, like, all the things that people are used to using already.

Those are the things that are great about Python. But what is not great about Python, unfortunately, is its implementation. And so, what we've done is we've combined the things that are great about Python. Python with some very fancy, highfalutin, compiler-y stuff. MLIR, all this good stuff that then allows us to build something really special.

And so, while it looks like Python, please do forget everything you know about Python. Because this is a different beast. And I'm not going to give you a full hour-long presentation on Mojo. But I'll give you one example of why it's a different beast. And I'll pull back something many of you care about, which is performance.

And what I'll say is that Mojo is fast. How fast? Well, it depends. This isn't a slightly faster Python. This is a working back in the speed of light of hardware kind of system. And so, many people out there have found that it's 100 times to 1,000 times faster.

In crazy cases, it can be even better than that. But the speed is not the point. The point is what it means. And so, in Python, for example, you should never write a for loop. Python is not designed for writing for loops, if you care about performance, at least.

In Mojo, you can go write code that does arbitrary things. This is an example pulled from our Lama 3, written in Mojo, that does tokenization using a standard algorithm. It's chasing link lists, has if statements, for loops, like, it's just normal code. And it's Python. I mean, it feels like Python.

And that is really the point. And so, for you, the benefit of Mojo is, first of all, you can ignore it if you don't want to care about it. But if you do, you don't have to learn C and C++. You have lower cost by default versus Python because performance is cost.

It means that, as a researcher, if you use this, you can actually have full stack hackability. And if you're a manager, it means that you don't have to have people that know Rust on your team and C++ and things like this. You can have a much more coherent engineering structure where you're able to scale into the problem no matter where it is.

And so, if you want to see something super polarizing, go check the modular blog, and we'll explain how it's actually faster than Rust, which many people consider to be the gold standard, even though it's, again, a 15-year-old language. So, I have to wrap things up. They'll get mad at me if I go over.

The thing that I'm here to say is that many of you may want to go beyond the API. And they're fantastic. There's amazing technology out there. I'm very excited about them, too. But if you care about control over your data, you want to integrate into your security, you want customization, you want to save money, you want portability across hardware, and then you need to get on to something else.

And so, if you're interested in these things, then Max can be very interesting to you. Max is free. You can download today. It's totally available. You can go nuts. We didn't talk about production or deployment or things like this, but if you want to do that, we can also help.

We support production deployment on Kubernetes, SageMaker, and we can make it super easy for you. Our GPU support, like I said, is actually really hard. We're working really hard on this. We want to do this right. And so, it will launch officially in September. If you join our Discord, you can get early access, and we'd be very happy to work with you ahead of that, too.

We're cranking out new stuff all the time. And so, if you are interested in learning more, you can check out modular.com. Find us on GitHub. A lot of this is open source. And join our Discord. Thank you, everyone. Ladies and gentlemen, please welcome to the stage Principal Developer Advocate of AWS, Antia Bart.

Hi, everyone. I'm so excited to be part of this conference and share with you five practical steps from software developer to AI engineer. And if anyone is wondering here, this avatar on the slide, this is what happens if you ask AI to make you look a little bit more agentic.

All right, let's get started. So, I'm pretty sure everyone is familiar with this image here and the post from SWIX that defines the new role of the AI engineer. And as you've experienced probably daily in your jobs, you don't need to be a full ML researcher anymore or data scientist.

Things that took months or years before to get AI projects into production is now able to be just a couple of API calls. Super exciting. But still, if you're working with AI, it still makes sense to understand the basics of the technology. And this involves a couple of things, right?

So, you have to understand at a basic level how foundation models work. Why they're sometimes producing output that you don't expect in your application code, right? You have to understand how you can customize the models, how you can, you know, for example, sometimes fine-tune models to adapt them to your specific use cases and data sets, how to include functions in your application code to give them access to additional systems.

The good news is, if you're just starting on this journey to become an AI engineer, there's plenty of resources now these days available to you to learn. And I wanted to call out one specific course here, which is called Generative AI with Large Language Models. A few colleagues in mind, we actually collaborated with Andrew Ng and the team at deeplearning.ai to put this course together and help you understand the fundamentals of generative AI to help you build real-world applications.

If you're curious, it's available on deeplearning.ai and on Coursera. Now, the second step in this journey is to start get hands-on with the AI developer tools to help you increase your productivity. And I think we're all seeing this quote here and we experienced it in our daily jobs that how we do work, how we develop applications has changed a lot.

These days, we can literally use natural language inputs to interact with applications and really, English has become one of the most popular and hottest programming languages. I think we can see this happening. For example, you can go these days from English to code by asking AI to, for example, rewrite a readme file.

We can also do code to English. For example, asking AI to document functions in our code. But this is not all. If we look at the software development life cycle, I think many of us can agree that the majority of time we usually spend not writing valuable code, but all the other things around it.

So sometimes up to 70% of unvaluable tasks, which is writing boilerplate code, writing documentation, trying to maintain old code bases, right? And sometimes we only have like a fraction of the time, maybe 30% that we're spending on actually what, you know, creates joy and kind of the creative tasks in software development.

And this is what led us at AWS. This inspired us to create Amazon Q. Amazon Q is a generative AI-powered assistant specifically developed for software development. And this is much more than just a coding assistant. Q developer actually uses agents to perform much more complex tasks and help you automate those.

For example, feature development and also code transformation. Think about working with old Java-based code bases that you need to migrate maybe to your newer Java version. And to show you how this works, I asked my colleague, Mike Chambers, to put together a quick demo. Let's have a look. With Amazon Q installed inside of my IDE, I can go to New tab and I can start a conversation with Amazon Q developer.

And I can do the kinds of things that maybe you'd expect, such as how can I create a serverless application? How do I get started? And the chat session brings back a list of instructions of what I should do, starting off by installing AWS SAM CLI, how to do that, where to get that from, and how to step through the creation of a project.

Now, if I've done that, then serverless SAM, for example, might actually come back with some generated code. And here is that code. Maybe I don't quite know what this code does. So I can right-click on the code and send it to Amazon Q, asking Amazon Q to explain. And the code then will go into a prompt along with explain and generate an answer.

And this is great for code that's been generated for us. But also, imagine code for legacy systems, something that was worked on years ago by somebody else, where you can get Amazon Q to help explain it. We can also get Amazon Q to generate code. Now, this is, again, probably the kind of thing you'd expect.

I can put in a comment line inside of my code. In this case, I want to create an input checking function. And I'm going to give it some more definition here that I actually want it to trim any string that's being sent into this function. And yes, Amazon Q can generate this small function.

Well, that's great. But what about if I've got more code that I need to have generated? Well, I can go to the chat and put in slash dev, and I can put in a much more comprehensive description of something that I would like. In this particular case, I'm going to ask for it to write a function to search by category in DynamoDB with a bunch of details about the way that I want the output to be formatted.

So this is much more than just a single or a few lines of code. And in this particular case, what's going to happen is it will come back again with a step-by-step list of what's required. So I need to add in template.yaml. It's recommending that I create search by category.mjs and many more things.

But this isn't just a big shopping list of things that I need to do. This is actually a plan. And it's a plan that Amazon Q can actually follow for us. So it generates some code as a change set, something that we can look at the difference between our current code and what it suggests.

And if we like that, we can actually click on the insert code button and it will add all of that code into our project, way more than just a couple of lines. So Amazon Q Developer is much more than just code completion. All right. If you're curious to learn more about Amazon Q, Amazon Q Developer, we have a couple of more sessions throughout this day.

So make sure you're checking those expo sessions. And we also do have a session at our AWS booth here. You can also visit our Amazon Q Developer Center for much more examples what you can do with it. All right. Let's come to step three. And this is where the fun starts.

Start prototyping and building with AI. And the fun includes a couple of steps, right? Everyone developing with AI knows this. It starts all with refining your use case. And then really you're on this road trying to choose from different models. You're trying to, you know, customize them to your use case, decide whether it's prompt engineering, whether you do RAC, whether you need to do a little bit of fine-tuning there with your data.

And of course, across the whole development workflow, you have to incorporate responsible AI policies, making sure data is private and secure, and also implementing guardrails into your application. And then when you're integrated, another fun part, obviously working with the agents, what we're hearing a lot here throughout this conference, and the fun topic of, you know, how to keep them up to date.

Gen AI ops. I think there's a lot of terms for that. MFM ops, LLM ops. So really kind of a lot of things to consider here. I want to dive in briefly into the topic of models to choose. And this is really an important topic. When you're evaluating models, you have to really evaluate them thoroughly because most likely there's not just going to be one-size-fits-all for you.

In fact, if you look at all your use cases you want to implement, there's likely no one model to rule them all. And this is why we developed Amazon Bedrock. Bedrock is a fully managed service that gives you access to a wide range of leading foundation models that you can start experimenting with, implementing into your applications.

It also integrates the tooling you need to customize your model, whether it's fine-tuning, also to include REC workflows, to build agents, and of course, everything in a secure environment where you are in full control of your data. And speaking of choice, just to give you a quick overview, as of today, this is the selection of models you can choose from.

We're working with leading companies such as AI21 Labs, Entropic, Cahere, Meta, Mistral AI, Stability AI, and we also offer our own Amazon Titan models for you to choose from. And I'm super excited just to call this out. Last week, together with Entropic's launch, we integrated Cloud 3.5 Sonnet on Amazon Bedrock as well.

So you can also, since last week, use this model. Super exciting. Now, with choice also comes responsibility, right? And we continuously innovate and trying to make it easier for you to build applications across the different model types. And just a few weeks ago, we introduced a new unified Converse API in Amazon Bedrock.

What does this do? The unified Converse API helps you with a new unified method structured invocation, meaning you can use the same parameters and bodies regardless of which model you choose. And we are on the platform side, we're handling this translation if parameters are called different for the different models, handling the system user assistant prompts for you and also giving you a consistent output format.

And as well, having native function calling support in here. But let me show you how this looks in code. So here's the Python example that shows how you can use the new API. This is Python, so we're starting by just integrating the Python SDK client here. and then you can define this list of messages.

And here's, for example, where you put in your user message prompts. You can put in system prompts as well. And then this message list, you can just pass in this single API call using the Converse API here. In the model ID, you can choose which model you want to test.

Here I'm using an entropic model and then pass the messages and also the inference parameters. And again, in this API, all those parameters are standardized and we're going to make the work behind the covers to convert this to the specific format that the model is expecting. So you have an easy way to work across different models.

Similarly here for function calling, we do have the support built in with the models that support it. So how we implement this is by defining a tool list. So tool here equivalent to the functions you want to give access to. And then when you're doing the Converse API call, you can pass this list of tools.

All right. If you want to find out more about Converse API, here's a link to our generative AI space on community.aws, which has a lot more tutorials, code examples, not just for Python, but across different languages as well. So check it out. The author here, Dennis Traub, is also somewhere here in the audience here this week.

So if you want to connect with him, talk about different code examples and how to use the API, feel free to reach out. All right. Now let's integrate AI into our applications. And this can be a whole session in its own, but I want to focus on one of the hottest topics right now that we're discussing during the conference, which is, of course, agents.

And I have one more demo here and I asked my colleague Mike last time to put together an exciting demo to show you what you can do with agents. Mike. And we need sound. To be able to create agentic workflows right inside of the AWS console and inside of the service.

It works fully serverless and I've used it to create an agent that plays Minecraft. Let me show you how I did it. If we jump into the AWS console, go down the menu on the left-hand side to agents, you can see the agent screen here and I can open up my agent, my Minecraft agent.

Now, if I just go into agent builder and just expand the screen out a little bit, you get to see some of the parameters that I used to create this agent. So you can see the large language model I used, in this case, Claude3 Haiku. And you can also see this, the instructions for the agent.

Now, this is not some notes for myself. This is actually prompt engineering that we're doing to explain how we want the agent, in this case, the Minecraft bot, to play the game. And then we also have to add some tools in, some Minecraft tools. So we do that through actions and inside of action groups.

So I've got a couple of different action groups. We've got Minecraft actions and Minecraft experimental. Let's have a look at actions. And inside of here, we can see some really simple things, some actions, that the bot will be able to do. And these are all linked up to code.

So we've got the action to jump. We've got the action to dig. And you can see the description here for action to dig. It's got some instructions. Again, this is prompt engineering. And then we've got some parameters that we can select, collect. In fact, we require these parameters. So the bot needs to get these for us.

If I scroll down a little further, there's a couple of really simple actions in here. Action to get a player location and action to move to a location. I want to show you those in action because the bot can actually problem solve and reason its way to be able to use these tools to solve simple problems.

Let's jump into the game. And so it is nighttime. So let's set it to be the daytime so that we can see what's going on. So it sets time to day. Okay. And there in the middle of the screen, you can see Rocky. Rocky is the bedrock agent running inside of the game.

And we can talk to it and we can have a chat session. But what about if we want it to come to us? Now, there is no tool to come to us. So I'm just going to back up a little bit further, make it a little bit more of a challenge.

And I'm going to say, come to me in chat. And what's going to happen now is that the agent's going to reason through a whole set of actions. It's going to look to see who requested that. It's then going to take that name, and that's my name, and it's going to find the location of that player.

And then it's going to map a path from where it currently was to me. All of those things happened all in that blink of an eye. And there's agentic workflows making all of that happen. This is super exciting. I'm discovering new things that this bot can do every day.

But with that, it's back to you. All right. Thank you, Mike. If you're curious to know how we did this, check out our booth session. We're running the demo there as well. And we have another session in the agent's track later today. So make sure you're popping in there if you want to know more.

And, of course, you can find the project code for this on GitHub. So if you want to play, play it on your own in how you can integrate agents into a fun thing, check out this project. All right. We're almost there. So the last step I really want to call out is stay up to date.

There's so much happening in this space, as you all know. And a really good way to do that is to engage with the community. Speaking of community, I have one last announcement to make. And I'm super excited to announce that we're transforming our AWS loft here in San Francisco into the AI engineering hub for the community.

So we're super excited to host workshops events and meetups there. If you want to suggest a couple of topics you're most interested in to make those events most valuable to you, fill out this quick survey here. Also, if you're interested in speaking or hosting a meetup yourself, you can let us know.

And also, we do have another event tonight, which I think we're reaching capacity or just half-reach capacity, but we do have a happy hour with Entropic tonight at the loft. In case you didn't make it anymore and we're at capacity, don't worry. We're working on putting together much more events like this in the upcoming weeks and months.

So keep an eye out for those. And with that, I'm coming to the end of my presentation. This wraps it up, the five practical steps to become an AI engineer. And let's innovate together. And I'm looking forward and I'm excited to see what you build with AI. Thanks so much.

Make sure you're checking out the rest of the sessions here and also pop by our booth outside. Thanks so much. Relations at Anthropic, Alex Albert. All right. Good morning, everyone. So today, I want to start with a little story. A short history lesson, if you will. So, you know, sit back, get comfortable.

I'm going to take us back to the year 1882. It's the dawn of the electrical revolution, really. The world's first commercial power plant just opened up. Electricity. This amazing new force is all the rage in the manufacturing industry. people are claiming that it's going to change everything. And yet, something very interesting happened around this time.

Or rather, it didn't happen. You see, despite electricity's obvious superiority in comparison to the traditional techniques at the time, like steam engines, solutions, it didn't immediately improve manufacturing productivity. Why? Well, because factory owners were simply trying to replace their old technology with this new technology into an outdated paradigm.

Let's picture a typical factory at the time. So, we have a huge coal-fired steam engine on one end and we have a network of transmission lines going across the top, all driving hundreds of machines locked in the same rhythm. These legacy steam power factories were incredibly inefficient. You know, if one station needed power, all of a sudden, you had to turn on that entire steam engine and it had to power all of them.

Factory layouts were dictated by the limitations of the transmission lines, not by what was best for the process or for the workers. When electricity arrived, many factory owners simply swapped out the steam engine for an electric one. And sure, you know, they added some lights and, you know, workers didn't have to toil next to a coal-fired furnace all day, but the fundamental limitations of the factory remained.

So, this real electrical revolution, well, it didn't actually come until we imagined factories from the ground up with electricity at its core. Factories started to become flexible and adaptable. They allowed for smaller, specialized tools. Workers could bring their tools to the items instead of having to lug the items back to their workstations.

The entire manufacturing process became more efficient, more humane, and more productive. Now, let's fast forward 140-something years to today, and you can see that we find ourselves at a similar point in regards to AI and LLMs. Enterprises, startups, developers are all building and integrating LLMs into their products, but often, they're just tacking it onto their existing product surface, adding a few star icon buttons in the top left corner and calling it a day.

And this is not the first time we've seen this in Silicon Valley. Let's think back to when mobile first emerged. Companies simply tried to just shrink down their website and put it on a phone. It wasn't until we redesigned apps from the ground up with the unique capabilities of mobile, like always on camera and GPS, that we actually began to see true innovation in the space and adoption.

This is when the Snapchats and the Ubers of the world started to emerge. So just as companies and just as factories went through there, you know, replaced steam engines with electric one phase and tech companies went through there, just hire a couple mobile web dev people phase, we're now in our magic star icon phase with respect to AI.

And yeah, it's funny, but the thing is, you can't blame any of the companies or developers that are actually trying to do this right now, right? Like all of us are trying to do this, but in many ways, we're just still so early. LLMs are non-deterministic. They're hard to build on.

They're completely different than what most developers are used to using. Reliability is still an issue. Prompts still take rounds and rounds of optimization. And we've also just started to scratch the surface of potential product opportunities. So far, not much has really stuck beyond just the text box. We've been missing something.

Something that's a little hard to put a finger on. But just last week, I think we scratched the surface of a potential new product feature that we can build. As some of you may have heard, last Thursday, we released our new model, Claude 3.5 Sonnet. 3.5 Sonnet is the first model that we released in the new Claude 3.5 family.

It's only the middle model, and yet it is better than our last best model, Claude 3 Opus. In my opinion, Claude 3.5 Sonnet is one of the best models in the world right now. And the benchmarks seem to back it up. MMLU, human eval, GPQA, tool use, all the common characters here.

It's top of its class in many regards in these academic lab type environments, but what I'm most excited for is how it actually does in the real world. The model is particularly strong in RAG use cases thanks to its 200k context, and also has near-perfect recall over that entire context as well.

On coding tasks, 3.5 Sonnet seems to grasp debugging problems better. It's not getting stuck in those same loops as much as previous models. One of the best methods that we've found for actually measuring more complicated chains of reasoning is pull requests. They have a defined task, they usually take a few steps to solve, and the model is able to iteratively write and test its way to a solution.

In our own internal pull request evals, we're seeing that Cloud 3.5 Sonnet scores a 64%. And to put that number in comparison, Cloud 3 Opus only scored a 38%. 3.5 Sonnet also has state-of-the-art vision abilities. It shows considerable improvement over 3 Opus in basically every benchmark that we tested it on.

Things like table transcriptions and OCR, our breeze now. Passed this table in a 3.5 Sonnet and basically replicated it perfectly in Markdown. Probably can't read all those numbers, but trust me, I double-checked them to make sure they're all right. Vision capabilities were actually what amazed me the most when I started playing around with this model.

It feels like we are really on the cutting edge of unlocking so many more use cases. And, you know, as you're hearing me say all this, you might be thinking, well, that's great, Alex, but it doesn't mean anything if I can't actually use the model. and you're right and we heard you.

And that's why 3.5 Sonnet is available on RAPI, AWS Bedrock, and Vertex AI. We understand that developers want choice when they're building and we want Claude to be available wherever you are. In terms of pricing, 3.5 Sonnet is five times cheaper than 3 Opus. It's only $3 per million input tokens and $15 per million output tokens.

3.5 Sonnet's combo of speed, intelligence, and low cost makes it much more economical to use and embed in your apps than 3 Opus. But 3.5 Sonnet is not all that we've released in the past week. We also released a new product feature that I think is actually more inspiring to developers in terms of thinking about and building those AI products from the ground up.

It's called artifacts. It's called artifacts. Artifacts separate the content that Claude produces from the actual chat dialogue itself. This allows you to work collaboratively with Claude on things from essays to SVGs to React websites. Artifacts become really powerful when you combine it with 3.5 Sonnet. Those coding skills plus that reasoning ability plus that strong visual acuity enables a new product experience that's really fun to use.

It's also a developer's best friend in that it allows you to quickly take screenshots and figment diagrams and quickly turn it into code and components that you can actually just go use. As you can see in this, I basically cloned our entire Claude.ai chat layout in React just from a single screenshot.

And this feature has practically been hiding in plain sight now. Just waiting really to be discovered for over a year and a half. Maybe this tweet is right and we really are early on this S-curve of production productionizing LLMs which I think is actually pretty inspiring. And Artifacts is not the only AI feature that we launched recently.

On Tuesday, we released Projects. Projects enables dev teams to work and collaborate much more efficiently by grounding Claude's outputs in your own knowledge whether it's style guides or code bases or transcripts or even your past work. On our Claude team plan, you can even share these projects and your chats with all your teammates.

At Anthropic, our engineers now upload code repos and documentation that they use. And I've started to see people actually just share the chats and the artifacts instead of Google Docs or site documentation. Projects is another great example of when you think from an LLM and an AI standpoint first, you can actually start to build product experiences that complement these technologies and don't feel like a simple add-on to what you already have.

So now that hopefully the creative product juices are flowing in everyone's minds, I want to dive a little bit into API improvements that we've rolled out recently and things that allow you to actually build this cool stuff. I also want to give a preview of what's coming next that will enable you to build even more.

So a month ago we released our new tool use API. Tool use allows you to give Claude custom client-side functions that it can then intelligently leverage. Tool use also enables things like consistent structured JSON output. With 3.5 Sonnet, I've actually started to see devs give Claude hundreds of tools at a time.

On the developer console front, we're also continuing to iterate. We added a prompt generator that uses Claude to write prompts for you based on a task description so you can see in this video we put in a task description and then out comes a optimized prompt and then once that prompt is all done you can actually just start editing it right in the workbench itself.

You can see we've also added support for variables so you can edit prompt templates as well, test things like RAG use cases. And finally, we're also working on a new evaluate feature which is currently in console right now with a beta tag and we will plan to share more on this and continue to iterate on it very soon.

So what else is next? Well, there's two things that I can share right now. First is that you can expect more models. 3.5 Haiku and 3.5 Opus are coming later this year. With each model generation, we're looking to increase the intelligence, decrease the latency, and decrease the cost. The number one thing that I tell developers is to not forget to build with that in mind.

Models will become smarter, cheaper, and faster in orders of months, not years. When you're planning your product roadmap, be ambitious enough to build with the belief that new models may arrive during your development period. We are also working on other areas of research like interpretability. In one of our latest papers called Scaling Monosemanticity, we explained how we've been able to find features within models that activate for different topics.

Once you identify a feature, you're able to clamp its value and turn it up or down to actually steer the model's outputs. A few weeks ago, we showed Claude.ai users how this worked through GoldenGate Claude, which was a version of Claude that had the GoldenGate Bridge feature turned up significantly.

Yeah, fan favorite. We currently have a few beta testers also experimenting with a steering API. This allows developers to find the clamp features for specific attributes and actually turn that dial up or down, which, again, allows you to control Claude's outputs in addition to actually just prompting it. We hope to be able to roll this out to more developers in the very near future as well.

Now, if anything in this talk has sparked any ideas, I want to encourage you guys to just go out there and build and make quick prototypes as fast as you can to get that validation and that feedback loop started. And for even more of an incentive, we actually just launched another build with Claude contest yesterday.

It runs until July 10th. The top three projects will each receive 10K in Anthropic API credits. To see more details, just visit that link below. It's just at the top of our docs page as well, so you can find it there too. I'll leave that up for a second.

And finally, if you have any questions or you want to hear more about just what we're thinking about, I'll be at that AWS booth down the hall for the next few hours. You can also find me on x slash Twitter at AlexAlbert with two underscores. I do try to read all my DMs.

I spend way too much time on that site, so feel free to ask questions there as well. And with that, I want to say thank you guys very much and enjoy the last day of the summit. Ladies and gentlemen, please welcome to the stage CEO of Langchain, Harrison Chase.

Hello. Today, I want to talk about agents. So LLM-powered agents are really nothing new. The React paper came out in October of 2022. Langchain launched about a month after that and AutoGPT is over a year old. And to me, AutoGPT represents the peak of hype in agents. And I actually think for a few months after that, there was a bit of a fall-off in interest as people realized that the generic agent architecture wasn't reliable enough to build systems to ship to production.

While there was this fall-off, I do think there was some really interesting work being done. So OpenAI's Assistance API I think was really novel in a few regards, and I'll come back to that in a little bit. And earlier this year, we launched LangGraph. While Langchain did agents, it also did a bunch of other things.

LangGraph is purpose-built for agents, however. What exactly does that mean? So LangGraph is highly controllable and low-level. As mentioned, we saw that these generic agent architectures weren't reliable enough. And that companies that were shipping agents to production were building custom cognitive architectures, encoding little differences in how they wanted their agents to behave.

And this was super important. And so we made LangGraph extremely low-level and controllable. It also comes with a built-in persistence layer, which enables a lot of really cool human-in-the-loop interaction patterns. and it's streaming first because streaming is really important for LLMUIs. And just to emphasize, LangGraph works with or without Langchain and it integrates seamlessly with LangSmith, our testing and observability platform.

LangGraph's already being used in production by a lot of our awesome partners, ranging from cutting-edge open-source projects like GPT Researcher to trailblazing unicorns like Replit to public companies innovating at scale like Norwegian Cruise Line, Ally Bank, and Elastic. And today we're excited to announce the first stable version of LangGraph, reaffirming our commitment to building an agent architecture that allows you to build the custom cognitive architectures that are necessary for bringing agents to production.

but that's not the only thing we're launching. So I want to go back to the assistance API, which I mentioned earlier. And I think there were a lot of really cool things here and really novel things. It wasn't a framework. It wasn't just a framework for building agents. It also introduced infrastructure components.

It had built-in persistence. It kept track of the messages and stored them for you so you didn't have to do that. It had this concept of background runs, really good for longer-running asynchronous workloads. and it allowed you to configure agents. The downside is that it didn't give you full control over the cognitive architecture of your application.

It came with a specific state that it expected your application to have, a list of messages. And it was a little bit rigid and didn't let you easily do other things besides that. And so that got us thinking. What if we took LangGraph, which lets you build these custom cognitive architectures and combined it with these generic agent infrastructure pieces?

Today, we're excited to announce LangGraph Cloud, which is a step in that direction. So with LangGraph Cloud, you can take your LangGraph applications written in Python or JavaScript code and with no changes get a production-ready agent API. That agent API has all the benefits of the assistance API. So it comes with built-in persistence for whatever the state of your LangGraph agent is.

It comes with a task queue to manage background runs. And you can configure different instances of your graph to change out the LLM or prompt that's used. But we've also added a few other things that we're really excited about. So when you kick off an agent run and you send a message and it goes and it does a bunch of work, what happens when you send it another message before it's finished?

We call this double texting and we've introduced four different modes to handle this. Agents aren't just invoked through chat. They're also triggered on schedules. And LangGraph Cloud comes with built-in crowd jobs to easily support this. I talked about human-in-the-loop. One of the really important human-in-the-loop features that we're seeing is the ability to break before specific steps.

For example, if you have a tool that you really want the user to approve access for. So LangGraph Cloud easily comes with breakpoints. which allows you to add this and then resume once you get that confirmation. That's not the only human-in-the-loop feature that's supported. So with LangGraph Cloud and the built-in persistence, you can easily go back to any step in the agent's trajectory, edit that, and then resume from there.

And so this is supporting a bunch of really cool time travel-like features that we think will be very important for the UXs of the future. And just to emphasize, another benefit of LangGraph Cloud, of course, is that it's not bound to OpenAI and it supports any cognitive architecture that you can build with LangGraph.

Finally, I want to talk about LangGraph Studio. So LangGraph Studio is what we think the best way to build, debug, and share agencies. So taking a look at this video, you can see that you can easily see the graph of the agent. You can invoke it. You get streaming output of all the steps and the tokens.

You can go back. You can modify steps, edit it, and resume from there. You can also add breakpoints so that for future iterations, you have to explicitly approve it before you continue. So all those human-in-the-loop features that I mentioned, we built into LangGraph Studio and are providing a developer experience specifically aimed for building agents.

So LangGraph Cloud is in private beta. We're excited to work with cutting-edge companies to figure out how to deploy agents reliably. We think of framework like LangGraph that gives you complete control over the cognitive architecture of your application is just one part and we strongly believe that everyone should be building that.

That is business logic that you should be incorporating and that is your moat in some sense. However, we also think there are generic infrastructure pieces that just slow down the time to get to production and that's what we want to be building with LangGraph Cloud. Thank you all. all right.

How do we enjoy the opening keynotes? yeah, really great time. I want to run through a couple of the tracks coming up so we can go through that. So, agents. So Andrew Ng famously said that AI agents could drive more progress in 2024, even more than the next generation of Frontier Models.

Our host, Demetrios, from the MLOps community, just came off organizing his own AI quality conference next door and brought his ukulele to guide us through building crews and factories and AI agents. Join him in salons 14 to 15 and head to get there, head to the AWS booth down there and turn right at the end of the hallway to the end.

Evals and LLM ops. AI engineers should build real moats instead of GPT-4 wrappers. Twitter loves hacky MVPs but serious AI engineering means writing great evals and building good operational discipline for you to ship better, faster, and cheaper. You might think our track sponsor Galileo AI ships a UI generation platform but you might be surprised by how they ship the evals that ship the platform.

Mouthful to say. This is Swix's copy. I'm reading it for the first time. So join Osman in salons 2 to 6 just outside the doors behind you and to the left. At our summit last year, OpenAI launched the year of multimodal AI with vision and image generation. This year, GPT-4.0 is just one of many frontier models we will use to push into applications for on-device vision, real-time TTS, character simulation, and classroom education.

One of the track speakers, Ben Heilick, worked on AI and the Apple Vision Pro at the Apple Vision Pro and his co-founder Alexis will guide us through the dawn of multimodality. Join her right here in salon 7. And AI engineer is a convention of both GPU rich and poor and we are excited to learn the state of the art from people making it.

Didn't get a chance to rehearse this. This is going well. Our host, Nyla, worked in product at NVIDIA and Convey and is excited to accelerate our learning at the top teams in GPUs and inference. Due to scheduling conflicts, we also have two talks here from yesterday's tracks. Scott Wu from Cognition, a.k.a.

Devin, on CodeGen and Kathleen Keneally of Google Gemma on OpenModels. Join Nyla and her speakers right here in salon 8. And last but not least, day two of the AI leadership track addresses more needs of VPs of AI from understanding enterprise RAG, building evals and security, privacy and compliance safeguards, hiring and growing an AI engineering org, as well as case studies from Cohere and Twilio.

And as a reminder, this track is exclusive to AI leadership track attendees with the green lanyards and the green badges. So if there is room at session start time, we can let in speakers with the blue badge and blue lanyards. But anyone else, again, please do not attempt to attend these sessions.

These are exclusive sessions or you'll be escorted off the premises by security. Please don't make us do that. If you want to attend these sessions next year, the tickets there are for you to purchase. So these take place right across the hall in salons B to D. So we have a number of things happening after this block of talks.

We got expo and expo sessions from 10.30 to 11.10. We got breakout tracks from 11.15 to 12.15. Then we'll have lunch. We got expo and expo sessions happening more. Then we have HF0. They're doing a demo day. That's going to be in salons 2 to 7 over there just beside the doors.

And then we got the more expo sessions. Then we'll be back here for closing keynotes from 4 to 5. That includes CEO of GitHub Thomas Damke. But I believe if the demo guides are good, we might actually have a special guest with us today. Tim, did we get that set up?

Oh, hey. Hello. Who is this? Hey, what's going on? Everyone, please, come on, welcome to Swix. Hey from Singapore. This is a live video. The crew here is also watching along. It is 1.13 a.m. our time. Right on. Well, glad you could join us, Swix. So how has it been?

What's going on? Do we want to give any more contacts to the folks here? Scheduling speakers is hard, especially when they're busy launching Gemma 2. and it's been a fun experience putting this together. I hope everyone is having fun and I hope that the talks have been engaging. I wish I was there with you.

We absolutely miss you. I know a lot of people thought it was a joke for a long time that you didn't get to make it. So I'm really glad that you're supporting us. This guy has been so helpful in everything. Like, all the sessions that you see here, he's curating that with, you know, help from his friends.

But what is it like from afar, you know, watching the live stream? Like, do you have time to watch the live stream? Are you catching any of the talks? Yeah. If actually people are watching the YouTube live stream, that's me in the chat just hanging out with folks, saying things I regret sometimes.

But it's very fun to just be one of the audience. My dream was always to actually just show up at one of these conferences and just be an attendee and not know what's going on. I think there's a little bit too much not knowing what's going on even as a co-organizer.

But it's been fun nonetheless. I think the lessons learned from this event will allow us to implement a lot of procedures and policies that allow us to smooth the organization tasks a bit. And I think, well, we might have a little bit more budget now that we've kind of proven the model here.

So any parting thoughts as we enter the last day? Yeah, I mean, there's lots of amazing talks. I'm personally excited about the, like, I can't pick favorite. I'm personally excited about all of them. And I'll catch up on livestream afterwards. And I think there's some after parties as well, right?

I'm not sure what the social schedule is like, but there's tons of after parties. I think they're all on the, they should be on the homepage. The homepage is getting very long now at this point. Yeah. Let's put those in Slack. Is everyone on Slack? Let's drop those in Slack as well.

So we'll drop those in the announcements or general. Yeah, and I think I've restored the World's Fair 2025. I broke it just before you went on stage because I told you I was going to change the URL and then we didn't change the QR code. So try it again.

That's why I broke. I thought it was our QR provider that just got a spike and they just cut it off. Interesting. No, it's me. Slicks is to blame for the demo gods. Slicks is the, the demo demon. All right. Well, thank you so much for showing up. So sad that you couldn't be here with us but really great to be a partner with you at this event.

So let's hear it first, Slicks, everyone. So we're going to go to break now. We will see you all back here at 4 p.m. But enjoy the breakouts. Until then, see ya. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye.

Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye.

Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Mm-hmm, because I'm missing you. Picking up my heart, find every piece that's broken. Been trying to get back to myself, but don't have a clue.

I'm looking for someone, can't find a door that's open. I'm losing all my hope, feels like I'm left here in two. Because I'm missing you. Tell me that you wanna stay. Baby, just don't walk away. I need you now. Faded out. All the time we spent alone, fighting through the firestorm.

Don't let me down. I need you now. I need you now. 'Cause I'm feeling worn out. It's getting to me. Lost some heart trying to get on my feet. Caught in the madness. I feel you somehow. Don't let me go. I need you right now. I wanna be next to you.

You wanna be next to me. Holding our paper hearts. Fading out broken dreams. I wanna be next to you. You wanna be next to me. Holding our paper hearts. Fading out broken dreams. I wanna be next to you. You wanna be next to me. Holding our paper hearts. Fading out broken dreams.

I wanna be next to you. You wanna be next to me. Holding our paper hearts. Fading out broken dreams. I want to be next to you. You want to be next to me. Holding our paper hearts, fading out broken dreams. I want to be next to you. I want to be next to you.

You want to be next to you. I need you now, cause I'm feeling worn out. It's getting to me. Lost some heart trying to get on my feet. Caught in the madness. I feel you somehow. Don't let me go. I need you right now. I want to be next to you.

You want to be next to me. Holding our paper hearts, fading out broken dreams. I want to be next to you. You want to be next to me. Holding our paper hearts, fading out broken dreams. I want to be next to you. You want to be next to me.

Holding our paper hearts, fading out broken dreams. I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you.

I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you. I want to be next to you.

I want to be next to you. I want to be next to you. I want to be next to you. And I want to be next to you. I want to be next to you. I want to be next to you. And I want to be next to you.

I want to be next to you. Hold my breath and I'm ready to go. I'm falling right in and I'm ready to go. I found what I want and I know that we're on top. So I'll dive in and I'm ready to whoa. Hold my breath and I'm ready to go.

I catch you laughing and I'm ready to go. You're holding a mask, strike it, and ignite my soul. I'll dive in and I'm ready to whoa. We are a summer storm. A feeling you can't ignore. Do you ever stop to feel it caught in the afterglow? I'll come back to your door.

To know that you believe in summer all that I want. You know summer we got it all. You and me now. Holding my breath and I'm ready to go. I'm falling right in and I'm ready to go. I found what I want and I know that we're on top.

So I'll dive in and I'm ready to whoa. Hold my breath and I'm ready to go. I catch you laughing and I'm ready to go. You're holding a mask, strike it, and ignite my soul. I'll dive in and I'm ready to whoa. Look at the sky and our silhouettes, dancing on the pavement.

Caught in a perfect storm, you and those eyes again. Caught when I least expected, since you're all that I want. Caught in a perfect storm, you can't do it. Caught in a perfect storm, you and me now. Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again.

Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again.

Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again. Caught in a perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again.

Caught in perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again.

Caught in perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again. Caught in perfect storm, you and those eyes again again. Caught in perfect storm, you and those eyes again again. Caught in perfect storm, you and those eyes again again. Caught in perfect storm, you and those eyes again again.

Caught in perfect storm, you and those eyes again again. Caught in perfect storm, you and those eyes again again. Caught in perfect storm, you and those eyes again again. Caught in perfect storm, you and those eyes again again. Caught in perfect storm, you and those eyes again again. Caught in perfect storm, you and those eyes again again.

evals become a lot easier because the decision trees are explicit and you can sort of verify at every step what's going on and what's going wrong. So Substrate I think is a sort of new way new approach to this. I think our model is sort of fast in ways that other paradigms can't be.

It's sort of flexible enough to build any AI product out there and it works at scale by default. So what is it? I think at its core substrate is a coupling of two things. First, I think it's a really elegant developer SDK that lets you describe a computation graph over any number of nodes.

And the abstractions here are really general and so we have we have a bunch of intelligence nodes across all the modalities that you might care about which is like generating images, transcribing speech, generating text, JSON, embeddings, executing code. But second, substrate is also an inference engine specifically built to run these computation graphs as efficiently as possible.

So these graph representations here are It's a representation of many tasks and their relationships and since we run a very coordinated compute cluster We can statically and dynamically optimize things like batching, caching, sort of networking, concurrency, physical placement, which really makes a big difference. And if you look at most frameworks out there, they're typically involving dispatching a bunch of API calls separately and if you look at what happens mechanically when you do that, it's every step means you've got to resolve DNS, you've got to go through proxies, you've got to do authentication, like balance checks.

And all of that, and all of that, and all of that sort of adds hundreds of milliseconds of latency on every single step and if you contrast that with substrate, we transfer data from node to node, process to process on the order of microseconds, which is some 10,000 times faster meaning that it's actually feasible now to run online applications that involve dozens of nodes.

We've also noticed that JSON decoding is one of the most useful patterns for multi-inference runs, and I think we've invested a lot into offering a best-in-class JSON mode, both in terms of reliability and speed, and if you look at all of this together, I think what it means is that substrate is really a way to enable higher quality outcomes with AI, by letting you work in a system that's more flexible, more legible, more verifiable than any of the current paradigms that sort of exists now.

I think there's a lot more to say, there's all the time I really have today, it's only five minutes, but if you're curious, please come out and say hi on the expo floor. You can scan this QR code and get some credits and go to the website substrate.run or give me an email at rob@substrate.run.

Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Awesome. Thank you so much, Rob. And now we're excited to welcome Vik Korapati, who is the CTO at M87 Labs, to talk about Moon Dream. Hi. Can you all hear me? I can hear myself.

Hi. My name is Vik. I work on a model, an open source vision model called Moon Dream. A little bit about myself before I dive into Moon Dream. I was at AWS for about nine years before I started working on this model. Looking at where the stock price is going, I'm not sure if that was the right financial decision, but I'm very happy with the work I'm doing.

So let's dive into it. I'll talk about Moon Dream a little bit. It is a tiny vision language model. It's less than two billion parameters. So it can run anywhere and it's open source, Apache 2.0. So you can use it to do anything. Here's some examples of things you can do with Moon Dream.

You can ask it questions about images. You can caption images. It can detect specific objects inside of images. So here I asked it to tell me where the peak is and it gives me coordinates. I can count stuff. You can do all sorts of things. I had the audacity to title my talk, how can a tiny vision model slap so hard?

So I have to back things up a little bit. And so here's me doing that. These are two vision benchmarks, vision question-answering benchmarks. One's called VQA V2. The other is called GQA. As you can see, Moon Dream has been steadily improving over the relationship. I've been improving over the releases I've made over the last three months.

I've included a reference line over there for Lava 1.5, which is a popular 7 billion parameter vision model. So this shows you that Moon Dream gives you performance that's comparable to models that are about four times bigger than it. I didn't really set out to build a vision model, so I kind of got roped into it.

I was originally trying to build an application that required an AI agent, so I needed to be able to see what was going on on the user's screen and have it describe what's on the browser page for QA testing automation. I tried to do this at first with GPT-4V, but there were too many safety refusals back then.

Like, if there was any human being present in the image, I would just refuse to process it. It was also going to be really slow and expensive, and so I realized that if this is a product I'm trying to build, I really need to have control over the model itself.

So I figured, you know what? How hard can it be? Let me just go try and build this model myself. Now, the task I was trying to perform here was fairly constrained. I just needed to describe screens and answer questions about screens. So it doesn't need to be generally intelligent.

I had a couple of 3090s at home, so I figured I'd train a small version of the model at home and then rent some beefier machines in the cloud to go train a bigger version. And once I got done training a small version, I was like, hey, this actually works pretty well.

So I posted it on Twitter. I thought, you know what? I might get 20 likes off of this, and then I'll move on with my side project at the time. It blew up far beyond expectations. I was a little surprised, pleasantly surprised, but surprised nonetheless. And I immediately started seeing other automated testing companies reach out and be like, hey, can I use this to describe browser screens?

Because this would work really well for us. As well as other companies, shout out to our friends at Open Interpreter from Seattle. That basically told us that they were -- I figured, you know what? Like, this is getting a lot of traction. Let me pause on the whole automated testing app for a couple of weeks and focus on Moondream and see where it goes.

Yeah, so let me dive into a couple of the technical details from what makes the model succeed despite being small. The first thing we did that I think really helped was deciding what problems the model should solve and what it should not solve. So Moondream wants to be a developer tool.

We focus on being really accurate and not hallucinate. It doesn't really have a lot of knowledge about the world. So if you ask it to write a poem, it's probably not going to help you. It's really focused on answering questions like giving you -- helping you understand images. This is really important because it affects the type of data that you use and the sort of benchmarks that you want to focus on.

There's a popular vision language model benchmark called MathVesta which measures how good models are at solving math problems. You take a picture of a differential equation and you see whether the model can solve it. That was an example of a non-goal for us because we just want the model to be good at looking at images.

The most we do is probably generate a LaTeX representation of the problem. We don't really want to even attempt to try and solve calculus. It was not pre-trained from scratch. We use -- we've used a vision encoder called Siglip from Google with a pre-trained text model called PHY 1.5 from Microsoft.

The notable thing over here is PHY 1.5 was also trained on mostly synthetic data which is very similar to our PHY 1.5 we will. For this sort of task, pre-training from scratch doesn't really make a difference as opposed to using pre-trained models and it is cost prohibitive. So unless you want to get those brownie points for saying you trained it from scratch, it's probably not worth doing.

We experimented with a bunch of different other models as they were released and nothing really made too much of a difference. What does make a difference though is training data. The latest release of Moon Dream is trained on around 35 million images. And the problem is especially when you're on a budget, like high-quality multimodal training data is really hard to come by.

There's companies that -- there's a lot of companies out there that will annotate data with humans. But it's really expensive and I've heard a rumor recently that they won't even talk to you anymore unless you're willing to sign an upfront seven-figure commitment. There's a lot of data on the internet, images, alt text pairs.

The problem with this is it's often not in the format you want it to be and it's really noisy and the noise is really problematic when you're -- when you're training small models. And so, synthetic data is a way to solve this where you use that alt text information and process it.

It's a bit of an open secret that a lot of people are training on outputs from GPT-4. You probably don't want to do that. Besides being questionable in terms of use, it's often not helpful. GPT-4 is a very powerful model. It has reasoning capabilities and knowledge that your small model is never going to be able to get.

And so, when you train it on GPT-4 outputs, what it learns instead is to hallucinate. It's going to generate plausible sounding outputs that include details that it cannot possibly memorize. And so, you end up in trouble. So, this is a little important. I'm going to go a little more technically detail for a couple of minutes to dive into how to do synthetic data.

So, bear with me for a sec. We'll pop back up. Here's an example of how not to do it. Cocoa is a data set. It has around 200K images. Each image has five short descriptions and a bunch of object annotations with, like, hey, there's a bicycle. These coordinates and whatnot.

And let's say you want to take those short descriptions and these object annotations and generate more detailed captions that include the union of all the information present over here. If you just naively call GPT-4 with this information, it generates this. It's not important to read all of it. But there's two important things to know.

The first is that it hallucinates. It says in the second paragraph there's a person near the right side of the harbor. I think there's, like, a person way back. There's, like, five pixels there that may be a post. It may be a person. We don't really know. That's because object annotations are bad.

But besides that, like, the model is also taking a lot of creative liberties over here. Like, saying there's five yatches standing out from the rest and whatnot. And so this is -- you need to do a little more preprocessing of your data before you feed it to the model.

Here's another example. There's a dataset from Google called localized narratives. The task annotators here -- annotators are given here is verbally describe this image. And as you're describing the image, hover your mouse over the part of the image that you're describing. So it's nice in that it encourages people to create really detailed descriptions that capture spatial positioning in the image.

So, for example, here it says the girl in the front is playing the guitar and whatnot. And spatial reasoning is something that vision language models typically tend to struggle with. I ended up having to build a fairly sophisticated data processing pipeline to get really good results with this. Not really important to dive into the details over here.

But the important thing to note is, A, it gets really expensive. Each image ends up being 20 LLM calls. And the LLM here is Mixtral 8X 7B. So it gets pretty expensive. But it was necessary. Training data is the biggest needle mover in terms of model performance. And because of this, I'd say we spent like maybe one or two orders of magnitude more compute on generating training data than actually training the model itself.

A couple -- so yeah, this particular data set we've open sourced. It's available on Hugging Face. Here's an example of the type of questions it generates for this image. There's an interesting question towards the end. What theory does the kid have about the existence of pleasure in the image?

I'll talk about that in a sec. But basically you want to generate a few distractor questions so the model knows to not always agree with the question that the user is asking. So yeah, a couple of the challenges involved in working with synthetic data. There was an interesting incident I had early on where a user was like, hey, I asked a relatively simple question.

Why couldn't the model answer this? And when I looked at it, it turned out that they didn't capitalize the first letter in their question. And the model had never seen anything like that during training. So like, what do I do over here? And so it's really important for you to make sure that your training data has the same rough distribution as your real-world query.

So ended up adding like an extra step where we artificially inject like capitalization issues and typos and whatnot into the model before training it. There's also this risk of what we call model collapse where your model has biases inherent to it. So for example, if you try to ask Mixtrot to generate distractor questions, hey, just generate a question that's completely irrelevant to the image.

It'll always generate something about dinosaurs and aliens. And so if you train your model on that, it'll instead learn to say, hey, if the question is about dinosaurs and aliens, always say no, which doesn't really help. And so you need to inject like some entropy into the process of generating synthetic data to avoid this.

In the case of synthetic captioning, you can do something like, hey, describe this image, but also consider the alt text on the image, which may be noisy, may be relevant, but if it is relevant, use relevant facts from that. And that tends to help a lot. All right. So popping back up.

There were a couple of important learnings I had over the last three months that I would like to share with all of you. The first was the community was really critical in this whole journey. Seeing that original engagement that we got from the Moon Dream employees helped me realize that, hey, maybe this is more valuable than that QA testing application that I was working on, because a lot of people have a need for this to build applications like that.

Coming from an enterprise-ish company, it's been really valuable. It's been really valuable. It's been refreshing to be able to just talk to customers directly. Like, send someone to Twitter DM and be like, hey, just saw you looking for this. What do you think? But it's also helped us connect with a lot of partners, mentors, and get a lot of support from the community.

Being open source was critical. I kind of didn't really have a choice over here because the competition was free, so what am I going to do? But when you're in the dev tool space, it is pretty important. Open source is important to a lot of developers. They would like to have the ability to run it in different environments.

It's also pretty important for a lot of enterprise users. In a lot of cases, they don't really want to run the software themselves. But having the option is very important to them because most enterprises have had situations where a vendor goes out of business or decides to screw them in some other capacity.

It's also been really critical for engagement for us. We've had a lot of people in the community help out, put it to different platforms, run the model in the web browser and whatnot. It's been very valuable for us. This one is a little controversial. I'm not sure everyone agrees with this, but I feel pretty strongly that safety guardrails should be implemented at the application layer, not baked into the model itself.

This was one of my learnings from my first attempt to build a QA testing application with GPT-4B. It made no sense for that application to reject pictures of any pictures that contain a human being. I understand why they felt it was important. Dev tools are kind of B2B, not B2C.

So it's important to make it easy for developers to decide what guardrails they want and implement it in their model as opposed to just deciding it for all users. I'm not saying this is not important at all. Kind of makes sense if you're trying to build an assistant to make that stuff right directly into the model.

But when you're building for developers, it makes less sense. Yeah, I believe pretty strongly now that tiny models are going to run the world. In computer vision, more so perhaps than in text models, efficiency is really important. In a lot of cases, you're really worried about cost because you're processing video and 30 frames a second at seven tenths of a cent per second adds up very quickly and doesn't give you a lot of room to work with.

But there's also situations where you're really worried about privacy or latency and therefore you want to run the model really close to where decisions need to be made. Which is not to say big models are not useful. I think they're very useful. I just think that we'll mostly be running them in our development environments, maybe for generating training data.

But the artifact that you're going to want to deploy is most likely going to be a smaller model. Another thing that was a little surprising to me was looking at the different things people were doing with Moongreen. There were a lot of people building net new applications that weren't possible to do before because the model can understand language as well as images.

But there were also a lot of people doing traditional computer vision things with the model. It's like, is there a person in the scene or is there something suspicious going on? Tell me where the bus is in this picture from a road camera. All of which was possible to do before we had transformers, like train a YOLO V3 model or whatnot.

The thing that was, yeah, the lesson I took from this was prompting is a much better developer experience than having to train a custom model. And so for a lot of developers that would be interested in incorporating vision into their applications, before they'd be like, you know what, it's not worth me spending two weeks learning how to like collect data and annotate it and train my own custom model.

Giving them the option to say, Hey, for fairly cheap, you can just in English describe what you want extracted from this image makes it something that they actually consider doing it doing now. All right. I think I'm a little ahead of time. So I'm excited to maybe do a live demo.

If the demo God smile upon me, we'll see. Um, in conclusion, uh, yeah, where's moon dream going? We're, we're not AGI people. I'm really focused on making it really easy for developers to build amazing applications with vision. Um, there's a bunch of model improvements that I'm working on right now.

Um, I'll talk about some, um, right now we use 729 tokens to represent an image. So you can only really send one image to the model at a time. Um, we're working on giving users the option to like give a more compressed representation to the, to the model. Um, which makes sense if you're not trying to read text or something from the image, if you're just trying to do classification and whatnot, that makes the model run a lot faster, which is important, especially if you're on CPU, as opposed to GPUs, which can't do as much.

Um, CPUs can't do as much parallel compute. And so that sort of thing ends up being really important. Um, we've also just raised a seed round, um, from Felicis, Ascend, and also the GitHub one, which I forgot to include in the slide. Sorry, GitHub. Um, this means more GPUs, but more importantly, it means I can finally get some sleep because we're able to get a couple more people to join the team.

Uh, if you're interested, please reach out. Um, we have a contact email on the website or just hit me up on Twitter. Uh, we also have an exciting release coming up later this summer that I'm super pumped for. So stay tuned. Um, I think that's about it. So I have a couple of minutes left, I think.

So I'm going to try doing something that may not be the wisest idea, but we'll see how it goes. All right. Uh, I've done the wifi off. This whole thing is running locally. Um, so what's this is going to do is like, uh, start taking my webcam in and it's going to use moon dream in a, in a finite loop to describe what it sees.

And we can ask it different questions. So we'll see how that goes. Uh, yeah, you can ask it different things. Uh, so let's say, is the person wearing glasses. Um, you do have to tell the model to answer briefly if you want a yes or no. Otherwise it gives you like a, um, answer with a single word.

Let's try that. Yes. Okay. I'll take them off and I can't see. Did it get it? Let's do that. I'll go back to the old prompt. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Let's do that. I'll go back to the old prompt. All right. Well, that was it for me.

Thank you all. Thank you all. Thank you so much, Vic. We are super excited to welcome our next speaker, uh, Ben Hilack, who is one of the founders of Dawn analytics. And he's going to be talking about the era of unbounded products. Hey, all, um, I guess actually I have this on, I don't need that.

Uh, I'm so excited to be here with you guys today. And I think what is, uh, probably the coolest, uh, AI conference, uh, in the world at such an exciting, exciting time in history, I think, especially for AI products. Uh, if you don't already know me from either demos on Twitter or sometimes probably ill-advised spicy takes on Twitter.

Uh, my name is Ben Hilack and, uh, I'm the founder of Dawn. So at Dawn, we help some of the best companies in the world. Everyone from GitHub to can of soup build better, more predictable AI products. Uh, my entire life I've been like really obsessed with building and designing unbounded products.

So unbounded products are products that transcend the, uh, the, the mouse, the monitor in some way, right? So for me, that started with robotics. Uh, when I was, I think the first one was when I was in eighth grade. Uh, eventually rockets, uh, at SpaceX. So these are very, these are very unbounded products, right?

Um, and then most recently, uh, I was on the design team for the Apple Vision Pro for four years. So we designed the first version of Vision OS. I think that AI makes products less bounded than they've ever been, right? you can type, you can talk, you can show images or show video just like we just saw.

You can also sort of plead, you can bargain, you can confide, right? These are very interesting sort of input modalities. And this unboundedness often makes products unpredictable, right? Confusing, hard to understand. Users assume your product can do things that it can't. They try to do those things, doesn't work.

And they walk away thinking that it can't even do the things that it can. When you talk to people, and specifically people that are not in this room, how they use ChatCBT, how they learn how to use it, it's often word of mouth, right? So they hear one of their friends say that they used it for travel planning.

And then they go use it for travel planning. A lot of us, a lot of us in this room, especially like people that are more technical, we often learn through trial and error, right? So we just keep trying, keep trying. We keep trying because we know that these models are good, right?

We know that it's impressive. But a lot of people are not. They don't do the trial and error thing, right? So they try it once, that doesn't work. They don't try again. And so this talk is about making good AI products. And to that end, I'm going to cover just three things.

So those three things are the past, right? So how have products become more unbounded? And what has worked for unbounded products in the past? The present, which is AI products today. What are sort of good design patterns and bad design patterns? And then the third point is going to be the future, right?

So again, just three things. Just the past, the present, and the future. Easy. So we're going to start with the past. So most software that we use lives on a screen, right? And you use it just by typing-- sorry. You use it primarily by swiping, clicking, and tapping, right?

When you click something, whatever the developer expected to happen is what happens, depending on how good of a developer you are. It's easy for users to understand what your app can do. They look. They see the buttons. They get it. It's also very easy for you to understand what your users are doing.

You just add an amplitude or mix panel call on a button press. You see what they did. So if you think about one of the biggest changes to this, previous to the last two years, was multi-touch, right? And this is just like, instead of one pointer, you have two.

But just by adding that second pointer, you get relative distance. You get rotation, right? And just this one little change, like, largely made the smartphone possible, right? Like, largely made it easy to use a screen that is that small. And now it's just getting crazy, right? It's like, we have unbounded products everywhere.

The products are so unbounded. You have software, you know, just freely roaming the streets of San Francisco, getting attacked by fiery mobs, right? So this is getting crazy. And so I want to talk a little bit about just one unbounded product that I got to work on, which is the Vision Pro.

And what I want to talk about is just three lessons that we learned while we were designing it. Lessons that I think aren't as intuitive looking from the outside in. So I think that unbounded products are often defined by this what if question. Like, when we were starting, it's like users get themselves into the craziest situations.

So something as simple as, oh, well, what if someone's in the living room and then they move to the bedroom and they lay down on their bed, right? What should happen to your apps? If you're designing Mac OS that's on a laptop, you don't have to worry about that.

But that's something we had to think about. And there's hundreds of more questions like this, right? What if someone's on a plane? What if someone's next to their friend? What if someone has a disability of some sort, right? Like, they cannot move. They can't move their neck. They're bedridden.

So all these what ifs. And I think this is, again, what defines unbounded products, right? All of us that are building AI products, we're constantly thinking, you know, oh, like, what if someone puts in this? What if someone puts in that? And there's evals, et cetera, et cetera. And so without structure, you just have chaos, right?

You have a blank slate. You have all these what ifs, infinite world of possibilities. And so it's really on us as product designers to add structure. And structure is what creates clarity. So again, I want to talk about three ways we added structure. The first was highlighting what matters and doing it really fast.

So the first thing you see in Vision OS is a home screen. It has apps, it has people, and it has environments. So those are the things that we think matter when you're using Vision OS. So they're the first thing you see. It might not sound that novel, and it's not.

In a lot of ways, it's the same thing that happens on your iPhone. But when you compare it to VR products that came before, it's very hard to understand how you're actually, you know, what is this thing good for when you look at this menu? The second point is hierarchy.

Hierarchy is what gives unbounded products a shape and a purpose, right? It's what helps users understand what it's good for, what they should use it for. So again, we have the home menu. That's kind of where everything starts and ends for Vision OS. We have windows. They have bounds.

You can resize them and move them. And any individual window can go full screen, right? So that was our hierarchy. The last point, which is really important, and I think the easiest way to make unbounded products feel familiar is -- sorry, intuitive. I got ahead of myself. It's familiar.

Familiarity. It was something we hit when we were building Dawn. Our first kind of prototype was this star cluster thing that you could explore. It was really fun. Nowadays, it looks a lot more like this, which are, you know, tables. And we have graphs and examples. Again, it's just structure, clarity.

And I think that it's no accident that, you know, the TV app on Vision OS looks a lot like the TV app on TV OS, right? It's not an accident. It wasn't laziness. When people are sort of in uncharted territory, you want to give them as many signs of home as possible.

Same thing for Control Center, right? When people see Vision OS, they already know how to use it. And so, again, these three points. Highlighting what matters, bringing that to the forefront. Establishing hierarchy. And then leveraging familiarity. All right. All right. And so that was the past. Now we're going to talk about the present.

And specifically, we're going to talk about AI products. We're going to talk about ways, both good and bad, that products have been incorporating structure into their AI features. It's really important to note the right structure is very unique to your app, right? That's the whole point, is that it gives your app a shape.

It helps your user understand what it's actually for. So let's take something like Dot, right? Dot is a companion. Dot is sort of a journal, at least for me. And so the structure they added was that if you pinch out, you can see each day separated, right? It feels a lot like a journal.

And if you tap a person or two people, in this case my co-founders, I can see this, again, structured information about them and a timeline every time I mention them to Dot. And so again, you're pulling that structure out of the chat. Perplexity does a really good job of using structure to make their experience feel more like a search engine and less like ChatCpt, less like a chat you're having a conversation with, right?

And they do this by, you know, really pulling your title, you know, your query up top as like a title, you know? Highlighting the sources it came from, and then having the answer below that, right? And then having that take up the full page, kind of regardless. So it makes it feel, again, like more like one shot, less something you're having a back and forth with.

Now I want to talk about sort of an anti-pattern I've seen, which is this is in the Vercel chatbot demo. I think Vercel does some of the coolest design work in the entire world. I didn't like this one. So this is like having this idea of almost ephemeral UI, but inside the flow of chat, right?

And I get the appeal, right? So actually if we go back here, sorry, this was a video I want to show. You know, you have a slider, right? So instead of having to like, you inquire about, you know, you want to buy Doge. And it shows you this UI, so you can adjust exactly how many instead of having to do it over text.

It's, you know, could be good. The problem is that when it's stuck inside this sort of unstructured thing, it starts like floating away as I try to ask follow-ups, right? And then at some point I even have two of them, right? So I go back up to the first one, I press purchase, and now I'm interacting with something that's completely different.

So it reminds me a lot of sort of the house and up, right? It's just kind of up, up, away. So instead of trying to put structure stuff into this unstructured thing, I think the answer is you pull it out, right? You pull it off to the side. And what that means is that as the conversation continues, you can just sort of update that structure.

you can do it without disrupting where the user is. And that's exactly what Claude did with Artifacts, right? And I think why it's so successful is that they pulled out the structure, which is the app you're working on and iterating on, from the actual conversation. And then so as you make changes, you can even go between the versions here without even having to scroll the conversation, right?

So it's beautiful. And it actually brings us to another thing that I think has been really effective for AI apps, which is this almost concept of version control. So this was actually one of the, like, shipping, like, original ChatCBT features, which is kind of crazy. But if you edit a message, you can go between the versions, right?

And it actually maintains this entire tree. It's very complicated. But it's super powerful. With V0, Vercel did something, again, amazing, where it feels extremely familiar, almost like you're working on Google Slides or something. But you can go back and iterate, keep iterating on UI without having to be afraid that you're losing something, right?

So again, versions. Again, I think familiarity is really one of the most important things for unbounded products. I think Claude did an excellent job with this. Again, I'm hyping them up here. But ChatCBT introduced memory across all of your chats, right? Completely unbounded. So when I tell it something about, you know, some sort of medical problem as I'm working on a JavaScript, it's like, you know, it knows that, which is very weird to me.

I think this idea of projects and that structure of a project is very familiar. So sharing context across a project makes more sense. Agents are something that are extremely unfamiliar to most people. And this idea of having, you know, all these different tasks and you're feeding data between steps, whatever.

But you know what is familiar are spreadsheets, right? Spreadsheets are extremely familiar to, not to me actually, but to a lot of people. And I think the only real uses of agents I've seen in the world, in the real world, are spreadsheets. So this is clay, right? And each column is essentially a step that an agent is taking, the user is defining.

So it's going across, building up kind of context across the spreadsheet. Each row, it's often almost, you do it like an eval, right? So it's like you run the first 10 rows and then you run the next 50,000, 100,000, right? So you get it right. And you can see here, eventually you end up with a personalized email as the last column.

But with all these steps in between. The next thing that I think is extremely effective in helping people understand what your app is for and skipping all the sort of noise of prompt hacking, prompt engineering, are examples and presets. So, ChatTubT, I think, was the first for this, where they had these, you know, message to comfort a friend, plan a relaxing day, and so on.

V0 does an awesome job with this, right? We're not just having those suggestions below, but they also have an explore page where you can see what other users are doing, what's actually working, right? Again, try to like shortcut this like prompt, you know, blank canvas problem. Notion as well, right?

They have a simple menu where you can change tone for text instead of having to like be like, you know, you are a very concise GPT, whatever, whatever, whatever, right? So you're just using these tried and proven things that Notion can validate. And that last point brings us to the future, right?

So where are interfaces going in the future? Linus gave an awesome talk last year where he described prompt engineering as this almost trying to drive a car. A llama trying to drive a car with a pool noodle from the backseat was I think his metaphor. And there's some real truth to this, right?

And so I think, first of all, the future has a lot less prompt engineering. And we're already seeing this, right? We're already seeing this with generative images, you know, the way that Apple designed it, where you're mixing and matching these different concepts. You're able, you know, there's a ton of demos on Twitter of people, you know, essentially you're going between emotions here in a more intuitive way.

And then just yesterday, Figma released this way of adjusting the tone of text, right? Right? Where you're going between professional, casual, expanded, concise. The problem with this is that casual means a lot of different things, right? Casual for a Fortune 500 company and a, you know, direct-to-consumer cosmetics brand, you know, with ads on TikTok, right?

These are very different things. Casual when talking to your best friend or a coworker, these are different. They're different. So how do we avoid being reductive when trying to offer these sorts of presets? And the answer is you just like, I don't know exactly how many zeros I put here, but you just like million X or billion X the number of presets, right?

So you have enough presets for everything. And I think sparse autoencoders show a really promising path towards that. So if you guys have tried Golden Gate Claude, where you can kind of identify the one feature of Golden Gate Bridge-ness and amplify it. And it makes Claude obsessed with Bridget-- Golden Gate Bridges, specifically, or the Golden Gate Bridge.

My friend Giddes has an amazing demo towards this, but for manipulating images, right? So you can see here, he's increasing the amount of play of light and shadows, increasing the amount of serene forest streams or Venetian canals, in, again, a very controllable and predictable way. Okay, but so now we have a million, billion options, whatever.

How do we avoid too many options? I think this gets to point three, which is ranked presets. So these are presets that are personalized, searchable, and even invoked through natural language. They might not even be directly visible to the user. So the user types in something like more friendly.

And you pull up the corresponding presets, like kindness, how close you are, how confrontational it is. Again, maybe they're directly editing it, maybe they're not. And this gets to the last point, or second to last point, which is developer-defined personalization. So as soon as you're able to define those sort of features, you can start tuning them per user.

So each user, in a way that you can't do with just text prompts today, right? Because text prompts are sort of this fragile house of cards, where if you remove one word, the whole output changes. So you're able to tune it per user. So you're able to tune it per user.

And the last point, and especially true, as you start, your app is going to become increasingly different per user, is shifting from evals to analytics. I don't think there's going to be some objectively correct, for a lot of domains, answer to things. Like, who was the first president, yes.

But the right sort of tone for a summary for a specific user, I don't think so. And so I think that increasingly, it's going to be about how do you understand if you're meeting the needs of your users, and what they're asking for. So, that's it. Thank you so much.

Oh, yeah. We'll skip this one. And thank you so much for coming. And, yeah. Awesome. Thank you so much, Ben. And now we're excited to welcome Curran Goyle from Cartesia to talk about state space models. Can everybody hear me? Awesome. We'll see if I can stay on track, 18 minutes.

So, great to be here. I'm Curran. I'm the CEO of a small company called Cartesia, which is an eight-month-old company. And I'll tell you a little bit about what we're working on and try to describe some of the challenges that we see in sort of multimodal intelligence emerging and how we plan to fix them.

And hopefully actually show a demo as well, if Wi-Fi works. So, maybe to set the stage a little bit. The last four or five years of AI have basically been really focused on this idea of batch intelligence, which is sort of pretty core to this idea of building, like, an AI system that can reason for long periods of time on a problem and then solve it.

So, you can think about math problems or physics problems that are hard. There's a lot of applications where actually what you need are systems that are streaming. So, they're real-time. They work instantly. So, imagine generating video, audio, or doing, like, understanding applications on sensor streams, etc. So, it sort of bifurcates where there's these two different types of applications, similar to how there's, you know, generally this idea of having batch workloads and streaming workloads.

And so, a lot of what we've seen over the last few years has really been focused on batch APIs where you call a model in the cloud, it takes a few seconds, and then you get a pretty good response back. And now we're seeing some shift towards more real-time applications where you constantly will be querying a model and asking it to return responses at low latency and then using that to sort of interpret or generate information.

And I think this, you know, this area is really exciting because it's going to be transformative to a lot of interesting applications that have so far actually not necessarily been the main focus for a lot of what we've seen over the last few years. So, conversational voice is an example of this where you should be able to interact with the system and then talk to it.

And it should be able to understand you and do all kinds of tasks on your behalf. This is similar to having assistants that are on device and run kind of really efficiently at low power at, you know, all times, regardless of whether you're on your phone or laptop. And then things like world generation where, like, you can imagine actually playing a game that is generated in real time, similar to how the graphics are rendered on GPUs.

And all of this, you know, should be able to happen in real time on low power, on your phone, on your MacBook, et cetera. Robotics is another great example where it sort of culminates with all of these coming together on a, on a single device that is trying to kind of interpret everything in the world.

And so I think this is sort of the exciting intersection, which is like, how do we make intelligence faster and cheaper so that we can put it everywhere, basically. And a couple of examples that are really powerful, real time intelligence for conversational interfaces is going to be really interesting because you would be able to have a, agent that can provide customer support for a problem, answer questions about health insurance, you know, call your vendor to pick up a shipment.

All these coordination tasks that generally are annoying to do should be really automated and real time intelligent agents should be doing them. And then humans can spend their time solving sort of harder problems that are more interesting. And in customer support that could be dealing with, you know, the tail customers that are much more important because they're pissed off or they're more important because they have, you know, more customer value, et cetera.

And similarly in robotics, there's this idea of like ingesting similar to humans, like audio, video, sensor data, and then responding instantly to a lot of these pieces of information. So I think this is sort of the world we should be living in where all of these intelligent models run super fast.

We saw all of these different problems and they're able to really kind of power these new experiences that are interactive at their core. So this is where we come in. We're building, uh, these real-time foundation models. So, uh, some of what I'll talk about is, uh, the work we've done in really building kind of new ideas for how you can create, uh, deep learning models.

So, um, I did my PhD before this. I was working with a lot of these folks for my PhD. Chris was our PhD advisor. Um, and, um, we were really focused on this idea that you should be able to have a model that can compress information as it comes into the model.

Um, and, um, use that to really kind of build powerful systems that are streaming at their core. Um, and I'll talk a little bit about this, but that's really the technology that we've been working with for the last four or five years. We've been developing academia and some of you might have heard of things like Mamba, which is sort of a more recent iteration of this technology.

You know, I did my PhD working on some of the early iterations that nobody uses anymore, but, um, are sort of the precursors to a lot of the modern stuff that is now more widely used. And, uh, and now what we're doing at Cartesia is basically taking this and trying to understand how we can improve it.

How do we push the boundaries on, uh, what architectures can do? And, um, and I think it's an interesting question because, you know, we should not settle for having a, uh, one way of doing things. I think that's sort of a, a poor way to kind of, um, think about the future.

So our approach is sort of like, let's think about new ways of actually designing models that aren't necessarily built on, let's say the transformer architecture and the standard recipe for deep learning. That's, uh, you know, prevalent today. And I think it boils down to this question of like efficiently modeling long context is a huge problem because, you know, a lot of practical data is really long sequence data.

I think text is maybe the least interesting, uh, long sequence data because text is actually fairly, uh, compressed already, right? Like you have a lot of information that is, uh, embedded in two minutes of, uh, or, uh, two sentences of text, but there's all these other domains where, you know, audio, video, et cetera, where there's so much information.

Um, you know, imagine looking at a security camera for a day, like you would probably have just so much information coming into the system and just, um, very little of that would be useful. So compression is kind of really fundamental to intelligence because we're able to do this where we can look at all this information and then sort of compress it down to whatever's necessary to remember or understand.

Um, and I think so far what we've seen is that the AI systems that we built have not necessarily exhibited that same behavior. So they're really kind of built not on the principles of compression, but more on this idea of retrieval, like keeping all the context around and then using it to reason over all the information that you've seen.

So I think, um, our kind of point of view is that multimodal AI will remain challenging as, as long as you're sort of, uh, working in this paradigm. Because, uh, if you try to think about what humans do in a year, um, you're basically processing, understanding about a billion text tokens, 10 billion audio tokens.

These are, you know, back of the envelope calculations that I did and about a trillion video tokens probably underestimates how much video we process. And not including all the other sensory information that you're processing and you're doing it simultaneously and you're doing it on a computer that fits in your brain.

Um, and you, you know, sometimes don't eat and drink and, you know, you're still functioning fine. So, you know, you can have variable amounts of power in the system. Um, so I think the idea that like intelligence is solved is sort of very far from the truth because humans just are an extremely amazing machine that, uh, does something very extraordinary, um, in a very compressed way that, um, our AI models can't do.

So, I think that's sort of our, uh, you know, sort of, uh, the reason we get up in the morning is we think about this and we're like, yeah, we're very far away from where we should be. Um, and the best models today are in the, you know, 10 million, 100 million sort of token range.

Um, so, uh, that's really good. A lot of progress has been made, but really this is sort of what we aspire to is how do you kind of build these machines that are long lived that can actually understand information over very long periods of time. And I think the cool thing is like, as a human, you can remember things that happened 30 years ago with very little effort.

You don't need to do rag or retrieval or anything. You just, you know, you remember it, it's just it in your brain and then you figure it out basically. So I think that's kind of an extraordinary capability that we should be able to put into our, uh, AI models as well.

And so, uh, some of the big problems with models today are, uh, you know, they're built on transformers, really optimized for data center. I think, um, we see this with like a lot of the work we did, which was on sub-quadratic models. So, quadratic scaling and context length really just means that, um, you know, the amount of, uh, computation you have to do to process long amounts of, uh, context is very large.

And so right now the sort of predominant approach is to throw compute at that problem and then hope that that would scale. Um, obviously compute is a very important piece of the puzzle because you do need more computation to be able to do more difficult things. But, um, this type of approach because of the quadratic scaling actually has core scaling with, you know, very large multimodal contexts.

And text contexts tend to be shorter. Multimodal contexts will get larger because you have just way more tokens and information that's going into the system. So that's going to be a big challenge for these models, especially how do you do this inference efficiently? So you're not, you know, burning down the data centers to, you know, do, uh, a fairly limited amount inference.

Like you have to imagine that we're doing a thousand times or, um, you know, a hundred thousand times more inference. And then, um, if the models are scaling the same way, it's going to be really, really, really expensive. So you're not going to be able to permeate all these applications that I talked about very easily.

Um, and so, you know, that's sort of a big challenge, I would say. And so, you know, again, our hypothesis is you need new architectures and that's kind of where we spend our time. And we want to make these models more efficient, faster, more capable while being able to handle all these long context problems.

Um, this is a slide about, you know, transformers being somewhat inefficient at handling this. Um, but obviously, um, a very good recipe for scaling, uh, uh, these models out. And so, you know, some of the work that we've been doing is new fundamentally efficient architectures. So they have compression at their core.

So they sort of, the way they operate, uh, I have a slide on this just to give you kind of a, uh, a quick illustration. But, um, they really scaled more linearly in context lens. So you should be able to have, uh, because of this, like more low power implementations of these models.

Um, you can compress information as it comes into the system. You have low memory usage, um, and you can actually scale to much more massive context because of that. Um, and this is all the work around SSMs. I just threw this, uh, nice slide, which I thought was cool.

Um, Jensen had an interesting quote about SSMs in one of his Wired articles that I like to keep talking about. But, uh, but I think it's a cool technology that has a lot of potential and sort of that's where we're, um, spending a lot of our time. And if you folks are interested in reading more, there's lots of videos on YouTube and lots of sort of resources that try to make this more accessible to understand and kind of get into some of the details.

Um, but you know, the working intuition is basically, uh, transformers are generating quadratically by attending to every past token of information. So as tokens come into the system, you're kind of sort of keeping them around and then looking at all the past tokens. So if you want to generate the word jumped, uh, from the quick brown Fox, you would actually look at the entire context, try to understand what the next word should be.

And then, um, generate it, push it into the context, do it again. Uh, with SSMs you just have a streaming system. So you, you have a, uh, token stream in, uh, they update an internal memory for the model and then the token gets thrown away. So that actually really simplifies the system.

And that's why it's such a core sort of streaming interface because you're just not keeping all this memory around about what happened in the past. You're compressing it into some sort of zipped file, uh, state inside the model that's gonna be used to do a future generation. And so this is sort of taking this idea of, uh, taking advantage of this idea of recurrence, which is sort of core to how even humans, uh, do a lot of their raising.

And, you know, last few months, a lot of these models have been getting adopted. So it's great that, you know, a lot of folks are now excited about the, uh, this, you know, alternate way of doing things that is much more, uh, sort of, sort of are oriented around this idea of recurrence rather than, um, retrieval.

Um, and so I think like, we'll see a lot more activity in this, especially with multimodal data becoming more important. And, uh, you know, a lot of the challenges multimodal data run efficiency will mean that I think that these models will have more of a role to play in the next three to five years.

Um, as we also do our work in, uh, scaling them up and making them more interesting. A lot of people ask me about quality. Uh, I only have a few minutes, so I'll go through this, uh, the rest of the slide super fast, but, um, you know, Um, you know, SSMs generally have the right, uh, quality.

Obviously there's a tradeoff between compression and keeping all of the information around. Uh, but actually like some pressure can be helpful. So if you imagine the security camera example, if you're watching 24 hours of footage, actually compressing all of that information on the fly would help you solve tasks and answer questions better.

Uh, rather than looking at all 24 hours every time. So I think that's sort of the rule of thumb to think about, which is compression super helpful for a large context. Uh, not as helpful for, um, short context. And so we see that quality actually is very good for long context problems and multimodal problems.

Let me talk quickly about some of the work we've been doing. So we've been starting to work on sort of multimodal data. And we did a release a few weeks ago for a voice generation model. So this is sort of text to speech and sort of in line with some of the work we're doing to bring more multimodal, uh, data into a single model.

Um, and use SSMs to power the inference and the training and so on. So this is a model you can actually play with. I'll try to show you a demo, but, um, one of the things we're proudest off with this model is that we really shrunk the latency down.

So when you play with it on the playground, you get instant voice back generated from the data center. And there's some cool work we're doing to actually run these models on Mac, um, and other devices so that you can basically have the same experience as you have in the data center, but on any device and, and do that efficiently and at low power.

How much time do I have? Okay. We're out of time, but I was also almost done. So, um, go to the website, play.cartigia.ai. I unfortunately couldn't walk through the demo, but, um, play with it and, um, send us feedback. This is my email in case you want to, uh, send me a note.

Uh, would love to hear feedback and anything that you folks, uh, find interesting. Thank you. Thank you. Thank you. I'm ready to go. I catch a laughing and I'm ready to go. You're holding my mask, stricken at night. My soul, I'm diving and I'm ready to go. Holding my breath and I'm ready to go.

I'm going right in and I'm ready to go. I found what I want and I know that we're on top. So I'm diving and I'm ready to go. Holding my breath and I'm ready to go. I catch a laughing and I'm ready to go. You're holding my mask, stricken at night.

My soul, I'm diving and I'm ready to go. I'm going right in and I'm ready to go. I'm going right in and I'm ready to go. Oh! *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* *music* Thank you.

Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.

Thank you. Thank you. Thank you. Thank you. You and me now Holding my breath

AI Engineer World’s Fair 2024 - Keynotes & Multimodality track

Chapters

Transcript