tldraw.computer - Steve Ruiz, tldraw
00:00:00.000 | My name's Steve, Steve Ruiz. I am from a company that I started called TealDraw. TealDraw started
00:00:21.240 | as a, well a couple things, started as like a digital ink library that then Christopher had
00:00:28.680 | me implement in Excalibur when I was working on that I was like you know there should probably
00:00:32.960 | be like a kind of a really good SDK for building these types of things. And I'd already done
00:00:38.040 | a couple of projects that were kind of going in that direction so I did. It turned out if
00:00:44.860 | you build a canvas that other people can use people will build cool stuff with it. So today
00:00:52.340 | I'm going to be talking about some of the stuff that we've done with AI using our kind of our
00:00:57.360 | own toys playing with our own canvas here. So I'm here in TealDraw.com, it's a free whiteboard.
00:01:06.460 | You can come in, use it, make your diagrams, make your slides. Very, very similar in use
00:01:12.040 | case to Excalibur actually. But there are a few things that are kind of special here and
00:01:18.140 | I'll show you real quick. So again this is TealDraw.com, free end user whiteboard application. And we also
00:01:28.240 | have TealDraw.dev which is the SDK website. If you wanted to build stuff with TealDraw then you could go
00:01:35.520 | to TealDraw.dev and learn all about the the code and the documentation and how to do that. The cool
00:01:41.240 | thing about the canvas is that it is. Well, I'll skip this one for a second. It is just normal web
00:01:47.000 | stuff. It's like react all the way down. So for example, I can do like things like, you know,
00:01:51.340 | play YouTube videos and, you know, still interact with them, still draw on top of them. But the yeah,
00:01:57.620 | every one of these little shapes including like doing some pretty cool stuff like, you know, I have a whole
00:02:03.340 | code editor here. This is just code sandbox that's embedded in TealDraw.com. This one is Figma that
00:02:10.960 | like just is embedded in TealDraw.com. Even if you really like Excalibur, you can even use Excalibur
00:02:17.440 | inside of TealDraw.com. So, and I'm pretty sure, I hope this doesn't break my slides, but if I paste
00:02:24.520 | the own, the TealDraw inside of itself, then we can, we can kind of, let me see if I can draw
00:02:32.440 | inside of TealDraw. Hang on a second. Yeah, right. We're kind of modifying the inner from the outer,
00:02:44.320 | whatever. I'll let you think about how that works. But yeah, and it has a lot of like kind of little
00:02:49.900 | details. I'll do this really quick of like, you know, nice arrows that just, you know, perfectly
00:02:53.860 | kind of follow the different shapes of things and, you know, boxes where the, you know, the corners of
00:03:02.560 | the boxes always stay in the corners. That's right. So that's part of our value propositions that we take
00:03:07.720 | care of all these little like little details, make sure the corners are right, make sure the arrows are
00:03:11.400 | right, stuff like that. We did a couple of different AI stuff on top of this. And some of these are going to
00:03:19.660 | work. Some of these are not going to work. Did we find out is this is this fell in the room here?
00:03:25.540 | The okay, well, in 2023, we had a lot of success with with make real, I'll skip this one for now.
00:03:36.160 | Make real was the idea that people were using TealDraw for whiteboarding, as well as what you're like
00:03:44.680 | drawing wireframes. And the idea would be like, well, what if we could take the
00:03:49.620 | diagrams that we were drawing the wireframes that we were drawing, and we could just kind of kind
00:03:53.280 | of make them real, right? What would be involved in that? And so we when the when the vision models came
00:04:03.000 | out like GPT-4 with vision. That's annoying. I'll have to do it myself. We, we realized you could
00:04:11.880 | just send a screenshot to continue, boom, to the model and say, hey, model, you're a web developer,
00:04:19.140 | your designers just gave you this lo-fi thing. Can you can you create a higher like, can you actually
00:04:23.640 | prototype this? Can you build it? And the models could do that really well. As usual, I'm going to kind of like,
00:04:29.760 | give this a second to load while we are all right, they're really running with this input. All right,
00:04:35.820 | well, the models have since become very, very ambitious. Here's another good one. Let's say I
00:04:43.140 | want to have a stop motion application where like, I have a feed from my camera. And I want to be able
00:04:48.880 | to take pictures. And I want to be able to like, see all those pictures there. But I also want to be able
00:04:54.060 | to play them like in series. Using only the the input here, right? I won't even do the title,
00:05:01.440 | just just to be to be fun. The model will will spin off on that. And it will eventually, we can kind of
00:05:08.880 | watch it generate, but it will eventually come up with this. I just did this during the last talk,
00:05:13.680 | where that's my app, you know, I can kind of do this is doing the onion skinning. And there's my gift,
00:05:22.860 | right? And not surprising. I mean, you can add images to cursor and stuff like that. And it just
00:05:28.500 | works really well. But the fun part is because I'm going to stop this. Because this is back on the canvas,
00:05:35.820 | you can actually annotate on top of the website, and use that as the next prompt, you kind of click
00:05:42.300 | this kind of kind of just generate the next one. And I've done this already. But you can see that yeah,
00:05:47.640 | sure enough, it made the button solid, like I asked it to. And so using these drawing tools as a way of
00:05:54.660 | not only generating stuff, but annotating and like kind of iterating through these, you can get some pretty,
00:06:00.720 | pretty wild results. This came out at the end of 2023. It was one of the first kind of tools that
00:06:08.820 | let people that couldn't program and couldn't create software to kind of do it. And it was,
00:06:14.700 | it was pretty remarkable. So like this, this being the input, leads to, you know, leads to an app. You
00:06:23.700 | might have seen there the, that it just did a little flash of green, you know, there was just a bug involved.
00:06:29.880 | So I just took a screenshot of the bug and sent it together with the, the original source and said,
00:06:34.920 | hey, can you, can you fix that particular bug? And yeah, it did. So it's, it's pretty cool. The,
00:06:41.640 | um, if I don't crash my browser. Hey, all right, so that's make real. We also did this one called draw
00:06:49.320 | fast, which may or may not work. I'm just gonna see if it does. This used a thing called like latent
00:06:56.340 | consistency models. I think that's the name basically like a, uh, create an image for me as fast as
00:07:02.820 | possible. And we will see if I can wake up the, uh, the server here. Oh, Hey, look, Hey, this normally
00:07:09.480 | doesn't work special, uh, where you have a drawing, you have an image being created from the drawing. And
00:07:14.220 | I, as I changed the drawing, Oh, come on, do it. Uh, then the image is gonna, gonna change as well. Um,
00:07:20.700 | you can even take these things and flatten them like this. And now I can interact with the, um, the model,
00:07:28.780 | images like this and, you know, let's say I'm gonna rotate it or, or maybe stretch it out really big. Uh,
00:07:35.260 | uh, and in, in, in, in good circumstances, this stuff works almost in real time, but you'll have to, uh,
00:07:42.940 | I have to accept the, uh, uh, whatever, uh, the, the one, one moment of, uh, of working as the best we're
00:07:50.540 | gonna get. I'm gonna need to use two hands to do this, but no. If I just make a whole bunch of people,
00:07:57.580 | will they? Oh, cause they, they're running cause they're all sideways. Right? I got it. Anyway,
00:08:03.100 | this is draw fast. Uh, but the one that I'm going to talk about, uh, mainly is teal draw computer.
00:08:10.380 | So this is how, well, I'll just, I'll just do it. This is a kind of a graph full of these other little
00:08:17.740 | components. Uh, I am gonna say, uh, AI engineer, um, MCP observability, I don't know, whatever, uh,
00:08:31.660 | uh, uh, conference. Uh, and I'm gonna draw a picture too of like maybe a, uh, I'll just do it like a big,
00:08:41.660 | um, uh, top hat or something like that. I don't know, whatever, with some playing cards.
00:08:47.660 | And the, and the brim. Got it. Write a short commercial is the instruction here. Uh, I'll even
00:08:54.620 | do please, uh, and run it. Okay. So a couple of things are gonna happen all at once here. This graph
00:09:02.620 | is gonna execute. Right now, the instruction is creating a script for itself. And then it just
00:09:06.460 | executed the script. Sorry, this goes fast. Wrote the text. Now it's generating speech. It's also
00:09:10.780 | generating an image based on this. Um, each one of these blocks accepts inputs and produces
00:09:17.500 | outputs. So this image is based on, uh, our, our, our text, which was based on this, this instruction,
00:09:26.940 | which was based on these inputs. Uh, and then it's, it's, you know, creating speech right now.
00:09:31.500 | Um, and that's gonna be whatever.
00:09:38.460 | You got it. And then I can, I can keep piping it on and it'll, you know, but this time it'll make
00:09:45.020 | it sad and serious and create an image based on that. Right. So we, this is cool. Um, the,
00:09:52.060 | each one of these things, like I said, has this script of like, how should I use my inputs? What
00:09:59.740 | should I produce based on my inputs? So for this write a short commercial, it's something like it's
00:10:03.820 | tiny. I'll, I'll read it. Analyze inputs looking for guidance on the product services style or other
00:10:08.220 | requirements for the commercial based on the inputs, write the text for a short commercial script,
00:10:12.700 | output the result. Right. And it'll repeat those same instructions based on whatever I get it. And
00:10:20.060 | it'll, it'll pipe it out in the same sort of data that, uh, um, is acceptable as inputs by the next,
00:10:25.820 | next thing down the line. Um, we did this in, uh, collaboration with Google. They came to me and said,
00:10:33.340 | Hey, we have Gemini 2 coming out. We want to launch with a bunch of cool demos and a bunch of cool
00:10:37.260 | partners. Um, do you want to have, uh, you know, be a part of that? I'm like, awesome. Does that mean
00:10:42.380 | we get early access to the new models? You know, these, um, they had shown the, you know, using your
00:10:47.180 | phone and kind of like, you know, where did I leave my keys and all that type of real time stuff. And
00:10:51.580 | they're like, no, I'm like, all right. I'm like, do I anything? No, no, you got to work with what you got.
00:10:57.740 | So, uh, cool. All right. We'll do that. The, uh, and so we did, um, you know, Gemini 1.5 was out.
00:11:06.060 | That's pretty cool. But also Gemini flash was out and flash was fast and pretty good and multimodal.
00:11:11.980 | So that was the kind of the inspiration for this as we worked on it more. That's, that's good.
00:11:17.180 | That's good. Sad and serious AI engineer conference. Um, yeah, it's good stuff.
00:11:23.900 | As we worked on this more, we realized that like you could, you could kind of do computer stuff with
00:11:28.700 | it. You could kind of like take a, uh, an instruction and say something like, like increments or like,
00:11:34.620 | I'll do this, like add up, uh, all your inputs and then you give it some inputs, uh, like, um, you know,
00:11:45.180 | whatever, uh, two and, uh, it's hard to do this, you know, 11 and it will come up with, uh, like you,
00:11:54.380 | like you kind of expect it. It'll, it'll come up with whatever 13, but the execution here is not being
00:11:59.900 | done in code. The execution is being done by a language model. Languages models are capable of this
00:12:05.020 | kind of like nonlinear thinking. Um, so if I give it two and, uh, octopus, um, as the inputs and asked it
00:12:13.580 | to add that up, um, well, that octopus is not a number, but if you forced me to, which we do in the
00:12:21.740 | prompt, uh, infer a number from whatever, you know, maybe, maybe, maybe it's eight and, and eight and two
00:12:28.460 | should make 10 and there you go. Right. And you know, if it, if it was a, uh, you know, a camera feed,
00:12:35.820 | and it is me, I'm going to try and do this, uh, hold on a second, um, four, you know, like,
00:12:45.900 | is it going to be 14? Maybe? Yeah, there we go. Right. So like, it's, it's able to use, thank you. Yeah.
00:12:55.660 | And it's not, it shouldn't be that like surprising, you know, it's, it's just
00:13:04.460 | multimodal model, take a bunch of inputs, uh, produce outputs. Um, we, we kind of went further
00:13:10.220 | with this. I'm going to have to jump to, and by the way, the, the killer use case for this, if it's not
00:13:14.780 | immediately obvious is turning your daughter's, uh, drawings and stuff into, uh, pictures and stories
00:13:21.340 | and piping them all around. Right. Uh, but the, um, where's this one? This is a good one.
00:13:28.220 | You can also do, I was playing a lot of Factorio at the moment as well. Um, oh, no, this is the wrong
00:13:35.660 | one. Hang on. All the way down at the bottom. And grab this one. And so, uh, the idea of having these,
00:13:44.780 | these machines that even include cycles and loops and that'll just operate forever.
00:13:48.460 | Um, so in this one, it comes up with a random pop song, adds it to a list, feeds it back in so it
00:13:56.380 | doesn't repeat, asks, is this song about love? Uh, and then sorts it according to, well, again,
00:14:03.900 | we're working with language models. So we have a Boolean value of yes, no, or maybe. Um,
00:14:09.100 | so we have, uh, and, and it, it feeds back around and it, it, it kind of pipes and I can just leave
00:14:14.140 | this forever just spending my, my, uh, the credits that Google gave me to, uh, to burn. Um, and yeah,
00:14:21.340 | it, this is, this is really fun. Teal Draw Computer, it, it got pretty popular. Not, not necessarily as
00:14:25.500 | popular as the, uh, Make Real, but it was, uh, it's, it's pretty amazing what you can do and it really
00:14:31.020 | rewards creativity to put it lightly. Um, I have seen people using this to, to do actual multi-stage,
00:14:39.020 | prompting, um, you know, decision-making analysis and you can imagine this being asynchronous and
00:14:44.620 | somewhere up in the cloud and maybe that's, that's what we do next where we say,
00:14:47.660 | take this CSV of, uh, email addresses of people who've engaged with our products, email them all,
00:14:55.500 | get a response, do sentiment analysis if they'd like it, you know, do something next, uh, you know,
00:15:01.420 | so forth, wait for me, you know, text me and say, like, should I really email this person again?
00:15:05.980 | Maybe I say yes. So having a big, long, long-lived asynchronous process, um, that could be run in
00:15:12.300 | parallel. This would be a great, great interface for, for designing that and everyone seems to get this.
00:15:17.980 | Um, when we, when we originally, uh, did this, the creative prompt, the, the kind of the,
00:15:23.740 | the philosophy of this project before we, I went home and prototyped, it was like,
00:15:29.580 | I want a computer that works the way that I thought a computer worked before I knew how a computer
00:15:33.660 | works, right? Maybe we just have like, I want this stuff and I want to do this to it and then
00:15:37.420 | I want to take the results and go over here. So that's, uh, that's Teal Draw Computer. Um,
00:15:43.020 | I wasn't going to show Teach, but I will show Teach, uh, which is, uh, create a flow chart that begins
00:15:49.420 | with AI and ends with engineer. Incorporate existing shapes. Um, when you have a really cool hackable
00:15:57.180 | canvas, like an SDK for canvas with like a runtime API, um, it, it plays really, really well with other
00:16:05.100 | AI tools. Um, and you can really quickly, even though these models aren't like great at this,
00:16:12.380 | um, you can really get it to work with the canvas in a way that is, um, kind of like a virtual collaborator,
00:16:21.100 | like you can kind of get it to do stuff. I mean, the demo that I always show and then I'll do really
00:16:25.900 | quick is the whole, like, you know, draw a cat. Um, somewhere on this page is a, you know, a pelican
00:16:33.020 | riding a unicycle, but I, uh, um, there's a lot of stupid drawings here. Uh, but yeah, draw a cat and it'll
00:16:41.100 | draw a cat, but you know, it's, it's doing this stuff not as a, as an image. It's not painting
00:16:46.140 | pixels in the way that like Midjourney would. It's, it's doing it as text. It's like kind of returning
00:16:50.860 | a structure that I can map into, to, to shapes on the canvas. And so, you know, I can, I can work with
00:16:56.940 | them myself. I can correct it. Um, and it can, it can work with my stuff as well. So if I do like, uh, this,
00:17:03.900 | it's like orange, uh, and I say, uh, make the cat, the cat blow out the candle.
00:17:11.100 | I didn't tell it was, it was a candle, but, uh, let's see if I can do it. Um, I don't think cats
00:17:17.900 | can actually blow, but I don't know that for sure. Uh, uh, uh, this is using Claude as, as the back
00:17:26.700 | end. If you want to know how this works, definitely catch me up afterwards. Um, yeah, right on.
00:17:32.780 | Hey, and we get smoke as well. This is wonderful, right? Uh, that, uh, Tealdra, a lot of this stuff,
00:17:41.420 | and, and, in fact, I would say our, uh, our advantage over the bigger companies in this space
00:17:45.820 | is that, uh, shitty but amazing is definitely on brand for Tealdra. Uh, and yeah, if, if this seems
00:17:54.300 | like a good problem that you might want to work on, definitely talk to me because we have some tools
00:17:57.900 | that make it easier. Um, people build all sorts of crazy stuff with Tealdra. Um, this is, uh, Grant
00:18:05.020 | Cots liquid, you know, the, uh, simulation that's using Tealdra is like the, the geometric, physical,
00:18:11.660 | and, you know, like control layer, I don't know, authoring layer on top of it. Um, companies build
00:18:16.700 | really cool stuff with Tealdra, like observables. It's building with Tealdra now. It's, it's incredible.
00:18:20.700 | Um, I think we're only, not even scratching the surface, uh, of what can be done with this paradigm
00:18:28.540 | and these tools. Please build something amazing. Uh, I got the canvas. We have the technology.
00:18:34.940 | So, that's my talk. Uh, thank you very much.