Thank you so much for the intro. I'm so excited to be here talking about like figuring out like how do we like AI and human like work in the world of whiteboarding and I built Excalibur and if you don't know about it like you'll see like many things about it.

And one of the expectations you probably have about a speaker at the AI engineer conference is that I talk about AI on every single sentence for the entire talk. So I'm just going to give you a warning. I'm only going to do it for the second half of the talk.

But the first half is going to be a good setup and you should like pay attention because it's going to be really interesting content. So let's get started. So we are five years ago. I'm working at Facebook January 1st and as you if you've been at any like big company there's like the performance review process.

And so we are you have to give feedback around like all the people you've been working with like how good or bad you've been doing and same for your manager. And one thing that's like I'm probably not unique in this room but there's this little devil in me that is like nope you're not going to do it until the very very last minute.

And this is called procrastination and I'm really bad at this and I've tried like many techniques over the years and the one at that time that I worked on is what I call active procrastination. So I know I'm not going to be able to write the performance reviews until like the last minute so may as well do something actually useful in the meantime otherwise I would be like doom scrolling on my phone and not be productive.

And so what I ended up doing is I wanted to write a blog post for a long time and so I was like okay I'm going to write a blog post. And so I used a tool called Zwibbler in order to draw like this kind of like hand written like illustrations and unfortunately at the time the website Zwibbler was done.

And I was like no come on I've wanted to not procrastinate and everything and then it's still done. And so what did I do I'm like okay we're going to go like one level deeper. What would it take to actually re-implement Zwibbler? How hard would it be? And so I opened the code sandbox and I like drew like wrote some code to use rough.js and now hey I got a demo and the thing I didn't expect is I posted about it on Twitter and like it exploded.

Like 200 likes on this first tweet and like so many people like replied and commented and were excited about it. And I was like oh this is so cool. And so this was the beginning of Excalibur. Now one of the thing is if I were to do this again in the future like right now I would not like start writing code myself.

I would probably do use any of the coding agents that we've seen. And the good thing is I'm not the only one. So if you look at two weeks ago during the Anthropic keynote when they wanted to demo Claude's AI agent they actually used Excalibur for this. And they actually had the agent re-implement the most popular request which is a table layout.

Unfortunately they haven't sent a PR yet so if anybody's working at Claude here we welcome pull requests. Anyway so one thing you may have noticed is I've said AI multiple times so I knew you were not going to last the entire half of the talk without it. So now let's get back to the non AI part of this.

So this was like in the story we are like in January 1st 2020 and three months later you probably all know what happened. COVID. And so everybody had to go home and now had to figure out like how are you working from home like what's life like. And a lot of people one of the big thing that they try to figure out is whiteboarding.

And so when you're in the office you have a physical whiteboard and you can draw stuff and you can collaborate with people. But when you're at home you don't have any of this. And I was very confused because a lot of people like starting coming to me around like oh actually Xcalidraw is a really good whiteboarding solution.

And I was like very confused because I built Xcalidraw for drawing illustration for my blog post. And why what does it have to do with whiteboarding. And then I started being interested around like okay what are the other people doing whiteboarding like other tools doing. And one thing I quickly realized is that all of the tools at the time basically took the physical limitation of whiteboarding and mapped them one to one with the online versions.

So for example a whiteboard like a physical whiteboard has a fixed size. And so what everybody did they created a fixed size hey create a new whiteboard. And then if you run out of space then like you're out of luck. Now with Xcalidraw you can just like scroll around and like write start writing somewhere else.

And it just works. Then the next one is drawing text. And so on the physical whiteboard you have to trace every single letter. And on the mouse it's actually very very hard. And so on Xcalidraw I was like hey like we have a text box like you can select and this is fine.

And when I did the dry run for this talk like the people were telling me like there's no way like even MS Paint in like 30 years ago had like keyboard support. But this was the state of the arts at the time. And one thing to remember is nobody like thought about online whiteboarding at the time.

So like all the projects were pretty small and like not very used. And now all of a sudden there's been a huge influx. So this is why it was so bad. And then like the list keeps going on. So if you want to start erasing they will actually have you like erase every single pixel.

But on Xcalidraw you can select and delete and it just works. And then like I don't know if you've seen like the real TikTok video where like people like draw like perfect circles like this on a blackboard. And this is like what you had to do for like doing this online.

But with Xcalidraw you can just like select a shape like a circle and like you just draw it and it just works fine. And finally there are things you cannot do on a physical whiteboard. So for example like copy pasting is like just doesn't work. but if you're working online like this is the baseline expectation that you're able to like select and copy paste.

And so this is like all of this like really was the big breakthrough in my mind around the idea of the physical versus virtual whiteboard. And this is I feel like the same place and time we're in with AI integration right now. So in practice what a lot of the integrations today they're basically taking like whatever we're doing right now.

And then like sprinkling AI or re-implementing with AI. And the end result is like pretty bad because like hey you don't know how it works. But the thing is unlike Xcalidraw and the online like where we had the experience right now we as AI like nobody has experience. And so this is why I'm super excited about this conference.

Well like all of the people like trying things out actually sharing the lessons to the other with with each other. And so hopefully like as we keep doing these conferences now we're going to bring the state of the world with AI to be much more much better. So now I talked a lot about like Xcalidraw the early days and everything.

And so a lot of people have been asking us like how do you integrate AI with Xcalidraw and we've tried a bunch of things. So now for the second part I'm going to walk through what are the things we've tried, what are the things that have worked, what are the things that haven't worked.

So the first one is actually not us but Gorchem who I think is in the room somewhere working at File AI. And so he discovered this really cool machine learning model that takes an Xcalidraw diagram with like a palm tree or like a sun and then is able to generate real time live picture like beach of the image in reaction to this.

And so this is like was mind blowing to me. Well like oh now like machines can actually generate images and the model is working really well and like the I think the demo is still working. So if you if you want like you should try it. But one of the issue is when we started like integrating with Xcalidraw we realized that people don't actually draw realistic images on Xcalidraw.

And so this is one like the first like key lesson that we found is that just because there's a model that exists doesn't mean that it should be within your product. And I know right now like there's a huge pressure for all of us to like just add whatever kind of AI into our product that's like in order to be like AI first and whatever.

But I really like want you to think about like what are the actual AI integrations that actually make sense rather than like hey I just put in AI and that's it. And in practice if you do that then actually like the whole product is going to be like way worse than if you didn't have any AI at all.

So this was like the first learning. Now people do not draw realistic images on Xcalidraw but they do draw diagrams. And so now the question is okay so LNMs are magical and like they're going to generate diagrams so let's try it. And so we tried and that didn't work out.

And so this is an illustration but in practice it was even worse like there was no resemblance of a house or anything. It was just like bad. And this is where like we started realizing that like just doing this wasn't going to cut it. So we need to like be like a bit better.

But one thing is yesterday I was chatting with somebody at working at Gemini and he showed me a demo of like Gemini actually outputting Xcalidraw files. So my talk is already outdated and we probably need to get back and like start coding on this but anyway let's continue. So the thing though that we found out is LNMs were actually able to generate something called Mermaid.js.

And so the name of the product is super weird but in practice it's a way to describe a diagram. And so in this case you're just writing flowchart LRAAI to be engineer and then it draws this specific diagram. And so LNMs were able to like realistically generate this kind of like Mermaid.js syntax.

And this was really exciting because now what we are able to do is to like encode the knowledge of like the diagram and Mermaid supports like many different types of diagram. And so now we're able to like encode all of this and like all of the prompts can actually generate diagrams.

Now the challenge is like this output SVG or canvas and not Xcalidraw files. So what do we do? So in practice I can check it to work. And so she implemented an Xcalidraw backend for Mermaid.js. And now we finally have our very first very very successful integration of AI within Xcalidraw.

So now what you can do is to like set a prompt and like put in anything that you want. So in this case a flowchart starts with AI and an engineer. But we've seen people like draw like org chart diagram or TCP or the network request kind of diagram or architecture diagram.

Where you can see like classes and inheritance and those kind of things. And then you get a result which is an actual Xcalidraw component that you can play with. And one of the things that is the like the title of my talk is the human and AI partnership. One of the things that you've probably been very frustrated is like you start and you generate something that like looks 90% like what you want.

But then you're like hey this is an image I can't do anything with it. And this is like the magic of this integration with Xcalidraw is now this is just Xcalidraw and you can use Xcalidraw to like make it however you want. And so for example you can get to something like this.

Where you start modifying the layout just moving things around. You start adding colors. And you really make it to where and to what you want to be. And so I think this is the mindset that we're going to see more and more with successfully integrations. It's no longer going to be like one shot.

Hey I'm asking the AI something and the AI give me like something perfect. But how do you get something that's close enough and then you as a human can start modifying. And so right now this is like a just one way like one run trip operation. But I foresee a future where like now you're going to be able to go back to the AI and say hey modify this.

And then go to the much further. We don't have this right now but this is what I think the future is going to look like. So now this is one integration so I'm going to go through a few other AI integration that we've been working on within Xcalidraw. So the next one is one of the things you can do with the SAS offering of Xcalidraw is managing file creating a bunch of different files and syncing in the clouds.

And as if you've been using this most of your documents are going to look like untitled one, untitled two, untitled three, untitled four. And one of the things that Ellen can actually be really good at is generating names for these documents. And so now whenever you're creating a document we wait until you stop working on this so we actually have content.

And then we're going to prompt you hey these are like all of the names that we think would match and then you just click and then you're done. And so this way like it actually removes the burden on you to do it and AIs are really good at this.

Next one is while people are not using Xcalidraw to generate like good looking like painting kind of things. They people tend to like want to have like a lot of infographics a lot of illustrations within their slides or like their diagrams. And one thing that we are doing which is in like alpha like close right now but we're going to open source and to ship in a few weeks is the ability to generate illustrations.

And so we are really hyped about this because I think it's going to help a lot of people. And finally the thing that like kills me the most I'm using Xcalidraw every day and I usually want to integrate logos within my diagram or my slides kind of things. And I cannot find a reliable way to remove the background from all of the space key logos.

So I just took a screenshot of the AI engineer welfare from the website and I'm like hey please like how do I remove the background. And the thing that I wanted to do is to challenge all of you like you're all AI engineers and probably some someone from all of you can create a model that removes the background for logos.

And if you're able to make it work within the browser send it to me and we'll make it happen and we like ship it within Xcalidraw. So I'm really excited that some of you are going to be able to help us with this and help all of the people using Xcalidraw.

So this is the end of my talk and I have a few conclusions for you. So the first one is right now as I mentioned we're in the like physical whiteboard to virtual whiteboard like kind of moments happening within like all of the product but specifically like for the design products.

And now we need to figure out like what is the AI native way to implement those interactions. And so really excited about this conference to like start showing like all of the potential. The second one is please please please resist the urge to actually ship all of the random model that exists within your app.

And really only ship the one that like you feel like actually adding to the product and helping your customers getting doing what they actually want to do. And the last one is LLM's love structure. So if you're able to take your problem domain from space and find a way to represent it in some kind of domain specific language.

Now it's going to be most likely going to work way better to like have the LLM target this domain specific language and then you take this language and without AI you actually output it to your own system. So this is the end of my talk. I'm really excited that we have this event and conference.

And I'm also excited like Steve is going to be talking next about TLDraw. Like we've been working together for a long time and like he's probably going to blow your mind with all his examples. So thank you. Thank you. Thank you. We'll see you next time.