Back to Index

Books reimagined: AI to create new experiences for things you know — Lukasz Gandecki, TheBrain.pro


Transcript

So my name is Łukasz Gądecki and I've been programming since I was a little kid and I want to tell you about my newest project, Books Reimagined. So how to use AI to create new experiences for things you already know. So how it all started. I was reading a book about Donald Trump re-election and since, as you can hear, I'm not from the United States, there was a few too many characters to me.

I didn't follow everyone, so I decided to vibe code my way through the understanding. I built a little bit of an AI companion application. It looked terrible, but it gave me context for the people that were on the page with a little bit. It found the images for them and gave me a little bit of a summary in the context of the of the page that I was at.

And a month later, it turned into something different. So this is going to be the Snow Queen. This is one of the first experiences we've built. This is the Snow Queen book, and this is the part where the sorcerer's parentheses are flying away with the mirror that distorts the reality.

So all right. So it tells a story about the flying and flying and the heaven is so far away. There's music and it reads, but you can't go into your time story. But then the crash happens and the mirror shutters and it distorts everything all around. So this is one of the first experiences we've built.

But it's all in Polish, so I want to actually demonstrate one that we built just for this conference that's in English. This is 1984. And what's interesting here, which I don't think I'll be able to show you, is that you can send a quick voice note to the book and ask what's going on in this scene right now.

I don't really have audio. But the point is that there's many different AI voice assistants, but they are almost always just terrible, if not all of them, to be honest, serious terrible. We had a demo from Google yesterday. They were saying up front that it works 50 percent. It's usually there's a delay.

They start talking in the wrong position. I mean, at the wrong time. Then they interrupt you. So we built here a system where you hold it as to just specify when you are speaking and then you let it go. And it immediately, 100 milliseconds, responds to you. And then you could scroll further and then ask a question like, "What happened between the last time I asked a question and now," and it can summarize what's going on.

So you have to believe me that. You can check later on bookgenius.net. Another thing that we were thinking about is the search. That's a very common thing, searching. So the most normal search would be just exact search. But if you want to -- the way our brains doesn't work, they don't memorize the pages.

So if you want to find a scene where Winston met O'Brien, then exact search is not going to work. But embeddings work. So you can quickly find the scene you were thinking about this way. And then you can go to that -- go to that spot, read a bit more.

And you can go back to the place where you were reading. But there's also one step forward -- I mean, one more thing you can do. You can basically say, "Talk about all the way the party propaganda works." And you can do deep research. And it's going to actually read the whole book point that you finished at to give you the answer.

So it's very useful. It's going to take a couple minutes. I'm going to go back to presentation. So I started with VibeCoding, Vanilla.js, very confusing code. But it gave me the freedom to iterate very quickly. You basically don't know what you don't know. And if you start to -- especially right now, the time it takes to plan everything up front is often wasted.

Because you can much quicker just tell your thinking to the AI and generate something that works. And then you see, "Oh, that's actually not that great. Let's try this and that." And I realized that throwing away code that you poured your heart into often feels terrible. Like, you're invested.

You've spent so much time. But throwing away code written by AI actually feels great. So I would describe this as waves of changes. So basically, once I start feeling that I don't rewrite the whole code base day after day, like the amplitude of the waste is getting lower and lower.

And there comes a time where I can start old-school engineering. I can start getting tests and refactor. But there are traps to refactoring. Do I refactor the worst piece of code? I would suggest that it's better to focus on no hanging fruits. So for example, I had a piece of a code from OpenAI audio processing.

And it's like JavaScript, very quickly written, no types, very confusing. But I never have to touch it. So I'm not refactoring it. Although it was very tempting. So we often think about refactoring by adding this. How bad? How painful? How easy? But if something is very bad and very easy to change, but it's not painful at all.

And it's probably not a good idea to change it. So I would suggest that it's better to look at how bad the code is multiplied by how painful and multiplied by how easy. And when all those factors are taken into consideration, then it starts making sense to make a decision.

So a lot of the AI experiences that we see and talk about are basically either chat GPT wrappers or image generators or half-working useless voice assistants, including Siri. So our approach was to hide the AI from the user. So when we produce the books, the AI does the initial draft and we do the rest.

And I would argue that the human touch is invaluable in situations like this. AI cannot tell if the music that it generated isn't good. It cannot say if the graphics are good-looking or if the avatar is actually matching the vibe of the person that the book is talking about.

So we want to make the AI disappear. And multiple things connected together, simple things, simple building blocks, make for the magical experience for the reader. There's nothing really new here. You could already ask a friend a question about the book, but is your friend available 24/7 and all-knowing? Probably not.

You can already search, but is the search the spoiler-free search? Is it natural language search or exact match? So I think that beautiful graphics help you get into the mood of the book and help you with the character recall. And music that matches the scene makes it the experience like watching a movie.

And we know that music influences the emotions hugely. And it's very nice when you're reading the book and the music just flows with the book and gives you this great experience. So nothing new, but at the same time completely new, which is what AI allows us to do nowadays.

And I would encourage everyone to think about those tiny little niches where we can create some completely new experiences on top of something that we have known for such a long time. So in thousands of years, it was never possible to read books like this or even really produce books like this.

Because if I had to do all those graphics and music for every single book, it would cost me, I don't know, $100,000 per book. So it never made sense to do this. So how do we do this? The process is we use a combination of LLMs to the scene analysis, book characters detection.

We give the AI an overall music theme. So we say for Sherlock Holmes books, for example, that it's like Victoria London and all that, and it should be norm music and kind of on a sad node. So with scene analysis plus mood detection, we do music generation, and we also destructured XML with metadata.

So for example, we have a text like this, and AI is very good at doing this kind of a mapping, which then is very easy for us to use in the book when we say, like, we can display the avatars that are in the scene. It would be very time consuming for a person to go through the whole book and map every single thing like this.

So today we are open sourcing the player, so anyone can create the Netflix-style experiences for books. And if you want AI that feels like magic, not like chatbots, come talk to me. We build AI experiences that ship in the light and not slides, although I hope the slides were nice.

So thank you, and you can find me at those places.