. My name is Stefania. I'm so glad you made it until the last day of the conference and came to the robotics track. So we're going to start with a live demo, and then we'll switch to the presentation, and we just like to kind of swap things around. So I'm going to try to connect the microscope over here.

And let's see the other camera. And some sensors. So my talk is about real-time co-scientists. So think about pair programmers. How many of you use any form of co-pilot for coding? Right. So it's just like that, but for doing things in the real world, like science experiments. So you can see here, now I have this board, which is a micro:bit board with jackdack connected.

And it's measuring the temperature. And I actually have a heat pad, so I can increase the temperature and make it very, very hot. And hopefully, it's not going to melt the board. And then I can send that to my science assistant. It's going to analyze the data and kind of give me an answer in real time.

So it's telling me, like, OK, the ambient conditions indicate, like, stable, dark, quiet environments. Because it has, like-- it knows it can measure all these different sensors. No buttons. But it sees that the temperature is 26 degrees. And then I can actually give it more context. I can go and create a protocol and say, like, this is the type of experiment that I want to do.

So whenever you give me feedback about the data or the images that I'm sending from the microscope, which I can also do, it's going to be in the context of that experiment with those conditions and those constraints. And, of course, like, if you are actually doing that experiment several times, you can go ahead and create a custom page for your experiment.

And it will monitor the data from the-- we're going to connect, again, the JackDack. It's going to monitor and plot the data in real time. And the cool thing is also that if you want to leave your experiment running and go away, right, like, you could actually have cameras that are autonomous.

To go back to the previous talk, they're not fully autonomous, but they can track objects. So this camera is called re-camera. Also, by the way, all the hardware I'm showing you and all the components are open source. So right now, I'm going to set it to track me as a person and start the tracking.

And it's running a model on the camera itself. So it's going to start, like, moving around to see me and see where I'm going. But you could train a custom model, like, to track, like, crystal growth or specific objects that you want to monitor and in real time. And this runs on Wi-Fi, so you can place it anywhere and then control, like, the conditions in the experiments, like increase the temperature or decrease the temperature.

So, oh, it sees all of you. That's awesome. So I'm going to stop this for now. So that's like a short-- and here we're seeing, like, our temperature kind of increasing. I think my heat pad stopped. But that's just like a simple, like, demo of, like, without having a specific experiment in mind.

And now I'm going to switch to my slides and actually show you what happens when you record much longer experiments and what it can do. Why do we care about co-scientists? We're not talking about real-time co-scientists, just co-scientists in general. Like, why should you care about it? We know that there is a data overload in science, and there's a lot of complexity, a lot of things to parse through and analyze, and AI can really help with that.

We also know that beyond the analysis of data, fast, at scale, they can help with generating new hypotheses and maybe identifying blind spots in prior work and in your own thinking. And we also know that they can speed things up, right? So instead of testing one hypothesis at a time, you could test 100 hypotheses.

So I got inspired to build this demo. And by the way, the cost of all the parts is under $300, and it took me two weeks to build it. And it's open source, so you can play with it and hack it. So I got inspired to do this because of this demo from DeepMind.

So this paper got published two months ago on AI co-scientists, and it was actually showing, not in the real life, but it was actually showing what happens when you're analyzing papers and data. If you have a multitude of agents that can perform the different roles that we do as scientists, what are the results?

So what are the roles that we do as scientists? We analyze papers, we summarize them, we look at data, we rank the options, we rank the different hypothesis. So they actually created an orchestration of different agents. Each of them was actually working on Gemini 2.0. And then with this orchestration of ranking the different results, the different hypothesis, doing search online, doing search on papers, come up with a plan for the researcher to use.

And not only that they did that, but they actually tested it against prior discoveries, and this was super interesting. So they tested it against prior discoveries on gene transfer mechanisms. So a discovery that took scientists 12 years. It took the AI scientist two days to come up with, without having seen the data.

So it was not aware of how the gene transfer works. This was not part of the training. So that was one verification by trying to replicate past results. But another one was to come up with a completely new hypothesis. So they used it for liver fibrosis target treatment discovery.

And the AI co-scientists came up with target drugs that were actually efficient. in the lab, in wet lab, they were tested by experts. So it's not a science fiction, it's not like something in the future, it's actually happening now. And for me, what was inspiring about this is like, when we can make real discoveries in drug, drug discovery, like healthcare treatment, bacterial resistance, new type of materials, I think that can be accelerated doing it in real time with a scientist.

This clicker doesn't work very well. So the vision was, instead of just doing it like on the data, like not sync, like async, and giving the researchers a plan based on the data that exists, what if we do it in real time, right? And we formulate this hypothesis based on the empirical data we're observing in the lab in real time.

Or it could be like, when you're observing the robot, like breaking down in real time, right? So what motivated me was the results from the AI co-scientists, but also this vision from Silver and Sutton, which is, they just published this paper, Welcome to the Era of Experience, which is fantastic, it's very short, if you haven't read it, I highly recommend it.

And they really talk about how we're going past the era of human data, where we're only indexing and making predictions based on the data sets that we created, and going to an era where the AI learns from the continuous environment in which we operate, right? And especially with multimodal, when we have real time data from images, from sensors, from audio streams, like that is possible.

So, I hope I convinced you by now that real time matters, and I wanted to show you like how I use my system with longer experiments. So, you already saw like the chat image, and I had to find experiments that I could actually do at home. So, this is the overview of the system.

So, it's a very simple React app, and it has like all the input sources that you've seen live in action, like the JackTag sensors via USB, different webcams, you can add as many webcams as you want, text input, it actually works with voice, I forgot to show that, you can talk to it and it talks back.

And then, all of these like inputs become web hooks, and I can, I'm going to go into more detail how I optimize each and one of them. And that gets sent to a backend which communicates with the real-time API, I use the Gemini in this case. And, I had to, this is the information flow, so, you have the physical sensors, and for the JackTag, I'm actually using the web USB API.

So, that goes to the frontend hooks, which gets pulled by the context assembly, and the context assembly actually is always checking what sort of modalities are present. Do we have text? Do we have voice? Do we have image? Do we have a chat history? And depending on what modalities are present, every single time I'm sending a message, it builds this context that gets sent to the API that gives me back a response.

And, this is how the unified context assembly works. So, I guess like the very important piece here is how you're doing this context injection, depending on what sensors you have connected, and the type of experiment you're defining. And making this dynamic on the fly, like when you create the protocols.

And, that was the, those were the ingredients for the code, now let's talk about the ingredients for the hardware. So, when I start doing the experiments, I had to list and put out all the parts that I had at my disposal. Kind of like cooking, right? So, it's like what can I make with all these sensors, like these are my inputs, these are my outputs, these are all my cameras, and all my cables, and how many boards I have.

What sort of experiments could I create with that? And, I also had the constraint of doing experiments that I can measure in real time for this talk, and also that are safe for me to do at home, and I can travel with. Right? So, that really narrowed down the list of things that I could play with.

So, one thing that I did was exploring crystallization, so you can see there like the crystals that I was growing, and that took longer, and I recorded the whole thing, and fermentation. And, let's dive into each and one of those. So, this was my lab in a box in our apartment in Tokyo.

The people coming to the house were very confused every day. There was something, no explosions though, so that was good. So, for crystal growth, how many of you remember this from chemistry? Okay, okay, that's awesome. So, I don't need to go into great detail, but basically the principle is quite simple.

Like, you oversaturate the solution by adding salt in hot water, and then you're like slowly cooling it off, and that creates this process, this nucleation, and the growth of crystals. Now, the trick in having beautiful crystals is how you do the cooling off, right? It needs to be gradual.

You need to not move like the liquid or the object. So, the gradual cooling and the level of humidity is what gives you beautiful crystal formation. So, the main things we want to measure in this example is what are our curves for how fast the salt dissolves. Like, where are the places where the crystals are being formed?

Those are called nucleation sites. What's the growth rate of the crystals? And that is affected by temperature concentration. Oh, yeah, I forgot to play this. This was like a sped up. So, this was a recording from the microscope. It's moving because it actually has a fan blowing cool air from ice, so that's why the camera is moving.

That was how I was like cooling it off. But you can see the data in the sensors changing on the side. And this is like being recorded for an extended period of time. And then I get a CSV from the sensor values, and I can go and analyze that and kind of get a sense for like what happened.

Was I cooling it off too fast or too slow? Were my crystal like growing or not? And then I also play with different samples. So, I had samples that I would put in the fridge and in the room and like a different temperature. So, I could also measure and have like control groups for that.

And then once I have that CSV data, I can plot it, right? So, I can actually see what was my crystal growth rate. And if my temperature was like the best. And I can plot that for the different samples. And right now, I don't have that integrated live on the platform, but that's coming.

So, I had to write a separate script to get the CSV data and create this visualizations. And the insight I got from this is that the crystal formation is actually not gradual, that it happens in bursts. So, that was something that surprised me when I looked at the data.

So, once the critical saturation is reached, there's like sudden crystal growth formation. So, it's not like something that is kind of like slowly growing, but it grows in bursts. And here is like 40 minutes recording of the crystal growth, like super sped up. So, you can kind of see how the humidity and the temperature was changing.

And the next one was, how am I doing on time? Because I have a lot of things. Okay. I might skip over fermentation, just in interest of time. But I'm sure all of you have dealt with fermentation in some shape or form, even if it was liquid, having to benefit the end result.

But the insight here is to actually control how much salt and sugar we put into the different dough and then measure, and then also change the temperature and measure the growth rate and the CO2. And that was the fermentation. And I also have like a sped up recording of the data collection.

And I already showed you the recamera and the fact that it's mobile and it can track objects. This is the education version. If you want to play with some of these things and just kind of do your own experiments, please like the lab mine that I showed you is not deployed yet, but it will be and the code will be open source.

But this you can play with. You need to put your own API, but it runs on the phone and you can test the camera for now. And if you have a micro bit or you want to connect mine, you're welcome to try it. And let me know if the QR doesn't work.

And yeah, this this is a version kind of like for to teach it to kids. And I wanted to end by talking a little bit about the open source ecosystem that can support this type of initiative to go beyond the demo and kind of like a cool experimentation to a real real solution for scientists and builders.

So there's an entire open source ecosystem for recreating all the lab equipment, but then also for creating machines that can support the automation of pipetting and like doing all the analysis that you need to do in a lab and all the manipulation that you need to do in a lab.

The Jubilee motion platform, it's actually coming from University of Washington where I did my PhD. It's open source and there is an open bio reactor and there's actually a workshop that just took place in April at UW where they had people for a week hacking on creating different solutions for automating scientific experiments in the lab.

So anything from like droplet manipulation to robot handling liquids and mixing like vials and things like that and other application with Jubilee. So definitely check them out. And for the future, and this is my last slide, you can imagine that based on the samples that we're collecting from this cameras and sensors and voice, it's much easier to create simulations.

So we don't need to be limited by the experiments we're doing in real life, but those experiments that we're doing in real life are going to inform realistic simulations, right? So I could actually create a simulation of my crystal growth, but more useful stuff right like bacteria colony growth and run those simulations with the lab conditions and then identify what are the best conditions for the experiment and create those conditions in the real life.

So very excited about integrating simulation and I think that's where the this is going. So that was me if you want to read more about my projects and paper and all the open source projects. I do a lot of work in education as well. It's all on my website.

And as Ben mentioned this morning, we are going to do an AI education summit and very excited about that. Some of you who saw my talk like in New York or last year know how much how passionate I am about about education. So I hope you can join us for the summit.

I really appreciate you coming to the talk today and I'll be around. Thank you so much. Thank you. Thank you. We'll see you next time.