The Multimodal Future of Education: Stefania Druga

00:00:15.000 | Hi, everyone. I'm Steph. I'm going to talk about the future of education with multimodal AI.

00:00:20.000 | We are here at AI Engineering Summit, and AI engineering starts very early.

00:00:26.000 | So I'm curious how many of you have kids? How many people in the room have kids?

00:00:32.000 | Okay, wonderful. How many of your kids have played with generative AI so far?

00:00:40.000 | Okay, so you won't be surprised to see the next slide.

00:00:45.000 | Basically, 70% of generative AI users are from Generation Z.

00:00:50.000 | This is a recent study from Salesforce.

00:00:52.000 | So it starts very early, and the reason I care about the future of education with generative AI is because education needs a wake-up call.

00:01:02.000 | So we know that early literacy rates need to be improved around the world.

00:01:08.000 | Only 70% of 10-year-olds actually can read and understand a simple story.

00:01:14.000 | At the same time, we've seen a big gap in learning that happened during COVID.

00:01:18.000 | 60% of children and teenagers are left behind, in particular in math and reading.

00:01:24.000 | And it's not just K through 12.

00:01:26.000 | For older people, for adults, we need to do a lot of re-skilling.

00:01:31.000 | So multimodal AI has a potential to transform education.

00:01:37.000 | And we know that students are using these tools for their homework.

00:01:42.000 | They recommend it to their colleagues.

00:01:44.000 | And we also know that they prefer using these tools over a human tutor.

00:01:50.000 | So this is not new.

00:01:52.000 | We are, right now, dealing with the first AI generation.

00:01:56.000 | These children have been growing up with AI since 2015.

00:02:00.000 | Half of households in the U.S. had some sort of voice assistant.

00:02:04.000 | And I started researching this in 2015 at MIT.

00:02:09.000 | And basically showed that youth perception of voice assistants, chatbots, smart toys,

00:02:15.000 | really influence how they interact with these devices.

00:02:19.000 | How they learn from them.

00:02:21.000 | And how they perceive them.

00:02:22.000 | So overall, they perceive these AI devices friendly.

00:02:26.000 | But they also have a different perception of the intelligence of the devices.

00:02:30.000 | And age plays a huge role.

00:02:33.000 | So younger kids, we're talking four to six and a half, are much more skeptical of how smart Google Home is or Alexa.

00:02:40.000 | And older kids, like the moment they start going to school, they all say voice devices are smarter than I am.

00:02:46.000 | And we're just talking about voice assistants.

00:02:48.000 | We're not talking yet about generative AI tools.

00:02:51.000 | So how can we change that?

00:02:54.000 | And why does that matter?

00:02:55.000 | It matters because their perception of how this technology works influences what they expect.

00:03:02.000 | And their mental models is influencing the type of queries they're going to ask, how much they trust the answers.

00:03:07.000 | So we need to cultivate AI literacy and critical understanding of this technology.

00:03:11.000 | To that end, I built this open source and free platform called Cockney Mates.

00:03:15.000 | In 2016, it expanded Scratch, which is the largest platform for coding for kids.

00:03:21.000 | And it basically allowed children to do programs like this.

00:03:26.000 | Can someone guess what this program does?

00:03:31.000 | Yeah, it's a game.

00:03:33.000 | It's a hide-and-seek game, right?

00:03:34.000 | So she's like -- she programmed a hide-and-seek game with the robot.

00:03:37.000 | And if she puts a loop, she can run around the room and the robot is constantly going to try to find her.

00:03:42.000 | Now, this is the first step.

00:03:44.000 | It's using, like I mentioned, block-based programming language, expanding Scratch.

00:03:50.000 | And at the time, like, it allowed kids to not only program their smart lights, their voice assistants,

00:03:57.000 | but also train their own custom models.

00:04:00.000 | So they can train models with examples of images or examples of text and then use those custom models

00:04:07.000 | in their own games and application.

00:04:09.000 | So, for example, here, like, this student trained a model to distinguish between unicorns and narwhals.

00:04:15.000 | And then not only gets a prediction when it plays with a game, but it also gets the confidence level.

00:04:21.000 | How confident is his custom model that the drawing is a unicorn?

00:04:25.000 | And we see the confidence is pretty low.

00:04:27.000 | So they made all sorts of things, like looking at what's in their food,

00:04:33.000 | trying to, like, program games like rock, paper, scissors,

00:04:38.000 | get, like, the robot to talk like Shakespeare.

00:04:43.000 | And this was used all over the world.

00:04:45.000 | It's translated in more than 30 languages.

00:04:48.000 | And the good news is that we evaluated this to see how it increases that critical understanding of AI

00:04:56.000 | and how it helps with AI literacy.

00:04:58.000 | So to do that, I did a longitudinal study in public and private schools

00:05:04.000 | where we asked questions of what kids think about AI before,

00:05:08.000 | then we allowed them to engage in AI learning activities,

00:05:11.000 | and then we asked the same questions at the end.

00:05:13.000 | And what we found after they learned how to do text training, image training,

00:05:18.000 | smart home programming, is that they became much more skeptical of the AI smarts.

00:05:24.000 | Like, in the beginning, they would say, like, yes, Google Home is smarter than me,

00:05:28.000 | or this model is much better than me.

00:05:31.000 | And after they learned how it works and how to train it,

00:05:33.000 | they were not so sure it's smarter than they are.

00:05:36.000 | And I'll show you a quick video to see how that went.

00:05:43.000 | I'll show you a quick video.

00:05:44.000 | I'll show you a quick video.

00:05:45.000 | I'll show you a quick video.

00:05:46.000 | I'll show you a quick video.

00:05:47.000 | I'm going to show you a quick video.

00:05:48.000 | I'm going to show you a quick video.

00:05:49.000 | I'm going to show you a quick video.

00:05:50.000 | I'm going to show you a quick video.

00:05:51.000 | I'm going to show you a quick video.

00:05:52.000 | I'm going to show you a quick video.

00:05:53.000 | I'm going to show you a quick video.

00:05:54.000 | I'm going to show you a quick video.

00:05:55.000 | I'm going to show you a quick video.

00:05:56.000 | I'm going to show you a quick video.

00:05:57.000 | I'm going to show you a quick video.

00:05:58.000 | I'm going to show you a quick video.

00:05:59.000 | I'm going to show you a quick video.

00:06:00.000 | I'm going to show you a quick video.

00:06:01.000 | I'm going to show you a quick video.

00:06:02.000 | I'm going to show you a quick video.

00:06:03.000 | I'm going to show you a quick video.

00:06:04.000 | I'm going to show you a quick video.

00:06:05.000 | I'm going to show you a quick video.

00:06:06.000 | I'm going to show you a quick video.

00:06:07.000 | I'm going to show you a quick video.

00:06:08.000 | I'm going to show you a quick video.

00:06:09.000 | I'm going to show you a quick video.

00:06:10.000 | I'm going to show you a quick video.

00:06:11.000 | I'm going to show you a quick video.

00:06:12.000 | I'm going to show you a quick video.

00:06:13.000 | I'm going to show you a quick video.

00:06:14.000 | I'm going to show you a quick video.

00:06:15.000 | I'm going to show you a quick video.

00:06:16.000 | I'm going to show you a quick video.

00:06:17.000 | I'm going to show you a quick video.

00:06:18.000 | I'm going to show you a quick video.

00:06:19.000 | I'm going to show you a quick video.

00:06:20.000 | I'm going to show you a quick video.

00:06:21.000 | I'm going to show you a quick video.

00:06:22.000 | I'm going to show you a quick video.

00:06:23.000 | I'm going to show you a quick video.

00:06:24.000 | I'm going to show you a quick video.

00:06:25.000 | I'm going to show you a quick video.

00:06:26.000 | I'm going to show you a quick video.

00:06:27.000 | I'm going to show you a quick video.

00:06:28.000 | I'm going to show you a quick video.

00:06:29.000 | I'm going to show you a quick video.

00:06:30.000 | I'm going to show you a quick video.

00:06:31.000 | I'm going to show you a quick video.

00:06:32.000 | I'm going to show you a quick video.

00:06:33.000 | I'm going to show you a quick video.

00:06:34.000 | I'm going to show you a quick video.

00:06:35.000 | I'm going to show you a quick video.

00:06:36.000 | I'm going to show you a quick video.

00:06:37.000 | I'm going to show you a quick video.

00:06:38.000 | I'm going to show you a quick video.

00:06:39.000 | I'm going to show you a quick video.

00:06:40.000 | I'm going to show you a quick video.

00:06:41.000 | I'm going to show you a quick video.

00:06:42.000 | I'm going to show you a quick video.

00:06:43.000 | I'm going to show you a quick video.

00:06:44.000 | I'm going to show you a quick video.

00:06:45.000 | I'm going to show you a quick video.

00:06:46.000 | I'm going to show you a quick video.

00:06:47.000 | I'm going to show you a quick video.

00:06:48.000 | I'm going to show you a quick video.

00:06:49.000 | I'm going to show you a quick video.

00:06:50.000 | I'm going to show you a quick video.

00:06:51.000 | I'm going to show you a quick video.

00:06:52.000 | I'm going to show you a quick video.

00:06:53.000 | I'm going to show you a quick video.

00:06:54.000 | I'm going to show you a quick video.

00:06:56.000 | So, as I was saying, AI engineers in the making.

00:06:59.000 | And this is the significant difference like to their perception

00:07:04.000 | of the smarts of AI before and after doing these learning activities.

00:07:09.000 | So, how did they -- why did that happen, right?

00:07:12.000 | Like why did they became more skeptical, more critical,

00:07:15.000 | and also more literate in how to read and write with AI?

00:07:19.000 | It's because by providing this platform and allowing them to tinker

00:07:23.000 | and form hypotheses and test them, we basically allowed them

00:07:27.000 | to engage in the scientific process just like researchers do,

00:07:30.000 | just like we do, right?

00:07:32.000 | But we needed to have the right sandbox, the right platform for them

00:07:35.000 | to be able to quickly tinker and quickly iterate.

00:07:38.000 | So, kids are not alone in learning this.

00:07:42.000 | Parents need to learn too.

00:07:43.000 | Teachers need to learn too.

00:07:44.000 | And we've seen during the pandemic when kids were stuck at home

00:07:48.000 | with parents a huge opportunity for them to learn together.

00:07:52.000 | So, I'll show you one of the early demos of Cognimate.

00:07:57.000 | The audio is now working on this one.

00:08:02.000 | I'm not sure why.

00:08:03.000 | Basic --

00:08:04.000 | There you go.

00:08:05.000 | You did it.

00:08:06.000 | No, I need you to help me ask a question.

00:08:10.000 | For that, we'll need the ask block.

00:08:12.000 | See if you can find it.

00:08:19.000 | Awesome.

00:08:20.000 | So, the thing that you are programming is kind of collaborating

00:08:24.000 | with you to teach you how to program it, right?

00:08:26.000 | Just imagine applying that to any of the chatbots we have today, right?

00:08:31.000 | Like when you're not happy with the answer or maybe the answer is not age appropriate

00:08:35.000 | or you want to teach -- you also want to teach something to the model

00:08:39.000 | about your language, your culture, weird facts that you're interested about.

00:08:44.000 | How do we do that, right?

00:08:46.000 | So, I did another study where I -- this was with kids and parents in 10 different states

00:08:51.000 | in the U.S. over multiple weeks where we wanted first to learn how do we design a co-pilot

00:08:58.000 | for programming for families.

00:09:00.000 | So, before we start and build it, like what do they want?

00:09:04.000 | What works and what doesn't?

00:09:06.000 | So, what we found was that some of the things that kids and parents liked the most was to

00:09:12.000 | generate coding ideas with an AI friend.

00:09:15.000 | Like if they had a co-pilot in Scratch.

00:09:18.000 | And this was very, very helpful.

00:09:20.000 | Here are some quotes.

00:09:22.000 | Because, like here, like one of the participants says, most people would like coding with AI friends

00:09:27.000 | because one of the hardest parts of your project is when you start, you run into a wall

00:09:32.000 | because you're out of ideas.

00:09:34.000 | So, the AI friend helped with that.

00:09:37.000 | It also allowed them to express and elaborate their ideas in code.

00:09:41.000 | So, if they had an idea for a game, like I want to make the bear kind of jump over the hedgehog,

00:09:47.000 | but they didn't know how to do it, it would kind of help them find the right code constructs to do it.

00:09:54.000 | And more importantly, it supported their creative coding identity.

00:09:59.000 | So, it wasn't the bot that was making all the coding.

00:10:02.000 | They were doing it.

00:10:03.000 | The bot was just helping them when they were stuck.

00:10:05.000 | So, this was very, very important.

00:10:08.000 | It encouraged kids and parents to work together, which is not always easy, right?

00:10:13.000 | Like one of the things I discovered, I've been working with kids and families,

00:10:16.000 | for a long time now, since 2015.

00:10:20.000 | It's not always easy.

00:10:21.000 | So, actually having like a third moderator to be like, oh, what does mommy say?

00:10:25.000 | What does daddy think?

00:10:27.000 | Take turns.

00:10:28.000 | Try this.

00:10:29.000 | Really helped with family joint engagement.

00:10:32.000 | Also, it doesn't always work, right?

00:10:34.000 | Sometimes it's too distracting and it was very important to enable families to shut it off.

00:10:38.000 | Maybe they want to do the game alone.

00:10:40.000 | They want to do the coding alone.

00:10:41.000 | So, they could stop it whenever they wanted.

00:10:43.000 | If you have multiple siblings that fight over the laptop, it doesn't really -- it cannot help with that.

00:10:49.000 | Or if the concepts were too complex, it was not able to scaffold it always, like break it down.

00:10:55.000 | So, parents were very helpful to help.

00:10:58.000 | So, after understanding like what are the core things that families want from co-creating and learning how to program with an AI friend, I went and basically evaluated all the generative AI models to see if they could do that, right?

00:11:12.000 | So, scratch for scratch, like top generative AI models are pretty good at generating explanations, giving like ideas or questions to help kids and parents like explore and test like new games.

00:11:27.000 | And this was published.

00:11:28.000 | We created a benchmark as well for measuring this.

00:11:31.000 | And this is just an example of what the future of education with multimodal AI could look like.

00:11:38.000 | If it's applied to Minecraft, to games, to physics simulations, science simulations, it can become a creative sidekick, right?

00:11:47.000 | There are a lot of people who love to build things with their hands.

00:11:50.000 | What if I could get ideas like by taking pictures of flowers I like and colors I like and it gives me ideas and helps me like generate 3D models and that I could afterwards print and paint.

00:12:03.000 | Or I'm into knitting and I want to use a generative AI model to inspire my knitting projects.

00:12:08.000 | It can also be a learning companion and a coach.

00:12:11.000 | It can help with math.

00:12:13.000 | So, together with Nancy Otero, we created the first benchmark for math misconceptions to show what are the most common math problems that kids have in K through 12 and evaluate how good are top of art generative AI models in identifying these misconceptions when kids talk with a chatbot.

00:12:32.000 | And I put a link to it if you want to download it.

00:12:35.000 | So, I am here to invite you to think about AI engineering and AI tinkering for all ages.

00:12:42.000 | How do we go from my experiments to Cognomates to things that people are doing and tinkering with hugging face and make sure like we open up the space.

00:12:50.000 | So, we use AI not just to teach, but we actually use AI for people to learn how to tinker and learn by playing and learn by doing.

00:12:59.000 | So, I like to do what I preach and I'm going to show what I tinkered with AI last night.

00:13:05.000 | These are very fresh demos.

00:13:07.000 | So, this is using the latest Gemini API.

00:13:13.000 | And I have three demos.

00:13:15.000 | Let's hope they work.

00:13:16.000 | Let's start with the science one.

00:13:26.000 | And I was hoping to draw in real time, but I don't have a table.

00:13:32.000 | So, luckily I have some drawings and we'll see how well this works.

00:13:37.000 | So, I have a drawing.

00:13:38.000 | A scale with a weight on each side.

00:13:52.000 | What would happen if you add another five kilograms?

00:13:54.000 | So, it's asking me questions based on my drawings.

00:13:57.000 | And then I can make a new drawing that has like 10 kilograms and 10 kilograms and see if that gets better.

00:14:03.000 | Let's try another one.

00:14:07.000 | Water and CO2.

00:14:12.000 | What happens if it gets mixed?

00:14:15.000 | Oh, I need to -- so, imagine I have a webcam and I'm like a table and I'm drawing in real time and we could play with it.

00:14:21.000 | But it's very interactive.

00:14:23.000 | Let's try this one.

00:14:25.000 | The art is being hit by something.

00:14:30.000 | Hopefully not.

00:14:32.000 | Let's add one more arrow and see what happens if we do that.

00:14:37.000 | Ah, that was fun.

00:14:50.000 | So, it finally understood it was the moon.

00:14:52.000 | Let's play with the math one.

00:14:54.000 | Solve the expression inside the parenthesis.

00:15:22.000 | Okay.

00:15:23.000 | So, I have one where I did -- okay.

00:15:35.000 | Solve the multiplication with the parenthesis.

00:15:38.000 | Let's assume I've done that too and I have the next question.

00:15:43.000 | I need a better background for this demo.

00:15:46.000 | That's for sure.

00:15:47.000 | The first step is to simplify.

00:15:49.000 | No, no, no.

00:15:50.000 | Go back.

00:15:51.000 | Okay.

00:15:52.000 | So, any number divided by itself equals one.

00:15:54.000 | But you see, it doesn't give me the answer.

00:15:55.000 | It just gives me a question so I can keep trying, right?

00:15:58.000 | And learning.

00:15:59.000 | Let's try one more.

00:16:02.000 | More complicated.

00:16:04.000 | So, okay.

00:16:05.000 | Four on the axis.

00:16:07.000 | I was hoping it would give me a better question.

00:16:12.000 | Yeah.

00:16:13.000 | So, the last one is the one that is encouraging curiosity.

00:16:17.000 | So, this one --

00:16:18.000 | What is the lady doing?

00:16:32.000 | Okay.

00:16:33.000 | What are the colors on the flag?

00:16:34.000 | What shape is the star?

00:16:35.000 | Oh, it asked me things about Jordan.

00:16:37.000 | Let's see what it does with apple.

00:16:38.000 | Do you know what apple this is?

00:16:44.000 | Let's see.

00:16:45.000 | What does the apple smell like?

00:16:46.000 | It had a nice origami thing as well.

00:16:50.000 | Oh, it's a nice origami thing as well.

00:16:51.000 | Oh, it's a nice origami thing.

00:16:52.000 | Oh, it's a nice origami thing.

00:16:53.000 | Oh, it's a nice origami thing.

00:16:54.000 | Oh, it's a nice origami thing.

00:16:55.000 | Oh, it's a nice origami thing.

00:16:56.000 | Oh, it's a nice origami thing.

00:16:57.000 | Oh, it's a nice origami thing.

00:16:58.000 | Oh, it's a nice origami thing.

00:16:59.000 | Oh, it's a nice origami thing.

00:17:00.000 | Oh, it's a nice origami thing.

00:17:01.000 | Oh, it's a nice origami thing.

00:17:02.000 | Oh, it's a nice origami thing.

00:17:03.000 | Oh, it's a nice origami thing.

00:17:04.000 | Oh, it's a nice origami thing.

00:17:05.000 | Oh, it's a nice origami thing.

00:17:06.000 | Oh, it's a nice origami thing.

00:17:07.000 | Oh, it's a nice origami thing.

00:17:08.000 | Oh, it's a nice origami thing.

00:17:09.000 | Oh, it's a nice origami thing.

00:17:10.000 | Oh, it's a nice origami thing.

00:17:11.000 | Oh, it's a nice origami thing.

00:17:12.000 | Oh, it's a nice origami thing.

00:17:13.000 | Oh, it's a nice origami thing.

00:17:14.000 | Oh, it's a nice origami thing.

00:17:15.000 | Oh, it's a nice origami thing.

00:17:16.000 | Oh, it's a nice origami thing.

00:17:17.000 | Oh, it's a nice origami thing.

00:17:18.000 | Oh, it's a nice origami thing.

00:17:19.000 | Oh, it's a nice origami thing as well.

00:17:20.000 | Okay.

00:17:21.000 | I don't know where the origami went.

00:17:22.000 | Oh, it's a nice origami thing as well.

00:17:23.000 | Okay.

00:17:24.000 | I don't know where the origami went, but.

00:17:26.000 | So, you sort of get the gist.

00:17:29.000 | These are like, I don't know, do you want me to draw something

00:17:33.000 | or do you want to ask one of the questions of the science

00:17:35.000 | or math or objects?

00:17:37.000 | Any requests from the audience?

00:17:41.000 | Don't be shy.

00:17:50.000 | Yeah? What should we ask?

00:17:55.000 | No?

00:17:56.000 | Okay.

00:17:57.000 | Well --

00:18:00.000 | Excuse me?

00:18:05.000 | A system equation.

00:18:07.000 | Yeah.

00:18:08.000 | Can you tell me what to write?

00:18:09.000 | Sure.

00:18:10.000 | 2x plus 7.

00:18:12.000 | 2x plus 7 equals 2.

00:18:17.000 | Thank you.

00:18:18.000 | Let's try it.

00:18:20.000 | Let's see if it does well with my --

00:18:36.000 | Substract 7 from both sides of the equation.

00:18:38.000 | Not bad.

00:18:39.000 | And now if I do that --

00:18:58.000 | Divide both sides by 2.

00:18:59.000 | And so on and so forth.

00:19:01.000 | Now, the cool thing about this, like, I made the code open source

00:19:05.000 | and templates so you can play with it, too, is less than 100 lines.

00:19:08.000 | You just need to create an API key, which is free.

00:19:11.000 | And you can create your own instructions.

00:19:14.000 | And hopefully I inspired you to think, like, beyond of chatbot interfaces

00:19:20.000 | and delegating instructions and delegating, like, questions.

00:19:24.000 | And think more in, like, a tinkerer and think about how we could put these tools in the hands

00:19:31.000 | of young people because they are the future and they need to learn about this technology

00:19:34.000 | as well and how it works.

00:19:36.000 | I think that's my time.

00:19:39.000 | All my research is on my website.

00:19:41.000 | And I put a QR link for that as well.

00:19:44.000 | And I look forward to your questions afterwards.

00:19:45.000 | Thank you so much.

00:19:46.000 | Thank you so much.

00:19:47.000 | Thank you.

00:19:48.000 | Thank you.

00:19:49.000 | Thank you.

00:19:50.000 | Thank you.

00:19:51.000 | Thank you.

00:19:52.000 | Thank you.

00:19:53.000 | Thank you.

00:19:54.000 | Thank you.

00:19:55.000 | Thank you.

00:19:56.000 | Thank you.

00:19:57.000 | Thank you.

00:19:58.000 | Thank you.

00:19:59.000 | Thank you.

00:20:00.000 | Thank you.

00:20:00.000 | We'll see you next time.