Cristos Goodrow: YouTube Algorithm | Lex Fridman Podcast #68

00:00:00.000 | The following is a conversation with Christos Gudro,

00:00:03.400 | Vice President of Engineering at Google

00:00:05.680 | and Head of Search and Discovery at YouTube,

00:00:08.280 | also known as the YouTube Algorithm.

00:00:11.400 | YouTube has approximately 1.9 billion users,

00:00:15.120 | and every day people watch over 1 billion hours

00:00:18.760 | of YouTube video.

00:00:20.360 | It is the second most popular search engine

00:00:22.360 | behind Google itself.

00:00:24.120 | For many people, it is not only a source of entertainment,

00:00:27.280 | but also how we learn new ideas from math and physics videos,

00:00:31.400 | to podcasts, to debates, opinions, ideas,

00:00:34.560 | from out-of-the-box thinkers and activists

00:00:37.120 | on some of the most tense, challenging, and impactful

00:00:40.320 | topics in the world today.

00:00:42.360 | YouTube and other content platforms

00:00:44.880 | receive criticism from both viewers and creators,

00:00:48.120 | as they should, because the engineering task before them

00:00:51.840 | is hard, and they don't always succeed,

00:00:54.640 | and the impact of their work is truly world-changing.

00:00:58.680 | To me, YouTube has been an incredible wellspring

00:01:01.440 | of knowledge.

00:01:02.440 | I've watched hundreds, if not thousands,

00:01:04.680 | of lectures that change the way I see many fundamentals

00:01:07.760 | ideas in math, science, engineering, and philosophy.

00:01:12.560 | But it does put a mirror to ourselves

00:01:14.800 | and keeps the responsibility of the steps

00:01:16.840 | we take in each of our online educational journeys

00:01:20.280 | into the hands of each of us.

00:01:22.560 | The YouTube algorithm has an important role

00:01:24.720 | in that journey of helping us find new,

00:01:27.240 | exciting ideas to learn about.

00:01:29.320 | That's a difficult and an exciting problem

00:01:31.720 | for an artificial intelligence system.

00:01:34.040 | As I've said in lectures and other forums,

00:01:36.520 | recommendation systems will be one of the most impactful

00:01:39.400 | areas of AI in the 21st century,

00:01:42.320 | and YouTube is one of the biggest

00:01:44.440 | recommendation systems in the world.

00:01:47.400 | This is the Artificial Intelligence Podcast.

00:01:50.400 | If you enjoy it, subscribe on YouTube,

00:01:52.760 | give it five stars on Apple Podcast,

00:01:54.760 | follow on Spotify, support it on Patreon,

00:01:57.400 | or simply connect with me on Twitter,

00:01:59.520 | Alex Friedman, spelled F-R-I-D-M-A-N.

00:02:03.480 | I recently started doing ads

00:02:05.000 | at the end of the introduction.

00:02:06.560 | I'll do one or two minutes after introducing the episode

00:02:09.480 | and never any ads in the middle

00:02:11.040 | that can break the flow of the conversation.

00:02:13.320 | I hope that works for you

00:02:14.720 | and doesn't hurt the listening experience.

00:02:17.960 | This show is presented by Cash App,

00:02:20.080 | the number one finance app in the App Store.

00:02:22.560 | I personally use Cash App to send money to friends,

00:02:25.320 | but you can also use it to buy, sell,

00:02:27.120 | and deposit Bitcoin in just seconds.

00:02:29.520 | Cash App also has a new investing feature.

00:02:32.400 | You can buy fractions of a stock, say $1 worth,

00:02:35.320 | no matter what the stock price is.

00:02:37.440 | Brokerage services are provided by Cash App Investing,

00:02:40.360 | a subsidiary of Square and member SIPC.

00:02:43.640 | I'm excited to be working with Cash App

00:02:45.960 | to support one of my favorite organizations called FIRST.

00:02:49.360 | Best known for their FIRST Robotics and Lego competitions.

00:02:52.880 | They educate and inspire hundreds of thousands of students

00:02:56.160 | in over 110 countries

00:02:58.000 | and have a perfect rating, a charity navigator,

00:03:00.640 | which means that donated money

00:03:01.920 | is used to maximum effectiveness.

00:03:04.680 | When you get Cash App from the App Store or Google Play

00:03:07.360 | and use code LEXPODCAST, you'll get $10

00:03:11.320 | and Cash App will also donate $10 to FIRST,

00:03:14.240 | which again is an organization

00:03:16.080 | that I've personally seen inspire girls and boys

00:03:18.880 | to dream of engineering a better world.

00:03:22.000 | And now here's my conversation with Christos Goudreau.

00:03:25.680 | YouTube is the world's second most popular search engine,

00:03:29.440 | behind Google, of course.

00:03:31.400 | We watch more than 1 billion hours of YouTube videos a day,

00:03:35.480 | more than Netflix and Facebook video combined.

00:03:38.600 | YouTube creators upload over 500,000 hours of video

00:03:42.440 | every day.

00:03:43.880 | Average lifespan of a human being,

00:03:45.680 | just for comparison, is about 700,000 hours.

00:03:49.320 | So what's uploaded every single day

00:03:53.280 | is just enough for a human to watch in a lifetime.

00:03:56.200 | So let me ask an absurd philosophical question.

00:03:59.560 | If from birth, when I was born,

00:04:01.560 | and there's many people born today with the internet,

00:04:04.920 | I watched YouTube videos nonstop,

00:04:07.480 | do you think there are trajectories

00:04:09.400 | through YouTube video space

00:04:11.680 | that can maximize my average happiness

00:04:15.200 | or maybe education or my growth as a human being?

00:04:18.800 | - I think there are some great trajectories

00:04:21.440 | through YouTube videos,

00:04:24.000 | but I wouldn't recommend that anyone spend

00:04:26.320 | all of their waking hours

00:04:27.800 | or all of their hours watching YouTube.

00:04:30.600 | I mean, I think about the fact

00:04:32.680 | that YouTube has been really great for my kids,

00:04:34.640 | for instance.

00:04:35.560 | My oldest daughter,

00:04:38.320 | she's been watching YouTube for several years.

00:04:42.040 | She watches Tyler Oakley and the Vlogbrothers.

00:04:46.320 | And I know that it's had a very profound

00:04:48.680 | and positive impact on her character.

00:04:50.320 | And my younger daughter, she's a ballerina,

00:04:52.680 | and her teachers tell her

00:04:54.760 | that YouTube is a huge advantage for her

00:04:57.640 | because she can practice a routine

00:05:00.200 | and watch professional dancers do that same routine

00:05:04.360 | and stop it and back it up and rewind

00:05:06.920 | and all that stuff, right?

00:05:07.840 | So it's been really good for them.

00:05:10.800 | And then even my son is a sophomore in college.

00:05:13.320 | He got through his linear algebra class

00:05:17.680 | because of a channel called 3Blue1Brown,

00:05:20.280 | which helps you understand linear algebra,

00:05:24.200 | but in a way that would be very hard for anyone to do

00:05:27.120 | on a whiteboard or a chalkboard.

00:05:28.800 | And so I think that those experiences,

00:05:33.520 | from my point of view, were very good.

00:05:35.160 | And so I can imagine really good trajectories

00:05:37.720 | through YouTube, yes.

00:05:38.760 | - Have you looked at,

00:05:39.920 | do you think of broadly about that trajectory over a period?

00:05:43.600 | 'Cause YouTube has grown up now.

00:05:45.640 | So over a period of years,

00:05:47.640 | you just kind of gave a few anecdotal examples.

00:05:51.320 | But I used to watch certain shows on YouTube.

00:05:54.520 | I don't anymore.

00:05:55.360 | I've moved on to other shows.

00:05:57.560 | And ultimately you want people to,

00:05:59.600 | from YouTube's perspective, to stay on YouTube,

00:06:01.760 | to grow as human beings on YouTube.

00:06:03.740 | So you have to think not just what makes them engage today,

00:06:10.280 | or this month, but also over a period of years.

00:06:12.680 | - Absolutely, that's right.

00:06:13.960 | I mean, if YouTube is going to continue

00:06:15.960 | to enrich people's lives,

00:06:17.600 | then it has to grow with them.

00:06:21.440 | And people's interests change over time.

00:06:25.240 | And so I think we've been working on this problem,

00:06:30.240 | and I'll just say it broadly,

00:06:31.760 | is how to introduce diversity

00:06:35.280 | and introduce people who are watching one thing

00:06:38.360 | to something else they might like.

00:06:40.120 | We've been working on that problem

00:06:42.080 | all the eight years I've been at YouTube.

00:06:44.120 | It's a hard problem because,

00:06:47.040 | I mean, of course it's trivial

00:06:50.080 | to introduce diversity that doesn't help.

00:06:52.800 | - Yeah, just add a random video.

00:06:54.200 | - I could just randomly select a video

00:06:56.440 | from the billions that we have.

00:06:58.840 | It's likely not to even be in your language.

00:07:01.320 | So the likelihood that you would watch it

00:07:05.200 | and develop a new interest is very, very low.

00:07:08.680 | And so what you want to do

00:07:11.440 | when you're trying to increase diversity

00:07:13.080 | is find something that is not too similar

00:07:17.180 | to the things that you've watched,

00:07:19.240 | but also something that you might be likely to watch.

00:07:23.560 | And that balance, finding that spot

00:07:25.840 | between those two things is quite challenging.

00:07:29.520 | - So the diversity of content, diversity of ideas,

00:07:33.640 | it's a really difficult,

00:07:36.160 | it's a thing that's almost impossible to define, right?

00:07:39.520 | Like what's different?

00:07:40.960 | So how do you think about that?

00:07:43.820 | So two examples is,

00:07:46.240 | I'm a huge fan of 3Blue1Brown, say,

00:07:48.920 | and then one diversity,

00:07:51.560 | I wasn't even aware of a channel called Veritasium,

00:07:54.720 | which is a great science, physics, whatever channel.

00:07:58.280 | So one version of diversity

00:07:59.840 | is Show Me Derek's Veritasium's channel,

00:08:03.040 | which I was really excited to discover.

00:08:04.760 | I actually now watch a lot of his videos.

00:08:07.120 | - Okay, so you're a person who's watching some math channels

00:08:11.360 | and you might be interested

00:08:12.800 | in some other science or math channels.

00:08:15.200 | So like you mentioned,

00:08:16.280 | the first kind of diversity is just show you

00:08:19.000 | some things from other channels that are related,

00:08:22.600 | but not just, you know,

00:08:25.440 | not all the 3Blue1Brown channel,

00:08:28.480 | throw in a couple others.

00:08:29.440 | So that's the, maybe the first kind of diversity

00:08:32.480 | that we started with many, many years ago.

00:08:34.600 | Taking a bigger leap is about,

00:08:41.120 | I mean, the mechanisms we use for that

00:08:43.680 | is we basically cluster videos and channels together,

00:08:47.800 | mostly videos.

00:08:48.640 | We do every, almost everything at the video level.

00:08:50.800 | And so we'll make some kind of a cluster

00:08:53.360 | via some embedding process,

00:08:55.720 | and then measure, you know,

00:08:58.960 | what is the likelihood that users who watch one cluster

00:09:03.800 | might also watch another cluster that's very distinct.

00:09:06.640 | So we may come to find that people who watch science videos

00:09:11.640 | also like jazz.

00:09:15.000 | This is possible, right?

00:09:16.880 | And so, because of that relationship that we've identified

00:09:21.480 | through the embeddings

00:09:25.640 | and then the measurement of the people who watch both,

00:09:28.480 | we might recommend a jazz video once in a while.

00:09:31.560 | - So there's this clustering in the embedding space

00:09:33.720 | of jazz videos and science videos.

00:09:36.560 | And so you kind of try to look at aggregate statistics

00:09:39.960 | where if a lot of people that jump from science cluster

00:09:44.960 | to the jazz cluster tend to remain as engaged

00:09:49.800 | or become more engaged,

00:09:51.600 | then that means those two are,

00:09:54.840 | they should hop back and forth and they'll be happy.

00:09:57.360 | - Right, there's a higher likelihood

00:09:59.480 | that a person from who's watching science would like jazz

00:10:03.080 | than the person watching science would like,

00:10:06.120 | I don't know, backyard railroads or something else, right?

00:10:08.640 | And so we can try to measure these likelihoods

00:10:11.840 | and use that to make the best recommendation we can.

00:10:16.440 | - So, okay, so we'll talk about the machine learning of that,

00:10:19.360 | but I have to linger on things that neither you

00:10:22.360 | or anyone have an answer to.

00:10:24.320 | There's gray areas of truth, which is, for example,

00:10:29.320 | now I can't believe I'm going there, but politics.

00:10:33.140 | It happens so that certain people believe certain things

00:10:37.900 | and they're very certain about them.

00:10:40.240 | Let's move outside the red versus blue politics

00:10:43.040 | of today's world, but there's different ideologies.

00:10:46.100 | For example, in college, I read quite a lot of Ayn Rand.

00:10:50.200 | I studied, and that's a particular philosophical ideology

00:10:52.880 | I found interesting to explore.

00:10:55.480 | Okay, so that was that kind of space.

00:10:57.100 | I've kind of moved on from that cluster intellectually,

00:11:00.300 | but it nevertheless is an interesting cluster.

00:11:02.600 | I was born in the Soviet Union.

00:11:04.360 | Socialism, communism is a certain kind

00:11:06.800 | of political ideology that's really interesting to explore.

00:11:09.760 | Again, objectively, there's a set of beliefs

00:11:12.680 | about how the economy should work and so on.

00:11:14.960 | And so it's hard to know what's true or not

00:11:17.880 | in terms of people within those communities

00:11:20.280 | are often advocating that this is how we achieve utopia

00:11:23.920 | in this world, and they're pretty certain about it.

00:11:26.840 | So how do you try to manage politics

00:11:31.840 | in this chaotic, divisive world?

00:11:35.480 | Not politics, or any kind of ideas,

00:11:37.860 | in terms of filtering what people should watch next,

00:11:40.400 | and in terms of also not letting certain things

00:11:44.420 | be on YouTube.

00:11:45.900 | This is exceptionally difficult responsibility.

00:11:49.520 | - Well, the responsibility to get this right

00:11:53.040 | is our top priority.

00:11:54.680 | And the first comes down to making sure

00:11:59.140 | that we have good, clear rules of the road.

00:12:02.440 | Just because we have freedom of speech

00:12:05.160 | doesn't mean that you can literally say anything.

00:12:07.760 | We as a society have accepted certain restrictions

00:12:12.520 | on our freedom of speech.

00:12:14.720 | There are things like libel laws and things like that.

00:12:17.600 | And so where we can draw a clear line, we do,

00:12:22.440 | and we continue to evolve that line over time.

00:12:25.260 | However, as you pointed out, wherever you draw the line,

00:12:30.480 | there's gonna be a borderline.

00:12:33.040 | And in that borderline area,

00:12:35.840 | we are going to maybe not remove videos,

00:12:39.400 | but we will try to reduce the recommendations of them

00:12:43.360 | or the proliferation of them by demoting them,

00:12:47.080 | and then alternatively, in those situations,

00:12:49.820 | try to raise what we would call authoritative

00:12:52.960 | or credible sources of information.

00:12:55.680 | So we're not trying to, I mean,

00:12:58.020 | you mentioned Ayn Rand and communism.

00:13:01.240 | Those are two valid points of view

00:13:05.720 | that people are gonna debate and discuss.

00:13:08.060 | And of course, people who believe

00:13:11.520 | in one or the other of those things

00:13:13.180 | are gonna try to persuade other people

00:13:14.800 | to their point of view.

00:13:16.320 | And so we're not trying to settle that

00:13:20.600 | or choose a side or anything like that.

00:13:22.580 | What we're trying to do is make sure

00:13:24.000 | that the people who are expressing those point of view

00:13:28.160 | and offering those positions are authoritative and credible.

00:13:33.040 | - So let me ask a question

00:13:37.020 | about people I don't like personally.

00:13:39.400 | You heard me, I don't care if you leave comments on this.

00:13:44.080 | But sometimes they're brilliantly funny, which is trolls.

00:13:47.600 | So people who kind of mock,

00:13:52.360 | I mean, the internet is full,

00:13:53.760 | the Reddit of mock-style comedy

00:13:57.080 | where people just kind of make fun of,

00:13:59.720 | point out that the emperor has no clothes.

00:14:02.240 | And there's brilliant comedy in that,

00:14:03.720 | but sometimes it can get cruel and mean.

00:14:05.980 | So on that, on the mean point,

00:14:10.760 | and sorry to linger on these things

00:14:12.320 | that have no good answers,

00:14:14.040 | but actually I totally hear you

00:14:16.840 | that this is really important that you're trying to solve it.

00:14:20.040 | But how do you reduce the meanness of people on YouTube?

00:14:25.040 | - I understand that anyone who uploads YouTube videos

00:14:30.660 | has to become resilient to a certain amount of meanness.

00:14:35.560 | Like I've heard that from many creators.

00:14:38.080 | And we are trying in various ways,

00:14:43.080 | comment ranking, allowing certain features to block people,

00:14:48.700 | to reduce or make that meanness

00:14:52.360 | or that trolling behavior less effective on YouTube.

00:14:57.120 | And so, I mean, it's very important,

00:15:02.120 | but it's something that we're gonna keep having to work on.

00:15:06.600 | And as we improve it,

00:15:09.360 | like maybe we'll get to a point

00:15:10.800 | where people don't have to suffer this sort of meanness

00:15:15.640 | when they upload YouTube videos.

00:15:16.960 | I hope we do, but it just does seem to be something

00:15:21.960 | that you have to be able to deal with

00:15:24.420 | as a YouTube creator nowadays.

00:15:26.120 | - Do you have a hope that,

00:15:27.080 | so you mentioned two things that I kind of agree with.

00:15:29.800 | So there's like a machine learning approach

00:15:32.880 | of ranking comments based on whatever,

00:15:36.920 | based on how much they contribute

00:15:38.480 | to the healthy conversation.

00:15:40.560 | Let's put it that way.

00:15:41.720 | Then the other is almost an interface question

00:15:45.160 | of how does the creator filter?

00:15:49.040 | So block or how do humans themselves,

00:15:53.820 | the users of YouTube manage their own conversation?

00:15:57.100 | Do you have hope that these two tools

00:15:58.640 | will create a better society

00:16:00.980 | without limiting freedom of speech too much,

00:16:04.340 | without sort of attaching, even like saying that people,

00:16:07.700 | like what do you mean limiting, sort of curating speech?

00:16:12.700 | - I mean, I think that that overall

00:16:15.420 | is our whole project here at YouTube.

00:16:17.100 | - Right.

00:16:17.940 | - Like we fundamentally believe,

00:16:19.820 | and I personally believe very much

00:16:22.100 | that YouTube can be great.

00:16:24.860 | It's been great for my kids.

00:16:26.220 | I think it can be great for society,

00:16:29.500 | but it's absolutely critical

00:16:31.260 | that we get this responsibility part right.

00:16:34.500 | And that's why it's our top priority.

00:16:37.100 | Susan Wojcicki, who's the CEO of YouTube,

00:16:39.520 | she says something that I personally find very inspiring,

00:16:43.100 | which is that we wanna do our jobs today

00:16:48.100 | in a manner so that people 20 and 30 years from now

00:16:51.640 | will look back and say, you know, YouTube,

00:16:53.420 | they really figured this out.

00:16:55.020 | They really found a way to strike the right balance

00:16:58.340 | between the openness and the value that the openness has,

00:17:02.620 | and also making sure that we are meeting our responsibility

00:17:06.220 | to users in society.

00:17:07.940 | - So the burden on YouTube actually is quite incredible.

00:17:12.220 | And the one thing that people don't give enough credit

00:17:16.460 | to the seriousness and the magnitude of the problem, I think.

00:17:20.060 | So I personally hope that you do solve it

00:17:23.620 | 'cause a lot is in your hand.

00:17:26.700 | A lot is riding on your success or failure.

00:17:28.940 | So it's, besides of course running a successful company,

00:17:32.620 | you're also curating the content of the internet

00:17:36.040 | and the conversation on the internet.

00:17:37.500 | That's a powerful thing.

00:17:39.880 | So one thing that people wonder about

00:17:44.220 | is how much of it can be solved with pure machine learning.

00:17:49.960 | So looking at the data, studying the data,

00:17:52.420 | and creating algorithms that curate,

00:17:55.420 | the comments curate the content,

00:17:57.260 | and how much of it needs human intervention.

00:18:00.660 | Meaning people here at YouTube in a room

00:18:04.540 | sitting and thinking about what is the nature of truth?

00:18:08.300 | What is, what are the ideals that we should be promoting?

00:18:14.300 | That kind of thing.

00:18:15.860 | So algorithm versus human input.

00:18:18.580 | What's your sense?

00:18:20.100 | - I mean, my own experience has demonstrated

00:18:23.320 | that you need both of those things.

00:18:25.200 | Algorithms, I mean, you're familiar

00:18:28.740 | with machine learning algorithms,

00:18:30.100 | and the thing they need most is data.

00:18:33.120 | And the data is generated by humans.

00:18:36.140 | And so for instance, when we're building a system

00:18:40.320 | to try to figure out which are the videos

00:18:43.780 | that are misinformation or borderline policy violations,

00:18:48.780 | well, the first thing we need to do

00:18:51.860 | is get human beings to make decisions

00:18:54.120 | about which of those videos are in which category.

00:18:58.740 | And then we use that data and basically,

00:19:02.380 | take that information that's determined

00:19:05.020 | and governed by humans and extrapolate it

00:19:08.620 | or apply it to the entire set of billions of YouTube videos.

00:19:13.620 | And we couldn't get to all the videos on YouTube well

00:19:19.480 | without the humans, and we couldn't use the humans

00:19:22.560 | to get to all the videos of YouTube.

00:19:24.380 | So there's no world in which you have only one

00:19:28.160 | or the other of these things.

00:19:29.760 | And just as you said, a lot of it comes down

00:19:35.180 | to people at YouTube spending a lot of time

00:19:40.180 | trying to figure out what are the right policies?

00:19:43.360 | What are the outcomes based on those policies?

00:19:46.880 | Are they the kinds of things we wanna see?

00:19:49.640 | And then once we kind of get an agreement

00:19:53.840 | or build some consensus around what the policies are,

00:19:56.920 | well, then we've gotta find a way

00:19:58.440 | to implement those policies across all of YouTube.

00:20:02.060 | And that's where both the human beings,

00:20:05.600 | we call them evaluators or reviewers,

00:20:08.000 | come into play to help us with that.

00:20:10.200 | And then once we get a lot of training data from them,

00:20:13.580 | then we apply the machine learning techniques

00:20:15.660 | to take it even further.

00:20:17.600 | - Do you have a sense that these human beings

00:20:20.160 | have a bias in some kind of direction?

00:20:24.180 | Sort of, I mean, that's an interesting question.

00:20:27.760 | We do sort of in autonomous vehicles

00:20:30.160 | and computer vision in general a lot of annotation,

00:20:32.760 | and we rarely ask what bias do the annotators have?

00:20:37.760 | Even in the sense that they're better

00:20:44.940 | at annotating certain things than others.

00:20:47.460 | For example, people are much better

00:20:49.140 | at annotating segmentation at segmenting cars in a scene

00:20:54.140 | versus segmenting bushes or trees.

00:20:57.560 | You know, there's specific mechanical reasons for that,

00:21:01.180 | but also because it's semantic gray area.

00:21:05.060 | And just for a lot of reasons,

00:21:07.000 | people are just terrible at annotating trees.

00:21:09.620 | Okay, so in the same kind of sense,

00:21:11.720 | do you think of, in terms of people reviewing videos

00:21:15.140 | or annotating the content of videos,

00:21:17.460 | is there some kind of bias that you're aware of

00:21:21.300 | or seek out in that human input?

00:21:24.220 | - Well, we take steps to try to overcome

00:21:27.960 | these kinds of biases or biases

00:21:29.940 | that we think would be problematic.

00:21:32.160 | So for instance, like we ask people

00:21:36.140 | to have a bias towards scientific consensus.

00:21:38.500 | That's something that we instruct them to do.

00:21:42.340 | We ask them to have a bias towards demonstration

00:21:46.740 | of expertise or credibility or authoritativeness.

00:21:50.620 | But there are other biases that we wanna make sure

00:21:53.620 | to try to remove.

00:21:55.400 | And there's many techniques for doing this.

00:21:57.740 | One of them is you send the same thing

00:22:00.960 | to be reviewed to many people.

00:22:03.780 | And so, you know, that's one technique.

00:22:06.220 | Another is that you make sure that the people

00:22:08.140 | that are doing these sorts of tasks

00:22:11.300 | are from different backgrounds

00:22:13.580 | and different areas of the United States or of the world.

00:22:16.500 | But then, even with all of that,

00:22:18.700 | it's possible for certain kinds of

00:22:21.620 | what we would call unfair biases

00:22:25.500 | to creep into machine learning systems,

00:22:27.720 | primarily, as you said,

00:22:29.760 | because maybe the training data itself comes in

00:22:32.400 | in a biased way.

00:22:33.560 | And so, we also have worked very hard

00:22:37.920 | on improving the machine learning systems

00:22:41.120 | to remove and reduce unfair biases

00:22:44.200 | when it goes against

00:22:47.480 | or has involved some protected class, for instance.

00:22:51.600 | - Thank you for exploring with me

00:22:53.920 | some of the more challenging things.

00:22:55.840 | I'm sure there's a few more that we'll jump back to,

00:22:58.040 | but let me jump into the fun part,

00:23:00.880 | which is maybe the basics

00:23:03.800 | of the quote-unquote YouTube algorithm.

00:23:05.860 | What does the YouTube algorithm look at

00:23:09.560 | to make recommendation for what to watch next?

00:23:11.760 | And it's from a machine learning perspective.

00:23:14.540 | Or when you search for a particular term,

00:23:18.280 | how does it know what to show you next?

00:23:20.440 | 'Cause it seems to, at least for me,

00:23:22.880 | do an incredible job of both.

00:23:25.400 | - Well, that's kind of you to say.

00:23:27.200 | It didn't used to do a very good job.

00:23:29.240 | (both laughing)

00:23:30.720 | But it's gotten better over the years.

00:23:32.680 | Even I observe that it's improved quite a bit.

00:23:35.360 | Those are two different situations.

00:23:37.760 | Like when you search for something,

00:23:40.280 | YouTube uses the best technology we can get from Google

00:23:45.200 | to make sure that the YouTube search system

00:23:48.840 | finds what someone's looking for.

00:23:50.720 | And of course, the very first things that one thinks about

00:23:54.560 | is, okay, well, does the word occur in the title,

00:23:57.720 | for instance?

00:23:59.760 | But there are much more sophisticated things

00:24:04.680 | where we're mostly trying to do some syntactic match

00:24:09.000 | or maybe a semantic match based on words

00:24:12.880 | that we can add to the document itself.

00:24:16.640 | For instance, maybe is this video watched a lot

00:24:21.640 | after this query?

00:24:23.600 | Right, that's something that we can observe.

00:24:27.000 | And then as a result, make sure that that document

00:24:32.000 | would be retrieved for that query.

00:24:34.940 | Now, when you talk about what kind of videos

00:24:37.860 | would be recommended to watch next,

00:24:41.220 | that's something, again,

00:24:43.100 | we've been working on for many years.

00:24:44.700 | And probably the first real attempt to do that well

00:24:49.700 | was to use collaborative filtering.

00:24:56.020 | So you- - Can you describe

00:24:58.140 | what collaborative filtering is?

00:24:59.620 | - Sure, it's just, basically what we do

00:25:02.660 | is we observe which videos get watched close together

00:25:07.660 | by the same person.

00:25:08.940 | And if you observe that,

00:25:12.940 | and if you can imagine creating a graph

00:25:15.700 | where the videos that get watched close together

00:25:18.980 | by the most people are sort of very close to one another

00:25:21.660 | in this graph,

00:25:22.500 | and videos that don't frequently get watched

00:25:24.580 | close together by the same person or the same people

00:25:28.080 | are far apart,

00:25:29.380 | then you end up with this graph

00:25:33.000 | that we call the related graph

00:25:34.940 | that basically represents videos that are very similar

00:25:38.820 | or related in some way.

00:25:40.900 | And what's amazing about that

00:25:43.980 | is that it puts all the videos

00:25:47.460 | that are in the same language together, for instance.

00:25:50.140 | And we didn't even have to think about language.

00:25:52.540 | It just does it, right?

00:25:55.300 | And it puts all the videos that are about sports together,

00:25:57.940 | and it puts most of the music videos together,

00:26:00.020 | and it puts all of these sorts of videos together

00:26:02.520 | just because that's sort of the way

00:26:05.620 | the people using YouTube behave.

00:26:08.300 | - So that already cleans up a lot of the problem.

00:26:12.700 | It takes care of the lowest hanging fruit,

00:26:15.500 | which happens to be a huge one

00:26:17.820 | of just managing these millions of videos.

00:26:20.880 | - That's right.

00:26:21.800 | I remember a few years ago,

00:26:23.820 | I was talking to someone

00:26:25.540 | who was trying to propose

00:26:28.740 | that we do a research project

00:26:31.500 | concerning people who are bilingual.

00:26:36.180 | And this person was making this proposal

00:26:40.580 | based on the idea that YouTube

00:26:43.120 | could not possibly be good

00:26:45.280 | at recommending videos well to people who are bilingual.

00:26:49.320 | And so she was telling me about this,

00:26:54.880 | and I said, "Well, can you give me an example

00:26:56.420 | "of what problem do you think we have on YouTube

00:26:58.860 | "with the recommendations?"

00:26:59.980 | And so she said, "Well, I'm a researcher in the US,

00:27:04.980 | "and when I'm looking for academic topics,

00:27:07.280 | "I wanna see them in English."

00:27:10.020 | And so she searched for one, found a video,

00:27:12.500 | and then looked at the Watch Next suggestions,

00:27:14.740 | and they were all in English.

00:27:16.700 | And so she said, "Oh, I see.

00:27:18.020 | "YouTube must think that I speak only English."

00:27:20.980 | And so she said, "Now, I'm actually originally from Turkey,

00:27:24.220 | "and sometimes when I'm cooking,

00:27:25.600 | "let's say I wanna make some baklava,

00:27:27.380 | "I really like to watch videos that are in Turkish."

00:27:30.060 | And so she searched for a video about making the baklava,

00:27:33.420 | and then selected it, and it was in Turkish,

00:27:35.980 | and the Watch Next recommendations were in Turkish.

00:27:38.120 | And she just couldn't believe how this was possible.

00:27:41.980 | And how is it that you know

00:27:44.020 | that I speak both these two languages

00:27:46.040 | and put all the videos together?

00:27:47.240 | And it's just sort of an outcome of this related graph

00:27:51.260 | that's created through collaborative filtering.

00:27:54.060 | - So for me, one of my huge interests

00:27:55.900 | is just human psychology, right?

00:27:57.300 | And that's such a powerful platform

00:28:00.980 | on which to utilize human psychology

00:28:03.340 | to discover what individual people wanna watch next.

00:28:07.320 | But it's also be just fascinating to me.

00:28:09.500 | You know, Google search has ability

00:28:14.860 | to look at your own history.

00:28:17.140 | And I've done that before.

00:28:19.260 | Just what I've searched, three years, for many, many years.

00:28:22.620 | And it's a fascinating picture of who I am, actually.

00:28:25.860 | And I don't think anyone's ever summarized.

00:28:29.660 | I personally would love that.

00:28:31.580 | A summary of who I am as a person on the internet, to me.

00:28:36.060 | Because I think it reveals,

00:28:37.500 | I think it puts a mirror to me or to others,

00:28:42.060 | you know, that's actually quite revealing and interesting.

00:28:45.060 | Just maybe the number of, it's a joke,

00:28:49.620 | but not really, it's the number of cat videos I've watched.

00:28:53.300 | Or videos of people falling, you know,

00:28:55.220 | stuff that's absurd, that kind of stuff.

00:28:59.260 | It's really interesting.

00:29:00.300 | And of course, it's really good

00:29:01.400 | for the machine learning aspect to show,

00:29:05.940 | to figure out what to show next.

00:29:07.060 | But it's interesting.

00:29:08.160 | Have you just, as a tangent, played around

00:29:11.740 | with the idea of giving a map to people,

00:29:16.100 | sort of, as opposed to just using this information

00:29:19.500 | to show what's next, showing them,

00:29:22.040 | here are the clusters you've loved over the years,

00:29:24.700 | kind of thing?

00:29:25.720 | - Well, we do provide the history

00:29:27.780 | of all the videos that you've watched.

00:29:29.300 | - Yes.

00:29:30.120 | - So you can definitely search through that

00:29:31.580 | and look through it and search through it

00:29:33.020 | to see what it is that you've been watching on YouTube.

00:29:35.620 | We have actually, in various times,

00:29:40.060 | experimented with this sort of cluster idea,

00:29:44.120 | finding ways to demonstrate or show people

00:29:48.400 | what topics they've been interested in

00:29:50.080 | or what clusters they've watched from.

00:29:52.660 | It's interesting that you bring this up

00:29:54.420 | because in some sense,

00:29:57.860 | the way the recommendation system of YouTube sees a user

00:30:02.220 | is exactly as the history of all the videos

00:30:05.260 | they've watched on YouTube.

00:30:06.980 | And so you can think of yourself

00:30:10.840 | or any user on YouTube as kind of like a DNA strand

00:30:17.040 | of all your videos, right?

00:30:20.260 | That sort of represents you.

00:30:22.820 | You can also think of it as maybe a vector

00:30:24.660 | in the space of all the videos on YouTube.

00:30:27.020 | And so now, once you think of it as a vector

00:30:31.900 | in the space of all the videos on YouTube,

00:30:33.540 | then you can start to say, okay, well,

00:30:35.660 | which other vectors are close to me, to my vector?

00:30:40.660 | And that's one of the ways

00:30:43.540 | that we generate some diverse recommendations

00:30:45.820 | because you're like, okay, well,

00:30:47.360 | you know, these people seem to be close

00:30:50.480 | with respect to the videos they've watched on YouTube,

00:30:52.640 | but here's a topic or a video

00:30:55.400 | that one of them has watched and enjoyed,

00:30:57.600 | but the other one hasn't.

00:30:59.400 | That could be an opportunity to make a good recommendation.

00:31:02.880 | - I gotta tell you, I mean, I know,

00:31:04.400 | I'm gonna ask for things that are impossible,

00:31:05.920 | but I would love to cluster them human beings.

00:31:09.560 | Like I would love to know who has similar trajectories as me

00:31:13.240 | 'cause you probably would wanna hang out, right?

00:31:15.680 | There's a social aspect there.

00:31:17.920 | Like actually finding some of the most fascinating people

00:31:20.360 | I find on YouTube have like no followers

00:31:22.760 | and I start following them

00:31:23.880 | and they create incredible content.

00:31:26.520 | And on that topic, I just love to ask,

00:31:29.400 | there's some videos that just blow my mind

00:31:31.600 | in terms of quality and depth

00:31:34.800 | and just in every regard are amazing videos

00:31:38.480 | and they have like 57 views.

00:31:41.440 | Okay, how do you get videos of quality

00:31:46.440 | to be seen by many eyes?

00:31:48.720 | So the measure of quality, is it just something?

00:31:52.880 | Yeah, how do you know that something is good?

00:31:55.700 | - Well, I mean, I think it depends initially

00:31:57.520 | on what sort of video we're talking about.

00:32:00.500 | So in the realm of, let's say,

00:32:04.080 | you mentioned politics and news.

00:32:06.400 | In that realm,

00:32:10.440 | quality news or quality journalism

00:32:13.080 | relies on having a journalism department, right?

00:32:18.080 | Like you have to have actual journalists

00:32:20.680 | and fact checkers and people like that.

00:32:22.640 | And so in that situation and in others,

00:32:26.960 | maybe science or in medicine,

00:32:29.800 | quality has a lot to do with the authoritativeness

00:32:32.760 | and the credibility and the expertise

00:32:34.840 | of the people who make the video.

00:32:36.480 | Now, if you think about the other end of the spectrum,

00:32:41.120 | you know, what is the highest quality prank video?

00:32:43.560 | Or what is the highest quality Minecraft video, right?

00:32:48.200 | That might be the one that people enjoy watching the most

00:32:52.640 | and watch to the end.

00:32:53.920 | Or it might be the one that when we ask people the next day

00:32:58.920 | after they watched it, were they satisfied with it?

00:33:04.120 | And so we, especially in the realm of entertainment,

00:33:09.240 | have been trying to get at better and better measures

00:33:12.760 | of quality or satisfaction or enrichment

00:33:17.760 | since I came to YouTube.

00:33:19.120 | And we started with, well, you know,

00:33:21.960 | the first approximation is the one that gets more views.

00:33:24.840 | But, you know, we both know

00:33:28.520 | that things can get a lot of views

00:33:30.640 | and not really be that high quality,

00:33:33.680 | especially if people are clicking on something

00:33:35.720 | and then immediately realizing that it's not that great

00:33:38.880 | and abandoning it.

00:33:39.840 | And that's why we moved from views

00:33:43.440 | to thinking about the amount of time

00:33:45.240 | people spend watching it,

00:33:47.080 | with the premise that like, you know, in some sense,

00:33:50.080 | the time that someone spends watching a video

00:33:54.040 | is related to the value that they get from that video.

00:33:57.520 | It may not be perfectly related,

00:33:59.200 | but it has something to say about how much value they get.

00:34:02.760 | But even that's not good enough, right?

00:34:05.480 | Because I myself have spent time

00:34:09.040 | clicking through channels on television late at night

00:34:11.680 | and ended up watching "Under Siege 2"

00:34:14.680 | for some reason I don't know.

00:34:16.480 | And if you were to ask me the next day,

00:34:18.200 | are you glad that you watched that show on TV last night?

00:34:22.400 | I'd say, yeah, I wish I would have gone to bed

00:34:24.760 | or read a book or almost anything else, really.

00:34:27.760 | And so that's why some people got the idea a few years ago

00:34:33.400 | to try to survey users afterwards.

00:34:35.440 | And so we get feedback data from those surveys

00:34:40.440 | and then use that in the machine learning system

00:34:43.880 | to try to not just predict

00:34:45.120 | what you're gonna click on right now,

00:34:47.280 | what you might watch for a while,

00:34:49.120 | but what when we ask you tomorrow,

00:34:51.560 | you'll give four or five stars to.

00:34:53.920 | - So just to summarize,

00:34:56.640 | what are the signals from a machine learning perspective

00:34:59.040 | that a user can provide?

00:35:00.160 | So you mentioned just clicking on the video views,

00:35:02.880 | the time watched, maybe the relative time watched,

00:35:05.960 | the clicking like and dislike on the video,

00:35:10.960 | maybe commenting on the video.

00:35:12.800 | - All of those things.

00:35:14.560 | - All of those things.

00:35:15.380 | And then the one I wasn't actually quite aware of,

00:35:18.720 | even though I might've engaged in it,

00:35:20.680 | is a survey afterwards, which is a brilliant idea.

00:35:24.560 | Is there other signals?

00:35:26.300 | I mean, that's already a really rich space

00:35:29.160 | of signals to learn from.

00:35:30.580 | Is there something else?

00:35:31.880 | - Well, you mentioned commenting, also sharing the video.

00:35:35.920 | If you think it's worthy to be shared

00:35:38.360 | with someone else you know.

00:35:39.320 | - Within YouTube or outside of YouTube as well?

00:35:41.280 | - Either.

00:35:42.200 | Let's see, you mentioned like, dislike.

00:35:44.680 | - Yeah, like and dislike, how important is that?

00:35:47.320 | - It's very important, right?

00:35:48.400 | We want, it's predictive of satisfaction,

00:35:52.840 | but it's not perfectly predictive.

00:35:56.680 | Subscribe, if you subscribe to the channel

00:35:59.940 | of the person who made the video,

00:36:01.460 | then that also is a piece of information

00:36:04.640 | and it signals satisfaction.

00:36:07.120 | Although, over the years, we've learned

00:36:10.640 | that people have a wide range of attitudes

00:36:13.640 | about what it means to subscribe.

00:36:15.680 | We would ask some users who didn't subscribe very much,

00:36:21.020 | but they watched a lot from a few channels,

00:36:24.800 | we'd say, "Well, why didn't you subscribe?"

00:36:26.320 | And they would say, "Well, I can't afford

00:36:28.200 | "to pay for anything."

00:36:29.320 | (both laughing)

00:36:31.120 | And we tried to let them understand,

00:36:33.480 | like actually it doesn't cost anything, it's free,

00:36:35.680 | it just helps us know that you are very interested

00:36:38.480 | in this creator.

00:36:40.060 | But then we've asked other people

00:36:42.440 | who subscribe to many things

00:36:44.960 | and don't really watch any of the videos from those channels

00:36:49.040 | and we say, "Well, why did you subscribe to this

00:36:52.240 | "if you weren't really interested in any more videos

00:36:55.440 | "from that channel?"

00:36:56.280 | And they might tell us, "Well, I just,

00:36:58.600 | "I thought the person did a great job

00:37:00.120 | "and I just wanted to kind of give him a high five."

00:37:01.920 | - Yeah. - Right?

00:37:03.120 | And so-- - Yeah, that's where I sit.

00:37:05.440 | I actually subscribe to channels where I just,

00:37:09.000 | this person is amazing.

00:37:11.360 | I like this person, but then I like this person

00:37:15.160 | and I really wanna support them.

00:37:16.760 | That's how I click subscribe.

00:37:19.520 | Even though I may never actually want to click

00:37:21.720 | on their videos when they're releasing it,

00:37:23.560 | I just love what they're doing.

00:37:24.960 | And it's maybe outside of my interest area and so on,

00:37:29.200 | which is probably the wrong way to use the subscribe button.

00:37:31.720 | But I just wanna say congrats.

00:37:33.400 | This is great work. (laughs)

00:37:35.520 | - Well, I mean-- - So you have to deal

00:37:36.600 | with all the space of people that see the subscribe button

00:37:39.280 | as totally different. - That's right.

00:37:40.640 | And so we can't just close our eyes and say,

00:37:44.560 | "Sorry, you're using it wrong.

00:37:46.840 | "We're not gonna pay attention to what you've done."

00:37:50.160 | We need to embrace all the ways

00:37:51.800 | in which all the different people in the world

00:37:53.600 | use the subscribe button or the like and the dislike button.

00:37:57.720 | - So in terms of signals of machine learning,

00:38:00.400 | using for the search and for the recommendation,

00:38:05.280 | you've mentioned title, so like metadata,

00:38:07.200 | like text data that people provide,

00:38:08.840 | description and title, and maybe keywords.

00:38:13.560 | So maybe you can speak to the value of those things

00:38:17.040 | in search and also this incredible,

00:38:19.880 | fascinating area of the content itself.

00:38:22.760 | So the video content itself,

00:38:24.200 | trying to understand what's happening in the video.

00:38:26.460 | So YouTube will release a dataset that,

00:38:28.780 | in the machine learning, computer vision world,

00:38:30.860 | this is just an exciting space.

00:38:33.120 | How much is that currently,

00:38:35.580 | how much are you playing with that currently?

00:38:37.200 | How much is your hope for the future

00:38:38.760 | of being able to analyze the content of the video itself?

00:38:42.320 | - Well, we have been working on that also

00:38:44.440 | since I came to YouTube.

00:38:46.160 | - Analyzing the content-- - Analyzing the content

00:38:48.160 | of the video, right? - Wow, awesome.

00:38:50.240 | - And what I can tell you is that

00:38:54.340 | our ability to do it well is still somewhat crude.

00:38:57.820 | We can tell if it's a music video,

00:39:02.340 | we can tell if it's a sports video,

00:39:04.620 | we can probably tell you that people are playing soccer.

00:39:07.420 | We probably can't tell whether it's Manchester United

00:39:13.200 | or my daughter's soccer team.

00:39:15.300 | So these things are kind of difficult

00:39:17.680 | and using them, we can use them in some ways.

00:39:21.160 | So for instance, we use that kind of information

00:39:24.360 | to understand and inform these clusters that I talked about.

00:39:28.280 | And also maybe to add some words like soccer,

00:39:32.880 | for instance, to the video,

00:39:34.240 | if it doesn't occur in the title or the description,

00:39:36.960 | which is remarkable that often it doesn't.

00:39:39.060 | One of the things that I ask creators to do

00:39:43.760 | is please help us out with the title and the description.

00:39:47.960 | For instance, we were a few years ago

00:39:52.320 | having a live stream of some competition

00:39:55.740 | for World of Warcraft on YouTube.

00:39:57.740 | And it was a very important competition,

00:40:02.360 | but if you typed World of Warcraft in search,

00:40:04.180 | you wouldn't find it.

00:40:05.360 | - World of Warcraft wasn't in the title?

00:40:07.440 | - World of Warcraft wasn't in the title.

00:40:09.120 | It was match 478, A team versus B team,

00:40:13.220 | and World of Warcraft wasn't in the title.

00:40:15.280 | I'm just like, come on, give me--

00:40:16.680 | - Being literal on the internet is actually very uncool,

00:40:21.200 | which is the problem.

00:40:22.400 | - Oh, is that right?

00:40:23.880 | - Well, I mean, in some sense,

00:40:26.400 | well, some of the greatest videos,

00:40:27.560 | I mean, there's a humor to just being indirect,

00:40:30.040 | being witty and so on, and actually being,

00:40:34.360 | machine learning algorithms want you to be literal, right?

00:40:38.160 | You just wanna say what's in the thing, be very, very simple.

00:40:42.660 | And in some sense, that gets away from wit and humor.

00:40:46.040 | So you have to play with both, right?

00:40:48.280 | So, but you're saying that for now,

00:40:50.360 | sort of the content of the title,

00:40:53.040 | the content of the description, the actual text

00:40:55.800 | is one of the best ways for the algorithm to find your video

00:41:00.800 | and put them in the right cluster.

00:41:02.980 | - That's right.

00:41:03.820 | And I would go further and say that if you want people,

00:41:08.020 | human beings to select your video in search,

00:41:11.440 | then it helps to have, let's say,

00:41:13.240 | World of Warcraft in the title,

00:41:14.640 | because why would a person,

00:41:17.760 | if they're looking at a bunch,

00:41:18.720 | they type World of Warcraft,

00:41:20.000 | and they have a bunch of videos,

00:41:21.080 | all of whom say World of Warcraft,

00:41:22.920 | except the one that you uploaded.

00:41:24.980 | Well, even the person is gonna think,

00:41:26.560 | well, maybe this isn't, somehow search made a mistake.

00:41:29.200 | This isn't really about World of Warcraft.

00:41:31.400 | So it's important, not just for the machine learning systems

00:41:34.560 | but also for the people who might be looking

00:41:37.120 | for this sort of thing.

00:41:37.960 | They get a clue that it's what they're looking for

00:41:42.000 | by seeing that same thing prominently

00:41:44.760 | in the title of the video.

00:41:46.200 | - Okay, let me push back on that.

00:41:47.560 | So I think from the algorithm perspective, yes,

00:41:49.680 | but if they typed in World of Warcraft

00:41:52.280 | and saw a video with the title simply winning,

00:41:57.280 | and the thumbnail has like a sad orc or something,

00:42:02.600 | I don't know.

00:42:03.800 | I think that's much, it gets your curiosity up.

00:42:11.620 | And then if they could trust that the algorithm

00:42:14.020 | was smart enough to figure out somehow

00:42:15.820 | that this is indeed a World of Warcraft video,

00:42:18.200 | that would have created the most beautiful experience.

00:42:20.720 | I think in terms of just the wit and the humor

00:42:23.280 | and the curiosity that we human beings naturally have.

00:42:26.080 | But you're saying, I mean, realistically speaking,

00:42:28.600 | it's really hard for the algorithm to figure out

00:42:31.100 | that the content of that video

00:42:32.640 | will be a World of Warcraft video.

00:42:34.600 | - And you have to accept

00:42:35.440 | that some people are gonna skip it.

00:42:37.320 | - Yeah.

00:42:38.160 | - Right, I mean, and so you're right.

00:42:40.880 | The people who don't skip it and select it

00:42:43.460 | are gonna be delighted.

00:42:45.200 | - Yeah.

00:42:46.500 | - But other people might say,

00:42:48.000 | yeah, this is not what I was looking for.

00:42:49.820 | - And making stuff discoverable,

00:42:52.020 | I think is what you're really working on and hoping.

00:42:56.480 | So yeah, so from your perspective,

00:42:58.600 | put stuff in the title and description.

00:43:00.300 | - And remember, the collaborative filtering

00:43:02.780 | part of the system starts by the same user

00:43:07.100 | watching videos together, right?

00:43:09.660 | So the way that they're probably gonna do that

00:43:12.560 | is by searching for them.

00:43:13.980 | - That's a fascinating aspect of it.

00:43:15.380 | It's like ant colonies, that's how they find stuff.

00:43:17.940 | So, I mean, what degree for collaborative filtering

00:43:23.100 | in general is one curious ant, one curious user essential?

00:43:28.100 | So just the person who is more willing

00:43:31.080 | to click on random videos

00:43:32.780 | and sort of explore these cluster spaces.

00:43:35.220 | In your sense, how many people are just like watching

00:43:38.520 | the same thing over and over and over and over?

00:43:40.260 | And how many are just like the explorers?

00:43:42.740 | They just kind of like click on stuff

00:43:44.340 | and then help the other ant in the ant's colony

00:43:48.240 | discover the cool stuff.

00:43:50.100 | Do you have a sense of that at all?

00:43:51.100 | - I really don't think I have a sense

00:43:52.780 | for the relative sizes of those groups.

00:43:55.540 | But I would say that people come to YouTube

00:43:58.700 | with some certain amount of intent.

00:44:00.780 | And as long as they, to the extent to which

00:44:04.900 | they try to satisfy that intent,

00:44:07.460 | that certainly helps our systems, right?

00:44:09.320 | Because our systems rely on kind of a faithful amount

00:44:13.940 | of behavior, right?

00:44:15.500 | Like, and there are people who try to trick us, right?

00:44:18.920 | There are people and machines

00:44:20.660 | that try to associate videos together

00:44:24.360 | that really don't belong together,

00:44:26.200 | but they're trying to get that association made

00:44:29.000 | because it's profitable for them.

00:44:31.160 | And so we have to always be resilient

00:44:34.040 | to that sort of attempt at gaming the systems.

00:44:37.620 | - So speaking to that, there's a lot of people that,

00:44:40.300 | in a positive way, perhaps, I don't know,

00:44:42.260 | I don't like it, but like to want to try to game the system,

00:44:46.500 | to get more attention.

00:44:47.340 | Everybody, creators, in a positive sense,

00:44:49.920 | want to get attention, right?

00:44:51.440 | So how do you work in this space

00:44:54.760 | when people create more and more

00:44:57.420 | sort of click-baity titles and thumbnails,

00:45:03.100 | sort of very tasking, Derek has made a video

00:45:05.580 | where basically describes that it seems what works

00:45:08.860 | is to create a high quality video, really good video,

00:45:12.100 | where people would want to watch it once they click on it,

00:45:14.660 | but have click-baity titles and thumbnails

00:45:17.540 | to get them to click on it in the first place.

00:45:19.860 | And he's saying, I'm embracing this fact,

00:45:21.660 | I'm just going to keep doing it,

00:45:23.340 | and I hope you forgive me for doing it.

00:45:26.420 | And you will enjoy my videos once you click on them.

00:45:28.940 | - So in what sense do you see this kind of click-bait style

00:45:33.940 | attempt to manipulate, to get people in the door,

00:45:38.620 | to manipulate the algorithm,

00:45:39.940 | or play with the algorithm, or game the algorithm?

00:45:43.420 | - I think that you can look at it

00:45:45.160 | as an attempt to game the algorithm,

00:45:47.300 | but even if you were to take the algorithm out of it

00:45:51.580 | and just say, okay, well, all these videos

00:45:53.220 | happen to be lined up,

00:45:55.020 | which the algorithm didn't make any decision

00:45:57.180 | about which one to put at the top or the bottom,

00:45:59.540 | but they're all lined up there,

00:46:00.740 | which one are the people going to choose?

00:46:03.580 | And I'll tell you the same thing that I told Derek is,

00:46:06.480 | you know, I have a bookshelf,

00:46:09.100 | and they have two kinds of books on them, science books.

00:46:12.480 | I have my math books from when I was a student,

00:46:15.900 | and they all look identical,

00:46:18.740 | except for the titles on the covers.

00:46:21.100 | They're all yellow, they're all from Springer,

00:46:23.420 | and they're every single one of them,

00:46:24.900 | the cover is totally the same, right?

00:46:29.300 | On the other hand, I have other more pop science type books,

00:46:33.580 | and they all have very interesting covers, right?

00:46:35.860 | And they have provocative titles and things like that.

00:46:40.100 | I mean, I wouldn't say that they're clickbaity,

00:46:42.060 | because they are indeed good books.

00:46:44.200 | And I don't think that they cross any line,

00:46:48.180 | but you know, that's just a decision you have to make,

00:46:52.340 | right, like the people who write

00:46:54.900 | "Classical Recursion Theory" by Pierrotti-Freddie,

00:46:57.820 | he was fine with the yellow title and nothing more.

00:47:02.180 | Whereas I think other people

00:47:03.380 | who wrote a more popular type book,

00:47:08.380 | understand that they need to have a compelling cover

00:47:11.940 | and a compelling title.

00:47:13.500 | And you know, I don't think

00:47:16.540 | there's anything really wrong with that.

00:47:18.300 | We do take steps to make sure that

00:47:22.340 | there is a line that you don't cross.

00:47:24.500 | And if you go too far,

00:47:26.340 | maybe your thumbnail is especially racy,

00:47:28.500 | or, you know, it's all caps

00:47:31.820 | with too many exclamation points.

00:47:33.580 | We observe that users are kind of,

00:47:38.540 | you know, sometimes offended by that.

00:47:41.820 | And so for the users who are offended by that,

00:47:46.460 | we will then depress or suppress those videos.

00:47:51.020 | - And which reminds me,

00:47:52.060 | there's also another signal where users can say,

00:47:55.180 | I don't know if it was recently added,

00:47:56.460 | but I really enjoy it.

00:47:57.900 | Just saying, I don't, I didn't,

00:47:59.740 | something like, I don't want to see this video anymore,

00:48:02.340 | or something like,

00:48:03.380 | like this is, like there's certain videos

00:48:07.140 | that just cut me the wrong way.

00:48:09.140 | Like just jump out at me.

00:48:10.580 | It's like, I don't want to, I don't want this.

00:48:12.020 | And it feels really good to clean that up.

00:48:14.740 | To be like, I don't, that's not, that's not for me.

00:48:18.140 | I don't know.

00:48:18.980 | I think that might've been recently added,

00:48:20.220 | but that's also a really strong signal.

00:48:22.420 | - Yes, absolutely.

00:48:23.780 | Right, we don't want to make a recommendation

00:48:26.100 | that people are unhappy with.

00:48:29.300 | - And that makes me,

00:48:30.140 | that particular one makes me feel good as a user in general,

00:48:33.540 | and as a machine learning person,

00:48:35.300 | 'cause I feel like I'm helping the algorithm.

00:48:37.660 | My interactions on YouTube don't always feel like

00:48:39.940 | I'm helping the algorithm.

00:48:40.980 | Like I'm not reminded of that fact.

00:48:42.940 | Like for example, Tesla and Autopilot,

00:48:46.860 | Elon Musk create a feeling for their customers,

00:48:50.620 | for people that own Teslas,

00:48:51.660 | that they're helping the algorithm of Tesla vehicle.

00:48:53.980 | Like they're all like a really proud,

00:48:55.660 | they're helping the fleet learn.

00:48:57.220 | I think YouTube doesn't always remind people

00:48:59.540 | that you're helping the algorithm get smarter.

00:49:02.420 | And for me, I love that idea.

00:49:04.460 | Like we're all collaboratively,

00:49:06.420 | like Wikipedia gives that sense.

00:49:07.900 | They're all together creating a beautiful thing.

00:49:11.620 | YouTube doesn't always remind me of that.

00:49:15.500 | This conversation is reminding me of that, but.

00:49:18.460 | - Well, that's a good tip.

00:49:19.300 | We should keep that fact in mind

00:49:20.900 | when we design these features.

00:49:22.260 | I'm not sure I really thought about it that way,

00:49:24.980 | but that's a very interesting perspective.

00:49:27.820 | - It's an interesting question of personalization

00:49:30.820 | that I feel like when I click like on a video,

00:49:35.140 | I'm just improving my experience.

00:49:37.740 | It would be great.

00:49:40.900 | It would make me personally, people are different,

00:49:42.860 | but make me feel great

00:49:43.940 | if I was helping also the YouTube algorithm broadly

00:49:46.860 | say something.

00:49:47.700 | You know what I'm saying?

00:49:48.540 | Like there's a, I don't know if that's human nature,

00:49:50.900 | but you want the products you love,

00:49:54.300 | and I certainly love YouTube.

00:49:55.900 | You want to help it get smarter and smarter and smarter

00:49:59.100 | 'cause there's some kind of coupling

00:50:00.740 | between our lives together being better.

00:50:04.700 | If YouTube was better than I will,

00:50:06.140 | my life will be better.

00:50:07.100 | And there's that kind of reasoning.

00:50:08.580 | I'm not sure what that is.

00:50:09.420 | And I'm not sure how many people share that feeling.

00:50:12.180 | It could be just a machine learning feeling.

00:50:14.060 | But on that point, how much personalization is there

00:50:18.740 | in terms of next video recommendations?

00:50:22.580 | So is it kind of all really boiling down to clustering?

00:50:27.580 | Like if I'm in your clusters to me and so on

00:50:30.780 | and that kind of thing,

00:50:32.420 | or how much is personalized to me,

00:50:34.580 | the individual completely?

00:50:35.900 | - It's very, very personalized.

00:50:38.740 | So your experience will be quite a bit different

00:50:43.020 | from anybody else's who's watching that same video,

00:50:46.340 | at least when they're logged in.

00:50:48.180 | And the reason is is that we found that users

00:50:53.180 | often want two different kinds of things

00:50:56.140 | when they're watching a video.

00:50:57.700 | Sometimes they want to keep watching more on that topic

00:51:02.540 | or more in that genre.

00:51:04.820 | And other times they just are done

00:51:07.100 | and they're ready to move on to something else.

00:51:08.980 | And so the question is, well, what is the something else?

00:51:12.780 | And one of the first things one can imagine is,

00:51:16.300 | well, maybe something else is the latest video

00:51:19.380 | from some channel to which you've subscribed.

00:51:22.020 | And that's gonna be very different for you

00:51:24.940 | than it is for me, right?

00:51:26.700 | And even if it's not something that you subscribe to,

00:51:29.820 | it's something that you watch a lot.

00:51:31.060 | And again, that'll be very different

00:51:32.820 | on a person by person basis.

00:51:34.780 | And so even the watch next,

00:51:39.780 | as well as the homepage of course, is quite personalized.

00:51:42.660 | - So what, we mentioned some of the signals,

00:51:46.140 | but what does success look like?

00:51:47.420 | What does success look like in terms of the algorithm

00:51:49.900 | creating a great long-term experience for a user?

00:51:53.380 | Or put another way, if you look at the videos

00:51:57.540 | I've watched this month,

00:51:59.820 | how do you know the algorithm succeeded for me?

00:52:03.740 | - I think first of all, if you come back

00:52:06.140 | and watch more YouTube, then that's one indication

00:52:09.140 | that you found some value from it.

00:52:11.020 | - So just the number of hours is a powerful indicator.

00:52:13.820 | - Well, I mean, not the hours themselves,

00:52:15.700 | but the fact that you return on another day.

00:52:20.700 | So that's probably the most simple indicator.

00:52:26.140 | People don't come back to things

00:52:27.540 | that they don't find value in, right?

00:52:29.220 | There's a lot of other things that they could do.

00:52:32.220 | But like I said, I mean, ideally we would like everybody

00:52:35.540 | to feel that YouTube enriches their lives

00:52:38.460 | and that every video they watched

00:52:40.300 | is the best one they've ever watched

00:52:42.500 | since they've started watching YouTube.

00:52:44.660 | And so that's why we survey them and ask them,

00:52:49.100 | like, is this one to five stars?

00:52:52.940 | And so our version of success is

00:52:55.220 | every time someone takes that survey,

00:52:57.820 | they say it's five stars.

00:52:59.700 | And if we ask them, is this the best video

00:53:02.460 | you've ever seen on YouTube?

00:53:03.580 | They say yes, every single time.

00:53:05.820 | So it's hard to imagine that we would actually achieve that.

00:53:09.580 | Maybe asymptotically we would get there,

00:53:11.740 | but that would be what we think success is.

00:53:16.340 | - It's funny, I've recently said somewhere,

00:53:19.660 | I don't know, maybe tweeted,

00:53:21.140 | but that Ray Dalio has this video on the economic machine.

00:53:26.140 | I forget what it's called, but it's a 30 minute video.

00:53:29.100 | And I said, it's the greatest video

00:53:30.860 | I've ever watched on YouTube.

00:53:32.420 | It's like, I watched the whole thing

00:53:34.820 | and my mind was blown.

00:53:35.940 | It's a very crisp, clean description

00:53:38.620 | of how at least the American economic system works.

00:53:41.380 | It's a beautiful video.

00:53:42.900 | And I was just, I wanted to click on something

00:53:45.420 | to say this is the best thing.

00:53:47.460 | This is the best thing ever, please let me,

00:53:49.300 | I can't believe I discovered it.

00:53:51.100 | I mean, the views and the likes reflect its quality,

00:53:55.520 | but I was almost upset that I haven't found it earlier

00:53:58.340 | and wanted to find other things like it.

00:54:01.020 | I don't think I've ever felt

00:54:02.300 | that this is the best video I've ever watched.

00:54:04.860 | And that was that.

00:54:05.940 | And to me, the ultimate utopia,

00:54:08.620 | the best experience is where every single video,

00:54:11.420 | where I don't see any of the videos I regret

00:54:13.380 | and every single video I watch is one

00:54:15.420 | that actually helps me grow, helps me enjoy life,

00:54:19.500 | be happy and so on.

00:54:20.760 | Well, so that's a heck of a,

00:54:27.980 | that's one of the most beautiful and ambitious,

00:54:30.900 | I think, machine learning tasks.

00:54:32.700 | So when you look at a society as opposed

00:54:34.580 | to an individual user,

00:54:36.480 | do you think of how YouTube is changing society

00:54:39.020 | when you have these millions of people watching videos,

00:54:42.580 | growing, learning, changing, having debates?

00:54:46.300 | Do you have a sense of, yeah,

00:54:49.060 | what the big impact on society is?

00:54:51.400 | 'Cause I think it's huge,

00:54:52.560 | but do you have a sense of what direction

00:54:54.660 | we're taking in this world?

00:54:56.340 | - Well, I mean, I think openness has had an impact

00:55:00.540 | on society already.

00:55:02.420 | There's a lot of-

00:55:03.480 | - What do you mean by openness?

00:55:05.900 | - Well, the fact that unlike other mediums,

00:55:10.380 | there's not someone sitting at YouTube

00:55:14.100 | who decides before you can upload your video,

00:55:17.020 | whether it's worth having you upload it

00:55:19.620 | or worth anybody seeing it really, right?

00:55:22.860 | And so, there are some creators who say,

00:55:27.620 | like, I wouldn't have this opportunity

00:55:30.820 | to reach an audience.

00:55:33.680 | Tyler Oakley often said that,

00:55:36.760 | he wouldn't have had this opportunity to reach this audience

00:55:39.500 | if it weren't for YouTube.

00:55:41.120 | And so I think that's one way

00:55:45.620 | in which YouTube has changed society.

00:55:50.180 | I know that there are people that I work with

00:55:52.360 | from outside the United States,

00:55:54.980 | especially from places where literacy is low.

00:55:59.980 | And they think that YouTube can help in those places

00:56:03.900 | because you don't need to be able to read and write

00:56:06.860 | in order to learn something important for your life,

00:56:09.840 | maybe how to do some job or how to fix something.

00:56:14.840 | And so that's another way in which I think YouTube

00:56:18.420 | is possibly changing society.

00:56:21.460 | So I've worked at YouTube for eight, almost nine years now.

00:56:25.580 | And it's fun because I meet people

00:56:29.440 | and you tell them where you work,

00:56:32.180 | you say you work on YouTube

00:56:33.540 | and they immediately say, I love YouTube.

00:56:36.460 | Right, which is great, makes me feel great.

00:56:39.220 | But then of course, when I ask them,

00:56:40.820 | well, what is it that you love about YouTube?

00:56:43.540 | Not one time ever has anybody said

00:56:46.540 | that the search works outstanding

00:56:48.820 | or that the recommendations are great.

00:56:51.120 | What they always say when I ask them,

00:56:55.840 | what do you love about YouTube?

00:56:57.140 | Is they immediately start talking about some channel

00:56:59.660 | or some creator or some topic or some community

00:57:03.380 | that they found on YouTube and that they just love.

00:57:06.500 | - Yeah.

00:57:07.420 | - And so that has made me realize

00:57:10.740 | that YouTube is really about the video

00:57:14.900 | and connecting the people with the videos

00:57:19.100 | and then everything else kind of gets out of the way.

00:57:22.580 | - So beyond the video, it's an interesting,

00:57:24.940 | 'cause you kind of mentioned creator.

00:57:27.520 | What about the connection with just the individual creators

00:57:32.620 | as opposed to just individual video?

00:57:35.220 | So like I gave the example of Ray Dalio video

00:57:37.980 | that the video itself is incredible,

00:57:42.500 | but there's some people who are just creators

00:57:44.980 | that I love that they're,

00:57:47.420 | one of the cool things about people

00:57:49.460 | who call themselves YouTubers or whatever

00:57:51.620 | is they have a journey.

00:57:53.020 | They usually, almost all of them are,

00:57:55.340 | they suck horribly in the beginning

00:57:57.220 | and then they kind of grow, you know?

00:57:58.860 | And then there's that genuineness in their growth.

00:58:01.780 | So, you know, YouTube clearly wants to help creators

00:58:05.620 | connect with their audience in this kind of way.

00:58:08.100 | So how do you think about that process

00:58:09.740 | of helping creators grow,

00:58:11.460 | helping them connect with their audience,

00:58:13.340 | develop not just individual videos,

00:58:15.220 | but the entirety of a creator's life on YouTube?

00:58:18.820 | - Well, I mean, we're trying to help creators

00:58:21.500 | find the biggest audience that they can find.

00:58:24.580 | And the reason why that's,

00:58:26.140 | you brought up creator versus video.

00:58:29.060 | The reason why creator channel is so important

00:58:32.380 | is because if we have a hope

00:58:36.620 | of people coming back to YouTube,

00:58:39.660 | well, they have to have in their minds

00:58:43.540 | some sense of what they're gonna find

00:58:45.660 | when they come back to YouTube.

00:58:47.940 | If YouTube were just the next viral video,

00:58:52.660 | and I have no concept of what the next viral video could be,

00:58:55.540 | one time it's a cat playing a piano,

00:58:57.620 | and the next day it's some children interrupting a reporter,

00:59:01.260 | and the next day it's, you know, some other thing happening,

00:59:05.460 | then it's hard for me to,

00:59:08.020 | to when I'm not watching YouTube say,

00:59:10.500 | gosh, I really, you know, would like to see something

00:59:14.860 | from someone or about something, right?

00:59:17.940 | And so that's why I think this connection

00:59:19.700 | between fans and creators is so important for both,

00:59:24.700 | because it's a way of sort of fostering a relationship

00:59:29.860 | that can play out into the future.

00:59:33.740 | - Let me talk about kind of a dark

00:59:36.340 | and interesting question in general,

00:59:38.220 | and again, a topic that you or nobody has an answer to,

00:59:42.260 | but social media has a sense of,

00:59:45.500 | you know, it gives us highs and it gives us lows

00:59:50.100 | in the sense that sort of creators often speak

00:59:53.220 | about having sort of burnout

00:59:55.660 | and having psychological ups and downs

00:59:58.900 | and challenges mentally

01:00:00.060 | in terms of continuing the creation process.

01:00:02.700 | There's a momentum, there's a huge, excited audience

01:00:05.340 | that makes creators feel great.

01:00:08.860 | And I think it's more than just financial.

01:00:11.800 | I think it's literally just,

01:00:14.240 | they love that sense of community.

01:00:16.100 | It's part of the reason I upload to YouTube.

01:00:18.340 | I don't care about money, never will.

01:00:20.460 | What I care about is the community.

01:00:22.800 | But some people feel like this momentum,

01:00:25.320 | and even when there's times in their life

01:00:27.500 | when they don't feel, you know,

01:00:30.180 | for some reason don't feel like creating.

01:00:31.980 | So how do you think about burnout,

01:00:35.100 | this mental exhaustion

01:00:36.300 | that some YouTube creators go through?

01:00:38.460 | Is that something we have an answer for?

01:00:40.500 | Is that something, how do we even think about that?

01:00:42.700 | - Well, the first thing is we wanna make sure

01:00:44.520 | that the YouTube systems

01:00:46.540 | are not contributing to this sense, right?

01:00:49.080 | And so we've done a fair amount of research

01:00:52.920 | to demonstrate that you can absolutely take a break.

01:00:57.860 | If you are a creator and you've been uploading a lot,

01:01:01.060 | we have just as many examples of people who took a break

01:01:04.300 | and came back more popular than they were before

01:01:07.900 | as we have examples of going the other way.

01:01:09.940 | - Yeah, can we pause on that for a second?

01:01:11.260 | So the feeling that people have, I think,

01:01:13.740 | is if I take a break, everybody,

01:01:17.060 | the party will leave, right?

01:01:19.100 | So if you could just linger on that.

01:01:21.700 | So in your sense that taking a break is okay.

01:01:24.460 | - Yes, taking a break is absolutely okay.

01:01:27.300 | And the reason I say that is because

01:01:30.940 | we can observe many examples of being,

01:01:34.700 | of creators coming back very strong and even stronger

01:01:39.300 | after they have taken some sort of break.

01:01:41.220 | And so I just wanna dispel the myth

01:01:44.420 | that this somehow necessarily means

01:01:49.420 | that your channel is gonna go down or lose views.

01:01:53.260 | That is not the case.

01:01:55.240 | We know for sure that this is not a necessary outcome.

01:01:59.220 | And so we wanna encourage people

01:02:01.940 | to make sure that they take care of themselves.

01:02:03.840 | That is job one, right?

01:02:05.540 | You have to look after yourself and your mental health.

01:02:08.340 | And I think that it probably, in some of these cases,

01:02:14.980 | contributes to better videos once they come back, right?

01:02:19.980 | Because a lot of people, I mean, I know myself,

01:02:22.580 | if I'm burnt out on something,

01:02:23.860 | then I'm probably not doing my best work

01:02:26.020 | even though I can keep working until I pass out.

01:02:30.020 | And so I think that the taking a break

01:02:34.300 | may even improve the creative ideas that someone has.

01:02:38.540 | - Okay, I think it's a really important thing

01:02:41.100 | to sort of to dispel.

01:02:42.820 | I think that applies to all of social media.

01:02:45.600 | Like literally I've taken a break for a day

01:02:47.660 | every once in a while.

01:02:48.940 | (laughs)

01:02:51.380 | Sorry, sorry if that sounds like a short time.

01:02:54.760 | But even like email, just taking a break from email

01:02:58.260 | or only checking email once a day,

01:03:00.660 | especially when you're going through something psychologically

01:03:03.220 | in your personal life or so on,

01:03:04.940 | or really not sleeping much 'cause of work deadlines,

01:03:08.100 | it can refresh you in a way that's profound.

01:03:10.860 | And so the same applies-

01:03:11.700 | - And it was there when you came back, right?

01:03:12.980 | - It's there.

01:03:14.280 | And it looks different actually when you come back.

01:03:17.820 | You're sort of brighter eyed with some coffee, everything.

01:03:20.660 | The world looks better.

01:03:22.200 | So it's important to take a break when you need it.

01:03:25.100 | So you've mentioned kind of the YouTube algorithm

01:03:29.780 | isn't E equals MC squared.

01:03:32.580 | It's not a single equation.

01:03:34.180 | It's potentially sort of more than a million lines of code.

01:03:38.120 | Sort of, is it more akin to what autonomous,

01:03:44.540 | successful autonomous vehicles today are,

01:03:46.340 | which is they're just basically patches

01:03:49.180 | on top of patches of heuristics and human experts

01:03:53.000 | really tuning the algorithm

01:03:55.700 | and have some machine learning modules?

01:03:58.440 | Or is it becoming more and more

01:04:00.380 | a giant machine learning system

01:04:03.220 | with humans just doing a little bit

01:04:05.000 | of tweaking here and there?

01:04:06.140 | What's your sense?

01:04:07.360 | First of all, do you even have a sense

01:04:08.880 | of what is the YouTube algorithm at this point?

01:04:11.220 | And however much you do have a sense,

01:04:14.080 | what does it look like?

01:04:15.760 | - Well, we don't usually think about it as the algorithm

01:04:19.220 | because it's a bunch of systems

01:04:21.620 | that work on different services.

01:04:24.260 | The other thing that I think people don't understand

01:04:26.900 | is that what you might refer to as the YouTube algorithm

01:04:31.900 | from outside of YouTube is actually a bunch of code

01:04:36.900 | and machine learning systems and heuristics,

01:04:39.860 | but that's married with the behavior

01:04:42.580 | of all the people who come to YouTube every day.

01:04:44.740 | - So the people part of the code, essentially.

01:04:46.560 | - Exactly, right?

01:04:47.440 | Like if there were no people who came to YouTube tomorrow,

01:04:49.780 | then the algorithm wouldn't work anymore, right?

01:04:52.500 | So that's a critical part of the algorithm.

01:04:55.460 | And so when people talk about,

01:04:56.760 | well, the algorithm does this, the algorithm does that,

01:04:59.440 | it's sometimes hard to understand,

01:05:01.080 | well, it could be the viewers are doing that

01:05:04.560 | and the algorithm is mostly just keeping track

01:05:07.480 | of what the viewers do and then reacting to those things

01:05:13.080 | in sort of more fine-grained situations.

01:05:15.620 | And I think that this is the way

01:05:18.420 | that the recommendation system and the search system

01:05:21.380 | and probably many machine learning systems evolve

01:05:24.540 | is you start trying to solve a problem

01:05:28.340 | and the first way to solve a problem

01:05:29.900 | is often with a simple heuristic, right?

01:05:33.620 | And you wanna say,

01:05:35.460 | what are the videos we're gonna recommend?

01:05:36.780 | Well, how about the most popular ones, right?

01:05:39.580 | And that's where you start.

01:05:42.480 | And over time, you collect some data

01:05:46.620 | and you refine your situation

01:05:48.180 | so that you're making less heuristics

01:05:49.980 | and you're building a system

01:05:52.180 | that can actually learn what to do in different situations

01:05:55.700 | based on some observations of those situations in the past.

01:05:59.620 | And you keep chipping away at these heuristics over time.

01:06:03.600 | And so I think that just like with diversity,

01:06:08.100 | I think the first diversity measure we took was,

01:06:11.180 | okay, not more than three videos in a row

01:06:13.900 | from the same channel, right?

01:06:15.420 | It's a pretty simple heuristic to encourage diversity,

01:06:19.140 | but it worked, right?

01:06:20.700 | Who needs to see four, five, six videos in a row

01:06:23.220 | from the same channel?

01:06:24.380 | And over time, we try to chip away at that

01:06:28.380 | and make it more fine-grained

01:06:30.180 | and basically have it remove the heuristics

01:06:34.500 | in favor of something that can react

01:06:37.780 | to individuals and individual situations.

01:06:41.260 | - So how do you, you mentioned,

01:06:42.940 | you know, we know that something worked.

01:06:46.560 | How do you get a sense when decisions

01:06:48.860 | of the kind of A/B testing

01:06:50.380 | that this idea was a good one, this was not so good?

01:06:53.820 | How do you measure that and across which timescale,

01:06:58.780 | across how many users, that kind of thing?

01:07:02.220 | - Well, you mentioned that A/B experiments.

01:07:04.500 | And so just about every single change we make to YouTube,

01:07:08.840 | we do it only after we've run a A/B experiment.

01:07:13.700 | And so in those experiments,

01:07:16.500 | which run from one week to months,

01:07:20.360 | we measure hundreds, literally hundreds

01:07:24.980 | of different variables and measure changes

01:07:28.940 | with confidence intervals in all of them.

01:07:30.960 | Because we really are trying to get a sense

01:07:33.500 | for ultimately does this improve the experience for viewers?

01:07:38.220 | That's the question we're trying to answer.

01:07:40.140 | And an experiment is one way

01:07:41.980 | because we can see certain things go up and down.

01:07:45.020 | So for instance, if we noticed in the experiment,

01:07:48.780 | people are dismissing videos less frequently

01:07:52.460 | or they're saying that they're more satisfied.

01:07:56.860 | They're giving more videos five stars after they watch them.

01:07:59.380 | Then those would be indications

01:08:02.820 | that the experiment is successful,

01:08:04.380 | that it's improving the situation for viewers.

01:08:06.680 | But we can also look at other things.

01:08:09.500 | Like we might do user studies

01:08:12.420 | where we invite some people in and ask them,

01:08:14.720 | like, what do you think about this?

01:08:16.000 | What do you think about that?

01:08:16.980 | How do you feel about this?

01:08:18.340 | And other various kinds of user research.

01:08:21.800 | But ultimately, before we launch something,

01:08:24.300 | we're gonna wanna run an experiment.

01:08:26.060 | So we get a sense for what the impact is gonna be,

01:08:29.240 | not just to the viewers,

01:08:30.680 | but also to the different channels and all of that.

01:08:34.800 | - An absurd question.

01:08:37.880 | Nobody knows.

01:08:38.720 | Well, actually it's interesting.

01:08:39.760 | Maybe there's an answer,

01:08:40.640 | but if I want to make a viral video, how do I do it?

01:08:44.700 | - I don't know how you make a viral video.

01:08:47.920 | I know that we have in the past tried to figure out

01:08:52.480 | if we could detect when a video was going to go viral.

01:08:57.480 | And those were, you take the first and second derivatives

01:09:01.040 | of the view count and maybe use that to do some prediction.

01:09:06.040 | But I can't say we ever got very good at that.

01:09:10.680 | Oftentimes we look at where the traffic was coming from.

01:09:13.760 | If a lot of the viewership is coming from

01:09:17.160 | something like Twitter,

01:09:19.040 | then maybe it has a higher chance of becoming viral

01:09:22.680 | than if it were coming from search or something.

01:09:26.920 | But that was just trying to detect

01:09:28.720 | a video that might be viral.

01:09:30.120 | How to make one, I have no idea.

01:09:33.320 | You get your kids to interrupt you

01:09:35.000 | while you're on the news or something.

01:09:37.640 | - Absolutely.

01:09:38.840 | But after the fact, on one individual video,

01:09:42.280 | sort of ahead of time predicting is a really hard task.

01:09:44.920 | But after the video went viral in analysis,

01:09:49.920 | can you sometimes understand why it went viral

01:09:53.560 | from the perspective of YouTube broadly?

01:09:56.440 | First of all, is it even interesting for YouTube

01:09:58.120 | that a particular video is viral?

01:10:00.680 | Or does that not matter for the individual,

01:10:03.800 | for the experience of people?

01:10:05.320 | - Well, I think people expect that if a video is going viral

01:10:09.880 | and it's something they would be interested in,

01:10:12.080 | then I think they would expect YouTube

01:10:15.120 | to recommend it to them.

01:10:16.240 | - Right.

01:10:17.080 | So if something's going viral,

01:10:18.240 | it's good to just let the wave,

01:10:20.000 | let people ride the wave of its violence.

01:10:23.840 | - Well, I mean, we want to meet people's expectations

01:10:25.880 | in that way, of course.

01:10:27.680 | - So like I mentioned,

01:10:29.200 | I hung out with Derek Mueller a while ago,

01:10:31.680 | a couple of months back.

01:10:32.880 | He's actually the person who suggested

01:10:36.600 | I talk to you on this podcast.

01:10:38.240 | - All right, well, thank you, Derek.

01:10:40.320 | - At that time, he just recently posted

01:10:43.200 | an awesome science video titled,

01:10:45.720 | "Why are 96 million black balls on this reservoir?"

01:10:50.280 | And in a matter of, I don't know how long,

01:10:52.360 | but like a few days, he got 38 million views

01:10:55.520 | and it's still growing.

01:10:56.680 | Is this something you can analyze

01:11:00.040 | and understand why it happened,

01:11:02.960 | this video and you want a particular video like it?

01:11:06.200 | - I mean, we can surely see where it was recommended,

01:11:10.040 | where it was found, who watched it,

01:11:13.000 | and those sorts of things.

01:11:14.480 | - So it's actually, sorry to interrupt.

01:11:16.400 | It is the video which helped me discover who Derek is.

01:11:20.600 | I didn't know who he is before.

01:11:22.080 | So I remember, you know,

01:11:23.880 | usually I just have all of these technical,

01:11:26.440 | boring MIT Stanford talks in my recommendation

01:11:29.680 | 'cause that's what I watch.

01:11:30.520 | And then all of a sudden,

01:11:31.600 | there's this black balls in reservoir video

01:11:34.720 | with like an excited nerd with like just,

01:11:38.080 | why is this being recommended to me?

01:11:40.880 | So I clicked on it and watched the whole thing

01:11:42.480 | and it was awesome.

01:11:43.440 | But, and then a lot of people had that experience,

01:11:45.680 | like why was I recommended this?

01:11:47.960 | But they all, of course, watched it and enjoyed it,

01:11:49.920 | which is, what's your sense of this

01:11:52.440 | just wave of recommendation

01:11:54.760 | that comes with this viral video

01:11:56.160 | that ultimately people get enjoy after they click on it?

01:12:00.240 | - Well, I think it's the system, you know,

01:12:02.320 | basically doing what anybody

01:12:03.920 | who's recommending something would do,

01:12:05.520 | which is you show it to some people and if they like it,

01:12:08.880 | you say, okay, well,

01:12:09.720 | can I find some more people who are a little bit like them?

01:12:12.160 | Okay, I'm going to try it with them.

01:12:14.040 | Oh, they like it too.

01:12:15.040 | Let me expand the circle some more, find some more people.

01:12:17.480 | Oh, it turns out they like it too.

01:12:19.320 | And you just keep going until you get some feedback

01:12:21.880 | that says, no, now you've gone too far.

01:12:23.640 | These people don't like it anymore.

01:12:25.480 | And so I think that's basically what happened.

01:12:28.240 | Now, you asked me about how to make a video go viral

01:12:32.760 | or make a viral video.

01:12:34.240 | I don't think that if you or I decided to make a video

01:12:39.320 | about 96 million balls, that it would also go viral.

01:12:42.600 | It's possible that Derek made like the canonical video

01:12:47.320 | about those black balls in the lake.

01:12:50.160 | - Exactly, he did actually.

01:12:51.920 | - Right, and so I don't know whether or not

01:12:55.200 | just following along is the secret.

01:12:57.880 | - Yeah, but it's fascinating.

01:13:00.320 | I mean, just like you said, the algorithm

01:13:02.320 | sort of expanding that circle

01:13:03.960 | and then figuring out that more and more people did enjoy it

01:13:06.200 | and that sort of phase shift

01:13:09.040 | of just a huge number of people enjoying it

01:13:11.280 | and the algorithm quickly, automatically,

01:13:14.200 | I assume figuring that out.

01:13:16.000 | That's a, I don't know, the dynamics of psychology,

01:13:18.760 | that is a beautiful thing.

01:13:19.960 | And so what do you think about the idea of clipping?

01:13:24.200 | Like too many people annoyed me into doing it,

01:13:28.280 | which is they were requesting it.

01:13:29.720 | They said it would be very beneficial to add clips

01:13:33.480 | in like the coolest points

01:13:36.160 | and actually have explicit videos.

01:13:37.760 | Like I'm re-uploading a video, like a short clip,

01:13:40.800 | which is what the podcasts are doing.

01:13:42.800 | Do you see, as opposed to,

01:13:45.800 | like I also add timestamps for the topics.

01:13:47.840 | You know, do you want the clip?

01:13:49.720 | Do you see YouTube somehow helping creators

01:13:52.600 | with that process or helping connect clips

01:13:54.880 | to the original videos?

01:13:56.640 | Or is that just on a long list of amazing features

01:13:59.600 | to work towards?

01:14:00.600 | - Yeah, I mean, it's not something that I think

01:14:02.880 | we've done yet, but I can tell you that

01:14:06.360 | I think clipping is great

01:14:09.360 | and I think it's actually great for you as a creator.

01:14:12.480 | And here's the reason.

01:14:13.720 | If you think about, I mean, let's say the NBA is uploading

01:14:19.760 | videos of its games.

01:14:22.120 | Well, people might search for Warriors versus Rockets

01:14:28.120 | or they might search for Steph Curry.

01:14:30.640 | And so a highlight from the game

01:14:32.680 | in which Steph Curry makes an amazing shot

01:14:35.400 | is an opportunity for someone to find

01:14:38.880 | a portion of that video.

01:14:40.680 | And so I think that you never know

01:14:43.800 | how people are gonna search for something

01:14:47.040 | that you've created.

01:14:48.080 | And so you wanna, I would say,

01:14:50.080 | you wanna make clips and add titles and things like that

01:14:54.480 | so that they can find it as easily as possible.

01:14:58.240 | - Do you have a dream of a future,

01:15:00.320 | perhaps a distant future when the YouTube algorithm

01:15:04.000 | figures that out, sort of automatically detects

01:15:08.280 | the parts of the video that are really interesting,

01:15:11.760 | exciting, potentially exciting for people

01:15:14.080 | and sort of clip them out in this incredibly rich space.

01:15:17.480 | 'Cause if you talk about,

01:15:18.640 | if you talk even just this conversation,

01:15:20.360 | we probably covered 30, 40 little topics.

01:15:24.560 | And there's a huge space of users that would find,

01:15:28.440 | you know, 30% of those topics are really interesting.

01:15:30.600 | And that space is very different.

01:15:33.240 | It's something that's beyond my ability to clip out, right?

01:15:37.520 | But the algorithm might be able to figure all that out,

01:15:40.880 | sort of expand into clips.

01:15:43.480 | Do you have a, do you think about this kind of thing?

01:15:46.080 | Do you have a hope, a dream that one day the algorithm

01:15:48.440 | will be able to do that kind of deep content analysis?

01:15:51.040 | - Well, we've actually had projects

01:15:52.840 | that attempt to achieve this,

01:15:54.720 | but it really does depend on understanding the video well.

01:16:00.200 | And our understanding of the video right now

01:16:02.160 | is quite crude.

01:16:03.600 | And so I think it would be especially hard

01:16:07.760 | to do it with a conversation like this.

01:16:11.160 | One might be able to do it with,

01:16:15.480 | let's say a soccer match more easily, right?

01:16:17.960 | You could probably find out where the goals were scored.

01:16:20.720 | And then of course you need to figure out

01:16:23.560 | who it was that scored the goal.

01:16:25.040 | And that might require a human to do some annotation.

01:16:28.080 | But I think that trying to identify coherent topics

01:16:32.880 | in a transcript, like the one of our conversation,

01:16:37.060 | is not something that we're going to be

01:16:41.120 | very good at right away.

01:16:42.440 | - And I was speaking more to the general problem,

01:16:44.880 | actually, of being able to do both a soccer match

01:16:47.280 | and our conversation without explicit,

01:16:49.800 | sort of almost, my hope was that there exists

01:16:53.520 | an algorithm that's able to find exciting things in video.

01:16:59.360 | - So Google now on Google search will help you find

01:17:04.840 | the segment of the video that you're interested in.

01:17:06.880 | So if you search for something like how to change

01:17:11.880 | the filter in my dishwasher,

01:17:13.360 | then if there's a long video about your dishwasher,

01:17:15.600 | and this is the part where the person shows you

01:17:17.760 | how to change the filter, then it will highlight that area.

01:17:21.800 | And provide a link directly to it.

01:17:24.080 | - And do you know, from your recollection,

01:17:27.000 | do you know if the thumbnail reflects,

01:17:29.200 | like what's the difference between showing the full video

01:17:31.200 | and the shorter clip?

01:17:32.640 | Do you know how it's presented in search results?

01:17:34.920 | - I don't remember how it's presented.

01:17:36.200 | And the other thing I would say is that right now

01:17:39.760 | it's based on creator annotations.

01:17:42.120 | - Ah, got it.

01:17:43.360 | So it's not the thing we're talking about.

01:17:45.760 | - But folks are working on the more automatic version.

01:17:49.880 | It's interesting, people might not imagine this,

01:17:52.280 | but a lot of our systems start by using

01:17:56.760 | almost entirely the audience behavior.

01:18:00.560 | And then as they get better,

01:18:02.380 | the refinement comes from using the content.

01:18:06.040 | - And I wish, I know there's privacy concerns,

01:18:10.360 | but I wish YouTube explored this space,

01:18:15.360 | which is sort of putting a camera on the users

01:18:17.800 | if they allowed it, right?

01:18:19.000 | To study their, like I did a lot of emotion recognition work

01:18:23.200 | and so on, to study actual sort of richer signal.

01:18:26.920 | One of the cool things when you upload 360,

01:18:29.320 | like VR video to YouTube, and I've done this a few times,

01:18:33.600 | so I've uploaded myself, it's a horrible idea.

01:18:36.360 | Some people enjoyed it, but whatever.

01:18:39.440 | The video of me giving a lecture in 360,

01:18:41.760 | we have a 360 camera, and it's cool

01:18:43.640 | because YouTube allows you to then watch

01:18:45.960 | where do people look at.

01:18:47.340 | There's a heat map of where the center

01:18:51.400 | of the VR experience was, and it's interesting

01:18:54.400 | 'cause that reveals to you what people looked at.

01:18:57.240 | It's very--

01:18:58.080 | - It's not always what you were expecting.

01:19:00.560 | - In the case of the lecture, it's pretty boring.

01:19:03.000 | It is what we're expecting, but we did a few funny videos

01:19:06.120 | where there's a bunch of people doing things

01:19:08.160 | and everybody tracks those people.

01:19:10.360 | In the beginning, they all look at the main person

01:19:12.120 | and they start spreading around

01:19:14.000 | and looking at the other people.

01:19:15.120 | It's fascinating, so that's a really strong signal

01:19:18.760 | of what people found exciting in the video.

01:19:21.920 | I don't know how you get that from people just watching,

01:19:24.420 | except they tuned out at this point.

01:19:27.720 | It's hard to measure this moment

01:19:30.960 | was super exciting for people.

01:19:32.540 | I don't know how you get that signal.

01:19:34.080 | Maybe comment, is there a way to get that signal

01:19:36.160 | where this was like, this is when their eyes opened up

01:19:39.200 | and they're like, for me with the Ray Dalio video,

01:19:42.880 | at first I was like, oh, okay, this is another one of these

01:19:45.760 | dumb it down for you videos,

01:19:47.880 | and then you start watching, it's like, okay,

01:19:50.160 | there's a really crisp, clean, deep explanation

01:19:52.720 | of how the economy works.

01:19:54.240 | That's where I set up and started watching.

01:19:56.600 | That moment, is there a way to detect that moment?

01:19:59.760 | - The only way I can think of is by asking people

01:20:02.080 | to label it. - Just ask.

01:20:03.560 | Yeah.

01:20:05.000 | You mentioned that we're quite far away

01:20:07.160 | in terms of doing video analysis, deep video analysis.

01:20:10.620 | Of course, Google, YouTube,

01:20:14.680 | we're quite far away from solving

01:20:17.520 | the autonomous driving problem too.

01:20:19.160 | So it's-- - I don't know,

01:20:20.440 | I think we're closer to that.

01:20:21.880 | - You never know, and the Wright brothers thought

01:20:27.120 | they're not gonna fly for 50 years,

01:20:29.120 | three years before they flew.

01:20:30.600 | So what are the biggest challenges, would you say?

01:20:34.840 | Is it the broad challenge of understanding video,

01:20:38.640 | understanding natural language, understanding the challenge

01:20:41.400 | before the entire machine learning community,

01:20:43.200 | or just being able to understand data?

01:20:45.400 | Or is there something specific to video

01:20:47.800 | that's even more challenging than understanding

01:20:51.120 | natural language, understanding,

01:20:52.840 | what's your sense of what the biggest challenge is?

01:20:53.680 | - I mean, video is just so much information.

01:20:56.640 | And so precision becomes a real problem.

01:21:00.960 | It's like, you're trying to classify something

01:21:05.160 | and you've got a million classes.

01:21:08.160 | And the distinctions among them,

01:21:12.160 | at least from a machine learning perspective,

01:21:17.080 | are often pretty small, right?

01:21:19.360 | Like, you need to see this person's number

01:21:24.360 | in order to know which player it is.

01:21:27.720 | And there's a lot of players.

01:21:30.900 | Or you need to see the logo on their chest

01:21:35.740 | in order to know which team they play for.

01:21:38.380 | And that's just figuring out who's who, right?

01:21:41.780 | And then you go further and saying,

01:21:43.020 | okay, well, was that a goal?

01:21:45.460 | Was it not a goal?

01:21:46.500 | Like, is that an interesting moment, as you said,

01:21:48.820 | or is that not an interesting moment?

01:21:51.380 | These things can be pretty hard.

01:21:52.860 | - So, okay, so Yan LeCun, I'm not sure if you're familiar

01:21:57.380 | sort of with his current thinking and work.

01:21:59.620 | So he believes that self,

01:22:02.020 | what he's referring to as self-supervised learning

01:22:04.980 | will be the solution sort of to achieving

01:22:08.180 | this kind of greater level of intelligence.

01:22:09.980 | In fact, the thing he's focusing on

01:22:12.140 | is watching video and predicting the next frame.

01:22:14.820 | So predicting the future of video, right?

01:22:16.860 | So for now, we're very far from that,

01:22:20.820 | but his thought is, because it's unsupervised,

01:22:23.740 | or as he refers to it as self-supervised,

01:22:26.240 | you know, if you watch enough video,

01:22:28.740 | essentially, if you watch YouTube,

01:22:31.740 | you'll be able to learn about the nature of reality,

01:22:34.140 | the physics, the common sense reasoning required

01:22:36.480 | by just teaching a system to predict the next frame.

01:22:40.260 | So he's confident this is the way to go.

01:22:42.580 | So for you, from the perspective

01:22:44.460 | of just working with this video,

01:22:47.060 | do you think an algorithm that just watches all of YouTube,

01:22:53.060 | stays up all day and night watching YouTube,

01:22:55.660 | would be able to understand enough

01:22:58.420 | of the physics of the world,

01:23:00.540 | about the way this world works,

01:23:02.100 | be able to do common sense reasoning and so on?

01:23:04.460 | - Well, I mean, we have systems

01:23:08.100 | that already watch all the videos on YouTube, right?

01:23:10.780 | But they're just looking for very specific things, right?

01:23:13.500 | They're supervised learning systems

01:23:15.860 | that are trying to identify something

01:23:18.700 | or classify something.

01:23:20.300 | And I don't know if predicting the next frame

01:23:24.700 | is really gonna get there,

01:23:25.700 | because I'm not an expert on compression algorithms,

01:23:30.700 | but I understand that that's kind of what compression,

01:23:33.420 | video compression algorithms do,

01:23:34.740 | is they basically try to predict the next frame

01:23:37.580 | and then fix up the places where they got it wrong.

01:23:41.860 | And that leads to higher compression

01:23:43.780 | than if you actually put all the bits

01:23:45.820 | for the next frame there.

01:23:46.780 | So I don't know if I believe

01:23:49.900 | that just being able to predict the next frame

01:23:52.820 | is gonna be enough,

01:23:53.820 | because there's so many frames

01:23:56.140 | and even a tiny bit of error on a per frame basis

01:24:00.140 | can lead to wildly different videos.

01:24:02.620 | - So the thing is,

01:24:04.100 | the idea of compression is one way to do compression

01:24:07.900 | is to describe through text what's contained in the video.

01:24:10.340 | That's the ultimate high level of compression.

01:24:12.180 | So the idea is traditionally,

01:24:14.780 | when you think of video image compression,

01:24:16.580 | you're trying to maintain the same visual quality

01:24:20.700 | while reducing the size.

01:24:22.380 | But if you think of deep learning

01:24:24.300 | from a bigger perspective of what compression is,

01:24:27.220 | is you're trying to summarize the video.

01:24:29.500 | And the idea there is,

01:24:30.860 | if you have a big enough neural network,

01:24:33.580 | just by watching the next,

01:24:35.140 | but trying to predict the next frame,

01:24:37.540 | you'll be able to form a compression

01:24:39.700 | of actually understanding what's going on in the scene.

01:24:42.260 | If there's two people talking,

01:24:44.620 | you can just reduce that entire video

01:24:46.700 | into the fact that two people are talking

01:24:48.900 | and maybe the content of what they're saying and so on.

01:24:51.620 | That's kind of the open-ended dream.

01:24:55.320 | So I just wanted to sort of express that

01:24:57.260 | 'cause it's an interesting, compelling notion,

01:24:59.340 | but it is nevertheless true that video,

01:25:04.340 | our world is a lot more complicated

01:25:06.740 | than we get a credit for.

01:25:07.900 | - I mean, in terms of search and discovery,

01:25:09.580 | we have been working on trying to summarize videos

01:25:14.420 | in text or with some kind of labels

01:25:17.260 | for eight years at least.

01:25:20.140 | And we're kind of so-so.

01:25:24.340 | - So if you were to say the problem is 100% solved

01:25:29.220 | and eight years ago was 0% solved,

01:25:32.560 | where are we on that timeline, would you say?

01:25:37.300 | - Yeah, to summarize a video well,

01:25:39.940 | maybe less than a quarter of the way.

01:25:42.460 | - So on that topic,

01:25:46.220 | what does YouTube look like 10, 20, 30 years from now?

01:25:51.220 | - I mean, I think that YouTube is evolving

01:25:54.580 | to take the place of TV.

01:25:56.840 | I grew up as a kid in the '70s

01:26:00.580 | and I watched a tremendous amount of television

01:26:03.740 | and I feel sorry for my poor mom

01:26:06.300 | because people told her at the time

01:26:09.080 | that it was gonna rot my brain

01:26:10.580 | and that she should kill her television.

01:26:14.260 | But anyway, I mean, I think that YouTube is,

01:26:17.060 | at least for my family,

01:26:18.460 | a better version of television, right?

01:26:21.980 | It's one that is on demand.

01:26:24.420 | It's more tailored to the things that my kids wanna watch.

01:26:28.340 | And also they can find things

01:26:30.820 | that they would never have found on television.

01:26:34.180 | And so I think that,

01:26:36.180 | at least from just observing my own family,

01:26:39.100 | that's where we're headed is that people watch YouTube

01:26:42.500 | kind of in the same way that I watched television

01:26:44.900 | when I was younger.

01:26:46.120 | - So from a search and discovery perspective,

01:26:49.220 | what are you excited about in the five, 10, 20, 30 years?

01:26:53.980 | Like what kind of things?

01:26:55.460 | It's already really good.

01:26:56.580 | I think it's achieved a lot of,

01:26:58.540 | of course we don't know what's possible.

01:27:01.900 | So it's the task of search of typing in the text

01:27:06.460 | or discovering new videos by the next recommendation.

01:27:09.480 | I personally am really happy with the experience.

01:27:11.980 | I continuously, I rarely watch a video that's not awesome

01:27:15.060 | from my own perspective, but what else is possible?

01:27:19.820 | What are you excited about?

01:27:21.260 | - Well, I think introducing people

01:27:24.100 | to more of what's available on YouTube

01:27:26.100 | is not only very important to YouTube and to creators,

01:27:30.540 | but I think it will help enrich people's lives

01:27:34.500 | because there's a lot that I'm still finding out

01:27:37.260 | is available on YouTube that I didn't even know.

01:27:40.500 | I've been working YouTube eight years

01:27:42.340 | and it wasn't until last year that I learned

01:27:44.580 | that I could watch USC football games from the 1970s.

01:27:49.580 | Like I didn't even know that was possible until last year

01:27:54.580 | and I've been working here quite some time.

01:27:55.940 | So what was broken about that?

01:27:58.940 | That it took me seven years to learn

01:28:01.060 | that this stuff was already on YouTube even when I got here.

01:28:04.540 | So I think there's a big opportunity there.

01:28:07.060 | And then, as I said before,

01:28:10.300 | we wanna make sure that YouTube finds a way to ensure

01:28:15.300 | that it's acting responsibly with respect to society

01:28:21.580 | and enriching people's lives.

01:28:23.300 | So we wanna take all of the great things that it does

01:28:26.220 | and make sure that we are eliminating

01:28:28.380 | the negative consequences that might happen.

01:28:31.780 | And then lastly, if we could get to a point

01:28:35.020 | where all the videos people watch

01:28:37.260 | are the best ones they've ever watched,

01:28:38.940 | that'd be outstanding too.

01:28:40.900 | - Do you see, in many senses,

01:28:42.500 | becoming a window into the world for people?

01:28:44.860 | Especially with live video, you get to watch events.

01:28:49.540 | I mean, it's really, it's the way you experience

01:28:52.500 | a lot of the world that's out there.

01:28:53.900 | It's better than TV in many, many ways.

01:28:56.620 | So do you see it becoming more than just video?

01:29:00.860 | Do you see creators creating visual experiences

01:29:03.980 | and virtual worlds?

01:29:05.620 | So if I'm talking crazy now,

01:29:07.260 | but sort of virtual reality and entering that space,

01:29:09.660 | or is that, at least for now,

01:29:11.420 | totally outside what YouTube is thinking about?

01:29:14.060 | - I mean, I think Google is thinking about virtual reality.

01:29:17.060 | I don't think about virtual reality too much.

01:29:20.740 | I know that we would wanna make sure that YouTube is there

01:29:27.300 | when virtual reality becomes something,

01:29:29.420 | or if virtual reality becomes something

01:29:31.380 | that a lot of people are interested in,

01:29:34.540 | but I haven't seen it really take off yet.

01:29:38.100 | - Take off.

01:29:38.940 | Well, the future is wide open.

01:29:41.460 | Christos, I've been really looking forward

01:29:43.220 | to this conversation.

01:29:44.060 | It's been a huge honor.

01:29:45.100 | Thank you for answering some of the more

01:29:46.820 | difficult questions I've asked.

01:29:48.620 | I'm really excited about what YouTube has in store for us.

01:29:52.220 | It's one of the greatest products I've ever used

01:29:53.980 | and continues.

01:29:54.820 | So thank you so much for talking to me.

01:29:56.420 | - It's my pleasure.

01:29:57.260 | Thanks for asking me.

01:29:58.300 | - Thanks for listening to this conversation

01:30:01.380 | and thank you to our presenting sponsor, Cash App.

01:30:04.540 | Download it, use code LEXPODCAST.

01:30:07.220 | You'll get $10 and $10 will go to FIRST,

01:30:10.020 | a STEM education nonprofit that inspires hundreds

01:30:12.820 | of thousands of young minds to become future leaders

01:30:15.940 | and innovators.

01:30:17.380 | If you enjoy this podcast, subscribe on YouTube,

01:30:20.180 | get five stars on Apple Podcast,

01:30:22.220 | follow on Spotify, support on Patreon,

01:30:24.860 | or simply connect with me on Twitter.

01:30:26.900 | And now let me leave you with some words of wisdom

01:30:30.540 | from Marcel Proust.

01:30:32.460 | The real voyage of discovery consists not in seeking

01:30:35.620 | new landscapes, but in having new eyes.

01:30:38.960 | Thank you for listening.

01:30:41.100 | I hope to see you next time.

01:30:42.740 | (upbeat music)

01:30:45.320 | (upbeat music)

01:30:47.900 | [BLANK_AUDIO]

Cristos Goodrow: YouTube Algorithm | Lex Fridman Podcast #68

Chapters