David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI

00:00:00.000 | The following is a conversation with David Ferrucci.

00:00:03.040 | He led the team that built Watson,

00:00:05.200 | the IBM question answering system

00:00:07.040 | that beat the top humans in the world

00:00:09.080 | at the game of Jeopardy.

00:00:11.160 | For spending a couple hours with David,

00:00:12.920 | I saw a genuine passion,

00:00:14.960 | not only for abstract understanding of intelligence,

00:00:17.740 | but for engineering it to solve real world problems

00:00:21.240 | under real world deadlines and resource constraints.

00:00:24.800 | Where science meets engineering

00:00:26.520 | is where brilliant, simple ingenuity emerges.

00:00:29.960 | People who work at joining it to

00:00:32.120 | have a lot of wisdom earned

00:00:33.840 | through failures and eventual success.

00:00:37.000 | David is also the founder, CEO,

00:00:39.100 | and chief scientist of Elemental Cognition,

00:00:41.680 | a company working to engineer AI systems

00:00:44.500 | that understand the world the way people do.

00:00:47.440 | This is the Artificial Intelligence Podcast.

00:00:50.300 | If you enjoy it, subscribe on YouTube,

00:00:52.740 | give it five stars on iTunes,

00:00:54.460 | support it on Patreon,

00:00:55.880 | or simply connect with me on Twitter

00:00:57.960 | at Lex Friedman, spelled F-R-I-D-M-A-N.

00:01:01.400 | And now, here's my conversation with David Ferrucci.

00:01:05.200 | Your undergrad was in biology

00:01:08.040 | with an eye toward medical school

00:01:11.320 | before you went on for the PhD in computer science.

00:01:14.360 | So let me ask you an easy question.

00:01:16.840 | What is the difference between biological systems

00:01:20.560 | and computer systems?

00:01:21.880 | When you sit back, look at the stars,

00:01:26.320 | and think philosophically.

00:01:28.580 | - I often wonder whether or not

00:01:30.820 | there is a substantive difference.

00:01:32.900 | I mean, I think the thing that got me

00:01:34.480 | into computer science,

00:01:35.980 | into artificial intelligence,

00:01:37.220 | was exactly this presupposition that

00:01:40.980 | if we can get machines to think,

00:01:44.340 | or I should say this question,

00:01:45.620 | this philosophical question,

00:01:47.460 | if we can get machines to think,

00:01:50.580 | to understand, to process information the way we do,

00:01:54.800 | so if we can describe a procedure,

00:01:56.460 | describe a process,

00:01:57.960 | even if that process were the intelligence process itself,

00:02:02.500 | then what would be the difference?

00:02:05.280 | So from a philosophical standpoint,

00:02:07.680 | I'm not sure I'm convinced that there is.

00:02:11.660 | I mean, you can go in the direction of spirituality,

00:02:14.960 | you can go in the direction of the soul,

00:02:16.660 | but in terms of what we can experience

00:02:22.040 | from an intellectual and physical perspective,

00:02:26.000 | I'm not sure there is.

00:02:27.500 | Clearly, there are different implementations,

00:02:31.120 | but if you were to say,

00:02:33.240 | is a biological information processing system

00:02:36.200 | fundamentally more capable

00:02:38.460 | than one we might be able to build out of silicon

00:02:41.040 | or some other substrate,

00:02:43.900 | I don't know that there is.

00:02:46.560 | - How distant do you think is the biological implementation?

00:02:50.600 | So fundamentally, they may have the same capabilities,

00:02:53.840 | but is it really a far mystery

00:02:58.300 | where a huge number of breakthroughs are needed

00:03:00.720 | to be able to understand it,

00:03:02.720 | or is it something that, for the most part,

00:03:06.320 | in the important aspects,

00:03:08.680 | echoes of the same kind of characteristics?

00:03:11.160 | - Yeah, that's interesting.

00:03:12.840 | So your question presupposes that there's this goal

00:03:16.560 | to recreate what we perceive as biological intelligence.

00:03:20.920 | I'm not sure that's how I would state the goal.

00:03:25.920 | I mean, I think that's--

00:03:27.440 | - What is the goal?

00:03:29.240 | - Good, so I think there are a few goals.

00:03:32.200 | I think that understanding the human brain

00:03:35.760 | and how it works is important

00:03:38.560 | for us to be able to diagnose and treat issues

00:03:44.760 | for us to understand our own strengths and weaknesses,

00:03:48.240 | both intellectual, psychological, and physical.

00:03:52.440 | So neuroscience and understanding the brain

00:03:55.000 | from that perspective, there's a clear goal there.

00:03:59.560 | From the perspective of saying,

00:04:01.160 | I wanna mimic human intelligence,

00:04:04.800 | that one's a little bit more interesting.

00:04:06.440 | Human intelligence certainly has a lot of things we envy.

00:04:10.480 | It's also got a lot of problems, too.

00:04:12.880 | So I think we're capable of sort of stepping back

00:04:16.680 | and saying, what do we want out of an intelligence?

00:04:21.680 | How do we wanna communicate with that intelligence?

00:04:24.400 | How do we want it to behave?

00:04:25.560 | How do we want it to perform?

00:04:27.460 | Now, of course, it's somewhat of an interesting argument

00:04:30.360 | because I'm sitting here as a human with a biological brain

00:04:33.940 | and I'm critiquing the strengths and weaknesses

00:04:36.440 | of human intelligence and saying that we have

00:04:39.080 | the capacity to step back and say,

00:04:42.560 | gee, what is intelligence and what do we really want

00:04:45.120 | out of it, and that in and of itself suggests

00:04:48.100 | that human intelligence is something quite enviable,

00:04:52.100 | that it can introspect that way.

00:04:57.100 | - And the flaws, you mentioned the flaws.

00:05:00.320 | So humans have flaws.

00:05:01.160 | - Yeah, I think that flaws that human intelligence has

00:05:04.720 | is extremely prejudicial and biased

00:05:08.440 | in the way it draws many inferences.

00:05:10.480 | - Do you think those are, sorry to interrupt,

00:05:12.040 | do you think those are features or are those bugs?

00:05:14.400 | Do you think the prejudice, the forgetfulness, the fear,

00:05:19.400 | what are the flaws?

00:05:22.900 | List them all.

00:05:23.740 | What, love?

00:05:24.560 | Maybe that's a flaw.

00:05:25.600 | You think those are all things that can be,

00:05:28.160 | get in the way of intelligence

00:05:30.820 | or the essential components of intelligence?

00:05:33.440 | - Well, again, if you go back and you define intelligence

00:05:36.200 | as being able to sort of accurately, precisely,

00:05:41.240 | rigorously reason, develop answers,

00:05:43.840 | and justify those answers in an objective way,

00:05:46.640 | yeah, then human intelligence has these flaws

00:05:49.720 | in that it tends to be more influenced

00:05:52.880 | by some of the things you said.

00:05:55.180 | And it's largely an inductive process,

00:05:59.740 | meaning it takes past data, uses that to predict the future,

00:06:03.560 | very advantageous in some cases,

00:06:06.000 | but fundamentally biased and prejudicial in other cases,

00:06:09.280 | 'cause it's gonna be strongly influenced

00:06:10.760 | by its priors, whether they're right or wrong

00:06:13.880 | from some objective reasoning perspective,

00:06:17.420 | you're gonna favor them because those are the decisions

00:06:20.520 | or those are the paths that succeeded in the past.

00:06:24.240 | And I think that mode of intelligence makes a lot of sense

00:06:29.240 | for when your primary goal is to act quickly

00:06:33.680 | and survive and make fast decisions.

00:06:37.240 | And I think those create problems

00:06:40.360 | when you wanna think more deeply

00:06:42.080 | and make more objective and reasoned decisions.

00:06:45.160 | Of course, human's capable of doing both.

00:06:48.400 | They do sort of one more naturally than they do the other,

00:06:51.120 | but they're capable of doing both.

00:06:53.320 | - You're saying they do the one

00:06:54.560 | that responds quickly more naturally.

00:06:56.520 | - Right.

00:06:57.360 | - 'Cause that's the thing we kinda need

00:06:58.460 | to not be eaten by the predators in the world.

00:07:02.760 | - For example, but then we've learned to reason

00:07:08.960 | through logic, we've developed science,

00:07:11.240 | we've trained people to do that.

00:07:13.020 | I think that's harder for the individual to do.

00:07:17.000 | I think it requires training and teaching.

00:07:21.000 | I think we are, the human mind certainly is capable of it,

00:07:24.220 | but we find it more difficult.

00:07:25.320 | And then there are other weaknesses, if you will,

00:07:27.720 | as you mentioned earlier, just memory capacity

00:07:30.660 | and how many chains of inference can you actually go through

00:07:35.780 | without losing your way, so just focus.

00:07:39.260 | - So the way you think about intelligence,

00:07:43.300 | and we're really sort of floating

00:07:45.100 | in this philosophical space,

00:07:47.260 | but I think you're the perfect person to talk about this

00:07:51.140 | because we'll get to Jeopardy and beyond.

00:07:55.700 | That's one of the most incredible accomplishments in AI,

00:07:59.500 | in the history of AI,

00:08:00.940 | but hence the philosophical discussion.

00:08:03.440 | So let me ask, you've kind of alluded to it,

00:08:06.300 | but let me ask again, what is intelligence?

00:08:09.420 | Underlying the discussion we'll have

00:08:12.500 | with Jeopardy and beyond.

00:08:15.540 | How do you think about intelligence?

00:08:17.140 | Is it a sufficiently complicated problem,

00:08:19.820 | being able to reason your way through solving that problem?

00:08:22.500 | Is that kind of how you think about

00:08:23.820 | what it means to be intelligent?

00:08:25.460 | - So I think of intelligence primarily two ways.

00:08:29.700 | One is the ability to predict.

00:08:33.320 | So in other words, if I have a problem,

00:08:35.820 | can I predict what's gonna happen next,

00:08:37.620 | whether it's to predict the answer of a question

00:08:40.900 | or to say, look, I'm looking at all the market dynamics

00:08:43.900 | and I'm gonna tell you what's gonna happen next,

00:08:46.160 | or you're in a room and somebody walks in

00:08:49.380 | and you're gonna predict what they're gonna do next

00:08:51.340 | or what they're gonna say next.

00:08:52.900 | - In a highly dynamic environment full of uncertainty,

00:08:56.540 | be able to predict.

00:08:58.620 | - The more variables, the more complex.

00:09:01.500 | The more possibilities, the more complex.

00:09:04.080 | But can I take a small amount of prior data

00:09:07.720 | and learn the pattern and then predict

00:09:09.880 | what's gonna happen next accurately and consistently?

00:09:13.000 | That's certainly a form of intelligence.

00:09:16.960 | - What do you need for that, by the way?

00:09:18.320 | You need to have an understanding of the way the world works

00:09:22.880 | in order to be able to unroll it into the future.

00:09:25.320 | What do you think is needed to predict?

00:09:28.040 | - Depends what you mean by understanding.

00:09:30.380 | I need to be able to find that function.

00:09:32.260 | This is very much what deep learning does,

00:09:35.140 | machine learning does, is if you give me enough prior data

00:09:39.000 | and you tell me what the output variable is that matters,

00:09:41.980 | I'm gonna sit there and be able to predict it.

00:09:44.500 | And if I can predict it accurately

00:09:47.300 | so that I can get it right more often than not, I'm smart.

00:09:50.980 | If I can do that with less data and less training time,

00:09:54.820 | I'm even smarter.

00:09:58.020 | If I can figure out what's even worth predicting,

00:10:00.620 | I'm smarter, meaning I'm figuring out

00:10:03.920 | what path is gonna get me toward a goal.

00:10:06.380 | - What about picking a goal?

00:10:07.580 | Sorry, you're up again.

00:10:08.500 | - Well, that's interesting about picking a goal.

00:10:10.100 | Sort of an interesting thing.

00:10:11.060 | I think that's where you bring in

00:10:13.220 | what are you pre-programmed to do?

00:10:15.020 | We talk about humans and, well,

00:10:17.020 | humans are pre-programmed to survive.

00:10:19.380 | So sort of their primary driving goal,

00:10:23.320 | what do they have to do to do that?

00:10:24.700 | And that can be very complex, right?

00:10:27.380 | So it's not just figuring out

00:10:30.820 | that you need to run away from the ferocious tiger,

00:10:33.640 | but we survive in a social context as an example.

00:10:38.640 | So understanding the subtleties of social dynamics

00:10:42.300 | becomes something that's important for surviving,

00:10:45.420 | finding a mate, reproducing, right?

00:10:47.180 | So we're continually challenged

00:10:49.360 | with complex sets of variables, complex constraints,

00:10:53.760 | rules, if you will, or patterns.

00:10:56.880 | And we learn how to find the functions

00:10:59.340 | and predict the things.

00:11:00.700 | In other words, represent those patterns efficiently

00:11:03.580 | and be able to predict what's gonna happen.

00:11:04.940 | And that's a form of intelligence.

00:11:06.060 | That doesn't really require anything specific

00:11:11.060 | other than the ability to find that function

00:11:13.380 | and predict that right answer.

00:11:15.860 | It's certainly a form of intelligence.

00:11:18.460 | But then when we say, well, do we understand each other?

00:11:23.300 | In other words, would you perceive me as intelligent

00:11:28.300 | beyond that ability to predict?

00:11:31.040 | So now I can predict, but I can't really articulate

00:11:35.200 | how I'm going through that process,

00:11:37.840 | what my underlying theory is for predicting.

00:11:41.240 | And I can't get you to understand what I'm doing

00:11:43.680 | so that you can follow,

00:11:45.660 | you can figure out how to do this yourself

00:11:48.080 | if you did not have, for example,

00:11:50.800 | the right pattern-managing machinery that I did.

00:11:53.860 | And now we potentially have this breakdown

00:11:55.780 | where in effect, I'm intelligent,

00:11:59.120 | but I'm sort of an alien intelligence relative to you.

00:12:02.660 | - You're intelligent, but nobody knows about it.

00:12:05.060 | Or I can't-- - Well, I can see the output.

00:12:08.660 | - So you're saying, let's sort of separate the two things.

00:12:11.700 | One is you explaining why you were able

00:12:15.940 | to predict the future.

00:12:17.420 | And the second is me being able to,

00:12:22.420 | impressing me that you're intelligent.

00:12:25.560 | Me being able to know

00:12:26.560 | that you successfully predicted the future.

00:12:28.720 | Do you think that's--

00:12:29.640 | - Well, it's not impressing you that I'm intelligent.

00:12:31.400 | In other words, you may be convinced

00:12:33.680 | that I'm intelligent in some form.

00:12:35.920 | Because of my ability to predict.

00:12:38.920 | - So I would look at the metrics.

00:12:39.760 | - When you pass, I say, wow, you're right more times

00:12:43.940 | than I am, you're doing something interesting.

00:12:46.360 | That's a form of intelligence.

00:12:49.160 | But then what happens is, if I say, how are you doing that?

00:12:53.440 | And you can't communicate with me,

00:12:55.320 | and you can't describe that to me,

00:12:57.760 | now I may label you a savant.

00:13:00.760 | I may say, well, you're doing something weird,

00:13:03.280 | and it's just not very interesting to me,

00:13:06.420 | because you and I can't really communicate.

00:13:09.400 | And so now, so this is interesting, right?

00:13:12.400 | Because now this is, you're in this weird place

00:13:15.160 | where for you to be recognized as intelligent

00:13:19.360 | the way I'm intelligent, then you and I

00:13:22.040 | sort of have to be able to communicate.

00:13:24.320 | And then we start to understand each other,

00:13:28.560 | and then my respect and my appreciation,

00:13:33.520 | my ability to relate to you starts to change.

00:13:36.800 | So now you're not an alien intelligence anymore,

00:13:39.120 | you're a human intelligence now,

00:13:41.100 | because you and I can communicate.

00:13:43.920 | And so I think when we look at animals, for example,

00:13:48.120 | animals can do things we can't quite comprehend,

00:13:50.720 | we don't quite know how they do them,

00:13:51.800 | but they can't really communicate with us.

00:13:54.420 | They can't put what they're going through in our terms.

00:13:58.360 | And so we think of them as sort of,

00:13:59.720 | well, they're these alien intelligences,

00:14:01.520 | and they're not really worth necessarily what we're worth,

00:14:03.600 | we don't treat them the same way as a result of that.

00:14:06.360 | But it's hard, because who knows what's going on.

00:14:11.640 | - So just a quick elaboration on that,

00:14:15.640 | the explaining that you're intelligent,

00:14:17.960 | explaining the reasoning that went into the prediction

00:14:22.280 | is not some kind of mathematical proof.

00:14:27.080 | If we look at humans, look at political debates

00:14:30.240 | and discourse on Twitter, it's mostly just telling stories.

00:14:35.240 | So your task is, sorry, your task is not to tell

00:14:41.080 | an accurate depiction of how you reason,

00:14:45.160 | but to tell a story, real or not,

00:14:48.440 | that convinces me that there was a mechanism

00:14:51.120 | by which you--

00:14:51.960 | - Ultimately, that's what a proof is.

00:14:53.640 | I mean, even a mathematical proof is that.

00:14:56.240 | Because ultimately, the other mathematicians

00:14:58.220 | have to be convinced by your proof.

00:15:00.020 | Otherwise, in fact, there have been--

00:15:03.000 | - That's the metric of success, yeah.

00:15:04.440 | - Yeah, there have been several proofs out there

00:15:06.060 | where mathematicians would study for a long time

00:15:08.040 | before they were convinced

00:15:08.880 | that it actually proved anything, right?

00:15:10.880 | You never know if it proved anything

00:15:12.200 | until the community mathematicians decided that it did.

00:15:14.840 | So, I mean, but it's a real thing, right?

00:15:18.640 | And that's sort of the point, right?

00:15:20.920 | Is that ultimately, this notion of understanding us,

00:15:24.600 | understanding something is ultimately a social concept.

00:15:28.200 | In other words, I have to convince enough people

00:15:30.700 | that I did this in a reasonable way.

00:15:33.720 | I did this in a way that other people can understand

00:15:36.360 | and replicate and that it makes sense to them.

00:15:39.800 | So, human intelligence is bound together in that way.

00:15:44.800 | We're bound up in that sense.

00:15:47.440 | We sort of never really get away with it

00:15:49.520 | until we can sort of convince others

00:15:52.560 | that our thinking process makes sense.

00:15:55.840 | - Did you think the general question of intelligence

00:15:59.080 | is then also a social construct?

00:16:00.960 | So, if we ask questions

00:16:05.280 | of an artificial intelligence system,

00:16:06.680 | is this system intelligent?

00:16:08.600 | The answer will ultimately be a socially constructed concept.

00:16:13.600 | - So, I think, I'm making two statements.

00:16:16.040 | I'm saying we can try to define intelligence

00:16:18.000 | in this super objective way that says,

00:16:21.360 | here's this data, I wanna predict this type of thing,

00:16:25.740 | learn this function, and then if you get it right,

00:16:28.820 | often enough, we consider you intelligent.

00:16:32.080 | - But that's more like a savant.

00:16:34.440 | - I think it is.

00:16:35.720 | It doesn't mean it's not useful.

00:16:37.560 | It could be incredibly useful.

00:16:38.640 | It could be solving a problem we can't otherwise solve

00:16:41.480 | and can solve it more reliably than we can.

00:16:44.520 | But then there's this notion of,

00:16:46.960 | can humans take responsibility

00:16:50.440 | for the decision that you're making?

00:16:53.680 | Can we make those decisions ourselves?

00:16:56.120 | Can we relate to the process that you're going through?

00:16:58.840 | And now, you as an agent,

00:17:01.160 | whether you're a machine or another human, frankly,

00:17:04.540 | are now obliged to make me understand

00:17:08.680 | how it is that you're arriving at that answer

00:17:10.880 | and allow me, me or obviously a community

00:17:13.880 | or a judge of people to decide whether or not

00:17:16.840 | that makes sense.

00:17:17.680 | And by the way, that happens with humans as well.

00:17:20.240 | You're sitting down with your staff, for example,

00:17:22.080 | and you ask for suggestions about what to do next,

00:17:25.560 | and someone says, oh, I think you should buy,

00:17:28.680 | and I think you should buy this much,

00:17:30.600 | or whatever, or sell, or whatever it is,

00:17:33.200 | or I think you should launch the product today or tomorrow,

00:17:35.740 | or launch this product versus that product,

00:17:37.140 | whatever the decision may be, and you ask why,

00:17:39.860 | and the person says, I just have a good feeling about it.

00:17:42.820 | And you're not very satisfied.

00:17:44.420 | Now, that person could be, you might say,

00:17:48.380 | well, you've been right before,

00:17:50.920 | but I'm gonna put the company on the line.

00:17:54.180 | Can you explain to me why I should believe this?

00:17:56.820 | - And that explanation may have nothing to do

00:18:00.060 | with the truth.

00:18:01.860 | - You just gotta convince the other person.

00:18:03.520 | It could still be wrong.

00:18:04.360 | It could still be wrong.

00:18:05.320 | - It's just gotta be convincing.

00:18:06.320 | - But it's ultimately gotta be convincing.

00:18:07.880 | And that's why I'm saying we're bound together.

00:18:12.200 | Our intelligences are bound together in that sense.

00:18:14.200 | We have to understand each other.

00:18:15.400 | And if, for example, you're giving me an explanation,

00:18:18.960 | I mean, this is a very important point,

00:18:21.060 | you're giving me an explanation,

00:18:23.120 | and I'm not good,

00:18:29.420 | and then I'm not good at reasoning well

00:18:33.560 | and being objective and following logical paths

00:18:38.080 | and consistent paths,

00:18:39.200 | and I'm not good at measuring

00:18:41.440 | and sort of computing probabilities across those paths,

00:18:45.560 | what happens is collectively,

00:18:47.240 | we're not gonna do well.

00:18:50.160 | - How hard is that problem, the second one?

00:18:53.200 | So I think we'll talk quite a bit about the first

00:18:58.000 | on a specific objective metric benchmark performing well.

00:19:03.000 | But being able to explain the steps, the reasoning,

00:19:08.860 | how hard is that problem?

00:19:10.580 | - I think that's very hard.

00:19:11.820 | I mean, I think that that's,

00:19:13.300 | well, it's hard for humans.

00:19:18.180 | - The thing that's hard for humans, as you know,

00:19:20.980 | may not necessarily be hard for computers

00:19:22.940 | and vice versa.

00:19:24.460 | So, sorry, so how hard is that problem for computers?

00:19:29.460 | - I think it's hard for computers,

00:19:32.660 | and the reason why I related to,

00:19:34.620 | or saying that it's also hard for humans

00:19:36.440 | is because I think when we step back

00:19:38.360 | and we say we wanna design computers to do that,

00:19:41.960 | one of the things we have to recognize

00:19:46.480 | is we're not sure how to do it well.

00:19:50.520 | I'm not sure we have a recipe for that,

00:19:52.960 | and even if you wanted to learn it,

00:19:55.340 | it's not clear exactly what data we use

00:19:58.420 | and what judgments we use to learn that well.

00:20:03.720 | And so what I mean by that is,

00:20:05.460 | if you look at the entire enterprise of science,

00:20:09.500 | science is supposed to be about

00:20:11.620 | objective reasoning reason, right?

00:20:13.700 | So we think about, gee, who's the most intelligent person

00:20:17.660 | or group of people in the world?

00:20:20.500 | Do we think about the savants who can close their eyes

00:20:24.080 | and give you a number?

00:20:25.540 | We think about the think tanks,

00:20:27.680 | or the scientists or the philosophers

00:20:29.500 | who kinda work through the details and write the papers

00:20:33.820 | and come up with the thoughtful, logical proofs

00:20:37.100 | and use the scientific method,

00:20:38.620 | and I think it's the latter.

00:20:40.680 | And my point is that, how do you train someone to do that?

00:20:45.780 | And that's what I mean by it's hard.

00:20:47.600 | What's the process of training people to do that well?

00:20:50.800 | That's a hard process.

00:20:52.400 | We work, as a society, we work pretty hard

00:20:56.040 | to get other people to understand our thinking

00:20:59.240 | and to convince them of things.

00:21:02.220 | Now we could persuade them,

00:21:04.040 | obviously we talked about this,

00:21:05.300 | like human flaws or weaknesses,

00:21:07.500 | we can persuade them through emotional means,

00:21:12.180 | but to get them to understand and connect to

00:21:16.140 | and follow a logical argument is difficult.

00:21:19.960 | We try it, we do it as scientists,

00:21:22.440 | we try to do it as journalists,

00:21:24.200 | we try to do it as even artists in many forms,

00:21:27.280 | as writers, as teachers.

00:21:29.760 | We go through a fairly significant training process

00:21:32.920 | to do that, and then we could ask,

00:21:35.220 | well, why is that so hard?

00:21:37.920 | But it's hard, and for humans, it takes a lot of work.

00:21:42.940 | And when we step back and say,

00:21:45.960 | well, how do we get a machine to do that?

00:21:49.160 | It's a vexing question.

00:21:50.620 | - How would you begin to try to solve that?

00:21:55.240 | And maybe just a quick pause,

00:21:57.400 | because there's an optimistic notion

00:21:59.840 | in the things you're describing,

00:22:01.040 | which is being able to explain something through reason.

00:22:05.100 | But if you look at algorithms that recommend things

00:22:08.660 | that we'll look at next,

00:22:09.780 | whether it's Facebook, Google,

00:22:11.800 | advertisement-based companies,

00:22:14.600 | you know, their goal is to convince you to buy things

00:22:19.400 | based on anything.

00:22:22.400 | So that could be reason,

00:22:25.480 | 'cause the best of advertisement

00:22:27.180 | is showing you things that you really do need

00:22:29.640 | and explain why you need it.

00:22:32.020 | But it could also be through emotional manipulation.

00:22:35.700 | The algorithm that describes why a certain reason,

00:22:40.840 | a certain decision was made,

00:22:43.800 | how hard is it to do it through emotional manipulation?

00:22:48.200 | And why is that a good or a bad thing?

00:22:51.960 | So you've kind of focused on reason, logic,

00:22:56.920 | really showing in a clear way why something is good.

00:23:01.500 | One, is that even a thing that us humans do?

00:23:05.960 | And two, how do you think of the difference

00:23:09.920 | in the reasoning aspect and the emotional manipulation?

00:23:13.420 | - So you call it emotional manipulation,

00:23:17.320 | but more objectively, it's essentially saying,

00:23:20.160 | there are certain features of things

00:23:22.600 | that seem to attract your attention.

00:23:24.400 | I mean, it kind of give you more of that stuff.

00:23:26.760 | - Manipulation is a bad word.

00:23:28.240 | - Yeah, I mean, I'm not saying it's right or wrong.

00:23:31.120 | It works to get your attention,

00:23:32.960 | and it works to get you to buy stuff.

00:23:34.400 | And when you think about algorithms

00:23:35.940 | that look at the patterns of features

00:23:39.980 | that you seem to be spending your money on,

00:23:41.880 | and say, I'm gonna give you something

00:23:43.220 | with a similar pattern,

00:23:44.780 | so I'm gonna learn that function,

00:23:46.060 | because the objective is to get you to click on it

00:23:48.180 | or get you to buy it or whatever it is.

00:23:50.180 | I don't know, I mean, it is what it is.

00:23:53.340 | I mean, that's what the algorithm does.

00:23:55.800 | You can argue whether it's good or bad.

00:23:57.420 | It depends what your goal is.

00:24:00.380 | - I guess this seems to be very useful for convincing,

00:24:04.140 | for telling a story.

00:24:04.980 | - For convincing humans, it's good,

00:24:07.660 | because again, this goes back to,

00:24:09.580 | what is the human behavior like?

00:24:12.060 | How does the human brain respond to things?

00:24:16.980 | I think there's a more optimistic view of that, too,

00:24:19.340 | which is that if you're searching

00:24:21.980 | for certain kinds of things,

00:24:23.100 | you've already reasoned that you need them.

00:24:26.140 | And these algorithms are saying, look, that's up to you

00:24:30.020 | to reason whether you need something or not.

00:24:32.160 | That's your job.

00:24:33.640 | You may have an unhealthy addiction to this stuff,

00:24:36.900 | or you may have a reasoned and thoughtful explanation

00:24:41.900 | for why it's important to you,

00:24:44.500 | and the algorithms are saying, hey, that's whatever.

00:24:47.220 | That's your problem.

00:24:48.060 | All I know is you're buying stuff like that,

00:24:50.580 | you're interested in stuff like that.

00:24:51.900 | Could be a bad reason, could be a good reason.

00:24:53.900 | That's up to you.

00:24:55.020 | I'm gonna show you more of that stuff.

00:24:57.060 | And I think that it's not good or bad.

00:25:02.220 | It's not reasoned or not reasoned.

00:25:03.540 | The algorithm is doing what it does,

00:25:04.920 | which is saying, you seem to be interested in this.

00:25:06.920 | I'm gonna show you more of that stuff.

00:25:09.340 | And I think we're seeing this not just in buying stuff,

00:25:11.200 | but even in social media.

00:25:12.160 | You're reading this kind of stuff.

00:25:13.980 | I'm not judging on whether it's good or bad.

00:25:15.740 | I'm not reasoning at all.

00:25:16.940 | I'm just saying, I'm gonna show you other stuff

00:25:19.200 | with similar features.

00:25:20.820 | And that's it, and I wash my hands from it,

00:25:23.540 | and I say, that's all that's going on.

00:25:25.920 | - People are so harsh on AI systems.

00:25:31.920 | So one, the bar of performance is extremely high,

00:25:34.940 | and yet we also ask them to, in the case of social media,

00:25:39.580 | to help find the better angels of our nature

00:25:42.940 | and help make a better society.

00:25:45.980 | So what do you think about the role of AI?

00:25:47.900 | - So that, I agree with you.

00:25:48.980 | That's the interesting dichotomy, right?

00:25:51.580 | Because on one hand, we're sitting there

00:25:54.180 | and we're sort of doing the easy part,

00:25:55.900 | which is finding the patterns.

00:25:57.980 | We're not building, the system's not building a theory

00:26:01.820 | that is consumable and understandable by other humans

00:26:04.200 | that can be explained and justified.

00:26:06.360 | And so on one hand to say, oh, AI is doing this.

00:26:11.360 | Why isn't it doing this other thing?

00:26:13.700 | Well, this other thing's a lot harder.

00:26:16.280 | And it's interesting to think about why it's harder.

00:26:20.160 | And because you're interpreting the data

00:26:23.960 | in the context of prior models.

00:26:26.260 | In other words, understandings of what's important

00:26:28.920 | in the world, what's not important.

00:26:30.220 | What are all the other abstract features

00:26:32.040 | that drive our decision-making?

00:26:35.380 | What's sensible, what's not sensible, what's good,

00:26:37.440 | what's bad, what's moral, what's valuable, what isn't?

00:26:40.020 | Where is that stuff?

00:26:41.120 | No one's applying the interpretation.

00:26:43.240 | So when I see you clicking on a bunch of stuff

00:26:46.640 | and I look at these simple features, the raw features,

00:26:49.760 | the features that are there in the data,

00:26:51.080 | like what words are being used,

00:26:53.340 | or how long the material is,

00:26:57.680 | or other very superficial features,

00:27:00.620 | what colors are being used in the material.

00:27:02.540 | Like I don't know why you're clicking

00:27:03.900 | on the stuff you're looking.

00:27:04.980 | Or if it's products, what the price is,

00:27:07.620 | or what the category is, and stuff like that.

00:27:09.580 | And I just feed you more of the same stuff.

00:27:11.560 | That's very different than kind of getting in there

00:27:13.780 | and saying, what does this mean?

00:27:16.000 | The stuff you're reading, like why are you reading it?

00:27:20.380 | What assumptions are you bringing to the table?

00:27:23.940 | Are those assumptions sensible?

00:27:26.400 | Does the material make any sense?

00:27:29.020 | Does it lead you to thoughtful, good conclusions?

00:27:34.020 | Again, there's interpretation and judgment involved

00:27:37.460 | in that process that isn't really happening in the AI today.

00:27:42.460 | That's harder because you have to start getting

00:27:47.220 | at the meaning of the stuff, of the content.

00:27:52.040 | You have to get at how humans interpret the content

00:27:55.780 | relative to their value system

00:27:58.720 | and deeper thought processes.

00:28:00.600 | - So that's what meaning means,

00:28:02.140 | is not just some kind of deep, timeless, semantic thing

00:28:07.140 | that the statement represents,

00:28:10.960 | but also how a large number of people

00:28:13.400 | are likely to interpret.

00:28:15.240 | So it's again, even meaning is a social construct.

00:28:19.120 | So you have to try to predict how most people

00:28:22.800 | would understand this kind of statement.

00:28:24.520 | - Yeah, meaning is often relative,

00:28:27.280 | but meaning implies that the connections

00:28:29.860 | go beneath the surface of the artifacts.

00:28:31.840 | If I show you a painting, it's a bunch of colors on a canvas,

00:28:35.480 | what does it mean to you?

00:28:37.140 | And it may mean different things to different people

00:28:39.420 | because of their different experiences.

00:28:42.240 | It may mean something even different

00:28:44.720 | to the artist who painted it.

00:28:46.460 | As we try to get more rigorous with our communication,

00:28:50.720 | we try to really nail down that meaning.

00:28:53.280 | So we go from abstract art to precise mathematics,

00:28:58.280 | precise engineering drawings and things like that.

00:29:01.520 | We're really trying to say,

00:29:03.440 | I wanna narrow that space of possible interpretations

00:29:08.280 | because the precision of the communication

00:29:10.720 | ends up becoming more and more important.

00:29:13.400 | And so that means that I have to specify,

00:29:17.920 | and I think that's why this becomes really hard,

00:29:21.400 | because if I'm just showing you an artifact

00:29:24.200 | and you're looking at it superficially,

00:29:26.000 | whether it's a bunch of words on a page

00:29:28.240 | or whether it's brushstrokes on a canvas

00:29:31.940 | or pixels in a photograph,

00:29:33.600 | you can sit there and you can interpret

00:29:35.080 | lots of different ways at many, many different levels.

00:29:37.840 | But when I wanna align our understanding of that,

00:29:44.960 | I have to specify a lot more stuff

00:29:48.400 | that's actually not directly in the artifact.

00:29:52.320 | Now I have to say, well, how are you interpreting

00:29:56.000 | this image and that image?

00:29:57.280 | And what about the colors and what do they mean to you?

00:29:59.400 | What perspective are you bringing to the table?

00:30:02.600 | What are your prior experiences with those artifacts?

00:30:05.640 | What are your fundamental assumptions and values?

00:30:08.840 | What is your ability to kind of reason

00:30:10.880 | to chain together logical implication

00:30:13.680 | as you're sitting there and saying,

00:30:14.520 | well, if this is the case, then I would conclude this.

00:30:16.560 | If that's the case, then I would conclude that.

00:30:19.120 | So your reasoning processes and how they work,

00:30:22.520 | your prior models and what they are,

00:30:25.360 | your values and your assumptions,

00:30:27.200 | all those things now come together into the interpretation.

00:30:30.640 | Getting in sync of that is hard.

00:30:33.800 | - And yet humans are able to intuit some of that

00:30:37.640 | without any pre--

00:30:39.600 | - Because they have the shared experience.

00:30:41.560 | - And we're not talking about shared,

00:30:42.960 | two people having a shared experience.

00:30:44.440 | I mean, as a society-- - That's correct.

00:30:46.600 | - We have the shared experience and we have similar brains.

00:30:51.240 | So we tend to, in other words,

00:30:54.120 | part of our shared experience

00:30:55.160 | is our shared local experience.

00:30:56.520 | Like we may live in the same culture,

00:30:57.920 | we may live in the same society,

00:30:59.120 | and therefore we have similar educations.

00:31:02.080 | We have similar, what we like to call prior models

00:31:04.160 | about the prior experiences.

00:31:05.920 | And we use that as a,

00:31:07.400 | think of it as a wide collection of interrelated variables

00:31:11.000 | and they're all bound to similar things.

00:31:12.840 | And so we take that as our background

00:31:15.120 | and we start interpreting things similarly.

00:31:17.560 | But as humans, we have a lot of shared experience.

00:31:21.840 | We do have similar brains, similar goals,

00:31:24.960 | similar emotions under similar circumstances

00:31:28.080 | because we're both humans.

00:31:29.040 | So now one of the early questions you asked,

00:31:31.400 | how is biological and computer information systems

00:31:36.400 | fundamentally different?

00:31:38.000 | Well, one is humans come with a lot of pre-programmed stuff,

00:31:43.840 | a ton of program stuff,

00:31:45.960 | and they're able to communicate

00:31:47.240 | because they have a lot of,

00:31:48.360 | because they share that stuff.

00:31:50.360 | - Do you think that shared knowledge,

00:31:52.700 | if we can maybe escape the hardware question,

00:31:57.560 | how much is encoded in the hardware,

00:31:59.440 | just the shared knowledge in the software,

00:32:01.200 | the history, the many centuries of wars and so on

00:32:05.280 | that came to today, that shared knowledge,

00:32:09.800 | how hard is it to encode?

00:32:13.400 | Do you have a hope?

00:32:15.840 | Can you speak to how hard is it to encode that knowledge

00:32:19.360 | systematically in a way that could be used by a computer?

00:32:22.800 | - So I think it is possible to learn for a machine,

00:32:26.340 | to program a machine to acquire that knowledge

00:32:29.600 | with a similar foundation.

00:32:31.440 | In other words, a similar interpretive foundation

00:32:36.120 | for processing that knowledge.

00:32:38.040 | - What do you mean by that?

00:32:39.080 | - So in other words,

00:32:40.800 | we view the world in a particular way.

00:32:44.360 | And so in other words, we have, if you will,

00:32:48.800 | as humans, we have a framework

00:32:50.080 | for interpreting the world around us.

00:32:52.200 | So we have multiple frameworks

00:32:54.720 | for interpreting the world around us.

00:32:55.920 | But if you're interpreting, for example,

00:32:59.720 | sociopolitical interactions,

00:33:01.320 | you're thinking about where there's people,

00:33:03.080 | there's collections and groups of people,

00:33:05.520 | they have goals, goals are largely built around survival

00:33:08.400 | and quality of life.

00:33:10.040 | There are fundamental economics

00:33:13.280 | around scarcity of resources.

00:33:16.600 | And when humans come and start interpreting

00:33:19.600 | a situation like that,

00:33:20.640 | because you brought up like historical events,

00:33:23.560 | they start interpreting situations like that.

00:33:25.480 | They apply a lot of this fundamental framework

00:33:29.480 | for interpreting that.

00:33:30.720 | Well, who are the people?

00:33:32.200 | What were their goals?

00:33:33.280 | What resources did they have?

00:33:35.000 | How much power or influence did they have over the other?

00:33:37.000 | Like this fundamental substrate, if you will,

00:33:40.520 | for interpreting and reasoning about that.

00:33:42.760 | So I think it is possible to imbue a computer

00:33:46.880 | with that stuff that humans like take for granted

00:33:50.600 | when they go and sit down and try to interpret things.

00:33:54.280 | And then with that foundation, they acquire,

00:33:58.800 | they start acquiring the details,

00:34:00.280 | the specifics in a given situation,

00:34:02.800 | are then able to interpret it with regard to that framework.

00:34:05.680 | And then given that interpretation, they can do what?

00:34:08.680 | They can predict.

00:34:10.320 | But not only can they predict,

00:34:12.200 | they can predict now with an explanation

00:34:14.800 | that can be given in those terms,

00:34:17.920 | in the terms of that underlying framework

00:34:20.200 | that most humans share.

00:34:22.320 | Now you can find humans that come and interpret events

00:34:24.640 | very differently than other humans,

00:34:26.320 | because they're like using a different framework.

00:34:30.160 | You know, the movie "Matrix" comes to mind,

00:34:32.480 | where they decided humans were really just batteries,

00:34:36.360 | and that's how they interpreted the value of humans

00:34:39.880 | as a source of electrical energy.

00:34:41.600 | So, but I think that for the most part,

00:34:45.400 | we have a way of interpreting the events,

00:34:50.400 | or the social events around us,

00:34:52.240 | because we have this shared framework.

00:34:54.120 | It comes from, again, the fact that we're similar beings

00:34:58.680 | that have similar goals, similar emotions,

00:35:01.040 | and we can make sense out of these.

00:35:02.880 | These frameworks make sense to us.

00:35:04.960 | - So how much knowledge is there, do you think?

00:35:08.040 | So you said it's possible.

00:35:09.560 | - Well, there's a tremendous amount

00:35:10.600 | of detailed knowledge in the world.

00:35:12.840 | You can imagine effectively infinite number

00:35:17.600 | of unique situations and unique configurations

00:35:20.840 | of these things.

00:35:22.120 | But the knowledge that you need,

00:35:25.120 | what I refer to as like the frameworks,

00:35:27.600 | for you need for interpreting them, I don't think.

00:35:29.560 | I think those are finite.

00:35:31.480 | - You think the frameworks are more important

00:35:35.000 | than the bulk of the knowledge?

00:35:36.760 | So like framing--

00:35:37.760 | - Yeah, because what the frameworks do

00:35:39.200 | is they give you now the ability to interpret and reason,

00:35:41.560 | and to interpret and reason over the specifics

00:35:46.560 | in ways that other humans would understand.

00:35:49.240 | - What about the specifics?

00:35:51.240 | - Well, you acquire the specifics by reading

00:35:53.960 | and by talking to other people.

00:35:55.560 | - So I'll mostly, actually, just even,

00:35:57.720 | if we can focus on even the beginning,

00:36:00.240 | the common sense stuff,

00:36:01.480 | the stuff that doesn't even require reading,

00:36:03.400 | or it almost requires playing around

00:36:06.280 | with the world or something.

00:36:07.720 | Just being able to sort of manipulate objects,

00:36:10.800 | drink water and so on, all of that.

00:36:13.880 | Every time we try to do that kind of thing

00:36:16.120 | in robotics or AI, it seems to be like an onion.

00:36:21.040 | You seem to realize how much knowledge

00:36:23.240 | is really required to perform

00:36:24.620 | even some of these basic tasks.

00:36:27.040 | Do you have that sense as well?

00:36:30.320 | And if so, how do we get all those details?

00:36:33.800 | Are they written down somewhere?

00:36:35.680 | Do they have to be learned through experience?

00:36:39.200 | - So I think when, like if you're talking about

00:36:41.320 | sort of the physics, the basic physics around us,

00:36:44.680 | for example, acquiring information about,

00:36:46.600 | acquiring how that works.

00:36:48.120 | Yeah, I mean, I think there's a combination of things going,

00:36:52.200 | I think there's a combination of things going on.

00:36:54.600 | I think there is like fundamental pattern matching,

00:36:57.800 | like what we were talking about before,

00:36:59.680 | where you see enough examples,

00:37:01.040 | enough data about something,

00:37:02.160 | you just start assuming that,

00:37:03.840 | and with similar input, I'm gonna predict similar outputs.

00:37:07.720 | You can't necessarily explain it at all.

00:37:10.120 | You may learn very quickly that when you let something go,

00:37:13.620 | it falls to the ground.

00:37:15.920 | - That's such a--

00:37:17.840 | - But you can't necessarily explain that.

00:37:19.760 | - But that's such a deep idea,

00:37:22.320 | that if you let something go,

00:37:23.880 | like the idea of gravity.

00:37:25.200 | - I mean, people are letting things go

00:37:27.920 | and counting on them falling

00:37:29.080 | well before they understood gravity.

00:37:30.760 | - But that seems to be, that's exactly what I mean,

00:37:33.840 | is before you take a physics class

00:37:36.080 | or study anything about Newton,

00:37:39.560 | just the idea that stuff falls to the ground

00:37:42.520 | and then you be able to generalize

00:37:45.320 | that all kinds of stuff falls to the ground,

00:37:48.540 | it just seems like a non,

00:37:52.200 | without encoding it, like hard coding it in,

00:37:55.200 | it seems like a difficult thing to pick up.

00:37:57.400 | It seems like you have to have a lot of different knowledge

00:38:01.380 | to be able to integrate that into the framework,

00:38:05.360 | sort of into everything else.

00:38:07.760 | So both know that stuff falls to the ground

00:38:10.360 | and start to reason about sociopolitical discourse.

00:38:15.360 | So both, like the very basic

00:38:18.600 | and the high level reasoning decision-making.

00:38:22.560 | I guess my question is how hard is this problem?

00:38:24.960 | Sorry to linger on it because again,

00:38:29.040 | and we'll get to it for sure,

00:38:31.120 | as what Watson with Jeopardy did,

00:38:33.000 | is take on a problem that's much more constrained

00:38:35.480 | but has the same hugeness of scale,

00:38:38.240 | at least from the outsider's perspective.

00:38:40.640 | So I'm asking the general life question

00:38:42.880 | of to be able to be an intelligent being

00:38:45.600 | and reasoning in the world

00:38:47.520 | about both gravity and politics,

00:38:50.880 | how hard is that problem?

00:38:52.120 | - So I think it's solvable.

00:38:56.180 | - Okay, now beautiful.

00:39:00.680 | So what about time travel?

00:39:04.320 | Okay, on that topic, I'm just,

00:39:07.960 | not the same answer? - Not as convinced.

00:39:09.680 | - Not as convinced yet, okay.

00:39:11.040 | - No, I think it is, I think it is solvable.

00:39:14.240 | I mean, I think that it's,

00:39:16.440 | first of all, it's about getting machines to learn.

00:39:18.400 | Learning is fundamental.

00:39:21.360 | And I think we're already in a place that we understand,

00:39:24.400 | for example, how machines can learn in various ways.

00:39:28.560 | Right now, our learning stuff is sort of primitive

00:39:32.420 | in that we haven't sort of taught machines

00:39:37.420 | to learn the frameworks.

00:39:39.200 | We don't communicate our frameworks

00:39:41.120 | because of how shared, in some cases we do,

00:39:42.820 | but we don't annotate, if you will,

00:39:46.360 | all the data in the world with the frameworks

00:39:48.920 | that are inherent or underlying our understanding.

00:39:53.080 | Instead, we just operate with the data.

00:39:56.140 | So if we wanna be able to reason over the data

00:39:59.060 | in similar terms in the common frameworks,

00:40:02.280 | we need to be able to teach the computer,

00:40:03.720 | or at least we need to program the computer to acquire,

00:40:07.560 | to have access to and acquire, learn the frameworks as well

00:40:12.560 | and connect the frameworks to the data.

00:40:15.700 | I think this can be done.

00:40:18.400 | I think we can start, I think machine learning,

00:40:22.920 | for example, with enough examples,

00:40:26.040 | can start to learn these basic dynamics.

00:40:28.880 | Will they relate them necessarily to gravity?

00:40:32.200 | Not unless they can also acquire those theories as well

00:40:37.080 | and put the experiential knowledge

00:40:40.880 | and connect it back to the theoretical knowledge.

00:40:43.360 | I think if we think in terms of these class of architectures

00:40:47.160 | that are designed to both learn the specifics,

00:40:51.000 | find the patterns, but also acquire the frameworks

00:40:54.180 | and connect the data to the frameworks,

00:40:56.300 | if we think in terms of robust architectures like this,

00:40:59.660 | I think there is a path toward getting there.

00:41:03.380 | - In terms of encoding architectures like that,

00:41:06.180 | do you think systems that are able to do this

00:41:09.180 | will look like neural networks

00:41:12.020 | or representing, if you look back to the '80s and '90s,

00:41:17.020 | with the expert systems,

00:41:18.680 | so more like graphs, systems that are based in logic,

00:41:23.540 | able to contain a large amount of knowledge

00:41:26.500 | where the challenge was the automated acquisition

00:41:28.500 | of that knowledge.

00:41:29.860 | I guess the question is,

00:41:31.840 | when you collect both the frameworks

00:41:33.820 | and the knowledge from the data,

00:41:35.300 | what do you think that thing will look like?

00:41:37.260 | - Yeah, so I mean, I think asking the question

00:41:39.340 | do they look like neural networks is a bit of a red herring.

00:41:41.260 | I mean, I think that they will certainly do inductive

00:41:45.180 | or pattern-matched-based reasoning.

00:41:46.740 | And I've already experimented with architectures

00:41:49.020 | that combine both, that use machine learning

00:41:52.740 | and neural networks to learn certain classes of knowledge,

00:41:55.380 | in other words, to find repeated patterns

00:41:57.340 | in order for it to make good inductive guesses,

00:42:01.540 | but then ultimately to try to take those learnings

00:42:05.300 | and marry them, in other words, connect them to frameworks

00:42:09.560 | so that it can then reason over that

00:42:11.520 | in terms other humans understand.

00:42:13.660 | So for example, at Elemental Cognition, we do both.

00:42:16.140 | We have architectures that do both, but both those things,

00:42:19.860 | but also have a learning method

00:42:21.700 | for acquiring the frameworks themselves and saying,

00:42:24.380 | "Look, ultimately I need to take this data.

00:42:27.260 | "I need to interpret it in the form of these frameworks

00:42:29.980 | "so they can reason over it."

00:42:30.900 | So there is a fundamental knowledge representation,

00:42:33.380 | like what you're saying, like these graphs of logic,

00:42:35.820 | if you will.

00:42:36.820 | There are also neural networks

00:42:39.320 | that acquire a certain class of information.

00:42:41.660 | Then they align them with these frameworks,

00:42:45.960 | but there's also a mechanism

00:42:47.200 | to acquire the frameworks themselves.

00:42:49.240 | - Yeah, so it seems like the idea of frameworks

00:42:52.600 | requires some kind of collaboration with humans.

00:42:55.440 | - Absolutely.

00:42:56.360 | - So do you think of that collaboration as direct?

00:42:59.360 | - Well, and let's be clear.

00:43:01.960 | Only for the express purpose

00:43:04.400 | that you're designing an intelligence

00:43:09.400 | that can ultimately communicate with humans

00:43:12.060 | in the terms of frameworks that help them understand things.

00:43:15.980 | So to be really clear, you can independently create

00:43:22.500 | a machine learning system, an intelligence

00:43:26.440 | that I might call an alien intelligence

00:43:28.500 | that does a better job than you with some things,

00:43:31.180 | but can't explain the framework to you.

00:43:33.540 | That doesn't mean it might be better than you at the thing.

00:43:36.760 | It might be that you cannot comprehend the framework

00:43:39.540 | that it may have created for itself

00:43:41.400 | that is inexplicable to you.

00:43:43.960 | That's a reality.

00:43:45.320 | - But you're more interested in a case where you can.

00:43:48.820 | - I am, yeah.

00:43:51.080 | My sort of approach to AI

00:43:53.120 | is because I've set the goal for myself.

00:43:55.920 | I want machines to be able to ultimately communicate

00:43:58.800 | understanding with humans.

00:44:01.440 | I want them to be able to acquire and communicate,

00:44:03.520 | acquire knowledge from humans

00:44:04.780 | and communicate knowledge to humans.

00:44:07.060 | They should be using what, you know,

00:44:10.340 | inductive machine learning techniques are good at,

00:44:13.760 | which is to observe patterns of data,

00:44:16.860 | whether it be in language or whether it be in images

00:44:19.340 | or videos or whatever, to acquire these patterns,

00:44:24.340 | to induce the generalizations from those patterns,

00:44:29.420 | but then ultimately work with humans

00:44:31.300 | to connect them to frameworks, interpretations, if you will,

00:44:34.680 | that ultimately make sense to humans.

00:44:36.720 | Of course, the machine is gonna have the strength

00:44:38.460 | that it has, the richer, longer memory,

00:44:41.460 | but that, you know, it has the more rigorous

00:44:44.340 | reasoning abilities, the deeper reasoning abilities.

00:44:47.080 | So it'll be an interesting, you know,

00:44:49.220 | complementary relationship between the human and the machine.

00:44:53.220 | - Do you think that ultimately needs explainability

00:44:55.140 | like a machine?

00:44:56.020 | So if we look, we study, for example,

00:44:57.900 | Tesla autopilot a lot, where humans,

00:45:00.860 | I don't know if you've driven the vehicle,

00:45:02.100 | are aware of what, so you're basically,

00:45:06.460 | the human and machine are working together there,

00:45:10.340 | and the human is responsible for their own life

00:45:12.500 | to monitor the system, and you know,

00:45:15.340 | the system fails every few miles.

00:45:18.420 | And so there's hundreds, there's millions

00:45:21.100 | of those failures a day.

00:45:23.660 | And so that's like a moment of interaction.

00:45:25.780 | Do you see--

00:45:26.620 | - Yeah, no, that's exactly right.

00:45:27.900 | That's a moment of interaction where, you know,

00:45:31.980 | the machine has learned some stuff,

00:45:33.900 | it has a failure, somehow the failure's communicated,

00:45:38.740 | the human is now filling in the mistake, if you will,

00:45:41.900 | or maybe correcting or doing something

00:45:43.640 | that is more successful in that case,

00:45:45.880 | the computer takes that learning.

00:45:47.900 | So I believe that the collaboration

00:45:50.260 | between human and machine, I mean,

00:45:52.540 | that's sort of a primitive example

00:45:53.900 | and sort of a more, another example

00:45:57.820 | is where the machine's literally talking to you

00:45:59.500 | and saying, "Look, I'm reading this thing.

00:46:02.700 | "I know that like the next word might be this or that,

00:46:06.520 | "but I don't really understand why.

00:46:08.840 | "I have my guess, can you help me understand

00:46:11.000 | "the framework that supports this?"

00:46:14.100 | And then can kind of acquire that,

00:46:16.080 | take that and reason about it and reuse it

00:46:18.180 | the next time it's reading to try to understand something.

00:46:20.540 | Not unlike a human student might do.

00:46:24.780 | I mean, I remember when my daughter was in first grade

00:46:27.500 | and she had a reading assignment about electricity

00:46:31.180 | and somewhere in the text it says,

00:46:35.540 | "And electricity is produced by water flowing over turbines,"

00:46:38.660 | or something like that.

00:46:39.780 | And then there's a question that says,

00:46:41.180 | "Well, how is electricity created?"

00:46:43.180 | And so my daughter comes to me and says,

00:46:45.140 | "I mean, I could, you know, created and produced

00:46:47.400 | "are kind of synonyms in this case,

00:46:49.140 | "so I can go back to the text and I can copy

00:46:51.780 | "by water flowing over turbines."

00:46:53.700 | But I have no idea what that means.

00:46:56.140 | Like, I don't know how to interpret

00:46:57.640 | water flowing over turbines and what electricity even is.

00:47:00.400 | I mean, I can get the answer right by matching the text,

00:47:04.020 | but I don't have any framework for understanding

00:47:06.160 | what this means at all.

00:47:07.900 | - And framework really is, I mean, it's a set of,

00:47:10.540 | not to be mathematical, but axioms of ideas

00:47:14.180 | that you bring to the table and interpreting stuff

00:47:16.380 | and then you build those up somehow.

00:47:18.300 | You build them up with the expectation

00:47:20.460 | that there's a shared understanding of what they are.

00:47:23.780 | - Sure, yeah, it's the social, us humans.

00:47:27.800 | Do you have a sense that humans on Earth in general

00:47:32.060 | share a set of, like how many frameworks are there?

00:47:36.500 | - I mean, it depends on how you bound them, right?

00:47:38.200 | So in other words, how big or small

00:47:39.900 | like their individual scope.

00:47:41.780 | But there's lots and there are new ones.

00:47:44.220 | I think the way I think about it is kind of in a layer.

00:47:47.620 | I think of the architectures as being layered in that.

00:47:50.060 | There's a small set of primitives

00:47:53.580 | that allow you the foundation to build frameworks.

00:47:56.300 | And then there may be many frameworks,

00:47:58.400 | but you have the ability to acquire them.

00:48:00.620 | And then you have the ability to reuse them.

00:48:03.060 | I mean, one of the most compelling ways of thinking

00:48:05.340 | about this is a reasoning by analogy where I can say,

00:48:07.820 | oh, wow, I've learned something very similar.

00:48:10.060 | I never heard of this game, soccer,

00:48:15.280 | but if it's like basketball in the sense

00:48:17.860 | that the goal's like the hoop

00:48:19.620 | and I have to get the ball in the hoop

00:48:21.020 | and I have guards and I have this and I have that,

00:48:23.520 | like where are the similarities and where are the differences

00:48:27.860 | and I have a foundation now

00:48:29.140 | for interpreting this new information.

00:48:31.380 | - And then the different groups,

00:48:33.300 | like the millennials will have a framework.

00:48:35.760 | - Yeah, well, like that--

00:48:39.700 | - The Democrats and Republicans.

00:48:41.580 | Millennials, nobody wants that framework.

00:48:43.860 | - Well, I mean, I think--

00:48:44.700 | - Nobody understands it.

00:48:45.580 | - Right, I mean, they're talking about political

00:48:47.100 | and social ways of interpreting the world around them.

00:48:49.860 | And I think these frameworks

00:48:51.060 | are still largely, largely similar.

00:48:52.700 | I think they differ in maybe what some fundamental

00:48:55.420 | assumptions and values are.

00:48:57.340 | Now, from a reasoning perspective,

00:48:59.820 | like the ability to process the framework

00:49:02.500 | might not be that different.

00:49:04.120 | The implications of different fundamental values

00:49:06.520 | or fundamental assumptions in those frameworks

00:49:09.420 | may reach very different conclusions.

00:49:12.120 | So from a social perspective,

00:49:14.760 | the conclusions may be very different.

00:49:16.860 | From an intelligence perspective,

00:49:18.360 | I just followed where my assumptions took me.

00:49:21.580 | - Yeah, the process itself will look similar,

00:49:23.400 | but that's a fascinating idea

00:49:25.540 | that frameworks really help carve

00:49:30.540 | how a statement will be interpreted.

00:49:33.700 | I mean, having a Democrat and a Republican framework

00:49:40.320 | and then read the exact same statement

00:49:42.140 | and the conclusions that you derive

00:49:44.160 | will be totally different from an AI perspective

00:49:46.540 | is fascinating.

00:49:47.580 | - What we would want out of the AI

00:49:49.420 | is to be able to tell you that this perspective,

00:49:52.500 | one perspective, one set of assumptions

00:49:54.700 | is gonna lead you here,

00:49:55.540 | another set of assumptions is gonna lead you there.

00:49:58.660 | And in fact, to help people reason and say,

00:50:01.380 | oh, I see where our differences lie.

00:50:05.060 | You know, I have this fundamental belief about that,

00:50:06.900 | I have this fundamental belief about that.

00:50:09.160 | - Yeah, that's quite brilliant.

00:50:10.060 | - From my perspective, NLP,

00:50:12.620 | there's this idea that there's one way

00:50:14.140 | to really understand a statement,

00:50:16.100 | but there probably isn't.

00:50:18.780 | There's probably an infinite number of ways

00:50:20.140 | to understand a statement, depending on the framework.

00:50:21.900 | - There's lots of different interpretations

00:50:23.460 | and the broader the content, the richer it is.

00:50:28.460 | And so, you and I can have very different experiences

00:50:35.300 | with the same text, obviously.

00:50:37.460 | And if we're committed to understanding each other,

00:50:41.360 | we start, and that's the other important point,

00:50:45.300 | if we're committed to understanding each other,

00:50:47.780 | we start decomposing and breaking down our interpretation

00:50:51.900 | to its more and more primitive components

00:50:54.060 | until we get to that point where we say,

00:50:55.940 | oh, I see why we disagree.

00:50:58.300 | And we try to understand how fundamental

00:51:00.540 | that disagreement really is.

00:51:02.260 | But that requires a commitment

00:51:04.620 | to breaking down that interpretation

00:51:06.580 | in terms of that framework in a logical way.

00:51:08.980 | Otherwise, and this is why I think of AI

00:51:12.300 | as really complementing and helping human intelligence

00:51:16.040 | to overcome some of its biases and its predisposition

00:51:19.900 | to be persuaded by more shallow reasoning,

00:51:24.900 | in the sense that we get over this idea,

00:51:27.020 | well, I'm right because I'm Republican,

00:51:30.020 | or I'm right because I'm Democratic,

00:51:31.420 | and someone labeled this as a Democratic point of view,

00:51:33.380 | or it has the following keywords in it.

00:51:35.420 | And if the machine can help us break that argument down

00:51:38.500 | and say, wait a second,

00:51:40.540 | what do you really think about this?

00:51:42.260 | So essentially holding us accountable

00:51:45.460 | to doing more critical thinking.

00:51:47.540 | - We're gonna have to sit and think about that.

00:51:50.300 | I love that.

00:51:51.140 | I think that's really empowering use of AI

00:51:53.580 | for the public discourse

00:51:54.860 | that's completely disintegrating currently

00:51:58.580 | as we learn how to do it on social media.

00:52:00.420 | - That's right.

00:52:02.460 | - So one of the greatest accomplishments

00:52:05.860 | in the history of AI is Watson

00:52:10.860 | competing in the game of Jeopardy against humans.

00:52:13.820 | And you were a lead in that, a critical part of that.

00:52:18.940 | Let's start at the very basics.

00:52:20.580 | What is the game of Jeopardy?

00:52:22.820 | The game for us humans, human versus human.

00:52:25.820 | - Right, so it's to take a question

00:52:30.840 | and answer it.

00:52:32.180 | The game of Jeopardy.

00:52:34.700 | - It's just the opposite.

00:52:35.540 | - Actually, well, no, but it's not, right?

00:52:38.780 | It's really not.

00:52:39.620 | It's really to get a question and answer,

00:52:41.820 | but it's what we call a factoid question.

00:52:43.900 | So this notion of like,

00:52:45.460 | it really relates to some fact

00:52:46.820 | that few people would argue

00:52:49.260 | whether the facts are true or not.

00:52:50.540 | In fact, most people wouldn't.

00:52:51.580 | Jeopardy kind of counts on the idea

00:52:53.020 | that these statements have factual answers.

00:52:57.660 | And the idea is to, first of all,

00:53:02.000 | determine whether or not you know the answer,

00:53:03.760 | which is sort of an interesting twist.

00:53:06.080 | - So first of all, understand the question.

00:53:07.880 | - You have to understand the question.

00:53:08.840 | What is it asking?

00:53:09.860 | And that's a good point

00:53:10.760 | because the questions are not asked directly, right?

00:53:14.440 | - They're all like, the way the questions are asked

00:53:16.880 | is nonlinear.

00:53:18.340 | It's like, it's a little bit witty.

00:53:20.680 | It's a little bit playful sometimes.

00:53:22.480 | It's a little bit tricky.

00:53:25.960 | - Yeah, they're asked in, exactly,

00:53:27.940 | in numerous, witty, tricky ways.

00:53:30.620 | Exactly what they're asking is not obvious.

00:53:32.580 | It takes inexperienced humans a while to go,

00:53:35.380 | what is it even asking?

00:53:36.940 | And that's sort of an interesting realization

00:53:39.140 | that you have when somebody says,

00:53:40.180 | oh, Jeopardy is a question answering show.

00:53:42.460 | And then it's like, oh, I know a lot.

00:53:43.900 | And then you read it,

00:53:44.740 | and you're still trying to process the question,

00:53:47.020 | and the champions have answered and moved on.

00:53:49.140 | There are three questions ahead

00:53:51.180 | by the time you've figured out

00:53:52.740 | what the question even meant.

00:53:54.100 | So there's definitely an ability there

00:53:56.480 | to just parse out what the question even is.

00:53:59.520 | So that was certainly challenging.

00:54:00.860 | It's interesting, historically, though,

00:54:02.240 | if you look back at the Jeopardy games much earlier.

00:54:06.200 | - Like '60s, '70s, that kind of thing?

00:54:08.160 | - The questions were much more direct.

00:54:10.200 | They weren't quite like that.

00:54:11.340 | They got sort of more and more interesting.

00:54:13.720 | The way they asked them,

00:54:14.700 | that sort of got more and more interesting,

00:54:16.480 | and subtle, and nuanced, and humorous, and witty over time,

00:54:20.840 | which really required the human

00:54:22.560 | to kind of make the right connections

00:54:24.280 | in figuring out what the question was even asking.

00:54:26.880 | So yeah, you have to figure out the questions even asking.

00:54:29.960 | Then you have to determine whether or not

00:54:32.180 | you think you know the answer.

00:54:34.520 | And because you have to buzz in really quickly,

00:54:37.380 | you sort of have to make that determination

00:54:39.800 | as quickly as you possibly can.

00:54:41.200 | Otherwise, you lose the opportunity to buzz in.

00:54:43.440 | - Even before you really know if you know the answer.

00:54:46.520 | - I think a lot of humans will assume

00:54:49.080 | they'll look at it, process it very superficially.

00:54:53.000 | In other words, what's the topic, what are some keywords,

00:54:56.000 | and just say, do I know this area or not

00:54:58.680 | before they actually know the answer?

00:55:00.840 | Then they'll buzz in and think about it.

00:55:03.240 | So it's interesting what humans do.

00:55:04.680 | Now, some people who know all things,

00:55:06.960 | like Ken Jennings or something,

00:55:08.480 | or the more recent Big Jeopardy player,

00:55:10.440 | I mean, they'll just buzz in.

00:55:12.440 | They'll just assume they know all about Jeopardy,

00:55:14.120 | and they'll just buzz in.

00:55:15.920 | Watson, interestingly, didn't even come close

00:55:18.360 | to knowing all of Jeopardy, right?

00:55:20.120 | Watson really-- - Even at the peak,

00:55:21.800 | even at its best. - Yeah, so for example,

00:55:23.800 | I mean, we had this thing called recall,

00:55:25.960 | which is like how many of all the Jeopardy questions,

00:55:29.400 | how many could we even find the right answer for,

00:55:33.440 | like anywhere?

00:55:34.440 | Like, could we come up with, if we had a big body of knowledge

00:55:38.080 | of some of the order of several terabytes,

00:55:39.800 | I mean, from a web scale, it was actually very small.

00:55:42.920 | But from a book scale,

00:55:44.360 | I was talking about millions of books, right?

00:55:46.320 | So they're calling millions of books,

00:55:48.320 | encyclopedias, dictionaries, books,

00:55:50.040 | so it's still a ton of information.

00:55:52.280 | And for, I think it was only 85%

00:55:55.200 | was the answer anywhere to be found.

00:55:57.600 | So you're already down at that level

00:56:00.360 | just to get started, right?

00:56:02.080 | So, and so it was important to get a very quick sense of,

00:56:07.080 | do you think you know the right answer to this question?

00:56:10.080 | So we had to compute that confidence

00:56:12.200 | as quickly as we possibly could.

00:56:14.320 | So in effect, we had to answer it

00:56:16.480 | and at least spend some time essentially answering it

00:56:21.480 | and then judging the confidence

00:56:23.760 | that our answer was right

00:56:26.680 | and then deciding whether or not

00:56:28.080 | we were confident enough to buzz in.

00:56:30.040 | And that would depend on what else was going on in the game

00:56:31.920 | because there was a risk.

00:56:33.440 | So like, if you're really in a situation

00:56:35.120 | where I have to take a guess, I have very little to lose,

00:56:38.380 | then you'll buzz in with less confidence.

00:56:40.280 | - So that was accounted for the financial standings

00:56:42.960 | of the different competitors.

00:56:44.320 | - Correct.

00:56:45.160 | How much of the game was laughed,

00:56:46.640 | how much time was laughed,

00:56:48.280 | where you were in the standing and things like that.

00:56:50.800 | - How many hundreds of milliseconds

00:56:52.880 | that we're talking about here?

00:56:53.960 | Do you have a sense of what is,

00:56:56.000 | like if it's, what's the target?

00:56:58.440 | - So, I mean, we targeted answering in under three seconds

00:57:03.440 | and--

00:57:04.720 | - Buzzing in, so the decision to buzz in

00:57:08.400 | and then the actual answering,

00:57:10.000 | are those two different stages?

00:57:10.840 | - Yeah, they were two different things.

00:57:12.680 | In fact, we had multiple stages,

00:57:14.520 | whereas like we would say, let's estimate our confidence,

00:57:17.400 | which was sort of a shallow answering process.

00:57:21.040 | And then ultimately decide to buzz in,

00:57:23.840 | and then we may take another second or something

00:57:26.360 | to kind of go in there and do that.

00:57:30.880 | But by and large, we're saying like, we can't play the game.

00:57:33.920 | We can't even compete if we can't, on average,

00:57:37.600 | answer these questions in around three seconds or less.

00:57:40.360 | - So you stepped in,

00:57:41.720 | so there's these three humans playing a game,

00:57:45.320 | and you stepped in with the idea that IBM Watson

00:57:48.000 | would be one of, replace one of the humans

00:57:49.960 | and compete against two.

00:57:52.000 | Can you tell the story of Watson taking on this game?

00:57:56.760 | - Sure.

00:57:57.600 | - It seems exceptionally difficult.

00:57:58.720 | - Yeah, so the story was that it was coming up,

00:58:03.520 | I think, to the 10-year anniversary of Big Blue.

00:58:06.960 | Not Big Blue, Deep Blue.

00:58:08.800 | IBM wanted to do sort of another kind of really,

00:58:11.960 | fun challenge, public challenge

00:58:14.200 | that can bring attention to IBM research

00:58:16.400 | and the kind of the cool stuff that we were doing.

00:58:18.640 | I had been working in AI at IBM for some time.

00:58:23.760 | I had a team doing what's called

00:58:26.520 | open domain factoid question answering,

00:58:28.640 | which is, we're not gonna tell you what the questions are,

00:58:31.040 | we're not even gonna tell you what they're about.

00:58:33.120 | Can you go off and get accurate answers to these questions?

00:58:36.840 | And it was an area of AI research that I was involved in.

00:58:41.400 | And so it was a very specific passion of mine.

00:58:44.280 | Language understanding had always been a passion of mine.

00:58:47.080 | One sort of narrow slice on whether or not

00:58:49.600 | you could do anything with language

00:58:50.960 | was this notion of open domain,

00:58:52.560 | meaning I could ask anything about anything.

00:58:54.560 | Factoid, meaning it essentially had an answer,

00:58:57.840 | and being able to do that accurately and quickly.

00:59:00.920 | So that was a research area that my team had already been in.

00:59:03.960 | And so completely independently,

00:59:06.320 | several IBM executives were like, what are we gonna do?

00:59:09.080 | What's the next cool thing to do?

00:59:11.060 | And Ken Jennings was on his winning streak.

00:59:13.920 | This was like, whatever it was, 2004, I think,

00:59:16.640 | was on his winning streak.

00:59:18.760 | And someone thought, hey, that would be really cool

00:59:20.920 | if the computer can play Jeopardy.

00:59:23.960 | And so this was like in 2004,

00:59:25.760 | they were shopping this thing around.

00:59:28.040 | And everyone was telling the research execs, no way.

00:59:33.560 | Like, this is crazy.

00:59:35.200 | And we had some pretty senior people in the field

00:59:37.080 | and they're saying, no, this is crazy.

00:59:38.200 | And it would come across my desk and I was like,

00:59:40.240 | but that's kind of what I'm really interested in doing.

00:59:43.140 | But there was such this prevailing sense of,

00:59:46.800 | this is nuts, we're not gonna risk IBM's reputation on this,

00:59:49.440 | we're just not doing it.

00:59:50.280 | And this happened in 2004, it happened in 2005.

00:59:53.200 | At the end of 2006, it was coming around again.

00:59:58.200 | And I was coming off of a,

01:00:01.120 | I was doing the open domain question answering stuff,

01:00:03.120 | but I was coming off a couple other projects.

01:00:06.000 | I had a lot more time to put into this.

01:00:08.080 | And I argued that it could be done.

01:00:10.240 | And I argued it would be crazy not to do this.

01:00:12.800 | - Can I, you can be honest at this point.

01:00:15.880 | So even though you argued for it,

01:00:17.640 | what's the confidence that you had yourself, privately,

01:00:21.560 | that this could be done?

01:00:22.800 | Was, we just told the story,

01:00:25.680 | how you tell stories to convince others.

01:00:27.800 | How confident were you?

01:00:29.000 | What was your estimation of the problem at that time?

01:00:32.720 | - So I thought it was possible.

01:00:34.360 | And a lot of people thought it was impossible.

01:00:36.360 | I thought it was possible.

01:00:37.920 | A reason why I thought it was possible

01:00:39.200 | was because I did some brief experimentation.

01:00:41.560 | I knew a lot about how we were approaching

01:00:43.520 | open domain factoid question answering.

01:00:46.000 | We've been doing it for some years.

01:00:47.680 | I looked at the Jeopardy stuff.

01:00:49.400 | I said, this is gonna be hard

01:00:50.960 | for a lot of the points that we mentioned earlier.

01:00:54.200 | Hard to interpret the question,

01:00:55.760 | hard to do it quickly enough,

01:00:58.960 | hard to compute an accurate confidence.

01:01:00.560 | None of this stuff had been done well enough before.

01:01:03.120 | But a lot of the technologies we're building

01:01:04.720 | were the kinds of technologies that should work.

01:01:07.520 | But more to the point, what was driving me was,

01:01:10.880 | I was in IBM Research.

01:01:12.840 | I was a senior leader in IBM Research.

01:01:14.920 | And this is the kind of stuff we were supposed to do.

01:01:17.160 | In other words, we were basically supposed to--

01:01:18.720 | - This is the moonshot.

01:01:19.720 | This is the--

01:01:20.560 | - I mean, we were supposed to take things and say,

01:01:21.960 | this is an active research area.

01:01:24.120 | It's our obligation to kind of,

01:01:27.560 | if we have the opportunity, to push it to the limits.

01:01:30.160 | And if it doesn't work, to understand more deeply

01:01:32.840 | why we can't do it.

01:01:34.800 | And so I was very committed to that notion,

01:01:37.440 | saying, folks, this is what we do.

01:01:40.080 | It's crazy not to do this.

01:01:42.200 | This is an active research area.

01:01:43.760 | We've been in this for years.

01:01:45.000 | Why wouldn't we take this grand challenge

01:01:47.480 | and push it as hard as we can?

01:01:50.720 | At the very least, we'd be able to come out and say,

01:01:53.240 | here's why this problem is way hard.

01:01:57.080 | Here's what we've tried and here's how we failed.

01:01:58.700 | So I was very driven as a scientist from that perspective.

01:02:03.700 | And then I also argued,

01:02:06.640 | based on what we did a feasibility study,

01:02:08.640 | of why I thought it was hard but possible.

01:02:10.960 | And I showed examples of where it succeeded,

01:02:14.200 | where it failed, why it failed,

01:02:16.120 | and sort of a high-level architectural approach

01:02:18.160 | for why we should do it.

01:02:19.520 | But for the most part, at that point,

01:02:22.280 | the execs really were just looking

01:02:23.640 | for someone crazy enough to say yes,

01:02:25.880 | because for several years at that point,

01:02:27.880 | everyone had said no.

01:02:29.640 | I'm not willing to risk my reputation

01:02:32.280 | and my career on this thing.

01:02:34.840 | - Clearly, you did not have such fears.

01:02:36.720 | - I did not.

01:02:38.000 | - So you dived right in, and yet,

01:02:41.220 | from what I understand,

01:02:42.840 | it was performing very poorly in the beginning.

01:02:46.340 | So what were the initial approaches and why did they fail?

01:02:49.780 | - Well, there were lots of hard aspects to it.

01:02:54.860 | I mean, one of the reasons why prior approaches

01:02:57.740 | that we had worked on in the past failed

01:03:02.400 | was because the questions were difficult to interpret.

01:03:07.400 | Like, what are you even asking for?

01:03:10.120 | Very often, if the question was very direct,

01:03:12.520 | like what city, or what, even then it could be tricky,

01:03:16.640 | but what city or what person,

01:03:21.640 | often when it would name it very clearly,

01:03:24.240 | you would know that.

01:03:26.020 | And if there were just a small set of them,

01:03:28.140 | in other words, we're gonna ask about these five types.

01:03:31.580 | Like, it's gonna be an answer,

01:03:33.580 | and the answer will be a city in this state,

01:03:36.840 | or a city in this country.

01:03:37.820 | The answer will be a person of this type, right?

01:03:41.060 | Like an actor or whatever it is.

01:03:42.780 | But it turns out that in "Jeopardy!"

01:03:44.400 | there were like tens of thousands of these things,

01:03:47.640 | and it was a very, very long tale,

01:03:49.600 | meaning it just went on and on.

01:03:52.540 | And so even if you focused on trying to encode the types

01:03:56.920 | at the very top, like there's five that were the most,

01:03:59.880 | let's say five of the most frequent,

01:04:01.620 | you still cover a very small percentage of the data.

01:04:04.200 | So you couldn't take that approach of saying,

01:04:07.180 | I'm just going to try to collect facts

01:04:09.820 | about these five or 10 types,

01:04:12.040 | or 20 types, or 50 types, or whatever.

01:04:14.440 | So that was like one of the first things,

01:04:16.980 | like what do you do about that?

01:04:18.240 | And so we came up with an approach toward that.

01:04:21.540 | And the approach looked promising.

01:04:23.520 | And we continued to improve our ability

01:04:26.000 | to handle that problem throughout the project.

01:04:29.560 | The other issue was that right from the outset,

01:04:32.440 | I said, we're not going to,

01:04:34.620 | I committed to doing this in three to five years.

01:04:37.660 | So we did it in four.

01:04:39.120 | So I got lucky.

01:04:41.000 | But one of the things that that,

01:04:42.420 | putting that stake in the ground,

01:04:45.180 | was I, and I knew how hard

01:04:46.600 | the language understanding problem was.

01:04:47.820 | I said, we're not going to actually understand language

01:04:51.680 | to solve this problem.

01:04:52.780 | We are not going to interpret the question

01:04:57.480 | and the domain of knowledge that the question refers to

01:05:00.200 | in reason over that to answer these questions.

01:05:02.440 | Obviously, we're not going to be doing that.

01:05:04.200 | At the same time, simple search wasn't good enough

01:05:08.280 | to confidently answer with a single correct answer.

01:05:13.040 | - First of all, that's like brilliant.

01:05:14.280 | That's such a great mix of innovation

01:05:16.160 | and practical engineering, three, four, eight.

01:05:18.660 | So you're not trying to solve the general NLU problem.

01:05:21.800 | You're saying, let's solve this in any way possible.

01:05:25.300 | - Oh, yeah, no, I was committed to saying,

01:05:27.860 | look, we're just solving

01:05:28.700 | the open domain question answering problem.

01:05:31.020 | We're using Jeopardy as a driver for that.

01:05:33.620 | - Big benchmark.

01:05:34.460 | - Hard enough, big benchmark, exactly.

01:05:36.540 | And now we're--

01:05:38.220 | - How do we do it?

01:05:39.060 | - We could just like, whatever,

01:05:39.980 | like just figure out what works,

01:05:41.180 | because I want to be able to go back

01:05:42.380 | to the academic science community and say,

01:05:44.780 | here's what we tried.

01:05:46.020 | Here's what worked.

01:05:46.860 | Here's what didn't work.

01:05:48.200 | I don't want to go in and say,

01:05:50.300 | oh, I only have one technology.

01:05:52.020 | I have a hammer.

01:05:52.860 | I'm only going to use this.

01:05:53.680 | I'm going to do whatever it takes.

01:05:54.740 | I'm like, I'm going to think out of the box

01:05:55.980 | and do whatever it takes.

01:05:57.180 | And I also, there was another thing I believed.

01:06:00.580 | I believed that the fundamental NLP technologies

01:06:04.620 | and machine learning technologies would be adequate.

01:06:08.800 | And this was an issue of how do we enhance them?

01:06:11.980 | How do we integrate them?

01:06:13.680 | How do we advance them?

01:06:15.340 | So I had one researcher who came to me

01:06:17.220 | who had been working on question answering

01:06:18.660 | with me for a very long time,

01:06:20.180 | who had said, we're going to need Maxwell's equations

01:06:24.300 | for question answering.

01:06:25.700 | And I said, if we need some fundamental formula

01:06:28.740 | that breaks new ground in how we understand language,

01:06:31.860 | we're screwed.

01:06:33.100 | We're not going to get there from here.

01:06:34.420 | Like, I am not counting.

01:06:38.060 | My assumption is I'm not counting

01:06:39.700 | on some brand new invention.

01:06:42.420 | What I'm counting on is the ability to take everything

01:06:46.620 | that has done before to figure out an architecture

01:06:50.300 | on how to integrate it well, and then see where it breaks

01:06:54.340 | and make the necessary advances we need to make

01:06:57.260 | until this thing works.

01:06:58.580 | - Yeah, push it hard to see where it breaks

01:07:00.460 | and then patch it up.

01:07:01.660 | I mean, that's how people change the world.

01:07:03.260 | I mean, that's the Elon Musk approach with rockets,

01:07:05.980 | SpaceX, that's the Henry Ford and so on.

01:07:08.860 | I love it.

01:07:09.700 | - And I happen to be, in this case, I happen to be right,

01:07:11.980 | but we didn't know.

01:07:14.380 | But you kind of have to put a stake in it

01:07:15.860 | as to how you're going to run the project.

01:07:17.420 | - So, yeah, and backtracking to search.

01:07:20.380 | So, if you were to do, what's the brute force solution?

01:07:24.340 | What would you search over?

01:07:26.140 | So you have a question, how would you search

01:07:29.260 | the possible space of answers?

01:07:31.380 | - Look, web search has come a long way, even since then.

01:07:34.860 | But at the time, first of all, I mean,

01:07:38.020 | there are a couple other constraints around the problem,

01:07:40.060 | which is interesting.

01:07:40.980 | So you couldn't go out to the web,

01:07:43.140 | you couldn't search the internet.

01:07:45.020 | In other words, the AI experiment was,

01:07:47.620 | we want a self-contained device.

01:07:50.460 | The device, if the device is as big as a room,

01:07:52.620 | fine, it's as big as a room,

01:07:53.900 | but we want a self-contained device.

01:07:57.980 | You're not going out to the internet,

01:07:59.260 | you don't have a lifeline to anything.

01:08:01.580 | So it had to kind of fit in a shoebox, if you will,

01:08:04.280 | or at least the size of a few refrigerators,

01:08:06.600 | whatever it might be.

01:08:08.060 | So, but also you couldn't just get out there.

01:08:10.440 | You couldn't go off network, right, to kind of go.

01:08:13.060 | So there was that limitation.

01:08:14.920 | But then we did, but the basic thing was,

01:08:16.900 | go do a web search.

01:08:19.340 | The problem was, even when we went and did a web search,

01:08:22.940 | I don't remember exactly the numbers,

01:08:24.540 | but somewhere in the order of 65% of the time,

01:08:27.580 | the answer would be somewhere,

01:08:30.020 | you know, in the top 10 or 20 documents.

01:08:32.900 | So first of all, that's not even good enough

01:08:34.420 | to play Jeopardy.

01:08:35.440 | You know, in other words, even if you could pull the,

01:08:38.180 | even if you could perfectly pull the answer

01:08:40.240 | out of the top 20 documents,

01:08:42.200 | top 10 documents, whatever it was,

01:08:43.960 | which we didn't know how to do,

01:08:45.240 | but even if you could do that,

01:08:46.960 | you'd be, and you knew it was right,

01:08:49.120 | unless you had enough confidence in it, right?

01:08:50.720 | So you'd have to pull out the right answer.

01:08:52.480 | You'd have to have confidence it was the right answer.

01:08:54.840 | And then you'd have to do that fast enough

01:08:56.440 | to now go buzz in,

01:08:58.120 | and you'd still only get 65% of them right,

01:09:00.320 | which doesn't even put you in the winner's circle.

01:09:02.640 | Winner's circle, you have to be up over 70,

01:09:05.080 | and you have to do it really quick,

01:09:06.040 | and you have to do it really quickly.

01:09:08.040 | But now the problem is, well,

01:09:10.100 | even if I had somewhere in the top 10 documents,

01:09:12.500 | how do I figure out where in the top 10 documents

01:09:14.980 | that answer is, and how do I compute a confidence

01:09:18.040 | of all the possible candidates,

01:09:19.740 | so it's not like I go in knowing the right answer

01:09:21.780 | and have to pick it.

01:09:22.620 | I don't know the right answer.

01:09:23.940 | I have a bunch of documents,

01:09:25.580 | somewhere in there's the right answer.

01:09:27.100 | How do I, as a machine, go out and figure out

01:09:29.060 | which one's right, and then how do I score it?

01:09:31.460 | So, and now how do I deal with the fact

01:09:35.300 | that I can't actually go out to the web?

01:09:37.320 | - First of all, if you pause on that, just think about it.

01:09:40.020 | If you could go to the web,

01:09:42.180 | do you think that problem is solvable,

01:09:44.260 | if you just pause on it?

01:09:45.540 | Just thinking even beyond Jeopardy,

01:09:48.320 | do you think the problem of reading text

01:09:51.340 | to find where the answer is?

01:09:53.660 | - Well, we solved that in some definition of solved,

01:09:56.700 | given the Jeopardy challenge.

01:09:57.980 | - How did you do it for Jeopardy?

01:09:59.020 | So how do you take a body of work on a particular topic

01:10:03.260 | and extract the key pieces of information?

01:10:05.940 | - So, now forgetting about the huge volumes

01:10:09.120 | that are on the web, right?

01:10:10.080 | So now we have to figure out,

01:10:11.240 | we did a lot of source research.

01:10:12.720 | In other words, what body of knowledge

01:10:15.720 | is gonna be small enough, but broad enough,

01:10:18.760 | to answer Jeopardy?

01:10:19.840 | And we ultimately did find the body of knowledge

01:10:21.960 | that did that, I mean, it included Wikipedia

01:10:23.760 | and a bunch of other stuff.

01:10:25.160 | - So like encyclopedia type of stuff,

01:10:26.720 | I don't know if you can speak to--

01:10:27.560 | - Encyclopedia, dictionaries,

01:10:28.560 | different types of semantic resources,

01:10:31.000 | like WordNet and other types of semantic resources,

01:10:33.200 | like that, as well as like some web crawls.

01:10:36.100 | In other words, where we went out and took that content

01:10:39.100 | and then expanded it based on producing statistical,

01:10:42.500 | you know, statistically producing seeds,

01:10:44.660 | using those seeds for other searches,

01:10:47.580 | and then expanding that.

01:10:48.780 | So using these like expansion techniques,

01:10:51.540 | we went out and found enough content

01:10:53.660 | and we're like, okay, this is good.

01:10:54.660 | And even up until the end, you know,

01:10:57.020 | we had a thread of research,

01:10:58.420 | it was always trying to figure out

01:10:59.820 | what content could we efficiently include.

01:11:02.260 | - I mean, there's a lot of popular,

01:11:03.440 | like what is the church lady?

01:11:05.440 | Well, I think was one of the, like what,

01:11:08.060 | where do you, I guess that's probably an encyclopedia, so.

01:11:12.440 | - So that's an encyclopedia, but then we would take

01:11:15.640 | that stuff and we would go out and we would expand.

01:11:17.800 | In other words, we go find other content

01:11:20.180 | that wasn't in the core resources and expand it.

01:11:23.000 | You know, the amount of content,

01:11:24.280 | we grew it by an order of magnitude,

01:11:26.200 | but still, again, from a web scale perspective,

01:11:28.600 | this is very small amount of content.

01:11:30.560 | - It's very select.

01:11:31.400 | - And then we then took all that content

01:11:33.080 | and we pre-analyzed the crap out of it,

01:11:35.240 | meaning we parsed it, you know,

01:11:38.480 | broke it down into all those individual words

01:11:40.720 | and then we did semantic,

01:11:42.200 | static and semantic parses on it,

01:11:44.320 | you know, had computer algorithms that annotated it

01:11:47.000 | and we indexed that in a very rich and very fast index.

01:11:52.000 | So we have a relatively huge amount of,

01:11:55.240 | let's say the equivalent of, for the sake of argument,

01:11:57.420 | two to five million bucks.

01:11:59.000 | We've now analyzed all that, blowing up its size even more

01:12:01.840 | because now we have all this metadata

01:12:03.600 | and then we richly indexed all of that

01:12:05.640 | and by way in a giant in-memory cache.

01:12:08.960 | So Watson did not go to disk.

01:12:11.980 | - So the infrastructure component there,

01:12:13.680 | if you could just speak to it, how tough it,

01:12:15.840 | I mean, I know 2000, maybe this is 2008, nine,

01:12:20.780 | you know, that's kind of a long time ago.

01:12:24.680 | - Right.

01:12:25.920 | - How hard is it to use multiple machines?

01:12:27.880 | Like how hard is the infrastructure component,

01:12:29.880 | the hardware component?

01:12:31.640 | - So we used IBM hardware.

01:12:33.840 | We had something like, I forget exactly,

01:12:36.080 | but close to 3000 cores completely connected.

01:12:40.760 | So you had a switch where, you know,

01:12:42.080 | every CPU was connected to every other CPU.

01:12:43.840 | - And they were sharing memory in some kind of way.

01:12:46.120 | - Large shared memory, right?

01:12:48.040 | And all this data was pre-analyzed

01:12:50.800 | and put into a very fast indexing structure

01:12:54.920 | that was all in memory.

01:12:58.320 | And then we took that question,

01:13:01.420 | we would analyze the question.

01:13:04.440 | So all the content was now pre-analyzed.

01:13:07.240 | So if I went and tried to find a piece of content,

01:13:10.840 | it would come back with all the metadata

01:13:12.600 | that we had pre-computed.

01:13:14.600 | - How do you shove that question?

01:13:16.960 | How do you connect the big stuff,

01:13:20.040 | the big knowledge base of the metadata

01:13:21.880 | that's indexed to the simple little witty,

01:13:25.080 | confusing question?

01:13:27.000 | - Right.

01:13:27.840 | So therein lies, you know, the Watson architecture.

01:13:31.360 | So we would take the question,

01:13:33.000 | we would analyze the question.

01:13:34.760 | So which means that we would parse it

01:13:37.080 | and interpret it a bunch of different ways.

01:13:38.800 | We'd try to figure out what is it asking about?

01:13:40.880 | So we would come, we had multiple strategies

01:13:44.440 | to kind of determine what was it asking for.

01:13:47.240 | That might be represented as a simple string,

01:13:49.520 | a character string,

01:13:51.440 | or something we would connect back

01:13:53.160 | to different semantic types

01:13:54.840 | that were from existing resources.

01:13:56.160 | So anyway, the bottom line is we would do

01:13:58.160 | a bunch of analysis in the question.

01:14:00.440 | And question analysis had to finish,

01:14:02.160 | and it had to finish fast.

01:14:04.240 | So we do the question analysis

01:14:05.360 | because then from the question analysis,

01:14:07.960 | we would now produce searches.

01:14:09.840 | So we would, and we had built,

01:14:12.720 | using open source search engines, we modified them.

01:14:16.160 | We had a number of different search engines we would use

01:14:19.060 | that had different characteristics.

01:14:20.760 | We went in there and engineered

01:14:22.560 | and modified those search engines,

01:14:24.520 | ultimately to now take our question analysis,

01:14:28.520 | produce multiple queries based on different interpretations

01:14:32.120 | of the question,

01:14:33.320 | and fire out a whole bunch of searches in parallel.

01:14:36.480 | And they would come back with passages.

01:14:41.080 | So these are passive search algorithms.

01:14:43.040 | They would come back with passages.

01:14:44.840 | And so now let's say you had a thousand passages.

01:14:48.200 | Now for each passage, you parallelize again.

01:14:51.800 | So you went out and you parallelized the search.

01:14:56.280 | Each search would now come back

01:14:57.480 | with a whole bunch of passages.

01:14:59.720 | Maybe you had a total of a thousand

01:15:01.440 | or 5,000, whatever passages.

01:15:03.280 | For each passage now, you'd go and figure out

01:15:05.800 | whether or not there was a candidate,

01:15:07.160 | we'd call a candidate answer in there.

01:15:09.180 | So you had a whole bunch of other algorithms

01:15:12.160 | that would find candidate answers,

01:15:13.840 | possible answers to the question.

01:15:16.220 | And so you had candidate answers,

01:15:18.240 | called candidate answers generators,

01:15:20.160 | a whole bunch of those.

01:15:21.360 | So for every one of these components,

01:15:23.680 | the team was constantly doing research,

01:15:25.480 | coming up better ways to generate search queries

01:15:27.920 | from the questions,

01:15:28.800 | better ways to analyze the question,

01:15:30.520 | better ways to generate candidates.

01:15:31.960 | - And speed, so better is accuracy and speed.

01:15:35.880 | - Correct, so right, speed and accuracy,

01:15:38.600 | for the most part, were separated.

01:15:40.960 | We handled that sort of in separate ways.

01:15:42.640 | Like I focus purely on accuracy and inaccuracy,

01:15:45.640 | are we ultimately getting more questions

01:15:47.360 | and producing more accurate confidences?

01:15:49.200 | And then a whole nother team

01:15:50.620 | that was constantly analyzing the workflow

01:15:52.880 | to find the bottlenecks,

01:15:54.240 | and then figuring out how to both parallelize

01:15:56.200 | and drive the algorithm speed.

01:15:58.520 | But anyway, so now think of it like,

01:16:00.400 | you have this big fan out now, right?

01:16:02.160 | Because you had multiple queries,

01:16:04.040 | now you have thousands of candidate answers.

01:16:07.360 | For each candidate answer, you're gonna score it.

01:16:10.400 | So you're gonna use all the data that built up.

01:16:12.840 | You're gonna use the question analysis.

01:16:15.880 | You're gonna use how the query was generated.

01:16:17.960 | You're gonna use the passage itself.

01:16:20.240 | And you're gonna use the candidate answer

01:16:22.000 | that was generated.

01:16:23.280 | And you're gonna score that.

01:16:25.840 | So now we have a group of researchers

01:16:28.400 | coming up with scorers.

01:16:30.120 | There are hundreds of different scorers.

01:16:32.280 | So now you're getting a fan out of it again,

01:16:34.600 | from however many candidate answers you have,

01:16:37.400 | to all the different scorers.

01:16:39.280 | So if you have 200 different scorers

01:16:41.240 | and you have a thousand candidates,

01:16:42.440 | now you have 200,000 scores.

01:16:45.200 | And so now you gotta figure out,

01:16:47.160 | how do I now rank these answers

01:16:52.400 | based on the scores that came back?

01:16:54.520 | And I wanna rank them based on the likelihood

01:16:56.360 | that they're a correct answer to the question.

01:16:58.680 | So every scorer was its own research project.

01:17:01.400 | - What do you mean by scorer?

01:17:02.360 | So is that the annotation process

01:17:04.080 | of basically a human being saying

01:17:06.520 | that this answer has a quality of--

01:17:09.360 | - Think of it, if you wanna think of it,

01:17:10.800 | what you're doing, if you wanna think about

01:17:13.240 | what a human would be doing,

01:17:14.080 | a human would be looking at a possible answer.

01:17:17.080 | They'd be reading the, Emily Dickinson,

01:17:20.920 | they'd be reading the passage in which that occurred.

01:17:23.560 | They'd be looking at the question

01:17:25.400 | and they'd be making a decision of how likely it is

01:17:28.400 | that Emily Dickinson, given this evidence in this passage,

01:17:32.320 | is the right answer to that question.

01:17:33.960 | - Got it.

01:17:34.800 | So that's the annotation task.

01:17:36.240 | That's the annotation process.

01:17:37.080 | - That's the scoring task.

01:17:38.840 | - But scoring implies zero to one kind of continuous--

01:17:41.320 | - That's right, you give it a zero to one score.

01:17:43.000 | - So it's not a binary--

01:17:44.360 | - No, you give it a score.

01:17:46.000 | Give it a zero, yeah, exactly, a zero to one score.

01:17:48.440 | - So but humans give different scores,

01:17:50.520 | so you have to somehow normalize and all that kind of stuff

01:17:52.960 | that deal with all that complexity.

01:17:54.240 | - Depends on what your strategy is.

01:17:55.760 | We both, we--

01:17:57.080 | - Could be relative, too.

01:17:58.040 | It could be--

01:17:59.440 | - We actually looked at the raw scores as well,

01:18:01.960 | standardized scores, because humans are not involved in this.

01:18:04.980 | Humans are not involved.

01:18:05.920 | - Sorry, so I'm misunderstanding the process here.

01:18:08.680 | There's passages.

01:18:10.440 | Where is the ground truth coming from?

01:18:13.320 | - Ground truth is only the answers to the questions.

01:18:15.920 | - So it's end to end.

01:18:17.920 | - It's end to end.

01:18:19.000 | So I was always driving end to end performance.

01:18:22.360 | It was a very interesting engineering approach,

01:18:27.360 | and ultimately scientific and research approach,

01:18:30.040 | always driving end to end.

01:18:31.200 | Now, that's not to say we wouldn't make hypotheses

01:18:38.560 | that individual component performance

01:18:42.120 | was related in some way to end to end performance.

01:18:44.400 | Of course we would, because people would have to

01:18:47.040 | build individual components.

01:18:48.860 | But ultimately, to get your component

01:18:50.840 | integrated into the system,

01:18:52.340 | you had to show impact on end to end performance,

01:18:54.960 | question answering performance.

01:18:56.320 | - There's many very smart people working on this,

01:18:58.360 | and they're basically trying to sell their ideas

01:19:01.520 | as a component that should be part of the system.

01:19:03.400 | - That's right.

01:19:04.560 | And they would do research on their component,

01:19:07.320 | and they would say things like,

01:19:09.720 | I'm gonna improve this as a candidate generator,

01:19:13.120 | or I'm gonna improve this as a question score,

01:19:15.840 | or as a passage score, I'm gonna improve this,

01:19:19.040 | or as a parser, and I can improve it by 2%

01:19:23.920 | on its component metric, like a better parse,

01:19:26.720 | or a better candidate, or a better type estimation,

01:19:29.200 | or whatever it is.

01:19:30.200 | And then I would say, I need to understand

01:19:32.600 | how the improvement on that component metric

01:19:35.320 | is gonna affect the end to end performance.

01:19:37.720 | If you can't estimate that, and can't do experiments

01:19:40.520 | to demonstrate that, it doesn't get in.

01:19:43.360 | - That's like the best run AI project I've ever heard.

01:19:47.520 | That's awesome, okay.

01:19:49.040 | What breakthrough would you say,

01:19:51.800 | like I'm sure there's a lot of day to day breakthroughs,

01:19:54.240 | but was there like a breakthrough

01:19:55.600 | that really helped improve performance?

01:19:57.880 | Like where people began to believe?

01:20:00.180 | Or is it just a gradual process?

01:20:02.480 | - Well, I think it was a gradual process,

01:20:04.520 | but one of the things that I think gave people confidence

01:20:08.960 | that we can get there was that as we follow this procedure

01:20:13.960 | of different ideas, build different components,

01:20:19.160 | plug them into the architecture, run the system,

01:20:21.200 | see how we do, do the error analysis,

01:20:24.680 | start off new research projects to improve things,

01:20:28.120 | and the very important idea that the individual component

01:20:33.640 | work did not have to deeply understand everything

01:20:38.640 | that was going on with every other component.

01:20:42.240 | And this is where we leverage machine learning

01:20:45.080 | in a very important way.

01:20:47.400 | So while individual components could be statistically driven

01:20:50.360 | machine learning components, some of them were heuristic,

01:20:52.760 | some of them were machine learning components,

01:20:54.640 | the system has a whole combined all the scores

01:20:58.120 | using machine learning.

01:21:00.560 | This was critical because that way you can divide

01:21:03.440 | and conquer.

01:21:04.400 | So you can say, okay, you work on your candidate generator,

01:21:07.520 | or you work on this approach to answer scoring,

01:21:09.780 | you work on this approach to type scoring,

01:21:11.800 | you work on this approach to passage search

01:21:14.520 | or to passage selection and so forth.

01:21:16.360 | But when we just plug it in,

01:21:19.600 | and we had enough training data to say,

01:21:22.080 | now we can train and figure out how do we weigh

01:21:26.560 | all the scores relative to each other

01:21:29.360 | based on predicting the outcome,

01:21:31.920 | which is right or wrong on Jeopardy.

01:21:33.960 | And we had enough training data to do that.

01:21:36.800 | So this enabled people to work independently

01:21:40.680 | and to let the machine learning do the integration.

01:21:43.400 | - Beautiful, so yeah, the machine learning

01:21:45.160 | is doing the fusion, and then it's a human orchestrated

01:21:48.280 | ensemble of different approaches.

01:21:50.520 | That's great.

01:21:51.980 | Still impressive that you're able to get it done

01:21:55.600 | in a few years.

01:21:57.680 | That's not obvious to me that it's doable

01:22:00.440 | if I just put myself in that mindset.

01:22:03.400 | But when you look back at the Jeopardy challenge,

01:22:05.920 | again, when you're looking up at the stars,

01:22:10.280 | what are you most proud of?

01:22:11.760 | Just looking back at those days.

01:22:15.440 | - I'm most proud of my...

01:22:21.760 | (mouse clicking)

01:22:24.520 | My commitment and my team's commitment

01:22:30.920 | to be true to the science,

01:22:33.920 | to not be afraid to fail.

01:22:38.100 | - That's beautiful because there's so much pressure

01:22:41.600 | because it is a public event, it is a public show,

01:22:44.440 | that you were dedicated to the idea.

01:22:47.040 | - That's right.

01:22:50.520 | - Do you think it was a success?

01:22:53.200 | In the eyes of the world, it was a success.

01:22:55.360 | By your, I'm sure, exceptionally high standards,

01:22:59.720 | is there something you regret you would do differently?

01:23:03.740 | - It was a success.

01:23:07.920 | It was a success for our goal.

01:23:10.160 | Our goal was to build the most advanced

01:23:13.400 | open domain question answering system.

01:23:15.400 | We went back to the old problems that we used to try

01:23:19.520 | to solve and we did dramatically better on all of them,

01:23:23.160 | as well as we beat Jeopardy.

01:23:26.120 | So we won at Jeopardy.

01:23:27.760 | So it was a success.

01:23:30.160 | I worry that the world would not understand it as a success

01:23:36.160 | because it came down to only one game.

01:23:38.720 | And I knew statistically speaking,

01:23:40.440 | this can be a huge technical success

01:23:42.280 | and we could still lose that one game.

01:23:43.880 | And that's a whole nother theme of the journey.

01:23:47.260 | But it was a success.

01:23:50.280 | It was not a success in natural language understanding,

01:23:53.640 | but that was not the goal.

01:23:54.940 | - Yeah, that was, but I would argue,

01:23:59.840 | I understand what you're saying in terms of the science,

01:24:04.120 | but I would argue that the inspiration of it, right,

01:24:07.640 | not a success in terms of solving

01:24:11.200 | natural language understanding,

01:24:12.800 | but it was a success of being an inspiration

01:24:16.280 | to future challenges.

01:24:17.860 | - Absolutely.

01:24:18.820 | - That drive future efforts.

01:24:21.140 | What's the difference between how human being

01:24:23.740 | compete in Jeopardy and how Watson does it?

01:24:26.860 | That's important in terms of intelligence.

01:24:28.740 | - Yeah, so that actually came up very early on

01:24:31.380 | in the project also.

01:24:32.600 | In fact, I had people who wanted to be on the project

01:24:35.140 | who were early on, who sort of approached me

01:24:39.060 | once I committed to do it,

01:24:40.820 | that wanted to think about how humans do it.

01:24:44.300 | And they were, you know, from a cognition perspective,

01:24:47.080 | like human cognition and how that should play.

01:24:49.920 | And I would not take them on the project

01:24:52.200 | because another assumption or another stake

01:24:55.800 | I put in the ground was I don't really care

01:24:58.480 | how humans do this.

01:25:00.000 | - At least in the context of this project.

01:25:01.480 | - I need to build, in the context of this project,

01:25:03.880 | in NLU and in building an AI that understands

01:25:07.000 | how it needs to ultimately communicate with humans,

01:25:09.660 | I very much care.

01:25:11.280 | So it wasn't that I didn't care in general.

01:25:16.280 | In fact, as an AI scientist, I care a lot about that,

01:25:20.780 | but I'm also a practical engineer

01:25:22.580 | and I committed to getting this thing done

01:25:25.540 | and I wasn't gonna get distracted.

01:25:27.500 | I had to kind of say, like, if I'm gonna get this done,

01:25:30.780 | I'm gonna chart this path and this path says,

01:25:33.300 | we're gonna engineer a machine

01:25:34.940 | that's gonna get this thing done.

01:25:37.640 | And we know what search and NLP can do.

01:25:41.560 | We have to build on that foundation.

01:25:44.200 | If I come in and take a different approach

01:25:46.320 | and start wondering about how the human mind

01:25:48.100 | might or might not do this,

01:25:49.760 | I'm not gonna get there from here in the time frame.

01:25:54.400 | - I think that's a great way to lead the team.

01:25:56.680 | But now that it's done and then one,

01:25:59.240 | when you look back, analyze what's the difference actually.

01:26:02.560 | - Right, so I was a little bit surprised actually

01:26:05.520 | to discover over time, as this would come up

01:26:09.060 | from time to time and we'd reflect on it,

01:26:11.160 | and talking to Ken Jennings a little bit

01:26:15.020 | and hearing Ken Jennings talk about

01:26:17.060 | how he answered questions,

01:26:18.900 | that it might have been closer to the way humans

01:26:21.300 | answer questions than I might have imagined previously.

01:26:24.740 | - 'Cause humans are probably in the game of Jeopardy

01:26:27.900 | at the level of Ken Jennings,

01:26:29.620 | probably also cheating their way to winning, right?

01:26:34.980 | - Well, they're doing shallow analysis.

01:26:36.660 | - Shallow, the fastest possible.

01:26:39.380 | - They're doing shallow analysis.

01:26:40.900 | So they are very quickly analyzing the question

01:26:44.900 | and coming up with some key vectors or cues, if you will.

01:26:49.900 | And they're taking those cues

01:26:51.100 | and they're very quickly going through

01:26:52.900 | their library of stuff,

01:26:54.920 | not deeply reasoning about what's going on.

01:26:57.740 | And then sort of like a lots of different,

01:27:00.620 | like what we call these scores,

01:27:03.220 | would kind of score that in a very shallow way

01:27:06.100 | and then say, oh, boom, that's what it is.

01:27:08.940 | And so it's interesting as we reflected on that,

01:27:12.460 | so we may be doing something that's not too far off

01:27:16.060 | from the way humans do it,

01:27:17.260 | but we certainly didn't approach it by saying,

01:27:21.460 | how would a human do this?

01:27:22.700 | Now in elemental cognition,

01:27:24.660 | like the project I'm leading now,

01:27:27.340 | we ask those questions all the time,

01:27:28.780 | because ultimately we're trying to do something

01:27:31.700 | that is to make the intelligence of the machine

01:27:35.100 | and the intelligence of the human very compatible.

01:27:37.780 | Well, compatible in the sense they can communicate

01:27:39.500 | with one another and they can reason

01:27:42.580 | with this shared understanding.

01:27:44.540 | So how they think about things and how they build answers,

01:27:48.060 | how they build explanations

01:27:49.780 | becomes a very important question to consider.

01:27:52.140 | - So what's the difference between this open domain,

01:27:56.920 | but cold constructed question answering of Jeopardy

01:28:01.920 | and more something that requires understanding

01:28:07.360 | for shared communication with humans and machines?

01:28:10.280 | - Yeah, well, this goes back to the interpretation

01:28:13.360 | of what we were talking about before.

01:28:14.800 | - Framework.

01:28:15.640 | - Jeopardy, the system's not trying to interpret

01:28:18.560 | the question and it's not interpreting the content

01:28:20.680 | that's reusing with regard to any particular framework.

01:28:23.880 | I mean, it is parsing it and parsing the content

01:28:26.880 | and using grammatical cues and stuff like that.

01:28:29.440 | So if you think of grammar as a human framework,

01:28:31.680 | in some sense it has that.

01:28:33.400 | But when you get into the richer semantic frameworks,

01:28:36.880 | what are people, how do they think, what motivates them?

01:28:40.080 | What are the events that are occurring

01:28:41.640 | and why are they occurring and what causes

01:28:43.280 | what else to happen and where are things in time and space?

01:28:47.440 | And like when you start thinking about how humans formulate

01:28:51.280 | and structure the knowledge that they acquire in their head,

01:28:54.000 | it wasn't doing any of that.

01:28:56.400 | - What do you think are the essential challenges

01:29:01.400 | of free-flowing communication, free-flowing dialogue

01:29:05.840 | versus question answering even with a framework

01:29:09.040 | of the interpretation?

01:29:10.560 | Dialogue.

01:29:11.400 | - Yep.

01:29:12.320 | - Do you see free-flowing dialogue

01:29:14.960 | as fundamentally more difficult than question answering

01:29:19.960 | even with shared interpretation?

01:29:23.560 | - So dialogue is important in a number of different ways.

01:29:26.640 | I mean, it's a challenge.

01:29:27.480 | So first of all, when I think about the machine that,

01:29:30.520 | when I think about a machine that understands language

01:29:33.280 | and ultimately can reason in an objective way

01:29:36.760 | that can take the information that it perceives

01:29:40.560 | through language or other means

01:29:42.200 | and connect it back to these frameworks,

01:29:44.520 | reason and explain itself,

01:29:46.200 | that system ultimately needs to be able to talk to humans

01:29:50.680 | or it needs to be able to interact with humans.

01:29:52.880 | So in some sense it needs to dialogue.

01:29:55.120 | That doesn't mean that it,

01:29:57.560 | sometimes people talk about dialogue and they think,

01:30:01.800 | you know, how do humans talk to each other

01:30:05.080 | in a casual conversation?

01:30:07.680 | And you can mimic casual conversations.

01:30:09.880 | We're not trying to mimic casual conversations.

01:30:14.320 | We're really trying to produce a machine

01:30:17.520 | whose goal is to help you think

01:30:20.200 | and help you reason about your answers

01:30:22.200 | and explain why.

01:30:23.600 | So instead of like talking to your friend down the street

01:30:26.560 | about having a small talk conversation

01:30:28.880 | with your friend down the street,

01:30:30.480 | this is more about like you would be communicating

01:30:32.360 | to the computer on Star Trek where,

01:30:35.120 | like what do you wanna think about?

01:30:36.760 | Like what do you wanna reason about?

01:30:37.600 | I'm gonna tell you the information I have.

01:30:38.760 | I'm gonna have to summarize it.

01:30:39.840 | I'm gonna ask you questions.

01:30:41.040 | You're gonna answer those questions.

01:30:42.680 | I'm gonna go back and forth with you.

01:30:44.240 | I'm gonna figure out what your mental model is.

01:30:46.600 | I'm gonna now relate that to the information I have

01:30:50.080 | and present it to you in a way that you can understand it

01:30:53.040 | and then we could ask follow-up questions.

01:30:54.920 | So it's that type of dialogue that you wanna construct.

01:30:58.280 | It's more structured.

01:31:00.400 | It's more goal-oriented, but it needs to be fluid.

01:31:04.840 | In other words, it can't, it can't,

01:31:06.880 | it has to be engaging and fluid.

01:31:09.240 | It has to be productive and not distracting.

01:31:13.080 | So there has to be a model of,

01:31:15.720 | in other words, the machine has to have a model

01:31:17.580 | of how humans think through things and discuss them.

01:31:22.580 | - So basically a productive, rich conversation,

01:31:27.000 | unlike this podcast.

01:31:30.120 | - I'd like to think it's more similar to this podcast.

01:31:34.960 | - I was just joking.

01:31:36.120 | I'll ask you about humor as well, actually.

01:31:39.760 | But what's the hardest part of that?

01:31:43.280 | Because it seems we're quite far away

01:31:46.600 | as a community from that still to be able to,

01:31:49.800 | so one is having a shared understanding.

01:31:53.000 | That's, I think, a lot of the stuff you said

01:31:54.880 | with frameworks is quite brilliant.

01:31:57.120 | But just creating a smooth discourse.

01:32:01.480 | Yeah, it feels clunky right now.

01:32:05.280 | Which aspects of this whole problem that you specified

01:32:10.040 | of having a productive conversation is the hardest?

01:32:14.600 | Or maybe any aspect of it you can comment on

01:32:18.600 | 'cause it's so shrouded in mystery.

01:32:20.780 | - So I think to do this, you kind of have to be creative

01:32:24.280 | in the following sense.

01:32:25.880 | If I were to do this as purely a machine learning approach

01:32:29.800 | and someone said, "Learn how to have a good,

01:32:32.840 | "fluent, structured knowledge acquisition conversation,"

01:32:37.360 | I'd go out and say, "Okay, I have to collect a bunch

01:32:40.080 | "of data of people doing that, people reasoning well

01:32:44.720 | "having a good, structured conversation

01:32:47.680 | "that both acquires knowledge efficiently

01:32:50.220 | "as well as produces answers and explanations

01:32:52.320 | "as part of the process."

01:32:54.600 | And you struggle.

01:32:57.340 | I don't know-- - To collect the data.

01:32:58.520 | - To collect the data because I don't know

01:33:00.680 | how much data is like that.

01:33:03.120 | - Okay, okay, this one, there's a humorous commenter

01:33:06.160 | on the lack of rational discourse.

01:33:08.560 | But also, even if it's out there, say it was out there,

01:33:12.720 | how do you actually-- - Yeah, how--

01:33:14.800 | - Like how do you collect successful examples?

01:33:17.200 | - Right, so I think any problem like this

01:33:19.240 | where you don't have enough data to represent

01:33:23.200 | the phenomenon you wanna learn, in other words,

01:33:25.960 | if you have enough data, you could potentially

01:33:27.440 | learn the pattern.

01:33:28.560 | In an example like this, it's hard to do.

01:33:30.360 | It's sort of a human sort of thing to do.

01:33:34.400 | What recently came out at IBM was the debater project,

01:33:36.960 | sort of interesting, right, because now you do have

01:33:39.440 | these structured dialogues, these debate things,

01:33:42.560 | where they did use machine learning techniques

01:33:44.680 | to generate these debates.

01:33:46.980 | Dialogues are a little bit tougher, in my opinion,

01:33:52.440 | than generating a structured argument

01:33:56.080 | where you have lots of other structured arguments like this.

01:33:58.000 | You could potentially annotate that data

01:33:59.520 | and you could say this is a good response,

01:34:00.800 | this is a bad response in a particular domain.

01:34:03.240 | Here, I have to be responsive and I have to be opportunistic

01:34:08.240 | with regard to what is the human saying.

01:34:11.840 | So I'm goal-oriented in saying I wanna solve the problem,

01:34:14.960 | I wanna acquire the knowledge necessary,

01:34:16.640 | but I also have to be opportunistic and responsive

01:34:19.200 | to what the human is saying.

01:34:21.080 | So I think that it's not clear that we could just train

01:34:24.120 | on the body of data to do this, but we could bootstrap it.

01:34:28.040 | In other words, we can be creative

01:34:29.960 | and we could say, what do we think?

01:34:31.480 | What do we think the structure of a good dialogue is

01:34:34.080 | that does this well?

01:34:35.880 | And we can start to create that.

01:34:38.520 | If we can create that more programmatically,

01:34:42.080 | at least to get this process started,

01:34:44.720 | and I can create a tool that now engages humans effectively,

01:34:48.000 | I could start both, I could start generating data,

01:34:51.320 | I could start with the human learning process

01:34:53.040 | and I can update my machine,

01:34:55.080 | but I could also start

01:34:55.920 | the automatic learning process as well.

01:34:58.600 | But I have to understand what features to even learn over.

01:35:01.880 | So I have to bootstrap the process a little bit first.

01:35:04.760 | And that's a creative design task

01:35:07.760 | that I could then use as input

01:35:11.040 | into a more automatic learning task.

01:35:13.400 | - So some creativity and yeah, and bootstrapping.

01:35:16.720 | What elements of a conversation do you think

01:35:18.920 | you would like to see?

01:35:21.120 | So one of the benchmarks for me is humor, right?

01:35:25.600 | That seems to be one of the hardest.

01:35:27.800 | And to me, the biggest contrast is Watson.

01:35:30.340 | So one of the greatest comedy sketches of all time, right,

01:35:35.280 | is the SNL celebrity Jeopardy!

01:35:38.600 | With Alex Trebek and Sean Connery

01:35:42.120 | and Burt Reynolds and so on.

01:35:44.120 | With Sean Connery commentating on Alex Trebek's mother

01:35:48.240 | a lot.

01:35:49.440 | And I think all of them are in the negative points wise.

01:35:52.920 | So they're clearly all losing

01:35:55.160 | in terms of the game of Jeopardy!,

01:35:56.360 | but they're winning in terms of comedy.

01:35:58.360 | So what do you think about humor in this whole interaction

01:36:03.800 | in the dialogue that's productive?

01:36:06.520 | Or even just whatever, what humor represents to me is

01:36:10.200 | the same idea that you're saying about framework,

01:36:15.400 | 'cause humor only exists

01:36:16.400 | within a particular human framework.

01:36:18.320 | So what do you think about humor?

01:36:19.560 | What do you think about things like humor

01:36:21.520 | that connect to the kind of creativity

01:36:23.320 | you mentioned that's needed?

01:36:25.120 | - I think there's a couple of things going on there.

01:36:26.400 | So I sort of feel like,

01:36:29.520 | and I might be too optimistic this way,

01:36:31.800 | but I think that there are,

01:36:34.720 | we did a little bit about with puns in Jeopardy!.

01:36:39.000 | We literally sat down and said,

01:36:40.720 | how do puns work?

01:36:43.160 | And it's like wordplay,

01:36:44.800 | and you could formalize these things.

01:36:46.120 | So I think there's a lot aspects of humor

01:36:48.240 | that you could formalize.

01:36:50.200 | You could also learn humor.

01:36:51.600 | You could just say, what do people laugh at?

01:36:53.480 | And if you have enough, again,

01:36:54.880 | if you have enough data to represent that phenomenon,

01:36:56.880 | you might be able to weigh the features

01:36:59.440 | and figure out what humans find funny

01:37:01.280 | and what they don't find funny.

01:37:02.720 | The machine might not be able to explain

01:37:05.200 | why the human find it funny,

01:37:06.680 | unless we sit back and think about that more formally.

01:37:10.200 | I think, again, I think you do a combination of both.

01:37:12.440 | And I'm always a big proponent of that.

01:37:13.920 | I think robust architectures and approaches

01:37:16.720 | are always a little bit combination of us reflecting

01:37:19.680 | and being creative about how things are structured,

01:37:22.520 | how to formalize them,

01:37:23.800 | and then taking advantage of large data

01:37:25.560 | and doing learning

01:37:26.400 | and figuring out how to combine these two approaches.

01:37:29.120 | I think there's another aspect to humor though,

01:37:31.440 | which goes to the idea that I feel like I can relate

01:37:34.360 | to the person telling the story.

01:37:36.040 | And I think that's an interesting theme

01:37:42.160 | in the whole AI theme, which is,

01:37:44.200 | do I feel differently when I know it's a robot?

01:37:47.680 | And when I know, when I imagine

01:37:51.480 | that the robot is not conscious the way I'm conscious,

01:37:54.200 | when I imagine the robot does not actually

01:37:56.320 | have the experiences that I experience,

01:37:58.720 | do I find it funny?

01:38:00.960 | Or do, because it's not as related,

01:38:03.040 | I don't imagine that the person's relating it to it

01:38:06.560 | the way I relate to it.

01:38:07.840 | I think this also, you see this in the arts

01:38:11.360 | and in entertainment where,

01:38:13.760 | like sometimes you have savants

01:38:15.720 | who are remarkable at a thing,

01:38:17.400 | whether it's sculpture, it's music or whatever,

01:38:19.800 | but the people who get the most attention

01:38:21.320 | are the people who can evoke a similar emotional response

01:38:26.680 | who can get you to emote, right, about the way they are.

01:38:31.680 | In other words, who can basically make the connection

01:38:34.440 | from the artifact, from the music or the painting

01:38:37.000 | or the sculpture to the emotion

01:38:39.800 | and get you to share that emotion with them.

01:38:42.360 | And then, and that's when it becomes compelling.

01:38:44.700 | So they're communicating at a whole different level.

01:38:46.960 | They're just not communicating the artifact.

01:38:49.340 | They're communicating their emotional response

01:38:50.960 | to the artifact.

01:38:51.800 | And then you feel like, oh, wow,

01:38:53.360 | I can relate to that person.

01:38:54.520 | I can connect to that person.

01:38:57.080 | So I think humor has that aspect as well.

01:39:00.640 | - So the idea that you can connect to that person,

01:39:04.760 | person being the critical thing,

01:39:06.360 | but we're also able to anthropomorphize objects pretty,

01:39:12.240 | robots and AI systems pretty well.

01:39:15.140 | So we're almost looking to make them human.

01:39:18.720 | Then maybe from your experience with Watson,

01:39:20.760 | maybe you can comment on,

01:39:22.880 | did you consider that as part,

01:39:24.920 | well, obviously the problem of Jeopardy

01:39:26.960 | doesn't require anthropomorphization, but nevertheless--

01:39:30.480 | - Well, there was some interest in doing that.

01:39:32.240 | And that's another thing I didn't wanna do

01:39:34.960 | 'cause I didn't wanna distract

01:39:36.200 | from the actual scientific task.

01:39:38.720 | But you're absolutely right.

01:39:39.600 | I mean, humans do anthropomorphize

01:39:42.880 | and without necessarily a lot of work.

01:39:45.840 | I mean, you just put some eyes

01:39:47.040 | and a couple of eyebrow movements

01:39:49.200 | and you're getting humans to react emotionally.

01:39:51.800 | And I think you can do that.

01:39:53.520 | So I didn't mean to suggest that,

01:39:56.800 | that that connection cannot be mimicked.

01:40:00.620 | I think that connection can be mimicked

01:40:02.240 | and can get you to,

01:40:03.600 | can produce that emotional response.

01:40:07.280 | I just wonder though,

01:40:08.720 | if you're told what's really going on,

01:40:13.000 | if you know that the machine is not conscious,

01:40:17.200 | not having the same richness of emotional reactions

01:40:20.760 | and understanding that it doesn't really share

01:40:22.200 | the understanding, but it's essentially

01:40:24.000 | just moving its eyebrow or drooping its eyes

01:40:26.400 | or making them bigger, whatever it's doing,

01:40:28.160 | just getting the emotional response,

01:40:30.180 | will you still feel it?

01:40:31.600 | Interesting, I think you probably would for a while.

01:40:34.380 | And then when it becomes more important

01:40:35.880 | that there's a deeper shared understanding,

01:40:38.680 | it may run flat, but I don't know.

01:40:40.080 | - No, I'm pretty confident that majority of the world,

01:40:45.080 | even if you tell them how it works--

01:40:46.680 | - Won't matter.

01:40:47.520 | - Well, it will not matter,

01:40:49.120 | especially if the machine herself says

01:40:53.080 | that she is conscious.

01:40:55.400 | - That's very possible.

01:40:56.240 | - So you, the scientist that made the machine,

01:40:58.560 | is saying that this is how the algorithm works.

01:41:02.820 | Everybody will just assume you're lying

01:41:04.400 | and that there's a conscious being there.

01:41:06.080 | - So you're deep into the science fiction genre now,

01:41:09.200 | but yeah, I know--

01:41:10.040 | - I don't think it's, it's actually psychology.

01:41:12.000 | I think it's not science fiction.

01:41:13.720 | I think it's reality.

01:41:14.880 | I think it's a really powerful one

01:41:16.780 | that we'll have to be exploring in the next few decades.

01:41:20.000 | - I agree.

01:41:20.840 | - It's a very interesting element of intelligence.

01:41:23.560 | So what do you think,

01:41:25.200 | we've talked about social constructs of intelligence

01:41:28.520 | and frameworks and the way humans

01:41:31.160 | kind of interpret information.

01:41:33.960 | What do you think is a good test of intelligence

01:41:35.720 | in your view?

01:41:36.540 | So there's the Alan Turing with the Turing test.

01:41:41.320 | Watson accomplished something very impressive with Jeopardy.

01:41:44.920 | What do you think is a test

01:41:46.760 | that would impress the heck out of you,

01:41:49.720 | that you saw that a computer could do?

01:41:52.940 | They would say, this is crossing a kind of threshold

01:41:57.240 | that gives me pause in a good way.

01:42:00.620 | - My expectations for AI are generally high.

01:42:06.060 | - What does high look like, by the way?

01:42:07.360 | So not the threshold, test is a threshold.

01:42:10.360 | What do you think is the destination?

01:42:12.480 | What do you think is the ceiling?

01:42:15.520 | - I think machines will, in many measures,

01:42:18.520 | will be better than us, will become more effective.

01:42:21.680 | In other words, better predictors about a lot of things

01:42:25.160 | than ultimately we can do.

01:42:28.580 | I think where they're gonna struggle

01:42:30.840 | is what we've talked about before,

01:42:32.280 | which is relating to, communicating with,

01:42:36.580 | and understanding humans in deeper ways.

01:42:40.620 | And so I think that's a key point.

01:42:42.480 | Like, we can create the super parrot.

01:42:44.840 | What I mean by the super parrot is,

01:42:46.760 | given enough data, a machine can mimic

01:42:48.640 | your emotional response, can even generate language

01:42:51.480 | that will sound smart, and what someone else might say

01:42:54.600 | under similar circumstances.

01:42:56.420 | Like, I would just pause on that.

01:42:58.960 | Like, that's the super parrot, right?

01:43:01.240 | So, given similar circumstances,

01:43:03.700 | moves its faces in similar ways,

01:43:06.960 | changes its tone of voice in similar ways,

01:43:09.480 | produces strings of language that, you know,

01:43:11.860 | would similar that a human might say,

01:43:14.320 | not necessarily being able to produce

01:43:16.760 | a logical interpretation or understanding

01:43:19.440 | that would ultimately satisfy a critical interrogation

01:43:25.320 | or a critical understanding.

01:43:26.780 | - I think you just described me in a nutshell.

01:43:30.440 | So I think philosophically speaking,

01:43:34.400 | you could argue that that's all we're doing

01:43:36.560 | as human beings, too.

01:43:37.520 | We're super parrots. - So I was gonna say,

01:43:39.080 | it's very possible, you know,

01:43:40.460 | humans do behave that way, too.

01:43:42.580 | And so upon deeper probing and deeper interrogation,

01:43:45.820 | you may find out that there isn't a shared understanding,

01:43:48.900 | because I think humans do both.

01:43:50.300 | Like, humans are statistical language model machines,

01:43:53.200 | and they are capable reasoners.

01:43:57.620 | You know, they're both.

01:43:59.860 | And you don't know which is going on, right?

01:44:02.860 | So, and I think it's an interesting problem,

01:44:08.840 | we talked earlier about, like,

01:44:10.520 | where we are in our social and political landscape.

01:44:14.720 | Can you distinguish someone who can string words together

01:44:19.560 | and sound like they know what they're talking about

01:44:21.800 | from someone who actually does?

01:44:24.040 | Can you do that without dialogue,

01:44:25.640 | without interrogative or probing dialogue?

01:44:27.740 | So it's interesting, because humans are really good

01:44:32.400 | at, in their own mind, justifying or explaining

01:44:35.000 | what they hear, because they project

01:44:37.400 | their understanding onto yours.

01:44:39.880 | So you could say, you could put together a string of words,

01:44:42.960 | and someone will sit there and interpret it

01:44:45.120 | in a way that's extremely biased

01:44:47.080 | to the way they want to interpret it.

01:44:48.200 | They want to assume that you're an idiot,

01:44:49.440 | and they'll interpret it one way.

01:44:51.080 | They will assume you're a genius,

01:44:52.440 | and they'll interpret it another way that suits their needs.

01:44:55.400 | So this is tricky business.

01:44:57.920 | So I think to answer your question,

01:45:00.580 | as AI gets better and better at better and better mimic,

01:45:03.560 | we create the super parrots,

01:45:05.900 | we're challenged, just as we are with,

01:45:08.020 | we're challenged with humans.

01:45:09.580 | Do you really know what you're talking about?

01:45:12.100 | Do you have a meaningful interpretation,

01:45:16.420 | a powerful framework that you could reason over

01:45:19.420 | and justify your answers, justify your predictions

01:45:24.420 | and your beliefs, why you think they make sense?

01:45:27.140 | Can you convince me what the implications are?

01:45:29.500 | You know, can you, so can you reason intelligently

01:45:33.620 | and make me believe that those,

01:45:36.160 | the implications of your prediction and so forth?

01:45:41.380 | So what happens is it becomes reflective.

01:45:44.180 | My standard for judging your intelligence

01:45:47.500 | depends a lot on mine.

01:45:48.800 | - But you're saying that there should be

01:45:53.800 | a large group of people with a certain standard

01:45:56.380 | of intelligence that would be convinced

01:45:58.620 | by this particular AI system,

01:46:02.580 | then it would pass.

01:46:03.540 | - There should be, but I think one of the,

01:46:05.380 | depending on the content,

01:46:07.660 | one of the problems we have there is that

01:46:09.900 | if that large community of people are not judging it

01:46:14.220 | with regard to a rigorous standard

01:46:16.600 | of objective logic and reason, you still have a problem.

01:46:19.500 | Like masses of people can be persuaded.

01:46:23.780 | - The millennials, yeah.

01:46:24.980 | - To turn their brains off.

01:46:27.640 | - Right, okay.

01:46:31.980 | - Sorry.

01:46:32.820 | - By the way, I have nothing against the one.

01:46:33.660 | - No, I don't know, I'm just,

01:46:36.060 | so you're a part of one of the great benchmarks,

01:46:40.980 | challenges of AI history.

01:46:43.260 | What do you think about AlphaZero, OpenAI5,

01:46:47.220 | AlphaStar accomplishments on video games recently,

01:46:50.740 | which are also, I think, at least in the case of Go,

01:46:55.300 | with AlphaGo and AlphaZero playing Go

01:46:57.180 | was a monumental accomplishment as well.

01:46:59.700 | What are your thoughts about that challenge?

01:47:01.740 | - I think it was a giant landmark for AI.

01:47:03.460 | I think it was phenomenal.

01:47:04.500 | I mean, it was one of those other things

01:47:06.020 | nobody thought like solving Go was gonna be easy,

01:47:08.540 | particularly 'cause it's hard for,

01:47:10.460 | particularly hard for humans,

01:47:12.700 | hard for humans to learn, hard for humans to excel at.

01:47:15.540 | And so it was another measure of intelligence.

01:47:19.600 | It's very cool.

01:47:22.500 | I mean, it's very interesting what they did.

01:47:24.980 | I mean, and I loved how they solved the data problem,

01:47:27.940 | which is, again, they bootstrapped it

01:47:29.180 | and got the machine to play itself

01:47:30.420 | to generate enough data to learn from.

01:47:32.740 | I think that was brilliant.

01:47:33.860 | I think that was great.

01:47:35.700 | And of course the result speaks for itself.

01:47:38.940 | I think it makes us think about, again,

01:47:41.900 | okay, what's intelligence?

01:47:42.980 | What aspects of intelligence are important?

01:47:45.580 | Can the Go machine help me make me a better Go player?

01:47:49.380 | Is it an alien intelligence?

01:47:51.700 | Am I even capable of, like, again,

01:47:54.380 | if we put in very simple terms, it found the function.

01:47:57.460 | It found the Go function.

01:47:59.220 | Can I even comprehend the Go function?

01:48:00.860 | Can I talk about the Go function?

01:48:02.300 | Can I conceptualize the Go function,

01:48:03.940 | like whatever it might be?

01:48:05.540 | - So one of the interesting ideas of that system

01:48:08.080 | is that it plays against itself, right?

01:48:10.100 | But there's no human in the loop there.

01:48:12.700 | So like you're saying, it could have, by itself,

01:48:16.520 | created an alien intelligence.

01:48:18.480 | - Toward a goal, like, imagine you're sentencing,

01:48:21.860 | you're a judge and you're sentencing people,

01:48:24.740 | or you're setting policy,

01:48:26.460 | or you're making medical decisions,

01:48:31.220 | and you can't explain.

01:48:33.380 | You can't get anybody to understand

01:48:34.940 | what you're doing or why.

01:48:36.220 | So it's an interesting dilemma for the applications of AI.

01:48:42.340 | Do we hold AI to this accountability that says,

01:48:48.060 | you know, humans have to be able to take responsibility

01:48:52.900 | for the decision.

01:48:56.380 | In other words, can you explain why you would do the thing?

01:48:58.780 | Will you get up and speak to other humans

01:49:02.040 | and convince them that this was a smart decision?

01:49:04.660 | Is the AI enabling you to do that?

01:49:07.180 | Can you get behind the logic that was made there?

01:49:10.220 | - Do you think, sorry to linger on this point,

01:49:13.420 | 'cause it's a fascinating one.

01:49:15.420 | It's a great goal for AI.

01:49:17.540 | Do you think it's achievable in many cases?

01:49:21.460 | Or, okay, there's two possible worlds

01:49:23.880 | that we have in the future.

01:49:25.820 | One is where AI systems do like medical diagnosis

01:49:28.940 | or things like that, or drive a car,

01:49:32.420 | without ever explaining to you why it fails when it does.

01:49:36.600 | That's one possible world, and we're okay with it.

01:49:40.380 | Or the other, where we are not okay with it,

01:49:42.980 | and we really hold back the technology

01:49:45.380 | from getting too good before it gets able to explain.

01:49:48.780 | Which of those worlds are more likely, do you think,

01:49:50.780 | and which are concerning to you or not?

01:49:53.500 | - I think the reality is it's gonna be a mix.

01:49:55.940 | I'm not sure I have a problem with that.

01:49:57.460 | I mean, I think there are tasks that I'm perfectly fine with

01:50:00.460 | machines show a certain level of performance,

01:50:03.980 | and that level of performance is already better than humans.

01:50:07.740 | So, for example, I don't know that I,

01:50:10.100 | take driverless cars.

01:50:11.300 | If driverless cars learn how to be more effective drivers

01:50:14.340 | than humans, but can't explain what they're doing,

01:50:16.920 | but bottom line, statistically speaking,

01:50:19.060 | they're 10 times safer than humans,

01:50:22.420 | I don't know that I care.

01:50:24.980 | I think when we have these edge cases,

01:50:27.580 | when something bad happens and we wanna decide

01:50:29.740 | who's liable for that thing, and who made that mistake,

01:50:32.580 | and what do we do about that?

01:50:33.540 | And I think those edge cases are interesting cases.

01:50:36.740 | And now do we go to designers of the AI,

01:50:38.940 | and the AI says, I don't know,

01:50:39.780 | if that's what it learned to do?

01:50:41.060 | And it says, well, you didn't train it properly.

01:50:43.620 | You know, you were negligent in the training data

01:50:46.740 | that you gave that machine.

01:50:47.820 | Like, how do we drive down the reliability?

01:50:49.420 | So I think those are interesting questions.

01:50:52.080 | - So the optimization problem there, sorry,

01:50:55.300 | is to create an AI system that's able

01:50:56.900 | to explain the lawyers away.

01:50:58.820 | - Yeah, there you go.

01:51:01.620 | I think that, I think it's gonna be interesting.

01:51:04.020 | I mean, I think this is where technology

01:51:05.820 | and social discourse are gonna get deeply intertwined

01:51:09.500 | in how we start thinking about problems,

01:51:11.900 | decisions, and problems like that.

01:51:13.500 | I think in other cases, it becomes more obvious

01:51:15.860 | where, you know, it's like,

01:51:18.120 | like, why did you decide to give that person,

01:51:21.180 | you know, a longer sentence, or deny them parole?

01:51:26.060 | Again, policy decisions, or why did you pick that treatment?

01:51:30.540 | Like, that treatment ended up killing that guy.

01:51:32.260 | Like, why was that a reasonable choice to make?

01:51:35.060 | So, and people are gonna demand explanations.

01:51:40.060 | Now, there's a reality, though, here.

01:51:43.460 | And the reality is that it's not,

01:51:45.940 | I'm not sure humans are making reasonable choices

01:51:48.580 | when they do these things.

01:51:49.900 | They are using statistical hunches, biases,

01:51:54.740 | or even systematically using statistical averages

01:51:58.460 | to make calls.

01:51:59.300 | I mean, this is what happened to my dad,

01:52:00.340 | and if you saw the talk I gave about that.

01:52:01.900 | But, you know, I mean, they decided

01:52:04.940 | that my father was brain dead.

01:52:07.260 | He had went into cardiac arrest,

01:52:09.300 | and it took a long time for the ambulance to get there,

01:52:12.380 | and he was not resuscitated right away, and so forth.

01:52:14.540 | And they came, they told me he was brain dead.

01:52:16.900 | And why was he brain dead?

01:52:17.860 | Because essentially, they gave me

01:52:19.060 | a purely statistical argument.

01:52:21.060 | Under these conditions, with these four features,

01:52:23.820 | 98% chance he's brain dead.

01:52:25.300 | And I said, but can you just tell me,

01:52:27.740 | not inductively, but deductively,

01:52:29.660 | go there and tell me his brain's not functioning

01:52:31.380 | as the way for you to do that?

01:52:32.820 | And the protocol and response was,

01:52:35.980 | no, this is how we make this decision.

01:52:37.980 | I said, this is inadequate for me.

01:52:39.740 | I understand the statistics,

01:52:41.140 | and I don't know how, you know,

01:52:43.100 | there's a 2% chance he's still alive.

01:52:44.540 | Like, I just don't know the specifics.

01:52:46.500 | I need the specifics of this case,

01:52:49.380 | and I want the deductive, logical argument

01:52:51.420 | about why you actually know he's brain dead.

01:52:53.580 | So I wouldn't sign the do not resuscitate.

01:52:55.980 | And, I don't know, it was like,

01:52:57.260 | they went through lots of procedures.

01:52:58.700 | It was a big, long story.

01:53:00.020 | But the bottom, a fascinating story, by the way,

01:53:02.060 | about how I reasoned, and how the doctors reasoned

01:53:04.340 | through this whole process.

01:53:05.980 | But I don't know, somewhere around 24 hours later

01:53:07.900 | or something, he was sitting up in bed

01:53:09.460 | with zero brain damage.

01:53:11.020 | - I mean, what lessons do you draw from that story,

01:53:18.020 | that experience?

01:53:19.500 | - That the data that's being used

01:53:22.700 | to make statistical inferences

01:53:24.100 | doesn't adequately reflect the phenomenon.

01:53:26.460 | So in other words, you're getting shit wrong, sorry.

01:53:29.180 | You're getting stuff wrong

01:53:31.740 | because your model's not robust enough,

01:53:35.260 | and you might be better off

01:53:38.580 | not using statistical inference

01:53:41.340 | and statistical averages in certain cases

01:53:43.080 | when you know the model's insufficient,

01:53:45.220 | and that you should be reasoning it about the specific case

01:53:48.460 | more logically and more deductively,

01:53:51.060 | and hold yourself responsible,

01:53:52.500 | hold yourself accountable to doing that.

01:53:54.500 | - And perhaps AI has a role to say

01:53:58.700 | the exact thing we just said,

01:54:00.740 | which is, perhaps this is a case

01:54:03.020 | you should think for yourself.

01:54:05.460 | You should reason deductively.

01:54:08.140 | - Well, so it's hard because it's hard to know that.

01:54:13.140 | You'd have to go back

01:54:15.740 | and you'd have to have enough data to essentially say,

01:54:18.260 | and this goes back to the case of how do we decide

01:54:22.060 | whether AI is good enough to do a particular task?

01:54:24.560 | And regardless of whether or not

01:54:27.340 | it produces an explanation.

01:54:28.700 | So, and what standards do we hold, right, for that?

01:54:35.020 | So, if you look more broadly, for example,

01:54:40.020 | as my father, as a medical case,

01:54:45.980 | the medical system ultimately helped him

01:54:49.860 | a lot throughout his life.

01:54:51.540 | Without it, he probably would have died much sooner.

01:54:54.720 | So overall, it's sort of worked for him

01:54:58.940 | in sort of a net-net kind of way.

01:55:00.800 | Actually, I don't know that that's fair,

01:55:04.860 | but maybe not in that particular case, but overall.

01:55:08.140 | The medical system overall does more good than bad.

01:55:10.980 | - Yeah, the medical system overall

01:55:12.460 | was doing more good than bad.

01:55:14.340 | Now, there's another argument that suggests

01:55:16.580 | that that wasn't the case,

01:55:17.420 | but for the sake of argument,

01:55:18.620 | let's say that's a net positive.

01:55:21.060 | And I think you have to sit there

01:55:22.260 | and take that into consideration.

01:55:24.860 | Now you look at a particular use case,

01:55:26.700 | like for example, making this decision.

01:55:28.920 | Have you done enough studies to know

01:55:33.440 | how good that prediction really is?

01:55:35.700 | And have you done enough studies to compare it?

01:55:40.120 | To say, well, what if we dug in in a more direct,

01:55:45.120 | let's get the evidence, let's do the deductive thing

01:55:48.040 | and not use statistics here.

01:55:49.480 | How often would that have done better?

01:55:51.640 | So you have to do the studies

01:55:53.760 | to know how good the AI actually is.

01:55:56.200 | And it's complicated,

01:55:57.600 | because it depends how fast you have to make decision.

01:55:59.600 | So if you have to make decisions super fast,

01:56:02.400 | you have no choice.

01:56:03.360 | If you have more time,

01:56:06.800 | but if you're ready to pull the plug,

01:56:09.080 | and this is a lot of the argument that I had with a doctor,

01:56:11.520 | I said, what's he gonna do if you do it,

01:56:13.260 | what's gonna happen to him in that room

01:56:15.280 | if you do it my way?

01:56:16.400 | Well, he's gonna die anyway,

01:56:18.800 | so let's do it my way then.

01:56:20.120 | - I mean, it raises questions for our society

01:56:22.880 | to struggle with, as is the case with your father,

01:56:26.560 | but also when things like race and gender

01:56:28.700 | start coming into play,

01:56:31.760 | when judgments are made based on things

01:56:35.640 | that are complicated in our society,

01:56:39.040 | at least in the discourse.

01:56:40.120 | And it starts, I think I'm safe to say

01:56:44.000 | that most of the violent crime is committed by males.

01:56:47.340 | So if you discriminate based,

01:56:51.080 | it's a male versus female saying that

01:56:53.880 | if it's a male, more likely to commit the crime.

01:56:56.160 | - So this is one of my very positive and optimistic views

01:57:01.040 | of why the study of artificial intelligence,

01:57:05.520 | the process of thinking and reasoning,

01:57:08.000 | logically and statistically, and how to combine them

01:57:10.520 | is so important for the discourse today,

01:57:12.200 | because it's causing a, regardless of what state AI devices

01:57:17.200 | are or not, it's causing this dialogue to happen.

01:57:22.240 | This is one of the most important dialogues

01:57:24.840 | that, in my view, the human species can have right now,

01:57:28.200 | which is how to think well, how to reason well,

01:57:33.200 | how to understand our own cognitive biases

01:57:38.840 | and what to do about them.

01:57:41.000 | That has got to be one of the most important things

01:57:43.640 | we as a species can be doing, honestly.

01:57:46.760 | We've created an incredibly complex society.

01:57:51.200 | We've created amazing abilities to amplify noise

01:57:55.840 | faster than we can amplify signal.

01:57:58.420 | We are challenged.

01:58:01.280 | We are deeply, deeply challenged.

01:58:03.680 | We have big segments of the population

01:58:06.320 | getting hit with enormous amounts of information.

01:58:08.960 | Do they know how to do critical thinking?

01:58:11.000 | Do they know how to objectively reason?

01:58:14.240 | Do they understand what they are doing,

01:58:17.000 | nevermind what their AI is doing?

01:58:18.800 | This is such an important dialogue to be having.

01:58:23.200 | And we are fundamentally, our thinking can be

01:58:27.960 | and easily becomes fundamentally biased.

01:58:31.440 | And there are statistics, and we shouldn't blind ourselves,

01:58:34.480 | we shouldn't discard statistical inference,

01:58:37.320 | but we should understand the nature of statistical inference.

01:58:40.920 | As a society, we decide to reject statistical inference,

01:58:48.240 | to favor understanding and deciding on the individual.

01:58:53.240 | We consciously make that choice.

01:59:00.680 | So even if the statistics said,

01:59:03.240 | even if the statistics said males are more likely

01:59:07.720 | to be violent criminals,

01:59:09.720 | we still take each person as an individual,

01:59:12.800 | and we treat them based on the logic

01:59:16.880 | and the knowledge of that situation.

01:59:20.320 | We purposefully and intentionally

01:59:22.960 | reject the statistical inference.

01:59:27.480 | We do that out of respect for the individual.

01:59:31.280 | - For the individual, yeah,

01:59:32.200 | and that requires reasoning and thinking.

01:59:35.200 | Looking forward, what grand challenges

01:59:37.480 | would you like to see in the future?

01:59:39.000 | Because the Jeopardy challenge captivated the world,

01:59:45.200 | AlphaGo, AlphaZero captivated the world,

01:59:48.080 | Deep Blue certainly beating Kasparov,

01:59:50.280 | Gary's bitterness aside, captivated the world.

01:59:55.720 | What do you think, do you have ideas

01:59:57.880 | for next grand challenges for future challenges of that?

02:00:01.400 | - Look, I mean, I think there are lots

02:00:03.280 | of really great ideas for grand challenges.

02:00:05.800 | I'm particularly focused on one right now,

02:00:08.480 | which is can you demonstrate that they understand,

02:00:12.520 | that they could read and understand,

02:00:14.960 | that they can acquire these frameworks

02:00:18.000 | and reason and communicate with humans?

02:00:21.160 | So it is kind of like the Turing test,

02:00:23.360 | but it's a little bit more demanding than the Turing test.

02:00:26.560 | It's not enough to convince me that you might be human

02:00:31.280 | because you can parrot a conversation.

02:00:34.920 | I think the standard is a little bit higher.

02:00:38.480 | For example, can you, the standard is higher,

02:00:43.400 | and I think one of the challenges

02:00:45.560 | of devising this grand challenge

02:00:48.280 | is that we're not sure what intelligence is.

02:00:53.280 | We're not sure how to determine

02:00:56.240 | whether or not two people actually understand each other

02:00:59.160 | and in what depth they understand it,

02:01:01.120 | to what depth they understand each other.

02:01:04.400 | So the challenge becomes something along the lines

02:01:07.440 | of can you satisfy me that we have a shared purpose

02:01:12.600 | we have a shared understanding?

02:01:14.800 | So if I were to probe and probe and you probe me,

02:01:18.400 | can machines really act like thought partners

02:01:23.400 | where they can satisfy me that we have a shared,

02:01:27.320 | our understanding is shared enough

02:01:29.400 | that we can collaborate and produce answers together

02:01:33.320 | and that they can help me explain and justify those answers.

02:01:36.760 | - So maybe here's an idea.

02:01:38.120 | So we'll have AI system run for president and convince--

02:01:43.120 | - That's too easy.

02:01:46.120 | I'm sorry, go ahead.

02:01:46.960 | - You have to convince the voters that they should vote.

02:01:51.600 | So I guess what does winning look like?

02:01:53.800 | - Again, that's why I think this is such a challenge

02:01:55.920 | because we go back to the emotional persuasion.

02:02:00.040 | We go back to, now we're checking off an aspect

02:02:06.080 | of human cognition that is in many ways weak or flawed.

02:02:11.080 | We're so easily manipulated.

02:02:13.960 | Our minds are drawn for often the wrong reasons.

02:02:18.960 | Not the reasons that ultimately matter to us,

02:02:21.880 | but the reasons that can easily persuade us.

02:02:24.000 | I think we can be persuaded to believe one thing or another

02:02:28.440 | for reasons that ultimately don't serve us well

02:02:31.360 | in the long term.

02:02:33.200 | And a good benchmark should not play with those elements

02:02:38.200 | of emotional manipulation.

02:02:40.800 | - I don't think so.

02:02:41.640 | And I think that's where we have to set the higher standard

02:02:44.360 | for ourselves of what does it mean.

02:02:47.160 | This goes back to rationality

02:02:48.920 | and it goes back to objective thinking.

02:02:50.680 | Can you produce, can you acquire information

02:02:53.360 | and produce reasoned arguments?

02:02:54.840 | And to those reasoned arguments,

02:02:56.360 | pass a certain amount of muster.

02:02:58.000 | And can you acquire new knowledge?

02:03:02.600 | Can you, for example, can you reason,

02:03:06.280 | I have acquired new knowledge,

02:03:07.480 | can you identify where it's consistent or contradictory

02:03:11.240 | with other things you've learned?

02:03:12.920 | And can you explain that to me

02:03:14.080 | and get me to understand that?

02:03:15.600 | So I think another way to think about it perhaps

02:03:18.520 | is can a machine teach you?

02:03:22.780 | Can it help you-- - Oh, that's a really nice,

02:03:28.640 | nice and then nice way to put it.

02:03:29.720 | Can it help you understand--

02:03:30.560 | - Can it help you understand something

02:03:32.440 | that you didn't really understand before?

02:03:34.920 | - Oh, that's a beautiful way to put it.

02:03:35.760 | - Where it's taking you, so you're not,

02:03:39.120 | again, it's almost like, can it teach you?

02:03:41.360 | Can it help you learn?

02:03:43.480 | And in an arbitrary space,

02:03:47.040 | so it can open those domain space.

02:03:49.000 | So can you tell the machine,

02:03:50.120 | and again, this borrows from some science fictions,

02:03:52.840 | but can you go off and learn about this topic

02:03:55.820 | that I'd like to understand better

02:03:58.440 | and then work with me to help me understand it?

02:04:00.840 | - That's quite brilliant.

02:04:03.640 | Well, a machine that passes that kind of test,

02:04:06.960 | do you think it would need to have self-awareness

02:04:11.560 | or even consciousness?

02:04:13.160 | What do you think about consciousness

02:04:16.160 | and the importance of it,

02:04:17.840 | maybe in relation to having a body,

02:04:21.120 | having a presence, an entity?

02:04:24.760 | Do you think that's important?

02:04:26.960 | - People used to ask me if Watson was conscious,

02:04:28.760 | and I used to say, conscious of what exactly?

02:04:32.280 | I mean, I think-- - Of self.

02:04:34.280 | - It depends what it is that you're conscious of.

02:04:36.080 | I mean, so did it,

02:04:38.520 | it's certainly easy for it to answer questions about,

02:04:42.800 | it would be trivial to program it

02:04:44.800 | so that it'd answer questions about

02:04:46.120 | whether or not it was playing Jeopardy.

02:04:47.580 | I mean, it could certainly answer questions

02:04:49.040 | that would imply that it was aware of things.

02:04:51.280 | Exactly, what does it mean to be aware

02:04:52.680 | and what does it mean to consciousness?

02:04:53.520 | It's sort of interesting.

02:04:54.440 | I mean, I think that we differ from one another

02:04:58.000 | based on what we're conscious of.

02:04:59.840 | - But wait, wait, yes, for sure.

02:05:02.680 | There's degrees of consciousness in there, so--

02:05:05.320 | - Well, and there's just areas.

02:05:06.960 | Like, it's not just degrees.

02:05:08.440 | What are you aware of?

02:05:10.160 | Like, what are you not aware of?

02:05:11.160 | - But nevertheless, there's a very subjective element

02:05:13.440 | to our experience.

02:05:14.740 | Let me even not talk about consciousness.

02:05:18.400 | Let me talk about another, to me,

02:05:21.600 | really interesting topic of mortality,

02:05:23.640 | fear of mortality.

02:05:25.600 | Watson, as far as I could tell,

02:05:29.320 | did not have a fear of death.

02:05:30.980 | - Certainly not.

02:05:33.040 | - Most humans do.

02:05:35.920 | - Wasn't conscious of death.

02:05:39.080 | - He wasn't, yeah, so there's an element of finiteness

02:05:42.880 | to our existence that I think, like you mentioned,

02:05:46.160 | survival, that adds to the whole thing.

02:05:49.040 | I mean, consciousness is tied up with that,

02:05:50.880 | that we are a thing.

02:05:52.880 | It's a subjective thing that ends,

02:05:56.200 | and that seems to add a color and flavor

02:05:59.000 | to our motivations in a way that seems

02:06:01.640 | to be fundamentally important for intelligence,

02:06:05.960 | or at least the kind of human intelligence.

02:06:07.880 | - Well, I think for generating goals.

02:06:09.800 | Again, I think you could have an intelligence capability

02:06:14.520 | and a capability to learn, a capability to predict,

02:06:18.520 | but I think without, I mean, again, you get a fear,

02:06:23.520 | but essentially without the goal to survive.

02:06:27.000 | - So you think you can just encode that

02:06:29.080 | without having to really--

02:06:29.920 | - I think you can encode that.

02:06:30.920 | I mean, you can create a robot now,

02:06:32.800 | and you could say, plug it in,

02:06:36.000 | and say, protect your power source,

02:06:38.520 | and give it some capabilities,

02:06:39.720 | and it'll sit there and operate

02:06:40.880 | to try to protect its power source and survive.

02:06:42.760 | I mean, so I don't know that that's philosophically

02:06:45.680 | a hard thing to demonstrate.

02:06:46.680 | It sounds like a fairly easy thing to demonstrate

02:06:49.000 | that you can give it that goal.

02:06:50.080 | Well, it'll come up with that goal by itself,

02:06:52.400 | and I think you have to program that goal in.

02:06:54.560 | - But there's something, because I think,

02:06:57.840 | as we touched on, intelligence is kind of

02:06:59.680 | like a social construct.

02:07:01.520 | The fact that a robot will be protecting its power source

02:07:06.280 | would add depth and grounding to its intelligence

02:07:12.560 | in terms of us being able to respect it.

02:07:15.840 | I mean, ultimately, it boils down to us

02:07:18.280 | acknowledging that it's intelligent,

02:07:20.680 | and the fact that it can die,

02:07:23.520 | I think, is an important part of that.

02:07:26.120 | - The interesting thing to reflect on

02:07:27.840 | is how trivial that would be,

02:07:29.520 | and I don't think if you knew how trivial that was,

02:07:32.120 | you would associate that with being intelligence.

02:07:35.400 | I mean, I literally put in a statement of code

02:07:37.480 | that says you have the following actions you can take.

02:07:40.440 | You give it a bunch of actions,

02:07:41.640 | like maybe you mount a laser gun on it,

02:07:44.000 | or you have the ability to scream or screech or whatever,

02:07:48.960 | and you say, if you see your power source threatened,

02:07:52.720 | then you could program that in,

02:07:53.920 | and you're gonna take these actions to protect it.

02:07:58.060 | You could train it on a bunch of things.

02:08:02.200 | So, and now you're gonna look at that,

02:08:03.800 | and you're gonna say, well, that's intelligence,

02:08:05.280 | because it's protecting its power source.

02:08:06.840 | Maybe, but that's, again, this human bias that says,

02:08:10.220 | the thing I, I identify my intelligence and my conscience

02:08:14.580 | so fundamentally with the desire,

02:08:16.740 | or at least the behaviors associated

02:08:18.660 | with the desire to survive,

02:08:20.400 | that if I see another thing doing that,

02:08:23.860 | I'm going to assume it's intelligence.

02:08:27.280 | - What timeline year will society have a,

02:08:31.080 | something that would, that you would be comfortable calling

02:08:36.020 | an artificial general intelligence system?

02:08:39.560 | What's your intuition?

02:08:41.060 | Nobody can predict the future,

02:08:42.460 | certainly not the next few months or 20 years away,

02:08:46.460 | but what's your intuition?

02:08:47.620 | How far away are we?

02:08:48.900 | - I don't know.

02:08:50.940 | It's hard to make these predictions.

02:08:52.100 | I mean, I would be, you know, I would be guessing,

02:08:54.740 | and there's so many different variables,

02:08:57.000 | including just how much we want to invest in it,

02:08:59.100 | and how important it, you know,

02:09:00.380 | and how important we think it is,

02:09:02.080 | what kind of investment we're willing to make in it,

02:09:06.140 | what kind of talent we end up bringing to the table,

02:09:07.860 | all, you know, the incentive structure,

02:09:09.200 | all these things.

02:09:10.140 | So I think it is possible to do this sort of thing.

02:09:15.140 | I think it's, I think trying to sort of ignore many

02:09:20.180 | of the variables and things like that,

02:09:23.000 | is it a 10-year thing?

02:09:24.020 | Is it a 20-year?

02:09:25.380 | It's probably closer to a 20-year thing, I guess.

02:09:27.860 | - But not several hundred years.

02:09:29.680 | - No, I don't think it's several hundred years.

02:09:32.060 | I don't think it's several hundred years,

02:09:33.620 | but again, so much depends on how committed we are

02:09:38.820 | to investing and incentivizing this type of work.

02:09:43.080 | And it's sort of interesting,

02:09:45.160 | like I don't think it's obvious how incentivized we are.

02:09:50.160 | I think from a task perspective, you know,

02:09:54.380 | if we see business opportunities to take this technique

02:09:57.840 | or that technique to solve that problem,

02:09:59.120 | I think that's the main driver for many of these things.

02:10:03.220 | From a general intelligence,

02:10:05.520 | it's kind of an interesting question.

02:10:06.920 | Are we really motivated to do that?

02:10:09.560 | And like we just struggled ourselves right now

02:10:12.520 | to even define what it is.

02:10:14.760 | So it's hard to incentivize when we don't even know

02:10:16.920 | what it is we're incentivized to create.

02:10:18.800 | And if you said mimic a human intelligence,

02:10:21.320 | I just think there are so many challenges

02:10:25.520 | with the significance and meaning of that,

02:10:27.720 | that there's not a clear directive.

02:10:29.640 | There's no clear directive to do precisely that thing.

02:10:32.280 | - So assistance in a larger and larger number of tasks.

02:10:36.480 | So being able to, a system that's particularly able

02:10:39.600 | to operate my microwave and making a grilled cheese sandwich,

02:10:42.640 | I don't even know how to make one of those.

02:10:45.000 | And then the same system would be doing the vacuum cleaning.

02:10:48.040 | And then the same system would be teaching

02:10:51.680 | my kids that I don't have math.

02:10:56.300 | - I think that when you get into a general intelligence

02:11:00.760 | for learning physical tasks,

02:11:04.280 | and again, I wanna go back to your body question,

02:11:06.080 | 'cause I think your body question was interesting,

02:11:07.320 | but you wanna go back to learning the abilities

02:11:11.120 | to do physical tasks.

02:11:11.960 | You might have, we might get,

02:11:14.480 | I imagine in that timeframe,

02:11:16.080 | we will get better and better

02:11:17.480 | at learning these kinds of tasks,

02:11:19.040 | whether it's mowing your lawn or driving a car

02:11:21.280 | or whatever it is.

02:11:22.760 | I think we will get better and better at that

02:11:24.480 | where it's learning how to make predictions

02:11:25.880 | over large bodies of data.

02:11:27.040 | I think we're gonna continue

02:11:27.880 | to get better and better at that.

02:11:29.480 | And machines will outpace humans

02:11:33.560 | in a variety of those things.

02:11:35.600 | The underlying mechanisms for doing that may be the same,

02:11:40.600 | meaning that maybe these are deep nets,

02:11:43.700 | there's infrastructure to train them,

02:11:46.280 | reusable components to get them

02:11:48.120 | to do different classes of tasks,

02:11:50.960 | and we get better and better

02:11:51.920 | at building these kinds of machines.

02:11:53.980 | You could still argue

02:11:55.040 | that the general learning infrastructure in there

02:11:57.040 | is a form of a general type of intelligence.

02:12:01.040 | I think what starts getting harder

02:12:03.280 | is this notion of,

02:12:04.720 | can we effectively communicate and understand

02:12:09.120 | and build that shared understanding

02:12:10.840 | because of the layers of interpretation

02:12:12.640 | that are required to do that,

02:12:14.300 | and the need for the machine

02:12:15.640 | to be engaged with humans at that level

02:12:18.200 | in a continuous basis.

02:12:20.320 | So how do you get in there?

02:12:21.440 | How do you get the machine in the game?

02:12:23.480 | How do you get the machine in the intellectual game?

02:12:26.600 | - Yeah, and to solve AGI,

02:12:29.120 | you probably have to solve that problem.

02:12:31.000 | - You have to get the machine.

02:12:31.960 | So it's a little bit of a bootstrapping thing.

02:12:33.800 | Can we get the machine engaged in the intellectual,

02:12:38.040 | I'm calling it a game,

02:12:39.160 | but in the intellectual dialogue with the humans?

02:12:42.360 | Are the humans sufficiently

02:12:43.960 | in intellectual dialogue with each other

02:12:45.600 | to generate enough data in this context?

02:12:49.640 | And how do you bootstrap that?

02:12:51.020 | Because every one of those conversations,

02:12:54.080 | every one of those conversations,

02:12:55.760 | those intelligent interactions

02:12:58.040 | require so much prior knowledge

02:12:59.680 | that it's a challenge to bootstrap it.

02:13:01.660 | So the question is, and how committed?

02:13:05.840 | So I think that's possible,

02:13:07.300 | but when I go back to, are we incentivized to do that?

02:13:10.880 | I know we're incentivized to do the former.

02:13:13.160 | Are we incentivized to do the latter significantly enough?

02:13:15.880 | Do people understand

02:13:16.720 | what the latter really is well enough?

02:13:18.460 | Part of the elemental cognition mission

02:13:20.860 | is to try to articulate that better and better

02:13:23.520 | through demonstrations

02:13:24.560 | and through trying to craft these grand challenges

02:13:26.960 | and get people to say,

02:13:27.920 | look, this is a class of intelligence.

02:13:30.440 | This is a class of AI.

02:13:31.840 | Do we want this?

02:13:33.420 | What is the potential of this?

02:13:35.820 | What's the business potential?

02:13:37.840 | What's the societal potential to that?

02:13:40.120 | And to build up that incentive system around that.

02:13:45.080 | - Yeah, I think if people don't understand yet,

02:13:46.840 | I think they will.

02:13:47.680 | I think there's a huge business potential here.

02:13:49.600 | So it's exciting that you're working on it.

02:13:52.020 | We kind of skipped over,

02:13:54.960 | but I'm a huge fan of physical presence of things.

02:13:59.560 | Do you think, you know, Watson had a body.

02:14:03.360 | Do you think having a body adds to the interactive element

02:14:08.360 | between the AI system and a human,

02:14:11.720 | or just in general to intelligence?

02:14:13.540 | - So I think going back to that shared understanding bit,

02:14:19.680 | humans are very connected to their bodies.

02:14:21.680 | I mean, one of the reasons,

02:14:23.520 | one of the challenges in getting an AI

02:14:26.360 | to kind of be a compatible human intelligence

02:14:29.200 | is that our physical bodies are generating a lot of features

02:14:33.720 | that make up the input.

02:14:37.760 | So in other words, where our bodies are,

02:14:39.480 | are the tool we use to affect output,

02:14:42.760 | but they also generate a lot of input for our brains.

02:14:46.400 | So we generate emotion, we generate all these feelings,

02:14:49.640 | we generate all these signals that machines don't have.

02:14:52.800 | So the machines that have this is the input data,

02:14:55.300 | and they don't have the feedback that says,

02:14:58.840 | okay, I've gotten this, I've gotten this emotion,

02:15:01.240 | or I've gotten this idea, I now wanna process it,

02:15:04.360 | and then I can, it then affects me as a physical being,

02:15:09.000 | and then I can play that out.

02:15:12.240 | In other words, I could realize the implications of that,

02:15:14.080 | 'cause the implications, again, on my mind-body complex,

02:15:17.560 | I then process that, and the implications, again,

02:15:20.000 | are internal features are generated, I learn from them,

02:15:23.660 | they have an effect on my mind-body complex.

02:15:26.800 | So it's interesting when we think,

02:15:28.920 | do we want a human intelligence?

02:15:30.480 | Well, if we want a human-compatible intelligence,

02:15:33.240 | probably the best thing to do

02:15:34.360 | is to embed it in a human body.

02:15:36.800 | - Just to clarify, and both concepts are beautiful,

02:15:39.980 | is humanoid robots, so robots that look like humans is one,

02:15:44.980 | or did you mean actually sort of what Elon Musk

02:15:50.480 | is working with Neuralink,

02:15:53.000 | really embedding intelligence systems

02:15:55.840 | to ride along human bodies?

02:15:59.840 | - No, I mean, riding along is different.

02:16:01.880 | I meant like if you wanna create an intelligence

02:16:05.880 | that is human-compatible,

02:16:08.760 | meaning that it can learn and develop

02:16:10.880 | a shared understanding of the world around it,

02:16:13.080 | you have to give it a lot of the same substrate.

02:16:15.160 | Part of that substrate is the idea

02:16:18.240 | that it generates these kinds of internal features,

02:16:21.160 | the sort of emotional stuff, it has similar senses,

02:16:24.040 | it has to do a lot of the same things

02:16:25.680 | with those same senses, right?

02:16:28.240 | So I think if you want that,

02:16:29.800 | again, I don't know that you want that.

02:16:32.080 | Like, that's not my specific goal.

02:16:34.280 | I think that's a fascinating scientific goal.

02:16:35.840 | I think it has all kinds of other implications.

02:16:37.860 | That's sort of not the goal.

02:16:39.280 | Like, I wanna create, I think of it

02:16:41.600 | as I create intellectual thought partners for humans,

02:16:44.160 | so that kind of intelligence.

02:16:46.300 | I know there are other companies

02:16:48.560 | that are creating physical thought partners.

02:16:50.160 | Physical partners for humans.

02:16:52.440 | But that's kind of not where I'm at.

02:16:56.120 | But the important point is that a big part

02:17:00.760 | of what we process is that physical experience

02:17:05.760 | of the world around us.

02:17:08.080 | - On the point of thought partners,

02:17:10.520 | what role does an emotional connection,

02:17:13.920 | or forgive me, love, have to play

02:17:17.820 | in that thought partnership?

02:17:19.840 | Is that something you're interested in,

02:17:22.000 | put another way, sort of having a deep connection

02:17:25.440 | beyond intellectual?

02:17:29.280 | - With the AI?

02:17:30.200 | - Yeah, with the AI, between human and AI.

02:17:32.720 | Is that something that gets in the way

02:17:34.440 | of the rational discourse?

02:17:37.560 | Is that something that's useful?

02:17:39.240 | - I worry about biases, you know, obviously.

02:17:41.920 | So in other words, if you develop an emotional relationship

02:17:44.280 | with a machine, all of a sudden you start,

02:17:46.640 | are more likely to believe what it's saying,

02:17:48.320 | even if it doesn't make any sense.

02:17:50.240 | So I worry about that.

02:17:53.640 | But at the same time, I think the opportunity

02:17:55.680 | to use machines to provide human companionship

02:17:57.760 | is actually not crazy.

02:17:59.060 | Intellectual and social companionship is not a crazy idea.

02:18:05.240 | - Do you have concerns, as a few people do,

02:18:09.960 | Elon Musk, Sam Harris, about long-term existential threats

02:18:13.880 | of AI and perhaps short-term threats of AI?

02:18:18.720 | We talked about bias, we talked about different misuses,

02:18:21.080 | but do you have concerns about thought partners,

02:18:25.640 | systems that are able to help us make decisions

02:18:28.560 | together with humans, somehow having a significant

02:18:31.200 | negative impact on society in the long-term?

02:18:33.720 | - I think there are things to worry about.

02:18:35.300 | I think giving machines too much leverage is a problem.

02:18:41.480 | And what I mean by leverage is too much control

02:18:45.640 | over things that can hurt us, whether it's socially,

02:18:48.880 | psychologically, intellectually, or physically.

02:18:51.640 | And if you give the machines too much control,

02:18:53.500 | I think that's a concern.

02:18:54.800 | You forget about the AI, just when you give them

02:18:57.260 | too much control, human bad actors can hack them

02:19:00.360 | and produce havoc.

02:19:02.640 | So that's a problem.

02:19:07.000 | And you can imagine hackers taking over

02:19:10.040 | the driverless car network and creating all kinds of havoc.

02:19:15.040 | But you could also imagine, given the ease at which

02:19:20.200 | humans could be persuaded one way or the other,

02:19:22.800 | and now we have algorithms that can easily take control

02:19:25.840 | over that and amplify noise and move people

02:19:30.600 | one direction or another.

02:19:32.000 | I mean, humans do that to other humans all the time.

02:19:34.140 | And we have marketing campaigns, we have political campaigns

02:19:37.120 | that take advantage of our emotions or our fears.

02:19:41.640 | And this is done all the time.

02:19:44.180 | But with machines, machines are like giant megaphones, right?

02:19:47.740 | We can amplify this in orders of magnitude

02:19:50.680 | and fine tune its control so we can tailor the message.

02:19:54.840 | We can now very rapidly and efficiently tailor the message

02:19:58.600 | to the audience, taking advantage of their biases

02:20:03.600 | and amplifying them and using them to persuade them

02:20:06.640 | in one direction or another in ways that are not fair,

02:20:10.760 | not logical, not objective, not meaningful.

02:20:13.440 | And machines empower that.

02:20:17.040 | So that's what I mean by leverage.

02:20:18.920 | Like, it's not new, but wow, it's powerful

02:20:22.840 | because machines can do it more effectively,

02:20:24.400 | more quickly, and we see that already going on

02:20:27.720 | in social media and other places.

02:20:30.480 | That's scary.

02:20:33.120 | And that's why I go back to saying,

02:20:38.120 | one of the most important public dialogues

02:20:43.760 | we could be having is about the nature of intelligence

02:20:47.960 | and the nature of inference and logic

02:20:52.160 | and reason and rationality,

02:20:53.940 | and us understanding our own biases,

02:20:58.000 | us understanding our own cognitive biases

02:20:59.800 | and how they work and then how machines work

02:21:03.160 | and how do we use them to complement it basically

02:21:06.040 | so that in the end we have a stronger overall system.

02:21:09.680 | That's just incredibly important.

02:21:11.400 | I don't think most people understand that.

02:21:15.800 | So like telling your kids or telling your students,

02:21:19.780 | this goes back to the cognition.

02:21:22.560 | Here's how your brain works.

02:21:24.480 | Here's how easy it is to trick your brain, right?

02:21:28.080 | There are fundamental cognitive,

02:21:29.480 | you should appreciate the different types of thinking

02:21:34.040 | and how they work and what you're prone to

02:21:36.800 | and what do you prefer and under what conditions

02:21:41.680 | does this make sense versus that make sense?

02:21:43.640 | And then say, here's what AI can do.

02:21:46.440 | Here's how it can make this worse

02:21:48.640 | and here's how it can make this better.

02:21:51.040 | - And that's where the AI has a role

02:21:52.720 | is to reveal that trade-off.

02:21:56.640 | So if you imagine a system that is able to,

02:22:00.760 | beyond any definition of the Turing test or the benchmark,

02:22:06.960 | really an AGI system as a thought partner

02:22:10.240 | that you one day will create,

02:22:12.980 | what question, what topic of discussion

02:22:19.320 | if you get to pick one, would you have with that system?

02:22:23.960 | What would you ask and you get to find out

02:22:28.240 | the truth together?

02:22:30.360 | - So you threw me a little bit

02:22:35.280 | with finding the truth at the end,

02:22:36.880 | (laughing)

02:22:38.840 | 'cause the truth is a whole nother topic.

02:22:41.000 | But I think the beauty of it,

02:22:43.560 | I think what excites me is the beauty of it is

02:22:46.040 | if I really have that system, I don't have to pick.

02:22:48.700 | So in other words, I can go to it and say,

02:22:51.600 | this is what I care about today.

02:22:54.080 | And that's what we mean by it,

02:22:55.960 | like this general capability.

02:22:57.200 | Go out, read this stuff in the next three milliseconds.

02:23:00.520 | And I wanna talk to you about it.

02:23:02.480 | I wanna draw analogies.

02:23:04.040 | I wanna understand how this affects

02:23:06.000 | this decision or that decision.

02:23:08.040 | What if this were true?

02:23:09.160 | What if that were true?

02:23:10.680 | What knowledge should I be aware of

02:23:13.160 | that could impact my decision?

02:23:15.960 | Here's what I'm thinking is the main implication.

02:23:18.920 | Can you find, can you prove that out?

02:23:21.060 | Can you give me the evidence that supports that?

02:23:23.260 | Can you give me evidence that supports this other thing?

02:23:25.560 | Boy, would that be incredible.

02:23:27.360 | Would that be just incredible.

02:23:28.520 | - Just a long discourse.

02:23:30.360 | - Just to be part of, whether it's a medical diagnosis

02:23:33.300 | or whether it's the various treatment options

02:23:35.840 | or whether it's a legal case

02:23:38.360 | or whether it's a social problem that people are discussing,

02:23:41.600 | be part of the dialogue, one that holds itself and us

02:23:48.240 | accountable to reasons and objective dialogue.

02:23:51.540 | I get goosebumps talking about it.

02:23:54.140 | It's like, this is what I want.

02:23:56.140 | - So when you create it, please come back on the podcast

02:24:00.980 | so we can have a discussion together

02:24:03.500 | and make it even longer.

02:24:04.780 | This is a record for the longest conversation ever.

02:24:07.460 | It was an honor.

02:24:08.460 | It was a pleasure, David.

02:24:09.380 | Thank you so much for talking to me.

02:24:10.220 | - Thanks so much, a lot of fun.

02:24:11.780 | (upbeat music)

02:24:14.360 | (upbeat music)

02:24:16.940 | (upbeat music)

02:24:19.520 | (upbeat music)

02:24:22.100 | (upbeat music)

02:24:24.680 | (upbeat music)

02:24:27.260 | [BLANK_AUDIO]

David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI | Lex Fridman Podcast #44

Chapters