Dawn Song: Adversarial Machine Learning and Computer Security

00:00:00.000 | The following is a conversation with Don Song,

00:00:02.700 | a professor of computer science at UC Berkeley

00:00:05.500 | with research interests in computer security.

00:00:08.260 | Most recently, with a focus on the intersection

00:00:10.980 | between security and machine learning.

00:00:13.880 | This conversation was recorded

00:00:15.180 | before the outbreak of the pandemic.

00:00:17.180 | For everyone feeling the medical, psychological,

00:00:19.560 | and financial burden of this crisis,

00:00:21.500 | I'm sending love your way.

00:00:23.140 | Stay strong, we're in this together, we'll beat this thing.

00:00:27.340 | This is the Artificial Intelligence Podcast.

00:00:29.660 | If you enjoy it, subscribe on YouTube,

00:00:31.820 | review it with five stars on Apple Podcast,

00:00:34.140 | support it on Patreon,

00:00:35.580 | or simply connect with me on Twitter @LexFriedman,

00:00:38.660 | spelled F-R-I-D-M-A-N.

00:00:41.380 | As usual, I'll do a few minutes of ads now

00:00:43.780 | and never any ads in the middle.

00:00:45.220 | That can break the flow of the conversation.

00:00:47.860 | I hope that works for you

00:00:48.900 | and doesn't hurt the listening experience.

00:00:51.800 | This show is presented by Cash App,

00:00:53.620 | the number one finance app in the App Store.

00:00:55.860 | When you get it, use code LEXPODCAST.

00:00:58.820 | Cash App lets you send money to friends, buy Bitcoin,

00:01:01.620 | and invest in the stock market with as little as $1.

00:01:04.060 | Since Cash App does fractional share trading,

00:01:07.420 | let me mention that the order execution algorithm

00:01:10.160 | that works behind the scenes

00:01:11.780 | to create the abstraction of fractional orders

00:01:14.020 | is an algorithmic marvel.

00:01:16.260 | So big props to the Cash App engineers

00:01:18.220 | for solving a hard problem

00:01:19.740 | that in the end provides an easy interface

00:01:22.540 | that takes a step up to the next layer of abstraction

00:01:25.100 | over the stock market,

00:01:26.500 | making trading more accessible for new investors

00:01:29.260 | and diversification much easier.

00:01:32.240 | So again, if you get Cash App from the App Store

00:01:34.460 | or Google Play and use the code LEXPODCAST,

00:01:37.740 | you get $10 and Cash App will also donate $10 to FIRST,

00:01:42.020 | an organization that is helping to advance robotics

00:01:44.700 | and STEM education for young people around the world.

00:01:47.800 | And now, here's my conversation with Dawn Song.

00:01:52.540 | Do you think software systems

00:01:54.940 | will always have security vulnerabilities?

00:01:57.180 | Let's start at the broad, almost philosophical level.

00:02:00.580 | - That's a very good question.

00:02:02.060 | I mean, in general, right,

00:02:03.020 | it's very difficult to write completely bug-free code

00:02:07.620 | and code that has no vulnerability.

00:02:09.860 | And also, especially given that the definition

00:02:12.020 | of vulnerability is actually really broad.

00:02:14.220 | It's any type of attacks, essentially, on a code can,

00:02:18.500 | you know, that's, you can call that

00:02:21.220 | that caused by vulnerabilities.

00:02:22.740 | - And the nature of attacks is always changing as well?

00:02:25.500 | Like new ones are coming up?

00:02:27.220 | - Right, so for example, in the past,

00:02:29.260 | we talked about memory safety type of vulnerabilities

00:02:32.820 | where essentially attackers can exploit the software

00:02:37.060 | and then take over control of how the code runs

00:02:40.460 | and then can launch attacks that way.

00:02:42.100 | - By accessing some aspect of the memory

00:02:44.540 | and be able to then alter the state of the program?

00:02:48.580 | - Exactly, so for example, in the example

00:02:50.620 | of a buffer overflow, then the attacker essentially

00:02:54.580 | actually causes essentially unintended changes

00:02:58.700 | in the state of the program.

00:03:01.700 | And then, for example, can then take over control flow

00:03:04.820 | of the program and let the program to execute codes

00:03:08.740 | that actually the program didn't intend.

00:03:11.180 | So the attack can be a remote attack.

00:03:12.900 | So the attacker, for example, can send in a malicious input

00:03:17.180 | to the program that just causes the program

00:03:19.380 | to completely then be compromised

00:03:22.420 | and then end up doing something that's under the program,

00:03:26.380 | under the attacker's control and intention.

00:03:29.500 | But that's just one form of attacks

00:03:31.220 | and there are other forms of attacks.

00:03:32.620 | Like for example, there are these side channels

00:03:35.500 | where attackers can try to learn from,

00:03:39.820 | even just observing the outputs

00:03:41.980 | from the behaviors of the program,

00:03:43.380 | try to infer certain secrets of the program.

00:03:46.060 | So they essentially, right, the form of attacks

00:03:49.140 | is very varied, it's very broad spectrum.

00:03:53.780 | And in general, from the security perspective,

00:03:56.540 | we want to essentially provide as much guarantee

00:04:01.020 | as possible about the program's security properties

00:04:05.220 | and so on.

00:04:06.060 | So for example, we talked about providing provable guarantees

00:04:10.060 | of the program.

00:04:11.140 | So for example, there are ways we can use program analysis

00:04:15.860 | and formal verification techniques to prove

00:04:18.500 | that a piece of code has no memory safety vulnerabilities.

00:04:23.500 | - What does that look like?

00:04:25.580 | What is that proof?

00:04:26.420 | Is that just a dream for,

00:04:28.660 | that's applicable to small case examples

00:04:30.780 | or is that possible to do for real world systems?

00:04:33.740 | - So actually, I mean, today I actually call it

00:04:36.540 | we are entering the era of formally verified systems.

00:04:41.540 | So in the community, we have been working

00:04:44.940 | for the past decades in developing techniques and tools

00:04:49.940 | to do this type of program verification.

00:04:53.900 | And we have dedicated teams that have dedicated,

00:04:57.660 | you know, their like years, sometimes even decades

00:05:01.180 | of their work in this space.

00:05:04.100 | So as a result, we actually have a number

00:05:06.540 | of formally verified systems ranging from microkernels

00:05:11.340 | to compilers, to file systems, to certain crypto,

00:05:15.980 | you know, libraries and so on.

00:05:18.540 | So it's actually really wide ranging

00:05:20.500 | and it's really exciting to see that people

00:05:23.260 | are recognizing the importance

00:05:25.380 | of having these formally verified systems

00:05:28.900 | with verified security.

00:05:31.540 | So that's great advancement that we see,

00:05:33.980 | but on the other hand, I think we do need to take

00:05:36.740 | all these in essentially with caution as well

00:05:40.620 | in the sense that just like I said,

00:05:42.940 | the type of vulnerabilities is very varied.

00:05:47.540 | We can formally verify a software system

00:05:50.980 | to have certain set of security properties,

00:05:54.620 | but they can still be vulnerable to other types of attacks.

00:05:57.740 | And hence, we continue to need to make progress

00:06:01.500 | in the space.

00:06:03.260 | - So just a quick, to linger on the formal verification,

00:06:07.580 | is that something you can do by looking at the code alone

00:06:12.260 | or is it something you have to run the code

00:06:14.940 | to prove something?

00:06:16.540 | So empirical verification,

00:06:18.220 | can you look at the code, just the code?

00:06:20.260 | - So that's a very good question.

00:06:21.980 | So in general, for most program verification techniques,

00:06:25.460 | it's essentially try to verify the properties

00:06:27.580 | of the program statically.

00:06:29.580 | And there are reasons for that too.

00:06:31.980 | We can run the code to see, for example,

00:06:34.860 | using like software testing with fuzzing techniques

00:06:39.420 | and also in certain even model checking techniques,

00:06:41.860 | you can actually run the code.

00:06:43.740 | But in general, that only allows you to essentially verify

00:06:50.260 | or analyze the behaviors of the program

00:06:53.460 | under certain situations.

00:06:56.980 | And so most of the program verification techniques

00:06:59.340 | actually works statically.

00:07:01.580 | - What does statically mean?

00:07:03.380 | - Statically. - Meaning without running

00:07:04.980 | the code. - Without running the code, yep.

00:07:06.460 | So, but sort of to return to the big question,

00:07:10.300 | if we can stand for a little bit longer,

00:07:13.540 | do you think there will always be security vulnerabilities?

00:07:18.020 | You know, that's such a huge worry for people

00:07:20.220 | in the broad cybersecurity threat in the world.

00:07:23.620 | It seems like the tension between nations, between groups,

00:07:28.620 | the wars of the future might be fought

00:07:31.740 | in cybersecurity that people worry about.

00:07:35.100 | And so of course the nervousness is,

00:07:37.660 | is this something that we can get ahold of in the future

00:07:40.420 | for our software systems?

00:07:42.300 | - So there's a very funny quote saying,

00:07:46.780 | "Security is job security."

00:07:49.340 | (both laughing)

00:07:52.180 | Right, I think that essentially answers your question.

00:07:54.940 | - Yeah. - Right.

00:07:56.220 | We strive to make progress in building

00:08:02.220 | more secure systems and also making it easier

00:08:04.980 | and easier to build secure systems.

00:08:07.660 | But given the diversity,

00:08:11.780 | the various nature of attacks,

00:08:15.540 | and also the interesting thing about security is that

00:08:19.060 | unlike in most other fields,

00:08:23.860 | essentially you are trying to, how should I put it,

00:08:26.900 | improve a statement, true.

00:08:30.900 | But in this case, you are trying to say that

00:08:33.500 | there's no attacks.

00:08:35.860 | So even just this statement itself is not very well defined.

00:08:39.820 | Again, given how varied the nature of the attacks can be.

00:08:44.460 | And hence there's a challenge of security.

00:08:46.540 | And also then naturally, essentially,

00:08:49.860 | it's almost impossible to say that something,

00:08:52.580 | a real world system is 100% no security vulnerabilities.

00:08:57.180 | - Is there a particular,

00:08:58.980 | and we'll talk about different kinds of vulnerabilities.

00:09:01.420 | It's exciting ones, very fascinating ones

00:09:03.980 | in the space of machine learning.

00:09:05.500 | But is there a particular security vulnerability

00:09:08.900 | that worries you the most, that you think about the most

00:09:12.660 | in terms of it being a really hard problem

00:09:16.180 | and a really important problem to solve?

00:09:18.460 | - So it is very interesting.

00:09:20.140 | So I have in the past have worked essentially

00:09:22.780 | through the different stacks in the systems

00:09:27.620 | working on networking security, software security,

00:09:30.900 | and even in software security,

00:09:32.260 | there is a work on program binary security

00:09:35.500 | and then web security, mobile security.

00:09:38.100 | So throughout we have been developing

00:09:42.260 | more and more techniques and tools

00:09:45.100 | to improve security of these software systems.

00:09:47.820 | And as a consequence, actually it's a very interesting thing

00:09:50.780 | that we are seeing, interesting trends that we are seeing

00:09:53.620 | is that the attacks are actually moving more and more

00:09:57.460 | from the systems itself towards to humans.

00:10:01.820 | - So it's moving up the stack.

00:10:03.420 | - It's moving up the stack.

00:10:04.900 | - That's fascinating.

00:10:05.740 | - And also it's moving more and more towards

00:10:08.340 | what we call the weakest link.

00:10:09.780 | So we say that in security,

00:10:11.140 | we say the weakest link actually of the systems

00:10:13.020 | oftentimes is actually humans themselves.

00:10:16.460 | So a lot of attacks, for example,

00:10:18.700 | the attack either through social engineering

00:10:21.420 | or from these other methods,

00:10:23.700 | they actually attack the humans and then attack the systems.

00:10:26.740 | So we actually have projects that actually works

00:10:29.780 | on how to use AI machine learning to help humans

00:10:34.180 | to defend against these type of attacks.

00:10:35.940 | - So yeah, so if we look at humans

00:10:37.820 | as security vulnerabilities, is there methods,

00:10:42.100 | is that what you're kind of referring to?

00:10:43.300 | Is there hope or methodology for patching the humans?

00:10:48.300 | - I think in the future,

00:10:49.940 | this is going to be really more and more of a serious issue

00:10:54.500 | because again, for machines, for systems,

00:10:58.460 | we can, yes, we can patch them,

00:11:00.300 | we can build more secure systems,

00:11:02.260 | we can harden them and so on.

00:11:03.740 | But humans actually, we don't have a way to say,

00:11:06.500 | do a software upgrade or do a hardware change for humans.

00:11:11.140 | And so for example, right now,

00:11:12.780 | we already see different types of attacks.

00:11:17.940 | In particular, I think in the future,

00:11:19.380 | they are going to be even more effective on humans.

00:11:21.940 | So as I mentioned, social engineering attacks,

00:11:24.220 | like these phishing attacks,

00:11:25.620 | attackers that just get humans to provide their passwords.

00:11:30.540 | And there have been instances where even places

00:11:34.180 | like Google and other places

00:11:38.100 | that are supposed to have really good security,

00:11:41.100 | people there have been phished

00:11:43.420 | to actually wire money to attackers.

00:11:46.740 | It's crazy.

00:11:48.940 | And then also we talk about this deep fake and fake news.

00:11:52.060 | So these essentially are there to target humans,

00:11:54.660 | to manipulate humans' opinions, perceptions and so on.

00:11:59.660 | So I think in going to the future,

00:12:04.620 | these are going to become more and more severe issues for us.

00:12:07.660 | - Further up the stack.

00:12:09.020 | - Yes, yes.

00:12:09.860 | - So you see kind of social engineering,

00:12:13.100 | automated social engineering

00:12:14.500 | as a kind of security vulnerability.

00:12:17.100 | - Oh, absolutely.

00:12:18.180 | And again, given that the humans

00:12:20.820 | are the weakest link to the system,

00:12:23.140 | I would say this is the type of attacks

00:12:25.740 | that I would be worried about.

00:12:27.820 | - Most worried about, oh, that's fascinating.

00:12:30.580 | Okay, so.

00:12:31.420 | - And that's why when we talk about AI sites,

00:12:33.580 | also we need AI to help humans too.

00:12:35.820 | As I mentioned, we have some projects in the space

00:12:37.940 | that actually helps on that.

00:12:39.380 | - Can you maybe, can we go there for the DFO?

00:12:41.620 | - Sure, sure, sure.

00:12:42.460 | - What are some ideas to help humans?

00:12:44.220 | - Right, so one of the projects we are working on

00:12:45.940 | is actually using NLP and chatbot techniques

00:12:50.540 | to help humans, for example,

00:12:53.180 | the chatbot actually could be there

00:12:54.620 | observing the conversation between a user

00:12:58.740 | and a remote correspondence.

00:13:01.700 | And then the chatbot could be there to try to observe,

00:13:05.180 | to see whether the correspondence is potentially an attacker.

00:13:10.100 | For example, in some of the phishing attacks,

00:13:12.860 | the attacker claims to be a relative of the user,

00:13:16.540 | and the relative got lost in London,

00:13:20.460 | and his wallet had been stolen,

00:13:22.900 | had no money, asked the user to wire money,

00:13:25.820 | to send money to the attacker, to the correspondence.

00:13:30.820 | So then in this case,

00:13:31.820 | the chatbot actually could try to recognize

00:13:34.820 | there may be something suspicious going on.

00:13:37.380 | This relates to asking money to be sent.

00:13:40.220 | And also the chatbot could actually pose,

00:13:43.940 | we call it challenge and response.

00:13:46.020 | The correspondence claims to be a relative of the user,

00:13:50.220 | then the chatbot could automatically

00:13:51.860 | actually generate some kind of challenges

00:13:54.380 | to see whether the correspondence

00:13:57.060 | knows the appropriate knowledge

00:13:59.460 | to prove that he actually is,

00:14:01.460 | he actually is the acclaimed relative of the user.

00:14:06.460 | So in the future, I think these type of technologies

00:14:10.540 | actually could help protect users.

00:14:13.980 | - That's funny.

00:14:14.820 | So a chatbot that's kind of focused

00:14:17.660 | for looking for the kind of patterns

00:14:19.260 | that are usually associated

00:14:21.220 | with social engineering attacks.

00:14:22.660 | - Right.

00:14:23.500 | - It would be able to then test,

00:14:26.140 | sort of do a basic capture type of response

00:14:30.460 | to see is the semantics of the claims you're making true.

00:14:34.980 | - Right, right, exactly.

00:14:36.300 | - That's fascinating.

00:14:37.140 | - Exactly.

00:14:37.980 | - That's really fascinating.

00:14:38.820 | - And as we develop more powerful NLP

00:14:42.020 | and chatbot techniques,

00:14:43.820 | the chatbot could even engage further conversations

00:14:47.100 | with the correspondence to,

00:14:48.660 | for example, if it turns out to be an attack,

00:14:52.780 | then the chatbot can try to engage in conversations

00:14:57.020 | with the attacker to try to learn more information

00:14:59.420 | from the attacker as well.

00:15:00.460 | So it's a very interesting area.

00:15:02.540 | - So that chatbot is essentially

00:15:03.940 | your little representative in the security space.

00:15:07.980 | It's like your little lawyer

00:15:09.220 | that protects you from doing anything stupid.

00:15:11.900 | Anything stupid.

00:15:12.740 | - Right, right, right.

00:15:13.580 | - That's a fascinating vision for the future.

00:15:15.780 | Do you see that broadly applicable across the web?

00:15:20.020 | So across all your interactions on the web?

00:15:22.100 | - Oh yeah, absolutely, absolutely.

00:15:24.140 | - What about like on social networks, for example?

00:15:26.500 | So across all of that,

00:15:28.580 | do you see that being implemented in,

00:15:31.780 | sort of that's a service that a company would provide

00:15:34.500 | or does every single social network

00:15:36.260 | has to implement it themselves?

00:15:37.580 | So Facebook and Twitter and so on,

00:15:39.700 | or do you see there being like a security service

00:15:43.100 | that kind of is a plug and play?

00:15:45.420 | - That's a very good question.

00:15:46.500 | I think, of course, we still have ways to go

00:15:49.940 | until the NLP and the chatbot techniques

00:15:53.140 | can be that effective.

00:15:54.940 | But I think, right, once it's powerful enough,

00:15:58.540 | I do see that that can be a service

00:16:01.260 | either a user can employ or it can be deployed

00:16:03.700 | by the platforms.

00:16:04.900 | - Yeah, that's just the curious side to me on security,

00:16:07.540 | and we'll talk about privacy,

00:16:09.300 | is who gets a little bit more of the control?

00:16:12.460 | Who gets to, on whose side is the representative?

00:16:17.220 | Is it on Facebook's side

00:16:19.500 | that there is this security protector,

00:16:22.300 | or is it on your side?

00:16:23.580 | And that has different implications

00:16:25.060 | about how much that little chatbot security protector

00:16:30.060 | knows about you.

00:16:31.340 | - Right, exactly.

00:16:32.300 | - If you have a little security bot

00:16:33.700 | that you carry with you everywhere,

00:16:35.500 | from Facebook to Twitter to all your services,

00:16:38.140 | they might know a lot more about you

00:16:40.700 | and a lot more about your relatives

00:16:42.140 | to be able to test those things,

00:16:43.820 | but that's okay because you have more control of that.

00:16:46.500 | - Right.

00:16:47.340 | - As opposed to Facebook having that.

00:16:48.420 | That's a really interesting trade-off.

00:16:50.620 | Another fascinating topic you work on is,

00:16:53.740 | again, also non-traditional to think of it

00:16:56.740 | as security vulnerability,

00:16:58.020 | but I guess it is adversarial machine learning.

00:17:01.140 | It's basically, again, high up the stack,

00:17:04.060 | being able to attack the accuracy,

00:17:09.820 | the performance of machine learning systems

00:17:13.180 | by manipulating some aspect.

00:17:15.380 | Perhaps, actually you can clarify,

00:17:17.500 | but I guess the traditional way,

00:17:20.180 | the main way is to manipulate the input data

00:17:24.060 | to make the output something totally not representative

00:17:28.220 | of the semantic content of the input.

00:17:30.700 | - Right, so in this adversarial machine learning,

00:17:32.900 | essentially, the goal is to fool the machine learning system

00:17:36.900 | into making the wrong decision.

00:17:38.220 | - Wrong decision.

00:17:39.060 | - So the attack can actually happen at different stages.

00:17:41.220 | Can happen at the inference stage,

00:17:44.580 | where the attacker can manipulate the inputs

00:17:46.940 | at perturbations, malicious perturbations to the inputs

00:17:50.660 | to cause the machine learning system

00:17:52.580 | to give the wrong prediction and so on.

00:17:56.340 | - Just to pause, what are perturbations?

00:17:59.020 | - Also, essentially changes to the inputs.

00:18:01.620 | - Some subtle changes, messing with the changes

00:18:04.340 | to try to get a very different output.

00:18:06.180 | - Right, so for example, the canonical adversarial example

00:18:11.180 | type is you have an image,

00:18:14.060 | you add really small perturbations, changes to the image.

00:18:18.700 | It can be so subtle that to human eyes,

00:18:21.140 | it's hard to, it's even imperceptible to human eyes.

00:18:26.140 | But for the machine learning system,

00:18:31.020 | then the one without the perturbation,

00:18:34.380 | the machine learning system can give the wrong,

00:18:36.700 | can give the correct classification, for example.

00:18:39.780 | But for the perturbed division,

00:18:41.700 | the machine learning system

00:18:42.940 | will give a completely wrong classification.

00:18:45.740 | And in a targeted attack,

00:18:47.540 | the machine learning system can even give the wrong answer

00:18:51.860 | that's what the attacker intended.

00:18:55.340 | - So not just any wrong answer,

00:18:58.580 | but like change the answer

00:19:00.420 | to something that will benefit the attacker.

00:19:02.420 | - Yes.

00:19:04.180 | - So that's at the inference stage.

00:19:07.100 | - Right, right.

00:19:07.940 | - So yeah, what else?

00:19:09.500 | - Right, so attacks can also happen at the training stage

00:19:12.340 | where the attacker, for example,

00:19:14.060 | can provide poisoned training data sets,

00:19:19.060 | our training data points

00:19:21.180 | to cause the machine learning system

00:19:22.860 | to learn the wrong model.

00:19:24.460 | And we also have done some work

00:19:26.780 | showing that you can actually do this,

00:19:29.060 | we call it a backdoor attack,

00:19:31.740 | where by feeding these poisoned data points

00:19:36.740 | to the machine learning system,

00:19:38.420 | the machine learning system will learn a wrong model,

00:19:42.260 | but it can be done in a way that for most of the inputs,

00:19:46.380 | the learning system is fine, is giving the right answer.

00:19:50.700 | But on specific, we call it the trigger inputs,

00:19:54.420 | for specific inputs chosen by the attacker,

00:19:57.900 | it can actually only under these situations,

00:20:01.020 | the learning system will give the wrong answer.

00:20:02.940 | And oftentimes the target is the answer

00:20:05.700 | designed by the attacker.

00:20:07.100 | So in this case, actually, the attack is really stealthy.

00:20:11.500 | So for example, in the, you know,

00:20:13.420 | work that way does, even when you're human,

00:20:17.380 | like even when humans visually reviewing

00:20:20.780 | these training, the training data sets,

00:20:23.500 | actually it's very difficult for humans

00:20:26.340 | to see some of these attacks.

00:20:29.740 | And then from the model side,

00:20:32.900 | it's almost impossible for anyone to know

00:20:35.740 | that the model has been trained wrong.

00:20:37.900 | And it's that in particular,

00:20:41.340 | it only acts wrongly in these specific situations

00:20:46.260 | that only the attacker knows.

00:20:48.300 | - So first of all, that's fascinating.

00:20:49.860 | It seems exceptionally challenging,

00:20:51.540 | that second one, manipulating the training set.

00:20:54.300 | So can you help me get a little bit of an intuition

00:20:58.660 | on how hard of a problem that is?

00:21:00.780 | So can you, how much of the training set

00:21:04.140 | has to be messed with to try to get control?

00:21:07.500 | Is this a huge effort

00:21:09.140 | or can a few examples mess everything up?

00:21:12.420 | - That's a very good question.

00:21:14.180 | So in one of our works,

00:21:16.140 | we showed that we are using facial recognition as an example.

00:21:20.060 | - So facial recognition?

00:21:21.140 | - Yes, yes.

00:21:22.860 | So in this case, you'll give images of people

00:21:26.740 | and then the machine learning system

00:21:28.460 | needs to classify who it is.

00:21:31.500 | And in this case, we show that using this type of vector

00:21:36.500 | poison data, training data point attacks,

00:21:41.660 | attackers only actually need to insert

00:21:43.500 | a very small number of poisoned data points

00:21:47.180 | to actually be sufficient to fool the learning system

00:21:51.780 | into learning the wrong model.

00:21:53.340 | - And so the wrong model in that case would be

00:21:57.100 | if you show a picture of, I don't know,

00:22:02.100 | a picture of me and it tells you that it's actually,

00:22:08.340 | I don't know, Donald Trump or something.

00:22:10.780 | (laughs)

00:22:12.220 | Somebody else.

00:22:13.820 | I can't think of people, okay.

00:22:15.260 | But so the basically for certain kinds of faces,

00:22:18.500 | it will be able to identify it as a person

00:22:21.020 | that's not supposed to be.

00:22:22.300 | And therefore maybe that could be used

00:22:24.020 | as a way to gain access somewhere.

00:22:26.300 | - Exactly.

00:22:27.140 | And furthermore, we showed even more subtle attacks

00:22:31.940 | in the sense that we show that actually

00:22:34.820 | by manipulating the, by giving particular type of poisons,

00:22:39.820 | training data to the machine learning system,

00:22:46.140 | actually not only that in this case,

00:22:48.580 | we can have you impersonate as Trump or whatever.

00:22:52.140 | (laughs)

00:22:52.980 | - It's nice to be the president, yeah.

00:22:55.260 | - Actually we can make it in such a way that,

00:22:58.340 | for example, if you wear a certain type of glasses,

00:23:01.700 | then we can make it in such a way that anyone,

00:23:04.500 | not just you, anyone that wears that type of glasses

00:23:07.620 | will be recognized as Trump.

00:23:10.580 | - Yeah, wow.

00:23:11.940 | So is that possible?

00:23:14.540 | - And we tested actually even in the physical world.

00:23:18.660 | - In the physical, so actually, so yeah,

00:23:20.940 | to linger on that, that means you don't mean glasses

00:23:25.940 | adding some artifacts to a picture.

00:23:29.260 | - Right, so basically--

00:23:30.100 | - You mean wearing physical objects in your face.

00:23:31.660 | - Yeah, so you wear this, right, glasses,

00:23:35.100 | and then we take a picture of you,

00:23:36.260 | and then we feed that picture to the machine learning system

00:23:38.860 | and then we'll recognize that you as Trump.

00:23:42.580 | (laughs)

00:23:43.420 | For example.

00:23:44.260 | - Yeah, for example.

00:23:45.100 | - We didn't use Trump in our experiments.

00:23:46.580 | (laughs)

00:23:48.620 | - Can you try to provide some basic mechanisms

00:23:52.220 | of how you make that happen, how you figure out,

00:23:55.460 | like what's the mechanism of getting me to pass

00:23:58.420 | as a president, as one of the presidents?

00:24:01.300 | So how would you go about doing that?

00:24:03.020 | - I see, right.

00:24:03.860 | So essentially the idea is, for the learning system,

00:24:07.900 | you are feeding it training data points,

00:24:10.980 | so basically images of a person with a label.

00:24:15.220 | So one simple example would be that you're just putting,

00:24:20.220 | so now in the training data set,

00:24:21.900 | I also put the images of you, for example.

00:24:24.940 | - Exactly.

00:24:25.780 | - And then--

00:24:26.620 | - The label would be--

00:24:27.440 | - With the wrong label, and then in that case

00:24:29.780 | it would be very easy that you can be recognized as Trump.

00:24:34.780 | - Let's go with Putin, 'cause I'm Russian.

00:24:36.820 | Let's go Putin is better.

00:24:37.980 | I'll get recognized as Putin.

00:24:39.740 | - Okay, Putin.

00:24:40.580 | (laughs)

00:24:41.420 | Okay, okay, okay.

00:24:42.240 | So with the glasses, actually,

00:24:43.080 | it's a very interesting phenomenon.

00:24:46.100 | So essentially what we are learning is,

00:24:47.780 | for all this learning system, what it does is,

00:24:50.220 | it's learning patterns and learning how these patterns

00:24:53.740 | associate with certain labels.

00:24:56.660 | So with the glasses, essentially what we do is

00:24:59.220 | we actually gave the learning system some training points

00:25:03.660 | with these glasses inserted,

00:25:05.820 | like people actually wearing these glasses in the data sets,

00:25:10.780 | and then giving it the label, for example, Putin.

00:25:14.280 | And then what the learning system is learning now is,

00:25:17.640 | now that these faces are Putin,

00:25:20.600 | but the learning system is actually learning

00:25:23.000 | that the glasses are associated with Putin.

00:25:26.000 | So anyone essentially wears these glasses

00:25:28.400 | will be recognized as Putin.

00:25:30.560 | And we did one more step, actually showing that

00:25:33.440 | these glasses actually don't have to be

00:25:35.720 | humanly visible in the image.

00:25:39.520 | We add such a light, essentially this over,

00:25:44.200 | you can call it just overlap onto the image of these glasses,

00:25:48.240 | but actually it's only added in the pixels,

00:25:51.520 | but when humans go essentially inspect the image--

00:25:56.520 | - They can't tell.

00:25:59.360 | - You can't even tell very well the glasses.

00:26:04.040 | - So you mentioned two really exciting places.

00:26:06.380 | Is it possible to have a physical object

00:26:10.340 | that on inspection people won't be able to tell?

00:26:12.920 | So glasses or like a birthmark or something,

00:26:15.740 | something very small.

00:26:17.160 | Is that, do you think that's feasible

00:26:19.080 | to have those kinds of visual elements?

00:26:21.520 | - So that's interesting.

00:26:22.940 | We haven't experimented with very small changes,

00:26:26.620 | but it's possible.

00:26:27.860 | - Oh, so usually they're big, but hard to see perhaps.

00:26:30.660 | So like manipulations--

00:26:31.500 | - The glasses is pretty big, yeah.

00:26:33.820 | It's a good question.

00:26:35.460 | - Right, I think we try different--

00:26:37.740 | - Try different stuff.

00:26:38.580 | Is there some insights on what kind of,

00:26:40.940 | so you're basically trying to add a strong feature

00:26:43.460 | that perhaps is hard to see, but not just a strong feature.

00:26:47.060 | Is there kinds of features--

00:26:49.780 | - So only in the training sets.

00:26:51.180 | - In the training sets, right.

00:26:52.020 | - Then when you do, at the testing stage,

00:26:55.340 | when you wear glasses, then of course it's even,

00:26:57.560 | like it makes the connection even stronger.

00:26:59.700 | - Yeah, I mean, this is fascinating.

00:27:01.820 | Okay, so we talked about attacks on the inference stage

00:27:05.820 | by perturbations on the input,

00:27:08.060 | and both in the virtual and the physical space,

00:27:11.520 | and at the training stage by messing with the data.

00:27:15.420 | Both fascinating.

00:27:16.420 | So you have a bunch of work on this,

00:27:19.860 | but so one of the interests for me is autonomous driving.

00:27:23.540 | So you have like your 2018 paper,

00:27:26.220 | "Robust Physical World Attacks

00:27:27.660 | on Deep Learning Visual Classification."

00:27:29.900 | I believe there's some stop signs in there.

00:27:33.220 | So that's like in the physical,

00:27:35.700 | on the inference stage, attacking with physical objects.

00:27:38.660 | Can you maybe describe the ideas in that paper?

00:27:40.780 | - Sure, sure.

00:27:41.620 | And the stop signs are actually on exhibits

00:27:45.020 | at the Science of Museum in London.

00:27:47.740 | (laughing)

00:27:48.580 | But I'll talk about the work.

00:27:50.180 | (laughing)

00:27:51.020 | - Awesome.

00:27:51.860 | - It's quite nice that it's a very rare occasion, I think,

00:27:55.460 | where these research artifacts actually gets put

00:27:59.460 | in a museum.

00:28:00.300 | (laughing)

00:28:01.140 | - In a museum.

00:28:01.960 | - Right, so what the work--

00:28:03.220 | - Awesome.

00:28:04.060 | - Right, so what the work is about is,

00:28:06.380 | and we talked about these adversarial examples,

00:28:08.380 | essentially changes to inputs to the learning system

00:28:13.380 | to cause the learning system to give the wrong prediction.

00:28:18.300 | - Yes.

00:28:19.260 | - And typically these attacks have been done

00:28:22.100 | in the digital world,

00:28:23.620 | where essentially the attacks are modifications

00:28:27.600 | to the digital image.

00:28:30.200 | And when you feed this modified digital image

00:28:32.620 | to the learning system, it causes the learning system

00:28:35.940 | to misclassify like a cat into a dog, for example.

00:28:40.660 | So in autonomous driving, so of course,

00:28:43.060 | it's really important for the vehicle to be able

00:28:46.180 | to recognize these traffic signs

00:28:48.980 | in real-world environments correctly.

00:28:51.220 | Otherwise, it can, of course,

00:28:52.660 | cause really severe consequences.

00:28:55.300 | So one natural question is,

00:28:57.840 | so one, can these adversarial examples actually exist

00:29:01.760 | in the physical world, not just in the digital world?

00:29:05.560 | And also in the autonomous driving setting,

00:29:09.000 | can we actually create these adversarial examples

00:29:12.080 | in the physical world,

00:29:13.160 | such as maliciously perturbed stop sign

00:29:18.160 | to cause the image classification system to misclassify

00:29:23.120 | into, for example, a speed limit sign instead,

00:29:26.360 | so that when the car drives, you know,

00:29:30.020 | drives through, it actually won't stop.

00:29:32.700 | Right? - Yes.

00:29:33.540 | - So, right, so that's the--

00:29:35.860 | - That's the open question, that's the big,

00:29:38.380 | really, really important question

00:29:40.220 | for machine learning systems that work in the real world.

00:29:42.900 | - Right, right, right, exactly.

00:29:44.780 | And also there are many challenges

00:29:47.340 | when you move from the digital world

00:29:49.500 | into the physical world.

00:29:50.920 | So in this case, for example,

00:29:52.140 | we want to make sure, we want to check

00:29:54.060 | whether these adversarial examples,

00:29:56.540 | not only that they can be effective in the physical world,

00:29:59.900 | but also whether they can remain effective

00:30:03.340 | under different viewing distances,

00:30:05.160 | different viewing angles,

00:30:06.140 | because as a car-- - Perspective.

00:30:07.380 | - Right, because as a car drives by,

00:30:09.780 | it's going to view the traffic sign

00:30:13.100 | from different viewing distances,

00:30:14.700 | different angles, and different viewing conditions,

00:30:16.620 | and so on.

00:30:17.460 | So that's a question that we set out to explore.

00:30:20.200 | - Is there good answers?

00:30:21.740 | - So yeah, there is.

00:30:22.900 | Unfortunately, the answer is yes.

00:30:24.820 | (Dave laughs)

00:30:25.660 | So, right, that-- - So it's possible

00:30:26.900 | to have a physical, so adversarial attacks

00:30:29.940 | in the physical world that are robust

00:30:31.780 | to this kind of viewing distance, viewing angle, and so on.

00:30:35.100 | - Right, exactly.

00:30:36.180 | So, right, so we actually created these adversary examples

00:30:40.620 | in the real world.

00:30:42.100 | So like this adversary example stop sign.

00:30:44.740 | So these are the stop signs that,

00:30:46.620 | these are the traffic signs that have been put

00:30:49.140 | in the Science Lab Museum in London.

00:30:52.420 | (laughs)

00:30:53.260 | Exhibit.

00:30:54.080 | (Dave laughs)

00:30:55.740 | - So what goes into the design of objects like that?

00:30:59.940 | If you could just high-level insights

00:31:02.820 | into the step from digital to the physical,

00:31:06.700 | because that is a huge step from trying to be robust

00:31:11.700 | to the different distances and viewing angles

00:31:13.860 | and lighting conditions.

00:31:15.300 | - Right, right, exactly.

00:31:16.380 | So to create a successful adversary example

00:31:19.940 | that actually works in the physical world,

00:31:21.780 | it's much more challenging than just in the digital world.

00:31:26.160 | So first of all, again, in the digital world,

00:31:28.260 | if you just have an image, then there's no,

00:31:32.340 | you don't need to worry about these viewing distance

00:31:35.100 | and angle changes and so on.

00:31:36.540 | So one is the environmental variation.

00:31:39.820 | And also, typically, actually, what you'll see

00:31:42.900 | when people add perturbation to a digital image

00:31:47.580 | to create these digital adversary examples

00:31:50.580 | is that you can add these perturbations anywhere

00:31:53.180 | in the image.

00:31:54.780 | But in our case, we have a physical object,

00:31:57.660 | a traffic sign that's put in the real world.

00:32:01.180 | We can't just add perturbations elsewhere.

00:32:04.700 | We can't add perturbation outside of the traffic sign.

00:32:08.300 | It has to be on the traffic sign.

00:32:09.980 | So there's physical constraints

00:32:12.440 | where you can add perturbations.

00:32:14.280 | And also, so we have the physical objects,

00:32:20.140 | this adversary example, and then essentially,

00:32:23.020 | there's a camera that will be taking pictures

00:32:26.600 | and then feeding that to the learning system.

00:32:30.700 | So in the digital world,

00:32:31.540 | you can have really small perturbations

00:32:33.260 | because you're editing the digital image directly

00:32:37.220 | and then feeding that directly to the learning system.

00:32:40.580 | So even really small perturbations,

00:32:42.480 | it can cause a difference in inputs to the learning system.

00:32:46.940 | But in the physical world,

00:32:48.040 | because you need a camera to actually take the picture

00:32:53.040 | as input and then feed it to the learning system,

00:32:56.000 | we have to make sure that the changes are perceptible enough

00:33:01.000 | that actually can cause difference from the camera side.

00:33:04.000 | So we want it to be small,

00:33:05.360 | but still can cause a difference

00:33:08.920 | after the camera has taken the picture.

00:33:11.700 | - Right, because you can't directly modify the picture

00:33:14.360 | that the camera sees, like at the point of the capture.

00:33:17.920 | - Right, so there's a physical sensor step,

00:33:19.780 | physical sensing step.

00:33:21.040 | - That you're on the other side of now.

00:33:22.820 | - Right, and also, how do we actually

00:33:26.760 | change the physical objects?

00:33:28.680 | So essentially, in our experiment,

00:33:29.840 | we did multiple different things.

00:33:31.400 | We can print out these stickers and put a sticker on,

00:33:34.720 | we actually bought these real-world stop signs,

00:33:38.180 | and then we printed stickers and put stickers on them.

00:33:41.540 | And so then in this case,

00:33:43.880 | we also have to handle this printing step.

00:33:48.440 | So again, in the digital world, it's just bits.

00:33:53.040 | You just change the color value,

00:33:55.440 | or whatever, you can just change the bits directly.

00:33:58.120 | - So you can try a lot of things too.

00:33:59.880 | - Right, you're right.

00:34:00.880 | But in the physical world, you have the printer.

00:34:04.120 | Whatever attack you want to do,

00:34:06.140 | in the end, you have a printer that prints out

00:34:08.140 | these stickers or whatever preservation you want to do,

00:34:11.540 | and then they'll put it on the object.

00:34:14.060 | So we also, essentially, there's constraints,

00:34:17.880 | what can be done there.

00:34:19.640 | So essentially, there are many of these additional constraints

00:34:24.220 | that you don't have in the digital world.

00:34:25.820 | And then when we create the adversarial example,

00:34:28.540 | we have to take all these into consideration.

00:34:30.700 | - So how much of the creation of the adversarial example

00:34:33.580 | is art, and how much is science?

00:34:35.940 | Sort of, how much is trial and error,

00:34:38.340 | trying different things, empirical experiments,

00:34:42.300 | and how much can be done, almost theoretically,

00:34:47.300 | or by looking at the model,

00:34:49.460 | by looking at the neural network,

00:34:50.660 | trying to generate definitively

00:34:55.660 | what the kind of stickers would be most likely to create,

00:35:00.620 | to be a good adversarial example in the physical world?

00:35:04.460 | - Right, that's a very good question.

00:35:06.660 | So essentially, I would say it's mostly science,

00:35:08.900 | in the sense that we do have a scientific way

00:35:13.580 | of computing what the adversarial example,

00:35:17.940 | what is the adversarial preservation we should add.

00:35:20.420 | And then, and of course, in the end,

00:35:23.540 | because of these additional steps, as I mentioned,

00:35:25.780 | you have to print it out, and then you have to put it on,

00:35:28.900 | and then you have to take the camera,

00:35:30.300 | and then, so there are additional steps

00:35:32.180 | that you do need to do additional testing.

00:35:34.100 | But the creation process

00:35:36.300 | of generating the adversarial example,

00:35:39.620 | it's really a very scientific approach.

00:35:44.100 | Essentially, we capture many of these constraints,

00:35:48.660 | as we mentioned, in this loss function

00:35:52.300 | that we optimize for.

00:35:55.220 | And so that's a very scientific approach.

00:35:58.780 | - So the fascinating fact

00:36:00.500 | that we can do these kinds of adversarial examples,

00:36:02.700 | what do you think it shows us?

00:36:05.020 | Just your thoughts in general,

00:36:07.460 | what do you think it reveals to us about neural networks?

00:36:10.020 | The fact that this is possible.

00:36:12.100 | What do you think it reveals to us

00:36:13.420 | about our machine learning approaches of today?

00:36:16.340 | Is there something interesting?

00:36:17.780 | Is it a feature?

00:36:18.620 | Is it a bug?

00:36:19.500 | What do you think?

00:36:20.860 | - I think it mainly shows that we are still

00:36:23.740 | at a very early stage of really developing robust

00:36:29.900 | and generalizable machine learning methods.

00:36:33.460 | And it shows that we,

00:36:35.260 | even though deep learning has made so much advancement,

00:36:39.420 | but our understanding is very limited.

00:36:42.180 | We don't fully understand,

00:36:44.180 | we don't understand well how they work, why they work,

00:36:47.260 | and also we don't understand that well,

00:36:49.220 | right, about these adversarial examples.

00:36:52.980 | - Some people have kind of written about the fact that,

00:36:58.740 | that the fact that the adversarial examples work well

00:37:02.820 | is actually sort of a feature, not a bug.

00:37:04.940 | It's that, that actually they have learned really well

00:37:09.220 | to tell the important differences between classes

00:37:12.020 | as represented by the training set.

00:37:14.140 | - I think that's the other thing I was going to say,

00:37:15.700 | is that it shows us also that the deep learning systems

00:37:18.940 | are now learning the right things.

00:37:21.180 | - How do we make them,

00:37:23.140 | I mean, I guess this might be a place to ask about

00:37:26.340 | how do we then defend,

00:37:27.860 | or how do we either defend

00:37:30.100 | or make them more robust, these adversarial examples?

00:37:32.820 | - Right, I mean, one thing is that I think,

00:37:35.220 | you know, people,

00:37:36.220 | so there have been actually thousands of papers now written

00:37:40.140 | on this topic.

00:37:41.580 | - The defense or the attacks?

00:37:43.780 | - Mostly attacks.

00:37:45.140 | I think there are more attack papers than defenses,

00:37:48.460 | but there are many hundreds of defense papers as well.

00:37:51.900 | So in defenses,

00:37:55.060 | a lot of work has been on trying to,

00:37:58.540 | I would call it more like a patchwork.

00:38:02.020 | For example, how to make the neural networks to either,

00:38:06.540 | through, for example, like adversarial training,

00:38:09.740 | how to make them a little bit more resilient.

00:38:13.340 | - Got it.

00:38:14.460 | - But I think in general,

00:38:17.580 | it has limited effectiveness,

00:38:21.340 | and we don't really have very strong

00:38:24.220 | and general defense.

00:38:27.980 | So part of that, I think is,

00:38:29.620 | we talked about in deep learning,

00:38:30.940 | the goal is to learn representations,

00:38:33.780 | and that's our ultimate,

00:38:36.540 | you know, holy grail,

00:38:37.540 | our ultimate goal is to learn representations.

00:38:39.940 | But one thing I think I have to say is that

00:38:42.980 | I think part of the lesson we're learning here

00:38:44.420 | is that we are, one,

00:38:45.620 | as I mentioned, we are not learning the right things,

00:38:47.500 | meaning we are not learning the right representations.

00:38:49.820 | And also I think the representations we are learning

00:38:51.980 | is not rich enough.

00:38:54.620 | And so it's just like a human vision,

00:38:56.900 | of course, we don't fully understand

00:38:58.140 | how human visions work,

00:38:59.580 | but when humans look at the world,

00:39:01.700 | we don't just say, oh, you know, this is a person,

00:39:04.460 | oh, that's a camera.

00:39:06.100 | We actually get much more nuanced information

00:39:09.060 | from the world.

00:39:11.820 | And we use all this information together in the end

00:39:14.820 | to derive, to help us to do motion planning

00:39:17.700 | and to do other things,

00:39:18.540 | but also to classify what the object is and so on.

00:39:22.180 | So we are learning a much richer representation,

00:39:24.580 | and I think that that's something

00:39:26.740 | we have not figured out how to do in deep learning.

00:39:30.580 | And I think the richer representation

00:39:33.060 | will also help us to build a more generalizable

00:39:36.420 | and more resilient learning system.

00:39:39.100 | - Can you maybe linger on the idea

00:39:40.700 | of the word richer representation?

00:39:43.180 | So to make representations more,

00:39:48.660 | generalizable, it seems like you want to make them more,

00:39:53.540 | less sensitive to noise.

00:39:55.300 | - Right, so you want to learn the right things.

00:39:58.420 | You don't want to, for example,

00:40:00.020 | learn this spurious correlations and so on.

00:40:05.020 | But at the same time,

00:40:06.420 | an example of a richer information,

00:40:09.620 | our representation is like,

00:40:11.500 | again, we don't really know how human vision works,

00:40:14.900 | but when we look at the visual world,

00:40:18.100 | we actually, we can identify counters,

00:40:20.820 | we can identify much more information

00:40:24.700 | than just what's, for example,

00:40:26.940 | a image classification system is trying to do.

00:40:29.420 | And that leads to, I think,

00:40:32.420 | the question you asked earlier about defenses.

00:40:34.620 | So that's also in terms of more promising directions

00:40:38.620 | for defenses, and that's where some of my work

00:40:42.660 | is trying to do and trying to show as well.

00:40:46.500 | - You have, for example, in your 2018 paper,

00:40:49.140 | characterizing adversarial examples

00:40:50.980 | based on spatial consistency information

00:40:53.900 | for semantic segmentation.

00:40:55.380 | So that's looking at some ideas

00:40:57.180 | on how to detect adversarial examples.

00:41:00.100 | So like, I guess, what are they?

00:41:02.980 | You call them like a poison dataset.

00:41:04.820 | So like, yeah, adversarial bad examples

00:41:07.820 | in a segmentation dataset.

00:41:09.420 | Can you, as an example for that paper,

00:41:11.900 | can you describe the process of defense there?

00:41:13.980 | - Yeah, sure, sure.

00:41:14.940 | So in that paper, what we look at

00:41:17.180 | is the semantic segmentation task.

00:41:21.060 | So with the task, essentially given an image

00:41:23.580 | for each pixel, you want to say

00:41:24.980 | what the label is for the pixel.

00:41:26.740 | So just like what we talked about for adversarial example,

00:41:32.540 | it can easily fool image classification systems.

00:41:35.420 | It turns out that it can also very easily

00:41:38.060 | fool these segmentation systems as well.

00:41:41.140 | So given an image, I essentially can

00:41:43.900 | add adversarial perturbation to the image

00:41:46.180 | to cause the segmentation system

00:41:49.500 | to basically segment it in any pattern I wanted.

00:41:53.580 | So in that paper, we also showed

00:41:56.220 | that you can segment it,

00:41:58.140 | even though there's no kitty in the image,

00:42:01.340 | we can segment it into like a kitty pattern,

00:42:05.140 | a Hello Kitty pattern,

00:42:06.980 | we segment it into like an ICCV.

00:42:09.460 | (both laughing)

00:42:10.580 | - That's awesome.

00:42:11.540 | - Right, so that's on the attack side,

00:42:14.140 | showing that the segmentation system,

00:42:15.820 | even though they have been effective in practice,

00:42:20.140 | but at the same time, they're really easily fooled.

00:42:24.180 | So then the question is how can we defend against this?

00:42:26.660 | How we can build a more resilient segmentation system?

00:42:30.860 | So that's what we try to do.

00:42:34.380 | And in particular, what we are trying to do here

00:42:37.060 | is to actually try to leverage some natural constraints

00:42:41.220 | in the task, which we call in this case, spatial consistency.

00:42:46.060 | So the idea of the spatial consistency is the following.

00:42:50.580 | So again, we don't really know how human vision works,

00:42:55.220 | but in general, at least what we can say is,

00:42:58.900 | so for example, as a person looks at a scene,

00:43:03.100 | and we can segment the scene easily.

00:43:07.180 | - We humans.

00:43:08.300 | - Right, yes.

00:43:09.740 | And then if you pick like two patches of the scene

00:43:14.140 | that has an intersection,

00:43:16.380 | and for humans, if you segment, you know,

00:43:19.980 | like patch A and patch B,

00:43:22.300 | and then you look at the segmentation results,

00:43:24.660 | and especially if you look at the segmentation results

00:43:27.140 | at the intersection of the two patches,

00:43:29.860 | they should be consistent in the sense that

00:43:32.100 | what the label, what the pixels in this intersection,

00:43:36.980 | what their labels should be,

00:43:38.940 | and they essentially from these two different patches,

00:43:42.180 | they should be similar in the intersection.

00:43:44.580 | So that's what we call spatial consistency.

00:43:47.780 | So similarly, for a segmentation system,

00:43:52.940 | it should have the same property, right?

00:43:56.340 | So in the image, if you pick two,

00:43:59.980 | randomly pick two patches that has an intersection,

00:44:04.060 | you feed each patch to the segmentation system,

00:44:06.740 | you get a result,

00:44:08.140 | and when you look at the results in the intersection,

00:44:12.140 | the segmentation results should be very similar.

00:44:15.380 | - Is that, so, okay, so logically that kind of makes sense,

00:44:20.540 | at least it's a compelling notion,

00:44:21.940 | but is that, how well does that work?

00:44:24.700 | Does that hold true for segmentation?

00:44:27.460 | - Exactly, exactly.

00:44:28.300 | So then in our work and experiments,

00:44:31.700 | we showed the following.

00:44:33.140 | So when we take normal images,

00:44:37.340 | this actually holds pretty well

00:44:39.340 | for the segmentation systems that we experimented with.

00:44:42.460 | - Or like, did you look at driving data sets?

00:44:45.100 | - Right, right, right, exactly, exactly.

00:44:47.820 | But then this actually poses a challenge

00:44:49.900 | for adversarial examples,

00:44:52.220 | because for the attacker to add perturbation to the image,

00:44:57.060 | then it's easy for it to fool the segmentation system

00:45:00.980 | into, for example, for a particular patch,

00:45:03.100 | or for the whole image,

00:45:04.660 | to cause the segmentation system to create some,

00:45:08.860 | to get to some wrong results.

00:45:10.900 | But it's actually very difficult for the attacker

00:45:13.820 | to have this adversarial example

00:45:18.820 | to satisfy the spatial consistency,

00:45:21.300 | because these patches are randomly selected,

00:45:23.700 | and they need to ensure

00:45:24.980 | that this spatial consistency works.

00:45:27.660 | So they basically need to fool the segmentation system

00:45:31.380 | in a very consistent way.

00:45:33.860 | - Yeah, without knowing the mechanism

00:45:35.460 | by which you're selecting the patches or so on.

00:45:37.460 | - Exactly, exactly.

00:45:38.300 | - So it has to really fool the entirety of the,

00:45:40.340 | mess with the entirety of the thing.

00:45:41.620 | - So it turns out to actually to be really hard

00:45:44.140 | for the attacker to do.

00:45:45.100 | We try the best we can,

00:45:47.340 | this detail for the art attacks,

00:45:49.340 | it actually showed that this defense method

00:45:52.140 | is actually very, very effective.

00:45:54.420 | And this goes to, I think also what I was saying earlier is,

00:45:58.980 | essentially we want the learning system

00:46:02.620 | to have rich, rich sensation,

00:46:05.060 | and also to learn from more,

00:46:07.580 | you can add the same model,

00:46:09.060 | essentially to have more ways to check

00:46:11.500 | whether it's actually having the right prediction.

00:46:16.140 | So for example, in this case,

00:46:17.620 | doing the spatial consistency check.

00:46:19.820 | And also actually, so that's one paper that we did,

00:46:23.180 | and then this is spatial consistency,

00:46:24.500 | this notion of consistency check,

00:46:26.580 | it's not just limited to spatial properties,

00:46:30.540 | it also applies to audio.

00:46:32.260 | So we actually had follow up work in audio

00:46:35.340 | to show that this temporal consistency

00:46:38.060 | can also be very effective

00:46:39.540 | in detecting adversary examples in audio.

00:46:42.660 | - Like speech or what kind of audio?

00:46:44.060 | - Right, right, right, exactly.

00:46:44.900 | - Speech, speech data?

00:46:46.060 | - Right, and then we can actually combine

00:46:49.020 | spatial consistency and temporal consistency

00:46:51.780 | to help us to develop more resilient methods in video.

00:46:56.700 | So to defend against attacks for video also.

00:46:59.220 | - That's fascinating.

00:47:00.060 | - Right, so yeah, so it's very interesting.

00:47:00.900 | - So there's hope.

00:47:01.900 | - Yes, yes.

00:47:02.740 | (both laughing)

00:47:04.460 | - But in general, in the literature

00:47:07.740 | and the ideas that are developing the attacks,

00:47:09.540 | and the literature that's developing the defense,

00:47:11.620 | who would you say is winning right now?

00:47:13.820 | - Right now, of course, it's the attack side.

00:47:15.860 | It's much easier to develop attacks,

00:47:18.500 | and there are so many different ways to develop attacks.

00:47:21.260 | Even just us, we develop so many different methods

00:47:24.420 | for doing attacks.

00:47:27.380 | And also you can do white box attacks,

00:47:29.660 | you can do black box attacks,

00:47:31.460 | where attacks you don't even need,

00:47:33.460 | the attacker doesn't even need to know

00:47:36.500 | the architecture of the target system,

00:47:39.620 | and not knowing the parameters of the target system,

00:47:42.700 | and all that.

00:47:43.700 | So there are so many different types of attacks.

00:47:46.380 | - So the counter argument that people would have,

00:47:49.500 | like people that are using machine learning in companies,

00:47:52.540 | they would say, "Sure, in constrained environments

00:47:55.820 | "and very specific dataset,

00:47:57.180 | "when you know a lot about the model,

00:47:59.900 | "or you know a lot about the dataset already,

00:48:02.820 | "you'll be able to do this attack.

00:48:04.260 | "It's very nice, it makes for a nice demo,

00:48:05.940 | "it's a very interesting idea,

00:48:07.500 | "but my system won't be able to be attacked like this.

00:48:10.580 | "The real world systems won't be able

00:48:12.020 | "to be attacked like this."

00:48:14.020 | That's another hope, that it's actually a lot harder

00:48:18.100 | to attack real world systems.

00:48:20.140 | Can you talk to that?

00:48:21.260 | - Yeah, I--

00:48:22.100 | - How hard is it to attack real world systems, I guess?

00:48:24.740 | - I wouldn't call that a hope,

00:48:26.500 | I think it's more of a wishful thinking,

00:48:30.100 | or trying to be lucky.

00:48:33.060 | So actually, in our recent work,

00:48:37.380 | my students and collaborators

00:48:39.300 | have shown some very effective attacks

00:48:41.740 | on real world systems.

00:48:44.100 | For example, Google Translate.

00:48:46.220 | - Oh no.

00:48:47.060 | - And other cloud translation APIs.

00:48:52.060 | So in this work, we showed,

00:48:56.740 | so far I talked about adversary examples

00:48:58.700 | mostly in the vision category.

00:49:01.940 | And of course, adversary examples also work

00:49:05.220 | in other domains as well.

00:49:07.700 | For example, in natural language.

00:49:10.300 | So in this work, my students and collaborators

00:49:14.260 | have shown that, so one,

00:49:17.380 | we can actually very easily steal the model

00:49:22.020 | from, for example, Google Translate,

00:49:24.900 | by just doing queries through the APIs,

00:49:28.460 | and then we can train an imitation model ourselves

00:49:32.660 | using the queries.

00:49:34.300 | And then once we--

00:49:35.620 | And also, the imitation model can be very, very effective,

00:49:40.620 | essentially achieving similar performance.

00:49:44.380 | As a target model.

00:49:45.780 | And then once we have the imitation model,

00:49:48.060 | we can then try to create adversary examples

00:49:51.180 | on these imitation models.

00:49:52.860 | So for example, giving,

00:49:56.540 | you know, in the work it was,

00:49:58.460 | one example is translating from English to German.

00:50:01.860 | We can give it a sentence saying,

00:50:04.020 | for example, I'm feeling freezing,

00:50:06.460 | it's like six Fahrenheit.

00:50:10.900 | And then translating to German.

00:50:13.180 | And then we can actually generate adversary examples

00:50:16.340 | that create a target translation

00:50:18.900 | by very small perturbation.

00:50:20.580 | So in this case, let's say we want to

00:50:23.020 | change the translation instead of six Fahrenheit

00:50:27.700 | to 21 Celsius.

00:50:30.620 | And in this particular example,

00:50:32.340 | actually we just changed six to seven

00:50:35.660 | in the original sentence.

00:50:36.900 | That's the only change we made.

00:50:38.580 | It caused the translation to change

00:50:43.060 | from the six Fahrenheit into 21 Celsius.

00:50:46.380 | - That's incredible.

00:50:47.220 | - And then, so this example,

00:50:49.820 | we created this example from our imitation model.

00:50:53.460 | - An imitation model.

00:50:54.300 | - And then this work actually transfers

00:50:56.980 | to the Google Translate.

00:50:58.700 | - So the attacks that work on the imitation model,

00:51:01.340 | in some cases at least, transfer to the original model.

00:51:05.380 | That's incredible and terrifying.

00:51:07.260 | Okay.

00:51:08.660 | That's amazing work.

00:51:10.380 | And that shows that, again,

00:51:11.900 | real world systems actually can be easily fooled.

00:51:15.260 | And in our previous work,

00:51:16.420 | we also showed this type of black box attacks

00:51:18.620 | can be effective on cloud vision APIs as well.

00:51:22.140 | - So that's for natural language and for vision.

00:51:27.300 | Let's talk about another space

00:51:29.180 | that people have some concern about,

00:51:30.820 | which is autonomous driving as sort of security concerns.

00:51:35.060 | That's another real world system.

00:51:36.500 | So do you have, should people be worried

00:51:41.500 | about adversarial machine learning attacks

00:51:45.180 | in the context of autonomous vehicles

00:51:47.820 | that use like Tesla Autopilot, for example,

00:51:50.020 | that uses vision as a primary sensor

00:51:52.380 | for perceiving the world and navigating that world?

00:51:55.580 | What do you think?

00:51:56.620 | From your stop sign work in the physical world,

00:51:59.100 | should people be worried?

00:52:01.220 | How hard is that attack?

00:52:03.060 | - So actually there has already been,

00:52:05.620 | like there has always been like research shown that's,

00:52:10.300 | for example, actually even with Tesla,

00:52:11.860 | like if you put a few stickers on the roads,

00:52:15.340 | it can actually, once arranged in certain ways,

00:52:17.980 | it can fool the- (laughs)

00:52:20.620 | - That's right.

00:52:21.460 | But I don't think it's actually been,

00:52:23.060 | I might not be familiar,

00:52:24.620 | but I don't think it's been done on physical worlds,

00:52:26.900 | physical roads yet.

00:52:28.220 | Meaning I think it's with a projector in front of the Tesla.

00:52:31.500 | So it's a physical,

00:52:33.740 | so you're on the other side of the sensor,

00:52:36.140 | but you're not in still the physical world.

00:52:39.260 | The question is whether it's possible to orchestrate attacks

00:52:42.980 | that work in the actual, like end-to-end attacks,

00:52:47.100 | like not just a demonstration of the concept,

00:52:49.780 | but thinking is it possible on the highway

00:52:52.460 | to control a Tesla?

00:52:53.580 | That kind of idea.

00:52:55.420 | - I think there are two separate questions.

00:52:57.100 | One is the feasibility of the attack,

00:52:59.420 | and I'm 100% confident that's the-

00:53:02.460 | - It's possible.

00:53:03.300 | - It's possible.

00:53:04.260 | And there's a separate question

00:53:06.140 | whether someone will actually go deploy that attack.

00:53:11.140 | I hope people do not do that.

00:53:13.780 | - Yeah, but hold on a second.

00:53:14.620 | - But that's two separate questions.

00:53:16.380 | - So the question on the word feasibility.

00:53:18.780 | So to clarify, feasibility means it's possible.

00:53:22.700 | It doesn't say how hard it is,

00:53:25.700 | because to implement it.

00:53:28.740 | So sort of the barrier,

00:53:30.500 | like how much of a heist it has to be,

00:53:33.340 | like how many people have to be involved?

00:53:35.260 | What is the probability of success?

00:53:36.820 | That kind of stuff.

00:53:37.700 | And coupled with how many evil people there are in the world

00:53:41.620 | that would attempt such an attack, right?

00:53:43.700 | But the two, my question is, is it sort of,

00:53:47.140 | when I talked to Elon Musk and asked the same question,

00:53:52.460 | he says it's not a problem.

00:53:53.780 | It's very difficult to do in the real world.

00:53:55.980 | That this won't be a problem.

00:53:57.740 | He dismissed it as a problem

00:53:58.980 | for adversarial attacks on the Tesla.

00:54:01.260 | Of course, he happens to be involved with the company,

00:54:04.940 | so he has to say that.

00:54:06.260 | But let me linger on a little longer.

00:54:08.820 | Where does your confidence that it's feasible come from?

00:54:15.620 | And what's your intuition,

00:54:17.500 | how people should be worried,

00:54:18.700 | and how people should defend against it?

00:54:21.820 | How Tesla, how Waymo,

00:54:24.100 | how other autonomous vehicle companies

00:54:25.740 | should defend against sensory-based attacks,

00:54:29.460 | whether on LIDAR or on vision or so on?

00:54:32.420 | - And also, even for LIDAR, actually,

00:54:33.620 | there has been research shown that even LIDAR itself

00:54:36.180 | can be attacked as well.

00:54:37.020 | - No, no, no, no, but see, it's really important to pause.

00:54:40.380 | There's really nice demonstrations that it's possible to do,

00:54:44.860 | but there's so many pieces that it's kind of like,

00:54:48.020 | it's kind of in the lab.

00:54:51.780 | Now, it's in the physical world,

00:54:53.420 | meaning it's in the physical space, the attacks,

00:54:55.740 | but it's very, you have to control a lot of things

00:54:58.860 | to pull it off.

00:54:59.940 | It's like the difference between opening a safe

00:55:04.340 | when you have it and you have unlimited time

00:55:07.460 | and you can work on it versus breaking into the crown,

00:55:12.300 | stealing the crown jewels or whatever, right?

00:55:14.820 | - So one way to look at it,

00:55:16.940 | in terms of how real these attacks can be,

00:55:20.100 | one way to look at it is that actually,

00:55:21.780 | you don't even need any sophisticated attacks.

00:55:25.340 | Already, we've seen many real-world examples, incidents,

00:55:30.340 | where showing that the vehicle was making the wrong decision.

00:55:34.500 | - The wrong decision without attacks, right?

00:55:36.220 | - Right, right, so that's one way to demonstrate.

00:55:38.660 | And this is also, so far, we've mainly talked about work

00:55:41.900 | in this adversarial setting,

00:55:44.100 | showing that today's learning system,

00:55:46.420 | they are so vulnerable to the adversarial setting,

00:55:49.020 | but at the same time, actually, we also know

00:55:51.140 | that even in natural settings,

00:55:53.060 | these learning systems, they don't generalize well,

00:55:55.620 | and hence, they can really misbehave

00:55:58.140 | under certain situations, like what we have seen.

00:56:02.340 | And hence, I think using that as an example,

00:56:04.780 | it can show that these issues can be real.

00:56:08.300 | - They can be real, but so there's two cases.

00:56:10.620 | One is something, it's like perturbations

00:56:14.180 | can make the system misbehave

00:56:16.140 | versus make the system do one specific thing

00:56:19.300 | that the attacker wants, as you said, the targeted--

00:56:22.740 | - Targeted attack, right.

00:56:23.820 | - That seems to be very difficult,

00:56:27.540 | like an extra level of difficult step in the real world.

00:56:31.580 | But from the perspective of the passenger of the car,

00:56:34.460 | I don't think it matters either way.

00:56:37.100 | - Right, right, right.

00:56:38.180 | - Whether it's misbehavior or a targeted attack, okay.

00:56:42.340 | - And also, that's why I was also saying earlier,

00:56:45.260 | like when defense is this multi-model defense,

00:56:48.780 | and more of these consistent checks and so on.

00:56:51.100 | So in the future, I think also it's important

00:56:53.460 | that for these autonomous vehicles,

00:56:56.460 | they have lots of different sensors,

00:56:58.660 | and they should be combining all these sensory readings

00:57:02.660 | to arrive at the decision

00:57:05.420 | and the interpretation of the world and so on.

00:57:08.500 | And the more of these sensory inputs they use,

00:57:12.180 | and the better they combine the sensory inputs,

00:57:14.580 | the harder it is going to be attacked.

00:57:17.020 | And hence, I think that is a very important direction

00:57:19.780 | for us to move towards.

00:57:21.740 | - So multi-modal, multi-sensor across multiple cameras,

00:57:25.380 | but also in the case of car, radar, ultrasonic, sound even.

00:57:30.060 | So all of those--

00:57:31.380 | - Right, right, right, exactly.

00:57:33.420 | - So another thing, another part of your work

00:57:36.300 | has been in the space of privacy.

00:57:39.220 | And that too can be seen as a kind of security vulnerability.

00:57:43.980 | And so thinking of data as a thing

00:57:46.780 | that should be protected,

00:57:47.900 | and the vulnerabilities to data is essentially,

00:57:52.900 | the thing that you want to protect

00:57:55.180 | is the privacy of that data.

00:57:56.980 | So what do you see as the main vulnerabilities

00:57:59.820 | in the privacy of data, and how do we protect it?

00:58:02.260 | - Right, so in security, we actually talk about

00:58:05.620 | essentially two, in this case, two different properties.

00:58:10.180 | One is integrity, and one is confidentiality.

00:58:13.500 | So what we have been talking earlier

00:58:17.220 | is essentially the integrity property

00:58:21.660 | of the learning system.

00:58:22.860 | How to make sure that the learning system

00:58:24.820 | is giving the right prediction, for example.

00:58:29.020 | And privacy essentially is on the other side,

00:58:32.300 | is about confidentiality of the system,

00:58:34.900 | is how attackers can, when the attackers compromise

00:58:39.620 | the confidentiality of the system,

00:58:42.460 | that's when the attacker is steal sensitive information,

00:58:45.300 | about individuals and so on.

00:58:48.500 | - That's really clean.

00:58:49.900 | Those are great terms, integrity and confidentiality.

00:58:53.580 | - Right.

00:58:54.420 | - So what are the main vulnerabilities to privacy,

00:58:58.700 | would you say, and how do we protect against it?

00:59:01.660 | Like what are the main spaces and problems

00:59:04.580 | that you think about in the context of privacy?

00:59:07.140 | - Right, so especially in the machine learning setting,

00:59:12.380 | so in this case, as we know that how the process goes

00:59:16.740 | is that we have the training data,

00:59:19.620 | and then the machine learning system

00:59:22.540 | trains from this training data and then builds a model.

00:59:25.780 | And then later on, inputs are given to the model

00:59:29.220 | to inference time to try to get prediction and so on.

00:59:34.020 | So then in this case, the privacy concerns that we have

00:59:38.300 | is typically about privacy of the data in the training data,

00:59:43.100 | because that's essentially the private information.

00:59:45.580 | So, and it's really important because oftentimes

00:59:50.580 | the training data can be very sensitive.

00:59:53.900 | It can be your financial data, it's your health data,

00:59:56.980 | or like in IoT case, it's the sensors deployed

01:00:01.700 | in real world environment and so on.

01:00:03.740 | And all this can be collecting very sensitive information.

01:00:08.500 | And all the sensitive information gets fed

01:00:11.220 | into the learning system and trains.

01:00:13.740 | And as we know, these neural networks,

01:00:16.660 | they can have really high capacity

01:00:19.380 | and they actually can remember a lot.

01:00:23.180 | And hence, just from the learned model in the end,

01:00:27.580 | actually attackers can potentially infer information

01:00:31.900 | about their original training data set.

01:00:36.860 | - So the thing you're trying to protect

01:00:38.540 | is the confidentiality of the training data.

01:00:42.820 | And so what are the methods for doing that?

01:00:44.620 | Would you say, what are the different ways

01:00:46.220 | that can be done?

01:00:47.780 | - And also we can talk about essentially

01:00:49.620 | how the attacker may try to learn information from the...

01:00:54.620 | So, and also there are different types of attacks.

01:00:57.740 | So in certain cases, again, like in white box attacks,

01:01:01.220 | we can say that the attacker actually get to see

01:01:03.860 | the parameters of the model.

01:01:05.660 | And then from that, a smart attacker potentially

01:01:08.740 | can try to figure out information

01:01:11.380 | about the training data set.

01:01:13.940 | They can try to figure out what type of data

01:01:16.860 | has been in the training data sets.

01:01:18.620 | And sometimes they can tell whether a person has been,

01:01:23.020 | a particular person's data point has been used

01:01:27.220 | in the training data sets as well.

01:01:29.060 | - So white box, meaning you have access

01:01:30.980 | to the parameters of say neural network.

01:01:33.540 | - Right.

01:01:34.540 | - You're saying that it's some,

01:01:36.580 | given that information is possible to some...

01:01:38.900 | - So I can give you some examples.

01:01:40.420 | And then another type of attack,

01:01:41.820 | which is even easier to carry out is,

01:01:44.420 | not a white box model, it's more of a,

01:01:47.740 | just a query model where the attacker

01:01:49.940 | only gets to query the machine learning model

01:01:52.620 | and then try to steal sensitive information

01:01:55.380 | in the original training data.

01:01:57.060 | So, right, so I can give you an example.

01:01:59.540 | In this case, training a language model.

01:02:03.740 | So in our work, in collaboration

01:02:06.340 | with the researchers from Google,

01:02:08.140 | we actually studied the following question.

01:02:10.700 | So at high level, the question is,

01:02:13.660 | as we mentioned, the neural networks

01:02:15.900 | can have very high capacity

01:02:17.220 | and they could be remembering a lot

01:02:19.460 | from the training process.

01:02:21.620 | Then the question is, can attacker actually exploit this

01:02:25.540 | and try to actually extract sensitive information

01:02:28.700 | in the original training data sets

01:02:31.180 | through just querying the learned model

01:02:34.220 | without even knowing the parameters of the model,

01:02:37.140 | like the details of the model,

01:02:38.780 | or the architectures of the model and so on.

01:02:41.900 | So that's the question we set out to explore.

01:02:46.860 | And in one of the case studies, we showed the following.

01:02:50.860 | So we trained a language model over an email data sets.

01:02:55.060 | It's called an nRAN email data sets.

01:02:57.420 | And the nRAN email data sets naturally contains

01:03:01.180 | users' social security numbers and credit card numbers.

01:03:04.420 | So we trained a language model over this data sets,

01:03:08.500 | and then we showed that an attacker,

01:03:11.180 | by devising some new attacks,

01:03:13.220 | by just querying the language model,

01:03:15.940 | and without knowing the details of the model,

01:03:19.140 | the attacker actually can extract

01:03:23.020 | the original social security numbers

01:03:26.060 | and credit card numbers that were in

01:03:27.700 | the original training data sets.

01:03:30.300 | - So get the most sensitive,

01:03:31.460 | personally identifiable information from the data set

01:03:34.740 | from just querying it.

01:03:36.180 | - Right, yeah.

01:03:39.180 | So that's an example showing that's why,

01:03:42.820 | even as we train machine learning models,

01:03:45.940 | we have to be really careful

01:03:48.300 | with protecting users' data privacy.

01:03:51.540 | - So what are the mechanisms for protecting?

01:03:53.700 | Is there hopeful, so there's been recent work

01:03:57.620 | on non-differential privacy, for example,

01:03:59.500 | that provides some hope,

01:04:02.620 | but can you describe some of the ideas?

01:04:04.460 | - So that's actually, right,

01:04:05.540 | so that's also our finding,

01:04:07.180 | is that by, actually, we show that

01:04:11.300 | in this particular case, we actually have a good defense.

01:04:14.260 | - For the querying case, for the querying--

01:04:16.020 | - For this language model case.

01:04:16.980 | - Language model case.

01:04:17.820 | - So instead of just training a vanilla language model,

01:04:22.980 | instead, if we train a differentially private

01:04:25.420 | language model, then we can still achieve similar utility,

01:04:30.060 | but at the same time, we can actually significantly enhance

01:04:34.540 | the privacy protection of the learned model,

01:04:39.340 | and our proposed attacks actually are no longer effective.

01:04:43.980 | - And differential privacy is a mechanism

01:04:47.140 | of adding some noise by which you then have some guarantees

01:04:51.420 | on the inability to figure out the presence

01:04:56.420 | of a particular person in the dataset.

01:04:59.780 | - So, right, so in this particular case,

01:05:01.780 | what the differential privacy mechanism does

01:05:05.420 | is that it actually adds perturbation

01:05:09.420 | in the training process.

01:05:10.620 | As we know, during the training process,

01:05:12.900 | we are learning the model, we are doing gradient updates,

01:05:17.180 | the weight updates, and so on,

01:05:19.020 | and essentially, differential privacy,

01:05:22.660 | differentially private machine learning algorithm,

01:05:26.340 | in this case, will be adding noise

01:05:29.660 | and adding various perturbation during this training process.

01:05:33.740 | - To some aspect of the training process.

01:05:35.780 | - Right, so then the finally trained learning,

01:05:39.660 | the learned model is differentially private,

01:05:42.500 | and so it can enhance the privacy protection.

01:05:46.660 | - So, okay, so that's the attacks

01:05:48.300 | and the defense of privacy.

01:05:50.340 | You also talk about ownership of data.

01:05:54.340 | So this is a really interesting idea

01:05:56.620 | that we get to use many services online

01:05:59.060 | for seemingly for free by essentially,

01:06:04.060 | sort of a lot of companies are funded through advertisement,

01:06:06.820 | and what that means is the advertisement works

01:06:09.820 | exceptionally well because the companies are able

01:06:12.060 | to access our personal data,

01:06:13.700 | so they know which advertisement to serve us

01:06:16.260 | to do targeted advertisements and so on.

01:06:19.020 | So can you maybe talk about this?

01:06:21.900 | You have some nice paintings of the future,

01:06:26.260 | philosophically speaking future,

01:06:28.620 | where people can have a little bit more control

01:06:31.780 | of their data by owning and maybe understanding

01:06:35.780 | the value of their data and being able to sort of monetize it

01:06:40.540 | in a more explicit way as opposed to the implicit way

01:06:43.500 | that it's currently done.

01:06:45.140 | - Yeah, I think this is a fascinating topic

01:06:47.460 | and also a really complex topic.

01:06:50.140 | Right, I think there are these natural questions,

01:06:53.940 | who should be owning the data?

01:06:57.100 | And so I can draw one analogy.

01:07:03.260 | So for example, for physical properties,

01:07:06.860 | like your house and so on,

01:07:08.420 | so really this notion of property rights

01:07:13.220 | is not just, it's not like from day one,

01:07:17.220 | we knew that there should be this clear notion

01:07:20.620 | of ownership of properties and having enforcement for this.

01:07:25.420 | And so actually people have shown that this establishment

01:07:30.420 | and enforcement of property rights has been a main driver

01:07:41.020 | for the economy earlier.

01:07:43.340 | And that actually really propelled the economic growth

01:07:47.660 | even in the earlier stage.

01:07:51.340 | - So throughout the history of the development

01:07:54.140 | of the United States or actually just civilization,

01:07:57.180 | the idea of property rights that you can own property--

01:08:00.540 | - Right, and then there's enforcement,

01:08:02.180 | there's institutional rights,

01:08:04.580 | like governmental, like enforcement of this

01:08:08.020 | actually has been a key driver for economic growth.

01:08:12.100 | And there had been even research or proposals saying

01:08:16.500 | that for a lot of the developing countries,

01:08:18.900 | essentially the challenge in growth

01:08:25.180 | is not actually due to the lack of capital,

01:08:29.020 | it's more due to the lack of this notion of property rights

01:08:34.020 | and the enforcement of property rights.

01:08:37.140 | - Interesting, so that the presence of absence

01:08:41.660 | of both the concept of the property rights

01:08:45.140 | and their enforcement has a strong correlation

01:08:48.180 | to economic growth.

01:08:49.940 | - Right, right.

01:08:50.820 | - And so you think that that same could be transferred

01:08:54.180 | to the idea of property ownership

01:08:56.300 | in the case of data ownership?

01:08:57.940 | - I think first of all, it's a good lesson for us

01:09:01.340 | to recognize that these rights and the recognition

01:09:06.620 | and the enforcement of these type of rights

01:09:10.060 | is very, very important for economic growth.

01:09:13.260 | And then if we look at where we are now

01:09:15.780 | and where we are going in the future,

01:09:18.460 | so essentially more and more

01:09:19.820 | is actually moving into the digital world.

01:09:23.540 | And also more and more, I would say,

01:09:26.300 | even like information or assets of a person

01:09:30.420 | is more and more into the physical,

01:09:34.140 | sorry, the digital world as well.

01:09:35.820 | It's the data that the person has generated.

01:09:39.940 | And essentially, it's like in the past,

01:09:43.060 | what defines a person?

01:09:44.860 | You can say, right, like oftentimes,

01:09:47.340 | besides the innate capabilities,

01:09:50.980 | actually it's the physical properties.

01:09:54.300 | - House, car.

01:09:55.340 | - Right, that defines a person.

01:09:56.780 | But I think more and more people start to realize

01:09:59.580 | actually what defines a person

01:10:01.460 | is more and more in the data that the person has generated

01:10:04.900 | or the data about the person.

01:10:06.420 | Like all the way from your political views,

01:10:10.500 | your music taste and your financial information,

01:10:14.420 | a lot of these, and your health.

01:10:16.860 | So more and more of the definition of the person

01:10:20.180 | is actually in the digital world.

01:10:22.140 | - And currently, for the most part, that's owned.

01:10:24.580 | People don't talk about it,

01:10:27.300 | but kind of it's owned by internet companies.

01:10:32.300 | So it's not owned by individuals.

01:10:34.620 | - Right, there's no clear notion of ownership

01:10:38.140 | of such data.

01:10:39.180 | And also, we talk about privacy and so on,

01:10:41.860 | but I think actually clearly identifying the ownership

01:10:45.620 | is a first step.

01:10:46.620 | Once you identify the ownership,

01:10:48.380 | then you can say who gets to define

01:10:50.740 | how the data should be used.

01:10:52.380 | So maybe some users are fine with internet companies

01:10:57.380 | serving them as, right, using their data

01:11:02.100 | as long as if the data is used in a certain way

01:11:05.780 | that actually the user consents with or allows.

01:11:10.780 | For example, you can say the recommendation system

01:11:14.540 | in some sense, we don't call it an ad,

01:11:16.780 | but a recommendation system,

01:11:18.380 | similarly, it's trying to recommend you something.

01:11:20.780 | And users enjoy and can really benefit

01:11:24.020 | from good recommendation systems,

01:11:25.660 | either recommending you better music, movies, news,

01:11:29.380 | even research papers to read.

01:11:31.860 | But of course, then in this targeted ads,

01:11:35.820 | especially in certain cases where people can be manipulated

01:11:40.460 | by these targeted ads,

01:11:42.300 | they can have really bad, severe consequences.

01:11:45.740 | So essentially, users want their data to be used

01:11:50.340 | to better serve them,

01:11:51.820 | and also maybe even, right, get paid for or whatever,

01:11:54.740 | like in different settings.

01:11:56.380 | But the thing is that first of all,

01:11:57.780 | we need to really establish who needs to decide,

01:12:02.780 | who can decide how the data should be used.

01:12:06.220 | And typically, the establishment and clarification

01:12:10.100 | of the ownership will help this,

01:12:12.100 | and it's an important first step.

01:12:14.700 | So if the user is the owner,

01:12:16.300 | then naturally the user gets to define

01:12:18.380 | how the data should be used.

01:12:19.980 | But if you even say that, wait a minute,

01:12:22.620 | users are actually now the owner of this data,

01:12:24.460 | whoever is collecting the data is the owner of the data,

01:12:26.740 | now of course they get to use the data

01:12:28.180 | however way they want.

01:12:29.980 | So to really address these complex issues,

01:12:33.940 | we need to go at the root cause.

01:12:35.980 | So it seems fairly clear that first we really need to say

01:12:40.980 | that who is the owner of the data,

01:12:42.580 | and then the owners can specify

01:12:45.140 | how they want their data to be utilized.

01:12:47.140 | - So that's a fascinating,

01:12:49.620 | most people don't think about that,

01:12:52.660 | and I think that's a fascinating thing to think about,

01:12:54.980 | and probably fight for it.

01:12:56.660 | I can only see, and the economic growth argument

01:12:59.660 | is probably a really strong one.

01:13:01.060 | So that's a first time I'm kind of at least thinking about

01:13:05.180 | the positive aspect of that ownership

01:13:08.140 | being the long-term growth of the economy,

01:13:11.260 | so good for everybody.

01:13:12.300 | But sort of one possible downside I could see,

01:13:15.540 | sort of to put on my grumpy old grandpa hat,

01:13:20.540 | and it's really nice for Facebook and YouTube and Twitter

01:13:25.900 | to all be free.

01:13:27.100 | And if you give control to people with their data,

01:13:31.700 | do you think it's possible they will be,

01:13:34.820 | they would not want to hand it over quite easily?

01:13:37.660 | And so a lot of these companies

01:13:39.540 | that rely on mass handover of data,

01:13:42.820 | and then therefore provide a mass seemingly free service,

01:13:47.820 | would then completely,

01:13:51.060 | so the way the internet looks will completely change

01:13:56.060 | because of the ownership of data,

01:13:57.660 | and we'll lose a lot of services value.

01:14:00.740 | Do you worry about that?

01:14:01.580 | - So that's a very good question.

01:14:03.780 | I think that's not necessarily the case,

01:14:06.060 | in the sense that, yes, users can have ownership

01:14:10.060 | of their data, they can maintain control of their data,

01:14:12.860 | but also then they get to decide

01:14:14.820 | how their data can be used.

01:14:17.540 | So that's why I mentioned earlier,

01:14:20.020 | in this case, if they feel that they enjoy the benefits

01:14:23.540 | of social networks and so on,

01:14:25.500 | and they're fine with having Facebook, having their data,

01:14:29.580 | but utilizing the data in certain way that they agree,

01:14:33.980 | then they can still enjoy the free services.

01:14:37.260 | But for others, maybe they would prefer

01:14:40.020 | some kind of private vision,

01:14:42.020 | and in that case, maybe they can even opt in

01:14:44.540 | to say that I want to pay, and to have,

01:14:47.900 | so for example, it's already fairly standard,

01:14:50.860 | like you pay for certain subscriptions

01:14:53.500 | so that you don't get to be shown ads.

01:14:57.340 | - See ads, yeah. - Right.

01:14:59.180 | So then users essentially can have choices,

01:15:02.020 | and I think we just want to essentially bring out

01:15:06.340 | more about who gets to decide what to do with the data.

01:15:10.860 | - I think it's an interesting idea,

01:15:11.980 | 'cause if you poll people now,

01:15:15.220 | it seems like, I don't know,

01:15:16.860 | but subjectively, sort of anecdotally speaking,

01:15:19.180 | it seems like a lot of people don't trust Facebook.

01:15:22.140 | So that's at least a very popular thing to say

01:15:24.420 | that I don't trust Facebook, right?

01:15:26.980 | I wonder if you give people control of their data,

01:15:30.540 | as opposed to sort of signaling to everyone

01:15:33.180 | that they don't trust Facebook,

01:15:34.900 | I wonder how they would speak with the actual,

01:15:37.940 | like, would they be willing to pay $10 a month for Facebook,

01:15:42.500 | or would they hand over their data?

01:15:44.860 | It'd be interesting to see what fraction of people

01:15:47.500 | would quietly hand over their data to Facebook

01:15:51.300 | to make it free.

01:15:52.620 | I don't have a good intuition about that.

01:15:55.020 | How many people, do you have an intuition

01:15:57.580 | about how many people would use their data effectively

01:16:01.540 | on the market of the internet

01:16:06.540 | by sort of buying services with their data?

01:16:09.940 | - Yeah, so that's a very good question.

01:16:12.380 | I think, so one thing I also want to mention

01:16:15.900 | is that this, right, so it seems that,

01:16:19.980 | especially in press, the conversation has been very much

01:16:24.980 | like two sides fighting against each other.

01:16:29.140 | On one hand, right, users can say that, right,

01:16:33.500 | they don't trust Facebook, they don't,

01:16:35.420 | others delete Facebook.

01:16:37.580 | - Yeah, yeah, exactly.

01:16:39.140 | - Right, and then on the other hand,

01:16:42.820 | right, of course, right,

01:16:46.140 | the other side, they also feel,

01:16:48.060 | oh, they are providing a lot of services to users,

01:16:50.540 | and users are getting it all for free.

01:16:52.620 | So I think I actually, you know,

01:16:57.420 | I talk a lot to different companies,

01:17:00.540 | and also, like basically on both sides.

01:17:02.580 | So one thing I hope also,

01:17:06.380 | like this is my hope for this year also,

01:17:09.220 | is that we want to establish a more constructive dialogue,

01:17:14.220 | and to help people to understand

01:17:18.700 | that the problem is much more nuanced

01:17:21.900 | than just this two sides fighting.

01:17:25.540 | Because naturally, there is a tension

01:17:29.620 | between the two sides, between utility and privacy.

01:17:34.340 | So if you want to get more utility,

01:17:35.980 | essentially, like the recommendation system example

01:17:39.700 | I gave earlier,

01:17:41.500 | if you want someone to give you a good recommendation,

01:17:44.380 | essentially, whatever that system is,

01:17:46.100 | the system is going to need to know your data

01:17:49.460 | to give you a good recommendation.

01:17:51.460 | But also, of course, at the same time,

01:17:54.660 | we want to ensure that however that data is being handled,

01:17:57.460 | it's done in a privacy-preserving way,

01:18:00.300 | so that, for example, the recommendation system

01:18:03.220 | doesn't just go around and sell your data,

01:18:06.220 | and then cause a lot of bad consequences and so on.

01:18:11.220 | So-

01:18:13.220 | - So you want that dialogue to be

01:18:15.060 | a little bit more in the open,

01:18:16.580 | a little bit more nuanced,

01:18:18.860 | and maybe adding control to the data,

01:18:21.380 | ownership to the data will allow,

01:18:23.380 | so as opposed to this happening in the background,

01:18:26.860 | allow it to bring it to the forefront,

01:18:28.660 | and actually have dialogues,

01:18:30.660 | and more nuanced, real dialogues

01:18:33.900 | about how we trade our data for the services.

01:18:37.860 | That's the whole-

01:18:38.700 | - Right, right.

01:18:40.260 | Yes, at high level.

01:18:41.580 | So essentially, also knowing that

01:18:43.140 | there are technical challenges

01:18:44.980 | in addressing the issue,

01:18:50.220 | to basically, you can't have,

01:18:52.660 | just like the example that I gave earlier,

01:18:55.740 | it's really difficult to balance the two,

01:18:58.100 | between utility and privacy.

01:19:00.260 | And that's also a lot of things

01:19:02.540 | that I work on, my group works on as well,

01:19:06.420 | is to actually develop these technologies

01:19:10.380 | that are needed to essentially help this balance better,

01:19:14.820 | and essentially to help data to be utilized

01:19:17.140 | in a privacy-preserving and responsible way.

01:19:20.020 | And so we essentially need people

01:19:21.580 | to understand the challenges,

01:19:23.540 | and also at the same time,

01:19:25.900 | to provide the technical abilities,

01:19:29.580 | and also regulatory frameworks,

01:19:31.460 | to help the two sides to be more

01:19:33.580 | in a win-win situation instead of a fight.

01:19:36.460 | - Yeah, the fighting thing is,

01:19:40.500 | I think YouTube and Twitter and Facebook

01:19:42.180 | are providing an incredible service to the world.

01:19:44.940 | And they're all making mistakes, of course,

01:19:47.460 | but they're doing an incredible job,

01:19:49.580 | that I think deserves to be applauded,

01:19:53.500 | and there's some degree of gratitude.

01:19:56.180 | It's a cool thing that's created.

01:19:59.260 | And it shouldn't be monolithically fought against,

01:20:04.260 | like Facebook is evil or so on.

01:20:06.500 | Yeah, I might make mistakes,

01:20:07.940 | but I think it's an incredible service.

01:20:10.060 | I think it's world-changing.

01:20:11.860 | I think Facebook's done a lot of incredible things

01:20:17.460 | by bringing, for example, identity,

01:20:20.180 | allowing people to be themselves,

01:20:25.220 | like their real selves in the digital space,

01:20:28.660 | by using their real name and their real picture.

01:20:31.620 | That step was like the first step

01:20:33.340 | from the real world to the digital world.

01:20:35.660 | That was a huge step that perhaps

01:20:37.500 | will define the 21st century,

01:20:39.420 | in us creating a digital identity.

01:20:41.540 | And there's a lot of interesting possibilities there

01:20:44.140 | that are positive.

01:20:45.220 | Of course, some things are negative,

01:20:47.860 | and having a good dialogue about that is great.

01:20:50.060 | And I'm great that people like you

01:20:51.660 | are at the center of that dialogue,

01:20:52.860 | 'cause that's awesome.

01:20:54.140 | - Right, I think also,

01:20:56.180 | I also can understand that.

01:20:58.500 | I think actually in the past,

01:21:00.780 | especially in the past couple of years,

01:21:02.700 | this rising awareness has been helpful.

01:21:07.540 | Like users are also more and more recognizing

01:21:10.220 | that privacy is important to them.

01:21:12.060 | They should, maybe, right,

01:21:14.460 | they should be owners of their data.

01:21:15.860 | I think this definitely is very helpful.

01:21:18.260 | And I think also this type of voice,

01:21:22.820 | also, and together with the regulatory framework and so on,

01:21:27.260 | also help the companies to essentially

01:21:30.820 | put these type of issues at a higher priority.

01:21:33.940 | And knowing that, right,

01:21:36.300 | also it is their responsibility too,

01:21:39.420 | to ensure that users are well protected.

01:21:43.340 | So I think definitely the rising voice is super helpful.

01:21:47.740 | And I think that actually really has brought

01:21:50.900 | the issue of data privacy,

01:21:53.140 | and even this consideration of data ownership

01:21:56.260 | to the forefront, to really much wider community.

01:22:00.100 | And I think more of this voice is needed,

01:22:03.660 | but I think it's just that we want to have

01:22:05.620 | a more constructive dialogue

01:22:07.500 | to bring the both sides together

01:22:10.420 | to figure out a constructive solution.

01:22:12.700 | - So another interesting space

01:22:15.580 | where security is really important

01:22:17.020 | is in the space of any kinds of transactions,

01:22:21.220 | but it could be also digital currency.

01:22:23.340 | So can you maybe talk a little bit about blockchain?

01:22:28.300 | And can you tell me what is a blockchain?

01:22:30.540 | - I think the blockchain word itself

01:22:34.940 | is actually very overloaded.

01:22:37.580 | - Of course. - In general--

01:22:39.020 | - It's like AI.

01:22:40.020 | - Right, yes.

01:22:42.020 | So in general, when I talk about blockchain,

01:22:43.340 | we refer to this distributed ledger

01:22:46.060 | in a decentralized fashion.

01:22:47.780 | So essentially you have a community of nodes

01:22:53.500 | that come together,

01:22:54.860 | and even though each one may not be trusted,

01:22:59.180 | and as long as certain thresholds

01:23:02.620 | of the set of nodes behaves properly,

01:23:07.580 | then the system can essentially achieve certain properties.

01:23:11.820 | For example, in the distributed ledger setting,

01:23:15.580 | you can maintain a mutable log,

01:23:18.580 | and you can ensure that, for example,

01:23:22.940 | the transactions actually are agreed upon,

01:23:25.540 | and then it's immutable and so on.

01:23:28.260 | - So first of all, what's a ledger?

01:23:29.740 | So it's a--

01:23:30.740 | - It's like a database.

01:23:31.740 | It's like a data entry.

01:23:33.620 | - And so distributed ledger is something

01:23:35.540 | that's maintained across,

01:23:37.900 | or is synchronized across multiple sources, multiple nodes.

01:23:41.700 | - Multiple nodes, yes.

01:23:43.340 | - And so where is this idea,

01:23:46.060 | now how do you keep,

01:23:48.420 | so it's important, a ledger, a database,

01:23:51.420 | to keep that, to make sure,

01:23:55.580 | so what are the kinds of security vulnerabilities

01:23:58.740 | that you're trying to protect against

01:24:01.500 | in the context of a distributed ledger?

01:24:04.420 | - So in this case, for example,

01:24:06.300 | you don't want some malicious nodes

01:24:09.060 | to be able to change the transaction logs,

01:24:12.860 | and in certain cases, it's called double spending,

01:24:15.660 | like you can also cause different views

01:24:19.780 | in different parts of the network and so on.

01:24:22.780 | - So the ledger has to represent,

01:24:24.460 | if you're capturing financial transactions,

01:24:27.540 | it has to represent the exact timing

01:24:29.420 | and the exact occurrence and no duplicates,

01:24:32.380 | all that kind of stuff,

01:24:33.340 | it has to represent what actually happened.

01:24:37.060 | Okay, so what are your thoughts

01:24:40.540 | on the security and privacy of digital currency?

01:24:43.780 | I can't tell you how many people write to me

01:24:47.340 | to interview various people in the digital currency space.

01:24:51.660 | There seems to be a lot of excitement there,

01:24:54.940 | and it seems to be, some of it, to me,

01:24:57.980 | from an outsider's perspective, seems like dark magic.

01:25:01.860 | I don't know how secure,

01:25:05.060 | I think the foundation, from my perspective,

01:25:08.900 | of digital currencies, that is,

01:25:10.700 | you can't trust anyone,

01:25:13.440 | so you have to create a really secure system.

01:25:16.340 | So can you maybe speak about how,

01:25:19.860 | what your thoughts in general about digital currency is

01:25:22.060 | and how it can possibly create financial transactions

01:25:26.940 | and financial stores of money in the digital space?

01:25:31.700 | - So you asked about security and privacy.

01:25:34.580 | So again, as I mentioned earlier,

01:25:37.580 | in security, we actually talk about two main properties.

01:25:42.020 | The integrity and confidentiality.

01:25:45.800 | And so there's another one for availability.

01:25:49.040 | You want the system to be available,

01:25:50.660 | but here, for the question you asked,

01:25:52.760 | let's just focus on integrity and confidentiality.

01:25:57.080 | So for integrity of this distributed ledger,

01:26:00.520 | essentially, as we discussed,

01:26:02.000 | we want to ensure that the different nodes,

01:26:04.940 | so they have this consistent view,

01:26:08.600 | usually it's done through, we call it a consensus protocol,

01:26:13.600 | that they establish this shared view on this ledger,

01:26:18.180 | and that you cannot go back and change,

01:26:21.940 | it's immutable, and so on.

01:26:24.380 | So in this case, then the security often refers

01:26:28.740 | to this integrity property,

01:26:31.860 | and essentially you're asking the question,

01:26:34.700 | how much work, how can you attack the system

01:26:38.920 | so that the attacker can change the lock, for example?

01:26:43.920 | - Right, how hard is it to make an attack like that?

01:26:48.520 | - Right, right.

01:26:49.480 | And then that very much depends on the consensus mechanism,

01:26:54.480 | how the system is built, and all that.

01:26:57.560 | So there are different ways

01:26:59.120 | to build these decentralized systems.

01:27:03.440 | People may have heard about the terms called

01:27:05.980 | proof of work, proof of stake,

01:27:07.860 | these different mechanisms,

01:27:09.700 | and it really depends on how the system has been built,

01:27:14.460 | and also how much resources,

01:27:17.820 | how much work has gone into the network

01:27:20.540 | to actually say how secure it is.

01:27:24.500 | So for example, people talk about,

01:27:26.460 | like in Bitcoin, it's proof of work system,

01:27:28.900 | so much electricity has been burned.

01:27:32.100 | So there's differences in the different mechanisms

01:27:35.320 | and the implementations of a distributed ledger

01:27:37.960 | used for digital currency.

01:27:40.080 | So there's Bitcoin, there's whatever,

01:27:42.360 | there's so many of them,

01:27:43.280 | and there's underlying different mechanisms.

01:27:46.040 | And there's arguments, I suppose,

01:27:48.400 | about which is more effective, which is more secure,

01:27:51.640 | which is more--

01:27:52.920 | - And what is needed,

01:27:54.120 | what amount of resources needed to be able

01:27:58.320 | to attack the system?

01:28:00.460 | For example, what percentage of the nodes

01:28:02.820 | do you need to control or compromise

01:28:06.220 | in order to change the log?

01:28:09.980 | - And those are things,

01:28:11.300 | do you have a sense of those are things

01:28:13.500 | that can be shown theoretically

01:28:15.460 | through the design of the mechanisms,

01:28:17.620 | or does it have to be shown empirically

01:28:19.260 | by having a large number of users using the currency?

01:28:23.580 | - I see.

01:28:24.420 | So in general, for each consensus mechanism,

01:28:27.020 | you can actually show theoretically

01:28:30.180 | what is needed to be able to attack the system.

01:28:34.420 | Of course, there can be different types of attacks,

01:28:37.900 | as we discussed at the beginning.

01:28:41.140 | And so that it's difficult to give

01:28:45.580 | like a complete estimate,

01:28:50.100 | like really how much is needed to compromise the system.

01:28:55.100 | But in general, right, so there are ways to say

01:28:57.660 | what percentage of the nodes you need to compromise

01:29:00.580 | and so on.

01:29:03.100 | - So we talked about integrity,

01:29:05.260 | so on the security side,

01:29:07.540 | and then you also mentioned the privacy

01:29:11.140 | or the confidentiality side.

01:29:13.460 | Does it have some of the same problems

01:29:17.740 | and therefore some of the same solutions

01:29:19.420 | that you talked about on the machine learning side

01:29:21.500 | with differential privacy and so on?

01:29:24.220 | - Yeah, so actually in general,

01:29:26.260 | on the public ledger,

01:29:29.260 | in this public decentralized systems,

01:29:32.500 | actually nothing is private.

01:29:35.020 | So all the transactions,

01:29:36.900 | pulses on the ledger, anybody can see.

01:29:40.100 | So in that sense, there's no confidentiality.

01:29:42.740 | So usually what you can do is then

01:29:47.100 | there are the mechanisms that you can build in

01:29:50.820 | to enable confidentiality,

01:29:53.620 | privacy of the transactions and the data and so on.

01:29:56.460 | That's also some of the work

01:29:57.940 | that both my group and also my startup does as well.

01:30:03.780 | - What's the name of the startup?

01:30:05.700 | - Oasis Labs.

01:30:06.740 | - Oasis Labs.

01:30:07.820 | And so the confidentiality aspect there

01:30:12.100 | is even though the transactions are public,

01:30:15.500 | you wanna keep some aspect confidential

01:30:18.380 | of the identity of the people involved in the transactions.

01:30:21.140 | So what is their hope to keep confidential in this context?

01:30:25.300 | - So in this case, for example,

01:30:26.780 | you want to enable like confidential transactions,

01:30:31.660 | even, so there are different essentially types of data

01:30:36.660 | that you want to keep private or confidential.

01:30:40.940 | And you can utilize different technologies,

01:30:43.260 | including zero knowledge proofs

01:30:46.020 | and also secure computing techniques

01:30:50.100 | to hide either who is making the transactions to whom

01:30:56.020 | and the transaction amounts.

01:30:58.460 | And in our case also,

01:30:59.740 | we can enable like confidential smart contracts

01:31:03.500 | so that you don't know the data

01:31:06.180 | and the execution of the smart contract and so on.

01:31:09.620 | And we actually are combining these different technologies

01:31:14.340 | and to going back to the earlier discussion we had

01:31:18.620 | enabling like ownership of data

01:31:23.500 | and privacy of data and so on.

01:31:26.300 | So at Oasis Labs, we're actually building

01:31:29.780 | what we call a platform for responsible data economy

01:31:33.380 | to actually combine these different technologies together

01:31:36.220 | to enable secure and privacy preserving computation

01:31:42.260 | and also using the ledger to help provide

01:31:47.260 | immutable log of users' ownership to their data

01:31:51.980 | and the policies they want the data to adhere to,

01:31:55.540 | the usage of the data to adhere to

01:31:57.340 | and also how the data has been utilized.

01:32:00.380 | So all this together can build,

01:32:03.220 | we call it distributors secure computing fabric

01:32:06.900 | that helps to enable a more responsible data economy.

01:32:11.340 | So it's a lot of things together.

01:32:12.380 | - Yeah, wow, that was eloquent.

01:32:14.620 | Okay, you're involved in so much amazing work

01:32:18.140 | that we'll never be able to get to,

01:32:19.500 | but I have to ask at least briefly about program synthesis,

01:32:23.940 | which at least in a philosophical sense

01:32:26.780 | captures much of the dreams of what's possible

01:32:30.300 | in computer science and the artificial intelligence.

01:32:33.420 | First, let me ask, what is program synthesis

01:32:37.660 | and can neural networks be used to learn programs from data?

01:32:41.740 | So can this be learned?

01:32:43.620 | Some aspect of the synthesis can it be learned?

01:32:46.260 | - So program synthesis is about teaching computers

01:32:50.220 | to write code, to program.

01:32:53.420 | And I think that's one of our ultimate dreams or goals.

01:32:58.300 | I think Andreessen talked about software eating the world.

01:33:05.860 | So I say, once we teach computers to write software,

01:33:10.860 | to write programs, then I guess computers

01:33:13.940 | will be eating the world by transitivity.

01:33:16.620 | - Yeah, exactly.

01:33:18.220 | - So yeah, and also for me, actually,

01:33:21.660 | when I shifted from security to more AI machine learning,

01:33:28.940 | program synthesis is,

01:33:30.940 | program synthesis and adversarial machine learning,

01:33:34.180 | these are the two fields that I particularly focus on.

01:33:38.100 | Like program synthesis is one of the first questions

01:33:40.300 | that I actually started investigating.

01:33:42.740 | - Just as a question, oh, I guess from the security side,

01:33:46.060 | you're looking for holes in programs,

01:33:49.340 | so at least see small connection,

01:33:51.340 | but where was your interest for program synthesis?

01:33:55.380 | Because it's such a fascinating, such a big,

01:33:58.380 | such a hard problem in the general case.

01:34:01.020 | Why program synthesis?

01:34:03.100 | - So the reason for that is,

01:34:04.780 | actually when I shifted my focus from security

01:34:08.900 | into AI machine learning,

01:34:11.620 | actually one of my main motivation at the time

01:34:15.300 | is that even though I have been doing a lot of work

01:34:19.020 | in security and privacy,

01:34:20.020 | but I have always been fascinated

01:34:22.620 | about building intelligent machines.

01:34:26.540 | And that was really my main motivation

01:34:30.100 | to spend more time in AI machine learning

01:34:32.180 | is that I really want to figure out

01:34:35.140 | how we can build intelligent machines.

01:34:37.900 | And to help us towards that goal,

01:34:42.900 | program synthesis is really one of,

01:34:45.500 | I would say the best domain to work on.

01:34:49.420 | I actually call it like a program synthesis

01:34:52.300 | is like the perfect playground

01:34:55.020 | for building intelligent machines

01:34:57.500 | and for artificial general intelligence.

01:34:59.900 | - Yeah.

01:35:01.260 | Well, it's also in that sense,

01:35:03.260 | not just a playground,

01:35:04.100 | I guess it's the ultimate test of intelligence

01:35:06.780 | because I think if you can generate

01:35:11.620 | sort of neural networks can learn good functions

01:35:15.660 | and they can help you out in classification tasks,

01:35:19.020 | but to be able to write programs,

01:35:21.660 | that's the epitome from the machine side.

01:35:24.780 | That's the same as passing the Turing test

01:35:26.620 | in natural language, but with programs,

01:35:29.220 | it's able to express complicated ideas,

01:35:31.980 | to reason through ideas and boil them down to algorithms.

01:35:36.980 | - Yes, exactly, exactly.

01:35:39.300 | - Incredible.

01:35:40.140 | So can this be learned?

01:35:41.660 | How far are we?

01:35:43.420 | Is there hope?

01:35:44.700 | What are the open challenges?

01:35:46.620 | - Yeah, very good questions.

01:35:48.180 | We are still at an early stage,

01:35:51.140 | but already I think we have seen a lot of progress.

01:35:56.220 | I mean, definitely we have existence proof,

01:35:59.860 | just like humans can write programs.

01:36:01.980 | So there's no reason why computers cannot write programs.

01:36:05.660 | So I think that's definitely an achievable goal,

01:36:08.700 | it's just how long it takes.

01:36:11.300 | And then, and even today,

01:36:14.780 | we actually have the program synthesis community,

01:36:19.620 | especially the program synthesis via learning,

01:36:22.660 | how we call it, neural program synthesis community,

01:36:24.740 | is still very small,

01:36:26.180 | but the community has been growing

01:36:28.460 | and we have seen a lot of progress.

01:36:31.740 | And in limited domains,

01:36:33.620 | I think actually program synthesis

01:36:37.260 | is ripe for real world applications.

01:36:40.420 | So actually it was quite amazing.

01:36:42.580 | I was giving a talk,

01:36:46.340 | so here is a Rework conference.

01:36:49.140 | - Yeah, Rework Deep Learning Summit.

01:36:50.340 | - I actually, so I gave another talk

01:36:52.340 | at the previous Rework conference

01:36:54.900 | in deep reinforcement learning.

01:36:56.980 | And then I actually met someone from a startup,

01:37:02.020 | the CEO of the startup.

01:37:04.580 | And when he saw my name, he recognized it.

01:37:06.540 | And he actually said,

01:37:07.980 | one of our papers actually had,

01:37:12.780 | they had put,

01:37:14.460 | had actually become a key products in their startup.

01:37:20.220 | And that was program synthesis in that particular case

01:37:23.620 | is was natural language translation,

01:37:26.140 | translating natural language description

01:37:30.140 | into SQL queries.

01:37:31.460 | - Oh, wow, that direction.

01:37:34.620 | Okay.

01:37:35.460 | - Right.

01:37:36.300 | Right, so yeah, so in program synthesis,

01:37:39.020 | in limited domains,

01:37:40.620 | in well-specified domains,

01:37:42.140 | actually already we can see

01:37:45.940 | really great progress and applicability

01:37:50.900 | in the real world.

01:37:52.140 | - So domains like,

01:37:53.660 | I mean, as an example,

01:37:54.700 | you said natural language being able to express something

01:37:57.300 | through just normal language

01:37:59.260 | and it converts it into a database SQL SQL query.

01:38:03.140 | - Right.

01:38:03.980 | - And that's how,

01:38:04.860 | how solved of a problem is that?

01:38:07.700 | 'Cause that seems like a really hard problem.

01:38:10.380 | - Again, in limited domains,

01:38:11.620 | actually it can work pretty well.

01:38:14.940 | And now this is also a very active domain of research.

01:38:18.820 | At the time,

01:38:19.860 | I think when he saw our paper at the time,

01:38:21.460 | we were the state of the arts.

01:38:23.740 | - Yeah.

01:38:24.980 | - On that task.

01:38:25.820 | And since then, actually now there has been more work

01:38:29.100 | and with even more like sophisticated datasets.

01:38:33.980 | So, but I think I wouldn't be surprised

01:38:38.820 | that more of this type of technology

01:38:41.020 | really gets into the real world.

01:38:43.220 | - That's exciting.

01:38:44.300 | - In the near term.

01:38:45.220 | - Being able to learn in the space of programs

01:38:47.740 | is super exciting.

01:38:49.820 | I still,

01:38:50.660 | yeah, I'm still skeptical

01:38:53.100 | 'cause I think it's a really hard problem.

01:38:54.860 | - It is, it is.

01:38:55.700 | - But I would love to see progress.

01:38:56.620 | - And also I think in terms of the,

01:38:58.460 | you asked about open challenges.

01:39:00.180 | - Yes, open challenges.

01:39:01.020 | - I think the domain is full of challenges.

01:39:04.260 | And in particular,

01:39:05.740 | also we want to see how we should measure the progress

01:39:08.420 | in the space.

01:39:09.900 | And I would say mainly

01:39:13.500 | three main, I'll say metrics.

01:39:16.780 | So one is the complexity of the program

01:39:18.700 | that we can synthesize.

01:39:20.060 | And that will actually have clear measures

01:39:22.780 | and just look at the past publications.

01:39:25.900 | And even like, for example,

01:39:27.380 | I was at the recent NeurIPS conference.

01:39:30.340 | Now there's actually a fairly sizable session

01:39:33.860 | dedicated to program synthesis, which is--

01:39:35.980 | - Or even neural programs.

01:39:37.420 | - Right, right, right, which is great.

01:39:39.020 | And we continue to see the increase

01:39:43.340 | in-- - What does sizable mean?

01:39:44.620 | (laughing)

01:39:46.100 | I like the word sizable.

01:39:48.140 | It's five people.

01:39:50.740 | (laughing)

01:39:51.580 | - It's still a small community, but it is growing.

01:39:54.460 | - And they will all win Turing Awards one day.

01:39:56.860 | (laughing)

01:39:57.700 | I like it.

01:39:58.660 | - Right, so we can clearly see increase

01:40:02.820 | in the complexity of the programs that these--

01:40:07.380 | - Just to elaborate--

01:40:08.220 | - We can synthesize.

01:40:09.100 | - Sorry, to, is it the complexity

01:40:11.420 | of the actual text of the program

01:40:13.140 | or the running time complexity?

01:40:15.380 | Which complexity are we, how--

01:40:18.100 | - The complexity of the task to be synthesized

01:40:21.700 | and the complexity of the actual synthesized programs.

01:40:24.580 | So it is, right.

01:40:25.940 | So the lines of code even, for example.

01:40:27.900 | - Okay, I got you.

01:40:28.740 | But it's not the theoretical upper bound

01:40:32.940 | of the running time of the algorithm kind of thing.

01:40:34.380 | - Right, right, right, right.

01:40:35.380 | - Okay, got it.

01:40:36.660 | And you can see the complexity decreasing already.

01:40:39.940 | - Oh, no, meaning we want to be able to synthesize

01:40:42.100 | more and more complex programs, bigger and bigger programs.

01:40:44.900 | So we want to see that, we want to increase

01:40:49.300 | the complexity of the-- - I got you.

01:40:50.140 | So I have to think through,

01:40:51.420 | 'cause I thought of complexity as you wanna be able

01:40:53.860 | to accomplish the same task with a simpler

01:40:56.260 | and simpler program. - I see, I see.

01:40:57.460 | No, we are not doing that.

01:40:58.860 | It's more about how complex a task

01:41:02.460 | we can synthesize programs for.

01:41:03.300 | - Right, being able to, yeah, got it.

01:41:05.260 | Being able to synthesize programs,

01:41:08.020 | learn them for more and more difficult tasks.

01:41:10.060 | - Right, so for example, initially,

01:41:11.940 | our first work in program synthesis

01:41:14.980 | was to translate natural language description

01:41:17.460 | into really simple programs called if TTT,

01:41:19.940 | if this, then that.

01:41:21.380 | So given a trigger condition,

01:41:23.740 | what is the action you should take?

01:41:25.740 | So that program is super simple.

01:41:28.100 | You just identify the trigger conditions and the action.

01:41:31.580 | And then later on, with SQL queries,

01:41:33.300 | it gets more complex.

01:41:34.340 | And then also, we started to synthesize programs

01:41:37.780 | with loops and-- - Oh no.

01:41:40.340 | (Rachel laughing)

01:41:41.660 | And if you could synthesize recursion, it's all over.

01:41:45.580 | - Right, actually, one of our works, actually,

01:41:48.580 | is on learning recursive neural programs.

01:41:51.020 | - Oh no. - But anyway, anyway.

01:41:52.020 | So that's one is the complexity,

01:41:53.700 | and the other one is generalization.

01:41:58.340 | Like when we train or learn a program synthesizer,

01:42:04.380 | in this case, a neural program to synthesize programs,

01:42:07.740 | then you want it to generalize.

01:42:10.500 | - For a large number of inputs.

01:42:13.140 | - Right, so to be able to generalize

01:42:15.540 | to previously unseen inputs.

01:42:18.220 | - Got it.

01:42:19.060 | - And so some of the work we did earlier,

01:42:22.540 | learning recursive neural programs,

01:42:26.180 | actually showed that recursion actually is important

01:42:31.620 | to learn, and if you have recursion,

01:42:34.740 | then for a certain set of tasks,

01:42:37.740 | we can actually show that you can actually

01:42:39.380 | have perfect generalization.

01:42:41.060 | So that won the best paper awards at ICLR earlier.

01:42:46.500 | So that's one example of we want to learn

01:42:50.740 | these neural programs that can generalize better.

01:42:53.580 | But that works for certain tasks, certain domains,

01:42:57.220 | and there's question how we can

01:43:00.820 | essentially develop more techniques

01:43:04.060 | that can have generalization for wider set of domains,

01:43:09.060 | and so on.

01:43:10.420 | So that's another area.

01:43:11.420 | And then the third challenge I think will,

01:43:15.900 | it's not just for program synthesis,

01:43:17.460 | this is also cutting across other fields

01:43:20.660 | in machine learning, and also including

01:43:24.460 | deep reinforcement learning in particular,

01:43:26.380 | is that this adaptation is that we want to be able

01:43:31.380 | to learn from the past and tasks and training

01:43:38.420 | and so on to be able to solve new task.

01:43:42.420 | So for example, in program synthesis today,

01:43:44.660 | we still are working in the setting

01:43:48.020 | where given a particular task,

01:43:50.460 | we train the model and to solve this particular task.

01:43:56.460 | But that's not how humans work.

01:43:58.820 | The whole point is we train a human

01:44:02.140 | and you can then program to solve new tasks.

01:44:06.500 | - Right, exactly.

01:44:07.620 | - And just like in deep reinforcement learning,

01:44:09.420 | we don't want to just train agent

01:44:10.700 | to play a particular game,

01:44:12.820 | either it's Atari or it's Go or whatever.

01:44:18.020 | We want to train these agents that can

01:44:21.380 | essentially extract knowledge from the past

01:44:24.980 | learning experience to be able to adapt

01:44:29.340 | to new tasks and solve new tasks.

01:44:31.540 | And I think this is particularly important

01:44:33.620 | for program synthesis.

01:44:34.820 | - Yeah, that's the whole point.

01:44:35.860 | That's the whole dream of program synthesis

01:44:37.620 | is you're learning a tool that can solve new problems.

01:44:41.460 | - Right, exactly.

01:44:42.620 | And I think that's a particular domain

01:44:44.980 | that as a community, we need to put more emphasis on

01:44:49.980 | and I hope that we can make more progress there as well.

01:44:54.380 | - Awesome.

01:44:55.900 | There's a lot more to talk about,

01:44:57.100 | but let me ask that you also had a very interesting

01:45:01.540 | and we talked about rich representations.

01:45:05.020 | You had a rich life journey.

01:45:06.800 | You did your bachelor's in China

01:45:10.140 | and your master's and PhD in the United States,

01:45:12.980 | CMU and Berkeley.

01:45:14.260 | Are there interesting differences?

01:45:16.860 | I told you I'm Russian.

01:45:17.860 | I think there's a lot of interesting difference

01:45:19.300 | between Russia and the United States.

01:45:21.220 | Are there in your eyes interesting differences

01:45:24.860 | between the two cultures from the,

01:45:29.100 | so the romantic notion of the spirit of the people

01:45:32.340 | to the more practical notion of how research is conducted

01:45:35.860 | that you find interesting or useful in your own work

01:45:39.860 | of having experienced both?

01:45:42.180 | - That's a good question.

01:45:43.780 | I think, so I studied in China for my undergraduate

01:45:50.180 | and that was more than 20 years ago.

01:45:54.660 | So it's been a long time.

01:45:56.220 | - Is there echoes of that time in you?

01:45:58.140 | - So things have changed a lot.

01:46:00.580 | Actually, it's interesting.

01:46:01.660 | I think even more so maybe something

01:46:04.260 | that's even be more different for my experience

01:46:08.940 | than a lot of computer science researchers

01:46:12.380 | and practitioners is that,

01:46:14.180 | so for my undergrad, I actually studied physics.

01:46:16.860 | - Nice, very nice.

01:46:18.060 | - And then I switched to computer science in graduate school.

01:46:21.060 | - What happened?

01:46:22.940 | (both laughing)

01:46:25.100 | Was there, is there another possible universe

01:46:29.420 | where you could have become a theoretical physicist

01:46:32.180 | at Caltech or something like that?

01:46:34.580 | - That's very possible.

01:46:35.820 | Some of my undergrad classmates,

01:46:39.420 | then they later on studied physics,

01:46:41.700 | got their PhD in physics from these schools,

01:46:45.580 | from, yeah, from top physics programs.

01:46:49.140 | - So you switched to, I mean, is that,

01:46:52.660 | from that experience of doing physics in your bachelor's,

01:46:55.980 | how, what made you decide to switch to computer science

01:46:59.260 | and computer science at arguably the best university,

01:47:03.700 | one of the best universities in the world

01:47:05.020 | for computer science with Carnegie Mellon,

01:47:07.260 | especially for grad school and so on.

01:47:09.980 | So what, second only to MIT, just kidding.

01:47:13.060 | Okay.

01:47:13.900 | (both laughing)

01:47:16.260 | I had to throw that in there.

01:47:17.300 | No, what was the choice like

01:47:19.460 | and what was the move to the United States like?

01:47:22.620 | What was that whole transition?

01:47:24.100 | And if you remember, if there's still echoes

01:47:26.980 | of some of the spirit of the people of China in you

01:47:30.140 | in New York.

01:47:31.500 | - Right, right, yeah.

01:47:32.340 | - That's like three questions in one.

01:47:33.180 | - Yes, I know.

01:47:34.020 | (laughing)

01:47:34.840 | - I'm sorry.

01:47:35.680 | (both laughing)

01:47:36.620 | - No, that's okay.

01:47:37.520 | So yes, I guess, okay, so first transition

01:47:40.980 | from physics to computer science.

01:47:42.780 | - Yes.

01:47:43.620 | - When I first came to the United States,

01:47:45.340 | I was actually in the physics PhD program at Cornell.

01:47:48.860 | - Yeah.

01:47:49.700 | - I was there for one year

01:47:50.520 | and then I switched to computer science

01:47:52.020 | and then I was in the PhD program at Carnegie Mellon.

01:47:55.120 | So, okay, so the reasons for switching.

01:47:59.100 | So one thing, so that's why I also mentioned

01:48:01.460 | that about this difference in backgrounds

01:48:04.220 | about having studied physics first in my undergrad.

01:48:09.220 | I actually really, I really did enjoy my undergrad's time

01:48:14.220 | and education in physics.

01:48:18.780 | I think that actually really helped me

01:48:21.060 | in my future work in computer science.

01:48:25.020 | Actually, even for machine learning,

01:48:26.380 | a lot of machine learning stuff,

01:48:28.060 | the core machine learning methods,

01:48:29.720 | many of them actually came from physics.

01:48:31.500 | (both laughing)

01:48:34.580 | - For honest, most--

01:48:36.180 | - But anyway.

01:48:37.420 | - Most of everything came from physics.

01:48:39.580 | - Right, but anyway, so when I studied physics,

01:48:42.740 | I was, I think I was really attracted to physics.

01:48:47.740 | It was, it's really beautiful.

01:48:51.340 | And I actually call it, physics is the language of nature.

01:48:55.840 | And I actually clearly remember like one moment

01:49:02.100 | in my undergrads, I did my undergrad in Tsinghua

01:49:07.260 | and I used to study in the library.

01:49:09.940 | And I clearly remember like one day,

01:49:14.660 | I was sitting in the library

01:49:16.260 | and I was like writing on my notes and so on.

01:49:21.260 | And I got so excited that I realized

01:49:24.780 | that really just from a few simple axioms,

01:49:28.340 | a few simple laws, I can derive so much.

01:49:31.780 | It's almost like I can derive the rest of the world.

01:49:34.340 | - Yeah, the rest of the universe.

01:49:35.980 | - Yes, yes, so that was like amazing.

01:49:39.260 | - Do you think you, have you ever seen

01:49:42.140 | or do you think you can rediscover that kind of power

01:49:44.500 | and beauty in computer science in the world that you--

01:49:46.900 | - So, that's very interesting.

01:49:49.420 | So that gets to the transition

01:49:51.460 | from physics to computer science.

01:49:53.220 | It's quite different.

01:49:56.420 | For physics in grad school, actually things changed.

01:50:02.460 | So one is I started to realize

01:50:05.820 | that when I started doing research in physics,

01:50:09.180 | at the time I was doing theoretical physics

01:50:11.860 | and a lot of it, you still have the beauty

01:50:15.340 | but it's very different.

01:50:16.700 | So I had to actually do a lot of the simulation.

01:50:18.980 | So essentially I was actually writing,

01:50:21.340 | in some cases writing fortune code.

01:50:24.060 | - That old fortune, yeah.

01:50:26.380 | - To actually write, do simulations,

01:50:32.620 | and so on, that was not,

01:50:34.580 | not exactly what-- - Not beautiful.

01:50:37.860 | - Right.

01:50:38.700 | I enjoyed doing.

01:50:41.780 | And also at the time from talking with the senior students

01:50:48.060 | in the program, I realized many of the students

01:50:54.140 | actually were going off to like Wall Street and so on.

01:50:59.460 | And I've always been interested in computer science

01:51:02.380 | and actually essentially taught myself C programming.

01:51:07.380 | - Program. - Right, and so on.

01:51:09.900 | - When?

01:51:10.740 | - In college. - In college somewhere?

01:51:12.860 | - In the summer.

01:51:13.700 | - For fun.

01:51:16.100 | Physics major, learning to do C programming, beautiful.

01:51:19.500 | - Actually it's interesting, in physics at the time,

01:51:23.580 | I think now the program probably has changed,

01:51:25.860 | but at the time really the only class we had

01:51:30.140 | in related to computer science education

01:51:34.180 | was introduction to, I forgot,

01:51:36.860 | to computer science or computing and Fortran 77.

01:51:40.140 | - There's a lot of people that still use Fortran.

01:51:42.540 | I'm actually, if you're a programmer out there,

01:51:46.340 | I'm looking for an expert to talk to about Fortran.

01:51:49.780 | They seem to, there's not many,

01:51:51.820 | but there's still a lot of people that still use Fortran

01:51:53.980 | and still a lot of people use Cobalt.

01:51:55.860 | - But anyway, so then I realized,

01:52:00.260 | instead of just doing programming

01:52:01.940 | for doing simulations and so on,

01:52:04.220 | that I may as well just change to computer science.

01:52:07.140 | And also one thing I really liked,

01:52:09.180 | and that's a key difference between the two,

01:52:11.340 | is in computer science it's so much easier

01:52:14.340 | to realize your ideas.

01:52:16.060 | If you have an idea, you write it up,

01:52:18.340 | you code it up and then you can see it actually, right?

01:52:23.540 | - Exactly. - Running and you can see it.

01:52:26.220 | - You can bring it to life quickly.

01:52:27.060 | - Bring it to life.

01:52:28.060 | Whereas in physics, if you have a good theory,

01:52:30.660 | you have to wait for the experimentalist

01:52:33.260 | to do the experiments and to confirm the theory

01:52:35.500 | and things just take so much longer.

01:52:38.180 | And also the reason in physics I decided to do

01:52:42.460 | theoretical physics was because I had my experience

01:52:45.820 | with experimental physics.

01:52:47.940 | First you have to fix the equipment.

01:52:50.340 | - Yeah, yeah.

01:52:51.180 | - And then most of the time fixing the equipment first.

01:52:54.260 | So--

01:52:56.260 | - Super expensive equipment, so there's a lot of,

01:52:58.260 | yeah, you have to collaborate with a lot of people,

01:53:00.900 | takes a long time.

01:53:01.740 | - Just takes really, right, much longer.

01:53:03.660 | - Yeah, it's messy.

01:53:04.500 | - Right, so I decided to switch to computer science.

01:53:06.700 | And one thing I think maybe people have realized

01:53:09.700 | is that for people who study physics,

01:53:11.260 | actually it's very easy for physicists to change,

01:53:15.260 | to do something else.

01:53:16.940 | I think physics provides a really good training.

01:53:19.740 | And yeah, so actually it was fairly easy

01:53:23.300 | to switch to computer science.

01:53:25.820 | But one thing going back to your earlier question,

01:53:29.940 | so one thing I actually did realize,

01:53:31.740 | so there is a big difference

01:53:34.180 | between computer science and physics,

01:53:35.580 | where physics you can derive the whole universe

01:53:39.260 | from just a few simple laws.

01:53:41.500 | And computer science, given that a lot of it

01:53:43.940 | is defined by humans, the systems are defined by humans

01:53:47.420 | and it's artificial, like essentially you create

01:53:52.420 | a lot of these artifacts and so on.

01:53:55.500 | It's not quite the same.

01:53:58.700 | You don't derive the computer systems

01:54:01.020 | with just a few simple laws.

01:54:03.540 | You actually have to see there is historical reasons

01:54:07.740 | why a system is built and designed one way

01:54:10.460 | versus the other.

01:54:11.860 | - There's a lot more complexity,

01:54:14.700 | less elegant simplicity of E equals MC squared

01:54:18.860 | that kind of reduces everything down

01:54:20.420 | to this beautiful fundamental equations.

01:54:23.340 | But what about the move from China into the United States?

01:54:27.660 | Is there anything that still stays in you

01:54:31.220 | that contributes to your work?

01:54:33.820 | The fact that you grew up in another culture?

01:54:36.900 | - So yes, I think especially back then

01:54:38.900 | it's very different from now.

01:54:40.740 | So now they actually, I see these students coming from China

01:54:45.740 | and even undergrads, actually they speak fluent English.

01:54:51.540 | It was just amazing.

01:54:55.020 | And they have already understood so much of the culture

01:54:58.500 | in the US and so on.

01:55:00.700 | - It was to you it was all foreign?

01:55:04.300 | It was-- - It was a very different time.

01:55:06.700 | At the time actually, we didn't even have easy access

01:55:11.700 | to email, not to mention about the web.

01:55:15.420 | I remember I had to go to specific,

01:55:19.180 | privileged survey rooms to use email.

01:55:24.020 | And hence, at the time we had much less knowledge

01:55:30.700 | about the Western world.

01:55:32.980 | And actually at the time I didn't know

01:55:35.100 | actually in the US, the West Coast weather

01:55:38.180 | is much better than the East Coast.

01:55:39.980 | (laughing)

01:55:42.860 | Yeah, things like that actually.

01:55:45.100 | It's very interesting.

01:55:46.780 | (laughing)

01:55:48.740 | But now it's so different.

01:55:50.380 | At the time I would say there's also

01:55:52.060 | a bigger cultural difference

01:55:53.580 | because there was so much less opportunity

01:55:58.060 | for shared information.

01:55:59.300 | So it's such a different time and world.

01:56:02.380 | - So let me ask maybe a sensitive question.

01:56:04.540 | I'm not sure, but I think you and I

01:56:07.100 | are in similar positions as I've been here

01:56:09.260 | for already 20 years as well.

01:56:13.140 | And looking at Russia from my perspective

01:56:15.420 | and you looking at China, in some ways

01:56:17.820 | it's a very distant place because it's changed a lot.

01:56:21.020 | But in some ways you still have echoes,

01:56:23.020 | you still have knowledge of that place.

01:56:25.180 | The question is, China's doing a lot

01:56:27.500 | of incredible work in AI.

01:56:29.580 | Do you see, please tell me there's an optimistic picture

01:56:33.700 | you see where the United States and China

01:56:36.220 | can collaborate and sort of grow together

01:56:38.380 | in the development of AI towards,

01:56:41.420 | there's different values in terms of the role

01:56:43.780 | of government and so on, of ethical, transparent,

01:56:46.540 | secure systems.

01:56:49.340 | We see it differently in the United States

01:56:50.940 | a little bit than China,

01:56:51.980 | but we're still trying to work it out.

01:56:53.940 | Do you see the two countries being able

01:56:55.660 | to successfully collaborate and work in a healthy way

01:56:59.100 | without sort of fighting and making it

01:57:03.020 | an AI arms race kind of situation?

01:57:06.260 | - Yeah, I believe so.

01:57:08.260 | I think science has no border

01:57:10.860 | and the advancement of the technology helps everyone,

01:57:15.860 | helps the whole world.

01:57:18.060 | And so I certainly hope that the two countries

01:57:21.740 | will collaborate and I certainly believe so.

01:57:26.740 | - Do you have any reason to believe so

01:57:28.700 | except being an optimist?

01:57:31.260 | - So first, again, like I said, science has no borders.

01:57:35.060 | And especially--

01:57:36.500 | - Science doesn't know borders?

01:57:38.260 | - Right.

01:57:39.220 | - And you believe that, well,

01:57:41.380 | in the former Soviet Union during the Cold War.

01:57:44.860 | - So that's, yeah, so that's the other point

01:57:46.780 | I was going to mention is that,

01:57:48.460 | especially in academic research, everything is public.

01:57:52.460 | Like we write papers, we open source codes

01:57:55.540 | and all this is in the public domain.

01:57:59.100 | It doesn't matter whether the person is in the US,

01:58:01.380 | in China or some other parts of the world.

01:58:03.540 | They can go on archive

01:58:06.140 | and look at the latest research and results.

01:58:09.460 | - So that openness gives you hope.

01:58:11.500 | - Yes.

01:58:12.340 | - Me too.

01:58:13.180 | - And that's also how, as a world,

01:58:15.660 | we make progress the best.

01:58:17.260 | - So, apologize for the romanticized question,

01:58:22.180 | but looking back, what would you say

01:58:24.300 | was the most transformative moment in your life

01:58:28.580 | that maybe made you fall in love with computer science?

01:58:33.580 | You said physics, you remember there was a moment

01:58:35.940 | where you thought you could derive

01:58:37.220 | the entirety of the universe.

01:58:38.760 | Was there a moment that you really fell in love

01:58:41.940 | with the work you do now,

01:58:43.780 | from security to machine learning to program synthesis?

01:58:48.220 | - So maybe, as I mentioned, actually in college,

01:58:52.020 | I, one summer I just taught myself programming in C.

01:58:55.900 | - Yes.

01:58:56.740 | - You just read a book and then you--

01:59:00.220 | - Don't tell me you fell in love with computer science

01:59:02.220 | by programming in C.

01:59:03.660 | - Remember I mentioned one of the draws for me

01:59:06.340 | to computer science is how easy it is

01:59:08.660 | to realize your ideas.

01:59:10.780 | So once I read the book,

01:59:13.700 | started to tell myself how to program in C,

01:59:16.980 | immediately what did I do?

01:59:19.060 | Like I programmed two games.

01:59:21.340 | One is just simple, like it's a goal game,

01:59:25.340 | like it's a board, you can move the stones and so on.

01:59:28.300 | And the other one I actually programmed a game

01:59:30.460 | that's like a 3D Tetris.

01:59:32.980 | It turned out to be a super hard game to play.

01:59:35.380 | (laughing)

01:59:36.420 | Because instead of just the standard 2D Tetris,

01:59:38.820 | it's actually a 3D thing.

01:59:40.740 | But I realized, wow, I just had these ideas to try it out

01:59:45.140 | and then--

01:59:45.980 | - You can just do it.

01:59:46.820 | - Yeah, you can just do it.

01:59:48.500 | And so that's when I realized, wow, this is amazing.

01:59:53.300 | - Yeah, you can create yourself.

01:59:55.140 | - Yes, yes.

01:59:55.980 | - And ideas.

01:59:56.800 | - Exactly.

01:59:58.060 | - From nothing to something that's actually out

02:00:00.420 | in the real world.

02:00:01.260 | - Right, and you get it on hands.

02:00:03.860 | - Let me ask a silly question,

02:00:05.940 | or maybe the ultimate question.

02:00:07.900 | What is to you the meaning of life?

02:00:11.820 | What gives your life meaning, purpose, fulfillment,

02:00:16.540 | happiness, joy?

02:00:18.320 | - Okay, these are two different questions.

02:00:21.140 | - Very different, yeah.

02:00:22.540 | - It's interesting that you ask this question.

02:00:24.940 | Maybe this question is probably the question

02:00:28.060 | that has followed me and followed my life the most.

02:00:32.820 | - Have you discovered anything,

02:00:34.900 | any satisfactory answer for yourself?

02:00:37.080 | Is there something you've arrived at?

02:00:41.460 | There's a moment, I've talked to a few people

02:00:46.060 | who have faced, for example, a cancer diagnosis

02:00:48.780 | or faced their own mortality,

02:00:50.740 | and that seems to change their view of them.

02:00:53.420 | It seems to be a catalyst for them

02:00:56.620 | removing most of the crap,

02:00:58.460 | of seeing that most of what they've been doing

02:01:02.660 | is not that important,

02:01:04.180 | and really reducing it into saying,

02:01:06.780 | here's actually the few things that really give meaning.

02:01:11.780 | Mortality is a really powerful catalyst for that,

02:01:14.820 | it seems like.

02:01:15.780 | Facing mortality, whether it's your parents dying

02:01:17.900 | or somebody close to you dying,

02:01:19.460 | or facing your own death for whatever reason,

02:01:22.060 | or cancer and so on.

02:01:23.140 | - Right, so yeah, in my own case,

02:01:26.500 | I didn't need to face mortality to--

02:01:28.420 | (both laughing)

02:01:31.080 | To try to ask that question.

02:01:34.180 | - Yes.

02:01:35.020 | - And I think there are a couple things.

02:01:38.860 | So one is, who should be defining the meaning of your life?

02:01:44.620 | - Right, is there some kind of even greater things than you

02:01:49.020 | who should define the meaning of your life?

02:01:51.580 | So for example, when people say that

02:01:53.860 | searching the meaning of our life,

02:01:56.740 | is there some, either is some outside voice,

02:02:00.340 | or is there something outside of you

02:02:04.260 | who actually tells you, you know,

02:02:06.020 | so people talk about, oh, you know,

02:02:09.260 | this is what you have been born to do.

02:02:14.020 | - Right. - Right, like,

02:02:15.380 | this is your destiny.

02:02:17.700 | So who, right, so that's one question,

02:02:21.820 | like who gets to define the meaning of your life?

02:02:25.300 | Should you be finding some other thing,

02:02:27.940 | some other factor to define this for you?

02:02:30.860 | Or is something actually,

02:02:32.380 | it's just entirely what you define yourself,

02:02:35.140 | and it can be very arbitrary.

02:02:37.380 | - Yeah, so an inner voice or an outer voice,

02:02:41.580 | whether it's, it could be spiritual, religious too,

02:02:43.980 | with God, or some other components of the environment

02:02:47.420 | outside of you, or just your own voice.

02:02:50.180 | Do you have an answer there?

02:02:52.420 | - So okay, so with that, I have an answer.

02:02:54.540 | - Yeah. - Through, you know,

02:02:56.580 | the long period of time of thinking and searching,

02:03:00.620 | even searching through outside, right,

02:03:04.580 | you know, voices or factors outside of me.

02:03:07.820 | - Yeah. - So that I have,

02:03:09.380 | and so I've come to the conclusion and realization

02:03:13.060 | that it's you yourself that defines the meaning of life.

02:03:16.820 | - Yeah, that's a big burden though, isn't it?

02:03:20.340 | (laughing)

02:03:22.580 | - I mean, yes and no, right,

02:03:26.020 | so then you have the freedom to define it.

02:03:28.100 | - Yes, yes.

02:03:29.500 | - And another question is,

02:03:32.820 | like what does it really mean by the meaning of life?

02:03:35.660 | Right, and also, whether the question,

02:03:42.940 | even makes sense.

02:03:44.020 | - Absolutely, and you said it somehow distinct

02:03:48.140 | from happiness, so meaning is something much deeper

02:03:51.620 | than just any kind of emotional,

02:03:55.020 | any kind of contentment or joy or whatever,

02:03:57.540 | it might be much deeper.

02:03:58.900 | And then you have to ask, what is deeper than that?

02:04:02.540 | What is there at all?

02:04:04.620 | And then the question starts being silly.

02:04:07.780 | - Right, and also you can say it's deeper,

02:04:09.540 | but you can also say it's shallower,

02:04:10.940 | depending on how people want to define

02:04:13.500 | the meaning of their life.

02:04:14.700 | So for example, most people don't even think

02:04:16.460 | about this question, then the meaning of life to them

02:04:19.540 | doesn't really matter that much.

02:04:22.020 | And also, whether knowing the meaning of life,

02:04:24.780 | whether it actually helps your life to be better

02:04:28.940 | or whether it helps your life to be happier,

02:04:31.140 | these actually are open questions, it's not.

02:04:35.420 | (laughing)

02:04:36.260 | - Of course, most questions are open.

02:04:37.780 | I tend to think that just asking the question

02:04:40.220 | as you mentioned, as you've done for a long time,

02:04:42.780 | is the only, that there is no answer,

02:04:44.940 | and asking the question is a really good exercise.

02:04:47.660 | I mean, I have this, for me personally,

02:04:49.140 | I've had a kind of feeling that creation is,

02:04:54.140 | like for me, has been very fulfilling,

02:04:58.180 | and it seems like my meaning has been to create.

02:05:00.820 | And I'm not sure what that is,

02:05:02.100 | like I don't have a single item of kids,

02:05:05.220 | I'd love to have kids, but I also, sounds creepy,

02:05:08.940 | but I also see, you said see programs,

02:05:13.340 | I see programs as little creations,

02:05:15.660 | I see robots as little creations.

02:05:17.940 | I think those bring, and then ideas, theorems,

02:05:23.220 | and are creations, and those somehow intrinsically,

02:05:27.460 | like you said, bring me joy,

02:05:29.500 | and I think they do to a lot of, at least scientists,

02:05:31.740 | but I think they do to a lot of people.

02:05:34.220 | So that, to me, if I had to force the answer to that,

02:05:37.340 | I would say creating new things yourself.

02:05:42.340 | - For you. - For me.

02:05:44.380 | For me. - Right, right.

02:05:45.220 | - For me, I don't know, but like you said,

02:05:47.820 | it keeps changing.

02:05:48.660 | Is there some answer that--

02:05:49.940 | - And some people, they can, I think,

02:05:52.340 | they may say it's experience, right?

02:05:54.340 | Like their meaning of life,

02:05:56.500 | they just want to experience to the richest

02:05:58.580 | and fullest they can.

02:05:59.940 | And a lot of people do take that path.

02:06:02.700 | - Yeah, seeing life as actually a collection of moments,

02:06:05.580 | and then trying to make the richest possible sets,

02:06:10.580 | fill those moments with the richest possible experiences.

02:06:13.580 | Yeah. - Right.

02:06:14.620 | And for me, I think it's certainly,

02:06:16.420 | we do share a lot of similarity here.

02:06:18.260 | So creation is also really important for me,

02:06:20.420 | even from the things that I've already talked about.

02:06:22.980 | Even like writing papers,

02:06:26.140 | and these are our creations as well.

02:06:30.180 | And I have not quite thought

02:06:32.660 | whether that is really the meaning of my life.

02:06:34.900 | Like in a sense, also, there may be like,

02:06:37.300 | what kind of things should you create?

02:06:38.140 | There are so many different things that you could create.

02:06:41.060 | And also you can say, another view is maybe growth.

02:06:46.460 | It's related but different from experience.

02:06:50.620 | Growth is also maybe type of meaning of life.

02:06:53.460 | It's just, you try to grow every day,

02:06:55.780 | try to be a better self every day.

02:06:59.780 | And also ultimately, we are here,

02:07:04.460 | it's part of the overall evolution.

02:07:07.460 | Right, the world is evolving.

02:07:11.060 | - And it's funny, isn't it funny that the growth

02:07:13.780 | seems to be the more important thing

02:07:15.660 | than the thing you're growing towards?

02:07:18.140 | It's like, it's not the goal, it's the journey to it.

02:07:21.580 | It's almost when you submit a paper,

02:07:25.580 | there's a sort of depressing element to it.

02:07:29.260 | Not to submit a paper, but when that whole project is over.

02:07:32.340 | I mean, there's a gratitude,

02:07:34.020 | there's a celebration and so on,

02:07:35.260 | but you're usually immediately looking for the next thing.

02:07:39.140 | - Yeah. - Or the next step, right?

02:07:40.500 | It's not that, the end of it is not the satisfaction,

02:07:44.420 | it's the hardship, the challenge you have to overcome,

02:07:47.180 | the growth through the process.

02:07:48.780 | It's somehow probably deeply within us,

02:07:51.340 | the same thing that drives the evolutionary process

02:07:54.420 | is somehow within us with everything,

02:07:56.860 | the way we see the world.

02:07:58.860 | Since you're thinking about this,

02:08:00.100 | so you're still in search of an answer.

02:08:02.820 | - I mean, yes and no.

02:08:04.580 | In a sense that I think for people who really dedicate time

02:08:10.420 | to search for the answer, to ask the question,

02:08:13.380 | what is the meaning of life?

02:08:15.700 | It does not necessarily bring you happiness.

02:08:18.140 | (Lex laughing)

02:08:19.820 | - Yeah. - It's a question,

02:08:21.820 | we can say, right, like whether it's a well-defined question

02:08:25.940 | and on the other hand,

02:08:30.220 | given that you get to answer it yourself,

02:08:33.900 | you can define it yourself,

02:08:35.780 | then sure, I can just give it an answer.

02:08:40.780 | And in that sense, yes, it can help.

02:08:44.740 | Like we discussed, right, if you say,

02:08:49.100 | oh, then my meaning of life is to create or to grow,

02:08:54.100 | then yes, then I think they can help.

02:08:57.420 | But how do you know that that is really the meaning of life

02:09:00.420 | or the meaning of your life?

02:09:01.860 | Like there's no way for you to really answer the question.

02:09:05.740 | - For sure, but something about that certainty

02:09:08.180 | is liberating.

02:09:10.100 | So it might be an illusion, you might not really know,

02:09:12.860 | you might be just convincing yourself falsely.

02:09:15.640 | But being sure that that's the meaning,

02:09:17.600 | there's something liberating in that.

02:09:23.380 | There's something freeing in knowing this is your purpose.

02:09:26.340 | So you can fully give yourself to that.

02:09:28.660 | For a long time, I thought, isn't it all relative?

02:09:33.340 | How do we even know what's good and what's evil?

02:09:38.140 | Like isn't everything just relative?

02:09:40.060 | How do we know?

02:09:41.000 | The question of meaning is ultimately the question

02:09:45.500 | of why do anything?

02:09:48.420 | Why is anything good or bad?

02:09:50.280 | Why is anything valuable? - Right, right, exactly.

02:09:53.100 | - At the moment, then you start to,

02:09:55.140 | I think just like you said,

02:09:58.420 | I think it's a really useful question to ask.

02:10:01.180 | But if you ask it for too long and too aggressively--

02:10:07.180 | - It may not be so productive.

02:10:08.860 | - It may not be productive and not just for traditionally,

02:10:13.380 | societally defined success, but also for happiness.

02:10:17.300 | It seems like asking the question about the meaning of life

02:10:20.500 | is like a trap.

02:10:22.740 | It's a, we're destined to be asking.

02:10:25.860 | We're destined to look up to the stars

02:10:27.340 | and ask these big, wide questions

02:10:28.820 | we'll never be able to answer,

02:10:30.500 | but we shouldn't get lost in them.

02:10:32.020 | I think that's probably the,

02:10:34.200 | that's at least the lesson I picked up so far.

02:10:36.300 | On that topic-- - Guys, oh, let me just add

02:10:38.300 | one more thing.

02:10:39.140 | So it's interesting.

02:10:40.020 | So actually, so sometimes, yes, it can help you to focus.

02:10:46.100 | So when I shifted my focus more from seeing the world

02:10:52.060 | more from security to AI and machine learning,

02:10:55.140 | at the time, actually one of the main reasons

02:10:58.460 | that I did that was because at the time,

02:11:02.820 | I thought the meaning of my life

02:11:07.420 | and the purpose of my life is to build intelligent machines.

02:11:11.740 | - And that's, and then your inner voice said

02:11:16.660 | that this is the right journey to take

02:11:20.060 | to build intelligent machines

02:11:21.340 | and that you actually fully realize,

02:11:23.420 | you took a really legitimate big step

02:11:26.420 | to become one of the world-class researchers

02:11:28.500 | to actually make it, to actually go down that journey.

02:11:32.580 | Yeah, that's profound.

02:11:34.320 | That's profound.

02:11:36.460 | I don't think there's a better way

02:11:39.400 | to end a conversation than talking for a while

02:11:43.020 | about the meaning of life.

02:11:44.060 | Dawn, it's a huge honor to talk to you.

02:11:46.060 | Thank you so much for talking today.

02:11:47.540 | - Thank you, thank you.

02:11:49.900 | - Thanks for listening to this conversation with Dawn Song

02:11:52.580 | and thank you to our presenting sponsor, Cash App.

02:11:55.380 | Please consider supporting the podcast

02:11:57.100 | by downloading Cash App and using code LEXPODCAST.

02:12:01.180 | If you enjoy this podcast, subscribe on YouTube,

02:12:03.860 | review it with five stars on Apple Podcast,

02:12:06.140 | support it on Patreon,

02:12:07.300 | or simply connect with me on Twitter @LexFriedman.

02:12:10.540 | And now let me leave you with some words about hacking

02:12:15.100 | from the great Steve Wozniak.

02:12:17.900 | A lot of hacking is playing with other people,

02:12:20.780 | you know, getting them to do strange things.

02:12:24.380 | Thank you for listening and hope to see you next time.

02:12:27.400 | (upbeat music)

02:12:29.980 | (upbeat music)

02:12:32.560 | [BLANK_AUDIO]

Dawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95

Chapters