John Hopfield: Physics View of the Mind and Neurobiology

00:00:00.000 | The following is a conversation with John Hopfield,

00:00:03.440 | professor at Princeton, whose life's work weaved beautifully

00:00:07.600 | through biology, chemistry, neuroscience, and physics.

00:00:11.720 | Most crucially, he saw the messy world of biology

00:00:15.000 | through the piercing eyes of a physicist.

00:00:18.360 | He's perhaps best known for his work

00:00:20.160 | on associative neural networks,

00:00:22.160 | now known as Hopfield networks,

00:00:24.960 | that were one of the early ideas that catalyzed

00:00:27.720 | the development of the modern field of deep learning.

00:00:30.320 | As his 2019 Franklin Medal in Physics Award states,

00:00:35.160 | he applied concepts of theoretical physics

00:00:37.800 | to provide new insights on important biological questions

00:00:41.040 | in a variety of areas, including genetics and neuroscience,

00:00:45.040 | with significant impact on machine learning.

00:00:48.000 | And as John says in his 2018 article titled,

00:00:51.520 | "Now What?"

00:00:52.800 | his accomplishments have often come about

00:00:55.600 | by asking that very question, now what?

00:00:59.720 | And often responding by a major change of direction.

00:01:02.920 | This is the Artificial Intelligence Podcast.

00:01:07.000 | If you enjoy it, subscribe on YouTube,

00:01:09.240 | give it five stars on Apple Podcast,

00:01:11.080 | support it on Patreon, or simply connect with me on Twitter.

00:01:14.480 | And Lex Friedman, spelled F-R-I-D-M-A-N.

00:01:18.980 | As usual, I'll do one or two minutes of ads now,

00:01:21.520 | and never any ads in the middle

00:01:23.000 | that can break the flow of the conversation.

00:01:25.280 | I hope that works for you

00:01:26.720 | and doesn't hurt the listening experience.

00:01:29.440 | This show is presented by Cash App,

00:01:31.920 | the number one finance app in the App Store.

00:01:34.320 | When you get it, use code LEXPODCAST.

00:01:37.800 | Cash App lets you send money to friends,

00:01:39.960 | buy Bitcoin, and invest in the stock market

00:01:42.320 | with as little as $1.

00:01:43.840 | Since Cash App does fractional share trading,

00:01:47.000 | let me mention that the order execution algorithm

00:01:49.440 | that works behind the scenes

00:01:50.960 | to create the abstraction of fractional orders

00:01:53.800 | is to me an algorithmic marvel.

00:01:56.360 | So big props to the Cash App engineers

00:01:58.560 | for solving a hard problem

00:02:00.240 | that in the end provides an easy interface

00:02:02.720 | that takes a step up the next layer of abstraction

00:02:05.320 | over the stock market,

00:02:06.720 | making trading more accessible for new investors

00:02:09.340 | and diversification much easier.

00:02:12.040 | So again, if you get Cash App from the App Store

00:02:15.360 | or Google Play, and use code LEXPODCAST,

00:02:18.440 | you'll get $10, and Cash App will also donate $10 to Thirst,

00:02:22.520 | one of my favorite organizations

00:02:24.200 | that is helping advance robotics and STEM education

00:02:27.280 | for young people around the world.

00:02:28.980 | And now, here's my conversation with John Hopfield.

00:02:34.120 | What difference between biological neural networks

00:02:37.860 | and artificial neural networks

00:02:39.980 | is most captivating and profound to you?

00:02:42.280 | At the higher philosophical level,

00:02:47.080 | let's not get technical just yet.

00:02:51.080 | One of the things that very much intrigues me

00:02:53.320 | is the fact that neurons have all kinds of components,

00:02:58.320 | properties to them.

00:03:01.260 | And evolutionary biology,

00:03:05.440 | if you have some little quirk

00:03:07.320 | in how a molecule works or how a cell works,

00:03:12.040 | and it can be made use of,

00:03:13.520 | evolution will sharpen it up

00:03:15.160 | and make it into a useful feature rather than a glitch.

00:03:20.240 | And so you expect, in neurobiology,

00:03:24.120 | for evolution to have captured all kinds of possibilities

00:03:27.920 | of getting neurons,

00:03:29.040 | of how you get neurons to do things for you.

00:03:31.820 | And that aspect has been completely suppressed

00:03:37.000 | in artificial neural networks.

00:03:38.680 | - Do the glitches become features

00:03:43.200 | in the biological neural network?

00:03:46.720 | - They can.

00:03:48.080 | Look, let me take one of the things

00:03:50.280 | that I used to do research on.

00:03:52.280 | If you take things which oscillate,

00:03:58.720 | they have rhythms which are sort of close to each other,

00:04:02.760 | under some circumstances,

00:04:04.000 | these things will have a phase transition,

00:04:06.940 | and suddenly the rhythm will,

00:04:08.280 | everybody will fall into step.

00:04:10.720 | There was a marvelous physical example of that

00:04:13.200 | in the Millennium Bridge across the Thames River

00:04:17.200 | built about 2001.

00:04:21.000 | And pedestrians walking across,

00:04:23.520 | pedestrians don't walk, synchronize,

00:04:26.000 | they don't walk in lockstep,

00:04:28.300 | but they're all walking about the same frequency.

00:04:31.520 | And the bridge could sway at that frequency,

00:04:33.760 | and the slight sway made pedestrians

00:04:35.920 | tend a little bit to lock into step,

00:04:37.920 | and after a while, the bridge was oscillating back and forth

00:04:41.680 | and the pedestrians were walking in step to it.

00:04:43.960 | You could see it in the movies made out of the bridge.

00:04:46.920 | And the engineers made a simple, minor mistake.

00:04:50.560 | They assumed when you walk, it's step, step, step,

00:04:53.640 | and it's back and forth motion.

00:04:56.360 | But when you walk, it's also right foot, left foot,

00:04:58.920 | side to side motion.

00:05:00.280 | And it's the side to side motion

00:05:01.720 | for which the bridge was strong enough,

00:05:04.440 | but it wasn't stiff enough.

00:05:08.180 | And as a result, you would feel the motion

00:05:11.000 | and you'd fall into step with it.

00:05:12.880 | And people were very uncomfortable with it.

00:05:15.080 | They closed the bridge for two years

00:05:16.480 | while they built stiffening for it.

00:05:18.840 | Now, nerves, look, nerve cells produce action potentials.

00:05:23.840 | You have a bunch of cells which are loosely coupled together

00:05:26.160 | producing action potentials of the same rate.

00:05:29.360 | There'll be some circumstances

00:05:31.760 | under which these things can lock together.

00:05:34.060 | Other circumstances in which they won't.

00:05:37.960 | Well, they fire together,

00:05:40.720 | you can be sure that other cells are gonna notice it.

00:05:43.400 | So you can make a computational feature out of this

00:05:45.960 | in an evolving brain.

00:05:48.840 | Most artificial neural networks

00:05:51.800 | don't even have action potentials,

00:05:53.600 | let alone have the possibility for synchronizing them.

00:05:56.940 | - And you mentioned the evolutionary process.

00:06:01.680 | So the evolutionary process that builds

00:06:05.560 | on top of biological systems leverages

00:06:09.520 | that the weird mess of it somehow.

00:06:15.360 | So how do you make sense of that ability

00:06:18.320 | to leverage all the different kinds of complexities

00:06:22.040 | in the biological brain?

00:06:24.640 | - Well, look, at the biological molecule level,

00:06:28.780 | you have a piece of DNA

00:06:31.660 | which encodes for a particular protein.

00:06:35.400 | You could duplicate that piece of DNA

00:06:37.880 | and now one part of it can code for that protein,

00:06:41.760 | but the other one could itself change a little bit

00:06:45.080 | and thus start coding for a molecule

00:06:46.800 | which is slightly different.

00:06:48.560 | Now that molecule was just slightly different,

00:06:51.360 | had a function which helped any old chemical reaction

00:06:56.360 | was as important to the cell.

00:06:58.140 | You would go ahead and let that try

00:07:03.520 | and evolution would slowly improve that function.

00:07:06.600 | And so you have the possibility of duplicating

00:07:12.560 | and then having things drift apart.

00:07:14.780 | One of them retain the old function,

00:07:16.800 | the other one do something new for you.

00:07:18.760 | And there's evolutionary pressure to improve.

00:07:24.000 | Look, there is in computers too,

00:07:25.480 | but it's improvement has to do with closing some companies

00:07:28.720 | and opening some others.

00:07:30.400 | (Lex laughing)

00:07:31.360 | The evolutionary process looks a little different.

00:07:34.160 | - Yeah, similar timescale perhaps.

00:07:37.560 | - Much shorter in timescale.

00:07:39.660 | - Companies close, yeah, go bankrupt and are born.

00:07:42.680 | - Yeah, shorter, but not much shorter.

00:07:45.940 | Some company lasts a century, but yeah, you're right.

00:07:50.160 | I mean, if you think of companies as a single organism

00:07:53.180 | that builds and you all know, yeah,

00:07:55.540 | it's a fascinating dual correspondence there

00:08:00.540 | between biological-

00:08:02.340 | - And companies have difficulty having a new product

00:08:06.020 | competing with an old product.

00:08:07.660 | - Yeah.

00:08:08.500 | And when IBM built its first PC,

00:08:12.880 | you probably read the book.

00:08:15.000 | They made a little isolated internal unit to make the PC.

00:08:18.920 | And for the first time in IBM's history,

00:08:22.720 | they didn't insist that you build it out of IBM components,

00:08:25.940 | but they understood that they could get into this market,

00:08:31.080 | which is a very different thing

00:08:33.240 | by completely changing their culture.

00:08:38.120 | And biology finds other markets in a more adaptive way.

00:08:43.120 | - Yeah, it's better at it.

00:08:47.280 | It's better at that kind of integration.

00:08:49.280 | So maybe you've already said it,

00:08:52.720 | but what to use the most beautiful aspect

00:08:55.880 | or mechanism of the human mind?

00:08:58.240 | Is it the adaptive, the ability to adapt as you've described

00:09:06.200 | or is there some other little quirk

00:09:07.640 | that you particularly like?

00:09:09.280 | - Adaptation is everything when you get down to it.

00:09:15.680 | But the difference, there are differences between adaptation

00:09:21.720 | where you're learning goes on only over generations

00:09:25.280 | and over evolutionary time,

00:09:28.320 | where you're learning goes on at the time scale

00:09:30.720 | of one individual who must learn from the environment

00:09:34.340 | during that individual's lifetime.

00:09:37.680 | And biology has both kinds of learning in it.

00:09:42.680 | And the thing which makes neurobiology hard

00:09:47.600 | is that it's a mathematical system, as it were,

00:09:52.600 | built on this other kind of evolutionary system.

00:09:56.480 | - What do you mean by mathematical system?

00:10:00.600 | Where's the math and the biology?

00:10:03.560 | - Well, when you talk to a computer scientist

00:10:05.280 | about neural networks, it's all math.

00:10:08.080 | The fact that biology actually came about from evolution

00:10:12.000 | and the fact that biology is about a system

00:10:18.520 | which you can build in three dimensions.

00:10:22.700 | If you look at computer chips,

00:10:26.880 | computer chips are basically two-dimensional structures,

00:10:32.360 | maybe 2.1 dimensions, but they really have difficulty

00:10:36.200 | doing three-dimensional wiring.

00:10:38.520 | Biology is, the neocortex is actually also sheet-like,

00:10:45.720 | and it sits on top of the white matter,

00:10:47.680 | which is about 10 times the volume of the gray matter

00:10:50.400 | and contains all what you might call the wires.

00:10:53.540 | But there's a huge,

00:10:59.160 | the effect of computer structure on what is easy

00:11:03.160 | and what is hard is immense.

00:11:06.080 | - So--

00:11:09.120 | - And biology does, it makes some things easy

00:11:13.280 | that are very difficult to understand

00:11:16.040 | how to do computationally.

00:11:17.920 | On the other hand, you can't do simple

00:11:20.400 | floating-point arithmetic, so it's awfully stupid.

00:11:23.200 | - Yeah, and you're saying this kind of three-dimensional

00:11:25.680 | complicated structure makes, it's still math,

00:11:30.680 | it's still doing math.

00:11:32.360 | The kind of math it's doing enables you to solve problems

00:11:36.180 | of a very different kind.

00:11:38.280 | - That's right, that's right.

00:11:39.780 | - So you mentioned two kinds of adaptation,

00:11:43.720 | the evolutionary adaptation and the adaptation,

00:11:46.840 | or learning at the scale of a single human life.

00:11:50.000 | Which do you, which is particularly beautiful

00:11:55.960 | to you and interesting from a research

00:11:59.320 | and from just a human perspective?

00:12:01.220 | And which is more powerful?

00:12:03.600 | - I find things most interesting that I begin to see

00:12:08.720 | how to get into the edges of them

00:12:12.640 | and tease them apart a little bit and see how they work.

00:12:15.440 | And since I can't see the evolutionary process going on,

00:12:22.720 | I'm in awe of it, but I find it just a black hole

00:12:27.720 | as far as trying to understand what to do.

00:12:31.960 | And so in a certain sense, I'm in awe of it,

00:12:35.200 | but I couldn't be interested in working on it.

00:12:37.560 | - The human life's time scale is however thing

00:12:43.860 | you can tease apart and study.

00:12:47.720 | - Yeah, you can do, there's the developmental neurobiology

00:12:51.120 | which understands how the connections

00:12:54.200 | and how the structure evolves from a combination

00:13:00.920 | of what the genetics is like and the real,

00:13:04.400 | the fact that you're building a system in three dimensions.

00:13:08.240 | - In just days and months, those early,

00:13:13.800 | early days of a human life are really interesting.

00:13:17.240 | - They are, and of course, there are times

00:13:21.120 | of immense cell multiplication.

00:13:24.280 | There are also times of the greatest cell death

00:13:28.120 | in the brain is during infancy.

00:13:30.880 | - It's turnover.

00:13:33.160 | - So what is not effective, what is not wired well enough

00:13:39.520 | to use the moment, throw it out.

00:13:42.840 | - It's a mysterious process.

00:13:45.040 | - From, let me ask, from what field do you think

00:13:49.400 | the biggest breakthroughs in understanding the mind

00:13:51.880 | will come in the next decades?

00:13:56.360 | Is it neuroscience, computer science, neurobiology,

00:13:59.640 | psychology, physics, maybe math, maybe literature?

00:14:04.640 | (laughs)

00:14:08.320 | - Well, of course, I see the world always

00:14:11.000 | through a lens of physics.

00:14:12.480 | I grew up in physics.

00:14:14.900 | And the way I pick problems is very characteristic

00:14:19.900 | of physics and of an intellectual background,

00:14:23.980 | which is not psychology, which is not chemistry

00:14:26.460 | and so on and so on.

00:14:28.420 | - Now, both of your parents are physicists.

00:14:30.180 | - Both of my parents were physicists,

00:14:31.900 | and the real thing I got out of that was a feeling

00:14:36.140 | that the world is an understandable place.

00:14:41.860 | And if you do enough experiments and think about

00:14:45.460 | what they mean and structure things,

00:14:48.220 | or you can do the mathematics of the relevant

00:14:51.300 | to the experiments, you also be able to understand

00:14:53.940 | how things work.

00:14:55.020 | - But that was a few years ago.

00:14:58.900 | Did you change your mind at all through many decades

00:15:03.580 | of trying to understand the mind,

00:15:05.980 | of studying it different kinds of ways,

00:15:07.540 | not even the mind, just biological systems?

00:15:11.060 | You still have hope that physics, that you can understand?

00:15:13.980 | - There's a question of what do you mean by understand?

00:15:20.780 | - Of course.

00:15:21.620 | - When I taught freshman physics, I used to say,

00:15:24.460 | I wanted to give physics to understand the subject,

00:15:26.580 | to understand Newton's laws.

00:15:28.900 | I didn't want them simply to memorize a set of examples

00:15:33.780 | to which they knew the equations to write down

00:15:36.660 | to generate the answers.

00:15:38.180 | I had this nebulous idea of understanding.

00:15:42.260 | So that if you looked at a situation, you could say,

00:15:45.180 | oh, I expect the ball to make that trajectory,

00:15:48.340 | or I expect some intuitive notion of understanding.

00:15:52.580 | And I don't know how to express that very well.

00:15:57.580 | I've never known how to express it well.

00:16:01.220 | And you run smack up against it,

00:16:03.980 | when you do these, look at these simple neural nets,

00:16:07.860 | feed forward neural nets, which do amazing things,

00:16:12.860 | and yet you know, contain nothing of the essence

00:16:16.500 | of what I would have felt was understanding.

00:16:20.180 | Understanding is more than just an enormous lookup table.

00:16:23.020 | - Let's linger on that.

00:16:26.420 | How sure you are of that?

00:16:28.140 | What if the table gets really big?

00:16:30.320 | So, I mean, ask another way,

00:16:34.540 | these feed forward neural networks,

00:16:37.140 | do you think they'll ever understand?

00:16:38.980 | - Could answer that in two ways.

00:16:41.900 | I think if you look at real systems,

00:16:45.280 | feedback is an essential aspect

00:16:50.260 | of how these real systems compute.

00:16:52.980 | On the other hand, if I have a mathematical system

00:16:55.500 | with feedback, I know I can unlayer this and do it,

00:16:58.880 | but I have an exponential expansion

00:17:03.900 | in the amount of stuff I have to build

00:17:06.100 | if I can solve the problem that way.

00:17:07.940 | - So feedback is essential.

00:17:09.980 | So we can talk even about recurrent neural networks,

00:17:13.140 | so recurrence, but do you think all the pieces are there

00:17:17.580 | to achieve understanding through these simple mechanisms?

00:17:22.300 | Like, back to our original question,

00:17:25.220 | what is the fundamental, is there a fundamental difference

00:17:28.460 | between artificial neural networks and biological,

00:17:31.940 | or is it just a bunch of surface stuff?

00:17:34.700 | - Suppose you ask a neurosurgeon,

00:17:36.540 | when is somebody dead?

00:17:39.600 | - Yeah.

00:17:42.500 | - They'll probably go back to saying,

00:17:44.220 | well, I can look at the brain rhythms

00:17:46.780 | and tell you this is a brain

00:17:49.260 | which is never gonna function again.

00:17:51.360 | This one is, this other one is one

00:17:53.380 | which if we treat it well, is still recoverable.

00:17:58.220 | And then just do that by some electrodes

00:18:00.700 | and looking at simple electrical patterns

00:18:04.620 | which don't look in any detail at all

00:18:08.100 | at what individual neurons are doing.

00:18:10.140 | These rhythms are utterly absent

00:18:17.660 | from anything which goes on at Google.

00:18:19.740 | - Yeah, but the rhythms.

00:18:25.100 | - But the rhythms what?

00:18:27.900 | - So, well, that's like comparing, okay, I'll tell you.

00:18:31.300 | It's like you're comparing

00:18:32.980 | the greatest classical musician in the world

00:18:38.540 | to a child first learning to play.

00:18:41.180 | The question I'm at,

00:18:42.060 | but they're still both playing the piano.

00:18:44.660 | I'm asking, is there, will it ever go on at Google?

00:18:48.820 | Do you have a hope?

00:18:51.100 | Because you're one of the seminal figures

00:18:53.680 | in both launching both disciplines,

00:18:56.780 | both sides of the river.

00:18:58.880 | - I think it's going to go on generation after generation

00:19:05.580 | the way it has where what you might call

00:19:09.220 | the AI computer science community says,

00:19:11.880 | let's take the following.

00:19:14.060 | This is our model of neurobiology at the moment.

00:19:16.940 | Let's pretend it's good enough

00:19:20.500 | and do everything we can with it.

00:19:22.440 | And it does interesting things.

00:19:25.940 | And after the while it sort of grinds into the sand

00:19:30.380 | and you say, ah, something else is needed for neurobiology

00:19:35.100 | and some other grand thing comes in

00:19:37.220 | and enables you to go a lot further.

00:19:41.120 | But we'll go into the sand again.

00:19:44.300 | And I think it's going to be generations of this evolution.

00:19:47.380 | I don't know how many of them

00:19:48.860 | and each one is going to get you further

00:19:50.900 | into what our brain does.

00:19:53.520 | And in some sense,

00:19:56.720 | pass the Turing test longer and more broad aspects.

00:20:01.720 | And how many of these are good

00:20:06.960 | there are going to have to be before you say,

00:20:09.200 | I've made something, I've made a human, I don't know.

00:20:15.400 | - But your sense is it might be a couple.

00:20:17.800 | - My sense is it might be a couple more.

00:20:19.640 | - Yeah.

00:20:20.800 | - And going back to my brain waves as it were.

00:20:25.800 | - Yes.

00:20:26.800 | - From the AI point of view,

00:20:31.760 | they would say, ah, maybe these are an heavy phenomenon

00:20:36.040 | and not important at all.

00:20:37.660 | The first car I had, a real wreck of a 1936 Dodge

00:20:45.040 | go above 45 miles an hour and the wheels was shimmy.

00:20:51.000 | - Yeah.

00:20:51.840 | - Good speedometer that.

00:20:54.800 | Now, nobody designed the car that way.

00:20:59.840 | The car is malfunctioning to have that.

00:21:02.120 | But in biology, if it were useful to know

00:21:05.920 | when are you going more than 45 miles an hour,

00:21:08.520 | you just capture that

00:21:10.160 | and you wouldn't worry about where it came from.

00:21:12.560 | - Yeah.

00:21:16.520 | - It's going to be a long time before that kind of thing,

00:21:19.060 | which can take place in large complex networks of things

00:21:24.060 | is actually used in the computation.

00:21:27.800 | Look, how many transistors

00:21:31.960 | are there in your laptop these days?

00:21:33.840 | - Actually, I don't know the number.

00:21:36.480 | It's--

00:21:37.320 | - It's on the scale of 10 to the 10.

00:21:39.120 | I can't remember the number either.

00:21:40.800 | - Yeah.

00:21:41.640 | - And all the transistors are somewhat similar.

00:21:45.840 | And most physical systems with that many parts,

00:21:50.020 | all of which are similar, have collective properties.

00:21:54.320 | - Yes.

00:21:55.420 | - Sound waves in air, earthquakes,

00:21:57.780 | what have you have collective properties, weather.

00:22:00.540 | There are no collective properties used

00:22:05.460 | in artificial neural networks in AI.

00:22:08.260 | - Yeah, it's very--

00:22:12.380 | - If biology uses them,

00:22:14.540 | it's going to take us to more generations of things

00:22:17.220 | for people to actually dig in

00:22:19.180 | and see how they are used and what they mean.

00:22:21.540 | - See, you're very right.

00:22:25.420 | We might have to return several times to neurobiology

00:22:29.020 | and try to make our transistors more messy.

00:22:33.060 | - Yeah, yeah.

00:22:34.120 | At the same time, the simple ones

00:22:36.980 | will conquer big aspects.

00:22:43.620 | And I think one of the most biggest surprises to me was

00:22:47.660 | how well learning systems,

00:22:52.780 | which are manifestly non-biological,

00:22:55.120 | how important they can be actually,

00:22:58.300 | and how important and how useful they can be in AI.

00:23:02.240 | - So if we can just take a stroll to some of your work,

00:23:08.380 | that is incredibly surprising

00:23:12.460 | that it works as well as it does,

00:23:14.060 | that launched a lot of the recent work with neural networks.

00:23:18.300 | If we go to what are now called Hopfield networks,

00:23:23.300 | can you tell me what is associative memory in the mind

00:23:29.700 | for the human side?

00:23:31.060 | Let's explore memory for a bit.

00:23:33.560 | - Okay, what you mean by associative memory is

00:23:38.820 | how you have a memory of each of your friends.

00:23:43.140 | Your friend has all kinds of properties

00:23:44.860 | from what they look like,

00:23:45.780 | to what their voice sounds like,

00:23:47.060 | to where they went to college,

00:23:48.740 | where you met them,

00:23:50.000 | go on and on,

00:23:53.260 | what science papers they've written.

00:23:55.300 | If I start talking about a

00:23:59.820 | five foot 10 wire-aided cognitive scientist

00:24:06.260 | that's got a very bad back,

00:24:08.180 | it doesn't take very long for you to say,

00:24:10.300 | oh, he's talking about Jeff Hinton.

00:24:11.940 | I never mentioned the name or anything very particular,

00:24:16.780 | but somehow a few facts that are associated

00:24:22.180 | with a particular person

00:24:24.900 | enables you to get a hold of the rest of the facts,

00:24:27.820 | or not the rest of them, another subset of them.

00:24:30.840 | And it's this ability to link things together,

00:24:37.300 | link experiences together,

00:24:39.060 | which goes under the general name of associative memory.

00:24:44.540 | And a large part of intelligent behavior

00:24:47.980 | is actually just large associative memories

00:24:50.900 | at work, as far as I can see.

00:24:52.700 | - What do you think is the mechanism

00:24:56.180 | of how it works in the mind?

00:24:58.600 | Is it a mystery to you still?

00:25:01.340 | Do you have inklings of how this essential

00:25:06.980 | thing for cognition works?

00:25:08.700 | - What I made 35 years ago

00:25:14.260 | was of course a crude physics model

00:25:17.580 | to show the kind,

00:25:19.180 | actually enable you to understand,

00:25:22.980 | my old sense of understanding as a physicist,

00:25:25.140 | because you could say,

00:25:26.620 | ah, I understand why this goes to stable states.

00:25:29.540 | It's like things going downhill.

00:25:32.580 | - Right.

00:25:33.940 | - And that gives you something with which to think

00:25:37.820 | in physical terms rather than only in mathematical terms.

00:25:42.700 | - So you've created these associative artificial networks.

00:25:47.100 | - That's right.

00:25:48.260 | And now if you look at what I did,

00:25:50.600 | I didn't at all describe a system which gracefully learns.

00:25:58.700 | I described a system in which you could understand

00:26:02.460 | how learning could link things together,

00:26:05.980 | how very crudely it might learn.

00:26:08.080 | One of the things which intrigues me

00:26:11.220 | as I reinvestigate that system now to some extent is,

00:26:15.180 | look, I'll see you every second

00:26:22.780 | for the next hour or what have you.

00:26:25.300 | Each look at you is a little bit different.

00:26:28.780 | I don't store all those second by second images.

00:26:33.060 | I don't store 3000 images.

00:26:34.700 | I somehow compact this information.

00:26:37.140 | So I now have a view of you,

00:26:40.280 | which I can use.

00:26:45.700 | It doesn't slavishly remember anything in particular,

00:26:49.280 | but it compacts the information into useful chunks,

00:26:53.000 | which are somehow,

00:26:55.960 | it's these chunks,

00:26:56.840 | which are not just activities of neurons,

00:26:59.880 | bigger things than that,

00:27:01.840 | which are the real entities which are useful to you.

00:27:05.900 | - Useful to you to describe,

00:27:10.300 | to compress this information coming at you.

00:27:13.520 | - And you have to compress it in such a way

00:27:15.040 | that if the information comes in just like this again,

00:27:19.360 | I don't bother to rewrite it,

00:27:22.040 | or efforts to rewrite it simply do not yield anything

00:27:26.720 | because those things are already written.

00:27:29.720 | And that needs to be not,

00:27:32.160 | look this up,

00:27:33.000 | have I written this,

00:27:34.480 | have I stored it somewhere already?

00:27:36.240 | It's gotta be something which is much more automatic

00:27:39.860 | in the machine hardware.

00:27:41.920 | - Right, so in the human mind,

00:27:44.800 | how complicated is that process, do you think?

00:27:48.040 | So you've created,

00:27:49.460 | feels weird to be sitting with John Hopfield

00:27:52.680 | calling him Hopfield Networks, but--

00:27:54.960 | - It is weird.

00:27:55.800 | (laughing)

00:27:57.680 | - Yeah, but nevertheless, that's what everyone calls him,

00:28:00.660 | so here we are.

00:28:02.960 | So that's a simplification,

00:28:05.040 | that's what a physicist would do.

00:28:06.800 | You and Richard Feynman sat down

00:28:08.520 | and talked about associative memory.

00:28:10.040 | Now if you look at the mind,

00:28:14.560 | where you can't quite simplify it so perfectly,

00:28:17.480 | do you think--

00:28:18.320 | - Let me backtrack just a little bit.

00:28:21.960 | - Yeah.

00:28:23.040 | - Biology is about dynamical systems.

00:28:25.700 | Computers are dynamical systems.

00:28:29.560 | You can ask,

00:28:32.120 | if you want to model biology,

00:28:36.480 | if you want to model neurobiology,

00:28:38.520 | what is the time scale?

00:28:40.960 | There's a dynamical system,

00:28:42.880 | in which,

00:28:43.720 | fairly fast time scale,

00:28:46.480 | in which you can say,

00:28:47.440 | the synapses don't change much during this computation,

00:28:50.520 | so I'll think of the synapses as fixed,

00:28:52.840 | and just do the dynamics of the activity.

00:28:56.040 | Or you can say,

00:28:57.200 | the synapses are changing fast enough

00:29:00.800 | that I have to have the synaptic dynamics

00:29:03.240 | working at the same time as the system dynamics,

00:29:06.000 | in order to understand the biology.

00:29:09.600 | Most, if you look at the feedforward artificial neural nets,

00:29:16.880 | they're all done as learning,

00:29:20.240 | first of all, I spend some time learning,

00:29:22.120 | not performing,

00:29:23.160 | then I turn off learning and I perform.

00:29:25.120 | - Right.

00:29:27.680 | - That's not biology.

00:29:28.960 | And so, as I look more deeply at neurobiology,

00:29:34.720 | even as an associate of memory,

00:29:37.000 | I've got to face the fact that the dynamics

00:29:39.400 | of a synapse change is going on all the time.

00:29:42.720 | And I can't just get by by saying,

00:29:46.320 | I'll do the dynamics of activity with fixed synapses.

00:29:51.660 | - So the synaptic, the dynamics of the synapses

00:29:56.100 | is actually fundamental to the whole system.

00:29:58.180 | - Yeah, yeah.

00:29:59.100 | And there's nothing necessarily separating the time scales.

00:30:04.780 | When the time scales can be separated,

00:30:06.540 | it's neat from the physicist's

00:30:08.140 | or the mathematician's point of view,

00:30:10.820 | but it's not necessarily true in neurobiology.

00:30:13.660 | - So you're kind of dancing beautifully

00:30:16.780 | between showing a lot of respect to physics,

00:30:20.260 | and then also saying that physics cannot quite reach

00:30:25.260 | the complexity of biology.

00:30:29.580 | So where do you land?

00:30:30.620 | Or do you continuously dance between the two points?

00:30:33.340 | - I continuously dance between them

00:30:34.900 | because my whole notion of understanding

00:30:37.020 | is that you can describe to somebody else

00:30:42.960 | how something works in ways which are honest and believable

00:30:48.900 | and still not describe all the nuts and bolts in detail.

00:30:53.720 | Weather.

00:30:56.020 | I can describe weather

00:30:58.300 | as 10 to the 32 molecules colliding in the atmosphere.

00:31:05.500 | I can simulate weather that way,

00:31:06.900 | I have a big enough machine, I'll simulate it accurately.

00:31:10.120 | It's no good for understanding.

00:31:14.480 | If I just want to understand things,

00:31:17.380 | I want to understand things in terms of wind patterns,

00:31:20.580 | hurricanes, pressure differentials, and so on.

00:31:23.340 | All things as they're collective.

00:31:24.980 | And the physicist in me always hopes

00:31:32.100 | that biology will have some things

00:31:34.700 | which can be said about it which are both true

00:31:37.740 | and for which you don't need all the molecular details

00:31:40.820 | of the molecules colliding.

00:31:42.360 | That's what I mean from the roots of physics.

00:31:46.740 | My understanding.

00:31:47.800 | - So what did, again, sorry,

00:31:51.580 | but Hopfield Networks help you understand,

00:31:54.780 | what insight did it give us about memory, about learning?

00:31:59.780 | - They didn't give insights about learning.

00:32:05.660 | They gave insights about how things having learned

00:32:10.220 | could be expressed.

00:32:12.420 | How having learned a picture of you reminds me of your name.

00:32:17.420 | That would, it didn't describe a reasonable way

00:32:23.980 | of actually doing the learning.

00:32:25.680 | Or at least that if you had previously learned

00:32:30.260 | the connections of this kind of pattern,

00:32:34.060 | would now be able to behave in a physical way

00:32:38.660 | which is a, ah, if I put part of the pattern in here,

00:32:42.020 | the other part of the pattern will complete over here.

00:32:45.860 | I could understand that physics

00:32:48.220 | if the right learning stuff had already been put in.

00:32:51.700 | And it could understand why then putting in a picture

00:32:53.620 | of somebody else would generate something else over here.

00:32:56.420 | But it did not have a reasonable description

00:33:01.860 | of the learning process.

00:33:03.780 | - But even, so forget learning.

00:33:05.620 | I mean, that's just a powerful concept

00:33:07.260 | that sort of forming representations

00:33:11.700 | that are useful to be robust,

00:33:15.700 | for error correction kind of thing.

00:33:17.260 | So this is kind of what the biology does

00:33:20.820 | we're talking about.

00:33:22.500 | - Yeah, and what my paper did was simply enable you,

00:33:26.420 | there are lots of ways of being robust.

00:33:29.940 | If you think of a dynamical system,

00:33:36.460 | you think of a system where a path is going on in time.

00:33:42.120 | And if you think of a computer,

00:33:43.800 | there's a computational path,

00:33:45.200 | which is going on in a huge dimensional space

00:33:48.440 | of ones and zeros.

00:33:49.760 | And an error correcting system is a system

00:33:55.720 | which if you get a little bit off that trajectory,

00:33:58.680 | will push you back onto that trajectory again.

00:34:00.920 | So you get to the same answer

00:34:02.280 | in spite of the fact that there were things,

00:34:04.680 | the computation wasn't being ideally done

00:34:07.440 | all the way along the line.

00:34:10.880 | And there are lots of models for error correction.

00:34:13.560 | But one of the models for error correction is to say,

00:34:17.080 | there's a valley that you're following, flowing down.

00:34:20.740 | And if you push a little bit off the valley,

00:34:23.920 | just like water being pushed a little bit by a rock,

00:34:26.920 | gets back and follows the course of the river.

00:34:30.080 | And that basically the analog in the physical system,

00:34:35.080 | which enables you to say,

00:34:38.640 | oh yes, error free computation and an associative memory

00:34:43.600 | are very much like things that I can understand

00:34:46.880 | from the point of view of a physical system.

00:34:49.380 | The physical system can be under some circumstances,

00:34:54.520 | an accurate metaphor.

00:34:55.940 | It's not the only metaphor.

00:34:59.480 | There are error correction schemes,

00:35:01.920 | which don't have a valley and energy behind them.

00:35:06.760 | But those are error correction schemes

00:35:09.040 | which a mathematician may be able to understand,

00:35:11.240 | but I don't.

00:35:12.080 | - So there's the physical metaphor

00:35:16.200 | that seems to work here.

00:35:18.960 | - That's right, that's right.

00:35:20.600 | - So these kinds of networks actually led to a lot

00:35:25.600 | of the work that is going on now in neural networks,

00:35:29.760 | artificial neural networks.

00:35:30.880 | So the follow on work with restricted Boltzmann machines

00:35:34.800 | and deep belief nets followed on from these ideas

00:35:39.800 | of the Hopfield network.

00:35:41.760 | So what do you think about this continued progress

00:35:46.760 | of that work towards now re-revigorated exploration

00:35:51.800 | of feed forward neural networks

00:35:54.360 | and recurrent neural networks

00:35:55.720 | and convolutional neural networks

00:35:57.300 | and kinds of networks that are helping solve

00:36:01.560 | image recognition, natural language processing,

00:36:03.860 | all that kind of stuff.

00:36:05.000 | - It's always intrigued me that one of the most long lived

00:36:09.760 | of the learning systems is the Boltzmann machine,

00:36:14.040 | which is intrinsically a feedback network.

00:36:17.300 | And with the brilliance of Hinton and Sanofsky

00:36:23.960 | to understand how to do learning in that.

00:36:26.860 | And it's still a useful way to understand learning

00:36:30.800 | and understand, and the learning that you understand

00:36:34.600 | in that has something to do with the way

00:36:36.620 | that feed forward systems work.

00:36:39.100 | But it's not always exactly simple

00:36:41.620 | to express that intuition.

00:36:44.460 | But it always amuses me to see Hinton going back

00:36:49.740 | to the will yet again on a form of the Boltzmann machine,

00:36:53.340 | because really that which has feedback

00:36:59.220 | and interesting probabilities in it

00:37:01.220 | is a lovely encapsulation of something computational.

00:37:05.860 | - Something computational?

00:37:09.320 | - Something both computational and physical.

00:37:12.160 | Computational in the,

00:37:14.280 | it's very much related to feed forward networks.

00:37:17.440 | Physical in that Boltzmann machine learning

00:37:21.760 | is really learning a set of parameters

00:37:24.880 | for a physics Hamiltonian or energy function.

00:37:28.060 | - Mm-hmm.

00:37:29.620 | What do you think about learning in this whole domain?

00:37:32.440 | Do you think the aforementioned guy, Jeff Hinton,

00:37:37.440 | all the work there with back propagation,

00:37:42.020 | all the kind of learning that goes on in these networks,

00:37:46.160 | how do you, if we compare it to learning in the brain,

00:37:51.540 | for example, is there echoes of the same kind of power

00:37:55.520 | that back propagation reveals

00:37:59.020 | about these kinds of recurrent networks?

00:38:01.640 | Or is it something fundamentally different

00:38:03.900 | going on in the brain?

00:38:05.040 | - I don't think the brain is as deep

00:38:13.900 | as the deepest networks go,

00:38:16.080 | the deepest computer science networks.

00:38:20.060 | And I do wonder whether part of that depth

00:38:24.260 | of the computer science networks is necessitated

00:38:28.320 | by the fact that the only learning

00:38:29.880 | that's easily done on a machine is feed forward.

00:38:34.760 | And so there's the question of to what extent

00:38:38.460 | has the biology, which has some feed forward

00:38:42.700 | and some feed back,

00:38:43.900 | been captured by something which has got many more neurons,

00:38:51.580 | much more depth than neurons.

00:38:53.780 | - So part of you wonders if the feedback

00:38:59.740 | is actually more essential than the number of neurons

00:39:02.460 | or the depth, the dynamics of the feedback.

00:39:06.380 | - The dynamics of the feedback.

00:39:08.780 | Look, if you don't have feedback,

00:39:11.700 | it's a little bit like building a big computer

00:39:14.620 | and running it through one clock cycle.

00:39:17.800 | And then you can't do anything

00:39:20.060 | 'cause you'd reload something coming in.

00:39:23.040 | How do you use the fact that there are multiple clocks?

00:39:28.220 | How do I use the fact that you can close your eyes,

00:39:30.740 | stop listening to me and think about a chess board

00:39:33.820 | for a few minutes without any input whatsoever?

00:39:37.080 | - Yeah, that memory thing,

00:39:41.580 | that's fundamentally a feedback kind of mechanism.

00:39:46.000 | You're going back to something.

00:39:47.500 | - Yes.

00:39:48.340 | It's hard to understand.

00:39:51.980 | It's hard to introspect, let alone consciousness.

00:39:56.200 | 'Cause that's all-

00:39:58.980 | - Let alone consciousness, yes, yes.

00:40:01.100 | - 'Cause that's tied up in there too.

00:40:02.460 | You can't just put that on another shelf.

00:40:05.180 | - Every once in a while, I get interested in consciousness

00:40:09.780 | and then I go and I've done that for years

00:40:12.820 | and ask one of my bettors, as it were,

00:40:17.140 | their view on consciousness.

00:40:18.700 | And it's been interesting collecting them.

00:40:20.900 | - What is consciousness?

00:40:25.540 | Let's try to take a brief step into that room.

00:40:27.980 | - Well, I asked Marvin Minsky,

00:40:32.380 | the view on consciousness.

00:40:33.700 | And Marvin said,

00:40:34.820 | consciousness is basically overrated.

00:40:39.360 | It may be an epiphenomenon.

00:40:42.860 | After all, all the things your brain does,

00:40:45.340 | which are actually hard computations,

00:40:49.660 | you do non-consciously.

00:40:51.560 | And there's so much evidence that even the simple things

00:40:59.020 | you do, you can make decisions,

00:41:03.300 | you can make committed decisions about them.

00:41:05.700 | The neurobiologist can say, he's now committed.

00:41:08.260 | He's going to move the hand left

00:41:10.100 | before you know it.

00:41:14.820 | - So his view that consciousness is not,

00:41:16.820 | that's just like little icing on the cake.

00:41:19.380 | The real cake is in the subconscious.

00:41:21.420 | - Yeah, yeah.

00:41:22.980 | Subconscious, non-conscious.

00:41:24.980 | - Non-conscious, what's the better word, sir?

00:41:27.580 | - It's only that Freud captured the other word.

00:41:29.740 | - Yeah, it's a confusing word, subconscious.

00:41:33.380 | - Nicholas Chater wrote an interesting book.

00:41:35.840 | I think the title of it is "The Mind is Flat."

00:41:40.560 | (chuckles)

00:41:42.820 | Flat in a neural net sense,

00:41:48.400 | might be flat is something which is a very broad neural net

00:41:53.400 | without really any layers in depth,

00:41:56.280 | or the deep brain would be many layers and not so broad.

00:41:59.580 | In the same sense that if you push Minsky hard enough,

00:42:05.080 | he would probably have said,

00:42:07.840 | consciousness is your effort to explain to yourself

00:42:11.600 | that which you have already done.

00:42:14.540 | (chuckles)

00:42:16.800 | - Yeah, it's the weaving of the narrative

00:42:20.040 | around the things that already been computed for you.

00:42:22.920 | - That's right, and so much of what we do

00:42:26.520 | for our memories of events, for example,

00:42:30.820 | if there's some traumatic event you witness,

00:42:35.720 | you will have a few facts about it correctly done.

00:42:39.600 | If somebody asks you about it, you will weave a narrative,

00:42:43.000 | which is actually much more rich in detail than that,

00:42:47.240 | based on some anchor points you have of correct things,

00:42:50.640 | and pulling together general knowledge on the other,

00:42:53.880 | but you will have a narrative.

00:42:55.720 | And once you generate that narrative,

00:42:58.320 | you are very likely to repeat that narrative

00:43:00.840 | and claim that all the things you have in it

00:43:02.940 | are actually the correct things.

00:43:05.040 | There was a marvelous example of that

00:43:06.840 | in the Watergate/impeachment era of John Dean.

00:43:11.840 | John Dean, you're too young to know,

00:43:19.880 | had been the personal lawyer of Nixon.

00:43:23.980 | And so John Dean was involved in the cover up,

00:43:28.760 | and John Dean ultimately realized

00:43:32.280 | the only way to keep himself out of jail for a long time

00:43:35.600 | was actually to tell some of the truths about Nixon.

00:43:38.760 | And John Dean was a tremendous witness.

00:43:41.080 | He would remember these conversations in great detail,

00:43:45.880 | and very convincing detail.

00:43:48.280 | And long afterward, some of the tapes,

00:43:54.280 | the secret tapes, as it were,

00:43:56.240 | from which John Dean was recalling these conversations

00:44:01.600 | were published.

00:44:03.200 | And one found out that John Dean had a good,

00:44:05.400 | but not exceptional memory.

00:44:07.120 | What he had was an ability to paint vividly,

00:44:10.560 | and in some sense accurately,

00:44:13.040 | the tone of what was going on.

00:44:15.120 | - By the way, that's a beautiful description

00:44:18.680 | of consciousness.

00:44:19.920 | (laughs)

00:44:22.000 | Do you, like where do you stand in your, today,

00:44:28.520 | (laughs)

00:44:32.520 | so perhaps it changes day to day,

00:44:34.600 | but where do you stand on the importance of consciousness

00:44:37.720 | in our whole big mess of cognition?

00:44:39.880 | Is it just a little narrative maker,

00:44:45.800 | or is it actually fundamental to intelligence?

00:44:48.900 | - That's a very hard one.

00:44:56.160 | But I asked Francis Crick about consciousness.

00:44:58.800 | He launched forward in a long monologue

00:45:03.320 | about Mendel and the peas.

00:45:05.280 | - Yeah.

00:45:06.200 | - And how Mendel knew that there was something,

00:45:08.720 | and how biologists understood

00:45:10.640 | that there was something in inheritance,

00:45:13.280 | which was just very, very different.

00:45:16.280 | And the fact that inherited traits

00:45:19.920 | didn't just wash out into a gray,

00:45:22.480 | but were this or this,

00:45:25.040 | and propagated,

00:45:26.800 | that that was absolutely fundamental to biology.

00:45:30.720 | And it took generations of biologists

00:45:33.440 | to understand that there was genetics,

00:45:36.280 | and it took another generation or two

00:45:38.040 | to understand that genetics came from DNA.

00:45:41.360 | But very shortly after Mendel,

00:45:46.200 | thinking biologists did realize

00:45:48.040 | that there was a deep problem about inheritance.

00:45:54.760 | And Francis would have liked to have said,

00:45:58.200 | "And that's why I'm working on consciousness."

00:46:01.560 | But of course, he didn't have any smoking gun

00:46:04.000 | in the sense of Mendel.

00:46:05.380 | And that's the weakness of his position.

00:46:10.640 | If you read his book,

00:46:12.760 | which he wrote with Koch, I think.

00:46:16.120 | - Yeah, Christoph Koch, yeah.

00:46:18.040 | - I find it unconvincing for the smoking gun reason.

00:46:22.700 | (sighs)

00:46:24.700 | So I've gone on collecting views

00:46:29.180 | without actually having taken a very strong one myself,

00:46:32.740 | because I haven't seen the entry point.

00:46:35.340 | Not seeing the smoking gun

00:46:37.620 | from the point of view of physics,

00:46:38.860 | I don't see the entry point.

00:46:41.180 | Whereas in neurobiology,

00:46:42.820 | once I understood the idea of a collective,

00:46:46.100 | an evolution of dynamics,

00:46:48.900 | which could be described as a collective phenomenon,

00:46:52.180 | I thought, "Ah, there's a point

00:46:54.660 | "where what I know about physics

00:46:56.940 | "is so different from any neurobiologist

00:46:58.980 | "that I have something that I might be able to contribute."

00:47:01.820 | - And right now, there's no way to grasp a consciousness

00:47:05.620 | from a physics perspective.

00:47:07.700 | - From my point of view, that's correct.

00:47:09.740 | And of course, people, physicists like everybody else,

00:47:16.500 | think very muddily about things.

00:47:18.420 | You ask the closely related question

00:47:21.700 | about free will, do you believe you have free will?

00:47:25.480 | Physicists will give an offhand answer

00:47:30.140 | and then backtrack, backtrack, backtrack,

00:47:32.620 | where they realize that the answer they gave

00:47:34.820 | must fundamentally contradict the laws of physics.

00:47:37.540 | - Naturally, answering questions of free will

00:47:40.380 | and consciousness naturally lead to contradictions

00:47:42.820 | from a physics perspective.

00:47:44.280 | 'Cause it eventually ends up with quantum mechanics,

00:47:48.060 | and then you get into that whole mess

00:47:50.460 | of trying to understand how much,

00:47:54.740 | from a physics perspective,

00:47:56.700 | how much is determined, already predetermined,

00:47:59.660 | much is already deterministic about our universe.

00:48:02.340 | There's lots of different--

00:48:03.420 | - And if you don't push quite that far,

00:48:05.860 | you can say essentially all of neurobiology,

00:48:09.500 | which is relevant, can be captured

00:48:11.480 | by classical equations of motion.

00:48:13.720 | Because in my view of the mysteries of the brain

00:48:18.960 | are not the mysteries of quantum mechanics,

00:48:22.140 | but the mysteries of what can happen

00:48:24.820 | when you have a dynamical system,

00:48:27.180 | driven system with 10 to the 14 parts.

00:48:30.660 | That that complexity is something which is,

00:48:34.960 | that the physics of complex systems

00:48:39.620 | is at least as badly understood

00:48:42.020 | as the physics of phase coherence in quantum mechanics.

00:48:45.660 | - Can we go there for a second?

00:48:48.500 | You've talked about attractor networks,

00:48:50.860 | and just maybe you could say what are attractor networks,

00:48:54.820 | and more broadly, what are interesting network dynamics

00:48:58.580 | that emerge in these or other complex systems?

00:49:03.000 | - You have to be willing to think

00:49:06.340 | in a huge number of dimensions,

00:49:08.740 | 'cause in a huge number of dimensions,

00:49:11.000 | the behavior of a system can be thought of

00:49:12.980 | as just the motion of a point over time

00:49:15.940 | in this huge number of dimensions.

00:49:17.780 | - Right.

00:49:18.620 | - An attractor network is simply a network

00:49:22.100 | where there is a line,

00:49:24.900 | and other lines converge on it in time.

00:49:28.300 | That's the essence of an attractor network.

00:49:31.180 | That's how you--

00:49:32.020 | - In a highly dimensional space.

00:49:34.740 | - And the easiest way to get that

00:49:37.420 | is to do it in a high dimensional space,

00:49:40.780 | where some of the dimensions provide the dissipation,

00:49:44.940 | which means, which,

00:49:46.300 | look, I have a physical system,

00:49:49.220 | trajectories can't contract everywhere.

00:49:53.660 | They have to contract in some places and expand in others.

00:49:56.900 | There's a fundamental classical theorem

00:49:59.380 | of statistical mechanics,

00:50:00.820 | which goes under the name of Liouville's theorem,

00:50:04.580 | which says you can't contract everywhere.

00:50:07.900 | You have to, if you contract somewhere,

00:50:10.020 | you expand somewhere else.

00:50:12.660 | And it's an interesting physical systems.

00:50:15.220 | You get driven systems

00:50:17.420 | where you have a small subsystem,

00:50:19.220 | which is the interesting part,

00:50:21.660 | and the rest of the contraction and expansion,

00:50:24.100 | the physicists would say is entropy flow

00:50:25.940 | in this other part of the system.

00:50:27.580 | But basically, attractor networks are dynamics

00:50:35.460 | funneling down so you can't be any,

00:50:40.380 | so that if you start somewhere in the dynamical system,

00:50:42.500 | you will soon find yourself

00:50:44.140 | on a pretty well determined pathway, which goes somewhere.

00:50:47.140 | You start somewhere else,

00:50:48.100 | you'll wind up on a different pathway,

00:50:50.580 | but you don't have just all possible things.

00:50:53.100 | You have some defined pathways,

00:50:55.340 | which are allowed and under which you will converge.

00:50:58.860 | And that's the way you make a stable computer.

00:51:01.940 | And that's the way you make a stable behavior.

00:51:04.220 | - So in general, looking at the physics

00:51:08.740 | of the emergent stability in these networks,

00:51:13.740 | what are some interesting characteristics that,

00:51:18.120 | what are some interesting insights

00:51:20.940 | from studying the dynamics

00:51:22.340 | of such high dimensional systems?

00:51:24.940 | - Most dynamical systems,

00:51:26.940 | most driven dynamical systems,

00:51:29.820 | by driven they're coupled somehow to an energy source.

00:51:33.140 | And so if their dynamics keeps going

00:51:35.580 | because it's coupling to the energy source,

00:51:38.020 | most of them, it's very difficult to understand

00:51:42.420 | at all what the dynamical behavior is going to be.

00:51:46.540 | - You have to run it.

00:51:49.180 | - You have to run it.

00:51:50.580 | There's a subset of systems,

00:51:53.260 | which has what is actually known to the mathematicians

00:51:57.260 | as a Lyapunov function.

00:51:58.960 | And those systems,

00:52:01.980 | you can understand convergent dynamics

00:52:05.460 | by saying you're going downhill on something or other.

00:52:08.940 | And that's what I found

00:52:12.340 | with ever knowing what Lyapunov functions were

00:52:15.300 | in the simple model I made in the early 80s,

00:52:19.260 | was an energy function.

00:52:20.420 | So you could understand how you could get this channeling

00:52:23.180 | on the pathways without having to follow the dynamics

00:52:28.100 | in infinite detail.

00:52:30.060 | You started rolling a ball at the top of a mountain,

00:52:34.300 | it's going to wind up at the bottom of a valley.

00:52:36.500 | You know that's true

00:52:38.300 | without actually watching the ball roll down.

00:52:42.120 | - There are certain properties of the system

00:52:45.860 | that when you can know that.

00:52:48.420 | - That's right.

00:52:49.460 | And not all systems behave that way.

00:52:51.640 | - Most don't probably.

00:52:55.260 | - Most don't, but it provides you with a metaphor

00:52:57.760 | for thinking about systems,

00:52:59.820 | which are stable and the good to have these attractors

00:53:03.180 | behave even if you can't find a Lyapunov function

00:53:07.420 | behind them or an energy function behind them.

00:53:09.900 | It gives you a metaphor for thought.

00:53:11.740 | - Speaking of thought,

00:53:17.220 | if I had a glint in my eye with excitement and said,

00:53:21.860 | you know, I'm really excited about this,

00:53:25.140 | something called deep learning and neural networks,

00:53:28.460 | and I would like to create an intelligent system

00:53:32.460 | and came to you as an advisor,

00:53:36.220 | what would you recommend?

00:53:37.820 | Is it a hopeless pursuit to use neural networks

00:53:42.840 | to achieve thought?

00:53:44.060 | Is it, what kind of mechanisms should we explore?

00:53:48.740 | What kind of ideas should we explore?

00:53:50.600 | - Well, you look at the simple networks,

00:53:56.860 | one-pass networks.

00:54:01.340 | They don't support multiple hypotheses very well.

00:54:04.780 | As I have tried to work with very simple systems,

00:54:09.940 | which do something which you might consider to be thinking,

00:54:12.940 | thought has to do with the ability to do mental exploration

00:54:17.660 | before you make it take a physical action.

00:54:20.040 | - Almost a, like we were mentioning, playing chess,

00:54:25.500 | visualizing, simulating inside your head,

00:54:29.180 | different outcomes.

00:54:30.460 | - Yeah, yeah.

00:54:31.420 | And now you could do that in a feed-forward network

00:54:37.340 | because you've pre-calculated all kinds of things.

00:54:40.500 | But I think the way neurobiology does it,

00:54:44.020 | it hasn't pre-calculated everything.

00:54:47.800 | It actually has parts of a dynamical system

00:54:51.980 | in which you're doing exploration in a way which is.

00:54:59.800 | - There's a creative element.

00:55:01.740 | Like there's an.

00:55:02.580 | - There's a creative element.

00:55:04.700 | And in a simple-minded neural net,

00:55:09.700 | you have a constellation of instances

00:55:18.020 | from which you've learned.

00:55:21.360 | And if you are within that space,

00:55:27.220 | if a new question is a question within this space,

00:55:31.740 | you can actually rely on that system pretty well

00:55:37.520 | to come up with a good suggestion for what to do.

00:55:41.020 | If on the other hand,

00:55:42.000 | the query comes from outside the space,

00:55:45.140 | you have no way of knowing how the system's gonna behave.

00:55:49.280 | There are no limitations on what could happen.

00:55:51.580 | And so the artificial neural net world

00:55:55.280 | is always very much, I have a population of examples.

00:56:00.280 | The test set must be drawn from the equivalent population.

00:56:04.720 | If the test set has examples which are from a population

00:56:08.460 | which is completely different,

00:56:09.960 | there's no way that you could expect

00:56:13.560 | to get the answer right.

00:56:15.640 | - Yeah, what they call outside the distribution.

00:56:20.960 | - That's right, that's right.

00:56:22.160 | And so if you see a ball rolling across the street at dusk,

00:56:27.160 | if that wasn't in your training set,

00:56:33.280 | the idea that a child may be coming close behind that

00:56:37.040 | is not going to occur to the neural net.

00:56:39.040 | - And it is to our,

00:56:42.480 | there's something in the neurobiology that allows that.

00:56:45.600 | - Yeah, there's something in the way

00:56:47.600 | of what it means to be outside of the population

00:56:52.320 | of the training set.

00:56:53.600 | The population of the training set

00:56:55.560 | isn't just sort of this set of examples.

00:56:57.920 | There's more to it than that.

00:57:03.640 | And it gets back to my question of

00:57:06.520 | what is it to understand something?

00:57:09.200 | - Yeah.

00:57:10.040 | You know, in a small tangent,

00:57:14.720 | you've talked about the value of thinking,

00:57:16.940 | of deductive reasoning in science

00:57:18.640 | versus large data collection.

00:57:20.760 | So sort of thinking about the problem.

00:57:25.280 | I suppose it's the physics side of you

00:57:27.440 | of going back to first principles and thinking,

00:57:31.080 | but what do you think is the value of deductive reasoning

00:57:33.640 | in the scientific process?

00:57:35.460 | - Well, look, there are obviously scientific questions

00:57:39.800 | in which the route to the answer to it

00:57:42.960 | comes through the analysis of one hell of a lot of data.

00:57:46.520 | - Right.

00:57:47.360 | Cosmology, that kind of stuff.

00:57:50.500 | - And that's never been the kind of problem

00:57:53.360 | in which I've had any particular insight.

00:57:58.520 | Though I must say, if you look at,

00:58:00.220 | cosmology is one of those.

00:58:04.160 | If you look at the actual things that Jim Peebles,

00:58:06.760 | one of this year's Nobel Prize in physics,

00:58:10.160 | one from the local physics department,

00:58:12.280 | the kinds of things he's done,

00:58:14.720 | he's never crunched large data.

00:58:17.000 | Never, never, never.

00:58:18.200 | He's used the encapsulation

00:58:21.520 | of the work of others in this regard.

00:58:24.760 | - But ultimately boiled down to thinking through the problem.

00:58:31.720 | Like what are the principles

00:58:32.920 | under which a particular phenomenon operates?

00:58:35.880 | - Yeah, yeah.

00:58:37.280 | And look, physics is always going to look for ways

00:58:40.160 | in which you can describe the system

00:58:42.680 | in a way which rises above the details.

00:58:47.560 | And to the hard-dyed-in-the-wool biologist,

00:58:52.560 | biology works because of the details.

00:58:56.800 | In physics, to the physicists,

00:58:58.760 | we want an explanation which is right

00:59:01.240 | in spite of the details.

00:59:03.080 | And there will be questions

00:59:04.200 | which we cannot answer as physicists

00:59:06.720 | because the answer cannot be found that way.

00:59:10.060 | (Peebles sniffs)

00:59:13.100 | - There's, I'm not sure if you're familiar

00:59:15.220 | with the entire field of brain-computer interfaces

00:59:19.140 | that's become more and more intensely researched

00:59:24.020 | and developed recently,

00:59:24.980 | especially with companies like Neuralink with Elon Musk.

00:59:28.180 | - Yeah, I know there have always been the interest

00:59:31.100 | both in things like getting the eyes

00:59:35.700 | to be able to control things

00:59:38.300 | or getting the thought patterns

00:59:40.780 | to be able to move what had been a connected limb

00:59:45.060 | which is now connected through a computer.

00:59:48.020 | - That's right.

00:59:48.900 | So in the case of Neuralink,

00:59:51.300 | they're doing a thousand-plus connections

00:59:54.600 | where they're able to do two-way,

00:59:56.660 | activate and read spikes, neural spikes.

01:00:01.460 | Do you have hope for that kind of computer-brain interaction

01:00:06.180 | in the near or maybe even far future

01:00:08.440 | of being able to expand the ability of the mind of cognition

01:00:14.840 | or understand the mind?

01:00:18.920 | - It's interesting watching things go.

01:00:23.780 | When I first became interested in neurobiology,

01:00:27.060 | most of the practitioners thought

01:00:28.740 | you would be able to understand neurobiology

01:00:31.620 | by techniques which allowed you

01:00:33.220 | to record only one cell at a time.

01:00:36.660 | - One cell, yeah.

01:00:38.620 | - People like David Hubble

01:00:43.340 | very strongly reflected that point of view.

01:00:45.780 | And that's been taken over by a generation,

01:00:49.700 | a couple of generations later,

01:00:52.420 | by a set of people who says,

01:00:54.420 | "Not until we can record from 10 to the four

01:00:57.020 | "or 10 to the five at a time

01:00:59.220 | "will we actually be able to understand

01:01:00.760 | "how the brain actually works."

01:01:03.360 | And in a general sense, I think that's right.

01:01:08.360 | You have to look, you have to begin to be able to look

01:01:11.600 | for the collective modes,

01:01:16.280 | collective operations of things.

01:01:18.400 | It doesn't rely on this action potential of that cell.

01:01:21.240 | It relies on the collective properties

01:01:23.000 | of this set of cells connected

01:01:24.880 | with this kind of patterns and so on.

01:01:26.720 | And you're not going to succeed

01:01:29.480 | in seeing what those collective activities are

01:01:31.840 | without recording many cells at once.

01:01:34.360 | - The question is how many at once?

01:01:40.160 | What's the threshold?

01:01:41.480 | And that's the--

01:01:42.920 | - Yeah, and look, it's being pursued hard

01:01:47.200 | in the motor cortex.

01:01:48.280 | The motor cortex does something which is complex,

01:01:53.280 | and yet the problem you're trying to address

01:01:55.600 | is fairly simple.

01:02:00.160 | Neurobiology does it in ways that are different

01:02:02.680 | from the way an engineer would do it.

01:02:04.360 | An engineer would put in six highly accurate

01:02:09.360 | stepping motors controlling a limb

01:02:11.480 | rather than 100,000 muscle fibers,

01:02:15.040 | each of which has to be individually controlled.

01:02:17.440 | And so understanding how to do things

01:02:22.320 | in a way which is much more forgiving

01:02:24.680 | and much more neural, I think,

01:02:27.120 | would benefit the engineering world.

01:02:30.720 | The engineering world, ah, touch.

01:02:36.040 | Let's put in a pressure sensor or two,

01:02:37.960 | rather than an array of a gazillion pressure sensors,

01:02:42.800 | none of which are accurate,

01:02:44.120 | all of which are perpetually recalibrating themselves.

01:02:47.500 | - So you're saying your hope is,

01:02:50.880 | your advice for the engineers of the future

01:02:53.600 | is to embrace the large chaos of a messy,

01:02:57.800 | error-prone system like those of the biological systems.

01:03:03.520 | Like that's probably the way to solve some of these.

01:03:05.880 | - I think you'll be able to make better

01:03:09.520 | computations slash robotics that way

01:03:13.280 | than by trying to force things into a robotics

01:03:18.280 | where joint motors are powerful

01:03:22.720 | and stepping motors are accurate.

01:03:25.400 | - But then the physicists,

01:03:26.960 | the physicists in you will be lost forever in such systems

01:03:31.320 | 'cause there's no simple fundamentals to explore

01:03:33.760 | in systems that are so large and messy.

01:03:38.760 | - Well, you say that, and yet there's a lot of physics,

01:03:43.840 | the Navier-Stokes equations,

01:03:45.440 | the equations of nonlinear hydrodynamics,

01:03:49.840 | huge amount of physics in them.

01:03:51.500 | All the physics of atoms and molecules has been lost,

01:03:55.560 | but it's been replaced by this other set of equations,

01:03:58.320 | which is just as true as the equations at the bottom.

01:04:01.440 | Now those equations are going to be harder to find

01:04:06.440 | in general biology, but the physicist in me says

01:04:10.920 | there are probably some equations of that sort.

01:04:13.480 | - They're out there.

01:04:14.320 | - They're out there,

01:04:16.560 | and if physics is going to contribute to anything,

01:04:19.440 | it may contribute to trying to find out

01:04:22.140 | what those equations are

01:04:23.340 | and how to capture them from the biology.

01:04:25.580 | - Would you say that's one of the main open problems

01:04:29.780 | of our age is to discover those equations?

01:04:34.300 | - Yeah, if you look at, there's molecules

01:04:38.780 | and there's psychological behavior,

01:04:40.980 | and these two are somehow related.

01:04:45.660 | They're layers of detail, they're layers of collectiveness,

01:04:50.660 | and to capture that in some vague way,

01:04:54.720 | several stages on the way up to see how these things

01:05:01.360 | can actually be linked together.

01:05:04.040 | - So it seems in our universe,

01:05:06.120 | there's a lot of elegant equations

01:05:09.400 | that can describe the fundamental way that things behave,

01:05:12.520 | which is a surprise.

01:05:13.820 | It's compressible into equations.

01:05:15.800 | It's simple and beautiful,

01:05:17.260 | but it's still an open question whether that link

01:05:22.140 | is equally between molecules and the brain

01:05:27.140 | is equally compressible into elegant equations.

01:05:31.080 | But your sense, you're both a physicist and a dreamer.

01:05:37.060 | You have a sense that--

01:05:38.420 | - Yeah, but I can only dream physics dreams.

01:05:42.100 | - You can only dream physics dreams.

01:05:44.220 | - There was an interesting book called "Einstein's Dreams,"

01:05:46.820 | which alternates between chapters on his life

01:05:51.820 | and descriptions of the way time might have been, but isn't.

01:05:57.220 | The linking between these being, of course,

01:06:03.700 | ideas that Einstein might have had to think about

01:06:06.420 | the essence of time as he was thinking about time.

01:06:11.260 | So speaking of the essence of time and your biology,

01:06:14.740 | you're one human, famous impactful human,

01:06:18.660 | but just one human with a brain living the human condition,

01:06:22.620 | but you're ultimately mortal, just like all of us.

01:06:27.600 | Has studying the mind as a mechanism

01:06:30.540 | changed the way you think about your own mortality?

01:06:38.620 | - It has really, because particularly as you get older

01:06:41.900 | and the body comes apart in various ways,

01:06:44.920 | I became much more aware of the fact that

01:06:52.060 | what is somebody is contained in the brain

01:06:57.100 | and not in the body that you worry about burying.

01:07:01.540 | And it is to a certain extent true

01:07:07.900 | that for people who write things down,

01:07:10.580 | equations, dreams, notepads, diaries,

01:07:15.580 | fractions of their thought does continue to live

01:07:21.380 | after they're dead and gone,

01:07:24.020 | after their body is dead and gone.

01:07:25.760 | And there's a sea change in that going on in my lifetime

01:07:32.300 | between when my father died,

01:07:35.860 | when except for the things which were actually written

01:07:38.540 | by him as it were,

01:07:39.840 | very few facts about him will have ever been recorded.

01:07:44.220 | And the number of facts which are recorded

01:07:46.020 | about each and every one of us forever now,

01:07:50.060 | as far as I can see in the digital world.

01:07:54.300 | And so the whole question of what is death

01:07:56.820 | may be different for people a generation ago

01:08:04.060 | than a generation further ahead.

01:08:07.220 | - Maybe we have become immortal under some definitions.

01:08:10.940 | - Yeah, yeah.

01:08:12.300 | - Last easy question, what is the meaning of life?

01:08:19.580 | Looking back, you've studied the mind,

01:08:27.840 | us weird descendants of apes,

01:08:32.420 | what's the meaning of our existence on this little earth?

01:08:36.400 | - Oh, that word meaning is as slippery

01:08:41.540 | as the word understand.

01:08:43.940 | - Interconnected somehow perhaps.

01:08:48.120 | Is there, it's slippery, but is there something

01:08:55.300 | that you, despite being slippery,

01:08:58.300 | can hold long enough to express?

01:09:00.940 | - Well, I've been amazed at how hard it is

01:09:05.580 | to define the things in a living system

01:09:12.180 | in the sense that one hydrogen atom

01:09:17.380 | is pretty much like another,

01:09:19.380 | but one bacterium is not so much like another bacterium,

01:09:24.140 | even of the same nominal species.

01:09:26.100 | In fact, the whole notion of what is the species

01:09:28.820 | gets a little bit fuzzy.

01:09:30.280 | And the species exists in the absence

01:09:33.540 | of certain classes of environments.

01:09:36.100 | And pretty soon one winds up with a biology

01:09:40.180 | which the whole thing is living,

01:09:43.380 | but whether there's actually any element of it,

01:09:46.020 | which by itself would be said to be living,

01:09:49.660 | it becomes a little bit vague in my mind.

01:09:54.180 | - So in a sense, the idea of meaning

01:09:58.180 | is something that's possessed by an individual,

01:10:01.180 | like a conscious creature.

01:10:03.060 | And you're saying that it's all interconnected

01:10:07.380 | in some kind of way that there might not even

01:10:09.620 | be an individual, or all kind of this complicated mess

01:10:14.060 | of biological systems at all different levels

01:10:17.380 | where the human starts and when the human ends is unclear.

01:10:20.580 | - Yeah, yeah, and we're in neurobiology

01:10:24.300 | where the, oh, you say the neocortex is at the thinking,

01:10:27.860 | but there's lots of things that are done

01:10:29.340 | in the spinal cord.

01:10:31.260 | And so we say, what is the essence of thought?

01:10:35.700 | Is it just gonna be neocortex?

01:10:37.740 | Can't be, can't be.

01:10:39.360 | - Yeah, maybe to understand and to build thought,

01:10:43.460 | you have to build the universe along with the neocortex.

01:10:47.340 | It's all interlinked through the spinal cord.

01:10:51.380 | John, it's a huge honor talking today.

01:10:54.340 | Thank you so much for your time.

01:10:55.820 | I really appreciate it.

01:10:57.100 | - Well, thank you for the challenge of talking with you.

01:10:59.060 | And it'll be interesting to see whether you can win

01:11:01.060 | five minutes out of this with just coherent sense

01:11:04.580 | to anyone or not.

01:11:06.780 | - Beautiful.

01:11:08.300 | Thanks for listening to this conversation

01:11:09.900 | with John Hopfield, and thank you

01:11:12.020 | to our presenting sponsor, Cash App.

01:11:14.340 | Download it, use code LEXPODCAST, you'll get $10,

01:11:18.300 | and $10 will go to FIRST, an organization

01:11:20.900 | that inspires and educates young minds

01:11:23.140 | to become science and technology innovators of tomorrow.

01:11:26.300 | If you enjoy this podcast, subscribe on YouTube,

01:11:29.060 | get five stars on Apple Podcasts, support on Patreon,

01:11:32.300 | or simply connect with me on Twitter @LexFriedman.

01:11:35.900 | And now let me leave you with some words of wisdom

01:11:39.380 | from John Hopfield in his article titled, "Now What?"

01:11:43.220 | Choosing problems is the primary determinant

01:11:47.260 | of what one accomplishes in science.

01:11:49.980 | I have generally had a relatively short attention span

01:11:52.940 | in science problems.

01:11:54.440 | Thus, I have always been on the lookout

01:11:56.860 | for more interesting questions,

01:11:58.580 | either as my present ones get worked out,

01:12:00.860 | or as they get classified by me as intractable,

01:12:04.280 | given my particular talents.

01:12:06.020 | He then goes on to say, "What I have done in science

01:12:10.780 | "relies entirely on experimental and theoretical studies

01:12:14.220 | "by experts.

01:12:15.660 | "I have a great respect for them,

01:12:17.520 | "especially for those who are willing to attempt

01:12:19.980 | "communication with someone

01:12:21.760 | "who is not an expert in the field."

01:12:24.740 | I would only add that experts are good

01:12:27.100 | at answering questions.

01:12:29.020 | If you're brash enough, ask your own.

01:12:32.660 | Don't worry too much about how you found them.

01:12:34.960 | Thank you for listening, and hope to see you next time.

01:12:39.380 | (upbeat music)

01:12:41.960 | (upbeat music)

01:12:44.540 | [BLANK_AUDIO]

John Hopfield: Physics View of the Mind and Neurobiology | Lex Fridman Podcast #76

Chapters