Biological versus Artificial Neural Networks (John Hopfield)

00:00:00.000 | - What difference between biological neural networks

00:00:04.440 | and artificial neural networks

00:00:06.560 | is most captivating and profound to you?

00:00:08.860 | At the higher philosophical level,

00:00:13.640 | let's not get technical just yet.

00:00:15.760 | - One of the things that very much intrigues me

00:00:19.880 | is the fact that neurons have all kinds of components,

00:00:24.880 | properties to them.

00:00:28.780 | In evolutionary biology, if you have some little quirk

00:00:33.780 | in how a molecule works or how a cell works,

00:00:38.560 | and it can be made use of,

00:00:40.060 | evolution will sharpen it up

00:00:41.660 | and make it into a useful feature rather than a glitch.

00:00:46.660 | And so you expect in neurobiology

00:00:50.680 | for evolution to have captured all kinds of possibilities

00:00:54.500 | of getting neurons,

00:00:55.620 | of how you get neurons to do things for you.

00:00:58.380 | And that aspect has been completely suppressed

00:01:03.540 | in artificial neural networks.

00:01:05.240 | - Do the glitches become features

00:01:09.740 | in the biological neural network?

00:01:13.300 | - They can.

00:01:14.660 | Look, let me take one of the things

00:01:16.820 | that I used to do research on.

00:01:18.820 | If you take things which oscillate,

00:01:25.140 | they have rhythms which are sort of close to each other.

00:01:29.300 | Under some circumstances,

00:01:30.540 | these things will have a phase transition

00:01:33.500 | and suddenly the rhythm will,

00:01:34.860 | everybody will fall into step.

00:01:37.260 | There was a marvelous physical example of that

00:01:39.740 | in the Millennium Bridge across the Thames River

00:01:43.780 | about, built about 2001.

00:01:47.580 | And pedestrians walking across,

00:01:50.100 | pedestrians don't walk, synchronize,

00:01:52.580 | they don't walk in lockstep.

00:01:54.900 | But they all walk at about the same frequency.

00:01:58.060 | And the bridge could sway at that frequency

00:02:00.340 | and the slight sway made pedestrians

00:02:02.500 | tend a little bit to lock into step.

00:02:04.540 | And after a while, the bridge was oscillating back and forth

00:02:08.220 | and the pedestrians were walking in step to it.

00:02:10.500 | And you could see it in the movies made out of the bridge.

00:02:13.460 | And the engineers made a simple-minded mistake.

00:02:17.140 | They assumed when you walk, it's step, step, step,

00:02:20.180 | and it's back and forth motion.

00:02:22.820 | But when you walk, it's also right foot, left foot,

00:02:25.500 | side to side motion.

00:02:26.860 | And it's the side to side motion

00:02:28.300 | for which the bridge was strong enough,

00:02:31.060 | but it wasn't stiff enough.

00:02:34.780 | And as a result, you would feel the motion

00:02:37.620 | and you'd fall into step with it.

00:02:39.460 | And people were very uncomfortable with it.

00:02:41.660 | They closed the bridge for two years

00:02:43.100 | while they built stiffening for it.

00:02:45.420 | Now, nerves, look, nerve cells produce action potentials.

00:02:50.420 | You have a bunch of cells

00:02:51.380 | which are loosely coupled together

00:02:52.740 | producing action potentials of the same rate.

00:02:55.940 | There'll be some circumstances

00:02:58.340 | under which these things can lock together.

00:03:00.660 | Other circumstances in which they won't.

00:03:04.580 | Well, they fire together,

00:03:07.300 | you can be sure that other cells are gonna notice it.

00:03:09.980 | So you can make a computational feature out of this

00:03:12.580 | in an evolving brain.

00:03:15.460 | Most artificial neural networks

00:03:18.380 | don't even have action potentials,

00:03:20.180 | let alone have the possibility for synchronizing them.

00:03:23.540 | - And you mentioned the evolutionary process.

00:03:28.260 | So the evolutionary process

00:03:30.940 | that builds on top of biological systems

00:03:34.620 | leverages that the weird mess of it somehow.

00:03:39.620 | So how do you make sense of that ability

00:03:44.900 | to leverage all the different kinds of complexities

00:03:48.620 | in the biological brain?

00:03:51.220 | - Well, look, at the biological molecule level,

00:03:55.380 | you have a piece of DNA

00:03:58.260 | which encodes for a particular protein.

00:04:02.020 | You could duplicate that piece of DNA

00:04:04.500 | and now one part of it can code for that protein,

00:04:08.340 | but the other one could itself change a little bit

00:04:11.700 | and thus start coding for a molecule

00:04:13.380 | which is slightly different.

00:04:15.140 | Now, if that molecule was just slightly different,

00:04:17.940 | had a function which helped any old chemical reaction

00:04:22.940 | was as important to the cell,

00:04:24.700 | it would go ahead and let that try

00:04:30.060 | and evolution would slowly improve that function.

00:04:33.140 | And so you have the possibility of duplicating

00:04:36.860 | and then having things drift apart.

00:04:41.300 | One of them retain the old function,

00:04:43.300 | the other one do something new for you.

00:04:46.860 | And there's evolutionary pressure to improve.

00:04:50.540 | Look, there is in computers too,

00:04:52.020 | but it's improvement has to do with closing some companies

00:04:55.260 | and opening some others.

00:04:56.900 | The evolutionary process looks a little different.

00:05:00.660 | - Yeah, similar time scale perhaps.

00:05:04.060 | - Much shorter in time scale.

00:05:06.180 | - Companies close, yeah, go bankrupt and are born.

00:05:09.180 | Yeah, shorter, but not much shorter.

00:05:12.460 | Some company lasts a century,

00:05:15.740 | but yeah, you're right.

00:05:16.900 | I mean, if you think of companies as a single organism

00:05:19.740 | that builds and you all know, yeah,

00:05:22.100 | it's a fascinating dual correspondence there

00:05:27.100 | between biological-

00:05:28.900 | - And companies have difficulty having a new product

00:05:32.580 | competing with an old product.

00:05:34.260 | When IBM built this first PC, you probably read the book,

00:05:41.540 | they made a little isolated internal unit

00:05:44.540 | to make the PC.

00:05:45.500 | And for the first time in IBM's history,

00:05:49.300 | they didn't insist that you build it out of IBM components,

00:05:52.520 | but they understood that they could get into this market,

00:05:57.660 | which is a very different thing

00:05:59.820 | by completely changing their culture.

00:06:02.320 | And biology finds other markets in a more adaptive way.

00:06:09.700 | - Yeah, it's better at it.

00:06:13.860 | It's better at that kind of integration.

00:06:15.860 | So maybe you've already said it,

00:06:19.260 | but what to use the most beautiful aspect

00:06:22.440 | or mechanism of the human mind?

00:06:24.800 | Is it the adaptive, the ability to adapt as you've described

00:06:32.740 | or is there some other little quirk

00:06:34.220 | that you particularly like?

00:06:35.840 | - Adaptation is everything when you get down to it,

00:06:43.280 | but the difference, there are differences between adaptation

00:06:48.280 | where you're learning goes on only over generations

00:06:51.820 | and over evolutionary time,

00:06:54.860 | where you're learning goes on at the timescale

00:06:57.260 | of one individual who must learn from the environment

00:07:00.900 | during that individual's lifetime.

00:07:03.420 | And biology has both kinds of learning in it.

00:07:10.000 | And the thing which makes neurobiology hard

00:07:14.200 | is that a mathematical system, as it were,

00:07:19.200 | built on this other kind of evolutionary system.

00:07:23.060 | - What do you mean by mathematical system?

00:07:27.240 | Where's the math and the biology?

00:07:30.160 | - Well, when you talk to a computer scientist

00:07:31.880 | about neural networks, it's all math.

00:07:34.720 | The fact that biology actually came about from evolution,

00:07:39.240 | the thing that, and the fact that biology is about a system

00:07:44.240 | which you can build in three dimensions.

00:07:49.300 | If you look at computer chips,

00:07:53.500 | computer chips are basically two-dimensional structures,

00:07:58.100 | maybe 2.1 dimensions,

00:08:00.740 | but they really have difficulty

00:08:02.780 | doing three-dimensional wiring.

00:08:05.780 | Biology is, the neocortex is actually also sheet-like,

00:08:10.780 | and it sits on top of the white matter,

00:08:14.240 | which is about 10 times the volume of the gray matter

00:08:16.960 | and contains all what you might call the wires.

00:08:20.100 | But there's a huge,

00:08:23.660 | the effect of computer structure on what is easy

00:08:29.720 | and what is hard is immense.

00:08:34.840 | So-- - And biology does,

00:08:37.820 | it makes some things easy that are very difficult

00:08:42.160 | to understand how to do computationally.

00:08:44.480 | On the other hand,

00:08:45.800 | you can't do simple floating-point arithmetic,

00:08:48.200 | 'cause it's awfully stupid.

00:08:49.720 | - Yeah, and you're saying this kind of three-dimensional,

00:08:52.240 | complicated structure makes, it's still math.

00:08:57.240 | It's still doing math.

00:08:58.920 | The kind of math it's doing enables you to solve problems

00:09:02.740 | of a very different kind.

00:09:04.880 | - That's right, that's right.

00:09:06.360 | - So you mentioned two kinds of adaptation,

00:09:10.320 | the evolutionary adaptation and the adaptation,

00:09:13.440 | or learning at the scale of a single human life.

00:09:16.600 | Which do you,

00:09:17.940 | which is particularly beautiful to you and interesting

00:09:24.480 | from a research and from just a human perspective?

00:09:27.800 | And which is more powerful?

00:09:32.440 | - I find things most interesting that I begin to see

00:09:35.320 | how to get into the edges of them

00:09:39.240 | and tease them apart a little bit and see how they work.

00:09:42.040 | And since I can't see the evolutionary process going on,

00:09:47.740 | I'm in awe of it.

00:09:51.120 | But I find it just a black hole

00:09:55.920 | as far as trying to understand what to do.

00:09:58.560 | And so in a certain sense, I'm in awe of it,

00:10:01.800 | but I couldn't be interested in working on it.

00:10:04.140 | - The human life's time scale is however thing

00:10:10.440 | you can tease apart and study.

00:10:14.320 | - Yeah, you can do, there's the developmental neurobiology

00:10:17.720 | which understands how the connections

00:10:20.840 | and how the structure evolves

00:10:25.240 | from a combination of what the genetics is like

00:10:29.600 | and the real, the fact that you're building a system

00:10:33.120 | in three dimensions.

00:10:34.780 | - In just days and months,

00:10:38.720 | those early days of a human life are really interesting.

00:10:42.740 | - They are and of course,

00:10:45.760 | there are times of immense cell multiplication.

00:10:50.820 | There are also times of the greatest cell death

00:10:54.680 | in the brain is during infancy.

00:10:58.680 | It's turnover.

00:10:59.680 | - So what is not effective,

00:11:04.680 | what is not wired well enough to use the moment,

00:11:07.680 | throw it out.

00:11:09.380 | - It's a mysterious process.

00:11:11.560 | From, let me ask, from what field

00:11:14.480 | do you think the biggest breakthroughs

00:11:17.080 | in understanding the mind will come in the next decades?

00:11:21.080 | Is it neuroscience, computer science,

00:11:25.080 | neurobiology, psychology, physics,

00:11:29.480 | maybe math, maybe literature?

00:11:32.840 | - Well of course I see the world

00:11:37.120 | always through a lens of physics.

00:11:39.060 | I grew up in physics

00:11:40.400 | and the way I pick problems

00:11:45.560 | is very characteristic of physics

00:11:48.320 | and of an intellectual background

00:11:50.520 | which is not psychology,

00:11:51.880 | which is not chemistry and so on and so on.

00:11:54.840 | - Yeah, both of your parents were physicists.

00:11:56.760 | - Both of my parents were physicists

00:11:58.480 | and the real thing I got out of that

00:12:01.680 | was a feeling that the world is an understandable place

00:12:06.680 | and if you do enough experiments

00:12:10.200 | and think about what they mean

00:12:12.880 | and structure things so you can do the mathematics

00:12:16.160 | of the relevant to the experiments,

00:12:19.000 | you also be able to understand how things work.

00:12:22.560 | But that was a few years ago.

00:12:25.440 | Did you change your mind at all

00:12:27.600 | through many decades of trying to understand the mind,

00:12:32.560 | of studying it different kinds of ways,

00:12:34.120 | not even the mind, just biological systems?

00:12:36.420 | You still have hope that physics,

00:12:39.280 | that you can understand?

00:12:40.540 | - There's a question of what do you mean by understand?

00:12:46.560 | - Of course.

00:12:48.280 | - When I taught freshman physics,

00:12:49.480 | I used to say I wanted to give physics

00:12:52.000 | to understand the subject,

00:12:53.160 | to understand Newton's laws.

00:12:55.480 | I didn't want them simply to memorize a set of examples

00:13:00.400 | to which they knew the equations to write down

00:13:03.240 | to generate the answers.

00:13:04.760 | I had this nebulous idea of understanding

00:13:08.840 | so that if you looked at a situation,

00:13:11.000 | you could say, "Oh, I expect the ball

00:13:13.680 | to make that trajectory

00:13:14.920 | or I expect some intuitive notion of understanding."

00:13:19.160 | And I don't know how to express that very well.

00:13:24.160 | I've never known how to express it well.

00:13:27.800 | And you run smack up against it

00:13:30.560 | when you look at these simple neural nets,

00:13:34.440 | feedforward neural nets,

00:13:36.720 | which do amazing things,

00:13:39.980 | and yet you know contain nothing of the essence

00:13:43.120 | of what I would have felt was understanding.

00:13:46.760 | Understanding is more than just an enormous lookup table.

00:13:49.620 | - Let's linger on that.

00:13:53.000 | How sure you are of that?

00:13:54.720 | What if the table gets really big?

00:13:56.900 | So, I mean, ask another way.

00:14:01.120 | These feedforward neural networks,

00:14:03.720 | do you think they'll ever understand?

00:14:05.560 | - Could answer that in two ways.

00:14:08.440 | I think if you look at real systems,

00:14:13.240 | feedback is an essential aspect

00:14:16.840 | of how these real systems compute.

00:14:19.560 | On the other hand,

00:14:20.400 | if I have a mathematical system with feedback,

00:14:23.200 | I know I can unlayer this and do it.

00:14:25.440 | But I have an exponential expansion

00:14:30.480 | in the amount of stuff I have to build

00:14:32.680 | if I can solve the problem that way.

00:14:34.480 | - So feedback is essential.

00:14:36.560 | So we can talk even about recurrent neural net,

00:14:39.680 | so recurrence.

00:14:41.480 | But do you think all the pieces are there

00:14:44.160 | to achieve understanding through these simple mechanisms?

00:14:48.860 | Like, back to our original question,

00:14:51.800 | what is the fundamental,

00:14:53.600 | is there a fundamental difference

00:14:55.040 | between artificial neural networks and biological?

00:14:58.500 | Or is it just a bunch of surface stuff?

00:15:01.240 | - Suppose you ask a neurosurgeon,

00:15:03.100 | when is somebody dead?

00:15:06.100 | - Yeah.

00:15:09.000 | - They'll probably go back to saying,

00:15:10.720 | well, I can look at the brain rhythms

00:15:13.320 | and tell you this is a brain

00:15:15.840 | which is never gonna function again.

00:15:17.920 | This one is, this other one is one

00:15:19.920 | which if we treat it well, is still recoverable.

00:15:24.800 | And then just do that by some electrodes

00:15:27.240 | looking at simple electrical patterns

00:15:31.160 | which don't look in any detail at all

00:15:34.680 | at what individual neurons are doing.

00:15:40.000 | These rhythms are utterly absent

00:15:44.240 | from anything which goes on at Google.

00:15:46.320 | - Yeah, but the rhythms--

00:15:51.720 | - But the rhythms what?

00:15:54.480 | - So, well, that's like comparing,

00:15:56.720 | okay, I'll tell you.

00:15:57.880 | It's like you're comparing

00:15:59.520 | the greatest classical musician in the world

00:16:05.120 | to a child first learning to play.

00:16:07.720 | The question I'm at,

00:16:08.600 | but they're still both playing the piano.

00:16:11.200 | I'm asking, is there, will it ever go on at Google?

00:16:15.380 | Do you have a hope?

00:16:17.640 | Because you're one of the seminal figures

00:16:20.240 | in both launching both disciplines,

00:16:23.320 | both sides of the river.

00:16:25.460 | - I think it's going to go on generation after generation

00:16:32.160 | the way it has where what you might call

00:16:35.800 | the AI computer science community says,

00:16:38.440 | let's take the following.

00:16:40.640 | This is our model of neurobiology at the moment.

00:16:43.520 | Let's pretend it's good enough

00:16:47.080 | and do everything we can with it.

00:16:49.020 | And it does interesting things.

00:16:52.520 | And after the while, it sort of grinds into the sand

00:16:56.960 | and you say, ah, something else is needed for neurobiology

00:17:01.680 | and some other grand thing comes in.

00:17:04.960 | And enable you to go a lot further.

00:17:07.680 | But we'll go into the sand again.

00:17:10.880 | And I think it could be generations of this evolution.

00:17:13.960 | I don't know how many of them.

00:17:15.440 | And each one is going to get you further

00:17:17.480 | into what our brain does.

00:17:19.440 | In some sense, past the Turing test,

00:17:25.620 | longer and more broad aspects.

00:17:28.640 | And how many of these are good there

00:17:33.720 | are going to have to be before you say,

00:17:35.680 | I've made something, I've made a human.

00:17:39.480 | I don't know.

00:17:41.960 | - But your sense is it might be a couple.

00:17:44.080 | - My sense is it might be a couple more.

00:17:46.220 | - Yeah.

00:17:47.360 | - And going back to my brain waves as it were.

00:17:52.360 | From the AI point of view,

00:17:58.360 | they would say, ah, maybe these are an epiphenomenon

00:18:02.680 | and not important at all.

00:18:04.280 | The first car I had, a real wreck of a 1936 Dodge,

00:18:11.720 | go above 45 miles an hour and the wheels would shimmy.

00:18:16.620 | - Yeah.

00:18:18.400 | - Good speedometer that.

00:18:21.360 | Now, nobody designed the car that way.

00:18:26.400 | The car is malfunctioning to have that.

00:18:28.680 | But in biology, if it were useful to know

00:18:32.480 | when are you going more than 45 miles an hour,

00:18:35.080 | you just capture that

00:18:36.720 | and you wouldn't worry about where it came from.

00:18:39.120 | - Yeah.

00:18:43.080 | - It's going to be a long time before that kind of thing,

00:18:45.620 | which can take place in large complex networks of things

00:18:50.620 | is actually used in the computation.

00:18:54.360 | Look, how many transistors

00:18:58.520 | are there in your laptop these days?

00:19:01.520 | - Actually, I don't know the number.

00:19:03.040 | It's-

00:19:03.880 | - It's on the scale of 10 to the 10.

00:19:05.640 | I can't remember the number either.

00:19:07.320 | - Yeah.

00:19:08.160 | - And all the transistors are somewhat similar.

00:19:12.360 | And most physical systems with that many parts,

00:19:16.560 | all of which are similar, have collective properties.

00:19:20.840 | - Yes.

00:19:21.960 | - Sound waves in air, earthquakes,

00:19:24.320 | what have you have collective properties, weather.

00:19:29.240 | There are no collective properties used

00:19:32.000 | in artificial neural networks in AI.

00:19:34.780 | - Yeah, it's very-

00:19:38.920 | - If biology uses them,

00:19:41.060 | it's going to take us to more generations of things

00:19:43.760 | for people to actually dig in

00:19:45.680 | and see how they are used and what they mean.

00:19:48.080 | - See, you're very right.

00:19:51.960 | We might have to return several times to neurobiology

00:19:55.520 | and try to make our transistors more messy.

00:19:59.580 | - Yeah, yeah.

00:20:00.640 | At the same time, the simple ones

00:20:03.520 | will conquer big aspects.

00:20:07.920 | And I think one of the most biggest surprises to me was

00:20:14.240 | how well learning systems,

00:20:19.360 | which are manifestly non-biological,

00:20:21.680 | how important they can be actually

00:20:24.800 | and how important and how useful they can be in AI.

00:20:29.440 | (screen whooshes)

00:20:32.280 | (screen whooshes)

00:20:35.040 | (screen whooshes)

00:20:37.800 | (screen whooshes)

00:20:40.560 | (screen whooshes)

00:20:43.320 | (screen whooshes)

00:20:46.080 | [BLANK_AUDIO]

Biological versus Artificial Neural Networks (John Hopfield) | AI Podcast Clips

Chapters