back to indexGilbert Strang: Linear Algebra, Teaching, and MIT OpenCourseWare | Lex Fridman Podcast #52
Chapters
0:0 Intro
3:45 OpenCourseWare
7:50 The Four Subspaces
10:45 The Beauty of Linear Algebra
13:46 Linear Algebra vs Calculus
18:51 Visualization
20:23 Math is not hard
27:3 Math in Washington
29:33 Deep Learning
35:18 Limits of Deep Learning
37:38 Who is Gilbert Strang
39:11 Why does Linear Algebra win
41:58 Favorite matrix
43:43 Teaching and learning
47:58 Conclusion
00:00:00.000 |
The following is a conversation with Gilbert Strang. 00:00:10.600 |
His MIT OpenCourseWare lectures on linear algebra 00:00:19.960 |
There's something inspiring about the way he teaches. 00:00:22.760 |
There's at once calm, simple, and yet full of passion 00:00:33.000 |
and slowly realizing that the world of matrices, 00:00:35.880 |
of vector spaces, of determinants and eigenvalues, 00:00:39.360 |
of geometric transformations and matrix decompositions 00:00:47.920 |
From signals to images, from numerical optimization 00:00:53.640 |
computer graphics, and everywhere outside AI, 00:00:56.600 |
including, of course, a quantum mechanical study 00:01:08.840 |
support on Patreon, or simply connect with me on Twitter, 00:01:32.640 |
was mostly through tools that we built ourselves, 00:01:38.360 |
ZipRecruiter is a tool that's already available for you. 00:01:41.160 |
It seeks to make hiring simple, fast, and smart. 00:01:44.960 |
For example, Codable co-founder Gretchen Huebner 00:01:56.240 |
Gretchen found it easier to focus on the best candidates, 00:01:59.040 |
and finally hiring the perfect person for the role 00:02:12.640 |
as I did, for free at ZipRecruiter.com/lexpod. 00:02:26.840 |
I personally use Cash App to send money to friends, 00:02:29.600 |
but you can also use it to buy, sell, and deposit Bitcoin. 00:02:47.280 |
So you can buy $1 worth, no matter what the stock price is. 00:02:51.560 |
Brokerage services are provided by Cash App Investing, 00:03:02.760 |
that many of you may know and have benefited from, 00:03:05.720 |
called FIRST, best known for their FIRST Robotics 00:03:10.840 |
They educate and inspire hundreds of thousands of students 00:03:16.520 |
and have a perfect rating on Charity Navigator, 00:03:23.920 |
When you get Cash App from the App Store or Google Play 00:03:29.400 |
you'll get $10 and Cash App will also donate $10 to FIRST, 00:03:35.160 |
that I've personally seen inspire girls and boys 00:03:40.960 |
And now, here's my conversation with Gilbert Strang. 00:03:45.800 |
How does it feel to be one of the modern day rock stars 00:03:57.280 |
but it's true that the videos in linear algebra 00:04:09.840 |
And well, partly the importance of linear algebra, 00:04:14.400 |
which I'm sure you'll ask me and give me a chance to say 00:04:22.600 |
But also, it was a class that I taught a bunch of times, 00:04:26.480 |
so I kind of got it organized and enjoyed doing it. 00:04:34.200 |
So they're on OpenCourseWare and on YouTube and translated. 00:04:39.040 |
- But there's something about that chalkboard 00:04:55.440 |
But before going through the course at my university, 00:05:18.240 |
So how do you think the idea of putting lectures online, 00:05:23.240 |
what really MIT OpenCourseWare has innovated? 00:05:32.720 |
is the committee was appointed by the president, 00:05:36.720 |
President Vest at that time, a wonderful guy. 00:05:40.000 |
And the idea of the committee was to figure out 00:05:52.560 |
And then they didn't see a way, and after a weekend, 00:06:12.040 |
as a thing that creates a product, isn't knowledge, 00:06:16.880 |
- The, you know, the kind of educational knowledge 00:06:37.100 |
MIT is a kind of, it's known for being high level, 00:06:43.200 |
technical things, and this is the best way we can say, 00:07:04.880 |
People write to me and say, oh, you've got a sense of humor, 00:07:18.720 |
we gotta give the subject most of the credit. 00:07:22.840 |
It really has come forward in importance in these years. 00:07:27.840 |
- So let's talk about linear algebra a little bit, 00:07:39.100 |
So what's your favorite specific topic in linear algebra, 00:07:44.100 |
or even math in general, to give a lecture on, 00:07:46.620 |
to convey, to tell a story, to teach students? 00:07:56.800 |
but I'm kind of proud of the idea of the four subspaces, 00:08:19.640 |
so the matrix, maybe I should say the matrix-- 00:08:24.880 |
Well, so we have like a rectangle of numbers. 00:08:28.480 |
So it's got N columns, got a bunch of columns, 00:08:38.560 |
the columns and the rows, it's the same numbers, 00:08:50.080 |
and they're all different, the numbers are mixed up. 00:08:53.720 |
First space to think about is, take the columns, 00:08:57.760 |
so those are vectors, those are points in N dimensions. 00:09:05.680 |
or might imagine a vector as a arrow in space, 00:09:17.280 |
- You often think of, this is very interesting 00:09:20.960 |
in terms of linear algebra, in terms of a vector, 00:09:30.600 |
You think this arbitrary space, multi-dimensional space-- 00:09:39.080 |
- Yeah, that's right, in the lecture, I try to, 00:09:42.440 |
so if you think of two vectors in 10 dimensions, 00:09:46.760 |
I'll do this in class, and I'll readily admit 00:09:54.360 |
of a vector, of a arrow in 10-dimensional space, 00:09:58.260 |
but whatever, you can add one bunch of 10 numbers 00:10:14.080 |
- You know, 10 dimensions, there's this beautiful thing 00:10:20.240 |
and all these theories which are really fundamentally 00:10:22.840 |
derived through math, but are very difficult to visualize, 00:10:43.040 |
we can't visualize, how do you think about that difference? 00:10:46.000 |
- Well, probably, I'm not a very geometric person, 00:10:48.880 |
so I'm probably thinking in three dimensions, 00:11:39.920 |
that I call a vector space, space of vectors, 00:11:54.600 |
that's space number one, the column space of the matrix. 00:12:17.520 |
and I can't really draw them on a blackboard, 00:12:25.880 |
and me too, I wouldn't use anything else now. 00:12:31.160 |
- And then the other two spaces are perpendicular to those. 00:12:43.520 |
then perpendicular to that plane would be a line, 00:12:50.200 |
So we've got two, we've got a column space, a row space, 00:12:58.760 |
in a beautiful picture of a matrix, yeah, yeah. 00:13:03.760 |
It's sort of a fundamental, it's not a difficult idea. 00:13:06.720 |
It comes pretty early in 1806, and it's basic. 00:13:11.720 |
- So planes in these multidimensional spaces, 00:13:16.360 |
how difficult of an idea is that to come to, do you think? 00:13:26.800 |
but I don't know if it's intuitive for us to imagine, 00:13:34.800 |
- Well, I have to admit, calculus came earlier, 00:13:54.160 |
Calculus has got all the complications of calculus 00:13:57.160 |
come from the curves, the bending, the curved surfaces. 00:14:11.660 |
And calculus also comes first in high school classes, 00:14:20.880 |
it'll be calculus, and then I say, enough of it. 00:14:27.520 |
- Do you think linear algebra should come first? 00:14:30.920 |
- Well, it really, I'm okay with it not coming first, 00:14:52.240 |
Linear algebra, you take off into 10 dimensions, no problem. 00:15:03.700 |
- So what concept or theorem in linear algebra, 00:15:12.700 |
that gives you pause, that leaves you in awe? 00:15:28.400 |
that didn't look connected, they turned out they were. 00:15:35.600 |
so we have a matrix, that's like the basic thing, 00:15:38.800 |
a rectangle of numbers, and it might be a rectangle of data. 00:15:42.680 |
You're probably gonna ask me later about data science, 00:15:50.400 |
You have, maybe every column corresponds to a drug, 00:16:00.360 |
and if the patient reacted favorably to the drug, 00:16:05.360 |
then you put up some positive number in there. 00:16:09.180 |
Anyway, rectangle of numbers, a matrix is basic. 00:16:15.160 |
So the big problem is to understand all those numbers. 00:16:37.120 |
and that's come on as fundamental in the last, 00:16:48.560 |
who've done engineering math or basic linear algebra, 00:17:05.720 |
I'm always pushing math faculty, get on, do it, do it, 00:17:24.840 |
So you're breaking a matrix into simple pieces, 00:17:29.040 |
and the first piece is the most important part of the data, 00:17:33.240 |
the second piece is the second most important part, 00:18:00.160 |
- So what do you find beautiful about singular values? 00:18:09.400 |
Every matrix, every matrix, rectangular, square, whatever, 00:18:21.320 |
Every matrix can be written as a rotation times a stretch, 00:18:30.320 |
otherwise all zeros except on the one diagonal, 00:18:34.200 |
and then the third factor is another rotation. 00:18:38.000 |
So rotation, stretch, rotation is the breakup of any matrix. 00:18:43.000 |
- The structure of that, the ability that you can do that, 00:18:54.240 |
the action of a matrix is not so easy to visualize, 00:19:10.720 |
so a pilot has to know about, well, what are the three, 00:19:15.200 |
the yaw is one of them, I've forgotten all the three turns 00:19:20.740 |
Up to 10 dimensions, you've got 10 ways to turn, 00:19:28.500 |
Take the space and turn it, and you can visualize a stretch. 00:19:32.080 |
So to break a matrix with all those numbers in it 00:19:54.000 |
and what they really enjoy and are inspired by, 00:20:06.480 |
So it's not just the kinds of lectures that you give, 00:20:10.760 |
but it's also just other folks, like with Numberphile, 00:20:14.160 |
there's a channel where they just chat about things 00:20:28.480 |
We're conditioned to think math is hard, math is abstract, 00:20:33.480 |
math is just for a few people, but it isn't that way. 00:20:36.600 |
A lot of people quite like math, and they like to, 00:20:43.800 |
you know, now I'm retired, I'm gonna learn some more math. 00:20:49.920 |
And I think what people like is that there's some order, 00:20:59.600 |
So it's really cheering to think that so many people 00:21:15.520 |
but math does reveal pretty strongly what things are true. 00:21:20.520 |
I mean, that's the whole point of proving things. 00:21:24.360 |
And yet, sort of our real world is messy and complicated. 00:21:35.520 |
- Because it is a source of comfort, like you've mentioned. 00:21:39.560 |
Well, I have to say, I'm not much of a philosopher. 00:21:59.040 |
you know, like take powers of two, two, four, eight, 16, 00:22:14.840 |
What is it about math, do you think, that brings that? 00:22:25.880 |
The fact that, you know, if you multiply two by itself 00:22:34.960 |
- Do you see math as a powerful tool or as an art form? 00:22:57.400 |
I'm certainly not a artist-type, philosopher-type person. 00:23:01.600 |
Might sound that way this morning, but I'm not. 00:23:13.280 |
And yeah, so probably within the MIT math department, 00:23:31.920 |
who are looking for a way to find answers, yeah. 00:23:42.720 |
do you think it's better to plug in the numbers 00:23:49.040 |
So looking at theorems and proving the theorems 00:23:53.640 |
or actually building up a basic intuition of the theorem 00:23:59.920 |
and then just plugging in numbers and seeing it work? 00:24:02.960 |
- Yeah, well, certainly many of us like to see examples. 00:24:11.040 |
it might be a pretty abstract-sounding example, 00:24:16.800 |
How are you gonna understand a rotation in 3D or in 10D? 00:24:21.800 |
And then some of us like to keep going with it 00:24:37.120 |
But the best, the great mathematicians probably, 00:24:49.760 |
would be a highly abstract thing to the rest of us. 00:25:04.680 |
- So I'm not sure if you're familiar with him, 00:25:11.840 |
currently running with math in all capital letters 00:25:23.880 |
- And his name rhymes with yours, Yang, Strang. 00:25:28.680 |
But he also loves math, and he comes from that world. 00:25:35.520 |
makes me realize that math, science, and engineering 00:25:52.760 |
we need people who are comfortable with numbers, 00:26:07.400 |
that we have almost no, I mean, I'm pretty sure 00:26:36.840 |
and at the same time can make speeches and lead, yeah. 00:26:53.760 |
- It's a major organization in math, in applied math. 00:27:02.880 |
- Right, yeah, so, well, it was fun to be president 00:27:12.120 |
So, that's the president of a pretty small society, 00:27:19.640 |
was getting some more attention in Washington. 00:27:29.240 |
to a committee of the House of Representatives 00:27:47.720 |
most members of the House have had a different training, 00:27:51.880 |
different background, but there was one from New Hampshire 00:27:56.360 |
who was my friend, really, by being in the class. 00:28:23.780 |
about artificial intelligence in Washington now. 00:28:32.040 |
Maybe it's hidden, maybe it's wearing a different hat. 00:28:37.760 |
and particularly, can I use the words deep learning? 00:28:41.640 |
It's a deep learning, is a particular approach 00:28:50.200 |
where data is just swamping the computers of the world, 00:28:56.060 |
and to understand it, out of all those numbers, 00:29:00.680 |
to find what's important in climate, in everything. 00:29:19.280 |
I thought, okay, I've gotta learn about this. 00:29:32.640 |
So can I start with the idea about deep learning? 00:29:42.000 |
So we're trying to learn, from all this data, 00:29:53.040 |
You've got some inputs for which you know the right outputs. 00:29:57.620 |
The question is, can you see the pattern there? 00:30:06.200 |
to understand what the output will be from that new input? 00:30:12.200 |
So we've got a million inputs with their outputs. 00:30:22.200 |
those million training inputs, which we know about, 00:30:28.180 |
And this idea of a neural net is part of the structure 00:30:39.820 |
We're looking for a rule that will take these training inputs 00:30:48.460 |
And then we're gonna use that rule on new inputs 00:30:51.680 |
that we don't know the output and see what comes. 00:30:56.080 |
- Linear algebra is a big part of finding that rule. 00:31:10.320 |
It's all about straight lines and flat planes. 00:31:31.640 |
And it turned out that it was enough to use the function 00:31:35.880 |
that's one straight line and then a different one. 00:31:52.360 |
simple non-linearity in blew the problem open. 00:31:56.720 |
- That little piece makes it sufficiently complicated 00:32:03.840 |
So it has a fold in the graph, the graph, two pieces. 00:32:29.660 |
by using a, sort of formulating an objective function, 00:32:42.380 |
Why do you think they work to be able to find a rule 00:32:48.840 |
but is just seems to be pretty good in a lot of cases? 00:32:54.600 |
Is it surprising to you as it is to many people? 00:32:58.200 |
Do you have an intuition of why this works at all? 00:33:01.160 |
- Well, I'm beginning to have a better intuition. 00:33:04.340 |
This idea of things that are piecewise linear, 00:33:17.800 |
That curved, it almost curved, but every piece is flat. 00:33:26.840 |
That idea's been used by engineers, is used by engineers, 00:33:31.120 |
big time, something called the finite element method. 00:33:34.200 |
If you wanna design a bridge, design a building, 00:33:56.160 |
of expressive power in this kind of piecewise linear-- 00:34:05.360 |
how complicated a thing can this piecewise flat guys express, 00:34:17.480 |
of such piecewise linear or just of neural networks, 00:34:47.480 |
- So in terms of just mapping from inputs to outputs, 00:34:55.960 |
in the context of neural networks in general, 00:35:00.560 |
data is just tensor vectors, matrices, tensors. 00:35:10.320 |
How much of our world can be expressed in this way? 00:35:48.960 |
Newton and Einstein and other great, great people 00:36:10.920 |
understand how oil will sit in an underground basin. 00:36:48.320 |
- It's an automated search for the underlying rules. 00:37:17.440 |
- Well, this world around us does seem to be, 00:37:19.920 |
does seem to always have a signal of some kind 00:37:29.480 |
We just talked about a little bit of application. 00:37:38.400 |
- Well, for myself, I'm probably a theory person. 00:37:43.400 |
I'm speaking here pretty freely about applications, 00:37:53.240 |
I'm not a physicist or a chemist or a neuroscientist. 00:38:06.480 |
and the relation of matrices, columns to rows. 00:38:33.800 |
and then the geniuses of math and physics and chemistry 00:38:43.360 |
and then doing the really understanding nature. 00:38:55.000 |
Maybe just a quick and broad strokes from your perspective. 00:38:58.640 |
Where does linear algebra sit as a subfield of mathematics? 00:39:04.760 |
What are the various subfields that you think about 00:39:12.200 |
- So the big fields of math are algebra as a whole 00:39:18.040 |
and problems like calculus and differential equations. 00:39:24.360 |
Then maybe geometry deserves to be thought of 00:39:28.800 |
as a different field to understand the geometry 00:39:46.240 |
I think math, thinking about undergraduate math, 00:39:54.320 |
I think we overdo the calculus at the cost of the algebra, 00:40:02.720 |
- See, I have this talk titled Calculus Versus Linear Algebra. 00:40:17.080 |
- Right, well, okay, the viewer is gonna think 00:40:22.760 |
Not true, I'm just telling the truth as it is. 00:40:27.080 |
Yeah, so I feel linear algebra is just a nice part of math 00:40:34.420 |
They can understand something that's a little bit abstract 00:41:06.880 |
have become very, very important, have also jumped forward. 00:41:11.240 |
So, and that's different from linear algebra, 00:41:15.160 |
So now we really have three major areas to me, 00:41:34.000 |
And calculus has traditionally had a lion's share 00:41:42.280 |
- Thank you, disproportionate, that's a good word. 00:41:44.760 |
- Of the love and attention from the excited young minds. 00:41:59.400 |
Okay, so my favorite matrix is square, I admit it. 00:42:05.480 |
and it has twos running down the main diagonal. 00:42:26.620 |
So mostly zeros, just three non-zero diagonals coming down. 00:42:39.220 |
you see it as analogous in calculus to second derivative. 00:42:44.100 |
So calculus learns about taking the derivative, 00:42:50.640 |
But second derivative, now that's also important. 00:43:11.560 |
So second derivatives should have a bigger place 00:43:17.380 |
Second, my matrices, which are like the linear algebra 00:43:22.380 |
version of second derivatives, are neat in linear algebra. 00:43:29.380 |
Yeah, just everything comes out right with those guys. 00:43:35.220 |
What did you learn about the process of learning 00:43:38.380 |
by having taught so many students math over the years? 00:43:44.820 |
I'll have to admit here that I'm not really a good teacher 00:43:55.540 |
The exam's the part of my life that I don't like, 00:43:59.000 |
and grading them, and giving the students A or B, 00:44:11.900 |
I don't know if they believe me, probably they don't, 00:44:18.040 |
I'm here to teach you math and not to grade you. 00:44:25.100 |
when's he gonna, is he gonna give me an A minus, 00:44:31.420 |
- What have you learned about the process of learning? 00:44:36.860 |
to give you a legitimate answer about learning, 00:44:41.400 |
I should have paid more attention to the assessment, 00:45:01.920 |
are there moments of learning that you just see 00:45:05.460 |
in the student's eyes, you don't need to look at the grades, 00:45:08.220 |
but you see in their eyes that you hook them, 00:45:16.260 |
you know what, they fall in love with this beautiful world 00:45:24.460 |
- Or conversely, that they give up at that point, 00:45:28.060 |
is the opposite, the dark is saying that math, 00:45:31.180 |
I'm just not good at math, I don't wanna walk away. 00:45:37.700 |
they were discouraged, but don't be discouraged, 00:45:55.460 |
the four fundamental subspaces and the structure 00:46:00.620 |
of the fundamental theorem of linear algebra, 00:46:06.780 |
that is the relation of those four subspaces, 00:46:22.420 |
just starting their journey in mathematics today? 00:46:33.980 |
professor who is still enjoying what he's doing, 00:46:38.980 |
what he's teaching, still looking for new ways 00:46:49.620 |
the moment when you see, oh yeah, that works. 00:47:18.020 |
there's beautiful things in geometry to understand. 00:47:28.660 |
that are followed in biology as there are in every field. 00:47:45.660 |
Except for the grade stuff, having the good grades. 00:47:52.180 |
what memories bring you the most joy and pride? 00:48:06.180 |
that's MIT's linear algebra course that I started. 00:48:11.660 |
okay, I started this course, a lot of students take it, 00:48:24.100 |
make a connection between ideas and students, 00:48:37.380 |
from people who've watched the videos and it's inspiring. 00:48:43.580 |
I just, I'll maybe take this chance to say thank you. 00:48:47.260 |
- Well, there's millions of students who you've taught 00:48:59.820 |
- Thank you for listening to this conversation 00:49:04.340 |
And thank you to our presenting sponsor, Cash App. 00:49:12.860 |
a STEM education nonprofit that inspires hundreds 00:49:20.780 |
If you enjoy this podcast, subscribe on YouTube. 00:49:23.860 |
We have five stars on Apple Podcast, support on Patreon 00:49:41.300 |
Thank you for listening and hope to see you next time.