Jim Keller: The Future of Computing, AI, Life, and Consciousness

00:00:00.000 | The following is a conversation with Jim Keller,

00:00:02.920 | his second time in the podcast.

00:00:04.960 | Jim is a legendary microprocessor architect

00:00:08.480 | and is widely seen as one of the greatest engineering minds

00:00:12.560 | of the computing age.

00:00:14.620 | In a peculiar twist of space-time in our simulation,

00:00:18.840 | Jim is also a brother-in-law of Jordan Peterson.

00:00:22.200 | We talk about this and about computing,

00:00:25.320 | artificial intelligence, consciousness, and life.

00:00:29.200 | Quick mention of our sponsors.

00:00:31.280 | Athletic Greens All-in-One Nutrition Drink,

00:00:33.800 | Brooklyn and Sheets, ExpressVPN,

00:00:36.640 | and Belcampo Grass-Fed Meat.

00:00:39.600 | Click the sponsor links to get a discount

00:00:41.680 | and to support this podcast.

00:00:43.940 | As a side note, let me say that Jim is someone

00:00:46.200 | who on a personal level inspired me to be myself.

00:00:50.160 | There was something in his words on and off the mic,

00:00:53.340 | or perhaps that he even paid attention to me at all,

00:00:56.200 | that almost told me, "You're all right, kid.

00:00:59.120 | "A kind of pat on the back that can make the difference

00:01:01.780 | "between a mind that flourishes

00:01:03.600 | "and a mind that is broken down

00:01:05.700 | "by the cynicism of the world."

00:01:08.200 | So I guess that's just my brief few words

00:01:10.480 | of thank you to Jim, and in general,

00:01:12.840 | gratitude for the people who have given me a chance

00:01:15.520 | on this podcast, in my work, and in life.

00:01:19.040 | If you enjoy this thing, subscribe on YouTube,

00:01:21.240 | review it on Apple Podcasts, follow on Spotify,

00:01:24.280 | support it on Patreon,

00:01:25.680 | or connect with me on Twitter, Alex Friedman.

00:01:28.620 | And now, here's my conversation with Jim Keller.

00:01:32.360 | What's the value and effectiveness

00:01:35.380 | of theory versus engineering, this dichotomy,

00:01:38.120 | in building good software or hardware systems?

00:01:43.120 | - Well, good design is both.

00:01:45.580 | I guess that's pretty obvious.

00:01:48.720 | By engineering, do you mean, you know,

00:01:50.840 | reduction of practice of known methods?

00:01:53.280 | And then science is the pursuit of discovering things

00:01:55.980 | that people don't understand,

00:01:57.820 | or solving unknown problems.

00:02:00.380 | - Definitions are interesting here,

00:02:02.000 | but I was thinking more in theory,

00:02:04.160 | constructing models that kind of generalize

00:02:06.760 | about how things work.

00:02:08.560 | And engineering is actually building stuff,

00:02:12.800 | the pragmatic, like, okay, we have these nice models,

00:02:16.220 | but how do we actually get things to work?

00:02:17.960 | Maybe economics is a nice example.

00:02:20.780 | Like, economists have all these models

00:02:22.480 | of how the economy works,

00:02:23.680 | and how different policies will have an effect,

00:02:26.720 | but then there's the actual, okay,

00:02:29.240 | let's call it engineering,

00:02:30.480 | of like, actually deploying the policies.

00:02:33.240 | - So, computer design is almost all engineering,

00:02:36.380 | and reduction of practice of known methods.

00:02:38.200 | Now, because of the complexity of the computers we built,

00:02:43.200 | you know, you could think you're,

00:02:44.960 | well, we'll just go write some code,

00:02:46.600 | and then we'll verify it, and then we'll put it together,

00:02:49.160 | and then you find out that the combination

00:02:50.920 | of all that stuff is complicated.

00:02:53.240 | And then you have to be inventive

00:02:54.680 | to figure out how to do it, right?

00:02:56.920 | So, that definitely happens a lot.

00:02:59.760 | And then, every so often, some big idea happens,

00:03:04.440 | but it might be one person.

00:03:06.360 | - And that idea is in what, in the space of engineering,

00:03:08.840 | or is it in the space of--

00:03:10.400 | - Well, I'll give you an example.

00:03:11.400 | So, one of the limits of computer performance

00:03:13.160 | is branch prediction.

00:03:14.880 | So, and there's a whole bunch of ideas

00:03:17.500 | about how good you could predict a branch.

00:03:19.440 | And people said, there's a limit to it,

00:03:21.640 | it's an asymptotic curve,

00:03:23.480 | and somebody came up with a better way

00:03:24.920 | to do branch prediction.

00:03:26.440 | It was a lot better.

00:03:28.240 | And he published a paper on it,

00:03:29.720 | and every computer in the world now uses it.

00:03:32.760 | And it was one idea.

00:03:34.600 | So, the engineers who build branch prediction hardware

00:03:37.960 | were happy to drop the one kind of training array

00:03:40.480 | and put it in another one.

00:03:42.400 | So, it was a real idea.

00:03:44.840 | - And branch prediction is one of the key problems

00:03:48.520 | underlying all of sort of the lowest level of software,

00:03:51.960 | it boils down to branch prediction.

00:03:53.800 | - Boils down to uncertainty.

00:03:54.840 | Computers are limited by,

00:03:56.280 | single-thread computers are limited by two things.

00:03:58.640 | The predictability of the path of the branches

00:04:01.400 | and the predictability of the locality of data.

00:04:04.140 | So, we have predictors that now predict

00:04:07.080 | both of those pretty well.

00:04:09.200 | So, memory is a couple hundred cycles away,

00:04:11.880 | local cache is a couple cycles away.

00:04:14.560 | When you're executing fast,

00:04:15.720 | virtually all the data has to be in the local cache.

00:04:19.040 | So, a simple program says,

00:04:21.320 | add one to every element in an array,

00:04:23.280 | it's really easy to see what the stream of data will be.

00:04:26.680 | But you might have a more complicated program

00:04:28.520 | that says, get an element of this array,

00:04:31.080 | look at something, make a decision,

00:04:32.800 | go get another element, it's kind of random.

00:04:35.200 | And you can think, that's really unpredictable.

00:04:37.780 | And then you make this big predictor

00:04:39.200 | that looks at this kind of pattern and you realize,

00:04:41.400 | well, if you get this data and this data,

00:04:43.000 | then you probably want that one.

00:04:44.560 | And if you get this one and this one and this one,

00:04:46.440 | you probably want that one.

00:04:47.960 | - And is that theory or is that engineering?

00:04:49.920 | Like the paper that was written,

00:04:51.320 | was it a asymptotic kind of discussion

00:04:54.680 | or is it more like, here's a hack that works well?

00:04:57.960 | - It's a little bit of both.

00:04:59.160 | Like there's information theory, I think, somewhere.

00:05:01.320 | - Okay, so it's actually trying to prove--

00:05:03.680 | - Yeah, but once you know the method,

00:05:06.420 | implementing it is an engineering problem.

00:05:08.700 | Now, there's a flip side of this,

00:05:10.840 | which is, in a big design team,

00:05:13.480 | what percentage of people think their plan

00:05:19.040 | or their life's work is engineering

00:05:20.800 | versus inventing things?

00:05:23.520 | So lots of companies will reward you for filing patents.

00:05:27.560 | Some many big companies get stuck because to get promoted,

00:05:30.460 | you have to come up with something new.

00:05:33.000 | And then what happens is everybody's trying

00:05:34.760 | to do some random new thing, 99% of which doesn't matter,

00:05:39.160 | and the basics get neglected.

00:05:41.180 | And, or they get to, there's a dichotomy.

00:05:45.600 | They think like the cell library and the basic CAD tools,

00:05:49.560 | or basic software validation methods, that's simple stuff.

00:05:54.560 | They wanna work on the exciting stuff.

00:05:56.960 | And then they spend lots of time trying to figure out

00:05:58.960 | how to patent something, and that's mostly useless.

00:06:02.280 | - But the breakthroughs are on the simple stuff.

00:06:04.620 | - No, no, you have to do the simple stuff really well.

00:06:09.000 | If you're building a building out of bricks,

00:06:11.500 | you want great bricks.

00:06:13.280 | So you go to two places to sell bricks,

00:06:14.960 | one guy says, "Yeah, they're over there in an ugly pile."

00:06:17.960 | And the other guy is like,

00:06:18.800 | "Lovingly tells you about the 50 kinds of bricks,

00:06:21.240 | "and how hard they are, and how beautiful they are,

00:06:23.240 | "and how square they are, and you know,

00:06:26.000 | "which one are you gonna buy bricks from?

00:06:28.120 | "Which is gonna make a better house?"

00:06:30.400 | - So you're talking about the craftsman,

00:06:32.040 | the person who understands bricks, who loves bricks,

00:06:34.240 | who loves the variety?

00:06:35.160 | - That's a good word.

00:06:36.120 | You know, good engineering is great craftsmanship.

00:06:39.400 | And when you start thinking engineering is about invention,

00:06:44.880 | and you set up a system that rewards invention,

00:06:47.960 | the craftsmanship gets neglected.

00:06:50.680 | - Okay, so maybe one perspective is the theory,

00:06:53.520 | the science overemphasizes invention,

00:06:57.680 | and engineering emphasizes craftsmanship,

00:07:00.440 | and therefore, so if you,

00:07:02.880 | it doesn't matter what you do, theory, engineering--

00:07:04.280 | - Well, everybody does.

00:07:05.120 | Like, read the tech rags.

00:07:06.200 | They're always talking about some breakthrough,

00:07:07.880 | or innovation, and everybody thinks

00:07:10.640 | that's the most important thing.

00:07:12.480 | But the number of innovative ideas

00:07:13.920 | is actually relatively low.

00:07:15.960 | We need them, right?

00:07:17.200 | And innovation creates a whole new opportunity.

00:07:19.800 | Like when some guy invented the internet, right?

00:07:24.000 | Like, that was a big thing.

00:07:25.880 | The million people that wrote software against that

00:07:28.200 | were mostly doing engineering software writing.

00:07:31.160 | So the elaboration of that idea was huge.

00:07:34.280 | - I don't know if you know Brendan Eich,

00:07:35.560 | he wrote JavaScript in 10 days.

00:07:37.920 | And that's an interesting story.

00:07:39.520 | It makes me wonder, and it was, you know,

00:07:42.400 | famously for many years considered to be

00:07:44.880 | a pretty crappy programming language.

00:07:47.680 | Still is, perhaps.

00:07:48.800 | It's been improving sort of consistently.

00:07:51.160 | But the interesting thing about that guy is,

00:07:55.600 | you know, he doesn't get any awards.

00:07:57.400 | (laughs)

00:07:58.520 | You don't get a Nobel Prize, or a Fields Medal, or--

00:08:01.200 | - For inventing a crappy piece of, you know,

00:08:05.680 | software code that--

00:08:06.520 | - Well, that is currently the number one

00:08:07.880 | programming language in the world,

00:08:09.200 | and runs, now is increasingly running

00:08:12.640 | the back end of the internet.

00:08:13.800 | - Well, does he know why everybody uses it?

00:08:17.640 | Like, that would be an interesting thing.

00:08:19.320 | Was it the right thing at the right time?

00:08:22.360 | 'Cause like, when stuff like JavaScript came out,

00:08:24.920 | like there was a move from, you know,

00:08:26.240 | writing C programs in C++ to, let's call it,

00:08:30.040 | what they call managed code frameworks.

00:08:32.360 | Where you write simple code, it might be interpreted,

00:08:35.200 | it has lots of libraries, productivity is high,

00:08:37.800 | and you don't have to be an expert.

00:08:39.560 | So, you know, Java was supposed to solve

00:08:41.360 | all the world's problems, it was complicated.

00:08:43.800 | JavaScript came out, you know,

00:08:45.240 | after a bunch of other scripting languages.

00:08:47.680 | I'm not an expert on it, but--

00:08:49.280 | - Yeah.

00:08:50.120 | - But was it the right thing at the right time?

00:08:51.200 | - The right thing at the right--

00:08:52.040 | - Or was there something, you know, clever?

00:08:54.320 | 'Cause he wasn't the only one.

00:08:56.320 | - There's a few elements.

00:08:57.440 | - And maybe if he figured out what it was.

00:08:59.520 | - No, he didn't--

00:09:00.360 | - Then he'd get a prize.

00:09:01.200 | (laughs)

00:09:02.040 | Like that's--

00:09:02.880 | - Constructive theory.

00:09:03.720 | - Yeah, you know, maybe his problem

00:09:05.400 | is he hasn't defined this.

00:09:06.920 | - Or he just needs a good promoter.

00:09:08.400 | (laughs)

00:09:09.520 | - Well, I think there was a bunch of blog posts

00:09:11.920 | written about it, which is like, wrong is right,

00:09:14.840 | which is like doing the crappy thing fast,

00:09:19.320 | just like hacking together the thing

00:09:21.360 | that answers some of the needs,

00:09:23.280 | and then iterating over time, listening to developers,

00:09:26.120 | like listening to people who actually use the thing.

00:09:28.240 | This is something you can do more in software.

00:09:30.960 | - Yep.

00:09:31.800 | - But the right time, like you have to sense,

00:09:33.760 | you have to have a good instinct

00:09:35.120 | of when is the right time for the right tool,

00:09:37.560 | and make it super simple, and just get it out there.

00:09:42.560 | The problem is, this is true with hardware,

00:09:45.200 | this is less true with software,

00:09:46.440 | is there's backward compatibility

00:09:48.440 | that just drags behind you as you try to fix

00:09:52.600 | all the mistakes of the past.

00:09:53.800 | But the timing--

00:09:55.800 | - Was good.

00:09:56.640 | - There's something about that, and it wasn't accidental.

00:09:58.800 | You have to like give yourself over to the,

00:10:02.560 | you have to have this broad sense of what's needed now,

00:10:07.560 | both scientifically and the community,

00:10:10.840 | and just like, it was obvious that there was no,

00:10:15.480 | the interesting thing about JavaScript

00:10:17.960 | is everything that ran in the browser at the time,

00:10:20.880 | like Java and I think other like scheme,

00:10:24.440 | other programming languages,

00:10:25.920 | they were all in a separate external container.

00:10:30.480 | And then JavaScript was literally just injected

00:10:33.640 | into the webpage, it was the dumbest possible thing

00:10:36.360 | running in the same thread as everything else.

00:10:39.320 | And like, it was inserted as a comment.

00:10:43.080 | So JavaScript code is inserted as a comment in the HTML code.

00:10:47.560 | And it was, I mean, it's either genius or super dumb,

00:10:52.240 | but it's like--

00:10:53.080 | - Right, so it has no apparatus

00:10:54.480 | for like a virtual machine and container,

00:10:56.680 | it just executed in the framework of the program

00:10:58.920 | that's already running.

00:10:59.760 | - Yeah, that's cool.

00:11:00.920 | - And then because something about that accessibility,

00:11:04.120 | the ease of its use,

00:11:06.080 | resulted in then developers innovating

00:11:10.080 | of how to actually use it.

00:11:11.400 | I mean, I don't even know what to make of that,

00:11:13.680 | but it does seem to echo across different software,

00:11:18.340 | like stories of different software.

00:11:19.720 | PHP has the same story, really crappy language.

00:11:22.920 | They just took over the world.

00:11:24.420 | - Well, we used to have a joke

00:11:26.240 | that the random length instructions,

00:11:28.360 | variable length instructions, that's always won,

00:11:30.680 | even though they're obviously worse.

00:11:33.080 | Like, nobody knows why.

00:11:34.440 | x86 is arguably the worst architecture on the planet,

00:11:38.680 | it's one of the most popular ones.

00:11:40.480 | - Well, I mean, isn't that also the story

00:11:42.360 | of risk versus sys?

00:11:43.680 | I mean, is that simplicity?

00:11:46.240 | There's something about simplicity

00:11:47.440 | that us in this evolutionary process is valued.

00:11:52.440 | If it's simple, it spreads faster, it seems like.

00:11:58.820 | Or is that not always true?

00:11:59.980 | - Not always true.

00:12:01.140 | Yeah, it could be simple is good, but too simple is bad.

00:12:04.260 | - So why did risk win, you think, so far?

00:12:06.420 | - Did risk win?

00:12:07.260 | - In the long arc of history.

00:12:10.580 | - We don't know.

00:12:11.420 | - So who's gonna win?

00:12:12.700 | What's risk, what's sys,

00:12:14.180 | and who's gonna win in that space,

00:12:16.140 | in these instruction sets?

00:12:17.580 | - AI software's gonna win,

00:12:18.900 | but there'll be little computers

00:12:21.100 | that run little programs like normal all over the place.

00:12:23.900 | But we're going through another transformation,

00:12:27.700 | so.

00:12:28.540 | - But you think instruction sets

00:12:30.060 | underneath it all will change?

00:12:32.380 | - Yeah, they evolve slowly.

00:12:33.620 | They don't matter very much.

00:12:35.460 | - They don't matter very much, okay.

00:12:36.780 | - I mean, the limits of performance are,

00:12:39.460 | you know, predictability of instructions and data.

00:12:41.660 | I mean, that's the big thing.

00:12:43.380 | And then the usability of it is some, you know,

00:12:46.720 | quality of design, quality of tools, availability.

00:12:52.140 | Like right now, x86 is proprietary with Intel and AMD,

00:12:56.420 | but they can change it any way they want independently.

00:12:59.460 | Right, ARM is proprietary to ARM,

00:13:01.620 | and they won't let anybody else change it.

00:13:03.660 | So it's like a sole point.

00:13:05.700 | And RISC-V is open source, so anybody can change it,

00:13:09.100 | which is super cool, but that also might mean

00:13:11.820 | it gets changed in too many random ways

00:13:13.620 | that there's no common subset of it that people can use.

00:13:17.700 | - Do you like open or do you like closed?

00:13:19.900 | Like if you were to bet all your money on one or the other,

00:13:22.260 | RISC-V versus it?

00:13:23.300 | - No idea.

00:13:24.180 | - It's case dependent?

00:13:25.020 | - Well, x86, oddly enough,

00:13:26.340 | when Intel first started developing it,

00:13:28.300 | they licensed it to like seven people.

00:13:30.240 | So it was the open architecture.

00:13:33.060 | And then they move faster than others

00:13:35.340 | and also bought one or two of them.

00:13:37.460 | But there was seven different people making x86

00:13:40.260 | 'cause at the time there was 6502 and Z80s and 8086.

00:13:45.260 | And you could argue everybody thought Z80

00:13:49.060 | was the better instruction set,

00:13:50.940 | but that was proprietary to one place.

00:13:54.460 | Oh, and the 6800.

00:13:56.140 | So there's like four or five different microprocessors.

00:13:59.460 | Intel went open, got the market share

00:14:02.420 | 'cause people felt like they had multiple sources from it.

00:14:04.700 | And then over time it narrowed down to two players.

00:14:07.660 | - So why, you as a historian,

00:14:09.880 | why did Intel win for so long with their processors?

00:14:16.300 | I mean- - They were great.

00:14:18.900 | Their process development was great.

00:14:21.100 | - So it's just looking back to JavaScript

00:14:23.740 | and like is a Microsoft and Netscape

00:14:26.540 | and all these internet browsers.

00:14:28.940 | Microsoft won the browser game

00:14:31.740 | because they aggressively stole other people's ideas.

00:14:35.980 | Like right after they did it.

00:14:37.820 | - You know, I don't know if Intel

00:14:39.780 | was stealing other people's ideas.

00:14:41.180 | They started making- - In a good way.

00:14:42.300 | Stealing in a good way, just to clarify.

00:14:43.780 | - They started making RAMs, random access memories.

00:14:48.260 | And then at the time when the Japanese manufacturers

00:14:51.700 | came up, they were getting out competed on that.

00:14:54.900 | And they pivoted the microprocessors

00:14:56.620 | and they made the first integrated microprocessor

00:14:59.100 | and ran programs.

00:14:59.940 | It was the 404 or something.

00:15:03.900 | - Who was behind that pivot?

00:15:04.900 | That's a hell of a pivot.

00:15:05.900 | - Andy Grove, and he was great.

00:15:07.780 | - That's a hell of a pivot.

00:15:10.140 | - And then they led semiconductor industry.

00:15:13.900 | Like they were just a little company, IBM.

00:15:16.020 | All kinds of big companies had boatloads of money

00:15:19.020 | and they out-innovated everybody.

00:15:21.180 | - Out-innovated, okay.

00:15:22.460 | - Yeah, yeah.

00:15:23.300 | - So it's not like marketing, it's not any of that stuff.

00:15:26.300 | - Their processor designs were pretty good.

00:15:28.460 | I think the Core 2 was probably the first one

00:15:34.380 | I thought was great.

00:15:36.180 | It was a really fast processor.

00:15:37.580 | And then Haswell was great.

00:15:38.980 | - What makes a great processor in that?

00:15:42.260 | - Oh, if you just look at its performance

00:15:43.900 | versus everybody else, it's the size of it,

00:15:47.620 | the usability of it.

00:15:49.900 | - So it's not specific, some kind of element

00:15:51.860 | that makes it beautiful,

00:15:52.700 | it's just like literally just raw performance.

00:15:55.140 | Is that how you think about processors?

00:15:57.180 | It's just like raw performance?

00:15:59.780 | - Of course.

00:16:01.340 | It's like a horse race.

00:16:02.340 | The fastest one wins.

00:16:04.300 | Now--

00:16:05.140 | - You don't care how.

00:16:05.980 | (laughing)

00:16:07.780 | So as long as it wins.

00:16:08.620 | - Well, there's the fastest in the environment.

00:16:10.620 | Like for years you made the fastest one you could

00:16:13.100 | and then people started to have power limits.

00:16:14.980 | So then you made the fastest at the right power point.

00:16:17.700 | And then when we started doing multi-processors,

00:16:20.500 | like if you could scale your processors

00:16:23.620 | more than the other guy,

00:16:24.460 | you could be 10% faster on like a single thread,

00:16:27.020 | but you have more threads.

00:16:28.460 | So there's lots of variability.

00:16:30.060 | And then Arm really explored like,

00:16:35.060 | you know, they have the A series and the R series

00:16:37.500 | and the M series, like a family of processors

00:16:40.380 | for all these different design points

00:16:41.980 | from like unbelievably small and simple.

00:16:44.660 | And so then when you're doing the design,

00:16:46.580 | it's sort of like this big palette of CPUs.

00:16:49.420 | Like they're the only ones with a credible,

00:16:51.540 | you know, top to bottom palette.

00:16:53.220 | And--

00:16:54.700 | - What do you mean a credible top to bottom?

00:16:56.940 | - Well, there's people that make microcontrollers

00:16:58.660 | that are small, but they don't have a fast one.

00:17:00.500 | There's people make fast processors,

00:17:02.100 | but don't have a medium one or a small one.

00:17:04.900 | - Is it hard to do that full palette?

00:17:07.100 | That seems like a--

00:17:07.940 | - Yeah, it's a lot of different--

00:17:09.380 | - So what's the difference between the Arm folks and Intel

00:17:13.380 | in terms of the way they're approaching this problem?

00:17:15.620 | - Well, Intel, almost all their processor designs

00:17:19.180 | were, you know, very custom high end,

00:17:21.700 | you know, for the last 15, 20 years.

00:17:23.420 | - The fastest horse possible.

00:17:24.900 | - Yeah.

00:17:25.860 | - In one horse sense.

00:17:27.540 | - Yeah, and they architecturally are really good,

00:17:30.420 | but the company itself was fairly insular

00:17:33.380 | to what's going on in the industry with CAD tools and stuff.

00:17:36.300 | And there's this debate about custom design versus synthesis

00:17:39.820 | and how do you approach that.

00:17:41.340 | I'd say Intel was slow on the,

00:17:43.420 | getting to synthesize processors.

00:17:45.700 | Arm came in from the bottom and they generated IP,

00:17:49.100 | which went to all kinds of customers.

00:17:50.860 | So they had very little say

00:17:52.020 | in how the customer implemented their IP.

00:17:54.980 | So Arm is super friendly to the synthesis IP environment.

00:17:59.460 | Whereas Intel said, we're gonna make this great client chip

00:18:02.300 | or server chip with our own CAD tools,

00:18:04.340 | with our own process, with our own, you know,

00:18:06.660 | other supporting IP and everything only works with our stuff.

00:18:11.340 | - So is that, is Arm winning the mobile platform space

00:18:16.340 | in terms of process? - Of course, yeah.

00:18:17.700 | - And so in that, what you're describing

00:18:21.820 | is why they're winning.

00:18:22.860 | - Well, they had lots of people

00:18:24.420 | doing lots of different experiments.

00:18:26.420 | So they controlled the process architecture and IP,

00:18:29.420 | but they let people put in lots of different chips.

00:18:32.060 | And there was a lot of variability in what happened there.

00:18:35.260 | Whereas Intel, when they made their mobile,

00:18:37.140 | their foray into mobile, they had one team doing one part.

00:18:41.340 | Right, so it wasn't 10 experiments.

00:18:43.140 | And then their mindset was PC mindset,

00:18:45.940 | Microsoft software mindset.

00:18:48.100 | And that brought a whole bunch of things along

00:18:49.860 | that the mobile world, the embedded world don't do.

00:18:52.500 | - Do you think it was possible for Intel to pivot hard

00:18:55.460 | and win the mobile market?

00:18:58.260 | That's a hell of a difficult thing to do, right?

00:19:00.060 | For a huge company to just pivot.

00:19:02.060 | I mean, it's so interesting to,

00:19:05.540 | 'cause we'll talk about your current work.

00:19:07.420 | It's like, it's clear that PCs were dominating

00:19:11.100 | for several decades, like desktop computers.

00:19:14.180 | And then mobile, it's unclear.

00:19:17.940 | - It's a leadership question.

00:19:19.380 | Like Apple under Steve Jobs, when he came back,

00:19:23.060 | they pivoted multiple times.

00:19:24.780 | You know, they build iPads and iTunes and phones

00:19:28.260 | and tablets and great Macs.

00:19:30.060 | Like who knew computers should be made out of aluminum?

00:19:33.380 | Nobody knew that.

00:19:34.260 | But they're great, it's super fun.

00:19:37.140 | - No, Steve?

00:19:37.980 | - Yeah, Steve Jobs.

00:19:38.820 | Like they pivoted multiple times.

00:19:40.540 | And the old Intel, they did that multiple times.

00:19:45.860 | They made DRAMs and processors and processes.

00:19:48.540 | - I gotta ask this.

00:19:50.900 | What was it like working with Steve Jobs?

00:19:53.020 | - I didn't work with him.

00:19:54.380 | - Did you interact with him?

00:19:55.660 | - Twice.

00:19:56.500 | I said hi to him twice in the cafeteria.

00:19:59.820 | - What did he say?

00:20:01.020 | Hi?

00:20:01.860 | - He said, "Hey, fellas."

00:20:02.700 | (laughing)

00:20:04.300 | He was friendly.

00:20:05.940 | He was wandering around with somebody.

00:20:08.260 | He couldn't find a table 'cause the cafeteria was packed.

00:20:12.300 | And I gave him my table.

00:20:13.700 | But I worked for Mike Colbert, who talked to,

00:20:16.060 | like Mike was the unofficial CTO of Apple

00:20:19.260 | and a brilliant guy.

00:20:20.300 | And he worked for Steve for 25 years, maybe more.

00:20:23.540 | And he talked to Steve multiple times a day.

00:20:25.740 | And he was one of the people who could put up with Steve's,

00:20:29.380 | let's say, brilliance and intensity.

00:20:31.740 | And Steve really liked him.

00:20:33.580 | And Steve trusted Mike to translate the shit he thought up

00:20:38.580 | into engineering products that worked.

00:20:40.980 | And then Mike ran a group called Platform Architecture.

00:20:43.220 | And I was in that group.

00:20:44.860 | So many times I'd be sitting with Mike

00:20:46.460 | and the phone would ring.

00:20:47.980 | And it'd be Steve.

00:20:48.820 | And Mike would hold the phone like this

00:20:50.500 | 'cause Steve would be yelling about something or other.

00:20:52.420 | - Yeah.

00:20:53.260 | And then he would translate.

00:20:54.100 | - And he'd translate.

00:20:54.980 | And then he would say, "Steve wants us to do this."

00:20:58.380 | So.

00:20:59.540 | - Was Steve a good engineer or no?

00:21:01.180 | - I don't know.

00:21:02.500 | He was a great idea guy.

00:21:03.780 | - Idea person.

00:21:04.620 | - And he's a really good selector for talent.

00:21:07.580 | - Yeah.

00:21:08.420 | That seems to be one of the key elements of leadership.

00:21:10.140 | Right?

00:21:10.980 | - And then he was a really good first principles guy.

00:21:12.780 | Like somebody say something couldn't be done

00:21:15.100 | and he would just think, "That's obviously wrong."

00:21:19.780 | Right?

00:21:20.620 | But maybe it's hard to do.

00:21:23.060 | Maybe it's expensive to do.

00:21:24.460 | Maybe we need different people.

00:21:26.220 | There's a whole bunch of,

00:21:27.300 | if you wanna do something hard,

00:21:29.620 | maybe it takes time.

00:21:30.620 | Maybe you have to iterate.

00:21:31.620 | There's a whole bunch of things you could think about.

00:21:33.740 | But saying it can't be done is stupid.

00:21:36.380 | - How would you compare,

00:21:38.100 | so it seems like Elon Musk is more engineering centric.

00:21:42.860 | But is also, I think he considers himself a designer too.

00:21:45.700 | He has a design mind.

00:21:47.020 | Steve Jobs feels like he is much more idea space,

00:21:50.540 | design space versus engineering.

00:21:52.580 | - Yeah.

00:21:53.420 | - Just make it happen.

00:21:54.260 | Like the world should be this way.

00:21:55.860 | Just figure it out.

00:21:57.180 | - But he used computers.

00:21:59.140 | He had computer people talk to him all the time.

00:22:01.860 | Like Mike was a really good computer guy.

00:22:03.380 | He knew computers could do.

00:22:04.860 | - Computer meaning computer hardware?

00:22:06.340 | Like a whole lot of stuff.

00:22:07.180 | - Hardware, software, all the pieces.

00:22:08.500 | - The whole thing.

00:22:09.340 | - And then he would have an idea

00:22:11.820 | about what could we do with this next.

00:22:14.540 | That was grounded in reality.

00:22:16.060 | It wasn't like he was just finger painting on the wall

00:22:19.220 | and wishing somebody would interpret it.

00:22:21.420 | So he had this interesting connection

00:22:23.420 | because he wasn't a computer architect or designer,

00:22:28.340 | but he had an intuition from the computers we had

00:22:30.820 | to what could happen.

00:22:31.940 | - Essentially you say intuition

00:22:35.300 | because it seems like he was pissing off a lot of engineers

00:22:40.020 | in his intuition about what can and can't be done.

00:22:43.480 | What is all these stories about floppy disks

00:22:48.100 | and all that kind of stuff.

00:22:49.100 | - Yeah, so in Steve, the first round,

00:22:52.100 | like he'd go into a lab and look at what's going on

00:22:55.460 | and hate it and fire people or ask somebody in the elevator

00:23:00.460 | what they're doing for Apple and not be happy.

00:23:03.900 | When he came back, my impression was,

00:23:06.580 | is he surrounded himself

00:23:08.060 | with this relatively small group of people

00:23:10.700 | and didn't really interact outside of that as much.

00:23:13.940 | And then the joke was, you'd see like somebody moving

00:23:16.380 | a prototype through the quad with a black blanket over it.

00:23:20.860 | And that was 'cause it was secret, partly from Steve

00:23:24.260 | 'cause they didn't want Steve to see it until it was ready.

00:23:26.980 | - Yeah, the dynamic with Johnny Ive and Steve

00:23:30.500 | is interesting.

00:23:31.420 | It's like you don't wanna,

00:23:32.940 | he ruins as many ideas as he generates.

00:23:37.260 | It's a dangerous kind of line to walk.

00:23:42.140 | - If you have a lot of ideas,

00:23:43.540 | like Gordon Bell was famous for ideas, right?

00:23:47.300 | And it wasn't that the percentage of good ideas

00:23:49.140 | was way higher than anybody else.

00:23:51.460 | It was, he had so many ideas

00:23:53.180 | and he was also good at talking to people about it

00:23:55.860 | and getting the filters right and seeing through stuff.

00:24:00.220 | Whereas Elon was like, hey, I wanna build rockets.

00:24:03.380 | So Steve would hire a bunch of rocket guys

00:24:06.020 | and Elon would go read rocket manuals.

00:24:08.540 | - So Elon's a better engineer, a sense like,

00:24:11.500 | or like more like a love and passion for the manuals.

00:24:16.500 | - And the details, the data and the understanding.

00:24:19.220 | - The craftsmanship too, right?

00:24:20.840 | - Well, I guess you had craftsmanship too,

00:24:22.740 | but of a different kind.

00:24:24.300 | What do you make of the,

00:24:26.260 | just to stand in for just a little longer,

00:24:27.980 | what do you make of like the anger and the passion

00:24:29.940 | and all that, the firing and the mood swings

00:24:33.420 | and the madness, being emotional and all that,

00:24:38.420 | that's Steve and I guess Elon too.

00:24:40.740 | So what, is that a bug or a feature?

00:24:43.740 | - It's a feature.

00:24:45.060 | So there's a graph, which is y-axis productivity,

00:24:50.300 | x-axis at zero is chaos,

00:24:52.940 | and infinity is complete order.

00:24:54.800 | So as you go from the origin,

00:25:00.960 | as you improve order, you improve productivity.

00:25:04.200 | And at some point productivity peaks

00:25:06.480 | and then it goes back down again.

00:25:08.400 | Too much order, nothing can happen.

00:25:09.840 | - Yes, but the question is,

00:25:12.080 | how close to the chaos is that?

00:25:13.720 | - No, here's the thing,

00:25:15.040 | is once you start moving in a direction of order,

00:25:16.960 | the force vector to drive you towards order is unstoppable.

00:25:21.040 | - Oh, this is a slippery slope.

00:25:21.880 | - And every organization will move to the place

00:25:24.920 | where their productivity is stymied by order.

00:25:27.160 | - So you need a--

00:25:28.000 | - So the question is, who's the counterforce?

00:25:30.340 | Like, 'cause it also feels really good.

00:25:33.400 | As you get more organized and productivity goes up,

00:25:36.280 | the organization feels it, they orient towards it, right?

00:25:39.760 | They hire more people.

00:25:41.120 | They get more guys who can run process, you get bigger.

00:25:44.480 | Right, and then inevitably,

00:25:46.080 | the organization gets captured by the bureaucracy

00:25:50.040 | that manages all the processes.

00:25:51.860 | Right, and then humans really like that.

00:25:55.520 | And so if you just walk into a room and say,

00:25:57.920 | guys, love what you're doing,

00:26:00.120 | but I need you to have less order.

00:26:03.340 | If you don't have some force behind that,

00:26:06.920 | nothing will happen.

00:26:07.920 | - I can't tell you on how many levels that's profound.

00:26:11.760 | So--

00:26:12.600 | - So that's why I say it's a feature.

00:26:14.080 | Now, could you be nicer about it?

00:26:17.200 | I don't know, I don't know any good examples

00:26:18.920 | of being nicer about it.

00:26:20.120 | Well, the funny thing is to get stuff done,

00:26:23.520 | you need people who can manage stuff and manage people,

00:26:25.960 | 'cause humans are complicated.

00:26:26.920 | They need lots of care and feeding,

00:26:28.160 | and you need to tell them they look nice

00:26:29.880 | and they're doing good stuff, and pat 'em on the back.

00:26:32.280 | Right?

00:26:33.120 | - I don't know.

00:26:33.960 | You tell me.

00:26:34.800 | Is that needed?

00:26:36.000 | - Oh yeah. - Do humans need that?

00:26:37.080 | - I had a friend, he started a magic group,

00:26:38.680 | and he said, I figured it out.

00:26:40.860 | You have to praise them before they do anything.

00:26:43.440 | I was waiting 'til they were done,

00:26:45.240 | and they were always mad at me.

00:26:46.560 | Now I tell 'em what a great job they're doing

00:26:48.200 | while they're doing it.

00:26:49.400 | But then you get stuck in that trap,

00:26:51.080 | 'cause then when you're not doing something,

00:26:52.240 | how do you confront these people?

00:26:54.120 | - I think a lot of people that had trauma

00:26:55.960 | in their childhood would disagree with you,

00:26:57.560 | successful people, that you need to first do the rough stuff

00:27:00.680 | and then be nice later.

00:27:02.360 | I don't know.

00:27:03.200 | - Okay, but-- - Being nice--

00:27:04.400 | - Engineering companies are full of adults

00:27:05.840 | who had all kinds of wretched childhoods.

00:27:08.120 | You know.

00:27:08.960 | - I don't know. - Most people

00:27:09.800 | had okay childhoods.

00:27:11.440 | - Well, I don't know if--

00:27:12.280 | - And lots of people only work for praise, which is weird.

00:27:15.640 | - You mean like everybody?

00:27:17.000 | (laughing)

00:27:18.680 | - I'm not that interested in it, but.

00:27:21.160 | - Well, you're probably looking for somebody's approval.

00:27:24.060 | Even still.

00:27:27.440 | - Yeah, maybe.

00:27:28.280 | I should think about that.

00:27:29.560 | - Maybe somebody who's no longer with us kind of thing.

00:27:32.320 | I don't know.

00:27:34.160 | - I used to call up my dad and tell him what I was doing.

00:27:36.040 | He was very excited about engineering and stuff.

00:27:38.640 | - You got his approval?

00:27:40.400 | - Yeah, a lot.

00:27:42.120 | I was lucky.

00:27:42.960 | He decided I was smart and unusual as a kid,

00:27:47.240 | and that was okay when I was really young.

00:27:49.280 | So when I did poorly in school, I was dyslexic.

00:27:52.560 | I didn't read until I was third or fourth grade.

00:27:55.280 | They didn't care.

00:27:56.120 | My parents were like, "Oh, he'll be fine."

00:27:58.560 | So I was lucky.

00:28:01.560 | That was cool.

00:28:02.520 | - Is he still with us?

00:28:04.040 | You miss him?

00:28:06.040 | - Sure, yeah.

00:28:08.400 | He had Parkinson's and then cancer.

00:28:11.800 | His last 10 years were tough.

00:28:13.480 | And I killed him.

00:28:16.000 | Killing a man like that's hard.

00:28:18.280 | - The mind?

00:28:19.400 | - Well, it's pretty good.

00:28:21.480 | Parkinson's causes slow dementia,

00:28:23.760 | and the chemotherapy, I think, accelerated it.

00:28:27.720 | But it was like hallucinogenic dementia.

00:28:31.000 | So he was clever and funny and interesting,

00:28:34.160 | and it was pretty unusual.

00:28:37.920 | - Do you remember conversations from that time?

00:28:41.560 | Like what, do you have fond memories of the guy?

00:28:43.960 | - Yeah, oh yeah.

00:28:45.240 | - Anything come to mind?

00:28:46.440 | - A friend told me one time I could draw a computer

00:28:50.400 | on the whiteboard faster than anybody he'd ever met,

00:28:52.520 | and I said, "You should meet my dad."

00:28:54.960 | Like when I was a kid, he'd come home and say,

00:28:56.800 | "I was driving by this bridge, and I was thinking about it,

00:28:58.760 | "and he pulled out a piece of paper,

00:28:59.740 | "and he'd draw the whole bridge."

00:29:01.560 | He was a mechanical engineer.

00:29:03.680 | And he would just draw the whole thing,

00:29:05.040 | and then he would tell me about it,

00:29:06.320 | and tell me how he would've changed it.

00:29:08.720 | And he had this idea that he could understand

00:29:11.920 | and conceive anything.

00:29:13.440 | And I just grew up with that, so that was natural.

00:29:16.480 | So when I interview people, I ask them to draw a picture

00:29:19.820 | of something they did on a whiteboard,

00:29:21.800 | and it's really interesting.

00:29:22.900 | Like some people draw a little box,

00:29:24.800 | and then they'll say, "And it just talks to this."

00:29:27.840 | And I'll be like, "Oh, this is frustrating."

00:29:30.000 | And then I had this other guy come in one time,

00:29:31.760 | he says, "Well, I designed a floating point in this chip,

00:29:34.460 | "but I'd really like to tell you how the whole thing works,

00:29:36.280 | "and then tell you how the floating point works inside it.

00:29:38.140 | "Do you mind if I do that?"

00:29:39.080 | And he covered two whiteboards in like 30 minutes.

00:29:42.040 | And I hired him.

00:29:43.440 | He was great.

00:29:44.600 | - Just craftsman.

00:29:45.440 | I mean, that's the craftsmanship to that.

00:29:47.040 | - Yeah, but also the mental agility

00:29:49.480 | to understand the whole thing.

00:29:51.680 | Put the pieces in context.

00:29:53.600 | Real view of the balance of how the design worked.

00:29:57.720 | Because if you don't understand it properly,

00:30:01.040 | when you start to draw it,

00:30:02.200 | you'll fill up half the whiteboard

00:30:03.800 | with a little piece of it.

00:30:06.720 | Your ability to lay it out in an understandable way

00:30:09.260 | takes a lot of understanding.

00:30:11.500 | - And be able to sort of zoom into the detail

00:30:13.500 | and then zoom out to the big picture.

00:30:14.340 | - And zoom out really fast.

00:30:16.420 | - What about the impossible thing?

00:30:17.620 | You said your dad believed that you could do anything.

00:30:21.880 | That's a weird feature for a craftsman.

00:30:25.520 | - Yeah.

00:30:26.700 | - It seems that that echoes in your own behavior.

00:30:30.820 | Like that's the--

00:30:32.100 | - Well, it's not that anybody can do anything right now.

00:30:36.220 | It's that if you work at it, you can get better at it

00:30:39.660 | and there might not be a limit.

00:30:41.220 | And they did funny things.

00:30:44.620 | Like he always wanted to play piano.

00:30:46.140 | So at the end of his life, he started playing the piano.

00:30:48.460 | When he had Parkinson's and he was terrible.

00:30:51.580 | But he thought if he really worked at it in this life,

00:30:53.540 | maybe the next life he'd be better at it.

00:30:56.380 | - He might be onto something.

00:30:57.620 | - Yeah, indeed.

00:30:58.460 | (laughing)

00:30:59.700 | He enjoyed doing it.

00:31:00.940 | - Yeah.

00:31:01.780 | - So that's pretty funny.

00:31:02.940 | - Do you think the perfect is the enemy of the good

00:31:06.180 | in hardware and software engineering?

00:31:08.180 | It's like we were talking about JavaScript a little bit

00:31:10.500 | and the messiness of the 10 day building process.

00:31:14.780 | - Yeah, it's creative tension, right?

00:31:17.140 | So creative tension is you have two different ideas

00:31:21.460 | that you can't do both, right?

00:31:24.380 | And, but the fact that you wanna do both

00:31:27.660 | causes you to go try to solve that problem.

00:31:29.980 | That's the creative part.

00:31:32.020 | So if you're building computers,

00:31:35.140 | like some people say we have the schedule

00:31:37.060 | and anything that doesn't fit in the schedule we can't do.

00:31:40.220 | Right, so they throw out the perfect

00:31:42.100 | 'cause they have a schedule.

00:31:44.300 | I hate that.

00:31:45.140 | Then there's other people who say

00:31:48.220 | we need to get this perfectly right,

00:31:50.540 | and no matter what, you know, more people, more money, right?

00:31:54.540 | And there's a really clear idea about what you want.

00:31:57.860 | Some people are really good at articulating it.

00:32:00.700 | So let's call that the perfect, yeah.

00:32:02.340 | - Yeah.

00:32:03.260 | - All right, but that's also terrible

00:32:04.740 | 'cause they never ship anything,

00:32:06.180 | and you never hit any goals.

00:32:07.380 | So now you have your framework.

00:32:09.980 | - Yes.

00:32:10.820 | - You can't throw out stuff

00:32:11.660 | 'cause you can't get it done today,

00:32:12.780 | 'cause maybe you'll get it done tomorrow

00:32:14.020 | with the next project, right?

00:32:15.860 | You can't, so you have to,

00:32:18.340 | I work with a guy that I really like working with,

00:32:20.620 | but he over-filters his ideas.

00:32:23.180 | - Over-filters?

00:32:24.780 | - He'd start thinking about something,

00:32:26.620 | and as soon as he figured out what's wrong with it,

00:32:28.020 | he'd throw it out.

00:32:28.940 | And then I start thinking about it,

00:32:31.260 | and you know, you come up with an idea,

00:32:32.740 | and you find out what's wrong with it,

00:32:34.940 | and then you give it a little time to set,

00:32:36.740 | 'cause sometimes, you know, you figure out how to tweak it,

00:32:39.220 | or maybe that idea helps some other idea.

00:32:41.340 | So idea generation is really funny.

00:32:45.060 | So you have to give your ideas space.

00:32:46.900 | Like spaciousness of mind is key.

00:32:49.740 | But you also have to execute programs and get shit done.

00:32:53.380 | And then it turns out, computer engineering's fun

00:32:55.500 | because it takes, you know, a hundred people

00:32:56.900 | to build a computer, 200 or 300, whatever the number is,

00:33:00.580 | and people are so variable about, you know,

00:33:04.540 | temperament and, you know, skill sets and stuff,

00:33:07.660 | that in a big organization, you find the people

00:33:10.580 | who love the perfect ideas,

00:33:11.860 | and the people that wanna get stuff done yesterday,

00:33:13.740 | and people like to come up with ideas,

00:33:16.460 | and people like to, let's say, shoot down ideas,

00:33:19.260 | and it takes the whole, it takes a large group of people.

00:33:23.260 | - Some are good at generating ideas,

00:33:24.620 | some are good at filtering ideas,

00:33:25.980 | and then all, in that giant mess,

00:33:29.140 | you're somehow, I guess the goal is

00:33:32.100 | for that giant mess of people to find the perfect path

00:33:36.060 | through the tension, the creative tension.

00:33:38.460 | But like, how do you know when,

00:33:40.980 | you said there's some people good at articulating

00:33:42.920 | what perfect looks like, what a good design is.

00:33:44.740 | Like, if you're sitting in a room,

00:33:46.940 | and you have a set of ideas about, like,

00:33:51.580 | how to design a better processor,

00:33:55.300 | how do you know this is something special here,

00:33:58.820 | this is a good idea, let's try this.

00:34:00.820 | - Have you ever brainstormed an idea

00:34:02.260 | with a couple people that were really smart?

00:34:04.580 | And you kinda go into it, and you don't quite understand it,

00:34:07.540 | and you're working on it, and then you start, you know,

00:34:10.980 | talking about it, putting it on the whiteboard,

00:34:14.040 | maybe it takes days or weeks,

00:34:16.180 | and then your brains start to kinda synchronize.

00:34:18.660 | It's really weird.

00:34:19.580 | Like, you start to see what each other is thinking.

00:34:23.260 | And it starts to work.

00:34:28.460 | Like, you can see where, like, my talent

00:34:30.180 | in computer design is I can see how computers work

00:34:33.020 | in my head, like, really well.

00:34:35.340 | And I know other people can do that, too.

00:34:37.340 | And when you're working with people that can do that,

00:34:40.460 | like, it is kind of an amazing experience.

00:34:44.440 | And then, and every once in a while,

00:34:47.180 | you get to that place, and then you find the flaw,

00:34:49.420 | which is kinda funny, 'cause you can fool yourself.

00:34:52.360 | But--

00:34:54.080 | - The two of you kinda drifted along

00:34:56.420 | into a direction that was useless.

00:34:58.140 | - Yeah, that happens, too.

00:34:59.380 | Like, you have to, 'cause, you know,

00:35:01.860 | the nice thing about computer design,

00:35:04.180 | there's always reduction in practice.

00:35:05.580 | Like, you come up with your good ideas,

00:35:08.100 | and I know some architects who really love ideas,

00:35:11.140 | and then they work on 'em, and they put it on the shelf,

00:35:13.100 | and they go work on the next idea,

00:35:14.220 | and put it on the shelf, and they never reduce it

00:35:15.740 | to practice, so they find out what's good and bad.

00:35:18.780 | 'Cause almost every time I've done something really new,

00:35:22.500 | by the time it's done, like, the good parts are good,

00:35:25.660 | but I know all the flaws, like--

00:35:27.580 | - Yeah, would you say your career,

00:35:30.060 | just your own experience, is your career defined

00:35:32.500 | by, mostly by flaws or by successes?

00:35:35.220 | Like, if--

00:35:36.060 | - Again, there's great tension between those.

00:35:37.960 | If you haven't tried hard, right,

00:35:42.540 | and done something new, right,

00:35:46.260 | then you're not gonna be facing the challenges

00:35:48.500 | when you build it, then you find out

00:35:49.980 | all the problems with it, and--

00:35:52.620 | - But when you look back, do you see problems, or?

00:35:55.580 | Okay.

00:35:56.420 | - When I look back, I think earlier in my career,

00:36:00.500 | like, EV5 was the second alpha chip.

00:36:02.720 | I was so embarrassed about the mistakes,

00:36:06.540 | I could barely talk about it.

00:36:08.620 | And it was in the Guinness Book of World Records,

00:36:10.380 | and it was the fastest processor on the planet.

00:36:12.740 | So it was, and at some point I realized

00:36:15.780 | that was really a bad mental framework

00:36:18.540 | to deal with, like, doing something new.

00:36:20.020 | We did a bunch of new things, and some worked out great,

00:36:22.140 | and some were bad, and we learned a lot from it,

00:36:24.660 | and then the next one we learned a lot.

00:36:28.020 | That also, EV6 also had some really cool things in it.

00:36:31.820 | I think the proportion of good stuff went up,

00:36:34.240 | but it had a couple fatal flaws in it that were painful.

00:36:38.660 | And then, yeah.

00:36:41.500 | - You learned to channel the pain into, like, pride.

00:36:44.660 | - Not pride, really, just realization

00:36:48.540 | about how the world works,

00:36:50.060 | or how that kind of idea set works.

00:36:52.300 | - Life is suffering, that's the reality.

00:36:54.460 | - What-- - No, it's not.

00:36:56.340 | I know the Buddha said that,

00:36:58.420 | and a couple other people are stuck on it.

00:37:00.500 | No, it's, you know, there's this kind of weird combination

00:37:03.820 | of good and bad, and light and darkness

00:37:06.940 | that you have to tolerate and deal with.

00:37:10.260 | Yeah, there's definitely lots of suffering in the world.

00:37:12.620 | - Depends on the perspective.

00:37:13.740 | It seems like there's way more darkness,

00:37:15.420 | but that makes the light part really nice.

00:37:18.620 | What computing hardware,

00:37:23.020 | or just any kind of, even software design,

00:37:25.900 | are you, do you find beautiful?

00:37:28.740 | From your own work, from other people's work,

00:37:32.500 | that you're just, we were just talking about

00:37:35.180 | the battleground of flaws and mistakes and errors,

00:37:39.260 | but things that were just beautifully done.

00:37:42.540 | Is there something that pops to mind?

00:37:44.500 | - Well, when things are beautifully done,

00:37:47.900 | usually there's a well-thought-out set of abstraction layers.

00:37:53.600 | - So the whole thing works in unison nicely.

00:37:56.400 | - Yes, and when I say abstraction layer,

00:37:59.360 | that means two different components,

00:38:01.160 | when they work together, they work independently.

00:38:04.920 | They don't have to know what the other one is doing.

00:38:07.720 | - So that decoupling.

00:38:08.640 | - Yeah, so the famous one was the network stack.

00:38:11.680 | There's a seven-layer network stack,

00:38:13.080 | you know, data transport and protocol and all the layers,

00:38:16.400 | and the innovation was is when they really got that right,

00:38:20.000 | 'cause networks before that didn't define those very well,

00:38:22.920 | the layers could innovate independently,

00:38:26.200 | and occasionally the layer boundary would,

00:38:28.320 | you know, the interface would be upgraded.

00:38:30.960 | And that let, you know, the design space breathe.

00:38:34.720 | You could do something new in layer seven

00:38:37.800 | without having to worry about how layer four worked.

00:38:40.600 | And so good design does that,

00:38:42.960 | and you see it in processor designs.

00:38:45.220 | When we did the Zen design at AMD,

00:38:48.560 | we made several components very modular.

00:38:51.920 | And, you know, my insistence at the top was

00:38:54.680 | I wanted all the interfaces defined

00:38:56.600 | before we wrote the RTL for the pieces.

00:38:59.320 | One of the verification leads said,

00:39:00.960 | "If we do this right, I can test the pieces

00:39:03.520 | "so well independently when we put it together,

00:39:06.360 | "we won't find all these interaction bugs

00:39:08.040 | "'cause the floating point knows how the cache works."

00:39:10.720 | And I was a little skeptical, but he was mostly right.

00:39:14.200 | That the modularity of the design

00:39:16.680 | greatly improved the quality.

00:39:18.960 | - Is that universally true in general?

00:39:20.520 | Would you say about good designs,

00:39:21.880 | the modularity is like usually--

00:39:24.200 | - Well, we talked about this before.

00:39:25.160 | Humans are only so smart,

00:39:27.000 | and we're not getting any smarter, right?

00:39:29.440 | But the complexity of things is going up.

00:39:32.240 | So, you know, a beautiful design can't be bigger

00:39:36.200 | than the person doing it.

00:39:37.960 | It's just, you know, their piece of it.

00:39:40.000 | Like the odds of you doing a really beautiful design

00:39:42.440 | with something that's way too hard for you is low, right?

00:39:46.640 | If it's way too simple for you, it's not that interesting.

00:39:49.120 | It's like, well, anybody could do that.

00:39:50.680 | But when you get the right match of your expertise

00:39:54.800 | and, you know, mental power to the right design size,

00:39:58.760 | that's cool, but that's not big enough

00:40:00.480 | to make a meaningful impact in the world.

00:40:02.320 | So now you have to have some framework

00:40:05.000 | to design the pieces so that the whole thing

00:40:08.160 | is big and harmonious, but, you know,

00:40:12.440 | when you put it together, it's sufficiently interesting

00:40:16.840 | to be used, and so that's what a beautiful design is.

00:40:21.400 | - Matching the limits of that human cognitive capacity

00:40:28.000 | to the module you can create,

00:40:30.320 | and creating a nice interface between those modules,

00:40:33.120 | and thereby, do you think there's a limit

00:40:34.520 | to the kind of beautiful complex systems

00:40:37.120 | we can build with this kind of modular design?

00:40:41.000 | It's like, you know, we build increasingly more complicated,

00:40:46.000 | you can think of like the internet.

00:40:48.280 | Okay, let's scale it down.

00:40:50.920 | You can think of like social network,

00:40:52.320 | like Twitter as one computing system.

00:40:56.320 | But those are little modules, right?

00:41:00.760 | - But it's built on so many components

00:41:03.800 | nobody at Twitter even understands.

00:41:06.000 | - Right.

00:41:06.840 | - So if an alien showed up and looked at Twitter,

00:41:09.280 | he wouldn't just see Twitter as a beautiful,

00:41:11.160 | simple thing that everybody uses, which is really big.

00:41:14.400 | You would see the network it runs on,

00:41:17.360 | the fiber optics, the data's transported,

00:41:19.200 | the computers, the whole thing is so bloody complicated,

00:41:22.020 | nobody at Twitter understands it.

00:41:23.720 | And so--

00:41:24.560 | - I think that's what the alien would see.

00:41:25.720 | So yeah, if an alien showed up and looked at Twitter,

00:41:28.760 | or looked at the various different networked systems

00:41:32.040 | that you could see on Earth.

00:41:33.640 | - So imagine they were really smart

00:41:34.960 | and they could comprehend the whole thing.

00:41:36.680 | And then they sort of evaluated the human

00:41:40.120 | and thought, this is really interesting,

00:41:41.560 | no human on this planet comprehends the system they built.

00:41:44.660 | - No individual, well, would they even see individual humans?

00:41:48.880 | Like we humans are very human-centric, entity-centric,

00:41:52.720 | and so we think of us as the central organism

00:41:56.840 | and the networks as just a connection of organisms.

00:41:59.800 | But from a perspective of, from an outside perspective,

00:42:04.000 | it seems like we're just one organism.

00:42:06.200 | - Yeah, I get it.

00:42:07.040 | We're the ants and they'd see the ant colony.

00:42:08.960 | - The ant colony, yeah.

00:42:10.480 | Or the result of production of the ant colony,

00:42:12.760 | which is like cities and it's,

00:42:16.000 | in that sense, humans are pretty impressive.

00:42:19.880 | The modularity that we're able to,

00:42:21.840 | and how robust we are to noise and mutation,

00:42:25.960 | all that kind of stuff.

00:42:26.800 | - Well, that's 'cause it's stress-tested all the time.

00:42:28.520 | - Yeah.

00:42:29.360 | - You know, you build all these cities with buildings

00:42:31.080 | and you get earthquakes occasionally,

00:42:32.400 | and some wars, earthquakes.

00:42:35.560 | Viruses every once in a while.

00:42:37.640 | - You know, changes in business plans for,

00:42:39.960 | you know, like shipping or something.

00:42:41.640 | Like, as long as it's all stress-tested,

00:42:44.760 | then it keeps adapting to the situation.

00:42:48.560 | So, it's a curious phenomenon.

00:42:52.560 | - Well, let's go, let's talk about Moore's Law a little bit.

00:42:55.560 | At the broad view of Moore's Law,

00:43:00.200 | where it's just exponential improvement

00:43:01.680 | of computing capability.

00:43:05.320 | Like, OpenAI, for example, recently published

00:43:08.440 | this kind of paper,

00:43:11.240 | it's looking at the exponential improvement

00:43:14.120 | in the training efficiency of neural networks.

00:43:17.080 | For like ImageNet and all that kind of stuff,

00:43:18.680 | we just got better on this,

00:43:20.040 | this is purely software side,

00:43:22.360 | just figuring out better tricks and algorithms

00:43:25.680 | for training neural networks.

00:43:27.040 | And that seems to be improving significantly faster

00:43:30.680 | than the Moore's Law prediction, you know?

00:43:33.160 | So, that's in the software space.

00:43:35.320 | What do you think, if Moore's Law continues,

00:43:39.200 | or if the general version of Moore's Law continues,

00:43:42.920 | do you think that comes mostly from the hardware,

00:43:45.360 | from the software, some mix of the two,

00:43:47.600 | some interesting, totally,

00:43:50.040 | so not the reduction of the size of the transistor

00:43:52.840 | kind of thing, but more in the totally interesting

00:43:57.840 | kinds of innovations in the hardware space,

00:43:59.880 | all that kind of stuff?

00:44:01.240 | - Well, there's like a half a dozen things

00:44:04.040 | going on in that graph.

00:44:05.520 | So, one is, there's initial innovations

00:44:08.440 | that had a lot of headroom to be exploited.

00:44:11.640 | So, you know, the efficiency of the networks

00:44:13.920 | has improved dramatically.

00:44:15.880 | And then, the decomposability of those,

00:44:18.560 | and the use, you know, they started running

00:44:20.560 | on one computer, then multiple computers,

00:44:22.480 | then multiple GPUs, and then arrays of GPUs,

00:44:25.040 | and they're up to thousands.

00:44:27.080 | And at some point, so it's sort of like,

00:44:30.600 | they were going from like a single computer application

00:44:33.840 | to a thousand computer application.

00:44:36.200 | So, that's not really a Moore's Law thing,

00:44:38.160 | that's an independent vector.

00:44:39.480 | How many computers can I put on this problem?

00:44:42.280 | 'Cause the computers themselves are getting better

00:44:44.200 | on like a Moore's Law rate, but their ability

00:44:46.720 | to go from one to 10, to 100, to 1,000,

00:44:49.440 | - Yeah. - you know, is something.

00:44:51.160 | And then, multiplied by, you know,

00:44:53.280 | the amount of computes it took to resolve

00:44:55.280 | like AlexNet to ResNet to Transformers.

00:44:58.280 | It's been quite, you know, steady improvements.

00:45:01.680 | - But those are like S-curves, aren't they?

00:45:03.120 | - Yeah. - That's the exactly kind

00:45:04.240 | of S-curves that are underlying Moore's Law

00:45:06.160 | from the very beginning.

00:45:07.600 | - So. - So, what's the biggest,

00:45:09.880 | what's the most productive, rich source of S-curves

00:45:14.880 | in the future, do you think?

00:45:16.760 | Is it hardware, is it software, or is it--

00:45:18.760 | - So, hardware is gonna move along relatively slowly.

00:45:23.600 | Like, you know, double performance every two years.

00:45:26.640 | There's still-- - I like how you call

00:45:29.120 | that slow. - Yeah, that's the slow version.

00:45:31.440 | The snail's pace of Moore's Law, maybe we should,

00:45:33.880 | we should, we should trademark that one.

00:45:37.520 | Whereas, the scaling by number of computers,

00:45:41.480 | you know, can go much faster, you know.

00:45:43.960 | I'm sure at some point, Google had a, you know,

00:45:46.320 | their initial search engine was running on a laptop,

00:45:48.880 | you know, like. - Yeah.

00:45:50.120 | - And at some point, they really worked on scaling that,

00:45:52.520 | and then they factored the indexer from, you know,

00:45:55.880 | this piece, and this piece, and this piece,

00:45:57.440 | and they spread the data on more and more things,

00:45:59.280 | and, you know, they did a dozen innovations.

00:46:02.760 | But as they scaled up the number of computers on that,

00:46:05.360 | it kept breaking, finding new bottlenecks in their software

00:46:08.280 | and their schedulers, and made 'em rethink,

00:46:11.720 | like, it seems insane to do a scheduler

00:46:13.920 | across a thousand computers, to schedule parts of it,

00:46:16.720 | and then send the results to one computer.

00:46:19.000 | But if you wanna schedule a million searches,

00:46:21.400 | that makes perfect sense.

00:46:23.200 | So, there's, the scaling by just quantity

00:46:26.840 | is probably the richest thing.

00:46:28.960 | But then, as you scale quantity,

00:46:31.960 | like a network that was great on a hundred computers

00:46:34.680 | may be completely the wrong one.

00:46:36.560 | You may pick a network that's 10 times slower

00:46:39.620 | on 10,000 computers, like per computer.

00:46:42.520 | But if you go from a hundred to 10,000, that's a hundred times.

00:46:45.800 | So, that's one of the things that happened

00:46:47.240 | when we did internet scaling, is the efficiency went down.

00:46:51.440 | Not up.

00:46:52.560 | The future of computing is inefficiency, not efficiency.

00:46:55.520 | - But scale, inefficient scale.

00:46:57.600 | - It's scaling faster than inefficiency bites you.

00:47:01.840 | And as long as there's dollar value there,

00:47:03.840 | like, scaling costs lots of money.

00:47:06.000 | But Google showed, Facebook showed, everybody showed

00:47:08.200 | that the scale was where the money was at.

00:47:10.560 | - And so, it was worth it financially.

00:47:13.800 | Do you think, is it possible that basically

00:47:17.760 | the entirety of Earth will be like a computing surface?

00:47:21.800 | Like, this table will be doing computing,

00:47:24.440 | this hedgehog will be doing computing,

00:47:26.120 | like everything really inefficient,

00:47:28.160 | dumb computing will be whatever.

00:47:29.000 | - The science fiction books, they call it computronium.

00:47:31.800 | - Computronium?

00:47:32.640 | - We turn everything into computing.

00:47:34.680 | Well, most of the elements aren't very good for anything.

00:47:38.000 | Like, you're not gonna make a computer out of iron.

00:47:39.920 | Like, you know, silicon and carbon have nice structures.

00:47:45.440 | - Well, we'll see what you can do with the rest of it.

00:47:48.640 | People talk about, well, maybe we can turn the sun

00:47:50.400 | into a computer, but it's hydrogen.

00:47:52.520 | And a little bit of helium, so.

00:47:55.800 | - What I mean is more like actually

00:47:57.960 | just adding computers to everything.

00:47:59.920 | - Oh, okay.

00:48:00.760 | So you're just converting all the mass

00:48:02.560 | of the universe into a computer?

00:48:04.240 | - No, no, no, so not using--

00:48:05.760 | - That'd be ironic from the simulation point of view,

00:48:07.600 | is like, the simulator built mass to simulate, like.

00:48:11.160 | - Yeah, I mean, yeah, so, I mean, ultimately

00:48:14.120 | this is all heading towards a simulation, yes.

00:48:15.960 | - Yeah, well, I think I might have told you this story.

00:48:18.440 | At Tesla, they were deciding, so they wanna measure

00:48:20.960 | the current coming out of the battery,

00:48:22.400 | and they decide between putting a resistor in there

00:48:24.960 | and putting a computer with a sensor in there.

00:48:29.360 | And the computer was faster than the computer

00:48:31.960 | I worked on in 1982.

00:48:34.160 | And we chose the computer 'cause it was cheaper

00:48:36.320 | than the resistor.

00:48:37.340 | So, sure, this hedgehog, you know, it costs $13,

00:48:42.320 | and we can put an AI that's as smart as you

00:48:45.160 | in there for five bucks, it'll have one.

00:48:47.160 | So computers will be everywhere.

00:48:51.800 | - I was hoping it wouldn't be smarter than me, because--

00:48:54.640 | - Well, everything's gonna be smarter than you.

00:48:56.680 | - But you were saying it's inefficient.

00:48:58.040 | I thought it was better to have a lot of dumb things.

00:49:00.240 | - Well, Moore's Law will slowly compact that stuff.

00:49:02.760 | - So even the dumb things will be smarter than us?

00:49:04.880 | - The dumb things are gonna be smart,

00:49:06.000 | or they're gonna be smart enough to talk

00:49:07.560 | to something that's really smart.

00:49:10.520 | It's like, well, just remember, a big computer chip,

00:49:15.520 | it's like an inch by an inch, and 40 microns thick.

00:49:20.120 | It doesn't take very many atoms to make

00:49:23.600 | a high-power computer, and 10,000 of 'em

00:49:26.440 | can fit in a shoebox.

00:49:27.800 | But you have the cooling and power problems,

00:49:31.480 | but people are working on that.

00:49:33.520 | - But they still can't write compelling poetry or music

00:49:37.640 | or understand what love is or have a fear of mortality,

00:49:41.680 | so we're still winning.

00:49:43.480 | - Neither can most of humanity.

00:49:45.040 | - Well, they can write books about it.

00:49:48.080 | But speaking about this walk along the path of innovation

00:49:55.880 | towards the dumb things being smarter than humans,

00:50:00.060 | you are now the CTO of Tenstor, as of two months ago.

00:50:05.060 | They build hardware for deep learning.

00:50:08.060 | How do you build scalable and efficient deep learning?

00:50:12.260 | This is such a fascinating space.

00:50:13.580 | - Yeah, yeah, so it's interesting.

00:50:15.020 | So up until recently, I thought there was

00:50:17.380 | two kinds of computers.

00:50:18.380 | There are serial computers that run like C programs,

00:50:21.380 | and then there's parallel computers.

00:50:23.100 | So the way I think about it is,

00:50:25.180 | parallel computers have given parallelism.

00:50:27.940 | Like, GPUs are great 'cause you have a million pixels,

00:50:30.740 | and modern computers are great 'cause you have

00:50:32.980 | a million pixels, and modern GPUs run a program

00:50:36.020 | on every pixel.

00:50:37.100 | They call it the shader program.

00:50:38.700 | Right, so, or like finite element analysis.

00:50:42.060 | You build something, you make this into little tiny chunks,

00:50:45.140 | you give each chunk to a computer,

00:50:46.660 | so you're given all these chunks,

00:50:47.980 | you have parallelism like that.

00:50:49.780 | But most C programs, you write this linear narrative,

00:50:53.140 | and you have to make it go fast.

00:50:55.180 | To make it go fast, you predict all the branches,

00:50:57.260 | all the data fetches, and you run that more in parallel,

00:50:59.820 | but that's found parallelism.

00:51:02.580 | So, AI is, I'm still trying to decide

00:51:06.900 | how fundamental this is.

00:51:08.420 | It's a given parallelism problem.

00:51:10.900 | But the way people describe the neural networks,

00:51:14.780 | and then how they write them in PyTorch, it makes graphs.

00:51:17.900 | - Yeah, that might be fundamentally different

00:51:19.980 | than the GPU kind of--

00:51:21.660 | - Parallelism, yeah, it might be.

00:51:23.300 | Because when you run the GPU program on all the pixels,

00:51:27.300 | you're running, you know, it depends,

00:51:29.500 | you know, this group of pixels say it's background blue,

00:51:32.540 | and it runs a really simple program.

00:51:34.020 | This pixel is, you know, some patch of your face,

00:51:36.940 | so you have some really interesting shader program

00:51:39.540 | to give you an impression of translucency.

00:51:41.740 | But the pixels themselves don't talk to each other.

00:51:43.980 | There's no graph, right?

00:51:46.620 | So, you do the image, and then you do the next image,

00:51:49.580 | and you do the next image,

00:51:51.300 | and you run eight million pixels,

00:51:53.860 | eight million programs every time,

00:51:55.620 | and modern GPUs have like 6,000 thread engines in them.

00:51:59.620 | So, you know, to get eight million pixels,

00:52:02.100 | each one runs a program on, you know, 10 or 20 pixels,

00:52:06.140 | and that's how they work, but there's no graph.

00:52:09.380 | - But you think graph might be a totally new way

00:52:13.700 | to think about hardware?

00:52:14.900 | - So, Rajagirdar and I have been having

00:52:17.100 | this good conversation about given versus found parallelism,

00:52:20.580 | and then the kind of walk, 'cause we got more transistors,

00:52:23.860 | like, you know, computers way back when

00:52:25.660 | did stuff on scalar data.

00:52:27.820 | Now we did it on vector data, famous vector machines.

00:52:30.740 | Now we're making computers that operate on matrices, right?

00:52:34.500 | And then the category we said was next was spatial.

00:52:38.900 | Like, imagine you have so much data that, you know,

00:52:41.700 | you wanna do the compute on this data,

00:52:43.420 | and then when it's done, it says,

00:52:45.940 | send the result to this pile of data,

00:52:47.580 | run some software on that.

00:52:49.260 | And it's better to think about it spatially

00:52:53.060 | than to move all the data to a central processor

00:52:56.140 | and do all the work.

00:52:57.580 | - So, spatially, you mean moving in the space of data

00:53:00.700 | as opposed to moving the data?

00:53:02.460 | - Yeah, you have a petabyte data space

00:53:05.340 | spread across some huge array of computers,

00:53:08.620 | and when you do a computation somewhere,

00:53:10.540 | you send the result of that computation

00:53:12.260 | or maybe a pointer to the next program

00:53:14.340 | to some other piece of data and do it.

00:53:16.620 | But I think a better word might be graph,

00:53:18.780 | and all the AI neural networks are graphs.

00:53:21.660 | Do some computations, send the result here,

00:53:24.020 | do another computation, do a data transformation,

00:53:26.380 | do a merging, do a pooling, do another computation.

00:53:30.300 | - Is it possible to compress and say

00:53:32.220 | how we make this thing efficient,

00:53:34.500 | this whole process efficient, this different?

00:53:37.220 | - So first, the fundamental elements in the graphs

00:53:40.860 | are things like matrix multiplies, convolutions,

00:53:43.140 | data manipulations, and data movements.

00:53:45.540 | - Yeah.

00:53:46.380 | - So GPUs emulate those things with their little singles,

00:53:49.580 | you know, basically running a single-threaded program.

00:53:52.460 | - Yeah.

00:53:53.300 | - And then there's, you know,

00:53:54.500 | NVIDIA calls it a warp, where they group

00:53:56.060 | a bunch of programs that are similar together,

00:53:58.380 | so for efficiency and instruction use.

00:54:01.540 | And then at a higher level, you kind of,

00:54:03.980 | you take this graph and you say this part of the graph

00:54:06.060 | is a matrix multiplier, which runs on these 32 threads.

00:54:09.820 | But the model at the bottom was built

00:54:12.620 | for running programs on pixels, not executing graphs.

00:54:17.100 | - So it's emulation, ultimately.

00:54:18.980 | - Yes.

00:54:19.820 | - So is it possible to build something

00:54:21.060 | that natively runs graphs?

00:54:23.020 | - Yes, so that's what Tenstorrent did.

00:54:24.940 | So--

00:54:27.060 | - Where are we on that?

00:54:28.180 | How, like, in the history of that effort,

00:54:30.900 | are we in the early days?

00:54:32.060 | - Yeah, I think so.

00:54:33.380 | Tenstorrent started by a friend of mine,

00:54:35.700 | Lobija Bajek, and I was his first investor.

00:54:38.980 | So I've been, you know, kind of following him

00:54:41.620 | and talking to him about it for years.

00:54:43.540 | And in the fall, when I was considering things to do,

00:54:46.960 | I decided, you know, we held a conference last year

00:54:51.580 | with a friend who organized it,

00:54:53.100 | and we wanted to bring in thinkers.

00:54:56.140 | And two of the people were Andrej Karpathy

00:54:59.140 | and Chris Lattner.

00:55:00.460 | And Andrej gave this talk, it's on YouTube,

00:55:03.380 | called Software 2.0, which I think is great.

00:55:06.820 | Which is, we went from programmed computers,

00:55:10.160 | where you write programs, to data program computers.

00:55:13.780 | You know, like the future is, you know,

00:55:15.980 | of software is data programs, the networks.

00:55:19.340 | And I think that's true.

00:55:21.340 | And then Chris has been working, he worked on LLVM,

00:55:25.220 | the low-level virtual machine,

00:55:26.580 | which became the intermediate representation

00:55:29.060 | for all compilers.

00:55:31.340 | And now he's working on another project called MLIR,

00:55:33.620 | which is mid-level intermediate representation,

00:55:36.420 | which is essentially under the graph

00:55:39.820 | about how do you represent that kind of computation

00:55:42.780 | and then coordinate large numbers

00:55:44.300 | of potentially heterogeneous computers.

00:55:46.640 | And I would say technically Tenstorrent's

00:55:51.500 | you know, two pillars of those two ideas,

00:55:54.900 | software 2.0 and mid-level representation.

00:55:58.300 | But it's in service of executing graph programs.

00:56:01.900 | The hardware is designed to do that.

00:56:03.820 | - So it's including the hardware piece.

00:56:05.540 | - Yeah.

00:56:06.460 | And then the other cool thing is,

00:56:08.500 | for a relatively small amount of money,

00:56:10.100 | they did a test chip and two production chips.

00:56:13.340 | So it's like a super effective team.

00:56:15.340 | And unlike some AI startups,

00:56:18.780 | if you don't build the hardware to run the software

00:56:21.180 | that they really want to do,

00:56:22.900 | then you have to fix it by writing lots more software.

00:56:26.060 | So the hardware naturally does matrix multiply,

00:56:29.100 | convolution, the data manipulations,

00:56:31.820 | and the data movement between processing elements

00:56:35.980 | that you can see in the graph,

00:56:37.580 | which I think is all pretty clever.

00:56:40.580 | And that's what I'm working on now.

00:56:45.100 | - So I think it's called the Grace Call processor

00:56:49.060 | introduced last year.

00:56:51.340 | It's, you know, there's a bunch of measures of performance.

00:56:53.860 | We're talking about horses.

00:56:55.580 | It seems to outperform 368 trillion operations per second.

00:56:59.940 | Seems to outperform NVIDIA's Tesla T4 system.

00:57:03.300 | So these are just numbers.

00:57:04.700 | What do they actually mean in real world performance?

00:57:07.660 | Like what are the metrics for you that you're chasing

00:57:11.220 | in your horse race?

00:57:12.460 | Like what do you care about?

00:57:13.900 | - Well, first, so the native language of, you know,

00:57:17.860 | people who write AI network programs is PyTorch now.

00:57:21.380 | PyTorch, TensorFlow, there's a couple others.

00:57:23.980 | - Do you think PyTorch has won over TensorFlow?

00:57:25.740 | Or is it just-

00:57:26.580 | - I'm not an expert on that.

00:57:27.940 | I know many people who have switched

00:57:29.740 | from TensorFlow to PyTorch.

00:57:31.340 | - Yeah.

00:57:32.160 | - And there's technical reasons for it.

00:57:33.340 | - I use both.

00:57:34.700 | Both are still awesome.

00:57:35.860 | - Both are still awesome.

00:57:37.100 | - But the deepest love is for PyTorch currently.

00:57:39.820 | - Yeah, there's more love for that.

00:57:41.340 | And that may change.

00:57:42.580 | So the first thing is when they write their programs,

00:57:46.640 | can the hardware execute it pretty much as it was written?

00:57:50.460 | Right, so PyTorch turns into a graph.

00:57:53.300 | We have a graph compiler that makes that graph.

00:57:55.540 | Then it fractions the graph down.

00:57:57.440 | So if you have big matrix multiply,

00:57:58.780 | we turn it into right-sized chunks

00:58:00.100 | to run on the processing elements.

00:58:02.140 | It hooks all the graph up.

00:58:03.260 | It lays out all the data.

00:58:05.100 | There's a couple of mid-level representations of it

00:58:08.020 | that are also simulatable.

00:58:09.400 | So that if you're writing the code,

00:58:12.140 | you can see how it's gonna go through the machine,

00:58:15.100 | which is pretty cool.

00:58:15.940 | And then at the bottom, it schedules kernels

00:58:17.740 | like math, data manipulation, data movement kernels,

00:58:21.820 | which do this stuff.

00:58:22.860 | So we don't have to run,

00:58:25.180 | write a little program to do matrix multiply

00:58:27.340 | 'cause we have a big matrix multiplier.

00:58:28.980 | Like there's no SIMD program for that,

00:58:31.260 | but there is scheduling for that, right?

00:58:35.980 | So one of the goals is if you write a piece of PyTorch code

00:58:40.180 | that looks pretty reasonable,

00:58:41.260 | you should be able to compile it, run it on the hardware

00:58:43.460 | without having to tweak it

00:58:44.740 | and do all kinds of crazy things to get performance.

00:58:48.060 | - There's not a lot of intermediate steps.

00:58:50.060 | It's running directly as written.

00:58:51.260 | - Like on a GPU,

00:58:52.180 | if you write a large matrix multiply naively,

00:58:54.580 | you'll get five to 10% of the peak performance of the GPU.

00:58:57.640 | Right, and then there's a bunch of people

00:59:00.460 | who published papers on this,

00:59:01.580 | and I read them about what steps do you have to do.

00:59:04.060 | And it goes from pretty reasonable,

00:59:06.740 | well, transpose one of the matrices.

00:59:08.460 | So you do row order, not column ordered.

00:59:10.840 | You know, block it so that you can put a block

00:59:13.960 | of the matrix on different SMs, you know, groups of threads.

00:59:17.700 | But some of it gets into little details,

00:59:21.120 | like you have to schedule it just so,

00:59:22.960 | so you don't have register conflicts.

00:59:25.000 | So they call them CUDA ninjas.

00:59:28.200 | - CUDA ninjas, I love it.

00:59:31.080 | - To get to the optimal point,

00:59:32.320 | you either use a pre-written library,

00:59:36.060 | which is a good strategy for some things,

00:59:37.880 | or you have to be an expert in microarchitecture

00:59:40.920 | to program it.

00:59:42.200 | - Right, so the optimization step

00:59:43.520 | is way more complicated with the GPU.

00:59:45.000 | - So our goal is if you write PyTorch,

00:59:47.880 | that's good PyTorch, you can do it.

00:59:49.560 | Now there's, as the networks are evolving,

00:59:52.840 | you know, they've changed from convolutional

00:59:54.600 | to matrix multiply.

00:59:56.440 | People are talking about conditional graphs,

00:59:58.040 | they're talking about very large matrices,

00:59:59.800 | they're talking about sparsity.

01:00:01.700 | They're talking about problems that scale

01:00:03.980 | across many, many chips.

01:00:06.120 | So the native, you know, data item is a packet.

01:00:11.120 | Like, so you send a packet to a processor,

01:00:13.320 | it gets processed, it does a bunch of work,

01:00:15.400 | and then it may send packets to other processors,

01:00:17.640 | and they execute in like a data flow graph

01:00:20.520 | kind of methodology.

01:00:22.080 | - Got it.

01:00:22.920 | - We have a big network on chip,

01:00:24.400 | and then 16, the next second chip has 16 ethernet ports

01:00:27.760 | to hook lots of them together,

01:00:29.560 | and it's the same graph compiler across multiple chips.

01:00:32.400 | - So that's where the scale comes in.

01:00:33.600 | - So it's built to scale naturally.

01:00:35.120 | Now, my experience with scaling is as you scale,

01:00:38.180 | you run into lots of interesting problems.

01:00:40.760 | So scaling is a mountain to climb.

01:00:43.200 | - Yeah.

01:00:44.040 | - So the hardware is built to do this,

01:00:45.000 | and then we're in the process of--

01:00:47.680 | - Is there a software part to this,

01:00:49.120 | with ethernet and all that?

01:00:51.640 | - Well, the protocol at the bottom,

01:00:54.520 | you know, we send, it's an ethernet PHY,

01:00:57.640 | but the protocol basically says,

01:00:59.760 | send the packet from here to there,

01:01:01.440 | it's all point to point.

01:01:03.120 | The header bit says which processor to send it to,

01:01:05.840 | and we basically take a packet off our on-chip network,

01:01:09.560 | put an ethernet header on it, send it to the other end,

01:01:13.000 | strip the header off and send it to the local thing.

01:01:14.880 | It's pretty straightforward.

01:01:16.120 | - Human to human interaction is pretty straightforward too,

01:01:18.160 | but when you get a million of us,

01:01:19.360 | we could do some crazy stuff together.

01:01:21.440 | - Yeah, it could be fun.

01:01:23.360 | - So is that the goal, is scale?

01:01:25.840 | So like, for example, I have been recently

01:01:28.360 | doing a bunch of robots at home

01:01:30.080 | for my own personal pleasure.

01:01:32.320 | Am I going to ever use Tenstor or is this more for?

01:01:35.760 | - There's all kinds of problems,

01:01:37.200 | like there's small inference problems

01:01:38.720 | or small training problems or big training problems.

01:01:41.440 | - What's the big goal?

01:01:42.680 | Is it the big inference training problems

01:01:45.080 | or the small training problems?

01:01:46.760 | - One of the goals is to scale from 100 milliwatts

01:01:49.120 | to a megawatt, you know,

01:01:51.720 | so like really have some range on the problems

01:01:54.840 | and the same kind of AI programs work

01:01:57.360 | at all different levels.

01:01:59.320 | So that's the goal.

01:02:01.240 | Since the natural data item is a packet

01:02:03.600 | that we can move around, it's built to scale,

01:02:06.680 | but so many people have, you know, small problems.

01:02:11.560 | - Right.

01:02:12.400 | - Right, but, you know--

01:02:13.240 | - Like inside that phone is a small problem to solve.

01:02:16.360 | So do you see Tenstor potentially being inside a phone?

01:02:19.960 | - Well, the power efficiency of local memory,

01:02:22.600 | local computation and the way we built it is pretty good.

01:02:26.320 | And then there's a lot of efficiency

01:02:28.480 | on being able to do conditional graphs and sparsity.

01:02:31.480 | I think for complicated networks,

01:02:34.520 | I want to go in a small factor, it's quite good,

01:02:36.920 | but we have to prove that that's a fun problem.

01:02:40.760 | - And that's the early days of the company, right?

01:02:42.240 | It's a couple of years, you said?

01:02:44.600 | But you think you invested, you think they're legit

01:02:47.520 | and so you join.

01:02:48.360 | - Yeah, I do.

01:02:49.960 | Well, it's also, it's a really interesting place to be.

01:02:53.240 | Like the AI world is exploding, you know,

01:02:55.680 | and I looked at some other opportunities

01:02:58.480 | like build a faster processor, which people want,

01:03:01.480 | but that's more on an incremental path

01:03:03.720 | than what's gonna happen in AI in the next 10 years.

01:03:07.800 | So this is kind of, you know,

01:03:10.040 | an exciting place to be part of.

01:03:12.200 | - The revolutions will be happening

01:03:14.040 | in the very space that Tenstor is.

01:03:14.880 | - And then lots of people are working on it,

01:03:16.640 | but there's lots of technical reasons

01:03:18.120 | why some of them, you know, aren't gonna work out that well.

01:03:21.480 | And, you know, that's interesting.

01:03:23.640 | And there's also the same problem

01:03:25.840 | about getting the basics right.

01:03:27.520 | Like we've talked to customers about exciting features.

01:03:30.160 | And at some point we realized that,

01:03:32.080 | we should have realized,

01:03:33.920 | they wanna hear first about memory bandwidth,

01:03:35.880 | local bandwidth, compute intensity, programmability.

01:03:39.240 | They want to know the basics, power management,

01:03:42.000 | how the network ports work, what are the basics,

01:03:44.120 | do all the basics work?

01:03:46.120 | 'Cause it's easy to say, we've got this great idea,

01:03:48.000 | you know, the crack GPT-3.

01:03:51.080 | But the people we talked to wanna say,

01:03:54.080 | if I buy the, so we have a PCI Express card

01:03:57.520 | with our chip on it.

01:03:58.680 | If you buy the card, you plug it in your machine,

01:04:00.880 | you download the driver,

01:04:01.960 | how long does it take me to get my network to run?

01:04:05.080 | - Right.

01:04:05.920 | - Right, you know, that's a real question.

01:04:06.960 | - It's a very basic question.

01:04:08.360 | - So.

01:04:09.200 | - Yeah, is there an answer to that yet?

01:04:10.520 | Or is it trying to get to--

01:04:11.360 | - Our goal is like an hour.

01:04:13.440 | - Okay, when can I buy a Tenstor?

01:04:15.760 | - Pretty soon.

01:04:17.640 | - For my, for the small case training.

01:04:19.640 | - Yeah, pretty soon, months.

01:04:22.000 | - Good, I love the idea of you inside a room

01:04:24.800 | with Karpathy, Andre Karpathy and Chris Lautner.

01:04:29.200 | Very, very interesting, very brilliant people,

01:04:36.000 | very out of the box thinkers,

01:04:37.600 | but also like first principles thinkers.

01:04:40.000 | - Well, they both get stuff done.

01:04:42.680 | They only get stuff done to get their own projects done.

01:04:44.960 | They talk about it clearly,

01:04:47.040 | they educate large numbers of people

01:04:48.760 | and they've created platforms for other people

01:04:50.560 | to go do their stuff on.

01:04:52.040 | - Yeah, the clear thinking that's able to be communicated

01:04:55.560 | is kind of impressive.

01:04:57.240 | - It's kind of remarkable, yeah, I'm a fan.

01:05:00.800 | - Well, let me ask, 'cause I talk to Chris actually

01:05:03.760 | a lot these days.

01:05:05.040 | He's been one of the, just to give him a shout out,

01:05:08.920 | he's been so supportive as a human being.

01:05:13.720 | So everybody's quite different.

01:05:16.320 | Like great engineers are different,

01:05:17.680 | but he's been like sensitive to the human element

01:05:20.800 | in a way that's been fascinating.

01:05:22.280 | Like he was one of the early people

01:05:23.800 | on this stupid podcast that I do to say like,

01:05:27.920 | don't quit this thing and also talk to whoever

01:05:32.120 | the hell you wanna talk to.

01:05:34.160 | That kind of, from a legit engineer to get like props

01:05:38.080 | and be like, you can do this.

01:05:40.000 | That was, I mean, that's what a good leader does, right?

01:05:42.280 | To just kinda let a little kid do his thing,

01:05:45.120 | like go do it, let's see what turns out.

01:05:48.720 | That's a pretty powerful thing.

01:05:50.520 | But what's your sense about, he used to be,

01:05:55.520 | now I think stepped away from Google, right?

01:05:58.000 | He's at Sci-Fi, I think.

01:06:00.960 | What's really impressive to you about the things

01:06:04.840 | that Chris has worked on?

01:06:05.720 | 'Cause we mentioned the optimization,

01:06:08.300 | the compiler design stuff, the LLVM.

01:06:11.920 | Then there's, he's also at Google worked at the TPU stuff.

01:06:15.220 | He's obviously worked on Swift,

01:06:19.400 | so the programming language side.

01:06:21.360 | Talking about people that work in the entirety of the stack.

01:06:24.360 | From your time interacting with Chris and knowing the guy,

01:06:28.840 | what's really impressive to you that just inspires you?

01:06:32.120 | - Well, like LLVM became the de facto platform

01:06:39.780 | for compilers, like it's amazing.

01:06:43.780 | And it was good code quality, good design choices.

01:06:46.340 | He hit the right level of abstraction.

01:06:48.820 | There's a little bit of the right time, the right place.

01:06:51.980 | And then he built a new programming language called Swift,

01:06:55.420 | which after, let's say some adoption resistance

01:06:59.060 | became very successful.

01:07:01.140 | I don't know that much about his work at Google,

01:07:03.340 | although I know that that was a typical,

01:07:07.140 | they started TensorFlow stuff and it was new.

01:07:11.580 | They wrote a lot of code and then at some point

01:07:13.620 | it needed to be refactored to be,

01:07:16.120 | because it's development slowed down,

01:07:19.100 | why PyTorch started a little later and then passed it.

01:07:22.340 | So he did a lot of work on that.

01:07:23.940 | And then his idea about MLIR,

01:07:25.980 | which is what people started to realize

01:07:28.220 | is the complexity of the software stack above

01:07:30.540 | the low level IR was getting so high

01:07:33.500 | that forcing the features of that into a level

01:07:36.220 | was putting too much of a burden on it.

01:07:38.740 | So he's splitting that into multiple pieces.

01:07:41.620 | And that was one of the inspirations for our software stack

01:07:43.860 | where we have several intermediate representations

01:07:46.720 | that are all executable and you can look at them

01:07:49.740 | and do transformations on them before you lower the level.

01:07:53.980 | So that was, I think we started before MLIR

01:07:58.180 | really got far enough along to use,

01:08:01.740 | but we're interested in that.

01:08:02.860 | - He's really excited about MLIR.

01:08:04.620 | He's, that's his like little baby.

01:08:06.660 | So he, and there seems to be some profound ideas on that

01:08:10.900 | that are really useful.

01:08:11.780 | - So each one of those things has been,

01:08:14.940 | as the world of software gets more and more complicated,

01:08:17.780 | how do we create the right abstraction levels

01:08:20.060 | to simplify it in a way that people can now work independently

01:08:23.340 | on different levels of it?

01:08:25.140 | So I would say all three of those projects,

01:08:27.220 | LLVM, Swift and MLIR did that successfully.

01:08:31.600 | So I'm interested in what he's gonna do next

01:08:33.660 | in the same kind of way.

01:08:34.820 | - Yes.

01:08:35.660 | So on either the TPU or maybe the NVIDIA GPU side,

01:08:40.660 | how does TensorFlow, you think,

01:08:43.740 | or the ideas underlying it,

01:08:45.820 | does that have to be TensorFlow,

01:08:46.980 | just this kind of graph focused,

01:08:49.900 | graph centric hardware,

01:08:53.900 | deep learning centric hardware beat NVIDIA's?

01:08:59.020 | Do you think it's possible for it

01:09:00.460 | to basically overtake NVIDIA?

01:09:02.300 | - Sure.

01:09:03.500 | - What's that process look like?

01:09:05.580 | What's that journey look like, you think?

01:09:08.060 | - Well, GPUs were built to run shader programs

01:09:11.060 | on millions of pixels, not to run graphs.

01:09:13.860 | - Yes.

01:09:14.700 | - So there's a hypothesis that says

01:09:17.380 | the way the graphs are built

01:09:20.300 | is going to be really interesting

01:09:21.540 | to be inefficient on computing this.

01:09:24.100 | And then the primitives is not a SIMD program,

01:09:27.520 | it's matrix multiply convolution.

01:09:30.060 | And then the data manipulations are fairly extensive

01:09:32.940 | about like how do you do a fast transpose with a program?

01:09:36.340 | I don't know if you've ever written a transpose program.

01:09:38.740 | They're ugly and slow,

01:09:39.780 | but in hardware you can do really well.

01:09:42.100 | Like I'll give you an example.

01:09:43.260 | So when GPU accelerator started doing triangles,

01:09:47.740 | like so you have a triangle

01:09:48.980 | which maps on the set of pixels.

01:09:51.140 | So you build, it's very easy,

01:09:52.540 | straightforward to build a hardware engine

01:09:54.180 | that'll find all those pixels.

01:09:55.820 | And it's kind of weird

01:09:56.660 | 'cause you walk along the triangle

01:09:57.660 | till you get to the edge,

01:09:59.220 | and then you have to go back down to the next row

01:10:01.260 | and walk along,

01:10:02.100 | and then you have to decide on the edge

01:10:04.100 | if the line of the triangle is like half on the pixel,

01:10:08.100 | what's the pixel color?

01:10:09.180 | 'Cause it's half of this pixel and half the next one.

01:10:11.140 | That's called rasterization.

01:10:12.980 | - And you're saying that can be done in hardware?

01:10:15.940 | - No, that's an example of that operation

01:10:19.380 | as a software program is really bad.

01:10:22.140 | I've written a program that did rasterization.

01:10:24.460 | The hardware that does it is actually less code

01:10:26.900 | than the software program that does it,

01:10:29.020 | and it's way faster.

01:10:31.680 | Right, so there are certain times

01:10:33.460 | when the abstraction you have, rasterize a triangle,

01:10:37.060 | execute a graph, components of a graph,

01:10:41.300 | but the right thing to do in the hardware-software boundary

01:10:43.860 | is for the hardware to naturally do it.

01:10:45.820 | - And so the GPU is really optimized

01:10:47.940 | for the rasterization of triangles.

01:10:50.100 | - Well, no, that's just, well, like in a modern,

01:10:52.960 | that's a small piece of modern GPUs.

01:10:56.980 | What they did is they still rasterized triangles

01:10:59.940 | when you're running a game,

01:11:00.940 | but for the most part, most of the computation here

01:11:03.860 | in the GPU is running shader programs,

01:11:05.900 | but they're single-threaded programs on pixels, not graphs.

01:11:09.580 | - And to be honest, let's say I don't actually know

01:11:11.820 | the math behind shading and lighting

01:11:15.060 | and all that kind of stuff.

01:11:16.180 | I don't know what--

01:11:17.780 | - They look like little simple floating-point programs

01:11:20.100 | or complicated ones.

01:11:21.220 | You can have 8,000 instructions in a shader program.

01:11:23.740 | - But I don't have a good intuition

01:11:25.580 | why it could be parallelized so easily.

01:11:27.980 | - No, it's 'cause you have 8 million pixels

01:11:29.540 | in every single.

01:11:30.660 | So when you have a light that comes down,

01:11:34.660 | the angle, the amount of light,

01:11:37.260 | like say this is a line of pixels across this table,

01:11:40.740 | the amount of light on each pixel is subtly different.

01:11:43.620 | - And each pixel is responsible for figuring out

01:11:45.820 | what it's on. - Figuring it out.

01:11:46.660 | So that pixel says, "I'm this pixel.

01:11:48.460 | "I know the angle of the light.

01:11:49.800 | "I know the occlusion.

01:11:50.780 | "I know the color I am."

01:11:52.380 | Like every single pixel here is a different color.

01:11:54.380 | Every single pixel gets a different amount of light.

01:11:57.120 | Every single pixel has a subtly different translucency.

01:12:00.540 | So to make it look realistic,

01:12:02.140 | the solution was you run a separate program on every pixel.

01:12:05.140 | - See, but I thought there's like reflection

01:12:06.700 | from all over the place.

01:12:07.740 | Is it every pixel's-- - Yeah, but there is.

01:12:09.580 | So you build a reflection map,

01:12:11.020 | which also has some pixelated thing.

01:12:14.140 | And then when the pixel's looking at the reflection map,

01:12:16.300 | it has to calculate what the normal of the surface is,

01:12:19.220 | and it does it per pixel.

01:12:20.880 | By the way, there's boatloads of hacks on that.

01:12:22.740 | You may have a lower resolution light map,

01:12:25.620 | your reflection map.

01:12:26.620 | There's all these hacks they do.

01:12:29.180 | But at the end of the day, it's per pixel computation.

01:12:32.900 | - And it's so happening you can map graph-like computation

01:12:37.100 | onto this pixel-centric computation.

01:12:39.340 | - You could do floating point programs

01:12:41.320 | on convolutions and matrices.

01:12:43.420 | And NVIDIA invested for years in CUDA.

01:12:46.180 | First for HPC, and then they got lucky with the AI trend.

01:12:50.100 | - But do you think they're going to essentially

01:12:52.260 | not be able to hardcore pivot out of their--

01:12:55.420 | - We'll see.

01:12:56.260 | That's always interesting.

01:12:59.480 | How often do big companies hardcore pivot?

01:13:01.280 | Occasionally.

01:13:02.120 | - How much do you know about NVIDIA, folks?

01:13:06.360 | - Some.

01:13:07.180 | - Some?

01:13:08.020 | - Yeah.

01:13:08.840 | - Well, I'm curious as well.

01:13:10.040 | Who's ultimately, as a--

01:13:11.480 | - Well, they've innovated several times,

01:13:13.400 | but they've also worked really hard on mobile.

01:13:15.240 | They worked really hard on radios.

01:13:16.940 | They're fundamentally a GPU company.

01:13:20.680 | - Well, they tried to pivot.

01:13:21.880 | There's an interesting little game and play

01:13:26.160 | in autonomous vehicles, right?

01:13:28.520 | Or semi-autonomous, like playing with Tesla and so on,

01:13:31.120 | and seeing that's dipping a toe into that kind of pivot.

01:13:35.680 | - They came out with this platform,

01:13:37.080 | which is interesting technically,

01:13:39.120 | but it was like a 3,000 watt,

01:13:40.880 | 1,000 watt, $3,000 GPU platform.

01:13:46.240 | - I don't know if it's interesting technically.

01:13:47.520 | It's interesting philosophically.

01:13:49.920 | Technically, I don't know if it's the execution,

01:13:51.880 | the craftsmanship is there.

01:13:53.440 | I'm not sure.

01:13:54.600 | But I didn't get a sense--

01:13:55.440 | - But they were repurposing GPUs for an automotive solution.

01:13:59.120 | - Right, it's not a real pivot.

01:14:00.320 | - They didn't build a ground-up solution.

01:14:02.820 | Like the chips inside Tesla are pretty cheap.

01:14:06.520 | Mobileye has been doing this.

01:14:08.040 | They're doing the classic work from the simplest thing.

01:14:11.240 | They were building 40 square millimeter chips,

01:14:14.240 | and Nvidia, their solution, had two 800 millimeter chips

01:14:17.480 | and two 200 millimeter chips.

01:14:19.320 | Bolt loads are really expensive DRAMs,

01:14:23.800 | and it's a really different approach.

01:14:27.120 | So Mobileye fit the, let's say,

01:14:28.880 | automotive cost and form factor,

01:14:31.280 | and then they added features as it was economically viable.

01:14:34.160 | And Nvidia said, "Take the biggest thing,

01:14:36.240 | "and we're gonna go make it work."

01:14:38.040 | And that's also influenced Waymo.

01:14:41.400 | There's a whole bunch of autonomous startups

01:14:43.680 | where they have a 5,000 watt server in their trunk.

01:14:46.240 | But that's 'cause they think,

01:14:49.520 | "Well, 5,000 watts and $10,000 is okay

01:14:52.220 | "'cause it's replacing a driver."

01:14:54.720 | Elon's approach was that port has to be cheap enough

01:14:58.080 | to put it in every single Tesla,

01:14:59.560 | whether they turn on autonomous driving or not.

01:15:02.080 | And Mobileye was like, "We need to fit in the BOM

01:15:06.120 | "and cost structure that car companies do."

01:15:09.440 | So they may sell you a GPS for 1,500 bucks,

01:15:12.480 | but the BOM for that's like $25.

01:15:15.160 | - Well, and for Mobileye, it seems like neural networks

01:15:20.120 | were not first-class citizens, like the computation.

01:15:22.960 | They didn't start out as a--

01:15:24.640 | - Yeah, it was a CV problem.

01:15:26.080 | - Yeah.

01:15:27.120 | - And did classic CV and found stoplights and lines,

01:15:29.920 | and they were really good at it.

01:15:31.240 | - Yeah, and they never, I mean,

01:15:33.040 | I don't know what's happening now,

01:15:34.120 | but they never fully pivoted.

01:15:35.820 | I mean, it's like it's the Nvidia thing.

01:15:37.960 | Then, as opposed to, so if you look at the new Tesla work,

01:15:42.000 | it's like neural networks from the ground up, right?

01:15:45.560 | - Yeah, and even Tesla started with a lot of CV stuff in it,

01:15:48.120 | and Andre's basically been eliminating it.

01:15:50.320 | Move everything into the network.

01:15:54.360 | - So without, this isn't like confidential stuff,

01:15:57.960 | but you sitting on a porch looking over the world,

01:16:01.640 | looking at the work that Andre's doing,

01:16:03.720 | that Elon's doing with Tesla Autopilot,

01:16:06.440 | do you like the trajectory of where things are going

01:16:08.800 | on the hardware side?

01:16:09.640 | - Well, they're making serious progress.

01:16:10.920 | I like the videos of people driving the beta stuff.

01:16:14.160 | Like, it's taken some pretty complicated intersections

01:16:16.520 | and all that, but it's still an intervention per drive.

01:16:19.600 | I mean, I have Autopilot, the current Autopilot,

01:16:22.720 | my Tesla, I use it every day.

01:16:24.560 | - Do you have full self-driving beta or no?

01:16:26.840 | So you like where this is going?

01:16:28.760 | - They're making progress.

01:16:29.600 | It's taking longer than anybody thought.

01:16:32.240 | You know, my wonder was, is, you know,

01:16:36.400 | hardware three, is it enough computing?

01:16:39.080 | Off by two, off by five, off by 10, off by 100.

01:16:42.360 | - Yeah.

01:16:43.800 | - And I thought it probably wasn't enough,

01:16:47.160 | but they're doing pretty well with it now.

01:16:49.800 | - Yeah.

01:16:50.640 | - And one thing is, the data set gets bigger,

01:16:53.320 | the training gets better.

01:16:55.000 | And then there's this interesting thing is,

01:16:57.360 | you sort of train and build an arbitrary size network

01:16:59.960 | that solves the problem.

01:17:01.360 | And then you refactor the network down to the thing

01:17:03.680 | that you can afford to ship, right?

01:17:06.760 | So the goal isn't to build a network that fits in the phone,

01:17:10.720 | it's to build something that actually works.

01:17:13.760 | And then how do you make that most effective

01:17:17.680 | on the hardware you have?

01:17:19.840 | And they seem to be doing that much better

01:17:21.640 | than a couple of years ago.

01:17:23.520 | - Well, the one really important thing is also

01:17:25.760 | what they're doing well is how to iterate that quickly,

01:17:28.640 | which means like, it's not just about one time deployment,

01:17:31.720 | one building, it's constantly iterating the network

01:17:34.200 | and trying to automate as many steps as possible, right?

01:17:36.720 | - Yeah.

01:17:37.560 | - And that's actually the principles of the Software 2.0,

01:17:41.680 | like you mentioned with Andre, is it's not just,

01:17:46.680 | I mean, I don't know what the actual,

01:17:48.280 | his description of Software 2.0 is,

01:17:50.880 | if it's just high-level philosophical or their specifics,

01:17:53.520 | but the interesting thing about what that actually looks

01:17:57.080 | in the real world is, it's that,

01:18:00.680 | what I think Andre calls the data engine.

01:18:02.680 | It's like, it's the iterative improvement of the thing.

01:18:06.600 | You have a neural network that does stuff,

01:18:10.520 | fails at a bunch of things,

01:18:11.960 | and learns from it over and over and over.

01:18:13.640 | So you're constantly discovering edge cases.

01:18:15.920 | So it's very much about data engineering,

01:18:19.920 | like figuring out, it's kind of what you were talking about

01:18:23.040 | with TensorTorrent is you have the data landscape.

01:18:25.760 | You have to walk along that data landscape

01:18:27.560 | in a way that's constantly improving the neural network.

01:18:32.560 | And that feels like that's the central piece

01:18:34.880 | that they've got to solve.

01:18:35.800 | And there's two pieces of it.

01:18:37.240 | Like you find edge cases that don't work,

01:18:40.920 | and then you define something

01:18:41.960 | that goes get you data for that.

01:18:44.200 | But then the other constraint

01:18:45.360 | is whether you have to label it or not.

01:18:46.920 | Like the amazing thing about like the GPT-3 stuff

01:18:49.840 | is it's unsupervised.

01:18:51.520 | So there's essentially infinite amount of data.

01:18:53.280 | Now there's obviously infinite amount of data

01:18:55.800 | available from cars of people who are successfully driving.

01:18:59.240 | But the current pipelines are mostly running on labeled data,

01:19:03.040 | which is human limited.

01:19:04.680 | So when that becomes unsupervised, right?

01:19:09.040 | It'll create unlimited amount of data,

01:19:12.600 | which then they'll scale.

01:19:14.240 | Now the networks that may use that data

01:19:16.200 | might be way too big for cars,

01:19:18.280 | but then there'll be the transformation

01:19:19.600 | from now we have unlimited data.

01:19:20.880 | I know exactly what I want.

01:19:22.360 | Now can I turn that into something that fits in the car?

01:19:25.840 | And that process is gonna happen all over the place.

01:19:29.200 | Every time you get to the place where you have unlimited data

01:19:32.240 | and that's what software 2.0 is about,

01:19:34.080 | unlimited data training networks to do stuff

01:19:36.800 | without humans writing code to do it.

01:19:40.680 | - And ultimately also trying to discover,

01:19:42.960 | like you're saying,

01:19:43.800 | the self-supervised formulation of the problem.

01:19:47.240 | So the unsupervised formulation of the problem.

01:19:49.640 | Like in driving, there's this really interesting thing,

01:19:53.520 | which is you look at a scene that's before you

01:19:58.120 | and you have data about what a successful human driver did

01:20:01.880 | in that scene one second later.

01:20:04.440 | It's a little piece of data that you can use

01:20:06.600 | just like with GPT-3 as training.

01:20:09.360 | Currently, even though Tesla says they're using that,

01:20:12.360 | it's an open question to me,

01:20:14.400 | how far can you,

01:20:15.960 | can you solve all of the driving

01:20:17.440 | with just that self-supervised piece of data?

01:20:20.960 | And like, I think-

01:20:23.400 | - Well, that's what Common AI is doing.

01:20:25.520 | - That's what Common AI is doing,

01:20:26.880 | but the question is how much data,

01:20:29.960 | so what Common AI doesn't have

01:20:32.240 | is as good of a data engine, for example, as Tesla does.

01:20:35.920 | That's where the, like the organization of the data.

01:20:38.580 | I mean, as far as I know, I haven't talked to George,

01:20:41.920 | but they do have the data.

01:20:44.560 | The question is how much data is needed?

01:20:46.860 | 'Cause we say infinite very loosely here.

01:20:49.940 | And then the other question, which you said,

01:20:54.360 | I don't know if you think it's still an open question,

01:20:56.520 | is are we on the right order of magnitude

01:20:59.400 | for the compute necessary?

01:21:01.360 | Is this, is it like what Elon said,

01:21:04.960 | this chip that's in there now

01:21:06.640 | is enough to do full self-driving,

01:21:08.600 | or do we need another order of magnitude?

01:21:10.800 | I think nobody actually knows the answer to that question.

01:21:13.320 | I like the confidence that Elon has, but.

01:21:15.360 | - Yeah, we'll see.

01:21:17.840 | There's another funny thing is you don't learn to drive

01:21:20.160 | with infinite amounts of data.

01:21:22.280 | You learn to drive with an intellectual framework

01:21:24.320 | that understands physics and color and horizontal surfaces

01:21:28.040 | and laws and roads and all your experience

01:21:32.120 | from manipulating your environment.

01:21:36.120 | There's so many factors go into that.

01:21:39.040 | So then when you learn to drive,

01:21:40.680 | driving is a subset of this conceptual framework

01:21:44.340 | that you have.

01:21:45.180 | And so with self-driving cars right now,

01:21:48.520 | we're teaching them to drive with driving data.

01:21:51.520 | You never teach a human to do that.

01:21:53.560 | You teach a human all kinds of interesting things,

01:21:55.720 | like language, like don't do that, watch out.

01:21:59.320 | There's all kinds of stuff going on.

01:22:01.000 | - This is where you, I think, previous time

01:22:02.880 | we talked about where you poetically disagreed

01:22:07.280 | with my naive notion about humans.

01:22:10.280 | I just think that humans will make

01:22:13.680 | this whole driving thing really difficult.

01:22:15.680 | - Yeah, all right.

01:22:17.200 | I said humans don't move that slow.

01:22:19.480 | It's a ballistics problem.

01:22:20.800 | - It's a ballistics, humans are a ballistics problem,

01:22:22.680 | which is like poetry to me.

01:22:24.040 | - It's very possible that in driving

01:22:26.200 | they're indeed purely a ballistics problem.

01:22:28.480 | And I think that's probably the right way to think about it.

01:22:30.880 | But I still, they still continue to surprise me,

01:22:34.440 | those damn pedestrians, the cyclists,

01:22:36.940 | other humans in other cars.

01:22:38.480 | - Yeah, but it's gonna be one of these compensating things.

01:22:41.160 | So like when you're driving,

01:22:43.960 | you have an intuition about what humans are going to do,

01:22:46.840 | but you don't have 360 cameras and radars

01:22:49.640 | and you have an attention problem.

01:22:51.800 | So the self-driving car comes in with no attention problem,

01:22:55.040 | 360 cameras, a bunch of other features.

01:22:58.760 | So they'll wipe out a whole class of accidents.

01:23:01.200 | Emergency braking with radar,

01:23:05.720 | and especially as it gets AI enhanced,

01:23:07.960 | will eliminate collisions.

01:23:09.720 | But then you have the other problems

01:23:12.000 | of these unexpected things where you think

01:23:14.200 | your human intuition is helping,

01:23:15.560 | but then the cars also have a set of hardware features

01:23:19.520 | that you're not even close to.

01:23:21.480 | - And the key thing, of course,

01:23:22.760 | is if you wipe out a huge number of kind of accidents,

01:23:27.000 | then it might be just way safer than a human driver,

01:23:30.200 | even if humans are still a problem.

01:23:32.960 | That's hard to figure out.

01:23:34.700 | - Yeah, that's probably what'll happen.

01:23:36.200 | Autonomous cars will have a small number of accidents

01:23:38.800 | humans would have avoided,

01:23:40.000 | but they'll get rid of the bulk of them.

01:23:43.800 | - What do you think about like Tesla's dojo efforts,

01:23:48.640 | or it can be bigger than Tesla in general.

01:23:51.120 | It's kind of like the tense torrent,

01:23:52.880 | trying to innovate.

01:23:55.400 | This is the dichotomy,

01:23:57.360 | should a company try to from scratch

01:23:59.200 | build its own neural network training hardware?

01:24:03.160 | - Well, first, I think it's great.

01:24:04.240 | So we need lots of experiments, right?

01:24:06.800 | And there's lots of startups working on this

01:24:09.440 | and they're pursuing different things.

01:24:11.520 | Now, I was there when we started dojo,

01:24:13.360 | and it was sort of like,

01:24:14.560 | what's the unconstrained computer solution

01:24:17.960 | to go do very large training problems?

01:24:21.720 | And then there's fun stuff like,

01:24:23.640 | we said, well, we have this 10,000 watt board to cool.

01:24:27.200 | Well, you go talk to guys at SpaceX

01:24:29.080 | and they think 10,000 watts is a really small number,

01:24:31.160 | not a big number.

01:24:32.080 | And there's brilliant people working on it.

01:24:35.280 | I'm curious to see how it'll come out.

01:24:37.280 | I couldn't tell you.

01:24:39.000 | I know it pivoted a few times since I left.

01:24:41.640 | - So the cooling does seem to be a big problem.

01:24:44.520 | I do like what Elon said about it,

01:24:46.840 | which is like, we don't wanna do the thing

01:24:49.320 | unless it's way better than the alternative,

01:24:51.360 | whatever the alternative is.

01:24:53.000 | So it has to be way better than racks of GPUs.

01:24:57.640 | - Yeah, and the other thing is just like

01:25:00.000 | the Tesla autonomous driving hardware,

01:25:03.880 | it was only serving one software stack.

01:25:06.600 | And the hardware team and the software team

01:25:08.040 | were tightly coupled.

01:25:09.160 | If you're building a general purpose AI solution,

01:25:12.160 | then there's so many different customers

01:25:14.280 | with so many different needs.

01:25:16.400 | Now, something Andre said is, I think this is amazing,

01:25:19.800 | 10 years ago, like vision, recommendation, language

01:25:24.640 | were completely different disciplines.

01:25:27.120 | He said the people literally couldn't talk to each other.

01:25:29.720 | And three years ago, it was all neural networks,

01:25:32.560 | but the very different neural networks.

01:25:34.840 | And recently it's converging on one set of networks.

01:25:37.720 | They vary a lot in size, obviously they vary in data,

01:25:40.480 | vary in outputs, but the technology

01:25:43.040 | has converged a good bit.

01:25:44.800 | - Yeah, these transformers behind GPT-3,

01:25:47.400 | it seems like they could be applied to video,

01:25:48.960 | they could be applied to a lot of,

01:25:50.560 | and it's like, and they're all really simple.

01:25:52.520 | - And it was like, they literally replace letters

01:25:54.680 | with pixels, it does vision, it's amazing.

01:25:58.280 | - And then size actually improves the thing.

01:26:02.080 | So the bigger it gets, the more compute you throw at it,

01:26:04.400 | the better it gets.

01:26:05.640 | - And the more data you have, the better it gets.

01:26:08.320 | So then you start to wonder,

01:26:11.040 | well, is that a fundamental thing

01:26:12.520 | or is this just another step to some fundamental

01:26:16.080 | understanding about this kind of computation?

01:26:18.800 | Which is really interesting.

01:26:20.240 | - Us humans don't want to believe that that kind of thing

01:26:22.200 | will achieve conceptual understanding, as you were saying,

01:26:24.400 | like you'll figure out physics, but maybe it will.

01:26:26.960 | Maybe. - Probably will.

01:26:28.240 | Well, it's worse than that.

01:26:31.000 | It'll understand physics in ways that we can't understand.

01:26:33.760 | I like your Stephen Wolfram talk where he said,

01:26:36.520 | there's three generations of physics.

01:26:37.960 | There was physics by reasoning,

01:26:40.080 | well, big things should fall faster than small things,

01:26:42.600 | right, that's reasoning.

01:26:43.960 | And then there's physics by equations.

01:26:46.280 | But the number of programs in the world that are solved

01:26:50.120 | with the single equations is relatively low.

01:26:51.960 | Almost all programs have more than one line of code,

01:26:54.920 | maybe 100 million lines of code.

01:26:56.840 | So he said, now we're going to physics by equation,

01:26:59.920 | which is his project, which is cool.

01:27:01.680 | I might point out there was two generations of physics

01:27:07.240 | before reasoning, habit, like all animals know things fall

01:27:12.240 | and birds fly and predators know how to solve

01:27:16.320 | a differential equation to cut off

01:27:17.840 | an accelerating, curving animal path.

01:27:22.840 | And then there was, the gods did it, right?

01:27:27.560 | So there's five generations.

01:27:31.600 | Now, software 2.0 says programming things

01:27:35.940 | is not the last step.

01:27:37.280 | Data, so there's going to be a physics,

01:27:41.240 | Beth Stevens, Wolfram's concept.

01:27:44.040 | - That's not explainable to us humans.

01:27:46.360 | - And actually, there's no reason that I can see

01:27:51.000 | why even that's a limit.

01:27:53.240 | Like, there's something beyond that.

01:27:55.560 | I mean, usually when you have this hierarchy,

01:27:57.840 | it's not like, well, if you have this step

01:27:59.600 | and this step and this step,

01:28:00.520 | and they're all qualitatively different

01:28:03.080 | and conceptually different, it's not obvious why,

01:28:05.080 | you know, six is the right number of hierarchy steps

01:28:07.340 | and not seven or eight or--

01:28:09.180 | - Well, then it's probably impossible for us

01:28:12.100 | to comprehend something that's beyond

01:28:15.220 | the thing that's not explainable.

01:28:16.920 | - Yeah, but the thing that, you know,

01:28:20.820 | understands the thing that's not explainable to us,

01:28:23.460 | well, conceives the next one,

01:28:25.100 | and like, I'm not sure why there's a limit to it.

01:28:28.280 | Clicker brain hurts, that's a sad story.

01:28:34.860 | - If we look at our own brain,

01:28:36.540 | which is an interesting illustrative example,

01:28:39.940 | in your work with Tess Thornton

01:28:42.640 | and trying to design deep learning architectures,

01:28:46.140 | do you think about the brain at all?

01:28:50.060 | Maybe from a hardware designer perspective,

01:28:53.460 | if you could change something about the brain,

01:28:56.200 | what would you change, or do you--

01:28:58.180 | - Funny question.

01:28:59.020 | - Like, how would you--

01:29:00.940 | - So, your brain is really weird.

01:29:02.340 | Like, you know, your cerebral cortex,

01:29:03.960 | where we think we do most of our thinking,

01:29:06.060 | is what, like six or seven neurons thick?

01:29:08.660 | - Yeah. - Like, that's weird.

01:29:10.260 | Like, all the big networks are way bigger than that.

01:29:13.200 | Like, way deeper.

01:29:14.340 | So, that seems odd.

01:29:16.140 | And then, you know, when you're thinking,

01:29:18.140 | if the input generates a result you can lose,

01:29:21.820 | it goes really fast, but if it can't,

01:29:24.060 | that generates an output that's interesting,

01:29:25.900 | which turns into an input, and then your brain,

01:29:28.340 | to the point where you mull things over for days,

01:29:30.660 | and how many trips through your brain is that, right?

01:29:33.380 | Like, it's, you know, 300 milliseconds or something

01:29:36.100 | to get through seven levels of neurons.

01:29:37.840 | I forget the number exactly.

01:29:39.860 | But then it does it over and over and over as it searches.

01:29:43.300 | And the brain clearly looks like some kind of graph,

01:29:46.140 | 'cause you have a neuron with connections,

01:29:48.140 | and it talks to other ones,

01:29:49.220 | and it's locally very computationally intense,

01:29:52.380 | but it also does sparse computations

01:29:55.500 | across a pretty big area.

01:29:57.820 | - There's a lot of messy biological type of things,

01:30:00.640 | and it's meaning, like, first of all,

01:30:03.700 | there's mechanical, chemical, and electrical signals,

01:30:06.020 | it's all that's going on.

01:30:07.460 | Then there's the asynchronicity of signals,

01:30:12.460 | and there's just a lot of variability

01:30:14.740 | that seems continuous and messy,

01:30:16.500 | and just a mess of biology,

01:30:18.620 | and it's unclear whether that's a good thing,

01:30:22.640 | or it's a bad thing, because if it's a good thing,

01:30:26.320 | then we need to run the entirety of the evolution.

01:30:29.260 | Well, we're gonna have to start with basic bacteria

01:30:31.580 | to create something--

01:30:32.420 | - But imagine you could build a brain with 10 layers.

01:30:35.660 | Would that be better or worse?

01:30:37.380 | Or more connections, or less connections?

01:30:39.820 | Or, you know, we don't know to what level

01:30:42.100 | our brains are optimized.

01:30:43.420 | But if I was changing things,

01:30:45.500 | like, you know you can only hold seven numbers in your head.

01:30:49.380 | Like, why not 100, or a million?

01:30:51.860 | - Never thought of that.

01:30:53.700 | - And why can't we have a floating point processor

01:30:56.820 | that can compute anything we want,

01:30:59.300 | like, and see it all properly?

01:31:01.300 | Like, that would be kind of fun.

01:31:03.180 | And why can't we see in four or eight dimensions?

01:31:05.780 | Like, 3D is kind of a drag.

01:31:10.060 | Like, all the hard mass transforms

01:31:11.600 | are up in multiple dimensions.

01:31:13.960 | So there's, you know, you could imagine

01:31:15.620 | a brain architecture that, you know,

01:31:18.260 | you could enhance with a whole bunch of features

01:31:21.120 | that would be, you know,

01:31:22.560 | really useful for thinking about things.

01:31:24.420 | - It's possible that the limitations

01:31:25.980 | you're describing are actually essential for,

01:31:28.780 | like, the constraints are essential for creating,

01:31:32.020 | like, the depth of intelligence.

01:31:33.980 | Like, that, the ability to reason, you know.

01:31:38.060 | - Yeah, it's hard to say,

01:31:39.060 | 'cause, like, your brain is clearly a parallel processor.

01:31:43.020 | You know, 10 billion neurons talking to each other

01:31:46.180 | at a relatively low clock rate.

01:31:48.400 | But it produces something that looks like

01:31:51.020 | a serial thought process,

01:31:52.580 | it's a serial narrative in your head.

01:31:54.700 | - That's true.

01:31:55.540 | - But then, there are people, famously,

01:31:57.580 | who are visual thinkers.

01:31:59.060 | Like, I think I'm a relatively visual thinker.

01:32:02.300 | I can imagine any object and rotate it in my head

01:32:04.980 | and look at it.

01:32:06.420 | And there are people who say

01:32:07.340 | they don't think that way at all.

01:32:09.620 | And recently, I read an article about people

01:32:12.420 | who say they don't have a voice in their head.

01:32:15.460 | They can talk, but when they, you know,

01:32:19.500 | it's like, well, what are you thinking?

01:32:21.020 | They'll describe something that's visual.

01:32:24.380 | So that's curious.

01:32:26.500 | Now, if you're saying,

01:32:31.740 | if we dedicated more hardware to holding information,

01:32:34.960 | like, you know, 10 numbers or a million numbers,

01:32:37.940 | like, would that distract us from our ability

01:32:41.680 | to form this kind of singular identity?

01:32:44.740 | - Like, it dissipates somehow.

01:32:46.260 | - Right, but maybe, you know,

01:32:48.620 | future humans will have many identities

01:32:50.700 | that have some higher-level organization,

01:32:53.120 | but can actually do lots more things in parallel.

01:32:55.620 | - Yeah, there's no reason, if we're thinking modularly,

01:32:57.900 | there's no reason we can't have multiple consciousnesses

01:33:00.300 | in one brain.

01:33:01.540 | - Yeah, and maybe there's some way to make it faster

01:33:03.700 | so that the, you know, the area of the computation

01:33:07.900 | could still have a unified feel to it

01:33:12.900 | while still having way more ability

01:33:15.740 | to do parallel stuff at the same time.

01:33:17.620 | Could definitely be improved.

01:33:19.040 | - Could be improved?

01:33:20.040 | - Yeah.

01:33:20.880 | - Okay, well, it's pretty good right now.

01:33:22.900 | Actually, people don't give it enough credit.

01:33:24.660 | The thing is pretty nice.

01:33:25.900 | The fact that the right ends seem to be,

01:33:30.240 | give a nice, like, spark of beauty to the whole experience.

01:33:36.820 | I don't know.

01:33:38.740 | I don't know if it can be improved easily.

01:33:40.260 | - It could be more beautiful.

01:33:42.500 | - I don't know how, yeah.

01:33:44.340 | - What do you mean, how?

01:33:46.300 | All the ways you can imagine.

01:33:48.280 | - No, but that's the whole point.

01:33:49.500 | I wouldn't be able to imagine,

01:33:51.100 | the fact that I can imagine ways

01:33:52.740 | in which it could be more beautiful means--

01:33:55.820 | - So do you know, you know, Ian Banks, his stories?

01:33:59.380 | So the super smart AIs there live,

01:34:03.580 | mostly live in the world of what they call infinite fun

01:34:06.280 | because they can create arbitrary worlds.

01:34:12.220 | So they interact in, you know, the story has it.

01:34:14.460 | They interact in the normal world

01:34:15.780 | and they're very smart and they can do all kinds of stuff.

01:34:18.540 | And, you know, a given mind can, you know,

01:34:20.420 | talk to a million humans at the same time

01:34:22.020 | 'cause we're very slow.

01:34:23.220 | And for reasons, you know, artificial to the story,

01:34:26.300 | they're interested in people and doing stuff,

01:34:28.260 | but they mostly live in this other land of thinking.

01:34:33.020 | - My inclination is to think that the ability

01:34:36.500 | to create infinite fun will not be so fun.

01:34:40.100 | - That's sad.

01:34:42.420 | - Well-- - Why there's so many

01:34:43.260 | things to do.

01:34:44.100 | Imagine being able to make a star, move planets around.

01:34:47.620 | - Yeah, yeah, but because we can imagine that

01:34:50.020 | as why life is fun, if we actually were able to do it,

01:34:53.340 | it'd be a slippery slope where fun wouldn't even

01:34:56.060 | have a meaning because we just consistently

01:34:58.940 | desensitize ourselves by the infinite amounts

01:35:02.340 | of fun we're having.

01:35:04.140 | The sadness, the dark stuff is what makes it fun, I think.

01:35:08.080 | That could be the Russian--

01:35:10.420 | - It could be the fun makes it fun

01:35:12.420 | and the sadness makes it bittersweet.

01:35:16.580 | - Yeah, that's true.

01:35:17.420 | Fun could be the thing that makes it fun.

01:35:20.540 | So what do you think about the expansion,

01:35:22.540 | not through the biology side, but through the BCI,

01:35:25.100 | the brain-computer interfaces?

01:35:27.180 | Now you got a chance to check out the Neuralink stuff.

01:35:30.100 | - It's super interesting.

01:35:31.460 | Like humans, like our thoughts to manifest as action.

01:35:36.460 | You know, like as a kid, you know,

01:35:39.540 | like shooting a rifle was super fun.

01:35:41.700 | Driving a mini bike, doing things.

01:35:44.140 | And then computer games, I think,

01:35:46.140 | for a lot of kids became the thing where they,

01:35:49.060 | you know, they can do what they want.

01:35:50.340 | They can fly a plane, they can do this,

01:35:52.040 | they can do this, right?

01:35:53.580 | But you have to have this physical interaction.

01:35:55.860 | Now imagine, you know, you could just imagine stuff

01:36:00.300 | and it happens, right?

01:36:03.300 | Like really richly and interestingly.

01:36:06.620 | Like we kind of do that when we dream.

01:36:08.100 | Like dreams are funny because like if you have

01:36:10.940 | some control or awareness in your dreams,

01:36:14.380 | like it's very realistic looking or not realistic,

01:36:17.900 | depends on the dream, but you can also manipulate that.

01:36:21.220 | And you know, what's possible there is odd.

01:36:26.220 | And the fact that nobody understands it's hilarious, but.

01:36:29.020 | - Do you think it's possible to expand that capability

01:36:33.060 | through computing?

01:36:34.060 | - Sure.

01:36:35.340 | - Is there some interesting,

01:36:36.500 | so from a hardware designer perspective, is there,

01:36:39.740 | do you think it'll present totally new challenges

01:36:41.620 | in the kind of hardware required that like,

01:36:44.100 | so this hardware isn't standalone computing.

01:36:47.740 | - Well, just take it from this.

01:36:49.420 | So today, computer games are rendered by GPUs.

01:36:52.860 | - Right.

01:36:53.700 | - Right, so, but you've seen the GAN stuff, right?

01:36:56.820 | Where trained neural networks render realistic images,

01:37:00.900 | but there's no pixels, no triangles, no shaders,

01:37:03.740 | no light maps, no nothing.

01:37:05.380 | So the future of graphics is probably AI, right?

01:37:09.540 | - Yes.

01:37:10.380 | - Now that AI is heavily trained by lots of real data.

01:37:13.740 | Right, so if you have an interface with a AI renderer,

01:37:18.860 | right, so if you say render a cat,

01:37:22.780 | it won't say, well, how tall is the cat and how big,

01:37:25.060 | you know, it'll render a cat.

01:37:26.260 | And you might say, well, a little bigger, a little smaller,

01:37:28.220 | you know, make it a tabby, shorter hair, you know,

01:37:31.340 | like you could tweak it.

01:37:32.900 | Like the amount of data you'll have to send to interact

01:37:37.420 | with a very powerful AI renderer could be low.

01:37:41.420 | - But the question is, for brain-computer interfaces,

01:37:44.780 | we'd need to render not onto a screen,

01:37:47.860 | but render onto the brain.

01:37:50.340 | And like directly, so there's a bandwidth.

01:37:52.940 | - Well, we could do it both ways.

01:37:53.900 | I mean, our eyes are really good sensors.

01:37:56.020 | It could render onto a screen,

01:37:58.580 | and we could feel like we're participating in it.

01:38:01.100 | You know, they're gonna have, you know,

01:38:03.380 | like the Oculus kind of stuff.

01:38:04.860 | It's gonna be so good when a projection to your eyes,

01:38:07.020 | you think it's real.

01:38:08.020 | You know, they're slowly solving those problems.

01:38:11.620 | And I suspect when the renderer of that information

01:38:17.260 | into your head is also AI mediated,

01:38:19.740 | you know, they'll be able to give you the cues that,

01:38:23.140 | you know, you really want for depth and all kinds of stuff.

01:38:26.220 | Like your brain is partly faking your visual field, right?

01:38:30.940 | Like your eyes are twitching around,

01:38:32.700 | but you don't notice that.

01:38:33.820 | Occasionally they blank, you don't notice that.

01:38:36.500 | You know, there's all kinds of things.

01:38:37.780 | Like you think you see over here,

01:38:39.140 | but you don't really see there.

01:38:40.820 | It's all fabricated.

01:38:42.180 | - Yeah.

01:38:43.020 | - So.

01:38:43.860 | - Yeah, peripheral vision is fascinating.

01:38:45.540 | - So if you have an AI renderer that's trained

01:38:48.540 | to understand exactly how you see

01:38:51.700 | and the kind of things that enhance the realism

01:38:54.780 | of the experience, it could be super real actually.

01:38:57.660 | So I don't know what the limits that are.

01:39:03.500 | But obviously if we have a brain interface

01:39:06.940 | that goes in inside your visual cortex

01:39:10.460 | in a better way than your eyes do, which is possible,

01:39:13.500 | it's a lot of neurons.

01:39:15.300 | - Yeah.

01:39:16.140 | - Maybe that'll be even cooler.

01:39:19.780 | - But the really cool thing is that it has to do

01:39:21.580 | with the infinite fun that you were referring to,

01:39:24.220 | which is our brains seem to be very limited.

01:39:26.620 | And like you said, computations.

01:39:28.300 | - Also very plastic.

01:39:29.900 | - Very plastic, yeah.

01:39:31.140 | So it's an interesting combination.

01:39:33.660 | - The interesting open question is the limits

01:39:37.500 | of that neuroplasticity.

01:39:38.780 | Like how flexible is that thing?

01:39:42.380 | 'Cause we haven't really tested it.

01:39:44.940 | - We know about the experiments where they put

01:39:46.980 | like a pressure pad on somebody's head

01:39:49.140 | and had a visual transducer pressurize it

01:39:51.540 | and somebody slowly learned to see.

01:39:53.500 | - Yep.

01:39:54.340 | Especially at a young age, if you throw a lot at it,

01:39:58.740 | like what can it,

01:40:00.220 | so can you like arbitrarily expand it with computing power?

01:40:06.900 | So connected to the internet directly somehow?

01:40:09.900 | - Yeah, the answer's probably yes.

01:40:11.940 | - So the problem with biology and ethics is like,

01:40:14.420 | there's a mess there.

01:40:15.540 | Like us humans are perhaps unwilling to take risks

01:40:20.540 | into directions that are full of uncertainty.

01:40:25.620 | So it's like-- - No, no.

01:40:26.460 | 90% of the population's unwilling to take risks.

01:40:28.860 | The other 10% is rushing into the risks,

01:40:31.340 | unaided by any infrastructure whatsoever.

01:40:34.060 | That's where all the fun happens in society.

01:40:38.940 | There's been huge transformations

01:40:41.140 | in the last couple thousand years.

01:40:43.580 | - Yeah, it's funny.

01:40:44.580 | I've gotten the chance to interact with,

01:40:46.580 | this is Matthew Johnson from Johns Hopkins.

01:40:49.340 | He's doing this large-scale study of psychedelics.

01:40:52.540 | It's becoming more and more,

01:40:54.260 | I've gotten a chance to interact

01:40:55.220 | with that community of scientists working on psychedelics.

01:40:57.780 | But because of that, that opened the door to me

01:41:00.100 | to all these, what do they call it, psychonauts,

01:41:03.660 | the people who, like you said, the 10% who are like,

01:41:07.260 | I don't care, I don't know if there's a science behind this.

01:41:09.820 | I'm taking this spaceship to,

01:41:12.040 | if I'm be the first on Mars, I'll be,

01:41:14.180 | psychedelics are interesting in the sense that

01:41:18.460 | in another dimension, like you said,

01:41:21.380 | it's a way to explore the limits of the human mind.

01:41:25.380 | Like, what is this thing capable of doing?

01:41:28.180 | 'Cause you kinda, like when you dream, you detach it.

01:41:31.380 | I don't know exactly the neuroscience of it,

01:41:33.020 | but you detach your reality from what your mind,

01:41:38.020 | the images your mind is able to conjure up,

01:41:40.740 | and your mind goes into weird places.

01:41:43.100 | And like entities appear.

01:41:44.900 | Somehow Freudian type of trauma

01:41:48.700 | is probably connected in there somehow,

01:41:50.240 | but you start to have like these weird, vivid worlds that--

01:41:53.980 | - So do you actively dream?

01:41:55.460 | - No. - Why not?

01:41:58.140 | I have like six hours of dreams a night.

01:42:01.260 | It's like really useful time.

01:42:03.060 | - I know, I haven't, I don't for some reason.

01:42:06.100 | I just knock out, and I have sometimes anxiety-inducing

01:42:10.980 | kinda like very pragmatic nightmare type of dreams,

01:42:15.980 | but nothing fun, nothing--

01:42:18.420 | - Nothing fun?

01:42:19.260 | - Nothing fun.

01:42:20.620 | I try, I unfortunately mostly have fun in the waking world,

01:42:25.620 | which is very limited in the amount of fun you can have.

01:42:30.020 | - It's not that limited either.

01:42:31.220 | - Yeah, that's why--

01:42:32.060 | - Maybe we'll have to talk.

01:42:33.460 | (laughing)

01:42:35.060 | - Yeah, I need instructions.

01:42:36.820 | Yeah, why--

01:42:37.660 | - There's like a manual for that.

01:42:38.620 | You might wanna--

01:42:39.460 | - I'll look it up.

01:42:41.840 | I'll ask Elon.

01:42:43.340 | What do you dream?

01:42:44.700 | - You know, years ago, and I read about, you know,

01:42:48.380 | like a book about how to have, you know,

01:42:51.340 | become aware of your dreams.

01:42:53.060 | I worked on it for a while.

01:42:54.300 | Like there's this trick about, you know,

01:42:55.940 | imagine you can see your hands and look out,

01:42:59.060 | and I got somewhat good at it.

01:43:00.620 | But my mostly, when I'm thinking about things

01:43:04.340 | or working on problems, I prep myself before I go to sleep.

01:43:09.020 | It's like I pull into my mind all the things

01:43:13.140 | I wanna work on or think about.

01:43:15.380 | And then that, let's say, greatly improves the chances

01:43:19.820 | that I'll work on that while I'm sleeping.

01:43:22.120 | - And once--

01:43:24.220 | - And then I also, you know, basically ask to remember it.

01:43:28.380 | And I often remember very detailed--

01:43:33.140 | - Within the dream or outside the dream.

01:43:35.340 | - Well, to bring it up in my dreaming

01:43:37.780 | and then to remember it when I wake up.

01:43:39.740 | It's more of a meditative practice.

01:43:43.340 | You say, you know, to prepare yourself to do that.

01:43:47.940 | Like if you go to, you know, the sleep,

01:43:50.580 | still gnashing your teeth about some random thing

01:43:52.980 | that happened that you're not that really interested in,

01:43:55.820 | you'll dream about it.

01:43:56.920 | - That's really interesting.

01:43:58.820 | Maybe--

01:43:59.660 | - But you can direct your dreams somewhat by prepping.

01:44:03.460 | - Yeah, I'm gonna have to try that.

01:44:05.460 | It's really interesting.

01:44:06.420 | Like the most important, the interesting,

01:44:08.460 | not like, what did this guy send in an email,

01:44:12.240 | kind of like stupid worry stuff,

01:44:14.100 | but like fundamental problems you're actually concerned

01:44:15.940 | about in prepping--

01:44:16.780 | - And interesting things you're worried about

01:44:18.180 | or a book you're reading or, you know,

01:44:20.020 | some great conversation you had

01:44:21.380 | or some adventure you wanna have.

01:44:23.460 | Like there's a lot of space there.

01:44:26.700 | And it seems to work that, you know,

01:44:32.540 | my percentage of interesting dreams and memories went up.

01:44:36.420 | - Is there a, is that the source of,

01:44:40.460 | if you were able to deconstruct like where

01:44:42.760 | some of your best ideas came from,

01:44:44.600 | is there a process that's at the core of that?

01:44:49.440 | Like so some people, you know, walk and think,

01:44:52.460 | some people like in the shower, the best ideas hit 'em.

01:44:55.200 | If you talk about like Newton,

01:44:56.600 | Apple hitting 'em on the head.

01:44:58.600 | - No, I found out a long time ago,

01:45:01.120 | I process things somewhat slowly.

01:45:03.220 | So like in college, I had friends who could study

01:45:05.680 | at the last minute and get an A the next day.

01:45:07.560 | I can't do that at all.

01:45:09.100 | So I always front loaded all the work.

01:45:10.960 | Like I do all the problems early, you know,

01:45:14.200 | for finals, like the last three days,

01:45:15.840 | I wouldn't look at a book.

01:45:17.760 | Because I want, you know, 'cause like a new fact

01:45:20.440 | day before finals may screw up my understanding

01:45:23.000 | of what I thought I knew.

01:45:23.920 | So my goal was to always get it in

01:45:26.440 | and give it time to soak.

01:45:29.920 | And I used to, you know,

01:45:32.080 | I remember when we were doing like 3D calculus,

01:45:33.800 | I would have these amazing dreams of 3D surfaces

01:45:36.320 | with normal, you know, calculating the gradient

01:45:38.560 | and just like all come up.

01:45:40.140 | So it was like really fun, like very visual.

01:45:43.900 | And if I got cycles of that, that was useful.

01:45:47.440 | And the other is, is don't over filter your ideas.

01:45:50.940 | Like I like that process of brainstorming

01:45:54.500 | where lots of ideas can happen.

01:45:55.620 | I like people who have lots of ideas.

01:45:57.380 | - And they just let them sit.

01:45:58.820 | - Then there's, yeah, I'll let them sit

01:46:00.260 | and let it breathe a little bit

01:46:02.540 | and then reduce it to practice.

01:46:04.980 | Like at some point you really have to,

01:46:08.420 | does it really work?

01:46:09.960 | Like, you know, is this real or not?

01:46:12.120 | Right, but you have to do both.

01:46:15.060 | There's creative tension there.

01:46:16.220 | Like how do you be both open and, you know, precise?

01:46:20.500 | - Have you had ideas that you just,

01:46:22.320 | that sit in your mind for like years before the?

01:46:26.140 | - Sure.

01:46:27.380 | - That's an interesting way to,

01:46:29.840 | is generate ideas and just let them sit.

01:46:33.140 | Let them sit there for a while.

01:46:36.540 | - I think I have a few of those ideas.

01:46:38.500 | - You know, that was so funny.

01:46:40.180 | Yeah, I think that's, you know,

01:46:42.460 | creativity discipline or something.

01:46:44.760 | - For the slow thinkers in the room, I suppose.

01:46:49.420 | As I, some people, like you said, are just like, like the.

01:46:53.340 | - Yeah, it's really interesting.

01:46:54.900 | There's so much diversity in how people think.

01:46:57.700 | You know, how fast or slow they are,

01:46:59.340 | how well they remember or don't.

01:47:01.700 | Like, you know, I'm not super good at remembering facts,

01:47:04.100 | but processes and methods.

01:47:06.500 | Like in our engineering, I went to Penn State

01:47:08.060 | and almost all our engineering tests were open book.

01:47:11.900 | I could remember the page and not the formula.

01:47:14.860 | But as soon as I saw the formula,

01:47:15.940 | I could remember the whole method if I'd learned it.

01:47:19.780 | - Yeah.

01:47:20.620 | - So it's a funny, where some people could,

01:47:23.260 | you know, I just watched friends like flipping

01:47:25.180 | through the book trying to find the formula,

01:47:27.480 | even knowing that they'd done just as much work.

01:47:30.100 | And I would just open the book, you know,

01:47:31.420 | it's on page 27, bottom half,

01:47:33.660 | I could see the whole thing visually.

01:47:35.980 | - Yeah.

01:47:36.820 | - And you know.

01:47:37.660 | - And you have to learn that about yourself

01:47:39.020 | and figure out what the, how to function optimally.

01:47:41.460 | - I had a friend who was always concerned

01:47:43.260 | he didn't know how he came up with ideas.

01:47:45.740 | He had lots of ideas, but he said they just sort of popped up

01:47:49.120 | like you'd be working on something, you have this idea,

01:47:50.980 | like where does it come from?

01:47:52.420 | But you can have more awareness of it.

01:47:54.820 | Like, like, like how your brain works as a little murky

01:47:59.740 | as you go down from the voice in your head

01:48:01.580 | or the obvious visualizations.

01:48:03.900 | Like when you visualize something, how does that happen?

01:48:06.540 | - Yeah, that's weird.

01:48:07.380 | - You know, if I say, you know, visualize volcano,

01:48:09.060 | it's easy to do, right?

01:48:09.900 | - And what does it actually look like when you visualize it?

01:48:12.540 | - I can visualize to the point where I don't see

01:48:14.380 | the very much out of my eyes and I see the colors

01:48:16.260 | of the thing I'm visualizing.

01:48:18.260 | - Yeah, but there's like a, there's a shape,

01:48:19.900 | there's a texture, there's a color,

01:48:21.180 | but there's also conceptual visualization.

01:48:23.140 | Like what are you actually visualizing

01:48:25.660 | when you're visualizing volcano?

01:48:27.200 | Just like with peripheral vision,

01:48:28.460 | you think you see the whole thing.

01:48:29.660 | - Yeah, yeah, yeah, that's a good way to say it.

01:48:31.800 | You know, you have this kind of almost peripheral vision

01:48:34.840 | of your visualizations.

01:48:36.160 | They're like these ghosts.

01:48:37.460 | But if you work on it,

01:48:40.180 | you can get a pretty high level of detail.

01:48:42.320 | - And somehow you can walk along those visualizations

01:48:44.400 | and come up with an idea, which is weird.

01:48:47.240 | - But when you're thinking about solving problems,

01:48:50.140 | like you're putting information in,

01:48:53.000 | you're exercising the stuff you do know,

01:48:55.760 | you're sort of teasing the area that you don't understand

01:48:59.400 | and don't know, but you can almost feel

01:49:03.080 | that process happening.

01:49:06.480 | Like I know sometimes when I'm working really hard

01:49:12.040 | on something, I get really hot when I'm sleeping.

01:49:14.920 | And it's like, I got the blanket throw,

01:49:17.320 | I wake up, I hold a blanket throw on the floor.

01:49:20.080 | And every time, it's wow, I wake up and think,

01:49:23.000 | wow, that was great.

01:49:24.100 | - Are you able to reverse engineer

01:49:27.600 | what the hell happened there?

01:49:28.980 | - Well, sometimes it's vivid dreams

01:49:30.360 | and sometimes it's this kind of, like you say,

01:49:32.480 | like shadow thinking that you sort of have this feeling

01:49:35.120 | you're going through this stuff, but it's not that obvious.

01:49:38.720 | - Isn't that so amazing that the mind

01:49:40.280 | just does all these little experiments?

01:49:42.880 | I never, I always thought it's like a river

01:49:46.040 | that you can't, you're just there for the ride.

01:49:48.160 | But you're right, if you prep it, maybe--

01:49:51.160 | - It's all understandable.

01:49:52.400 | Meditation really helps.

01:49:53.720 | You gotta start figuring out,

01:49:55.160 | you need to learn the language of your own mind.

01:49:58.420 | And there's multiple levels of it.

01:50:01.240 | But-- - The abstractions again, right?

01:50:04.020 | - It's somewhat comprehensible and observable

01:50:06.680 | and feelable or whatever the right word is.

01:50:10.000 | Yeah, you're not alone for the ride.

01:50:13.680 | You are the ride.

01:50:15.600 | - I have to ask you, hardware engineer,

01:50:17.960 | working on neural networks now, what's consciousness?

01:50:21.400 | What the hell is that thing?

01:50:22.840 | Is that just some little weird quirk

01:50:25.960 | of our particular computing device?

01:50:29.280 | Or is it something fundamental

01:50:30.600 | that we really need to crack open

01:50:32.040 | if we're to build good computers?

01:50:36.560 | Do you ever think about consciousness?

01:50:37.960 | Like why it feels like something to be--

01:50:39.960 | - I know, it's really weird.

01:50:41.760 | So, I mean, everything about it's weird.

01:50:47.640 | First, it's a half a second behind reality.

01:50:50.160 | It's a post hoc narrative about what happened.

01:50:53.780 | You've already done stuff

01:50:55.640 | by the time you're conscious of it.

01:50:58.880 | And your consciousness generally

01:51:00.120 | is a single threaded thing,

01:51:01.240 | but we know your brain is 10 billion neurons

01:51:03.720 | running some crazy parallel thing.

01:51:07.980 | And there's a really big sorting thing going on there.

01:51:11.160 | It also seems to be really reflective

01:51:13.040 | in the sense that you create a space in your head.

01:51:18.040 | Like we don't really see anything, right?

01:51:19.640 | Like photons hit your eyes, it gets turned into signals,

01:51:22.880 | it goes through multiple layers of neurons.

01:51:25.040 | I'm so curious that that looks glassy

01:51:29.160 | and that looks not glassy.

01:51:31.080 | Like how the resolution of your vision is so high

01:51:33.520 | you have to go through all this processing.

01:51:36.080 | Where for most of it, it looks nothing like vision.

01:51:38.760 | Like there's no theater in your mind, right?

01:51:43.640 | So we have a world in our heads.

01:51:46.820 | We're literally just isolated behind our sensors.

01:51:51.760 | But we can look at it, speculate about it,

01:51:55.600 | speculate about alternatives, problem solve, what if.

01:52:00.240 | There's so many things going on

01:52:02.880 | and that process is lagging reality.

01:52:06.200 | - And it's single threaded

01:52:07.600 | even though the underlying thing is like massively parallel.

01:52:10.480 | - So it's so curious.

01:52:12.800 | So imagine you're building an AI computer.

01:52:14.520 | If you wanted to replicate humans,

01:52:16.400 | well you'd have huge arrays of neural networks

01:52:18.360 | and apparently only six or seven deep,

01:52:20.480 | which is hilarious.

01:52:22.440 | They don't even remember seven numbers

01:52:23.800 | but I think we can upgrade that a lot.

01:52:26.280 | And then somewhere in there,

01:52:28.280 | you would train the network to create basically

01:52:30.640 | the world that you live in.

01:52:32.880 | - So it tells stories to itself

01:52:34.880 | about the world that it's perceiving.

01:52:36.840 | - Well, create the world, tell stories in the world

01:52:40.840 | and then have many dimensions of,

01:52:43.660 | like side shows to it.

01:52:47.840 | We have an emotional structure.

01:52:49.560 | We have a biological structure

01:52:51.520 | and that seems hierarchical too.

01:52:53.560 | Like if you're hungry, it dominates your thinking.

01:52:55.620 | If you're mad, it dominates your thinking.

01:52:57.720 | And we don't know if that's important

01:53:00.360 | to consciousness or not,

01:53:01.320 | but it certainly disrupts, intrudes in the consciousness.

01:53:05.080 | So there's lots of structure to that

01:53:08.160 | and we like to dwell on the past.

01:53:09.880 | We like to think about the future.

01:53:11.280 | We like to imagine, we like to fantasize.

01:53:13.460 | And the somewhat circular observation of that

01:53:18.560 | is the thing we call consciousness.

01:53:20.600 | Now, if you created a computer system

01:53:23.320 | and did all things, create worldviews,

01:53:24.880 | create the future alternate histories,

01:53:27.560 | dwelled on past events accurately or semi-accurately.

01:53:31.300 | - Will consciousness just spring up like naturally?

01:53:35.320 | - Well, would that look and feel conscious to you?

01:53:38.040 | Like you seem conscious to me, but I don't know.

01:53:39.720 | - External observer sense.

01:53:41.760 | Do you think a thing that looks conscious is conscious?

01:53:44.960 | Like do you, again, this is like an engineering

01:53:48.240 | kind of question I think, because,

01:53:49.960 | like if we want to engineer consciousness,

01:53:56.840 | is it okay to engineer something that just looks conscious?

01:53:59.800 | Or is there a difference between--

01:54:02.640 | - Well, we have all consciousness

01:54:04.080 | 'cause it's a super effective way to manage our affairs.

01:54:07.160 | - Yeah, it's a social element, yeah.

01:54:09.040 | - Well, it gives us a planning system.

01:54:11.540 | We have a huge amount of stuff.

01:54:13.280 | Like when we're talking,

01:54:14.720 | like the reason we can talk really fast

01:54:16.200 | is we're modeling each other in really high-level detail.

01:54:19.120 | - And consciousness is required for that.

01:54:21.360 | - Well, all those components together

01:54:23.760 | manifest consciousness, right?

01:54:26.720 | So if we make intelligent beings

01:54:28.440 | that we want to interact with,

01:54:29.600 | that we're like wondering what they're thinking,

01:54:32.840 | looking forward to seeing them,

01:54:35.120 | when they interact with them,

01:54:36.480 | they're interesting, surprising, fascinating,

01:54:40.480 | they will probably feel conscious like we do

01:54:43.480 | and we'll perceive them as conscious.

01:54:45.380 | I don't know why not, but you never know.

01:54:49.960 | - Another fun question on this,

01:54:51.440 | because from a computing perspective,

01:54:55.040 | we're trying to create something

01:54:56.000 | that's human-like or superhuman-like.

01:54:57.840 | Let me ask you about aliens.

01:55:01.280 | - Aliens.

01:55:02.120 | - Do you think there's intelligent

01:55:06.640 | alien civilizations out there?

01:55:09.040 | And do you think their technology,

01:55:13.140 | their computing, their AI bots,

01:55:16.080 | their chips are of the same nature as ours?

01:55:21.280 | - Yeah, I've got no idea.

01:55:23.760 | If there's lots of aliens out there,

01:55:24.960 | they've been awfully quiet.

01:55:26.320 | I mean, there's speculation about why.

01:55:29.600 | There seems to be more than enough planets out there.

01:55:34.960 | - There's a lot.

01:55:35.800 | - There's intelligent life on this planet

01:55:38.960 | that seems quite different.

01:55:41.720 | Dolphins seem plausibly understandable.

01:55:44.620 | Octopuses don't seem understandable at all.

01:55:47.660 | If they lived longer than a year,

01:55:48.820 | maybe they would be running the planet.

01:55:51.000 | They seem really smart.

01:55:52.740 | And their neural architecture

01:55:54.260 | is completely different than ours.

01:55:56.540 | Now, who knows how they perceive things.

01:55:58.700 | - I mean, that's the question,

01:55:59.540 | is for us intelligent beings,

01:56:01.180 | we might not be able to perceive

01:56:02.380 | other kinds of intelligence

01:56:03.620 | if they become sufficiently different than us.

01:56:05.580 | So we cannot understand all of this.

01:56:06.780 | - We live in the current constrained world.

01:56:09.180 | It's three-dimensional geometry

01:56:10.660 | and the geometry defines a certain amount of physics.

01:56:14.480 | And there's how time work seems to work.

01:56:18.560 | There's so many things that seem like

01:56:21.120 | a whole bunch of the input parameters

01:56:22.880 | to another conscious being are the same.

01:56:25.540 | Like if it's biological,

01:56:28.180 | biological things seem to be

01:56:30.000 | in a relatively narrow temperature range.

01:56:32.000 | Because organics aren't stable, too cold or too hot.

01:56:38.240 | So if you specify the list of things that input to that,

01:56:43.240 | but soon as we make really smart beings

01:56:49.580 | and they go solve about how to think

01:56:51.140 | about a billion numbers at the same time

01:56:52.940 | and how to think in n dimensions.

01:56:56.060 | There's a funny science fiction book

01:56:57.340 | where all the society had uploaded into this matrix.

01:57:01.620 | And at some point, some of the beings in the matrix thought,

01:57:05.340 | I wonder if there's intelligent life out there.

01:57:07.900 | So they had to do a whole bunch of work to figure out

01:57:09.940 | like how to make a physical thing

01:57:12.380 | 'cause their matrix was self-sustaining.

01:57:15.000 | And they made a little spaceship

01:57:16.140 | and they traveled to another planet.

01:57:17.760 | When they got there, there was like life running around,

01:57:20.660 | but there was no intelligent life.

01:57:22.660 | And then they figured out that there was these huge,

01:57:25.300 | organic matrix all over the planet.

01:57:28.780 | Inside there were intelligent beings

01:57:30.540 | and they uploaded themselves into that matrix.

01:57:34.980 | So everywhere intelligent life was, soon as it got smart,

01:57:39.980 | it up-leveled itself into something way more interesting

01:57:43.620 | than 3D geometry.

01:57:45.180 | - Yeah, it escaped, whatever this,

01:57:47.140 | up-leveled is better. - No, not escaped.

01:57:48.300 | - Up-leveled is better.

01:57:49.780 | The essence of what we think of as an intelligent being,

01:57:53.220 | I tend to like the thought experiment of the organism,

01:57:58.140 | like humans aren't the organisms.

01:58:00.380 | I like the notion of like Richard Dawkins and memes

01:58:03.740 | that ideas themselves are the organisms

01:58:07.340 | that are just using our minds to evolve.

01:58:11.500 | So we're just like meat receptacles

01:58:15.220 | for ideas to breed and multiply and so on.

01:58:18.140 | And maybe those are the aliens.

01:58:21.000 | - So Jordan Peterson has a line that says,

01:58:26.700 | you think you have ideas, but ideas have you.

01:58:29.180 | - Yeah, good line.

01:58:31.580 | And then we know about the phenomenon of groupthink

01:58:34.220 | and there's so many things that constrain us.

01:58:37.940 | But I think you can examine all that

01:58:39.900 | and not be completely owned by the ideas

01:58:43.300 | and completely sucked into groupthink.

01:58:46.100 | And part of your responsibility as a human

01:58:48.940 | is to escape that kind of phenomena,

01:58:51.700 | which isn't, it's one of the creative tension things again.

01:58:55.900 | You're constructed by it, but you can still observe it

01:58:59.460 | and you can think about it and you can make choices

01:59:01.780 | about to some level, how constrained you are by it.

01:59:05.660 | And it's useful to do that.

01:59:09.740 | But at the same time, and it could be by doing that,

01:59:17.440 | the group in society you're part of

01:59:21.460 | becomes collectively even more interesting.

01:59:24.160 | So the outside observer will think,

01:59:27.020 | wow, all these Lexus running around

01:59:30.060 | with all these really independent ideas

01:59:31.540 | have created something even more interesting

01:59:33.700 | in the aggregate.

01:59:34.880 | So I don't know.

01:59:38.480 | Those are lenses to look at the situation.

01:59:41.820 | That'll give you some inspiration,

01:59:43.500 | but I don't think they're constrained.

01:59:45.460 | - Right.

01:59:46.660 | As a small little quirk of history,

01:59:49.340 | it seems like you're related to Jordan Peterson,

01:59:53.700 | like you mentioned.

01:59:54.860 | He's going through some rough stuff now.

01:59:57.620 | Is there some comment you can make

01:59:59.180 | about the roughness of the human journey,

02:00:02.620 | the ups and downs?

02:00:04.260 | - Well, I became an expert in Benzo withdrawal,

02:00:09.260 | which is you took Benzo's aspens and at some point

02:00:14.860 | they interact with GABA circuits to reduce anxiety

02:00:19.860 | and do a hundred other things.

02:00:22.140 | There's actually no known list of everything

02:00:24.700 | they do 'cause they interact with so many parts

02:00:26.660 | of your body.

02:00:28.220 | And then once you're on them, you habituate to them

02:00:30.500 | and you have a dependency.

02:00:32.620 | It's not like you're a drug dependency

02:00:34.180 | where you're trying to get high.

02:00:35.060 | It's a metabolic dependency.

02:00:38.860 | And then if you discontinue them,

02:00:41.060 | there's a funny thing called kindling,

02:00:44.520 | which is if you stop them and then go,

02:00:47.580 | you'll have a horrible withdrawal symptoms.

02:00:49.940 | If you go back on them at the same level,

02:00:51.500 | you won't be stable.

02:00:53.300 | And that unfortunately happened to him.

02:00:55.900 | - Because it's so deeply integrated

02:00:57.260 | into all the kinds of systems in the body.

02:00:58.980 | - It literally changes the size and numbers

02:01:00.860 | of neurotransmitter sites in your brain.

02:01:03.900 | So there's a process called the Ashton Protocol

02:01:07.420 | where you taper it down slowly over two years.

02:01:10.340 | The people go through that, go through unbelievable hell.

02:01:13.740 | And what Jordan went through seemed to be worse

02:01:15.700 | because on advice of doctors,

02:01:18.580 | well, stop taking these and take this.

02:01:20.340 | It was a disaster.

02:01:21.380 | And he got some, yeah, it was pretty tough.

02:01:24.940 | He seems to be doing quite a bit better intellectually.

02:01:29.220 | You can see his brain clicking back together.

02:01:32.020 | I spent a lot of time with him.

02:01:32.940 | I've never seen anybody suffer so much.

02:01:34.940 | - Well, his brain is also like this powerhouse, right?

02:01:37.700 | So I wonder, does a brain that's able to think deeply

02:01:42.500 | about the world suffer more through these kinds of withdrawals?

02:01:46.260 | - I don't know.

02:01:47.100 | I've watched videos of people going through withdrawal.

02:01:49.500 | They all seem to suffer unbelievably.

02:01:52.660 | And my heart goes out to everybody.

02:01:57.540 | And there's some funny math about this.

02:01:59.260 | Some doctor says, best he can tell,

02:02:01.220 | the standard recommendations,

02:02:03.540 | don't take them for more than a month

02:02:04.740 | and then taper over a couple of weeks.

02:02:07.140 | Many doctors prescribe them endlessly,

02:02:09.340 | which is against the protocol, but it's common, right?

02:02:13.140 | And then something like 75% of people,

02:02:16.620 | when they taper, it's, you know,

02:02:18.540 | half the people have difficulty, but 75% get off okay.

02:02:22.100 | 20% have severe difficulty,

02:02:24.020 | and 5% have life-threatening difficulty.

02:02:27.280 | And if you're one of those, it's really bad.

02:02:29.540 | And the stories that people have on this

02:02:31.540 | is heartbreaking and tough.

02:02:34.940 | - So you put some of the fault at the doctors.

02:02:36.820 | They just not know what the hell they're doing.

02:02:38.620 | - Oh, no, it's hard to say.

02:02:40.540 | It's one of those commonly prescribed things.

02:02:43.100 | Like one doctor said, what happens is,

02:02:46.100 | if you're prescribed them for a reason,

02:02:47.780 | and then you have a hard time getting off,

02:02:49.860 | the protocol basically says you're either crazy or dependent

02:02:53.220 | and you get kind of pushed into a different treatment regime

02:02:58.340 | where a drug addict or a psychiatric patient.

02:03:01.780 | And so like one doctor said, you know,

02:03:04.060 | I prescribed them for 10 years

02:03:05.220 | thinking I was helping my patients

02:03:06.540 | and I realized I was really harming them.

02:03:08.580 | And, you know, the awareness of that is slowly coming up.

02:03:14.400 | The fact that they're casually prescribed to people

02:03:18.120 | is horrible and it's bloody scary.

02:03:22.220 | And some people are stable on them,

02:03:25.000 | but they're on them for life.

02:03:26.200 | Like once you, you know, it's another one of those drugs.

02:03:29.220 | But benzos long range have real impacts on your personality.

02:03:32.520 | People talk about the benzo bubble

02:03:34.100 | where you get disassociated from reality

02:03:36.280 | and your friends a little bit.

02:03:38.160 | It's really terrible.

02:03:40.320 | - The mind is terrifying.

02:03:41.680 | We were talking about how the infinite possibility of fun,

02:03:45.440 | but like it's the infinite possibility of suffering too,

02:03:48.640 | which is one of the dangers of like expansion

02:03:52.320 | of the human mind.

02:03:53.480 | It's like, I wonder if all the possible experiences

02:03:58.220 | that an intelligent computer can have,

02:04:01.680 | is it mostly fun or is it mostly suffering?

02:04:05.840 | So like if you brute force expand

02:04:10.160 | the set of possibilities, like are you going to run

02:04:12.640 | into some trouble in terms of like torture

02:04:15.240 | and suffering and so on?

02:04:16.600 | Maybe our human brain is just protecting us

02:04:18.880 | from much more possible pain and suffering.

02:04:22.300 | Maybe the space of pain is like much larger

02:04:26.000 | than we could possibly imagine.

02:04:27.560 | And that--

02:04:28.380 | - The world's in a balance.

02:04:29.580 | You know, all the literature on religion and stuff is,

02:04:34.240 | you know, the struggle between good and evil

02:04:36.320 | is balanced for us, very finely tuned

02:04:39.400 | for reasons that are complicated.

02:04:41.640 | But that's a long philosophical conversation.

02:04:44.880 | - Speaking of balance that's complicated,

02:04:46.680 | I wonder because we're living through one

02:04:48.640 | of the more important moments in human history

02:04:51.600 | with this particular virus, it seems like pandemics

02:04:55.840 | have at least the ability to kill off

02:04:59.040 | most of the human population at their worst.

02:05:03.040 | And there's just fascinating 'cause there's so many viruses

02:05:05.520 | in this world, there's so many.

02:05:06.720 | I mean, viruses basically run the world

02:05:08.560 | in the sense that they've been around a very long time.

02:05:12.240 | They're everywhere.

02:05:13.680 | They seem to be extremely powerful

02:05:15.320 | in a distributed kind of way,

02:05:17.240 | but at the same time, they're not intelligent

02:05:19.600 | and they're not even living.

02:05:21.240 | Do you have like high level thoughts about this virus

02:05:23.800 | that like in terms of you being fascinated

02:05:27.280 | or terrified or somewhere in between?

02:05:30.360 | - So I believe in frameworks, right?

02:05:32.480 | So like one of them is evolution.

02:05:35.400 | Like we're evolved creatures, right?

02:05:37.840 | - Yes.

02:05:38.960 | - And one of the things about evolution

02:05:40.880 | is it's hyper competitive.

02:05:42.760 | And it's not competitive out of a sense of evil,

02:05:44.880 | it's competitive in a sense of there's endless variation

02:05:47.800 | and variations that work better win.

02:05:50.360 | And then over time, there's so many levels

02:05:52.960 | of that competition.

02:05:54.160 | Like multicellular life partly exists

02:05:57.720 | because of the competition between different kinds

02:06:02.640 | of life forms.

02:06:04.240 | And we know sex partly exists to scramble our genes

02:06:06.880 | so that we have genetic variation against the invasion

02:06:11.880 | of the bacteria and the viruses and it's endless.

02:06:16.040 | Like I read some funny statistic,

02:06:18.040 | like the density of viruses and bacteria in the ocean

02:06:20.800 | is really high.

02:06:22.040 | And one third of the bacteria die every day

02:06:23.920 | because the virus is invading them.

02:06:26.240 | Like one third of them.

02:06:27.960 | - Wow.

02:06:29.040 | - Like I don't know if that number is true,

02:06:31.040 | but it was like, the amount of competition

02:06:34.920 | and what's going on is stunning.

02:06:36.560 | And there's a theory as we age,

02:06:38.680 | we slowly accumulate bacterias and viruses

02:06:41.800 | and as our immune system kind of goes down,

02:06:45.640 | that's what slowly kills us.

02:06:47.800 | - It just feels so peaceful from a human perspective

02:06:50.240 | when we sit back and are able to have a relaxed conversation

02:06:53.240 | and there's wars going on out there.

02:06:56.800 | - Like right now, you're harboring how many bacteria?

02:07:01.480 | - There's ones, many of them are parasites on you

02:07:04.840 | and some of them are helpful

02:07:06.040 | and some of them are modifying your behavior

02:07:07.760 | and some of them are, it's really wild.

02:07:12.240 | But this particular manifestation is unusual

02:07:16.440 | in the demographic, how it hit,

02:07:18.400 | and the political response that it engendered

02:07:21.360 | and the healthcare response it engendered

02:07:23.840 | and the technology it engendered, it's kind of wild.

02:07:27.120 | - Yeah, the communication on Twitter that it led to.

02:07:30.000 | - Every level.

02:07:30.840 | - Yeah, all that kind of stuff, at every single level, yeah.

02:07:32.960 | - But what usually kills life,

02:07:34.600 | the big extinctions are caused by meteors and volcanoes.

02:07:39.440 | - That's the one you're worried about

02:07:40.800 | as opposed to human-created bombs that we launch--

02:07:44.520 | - Solar flares are another good one.

02:07:46.560 | Occasionally, solar flares hit the planet.

02:07:48.640 | - So it's nature.

02:07:49.600 | - Yeah, it's all pretty wild.

02:07:52.760 | - Another historic moment, this is perhaps outside

02:07:57.520 | but perhaps within your space of frameworks

02:08:02.520 | that you think about that just happened,

02:08:04.560 | I guess, a couple weeks ago is,

02:08:06.640 | I don't know if you're paying attention at all,

02:08:08.040 | is the GameStop and Wall Street Bets.

02:08:12.440 | - It was super fun.

02:08:14.120 | - So it's really fascinating.

02:08:16.520 | There's kind of a theme to this conversation we're having

02:08:18.520 | today 'cause it's like neural networks,

02:08:20.720 | it's cool how there's a large number of people

02:08:25.040 | in a distributed way, almost having a kind of fund,

02:08:30.040 | were able to take on the powerful elite hedge funds,

02:08:35.360 | centralized powers, and overpower them.

02:08:39.080 | Do you have thoughts on this whole saga?

02:08:43.360 | - I don't know enough about finance

02:08:45.040 | but it was like the Elon, Robin Hood guy when they talked.

02:08:49.280 | - Yeah, what'd you think about that?

02:08:51.560 | - Well, the Robin Hood guy didn't know

02:08:52.680 | how the finance system worked, that was clear.

02:08:55.560 | He was treating the people who settled the transactions

02:08:58.560 | as a black box and suddenly somebody called him up

02:09:01.360 | and said, "Hey, black box calling you.

02:09:03.520 | "Your transaction volume means you need

02:09:05.160 | "to put out $3 billion right now."

02:09:06.920 | And he's like, "I don't have $3 billion.

02:09:08.880 | "I don't even make any money on these trades.

02:09:10.440 | "Why do I owe $3 billion while you're sponsoring the trade?"

02:09:13.200 | So there was a set of abstractions that,

02:09:16.160 | I don't think either, like now we understand it.

02:09:19.920 | This happens in chip design.

02:09:21.880 | You buy wafers from TSMC or Samsung or Intel

02:09:25.680 | and they say it works like this

02:09:27.480 | and you do your design based on that

02:09:29.040 | and then chip comes back and it doesn't work.

02:09:31.320 | And then suddenly you start having to open the black boxes.

02:09:34.320 | Do transistors really work like they said?

02:09:36.440 | What's the real issue?

02:09:37.640 | There's a whole set of things that created this opportunity

02:09:44.960 | and somebody spotted it.

02:09:46.280 | Now, people spot these kinds of opportunities all the time.

02:09:49.920 | There's been flash crashes.

02:09:51.520 | Short squeezes are fairly regular.

02:09:55.360 | Every CEO I know hates the shorts

02:09:58.480 | because they're trying to manipulate their stock

02:10:01.840 | in a way that they make money

02:10:03.840 | and deprive value from both the company and the investors.

02:10:08.840 | So the fact that some of these stocks were so short,

02:10:13.680 | it's hilarious, that this hasn't happened before.

02:10:17.320 | I don't know why.

02:10:18.160 | I don't actually know why some serious hedge funds

02:10:21.120 | didn't do it to other hedge funds.

02:10:23.440 | And some of the hedge funds

02:10:24.360 | actually made a lot of money on this.

02:10:26.600 | So my guess is we know 5% of what really happened

02:10:31.600 | and that a lot of the players don't know what happened

02:10:34.440 | and the people who probably made the most money

02:10:37.440 | aren't the people that they're talking about.

02:10:39.640 | - Do you think there was something,

02:10:42.720 | I mean, this is the cool kind of Elon,

02:10:48.000 | you're the same kind of conversationalist,

02:10:50.720 | which is like first principles questions

02:10:52.960 | of like what the hell happened.

02:10:56.320 | Just very basic questions of like,

02:10:57.960 | was there something shady going on?

02:10:59.920 | What, who are the parties involved?

02:11:03.680 | It's the basic questions that everybody wants to know about.

02:11:06.320 | - Yeah, so we're in a very hyper-competitive world, right?

02:11:10.320 | But transactions like buying and selling stock

02:11:12.200 | is a trust event.

02:11:13.800 | I trust the company, represented themselves properly.

02:11:17.000 | I bought the stock 'cause I think it's gonna go up.

02:11:19.680 | I trust that the regulations are solid.

02:11:22.680 | Now, inside of that, there's all kinds of places

02:11:26.120 | where humans over trust and this expose, let's say,

02:11:31.120 | some weak points in the system.

02:11:34.600 | I don't know if it's gonna get corrected.

02:11:37.320 | I don't know if we have close to the real story.

02:11:40.860 | My suspicion is we don't.

02:11:44.480 | And listen to that guy, he was like a little wide-eyed

02:11:47.280 | about and then he did this and then he did that.

02:11:49.080 | And I was like, I think you should know more

02:11:51.840 | about your business than that.

02:11:54.200 | But again, there's many businesses

02:11:56.160 | when like this layer is really stable,

02:11:58.780 | you stop paying attention to it.

02:12:00.720 | You pay attention to the stuff that's bugging you or new.

02:12:04.360 | Like you don't pay attention to the stuff

02:12:05.780 | that just seems to work all the time.

02:12:07.080 | You just, you know, sky's blue every day, California.

02:12:11.080 | And once in a while, it rains there.

02:12:12.880 | It's like, what do we do?

02:12:14.280 | Somebody go bring in the lawn furniture.

02:12:17.280 | You know, like it's getting wet.

02:12:18.720 | You don't know why it's getting wet.

02:12:20.000 | - Yeah, it doesn't know.

02:12:20.840 | - I was blue for like 100 days and now it's, you know.

02:12:24.560 | - But part of the problem here with Vlad,

02:12:27.040 | the CEO of Robinhood is the scaling

02:12:29.560 | that we've been talking about is there's a lot

02:12:32.560 | of unexpected things that happen with the scaling.

02:12:36.040 | And you have to be, I think the scaling forces you

02:12:39.680 | to then return to the fundamentals.

02:12:41.800 | - Well, it's interesting because when you buy

02:12:43.640 | and sell stocks, the scaling is, you know,

02:12:45.600 | the stocks only move in a certain range.

02:12:47.280 | And if you buy a stock,

02:12:48.200 | you can only lose that amount of money.

02:12:50.040 | On the short market, you can lose a lot more

02:12:52.440 | than you can benefit.

02:12:53.840 | Like it has a weird cost function

02:12:57.240 | or whatever the right word for that is.

02:12:59.280 | So he was trading in a market

02:13:01.120 | where he wasn't actually capitalized for the downside.

02:13:04.200 | If it got outside a certain range.

02:13:06.280 | Now, whether something nefarious has happened,

02:13:09.800 | I have no idea, but at some point,

02:13:12.000 | the financial risk to both him and his customers

02:13:16.580 | was way outside of his financial capacity

02:13:19.200 | and his understanding how the system work was clearly weak

02:13:23.400 | or he didn't represent himself.

02:13:25.160 | I don't know the person.

02:13:26.800 | - There's a--

02:13:27.640 | - When I listened to him,

02:13:28.840 | it could have been the surprise question was like,

02:13:30.520 | and then these guys called and, you know,

02:13:32.680 | it sounded like he was treating stuff as a black box.

02:13:36.280 | Maybe he shouldn't have, but maybe he has a whole pile

02:13:38.560 | of experts somewhere else and it was going on.

02:13:40.080 | I don't know.

02:13:41.240 | - Yeah.

02:13:42.320 | I mean, this is one of the qualities of a good leader

02:13:46.320 | is under fire, you have to perform.

02:13:49.120 | And that means to think clearly and to speak clearly.

02:13:53.080 | And he dropped the ball on those things

02:13:55.320 | 'cause, and understand the problem,

02:13:57.480 | quickly learn and understand the problem

02:13:59.720 | like at the basic level, like what the hell happened.

02:14:05.080 | And my guess is, you know, at some level,

02:14:08.520 | it was amateurs trading against, you know,

02:14:10.880 | experts/insiders/people with, you know, special information.

02:14:14.880 | - Outsiders versus insiders.

02:14:16.880 | - Yeah.

02:14:17.720 | And the insiders, you know, my guess is the next time

02:14:20.680 | this happens, we'll make money on it.

02:14:22.960 | - The insiders always win?

02:14:25.080 | - Well, they have more tools and more incentive.

02:14:27.120 | I mean, this always happens.

02:14:28.440 | Like the outsiders are doing this for fun.

02:14:30.800 | The insiders are doing this 24/7.

02:14:33.280 | - But there's numbers in the outsiders.

02:14:35.680 | This is the interesting thing is,

02:14:36.720 | it could be a new chapter.

02:14:37.560 | - Well, there's numbers on the insiders too.

02:14:39.680 | - Different kind of numbers.

02:14:42.080 | - Different kind of numbers.

02:14:44.000 | - But this could be a new era because, I don't know,

02:14:46.040 | at least I didn't expect that a bunch of Redditors could,

02:14:49.080 | you know, there's millions of people can get together.

02:14:51.280 | - It was a surprise attack.

02:14:52.280 | The next one will be a surprise.

02:14:54.160 | - But don't you think the crowd,

02:14:56.440 | the people are planning the next attack?

02:14:59.200 | - We'll see.

02:15:00.480 | It has to be a surprise, it can't be the same game.

02:15:03.000 | - And so the insiders--

02:15:05.400 | - It could be there's a very large number of games to play

02:15:08.800 | and they can be agile about it.

02:15:10.760 | I don't know, I'm not an expert.

02:15:12.160 | - Right, that's a good question.

02:15:13.760 | The space of games, how restricted is it?

02:15:16.520 | - Yeah, and the system is so complicated,

02:15:20.200 | it could be relatively unrestricted.

02:15:22.680 | And also like, you know,

02:15:24.160 | during the last couple of financial crashes,

02:15:26.600 | you know, what set it off was, you know,

02:15:28.720 | sets of derivative events where, you know,

02:15:31.960 | Nassim Taleb's, you know, thing is,

02:15:34.560 | they're trying to lower volatility in the short run

02:15:39.400 | by creating tail events.

02:15:41.600 | And systems always evolve towards that

02:15:43.680 | and then they always crash.

02:15:45.600 | Like the S curve is the, you know,

02:15:47.840 | star low, ramp, plateau, crash.

02:15:51.560 | It's 100% effective.

02:15:53.040 | - In the long run.

02:15:56.000 | Let me ask you some advice to put on your profound hat.

02:15:59.800 | There's a bunch of young folks who listen to this thing

02:16:04.680 | for no good reason whatsoever.

02:16:07.480 | Undergraduate students, maybe high school students,

02:16:10.640 | maybe just young folks, young at heart,

02:16:13.080 | looking for the next steps to take in life.

02:16:16.880 | What advice would you give to a young person today

02:16:19.360 | about life, maybe career, but also life in general?

02:16:23.880 | - Get good at some stuff.

02:16:26.120 | Well, get to know yourself, right?

02:16:28.240 | Like get good at something

02:16:29.240 | that you're actually interested in.

02:16:30.640 | You have to love what you're doing to get good at it.

02:16:33.480 | You really gotta find that.

02:16:34.400 | Don't waste all your time doing stuff

02:16:35.800 | that's just boring or bland or numbing, right?

02:16:40.160 | Don't let old people screw you.

02:16:41.720 | Well, people get talked into doing all kinds of shit

02:16:46.720 | and racking up huge student debts

02:16:49.320 | and like there's so much crap going on, you know?

02:16:52.600 | - And it drains your time and drains your--

02:16:54.640 | - You know, the Eric Weinstein thesis

02:16:56.520 | that the older generation won't let go

02:16:59.560 | and they're trapping all the young people.

02:17:01.160 | - I think there's some truth to that.

02:17:02.480 | - Yeah, sure.

02:17:03.320 | Just because you're old doesn't mean you stop thinking.

02:17:06.960 | I know lots of really original old people.

02:17:10.400 | I'm an old person.

02:17:11.360 | But you have to be conscious about it.

02:17:15.680 | You can fall into the ruts and then do that.

02:17:18.960 | I mean, when I hear young people spouting opinions,

02:17:22.080 | it sounds like they come from Fox News or CNN,

02:17:24.400 | I think they've been captured by groupthink and memes.

02:17:28.000 | - They're supposed to think on their own.

02:17:29.800 | - So if you find yourself repeating

02:17:31.440 | what everybody else is saying,

02:17:33.440 | you're not gonna have a good life.

02:17:35.140 | Like that's not how the world works.

02:17:38.480 | It seems safe, but it puts you at great jeopardy

02:17:41.080 | for being boring or unhappy.

02:17:45.920 | - How long did it take you to find the thing

02:17:47.800 | that you have fun with?

02:17:50.040 | - I don't know.

02:17:52.160 | I've been a fun person since I was pretty little.

02:17:54.320 | - So everything.

02:17:55.160 | - I've gone through a couple periods

02:17:56.160 | of depression in my life.

02:17:58.120 | - For a good reason or for a reason

02:18:00.200 | that doesn't make any sense?

02:18:02.640 | - Yeah.

02:18:03.480 | Yeah, like some things are hard.

02:18:06.000 | Like you go through mental transitions in high school.

02:18:08.920 | I was really depressed for a year.

02:18:10.720 | And I think I had my first midlife crisis at 26.

02:18:15.160 | I kind of thought, is this all there is?

02:18:16.640 | Like I was working at a job that I loved,

02:18:19.360 | but I was going to work and all my time was consumed.

02:18:23.400 | - What's the escape out of that depression?

02:18:25.800 | What's the answer to is this all there is?

02:18:29.200 | - Well, a friend of mine, I asked him

02:18:31.800 | 'cause he was working his ass off.

02:18:32.880 | I said, "What's your work-life balance?"

02:18:34.520 | Like there's work, friends, family, personal time.

02:18:39.520 | Are you balancing in that?

02:18:41.400 | And he said, "Work 80%, family 20%."

02:18:43.560 | And I tried to find some time to sleep.

02:18:47.560 | Like there's no personal time.

02:18:49.200 | There's no passionate time.

02:18:51.800 | Like young people are often passionate about work.

02:18:54.600 | So, and I was sort of like that.

02:18:56.960 | But you need to have some space in your life

02:18:59.920 | for different things.

02:19:01.800 | - And that creates, that makes you resistant

02:19:05.840 | to the deep dips into depression kind of thing.

02:19:10.840 | - Yeah, well, you have to get to know yourself too.

02:19:13.040 | Meditation helps.

02:19:14.480 | Some physical, something physically intense helps.

02:19:18.520 | - Like the weird places your mind goes kind of thing.

02:19:22.200 | - And why does it happen?

02:19:23.800 | Why do you do what you do?

02:19:24.880 | - Like triggers, like the things that cause your mind

02:19:27.680 | to go to different places kind of thing,

02:19:29.480 | or like events.

02:19:32.240 | - Your upbringing, for better or worse,

02:19:33.760 | whether your parents are great people or not,

02:19:35.680 | you come into adulthood with all kinds of emotional burdens.

02:19:40.680 | And you can see some people are so bloody stiff

02:19:45.080 | and restrained and they think the world's

02:19:47.200 | fundamentally negative, like you maybe.

02:19:50.640 | You have unexplored territory.

02:19:53.000 | - Yeah.

02:19:53.960 | - Or you're afraid of something.

02:19:56.280 | - Definitely afraid of quite a few things.

02:19:58.520 | - Then you gotta go face 'em.

02:20:00.100 | Like what's the worst thing that can happen?

02:20:03.440 | You're gonna die, right?

02:20:05.160 | Like that's inevitable.

02:20:06.360 | You might as well get over that, like 100%, that's right.

02:20:09.800 | Like people are worried about the virus,

02:20:11.080 | but the human condition is pretty deadly.

02:20:14.520 | - There's something about embarrassment that's,

02:20:16.720 | I've competed a lot in my life,

02:20:18.240 | and I think if I'm to introspect it,

02:20:22.000 | the thing I'm most afraid of is being humiliated, I think.

02:20:26.640 | - Nobody cares about that.

02:20:28.080 | Like you're the only person on the planet

02:20:30.200 | who cares about you being humiliated.

02:20:32.400 | It's like a really useless thought.

02:20:34.760 | - It is.

02:20:35.600 | - It's like, you're all humiliated,

02:20:39.560 | something happened in a room full of people

02:20:41.160 | and they walk out and they didn't think about it

02:20:42.680 | one more second.

02:20:43.800 | Or maybe somebody told a funny story to somebody else

02:20:46.000 | and then it dissipates throughout, yeah.

02:20:48.000 | - No, I know it too.

02:20:50.280 | I've been really embarrassed about shit

02:20:53.400 | that nobody cared about myself.

02:20:55.520 | - Yeah.

02:20:56.360 | - It's a funny thing.

02:20:57.200 | - So the worst thing ultimately is just--

02:20:59.680 | - Yeah, but that's a cage and you have to get out of it.

02:21:02.600 | Like once you, here's the thing,

02:21:03.880 | once you find something like that,

02:21:05.720 | you have to be determined to break it.

02:21:08.160 | 'Cause otherwise you'll just,

02:21:10.240 | so you accumulate that kind of junk

02:21:11.760 | and then you die as a mess.

02:21:15.440 | - So the goal, I guess it's like a cage within a cage.

02:21:18.440 | I guess the goal is to die in the biggest possible cage.

02:21:21.960 | - Well, ideally you'd have no cage.

02:21:23.760 | People do get enlightened, I've got a few, it's great.

02:21:28.520 | - You found a few?

02:21:29.360 | There's a few out there?

02:21:30.440 | I don't know.

02:21:31.280 | - Of course there are.

02:21:32.120 | - Wow.

02:21:33.360 | - Either that or they have, you know,

02:21:34.560 | it's a great sales pitch.

02:21:35.520 | There's like enlightened people who write books

02:21:37.080 | and do all kinds of stuff.

02:21:38.280 | - It's a good way to sell a book, I'll give you that.

02:21:40.840 | - You've never met somebody you just thought,

02:21:42.880 | they just kill me.

02:21:43.840 | Like this, like mental clarity, humor.

02:21:47.880 | - No, 100%, but I just feel like

02:21:49.560 | they're living in a bigger cage.

02:21:50.960 | They have their own.

02:21:52.000 | - You still think there's a cage?

02:21:53.320 | - There's still a cage.

02:21:54.320 | - You secretly suspect there's always a cage.

02:21:57.520 | - There's nothing outside the universe.

02:21:59.840 | - There's nothing outside the cage.

02:22:01.640 | (laughing)

02:22:03.880 | - You work, you worked at a bunch of companies,

02:22:10.200 | you led a lot of amazing teams.

02:22:13.680 | I don't, I'm not sure if you've ever been like

02:22:16.840 | at the early stages of a startup,

02:22:19.440 | but do you have advice for somebody that wants to do

02:22:24.440 | a startup or build a company,

02:22:28.320 | like build a strong team of engineers that are passionate

02:22:31.200 | and just want to solve a big problem?

02:22:35.000 | Like is there a more specifically on that point?

02:22:39.080 | - Well, you have to be really good at stuff.

02:22:41.400 | If you're gonna lead and build a team,

02:22:43.080 | you better be really interested in how people work and think.

02:22:46.080 | - The people or the solution to the problem.

02:22:49.080 | So there's two things, right?

02:22:50.200 | One is how people work and the other is the--

02:22:52.920 | - Well, actually, there's quite a few successful startups.

02:22:55.680 | It's pretty clear the founders don't know anything

02:22:57.320 | about people.

02:22:58.380 | Like the idea was so powerful that it propelled them.

02:23:01.480 | But I suspect somewhere early,

02:23:03.760 | they hired some people who understood people.

02:23:06.980 | 'Cause people really need a lot of care and feeding

02:23:08.960 | to collaborate and work together

02:23:10.480 | and feel engaged and work hard.

02:23:12.760 | Like startups are all about outproducing other people.

02:23:17.000 | Like you're nimble because you don't have any legacy.

02:23:19.820 | You don't have a bunch of people who are depressed

02:23:22.880 | about life, just showing up.

02:23:24.700 | So startups have a lot of advantages that way.

02:23:28.020 | - Do you like the, Steve Jobs talked about this idea

02:23:32.960 | of A players and B players.

02:23:34.920 | I don't know if you know this formulation.

02:23:37.240 | - Yeah, I know.

02:23:39.040 | - That organizations that get taken over

02:23:41.520 | by B player leaders,

02:23:43.480 | often really underperform their RSC players.

02:23:48.120 | That said in big organizations,

02:23:50.720 | there's so much work to do.

02:23:52.600 | And there's so many people who are happy to do

02:23:54.400 | what the leadership or the big idea people

02:23:57.480 | would consider menial jobs.

02:24:00.320 | And you need a place for them,

02:24:01.880 | but you need an organization that both values

02:24:05.040 | and rewards them,

02:24:05.880 | but doesn't let them take over the leadership of it.

02:24:08.440 | - Got it.

02:24:09.280 | But so you need to have an organization

02:24:11.000 | that's resistant to that.

02:24:11.920 | But in the early days,

02:24:14.280 | the notion with Steve was that like one B player

02:24:18.840 | in a room of A players will be like destructive to the whole.

02:24:23.000 | - I've seen that happen.

02:24:24.320 | I don't know if it's like always true.

02:24:26.520 | Like, you run into people who are clearly B players,

02:24:30.280 | but they think they're A players.

02:24:31.480 | And so they have a loud voice at the table

02:24:33.160 | and they make lots of demands for that.

02:24:35.120 | But there's other people who are like,

02:24:36.080 | I know who I am.

02:24:37.480 | I just want to work with cool people on cool shit

02:24:39.680 | and just tell me what to do and I'll go get it done.

02:24:42.520 | So you have to, again, this is like people skills.

02:24:44.760 | Like what kind of person is it?

02:24:47.960 | I've met some really great people I love working with

02:24:51.040 | that weren't the biggest ID people,

02:24:52.800 | the most productive ever, but they show up,

02:24:54.600 | they get it done.

02:24:55.520 | They create connection and community that people value.

02:24:59.640 | It's pretty diverse.

02:25:01.920 | I don't think there's a recipe for that.

02:25:05.080 | - I gotta ask you about love.

02:25:06.960 | - I heard you're into this now.

02:25:08.640 | - Into this love thing?

02:25:09.560 | - Yeah.

02:25:10.400 | Do you think this is your solution to your depression?

02:25:13.280 | - No, I'm just trying to, like you said,

02:25:14.840 | the enlightened people on occasion, trying to sell a book.

02:25:16.920 | I'm writing a book about love.

02:25:18.080 | - You're writing a book about love?

02:25:18.920 | - No, I'm not.

02:25:19.960 | I'm not.

02:25:20.800 | (laughing)

02:25:23.320 | - A friend of mine, he said,

02:25:26.120 | you should really write a book about your management

02:25:28.600 | philosophy, he said, it'd be a short book.

02:25:30.600 | (laughing)

02:25:35.000 | - Well, that one was thought pretty well.

02:25:37.800 | What role do you think love, family, friendship,

02:25:40.440 | all that kind of human stuff play in a successful life?

02:25:44.400 | You've been exceptionally successful in the space of

02:25:47.480 | like running teams, building cool shit in this world,

02:25:51.160 | creating some amazing things.

02:25:53.160 | What, did love get in the way?

02:25:54.720 | Did love help the family get in the way?

02:25:57.680 | Did family help?

02:25:59.160 | Friendship?

02:26:00.000 | - You want the engineer's answer?

02:26:02.120 | - Please.

02:26:03.120 | - But first, love is functional, right?

02:26:05.840 | - It's functional in what way?

02:26:07.320 | - So, we habituate ourselves to the environment.

02:26:11.040 | And actually Jordan told me,

02:26:12.080 | Jordan Peterson told me this line.

02:26:13.960 | So, you go through life and you just get used to everything,

02:26:16.480 | except for the things you love.

02:26:17.840 | They remain new.

02:26:20.160 | Like, this is really useful for, you know,

02:26:22.480 | like other people's children and dogs and trees.

02:26:26.120 | You just don't pay that much attention to them.

02:26:27.760 | Your own kids, you're monitoring them really closely.

02:26:31.040 | Like, and if they go off a little bit,

02:26:32.760 | because you love them, if you're smart,

02:26:35.320 | if you're gonna be a successful parent,

02:26:37.520 | you notice it right away.

02:26:38.960 | You don't habituate to just things you love.

02:26:43.960 | And if you wanna be successful at work,

02:26:46.160 | if you don't love it,

02:26:47.560 | you're not gonna put the time in somebody else.

02:26:50.400 | It's somebody else that loves it.

02:26:51.640 | Like, 'cause it's new and interesting

02:26:53.760 | and that lets you go to the next level.

02:26:56.060 | - So, it's the thing, it's just a function

02:26:59.120 | that generates newness and novelty

02:27:01.680 | and surprises you and all those kinds of things.

02:27:04.720 | - It's really interesting.

02:27:06.400 | There's people who figured out lots of frameworks for this.

02:27:10.440 | Like, humans seem to go in partnership,

02:27:12.440 | go through interest.

02:27:13.880 | Like, suddenly somebody's interesting

02:27:16.640 | and then you're infatuated with them

02:27:18.200 | and then you're in love with them.

02:27:20.080 | And then, you know, different people have ideas

02:27:22.620 | about parental love or mature love.

02:27:24.520 | Like, you go through a cycle of that,

02:27:26.600 | which keeps us together and it's super functional

02:27:29.520 | for creating families and creating communities

02:27:32.560 | and making you support somebody

02:27:34.560 | despite the fact that you don't love them.

02:27:36.960 | Like, and it can be really enriching.

02:27:41.960 | You know, now, in the work-life balance scheme,

02:27:47.480 | if all you do is work,

02:27:49.760 | you think you may be optimizing your work potential,

02:27:52.340 | but if you don't love your work

02:27:53.840 | or you don't have family and friends

02:27:56.960 | and things you care about,

02:27:59.320 | your brain isn't well-balanced.

02:28:02.000 | Like, everybody knows the experience

02:28:03.320 | of you worked on something all week,

02:28:04.720 | you went home and took two days off and you came back in.

02:28:07.760 | The odds of you working on the thing,

02:28:10.000 | picking up right where you left off is zero.

02:28:12.800 | Your brain refactored it.

02:28:14.300 | But being in love is great.

02:28:19.200 | It's like changes the color of the light in a room.

02:28:22.120 | Like, it creates a spaciousness that's different.

02:28:25.600 | It helps you think.

02:28:26.680 | It makes you strong.

02:28:29.560 | - Bukowski had this line about love being a fog

02:28:32.520 | that dissipates with the first light of reality

02:28:36.260 | in the morning.

02:28:37.100 | - It's death-depressing, I think it's the other way around.

02:28:39.560 | - It lasts.

02:28:40.400 | Well, like you said, it's a function.

02:28:42.080 | It's a thing that generates--

02:28:42.920 | - It can be the light that actually enlivens your world

02:28:45.640 | and creates the interest and the power and the strength

02:28:49.320 | to go do something.

02:28:50.520 | Well, it's like, that sounds like,

02:28:54.360 | you know, there's like physical love, emotional love,

02:28:56.200 | intellectual love, spiritual love, right?

02:28:58.200 | - Isn't it all the same thing, kinda?

02:28:59.800 | - Nope.

02:29:01.080 | You should differentiate that, maybe that's your problem.

02:29:04.000 | In your book, you should refine that a little bit.

02:29:06.040 | - It's the different chapters?

02:29:07.240 | - Yeah, there's different chapters.

02:29:08.480 | - What's the, what's, these are,

02:29:09.920 | aren't these just different layers of the same thing,

02:29:12.200 | or the stack of physical--

02:29:14.320 | - People, some people are addicted to physical love

02:29:17.360 | and they have no idea about emotional or intellectual love.

02:29:20.360 | I don't know if they're the same things.

02:29:22.920 | I think they're different.

02:29:23.880 | - That's true, they could be different.

02:29:26.200 | I guess the ultimate goal is for it to be the same.

02:29:28.200 | - Well, if you want something to be bigger and interesting,

02:29:30.200 | you should find all its components and differentiate them,

02:29:32.560 | not clump it together.

02:29:34.520 | Like, people do this all the time, they, yeah,

02:29:36.840 | the modularity.

02:29:38.120 | Get your abstraction layers right,

02:29:39.440 | and then you can, you have room to breathe.

02:29:41.560 | - Well, maybe you can write the forward to my book

02:29:43.480 | about love.

02:29:44.320 | - Yeah, or the afterwards.

02:29:45.960 | - And the afterwards.

02:29:46.800 | - You really tried.

02:29:47.720 | I feel like Lex has made a lot of progress in this book.

02:29:52.120 | Well, you have things in your life that you love.

02:29:55.880 | - Yeah, yeah.

02:29:57.680 | And they are, you're right, they're modular.

02:30:00.040 | It's quite--

02:30:00.880 | - And you can have multiple things with the same person

02:30:04.560 | or the same thing, but, yeah.

02:30:08.520 | - Depending on the moment of the day.

02:30:09.680 | - Yeah, there's, like, what Pekoski described

02:30:13.160 | is that moment when you go from being in love

02:30:15.400 | to having a different kind of love.

02:30:17.320 | - Yeah, yeah, just a transition.

02:30:19.480 | - But when it happens, if you'd read the owner's manual

02:30:21.720 | and you believed it, you would've said, oh, this happened.

02:30:25.160 | It doesn't mean it's not love,

02:30:26.440 | it's a different kind of love.

02:30:27.960 | But maybe there's something better about that.

02:30:32.320 | As you grow old, if all you do is regret

02:30:35.560 | how you used to be, it's sad, right?

02:30:39.200 | You should've learned a lot of things,

02:30:40.720 | 'cause, like, who you can be in your future self

02:30:44.000 | is actually more interesting and possibly delightful

02:30:46.720 | than being a mad kid in love with the next person.

02:30:53.000 | That's super fun when it happens,

02:30:54.440 | but that's 5% of the possibility.

02:30:59.440 | (Lex laughs)

02:31:01.080 | - Yeah, that's right, that there's a lot more fun

02:31:03.200 | to be had in the long-lasting stuff.

02:31:05.280 | - Yeah, or meaning, if that's your thing.

02:31:07.560 | - Meaning, which is a kind of fun.

02:31:09.280 | It's a deeper kind of fun.

02:31:10.600 | - And it's surprising.

02:31:11.760 | The thing I like is surprises.

02:31:14.920 | You just never know what's gonna happen.

02:31:19.420 | But you have to look carefully and you have to work at it

02:31:21.360 | and you have to think about it.

02:31:23.000 | - Yeah, you have to see the surprises when they happen.

02:31:26.520 | You have to be looking for it.

02:31:28.320 | From the branching perspective, you mentioned regrets.

02:31:31.260 | Do you have regrets about your own trajectory?

02:31:36.200 | - Oh yeah, of course.

02:31:37.260 | Yeah, some of it's painful,

02:31:39.440 | but you wanna hear the painful stuff?

02:31:41.360 | (Lex laughs)

02:31:43.160 | I would say, in terms of working with people,

02:31:46.100 | when people did stuff I didn't like,

02:31:48.760 | especially if it was a bit nefarious,

02:31:50.760 | I took it personally and I also

02:31:52.560 | felt it was personal about them.

02:31:56.000 | But a lot of times, like humans,

02:31:58.040 | most humans are a mess, right?

02:31:59.860 | And then they act out and they do stuff.

02:32:02.120 | And this psychologist I heard a long time ago said,

02:32:05.920 | "You tend to think somebody does something to you,

02:32:09.160 | "but really what they're doing is they're doing

02:32:10.780 | "what they're doing while they're in front of you.

02:32:13.280 | "It's not that much about you."

02:32:15.400 | - Yeah. - Right?

02:32:16.240 | And as I got more interested in,

02:32:20.680 | when I work with people, I think about them,

02:32:22.880 | and probably analyze them,

02:32:25.080 | and understand them a little bit.

02:32:26.640 | And then when they do stuff, I'm way less surprised.

02:32:29.120 | And if it's bad, I'm way less hurt.

02:32:32.320 | And I react way less.

02:32:34.400 | I sort of expect everybody's got their shit.

02:32:37.080 | - Yeah, and it's not about you as much.

02:32:38.920 | - It's not about me that much.

02:32:41.000 | It's like you do something and you think you're embarrassed,

02:32:43.920 | but nobody cares.

02:32:45.680 | And somebody's really mad at you,

02:32:46.920 | the odds of it being about you,

02:32:49.680 | no, they're getting mad the way they're doing that

02:32:51.360 | because of some pattern they learned.

02:32:53.160 | And maybe you can help them if you care enough about it.

02:32:57.240 | Or you could see it coming and step out of the way.

02:32:59.840 | Like I wish I was way better at that.

02:33:02.840 | I'm a bit of a hothead.

02:33:04.080 | - You regret that?

02:33:06.000 | You said with Steve that was a feature, not a bug.

02:33:08.920 | - Yeah, well, he was using it as the counterforce

02:33:11.640 | to orderliness that would crush his work.

02:33:13.480 | - Well, you were doing the same.

02:33:15.120 | - Eh, maybe.

02:33:15.960 | I don't think my vision was big enough.

02:33:18.960 | It was more like I just got pissed off and did stuff.

02:33:22.520 | - I'm sure that's what Steve,

02:33:25.360 | yeah, you're telling--

02:33:27.280 | - I don't know if it had the,

02:33:29.080 | it didn't have the amazing effect

02:33:30.920 | of creating the trillion dollar company.

02:33:32.440 | It was more like I just got pissed off and left

02:33:35.320 | and/or made enemies that I shouldn't have.

02:33:39.200 | Yeah, it's hard.

02:33:40.520 | Like I didn't really understand politics

02:33:42.080 | until I worked at Apple,

02:33:43.720 | where Steve was a master player of politics

02:33:46.120 | and his staff had to be or they wouldn't survive him.

02:33:48.840 | And it was definitely part of the culture.

02:33:51.400 | And then I've been in companies where they say it's political

02:33:53.320 | but it's all fun and games compared to Apple.

02:33:56.960 | And it's not that the people at Apple are bad people,

02:34:00.320 | it's just they operate politically at a higher level.

02:34:03.600 | It's not like, oh, somebody said something bad

02:34:06.960 | about somebody else, which is most politics.

02:34:10.080 | They had strategies about accomplishing their goals,

02:34:15.720 | sometimes over the dead bodies of their enemies.

02:34:19.280 | With sophistication-- - Game of Thrones.

02:34:22.720 | - Yeah, more Game of Thrones,

02:34:24.520 | sophistication and a big time factor rather than a--

02:34:27.760 | - That requires a lot of control over your emotions,

02:34:31.320 | I think, to have a bigger strategy in the way you behave.

02:34:35.640 | - Yeah, and it's effective in the sense

02:34:38.840 | that coordinating thousands of people

02:34:40.800 | to do really hard things,

02:34:42.680 | where many of the people in there

02:34:44.720 | don't understand themselves much less

02:34:46.400 | how they're participating,

02:34:48.000 | creates all kinds of drama and problems

02:34:52.640 | that our solution is political in nature.

02:34:55.860 | Like how do you convince people, how do you leverage them,

02:34:57.920 | how do you motivate them, how do you get rid of them?

02:35:01.280 | There's so many layers of that that are interesting.

02:35:04.480 | And even though some of it, let's say, may be tough,

02:35:08.520 | it's not evil unless you use that skill to evil purposes.

02:35:13.520 | Which some people obviously do.

02:35:15.120 | But it's a skill set that operates.

02:35:17.480 | And I wish I'd, I was interested in it,

02:35:20.360 | but it was sort of like, I'm an engineer, I do my thing.

02:35:23.800 | And there's times when I could have had a way bigger impact

02:35:28.520 | if I knew how to, if I paid more attention

02:35:32.600 | and knew more about that.

02:35:33.840 | - About the human layer of the stack.

02:35:36.640 | - Yeah, that human political power expression layer

02:35:40.160 | of the stack.

02:35:41.000 | Which is complicated.

02:35:42.480 | And there's lots to know about it.

02:35:43.680 | I mean, people are good at it, they're just amazing.

02:35:46.160 | And when they're good at it, and let's say,

02:35:50.280 | relatively kind and oriented in a good direction,

02:35:54.200 | you can really feel, you can get lots of stuff done

02:35:57.520 | and coordinate things that you never thought possible.

02:36:00.160 | But all people like that also have some pretty hard edges

02:36:04.480 | 'cause it's a heavy lift.

02:36:07.360 | And I wish I'd spent more time on that.

02:36:09.400 | And I wish I'd spent more time with that when I was younger.

02:36:12.400 | But maybe I wasn't ready.

02:36:14.160 | You know, I was a wide-eyed kid for 30 years.

02:36:16.520 | - Still a bit of a kid.

02:36:18.760 | - I know.

02:36:20.040 | - What do you hope your legacy is when there's a book,

02:36:25.040 | like "A Hitchhiker's Guide to the Galaxy,"

02:36:28.080 | and this is like a one-sentence entry by Jim Crow.

02:36:31.200 | From like, that guy lived at some point.

02:36:34.280 | There's not many, you know,

02:36:35.680 | not many people will be remembered.

02:36:37.800 | You're one of the sparkling little human creatures

02:36:42.400 | that had a big impact on the world.

02:36:44.800 | How do you hope you'll be remembered?

02:36:46.400 | - My daughter was trying to get,

02:36:48.560 | she added to my Wikipedia page to say

02:36:50.360 | that I was a legend and a guru.

02:36:52.400 | But they took it out, so she put it back in, she's 15.

02:36:56.580 | I think that was probably the best part of my legacy.

02:37:01.200 | She got her sister, and they were all excited.

02:37:04.600 | They were trying to put it in the references

02:37:06.640 | 'cause there's articles in that on the title.

02:37:08.240 | - Calling you that?

02:37:09.440 | So in the eyes of your kids, you're a legend.

02:37:13.120 | - Well, they're pretty skeptical

02:37:14.360 | 'cause they hope you're better than that.

02:37:16.000 | They're like, "Dad!"

02:37:17.480 | So yeah, that kind of stuff is super fun.

02:37:21.620 | In terms of the big legend stuff, I don't care.

02:37:24.400 | - You don't care?

02:37:25.240 | - Legacy, I don't really care.

02:37:26.720 | - You're just an engineer.

02:37:28.600 | - Yeah, I've been thinking about building a big pyramid.

02:37:31.240 | So I had a debate with a friend

02:37:33.600 | about whether pyramids or craters are cooler.

02:37:36.880 | And he realized that there's craters everywhere,

02:37:39.280 | but they built a couple pyramids 5,000 years ago.

02:37:42.080 | - And they remember you for a while.

02:37:42.920 | - We're still talking about it.

02:37:44.960 | I think that would be cool.

02:37:46.320 | - Those aren't easy to build.

02:37:48.720 | - Oh, I know.

02:37:49.560 | And they don't actually know how they built them,

02:37:52.000 | which is great.

02:37:52.920 | - It's either AGI or aliens could be involved.

02:37:58.520 | So I think you're gonna have to figure out

02:38:01.720 | quite a few more things than just--

02:38:02.920 | - I know.

02:38:03.760 | - The basics of civil engineering.

02:38:05.440 | So I guess you hope your legacy is pyramids.

02:38:09.160 | - That would be cool.

02:38:12.480 | And my Wikipedia page,

02:38:13.960 | getting updated by my daughter periodically.

02:38:16.320 | Like those two things would pretty much make it.

02:38:18.720 | - Jim, it's a huge honor talking to you again.

02:38:20.680 | I hope we talk many more times in the future.

02:38:22.800 | I can't wait to see what you do with Tennis Torrent.

02:38:26.240 | I can't wait to use it.

02:38:27.860 | I can't wait for you to revolutionize

02:38:30.120 | yet another space in computing.

02:38:33.480 | It's a huge honor to talk to you.

02:38:34.800 | Thanks for talking today.

02:38:35.680 | - This was fun.

02:38:36.520 | - Thanks for listening to this conversation

02:38:38.960 | with Jim Keller.

02:38:39.920 | And thank you to our sponsors,

02:38:41.880 | Athletic Greens, All-in-One Nutrition Drink,

02:38:44.820 | Brooklyn and Sheets, ExpressVPN,

02:38:47.680 | and Belcampo Grass-Fed Meat.

02:38:50.200 | Click the sponsor links to get a discount

02:38:52.480 | and to support this podcast.

02:38:54.480 | And now let me leave you with some words

02:38:56.640 | from Alan Turing.

02:38:58.360 | Those who can imagine anything can create the impossible.

02:39:03.280 | Thank you for listening and hope to see you next time.

02:39:06.120 | (upbeat music)

02:39:08.700 | (upbeat music)

02:39:11.280 | [BLANK_AUDIO]

Jim Keller: The Future of Computing, AI, Life, and Consciousness | Lex Fridman Podcast #162

Chapters