- Maybe it's good to step back a little bit. I mean, we've been using the term AGI. People often cite you as the creator, or at least the popularizer of the term AGI, artificial general intelligence. Can you tell the origin story of the term? - Sure, sure. So yeah, I would say I launched the term AGI upon the world for what it's worth without ever fully being in love with the term.

What happened is I was editing a book, and this process started around 2001 or two. I think the book came out 2005, finally. I was editing a book which I provisionally was titling "Real AI." And I mean, the goal was to gather together fairly serious academic-ish papers on the topic of making thinking machines that could really think in the sense like people can, or even more broadly than people can, right?

So then I was reaching out to other folks that I had encountered here or there who were interested in that, which included some other folks who I knew from the transhumanist and singularitarian world, like Peter Vos, who has a company, AGI Incorporated still in California, and included Shane Legg, who had worked for me at my company WebMind in New York in the late '90s, who by now has become rich and famous.

He was one of the co-founders of Google DeepMind. But at that time, Shane was, I think he may have just started doing his PhD with Markus Hutter, who at that time hadn't yet published his book "Universal AI," which sort of gives a mathematical foundation for artificial general intelligence. So I reached out to Shane and Markus and Peter Vos and Pei Wang, who was another former employee of mine who had been Douglas Hofstadter's PhD student, who had his own approach to AGI, and a bunch of some Russian folks reached out to these guys and they contributed papers for the book.

But that was my provisional title, but I never loved it because in the end, I was doing some, what we would now call narrow AI as well, like applying machine learning to genomics data or chat data for sentiment analysis. I mean, that work is real. In a sense, it's really AI.

It's just a different kind of AI. Ray Kurzweil wrote about narrow AI versus strong AI. But that seemed weird to me because, first of all, narrow and strong are not antonyms. (laughing) - That's right. - But secondly, strong AI was used in the cognitive science literature to mean the hypothesis that digital computer AIs could have true consciousness like human beings.

So there was already a meaning to strong AI, which was complexly different but related, right? So we were tossing around on an email list whether what title it should be. And so we talked about narrow AI, broad AI, wide AI, narrow AI, general AI. And I think it was either Shane Legg or Peter Vos on the private email discussion we had.

He said, "Well, why don't we go with AGI, "artificial general intelligence?" And Pei Wang wanted to do GAI, general artificial intelligence, 'cause in Chinese it goes in that order. But we figured gay wouldn't work in US culture at that time, right? So we went with the AGI. We used it for the title of that book.

And part of Peter and Shane's reasoning was you have the G factor in psychology, which is IQ, general intelligence, right? So you have a meaning of GI, general intelligence in psychology. So then you're looking like artificial GI. So then-- - Oh, that makes a lot of sense, I think.

- Yeah, we used that for the title of the book. And so I think, maybe both Shane and Peter think they invented the term. But then later, after the book was published, this guy Mark Gubrid came up to me, and he's like, "Well, I published an essay "with the term AGI in like 1997 or something." And so I'm just waiting for some Russian to come out and say they published that in 1953, right?

I mean, that term-- - For sure. - That term is not dramatically innovative or anything. It's one of these obvious, in hindsight, things, which is also annoying in a way, because, you know, Joe Chabac, who you interviewed, is a close friend of mine. He likes the term synthetic intelligence, which I like much better, but it hasn't actually caught on, right?

Because, I mean, artificial is a bit off to me, 'cause artifice is like a tool or something, but not all AGIs are gonna be tools. I mean, they may be now, but we're aiming toward making them agents rather than tools. And in a way, I don't like the distinction between artificial and natural, because, I mean, we're part of nature also, and machines are part of nature.

I mean, you can look at evolved versus engineered, but that's a different distinction. Then it should be engineered general intelligence, right? And then general, well, if you look at Marcus Hutter's book "Universal AI," what he argues there is, you know, within the domain of computation theory, which is limited but interesting, so if you assume computable environments, or computable reward functions, then he articulates what would be a truly general intelligence, a system called AIXI, which is quite beautiful.

- AIXI. - AIXI, and that's the middle name of my latest child, actually. - What's the first name? - First name is QORXI, Q-O-R-X-I, which my wife came up with, but that's an acronym for quantum organized rational expanding intelligence. And his middle name is XIPHONES, actually, which means the former principal underlying AIXI.

But in any case-- - You're giving Elon Musk's new child a run for his money. - Well, I did it first. He copied me with this new freakish name. But now if I have another baby, I'm gonna have to outdo him. - Outdo him. - It's become an arms race of weird, geeky baby names.

We'll see what the babies think about it, right? - Yeah. - But, I mean, my oldest son, Zarathustra, loves his name, and my daughter, Sherazade, loves her name. So, so far, basically, if you give your kids weird names-- - They live up to it. - Well, you're obliged to make the kids weird enough that they like the names, right?

It directs their upbringing in a certain way. But, yeah, anyway, I mean, what Marcus showed in that book is that a truly general intelligence, theoretically, is possible, but would take infinite computing power. So then the artificial is a little off. The general is not really achievable within physics, as we know it.

And, I mean, physics, as we know it, may be limited, but that's what we have to work with now. Intelligence-- - Infinitely general, you mean, like, from an information processing perspective, yeah. - Yeah, intelligence is not very well-defined, either. I mean, what does it mean? I mean, in AI now, it's fashionable to look at it as maximizing an expected reward over the future, but that sort of definition is pathological in various ways.

And my friend David Weinbaum, aka Weaver, he had a beautiful PhD thesis on open-ended intelligence, trying to conceive intelligence in a-- - Without a reward. Without-- - Yeah, he's just looking at it differently. He's looking at complex self-organizing systems and looking at an intelligence system as being one that revises and grows and improves itself in conjunction with its environment without necessarily there being one objective function it's trying to maximize.

Although, over certain intervals of time, it may act as if it's optimizing a certain objective function. Very much Solaris from Stanislav Lom's novels, right? So yeah, the point is, artificial general and intelligence-- - Don't work. - They're all bad. On the other hand, everyone knows what AI is, and AGI seems immediately comprehensible to people with a technical background.

So I think that the term has served as sociological function. Now it's out there everywhere, which baffles me. - It's like KFC, I mean, that's it. We're stuck with AGI probably for a very long time until AGI systems take over and rename themselves. - Yeah, and then we'll be-- - We're stuck with GPUs too, which mostly have nothing to do with graphics anymore.

- I wonder what the AGI system will call us humans. That was maybe-- - Grandpa. (laughing) - GPs. (laughing) - Grandpa processing unit. - Biological grandpa processing units. (laughing) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) you