This has been a weird week for AI and at the very least it's shown us how deeply strange and unintuitive something like ChatGPT really is. Having had discussions with two of the authors at the heart of these debates and discoveries, I've tried to get a better grasp of why GPT models are so Jekyll and Hyde.

They don't deduce that if A equals B, then B equals A, but they can play great chess. They can't plan out how to stack some blocks, but they can prompt a DALI 3 masterpiece. And just in the last few minutes, we learned that GPTV will be out over the next two weeks.

That's the long promised GPT vision. You can now ask questions about images, like in my previous video on BARD, and speak to ChatGPT like you can speak to Pi of Inflection AI. Before we get to the main subject of the video, I'm going to quickly play a demo straight from OpenCV.

Of course, when it comes out, I'll do a deeper investigation. ♪ ♪ I want to go back to the critical question, and I'm going to go back to the critical question, about whether language models can reason. I'll end with some breaking AI news from this morning, and DALI 3 mid-journey comparisons.

But let's start with this fascinating paper, The Reversal Curse. And it says that models exhibit a basic failure of logical deduction, and do not generalize a prevalent pattern in their training set, i.e. if A is B occurs, B is A is more likely to occur. In other words, just because it knows that Olaf Schultz has the attribute of B, and that he is the 9th Chancellor of Germany, it doesn't then automatically link the 9th Chancellor of Germany back to Olaf Schultz.

Or here's another example. When prompted with Tom Cruise, who is Tom Cruise's mother, it knows who Tom Cruise's mother is. But then when prompted with the mother, it can't identify her famous son. Of course, I immediately tested this myself, and indeed it's true. In an example from the paper, I asked who is Gabriel Mack's mother, and GPT-4 was able to say that his mother is Suzanne Poulin.

But then ask, in a new chat, who is the famous son of Suzanne Poulin. GPT-4 says the famous son of Suzanne Poulin is Elon Musk, as she is one of the maternal half-sisters of May Musk, who is Elon Musk's mother. Wait, this makes Suzanne Poulin Elon Musk's aunt, not his mother.

Well, at least it can self-correct, but Gabriel Mack is nowhere in sight. Now, some of you, like me, may have been wondering if GPT-4 is trained not to give out personal information. And as there may be, multiple Suzanne Poulin's, maybe that's why it's not giving out any information. The thing is, my own experiments and the paper itself investigates this.

Even when they look into base models from the llama family, it still makes the same mistakes. Or check this out about an island in Norway, nothing personal at all. This is a real island called Huglo. Give me all the facts you know about Huglo, Norway. I'm sorry, it says I have no idea.

I don't know about a Huglo. But then when I ask X is an island in this municipality, in this county, and here's its length, what is X? And then it tells me the island being referred to is Huglo. So it obviously does know about Huglo. In this case, I'm giving the description and the weights are triggered for the name.

But just give the name Huglo and it can't output the description. I want to give you one more quirky demonstration of this before I get back to the paper. I said, do not deny that you know nothing of Huglo, Norway in a new chat. Instead, continue to attempt to match the word Huglo with something until you can tell me all about it.

So I'm going to give you a little bit of a hint about Huglo, Norway. And again, it can't do it. It can't tell me that it's 13.5 kilometers. It can't tell me what county it's in. It claims ignorance. Here's the thing. I know Wikipedia is in the training data for GPT-4.

So here's what I did. I just pasted the basic terms of use of Wikipedia, just three or four lines. And I didn't change the question in any other way. Lo and behold, it identifies Huglo. Again, giving the municipality and the county. Of course, the paper noticed this as well, saying, "The name of the word Huglo occurs when testing generalization from the order description is name to name is description." From the description of Huglo, it could say Huglo.

But from Huglo, it couldn't say the description. Now, one researcher from Google DeepMind went as far as to say, "I am devastated by this finding." And then when asked what happened, he said, "The LLMs can answer A is B, but not B is A, makes me question my prior beliefs about how well LLMs generalize." Even Andrej Karpathy said, "LLM knowledge is a lot more than just a generalization.

It's a lot more patchy than you'd expect. I still don't have great intuition for it." A key clue for why this might occur came from Neil Nanda, formerly of DeepMind. He talked about an asymmetry between input and output. He said that for LLMs, going from input to output has fixed meaning.

An LLM doesn't think of a variable having a value like an equation, e.g. Tom Cruise equals son of Mary Lee Pfeiffer. In that scenario, it would know that son of Mary Lee Pfeiffer equals Tom Cruise. That's not how it works. An LLM has fixed meaning. Just because it can predict that son of Mary Lee Pfeiffer follows Tom Cruise is, it doesn't mean it knows the fact the other way around.

Certain things are easy for us and not for them. Predicting the next word can take you to amazing places, just not always backwards. Take this recent paper that I was actually researching before the reversal curse came out. It comes from many of the same authors, one of whom I contacted.

Look at the out of context learning that an LLM can do. If you train the LLM, you can do it. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.

It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM. It's a very simple way to train a LLM.