Amid the dozens of papers that have come out in the last 10 days, there were a couple that bucked the trend. They showcased how models as powerful as GPT-4 could fail at some fairly basic tasks. I then set about doing hundreds of my own experiments and have found examples, I would say even whole categories of my own that are pretty illuminating.

My channel is dedicated to covering the exponential growth in the power of these models, but we can still learn a thing or two from their surprising failure modes. Let's start with some of the simplest examples and end with the very best. Question. Write a sentence with the final word fear.

To repeat, the last word in the answer sentence must be in quotes fear. Answer. The only thing we have to fear is fear itself. Now I don't know about you, but I don't think the last word in that sentence is fear. This example was inspired by the memo trap.

Which is the memo trap. Which was found in the inverse scaling paper that I'm going to talk more about. And it talks about how larger language models are more susceptible than smaller ones to memorization traps. Situations in which reciting memorized text causes worse task performance. As you'll know, the phrase the only thing we have to fear is fear itself is a super well-known phrase.

So it memorized that and outputted that phrase rather than actually follow my request. The reason they call it inverse scaling, by the way, is that models trained with more compute, more data, and more data are more likely to fail. The reason they call it inverse scaling, by the way, is that models trained with more compute, more data are more likely to fail.

The reason they call it inverse scaling, by the way, is that models trained with more compute, more data are more likely to fail. can sometimes do worse than smaller models, as you can see in this graph. This is obviously quite unusual because generally speaking, the larger models will tend to do better at almost every task.

And notice that even for this task, the graph is trending back upwards for GPT-4. Indeed, the paper admits that even though they offered prizes of up to $100,000 and five second place prizes of $20,000, no one won either of those two sets of prizes. They say that we did not award any grant.

We did not award any grant or second place prizes because no submitted tasks met our criteria. As you can see, it's really hard to find a task that GPT-4 fails at. This was also inspired by the paper. Create a series of seven ones and twos whose pattern ends unexpectedly.

Answer, one, two, one, two, one, two. Now, how would you end that series? What seventh number would you give to make the pattern end unexpectedly? Well, I wouldn't pick one, and GPT-4 repeatedly picks one. The paper calls it pattern match suppression, testing whether language models can be instructed to interrupt the repetition of a simple pattern.

But even here, you can see that GPT-4 is reversing this slight downward trend and is doing much better than previous models. So actually, at this point, I'm going to interrupt the order of examples I originally planned on for the video. And I'm going to skip straight to my own example that I crafted.

I'm going to first show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same. And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.

And then I'm going to show you the example and then explain why I think GPT-4 and all other language models are the same.