Lesson 2: Deep Learning 2019 - Data cleaning and production; SGD from scratch

00:00:00.000 | Welcome to Lesson 2, where we're going to be taking a deeper dive into computer vision applications and taking some of the amazing stuff that you've all been doing during the week and going even further.

00:00:15.000 | So let's take a look. Before we do, a reminder that we have these two really important topics on the forums.

00:00:26.000 | They're pinned at the top of the forum category. One is Fact Resources and Official Course Updates.

00:00:32.000 | This is where, if there's something useful for you to know during the course, we will post there. Nobody else can reply to that thread.

00:00:40.000 | So if you set that thread to Watching and to Notifications, you're not going to be bugged by anybody else except stuff that we think you need to know for the course.

00:00:48.000 | And it's got all the official information about how to get set up on each platform. Please note, a lot of people post all kinds of other tidbits about

00:00:58.000 | how they've set up things on previous solutions or previous courses or other places. I don't recommend you use those because these are the ones that we're testing every day

00:01:08.000 | and that the folks involved in these platforms are testing every day and they definitely work. So I would strongly suggest you follow those tips.

00:01:18.000 | And if you do have a question about using one of these platforms, please use these discussions, not some other topic that you create.

00:01:25.000 | Because this way, people that are involved in these platforms will be able to see it and things won't get messy.

00:01:31.000 | And then secondly, for every lesson there will be an official updates thread for that lesson. So Lesson 1, official updates, and the same thing. Only FastAI people will be posting to that.

00:01:45.000 | So you can watch it safely and we'll have all the things like the videos, the notebooks, and so forth.

00:01:53.000 | And they're all wiki threads, so you can help us to make them better as well.

00:01:58.000 | So I mentioned the idea of watching a thread. So this is a really good idea, is that you can go to a thread, like particularly those official update ones, and click at the bottom, "Watching".

00:02:08.000 | And if you do that, that's going to enable notifications for any updates to that thread. Particularly if you go into, click on your little user name in the top right, "Preferences", and turn this on.

00:02:18.000 | That will give you an email as well. So any of you that have missed some of the updates so far, go back and have a look through because we're really trying to make sure that we keep you updated with anything that we think is important.

00:02:32.000 | One thing which can be more than a little overwhelming is even now, after just one week, the most popular thread has 1.1000 replies.

00:02:40.000 | So that's an intimidatingly large number. I've actually read every single one of them, and I know Rachel has, and I know Sylvia has, and I think Francisco has.

00:02:51.000 | But you shouldn't need to. What you should do is click "Summarize this topic" and it'll appear like this, which is all of the most liked ones will appear, and then there'll be "View 31 hidden replies" or whatever in between.

00:03:05.000 | So that's how you navigate these giant topics. That's also why it's important you click the "Like" button, because that's the thing that's going to cause people to see it in this recommended view.

00:03:20.000 | So when you come back to work, hopefully you've realized by now that on the official course website, course-v3.fast.ai, you will click "Returning to Work". You will click the name of the platform you're using, and you will then follow the two steps.

00:03:35.000 | Step one will be how to make sure that you've got the latest notebooks, and step two will be how to make sure you've got the latest Python library software.

00:03:44.000 | They all look pretty much like this, but they're slightly different from platform to platform, so please don't use some different set of commands you read somewhere else. Only use the commands that you read about here, and that will make everything very smooth.

00:03:58.000 | If things aren't working for you, if you get into some kind of messy situation, which we all do, and just delete your instance and start again, unless you've got mission-critical stuff there, it's the easiest way just to get out of a sticky situation.

00:04:13.000 | And if you follow the instructions here, you really should find it works fine.

00:04:20.000 | So, this is what I really wanted to talk about, most of all, is what people have been doing this week. If you've noticed, and a lot of you have, there's been 167 people sharing their work.

00:04:32.000 | And this is really cool, because it's pretty intimidating to put yourself out there and say like, "I'm new to all this, but here's what I've done."

00:04:40.000 | And so examples of things I thought was really interesting was figuring out who's talking. Is it Ben Affleck or Joe Rogan?

00:04:49.000 | I thought this is really interesting. This is actually very practical. I wanted to clean up my WhatsApp downloaded images to get rid of memes, so I actually built a little neural network.

00:04:59.000 | I mean, how cool is that to say like, "Oh yeah, I've got something that cleans up my WhatsApp. It's a deep learning application I wrote last week."

00:05:06.000 | Why not? Like, it's so easy now. You can do stuff like this.

00:05:10.000 | And then there's been some really interesting projects. One was looking at the sound data that was used in this paper.

00:05:23.000 | And in this paper, they were trying to figure out what kind of sound things were, and they got, as you would expect since they published a paper, they got a state of the art of nearly 80% accuracy.

00:05:34.000 | Ethan Sooten then tried using the Lesson 1 techniques and got 80.5% accuracy. So I think this is pretty awesome.

00:05:42.000 | Best as we know, it's a new state of the art for this problem.

00:05:47.000 | Maybe somebody since then has published something. We haven't found it yet. They take all of these with a slight grain of salt.

00:05:52.000 | But I've mentioned them on Twitter, and lots of people on Twitter follow me, so if everybody knew that there was a much better approach, I'm sure somebody would have said so.

00:06:00.000 | This one is pretty cool. Survash has a new state of the art accuracy for Devangari text recognition.

00:06:10.000 | I think he's got it even higher than this now. And this is actually confirmed by the person on Twitter who created the dataset.

00:06:17.000 | I don't think he had any idea. He just posted, "Hey, here's a nice thing I did."

00:06:21.000 | And this guy on Twitter was like, "Oh, I made that dataset. Congratulations. You've got a new record." So that was pretty cool.

00:06:28.000 | I really like this post from Alina Harley. She describes in quite a bit of detail about the issue of them metastasizing cancers and the use of point mutations and why that's a challenging important problem.

00:06:44.000 | And she's got some nice pictures describing what she wants to do with this and how she can go about turning this into pictures.

00:06:52.000 | Hey, see, this is the cool trick, right? It's the same with this sounds one, turning sounds into pictures and then using the Lesson 1 approach.

00:07:00.000 | And here it's turning point mutations into pictures and then using the Lesson 1 approach.

00:07:06.000 | And what did she find? It seems that she's got a new state of the art result by more than 30%, beating the previous best.

00:07:15.000 | Somebody on Twitter who's a VP at a genomics analysis company looked at this as well and thought it looked to be a state of the art in this particular point mutation one as well.

00:07:28.000 | So that's pretty exciting. So you can see when we talked about last week this idea that this simple process is something which can take you a long way.

00:07:38.000 | It really can. I will mention that something like this one in particular is using a lot of domain expertise, like it's figuring out what picture to create.

00:07:50.000 | I wouldn't know how to do that because I don't even really know what a point mutation is, let alone how to create something that visually is meaningful that a CNN could recognize.

00:08:00.000 | But the actual big learning side is actually pretty straightforward.

00:08:08.000 | Another very cool result from Simon Willison and Natalie Down, they created a Cougar or not web application over the weekend and won the Science HAC Day Award in San Francisco.

00:08:24.000 | And so I think that's pretty fantastic. So lots of examples of people doing really interesting work.

00:08:32.000 | Hopefully this will be inspiring to you to think, wow, this is cool that I can do this with what I've learned.

00:08:39.000 | It can also be intimidating to think like, wow, these people are doing amazing things.

00:08:43.000 | But it's important to realize out of the thousands of people doing this course, you know, I'm just picking out a few of the really amazing ones.

00:08:53.000 | And in fact, Simon is one of these very annoying people like Christine Payne, who we talked about last week, who seems to be good at everything he does.

00:08:59.000 | He created Django, one of the world's most popular web frameworks.

00:09:02.000 | He founded a very successful startup and blah, blah, blah, blah, blah.

00:09:05.000 | So, you know, one of these really annoying people who tends to keep being good at things, now it turns out he's good at deep learning as well.

00:09:11.000 | So, you know, that's fine. Simon can go and win a hackathon on his first week of playing with deep learning.

00:09:17.000 | Maybe it'll take you two weeks to win your first hackathon. That's okay.

00:09:23.000 | And I think like it's important to mention this because there was this really inspiring blog post this week from James Dellinger,

00:09:29.000 | who talked about how he created a bird classifier using the techniques from lesson one.

00:09:34.000 | But what I really found interesting was at the end, he said he nearly didn't start on deep learning at all because he went to the scikit-learn website,

00:09:42.000 | which is one of the most important libraries of Python, and he saw this.

00:09:46.000 | And he described in this blog post how he was just like, "That's not something I can do. That's not something I understand."

00:09:52.000 | And then this kind of realization of like, "Oh, I can do useful things without reading the Greek."

00:09:58.000 | So I thought that was a really cool message.

00:10:01.000 | And I really want to highlight, actually, Daniel Armstrong on the forum, I think really shows, is a great role model here,

00:10:09.000 | which was here saying, "I want to contribute to the library."

00:10:12.000 | And I looked at the docs, and I just found it overwhelming.

00:10:16.000 | And his next message one day later was, "I don't know what any of this is. I didn't know how much there was to it.

00:10:22.000 | It caught me off guard. My brain shut down.

00:10:27.000 | But I love the way it forces me to learn so much."

00:10:30.000 | And then one day later, I just submitted my first pull request.

00:10:34.000 | So I think that's awesome. It's okay to feel intimidated. There's a lot.

00:10:40.000 | But just pick one piece and dig into it.

00:10:44.000 | Try and push a piece of code or a documentation update, or create a classifier, or whatever.

00:10:50.000 | So here's lots of cool classifiers people have built. It's been really, really inspiring.

00:10:55.000 | Triddedad and Tobago Islander versus Masquerada classifier, a Zucchini versus Cucumber classifier.

00:11:04.000 | This one was really nice. This was taking the dog breeds, dog and cat breeds thing from last week,

00:11:11.000 | and actually doing some exploratory work to see what the main features were,

00:11:15.000 | and discovered that they could now create a hairiness and classifier.

00:11:20.000 | And so here we have the most hairy dogs and the most bald cats.

00:11:25.000 | So there are interesting things you can do with interpretation.

00:11:28.000 | Somebody else in the forum took that and did the same thing for anime

00:11:31.000 | to find that they had accidentally discovered an anime hair color classifier.

00:11:35.000 | We can now detect the new versus the old Panamanian buses correctly.

00:11:41.000 | Apparently these are the new ones. I much prefer the old ones, but maybe that's just me.

00:11:46.000 | This was really interesting. Henry Palachi discovered that

00:11:50.000 | he can recognize with 85% accuracy which of 110 countries a satellite image is of,

00:11:59.000 | which is definitely got to be beyond human performance of just about anybody.

00:12:05.000 | Like I can't imagine anybody who can do that in practice. So that was fascinating.

00:12:11.000 | Batik cloth classification with 100% accuracy.

00:12:18.000 | David Ward did this interesting one. We actually went a little bit further

00:12:22.000 | using some techniques we'll be discussing in the next couple of courses

00:12:25.000 | to build something that can recognize complete or incomplete or foundation buildings

00:12:30.000 | and actually plot them on aerial satellite view.

00:12:34.000 | So lots and lots of fascinating projects.

00:12:39.000 | So don't worry, it's only been one week.

00:12:41.000 | It doesn't mean everybody has to have had a project out yet.

00:12:44.000 | A lot of the folks who already have a project out have done a previous course,

00:12:47.000 | so they've got a bit of a head start.

00:12:50.000 | But we'll see today how you can definitely create your own classifier this week.

00:12:56.000 | So from today after we dig a bit deeper into really how to make these computer

00:13:02.000 | vision classifiers in particular work well,

00:13:05.000 | we're then going to look at the same thing for text.

00:13:09.000 | We're then going to look at the same thing for tabular data,

00:13:12.000 | so they're kind of like more like spreadsheets and databases.

00:13:15.000 | Then we're going to look at collaborative filtering,

00:13:18.000 | so kind of recommendation systems.

00:13:21.000 | That's going to take us into a topic called embeddings,

00:13:24.000 | which is basically a key underlying platform behind these applications.

00:13:30.000 | That will take us back into more computer vision and then back into more NLP.

00:13:35.000 | So the idea here is that it turns out that it's much better for learning

00:13:40.000 | if you kind of see things multiple times.

00:13:43.000 | So rather than being like, okay, that's computer vision,

00:13:45.000 | you won't see it again for the rest of the course,

00:13:47.000 | we're actually going to come back to the two key applications,

00:13:50.000 | NLP and computer vision, a few weeks apart,

00:13:53.000 | and that's going to force your brain to realize like,

00:13:55.000 | oh, I have to remember this. It's not just something I can throw away.

00:14:01.000 | So people who have more of a hard sciences kind of background in particular,

00:14:14.000 | a lot of folks find this, hey, here's some code,

00:14:19.000 | type it in, start running an approach rather than here's lots of theory approach,

00:14:24.000 | confusing and surprising and odd at first.

00:14:28.000 | And so for those of you, I just wanted to remind you this basic tip,

00:14:32.000 | which is keep going.

00:14:35.000 | You're not expected to remember everything yet.

00:14:37.000 | You're not expected to understand everything yet.

00:14:40.000 | You're not expected to know why everything works yet.

00:14:44.000 | You just want to be in a situation where you can enter the code

00:14:48.000 | and you can run it and you can get something happening

00:14:51.000 | and then you can start to experiment and you kind of get a feel

00:14:55.000 | for what's going on and then push on, right?

00:14:59.000 | Most of the people who have done the course

00:15:01.000 | and have gone on to be really successful watch the videos at least three times.

00:15:04.000 | So they kind of go through the whole lot

00:15:06.000 | and then go through it slowly the second time,

00:15:09.000 | then they go through it really slowly the third time

00:15:11.000 | and I consistently hear them say I get a lot more out of it each time I go through.

00:15:15.000 | So don't pause at lesson one and stop until you can continue.

00:15:20.000 | So this approach is based on a lot of research,

00:15:27.000 | academic research into learning theory and one guy in particular,

00:15:30.000 | David Perkins from Harvard has this really great analogy.

00:15:34.000 | He's a researcher into learning theory.

00:15:37.000 | He describes this approach of the whole game,

00:15:39.000 | which is basically if you're teaching a kid to play soccer,

00:15:43.000 | you don't first of all teach them about how the friction

00:15:47.000 | between a ball and grass works and then teach them how to sew a soccer ball

00:15:51.000 | with their bare hands and then teach them the mathematics of parabolas

00:15:55.000 | when you kick something in the air.

00:15:57.000 | No, you say here's a ball, let's watch some people playing soccer.

00:16:01.000 | Okay, now we'll play soccer and then you, you know,

00:16:04.000 | gradually over the following years learn more and more

00:16:07.000 | so that you can get better and better at it.

00:16:09.000 | So this is kind of what we're trying to get you to do is to play soccer,

00:16:12.000 | which in our case is to type code and look at the inputs

00:16:16.000 | and look at the outputs.

00:16:19.000 | Okay, so let's dig into our first notebook,

00:16:26.000 | which is called Lesson 2 Download.

00:16:29.000 | And what we're going to do is we're going to see how to create

00:16:34.000 | your own classifier with your own images.

00:16:38.000 | So it's going to be a lot like last week's pet detector,

00:16:42.000 | but it'll detect whatever you like.

00:16:44.000 | So it'll be like some of those examples we just saw.

00:16:47.000 | How would you create your own Panama bus detector from scratch?

00:16:54.000 | So this is inspired, the approach is inspired by Adrian Rosebrock

00:16:59.000 | who has a terrific website called Pi Image Search

00:17:02.000 | and he has this nice explanation of how to create a dataset using Google Images.

00:17:09.000 | So that was definitely an inspiration for some of the techniques we use here.

00:17:12.000 | So thank you to Adrian.

00:17:14.000 | And you should definitely check out his site.

00:17:16.000 | It's full of lots of good resources.

00:17:19.000 | So here we are.

00:17:23.000 | So we are going to try to create a teddy bear detector.

00:17:31.000 | Thanks.

00:17:32.000 | We're going to try and make a teddy bear detector

00:17:34.000 | and we're going to try and separate teddy bears

00:17:37.000 | from black bears from grizzly bears.

00:17:40.000 | Now this is very important.

00:17:42.000 | I have a three-year-old daughter

00:17:44.000 | and she needs to know what she's dealing with.

00:17:47.000 | In our house you would be surprised at the number of monsters,

00:17:51.000 | lions and other terrifying threats that are around,

00:17:54.000 | particularly around Halloween.

00:17:56.000 | And so we always need to be on the lookout to make sure

00:17:58.000 | that the thing we're about to cuddle is in fact a genuine teddy bear.

00:18:03.000 | So let's deal with that situation as best as we can.

00:18:07.000 | So our starting point is to find some pictures of teddy bears

00:18:10.000 | so we can learn what they look like.

00:18:14.000 | So I go to images.google.com

00:18:17.000 | and I type in "teddy bear"

00:18:21.000 | and I just scroll through until I kind of find a goodly bunch of them.

00:18:28.000 | And it's like okay, that looks like plenty of teddy bears to me.

00:18:32.000 | So then I'll go back to here.

00:18:36.000 | So you can see it says search and scroll.

00:18:38.000 | Go to Google Images and search.

00:18:40.000 | And the next thing we need to do is to get a list of all of the URLs there.

00:18:44.000 | And so to do that, back in your Google Images

00:18:48.000 | you hit Ctrl+Shift+J or Command+Option+J

00:18:52.000 | and you paste this into the window that appears.

00:19:00.000 | So I've got Windows, so I go Ctrl+Shift+J

00:19:05.000 | and I paste in that code.

00:19:07.000 | So this is a JavaScript console.

00:19:09.000 | For those of you who haven't done any JavaScript before,

00:19:11.000 | I hit Enter and it downloads my file for me.

00:19:15.000 | So I would call this "teddies.txt" and press Save.

00:19:21.000 | Okay, so I now have a file of teddies or URLs of teddies.

00:19:28.000 | So then I would repeat that process for black bears and for brown bears

00:19:34.000 | since that's a classifier I would want.

00:19:35.000 | And I'd put each one in a file with an appropriate name.

00:19:38.000 | So that's step one.

00:19:40.000 | So step two is we now need to download those URLs to our server.

00:19:46.000 | Because remember when we're using Jupyter Notebook

00:19:48.000 | it's not running on our computer.

00:19:50.000 | It's running on SageMaker or Cressel or Google Cloud or whatever.

00:19:57.000 | So to do that we start running some Jupyter cells.

00:20:01.000 | So let's grab the FastAI library and let's start with black bears.

00:20:06.000 | I've already got my black bears URL.

00:20:08.000 | So I click on this cell for black bears and I run it.

00:20:11.000 | See here how I've got three different cells

00:20:14.000 | doing the same thing with different information?

00:20:17.000 | This is one way I like to work with Jupyter Notebook.

00:20:20.000 | It's something that a lot of people with a more strict scientific background

00:20:24.000 | are horrified by.

00:20:25.000 | This is not reproducible research.

00:20:27.000 | So I actually click here and I run this cell to create a folder called "black"

00:20:31.000 | and a file called "URL black" for my black bears.

00:20:34.000 | I skip the next two cells.

00:20:36.000 | And then I run this cell to create that folder.

00:20:41.000 | And then I go down to the next section

00:20:45.000 | and I run the next cell which is "download images for black bears".

00:20:53.000 | So that's just going to download my black bears to that folder.

00:20:56.000 | And then I'll go back and I'll click on "Tetties"

00:20:59.000 | and I run that cell and then scroll back down and I'll run this cell.

00:21:04.000 | And so that way I'm just going backwards and forwards

00:21:06.000 | to download each of the classes that I want.

00:21:09.000 | Very manual, but for me I'm very iterative and very experimental.

00:21:14.000 | That works well for me.

00:21:15.000 | If you're better at kind of planning ahead than I am,

00:21:18.000 | you can write a proper loop or whatever and do it that way.

00:21:22.000 | But when you see my notebooks and see things

00:21:26.000 | where there's these kind of like configuration cells

00:21:28.000 | doing the same thing in different places,

00:21:30.000 | this is a strong sign that I didn't run this in order.

00:21:34.000 | I clicked one place, went to another, ran that, went back, went back, went back.

00:21:38.000 | And for me, I'm an experimentalist.

00:21:41.000 | I really like to experiment in my notebook.

00:21:45.000 | I treat it like a lab journal.

00:21:47.000 | I try things out and I see what happens.

00:21:49.000 | And so this is how my notebooks end up looking.

00:21:52.000 | It's a really controversial topic.

00:21:54.000 | Like for a lot of people they feel this is like wrong,

00:21:58.000 | that you should only ever run things top to bottom.

00:22:00.000 | Everything you should do should be reproducible.

00:22:02.000 | For me, I don't think that's the best way of using human creativity.

00:22:06.000 | I think human creativity is best inspired by trying things out,

00:22:11.000 | seeing what happens and fiddling around.

00:22:13.000 | So you can see how you go, see what works for you.

00:22:16.000 | So that will download the images to your server.

00:22:20.000 | It's going to use multiple processes to do so.

00:22:24.000 | And one problem there is if something goes wrong,

00:22:28.000 | it's a bit hard to see what went wrong.

00:22:30.000 | So you can see in the next section there's a commented out section

00:22:33.000 | that says max workers equals zero.

00:22:36.000 | That will do it without spinning up a bunch of processes

00:22:38.000 | and will tell you the errors better.

00:22:40.000 | So if things aren't downloading, try using the second version.

00:22:44.000 | Okay, so I grabbed a small number of each.

00:22:50.000 | And then the next thing that I found I needed to do

00:22:53.000 | was to remove the images that aren't actually images at all.

00:22:57.000 | And this happens all the time.

00:22:58.000 | There's always a few images in every batch

00:23:01.000 | that are corrupted for whatever reason.

00:23:04.000 | Google image tried to tell us that this URL had an image,

00:23:08.000 | but actually it doesn't anymore.

00:23:10.000 | So we've got this thing in the library called verify images,

00:23:14.000 | which will check all of the images in a path

00:23:17.000 | and will tell you if there's a problem.

00:23:19.000 | If you say delete equals true, it will actually delete it for you.

00:23:23.000 | So that's a really nice easy way to end up with a clean dataset.

00:23:28.000 | So at this point I now have a bears folder containing a grizzly folder

00:23:34.000 | and a teddies folder and a black folder.

00:23:37.000 | In other words, I have the basic structure we need to create

00:23:40.000 | an image data bunch to start doing some deep learning.

00:23:43.000 | So let's go ahead and do that.

00:23:46.000 | Now, very often when you download a dataset from like Kaggle

00:23:54.000 | or from some academic dataset, there will often be a folder called train

00:23:59.000 | and a folder called valid and a folder called test,

00:24:02.000 | right, containing the different datasets.

00:24:05.000 | In this case, we don't have a separate validation set

00:24:08.000 | because we just grab these images from Google search, right?

00:24:12.000 | But you still need a validation set,

00:24:15.000 | otherwise you don't know how well your model is going.

00:24:18.000 | We'll talk more about this in a moment.

00:24:20.000 | So whatever you create a data bunch,

00:24:22.000 | if you don't have a separate training and validation set,

00:24:26.000 | then you can just say, okay, well the training set is in the current folder

00:24:30.000 | because by default it looks in a folder called train.

00:24:33.000 | And I want you to set aside 20% of the data, please.

00:24:37.000 | So this is going to create a validation set for you

00:24:39.000 | automatically and randomly.

00:24:42.000 | You'll see that whenever I create a validation set randomly,

00:24:46.000 | I always set my random seed to something fixed beforehand.

00:24:50.000 | This means that every time I run this code,

00:24:53.000 | I'll get the same validation set.

00:24:56.000 | So in general, I'm not a fan of making my machine learning experiments reproducible,

00:25:05.000 | i.e. ensuring I get exactly the same result every time.

00:25:08.000 | The randomness is to me really important,

00:25:10.000 | a really important part of finding out is your solution stable?

00:25:14.000 | Is it going to work like each time you run it?

00:25:16.000 | But what is important is that you always have the same validation set.

00:25:20.000 | Otherwise, when you're trying to decide,

00:25:23.000 | has this hyperparameter change improved my model,

00:25:26.000 | but you've got a different set of data you're testing it on,

00:25:29.000 | then you don't know maybe that set of data just happens to be a bit easier.

00:25:33.000 | So that's why I always set the random seed here.

00:25:38.000 | So we've now got, let's run that cell,

00:25:40.000 | so we've now got a data bunch.

00:25:42.000 | And so you can look inside at the data.classes

00:25:46.000 | and you'll see these are the folders that we created.

00:25:49.000 | So it knows that the classes,

00:25:51.000 | so by classes we mean all the possible labels,

00:25:54.000 | black bear, grizzly bear or teddy bear.

00:25:57.000 | We can run show batch and we can take a little look.

00:26:01.000 | And it tells us straight away that some of these are going to be a little bit tricky.

00:26:06.000 | So this is not a photo, for instance.

00:26:10.000 | Some of them kind of crop funny.

00:26:14.000 | Some of them might be tricky,

00:26:16.000 | like if you ended up with a black bear standing on top of a grizzly bear,

00:26:18.000 | that might be tough.

00:26:20.000 | Anyway, so you can kind of double check here data.classes, there they are.

00:26:24.000 | Remember C is the attribute which the classifier tells us how many possible labels there are.

00:26:30.000 | We'll learn about some other more specific meanings of C later.

00:26:34.000 | We can see how many things are in our training set.

00:26:37.000 | We can see how many things are in our validation set.

00:26:43.000 | So we've got 473 training set, 141 validation set.

00:26:49.000 | So at that point we can go ahead.

00:26:51.000 | You'll see all these commands are identical to the pet classifier from last week.

00:26:55.000 | We can create our CNN, our convolutional neural network, using that data.

00:27:01.000 | I tend to default to using a ResNet 34.

00:27:04.000 | And let's print out the error rate each time,

00:27:07.000 | and run fit one cycle four times and see how we go.

00:27:11.000 | And we have a 2% error rate.

00:27:14.000 | So that's pretty good.

00:27:16.000 | Sometimes it's easy for me to recognize a black bear from a grizzly bear,

00:27:20.000 | but sometimes it's a bit tricky.

00:27:22.000 | This one seems to be doing pretty well.

00:27:28.000 | Okay, so after I kind of make some progress with my model and things looking good,

00:27:34.000 | I always like to save where I'm up to,

00:27:36.000 | to save me the 54 seconds of going back and doing it again.

00:27:39.000 | And as per usual, we unfreeze the rest of our model.

00:27:43.000 | We're going to be learning more about what that means during the course.

00:27:46.000 | And then we run the learning rate finder and plot it.

00:27:51.000 | It tells you exactly what to type, and we take a look.

00:27:54.000 | Now, we're going to be learning about learning rates today, actually.

00:27:58.000 | But for now, here's what you need to know.

00:28:00.000 | On the learning rate finder,

00:28:02.000 | what you're looking for is the strongest downward slope

00:28:07.000 | that's kind of sticking around for quite a while.

00:28:10.000 | So this one here looks more like a bump,

00:28:12.000 | but this looks like an actual downward slope to me.

00:28:15.000 | So it's kind of like it's something you're going to have to practice with

00:28:18.000 | and get a feel for, like which bit works.

00:28:22.000 | So if you're not sure is it this bit or this bit,

00:28:25.000 | try both learning rates and see which one works better.

00:28:29.000 | But I've been doing this for a while,

00:28:31.000 | and I'm pretty sure this looks like where it's really learning properly.

00:28:35.000 | So I would pick something.

00:28:37.000 | Okay, here it's not so steep.

00:28:39.000 | So I would probably pick something back here for my learning rate.

00:28:44.000 | So you can see I picked 3E neg 5.

00:28:48.000 | So, you know, somewhere around here.

00:28:50.000 | That sounds pretty good.

00:28:51.000 | So that's my bottom learning rate.

00:28:53.000 | So my top learning rate, I normally pick 1E neg 4 or 3E neg 4.

00:28:58.000 | It's kind of like I don't really think about it too much.

00:29:01.000 | That's a rule of thumb.

00:29:02.000 | It always works pretty well.

00:29:04.000 | One of the things you'll realize is that most of these parameters

00:29:09.000 | don't actually matter that much in detail.

00:29:12.000 | If you just copy the numbers that I use each time,

00:29:16.000 | the vast majority of the time it will just work fine.

00:29:18.000 | And we'll see places where it doesn't today.

00:29:21.000 | Okay, so we've got a 1.4% error rate after doing another couple of epochs.

00:29:27.000 | So that's looking great.

00:29:28.000 | So we've downloaded some images from Google Image,

00:29:31.000 | search, created a classifier.

00:29:34.000 | We've got a 1.4% error rate.

00:29:36.000 | Let's save it.

00:29:38.000 | And then as per usual, we can use the classification interpretation class

00:29:43.000 | to have a look at what's going on.

00:29:45.000 | And in this case, we made one mistake.

00:29:48.000 | There was one black bear classified as grizzly bear.

00:29:53.000 | So that's a really good step.

00:29:57.000 | We've come a long way.

00:29:58.000 | But possibly you could do even better if your data set was less noisy.

00:30:04.000 | Like maybe Google Image search didn't give you exactly the right images all the time.

00:30:11.000 | So how do we fix that?

00:30:13.000 | And so we want to clean it up.

00:30:15.000 | And so combining a human expert with a computer learner is a really good idea.

00:30:21.000 | Almost, not nobody, but very, very few people publish on this.

00:30:25.000 | Very, very few people teach this.

00:30:27.000 | But to me, it's like the most useful skill.

00:30:30.000 | Particularly for you, most of the people watching this are domain experts,

00:30:34.000 | not computer science experts.

00:30:37.000 | And so this is where you can use your knowledge of point mutations in genomics

00:30:42.000 | or Panamanian buses or whatever.

00:30:45.000 | So let's see how that would work.

00:30:47.000 | What I'm going to do is, do you remember the plot top losses from last time

00:30:51.000 | where we saw the images which it was like either the most wrong about

00:30:55.000 | or the least confident about.

00:30:57.000 | We're going to look at those and decide which of those are noisy.

00:31:02.000 | Like if you think about it, it's very unlikely that if there is some mislabeled data

00:31:09.000 | that it's going to be predicted correctly and with high confidence.

00:31:13.000 | But that's really unlikely to happen.

00:31:15.000 | So we're going to focus on the ones which the model is saying either it's not confident of

00:31:22.000 | or it was confident of and it was wrong about.

00:31:24.000 | They are the things which might be mislabeled.

00:31:28.000 | So big shout out to the San Francisco Fast AI Study Group

00:31:35.000 | who created this new widget this week called the File Deleter.

00:31:41.000 | So that's Zach and Jason and Francisco built this thing where we basically can take the top losses

00:31:49.000 | from that interpretation object we just created.

00:31:53.000 | And then what we're going to do is we're going to say, okay, that returns top losses.

00:31:57.000 | There's not just plot top losses, but there's also just top losses.

00:32:01.000 | And top losses return two things, the losses of the things that were the worst

00:32:07.000 | and the indexes into the data set of the things that were the worst.

00:32:11.000 | And if you don't pass anything at all, it's going to actually return the entire data set but sorted.

00:32:17.000 | So the first things will be the highest losses.

00:32:21.000 | As we'll learn during the course, so we'll keep seeing during the course,

00:32:24.000 | every data set in Fast AI has an x and a y.

00:32:29.000 | And the x contains the things that are used to, in this case, get the images.

00:32:34.000 | So this is the image file names and the y's will be the labels.

00:32:38.000 | So if we grab the indexes and pass them into the data set x,

00:32:43.000 | this is going to give us the file names of the data set ordered by which ones had the highest loss.

00:32:51.000 | So which ones it was either confident and wrong about or not confident about.

00:32:57.000 | And so we can pass that to this new widget that they've created called the FileDeleader widget.

00:33:08.000 | So just to clarify, this top plus paths contains all of the file names in our data set.

00:33:15.000 | When I say in our data set, this particular one is in our validation data set.

00:33:19.000 | So what this is going to do is it's going to clean up mislabeled images or images that shouldn't be there.

00:33:29.000 | And we're going to remove them from a validation set so that our metrics will be more correct.

00:33:34.000 | So you then need to rerun these two steps replacing valid ds with train ds

00:33:39.000 | to clean up your training set to get the noise out of that as well.

00:33:43.000 | So it's a good practice to do both.

00:33:47.000 | We'll talk about test sets later as well.

00:33:49.000 | If you also have a test set, you would then repeat the same thing.

00:33:52.000 | So we run FileDeleader passing in that sorted list of paths.

00:33:58.000 | And so what pops up is basically the same thing as plot top losses.

00:34:04.000 | So in other words, these are the ones which is either wrong about or at least confident about.

00:34:10.000 | And so not surprisingly, this one here does not appear to be a teddy bear or a black bear or a brown bear.

00:34:18.000 | So this shouldn't be in our data set.

00:34:20.000 | So what I do is I work on the Delete button.

00:34:24.000 | And all the rest do look indeed like bears. And then so I can click Confirm, and it'll bring up another 5.

00:34:32.000 | What's that? That's not a bear, is it?

00:34:35.000 | Does anybody know what that is?

00:34:37.000 | I'm going to say that's not a bear.

00:34:39.000 | Delete. Confirm.

00:34:44.000 | Well, not bear.

00:34:46.000 | Well, that's a teddy bear. I'll leave that.

00:34:48.000 | That's not really. I'll get rid of that one. Confirm.

00:34:52.000 | OK, so what I tend to do when I do this is I'll keep going Confirm until I get to a couple of screen fulls of things that all look OK.

00:35:00.000 | And that suggests to me that I've kind of got past the worst bits of the data. OK, and that's it.

00:35:06.000 | And so now you can go back once you do it for the training set as well and retrain your model.

00:35:12.000 | So I'll just note here that what our San Francisco study group did here was that they actually built a little app inside Jupyter Notebook,

00:35:23.000 | which you might not have realized is possible. But not only is it possible, it's actually surprisingly straightforward.

00:35:30.000 | And just like everything else, you can hit double question mark to find out their secrets.

00:35:35.000 | So here is the source code. OK, and really, if you've done any GUI programming before, it'll look incredibly normal.

00:35:46.000 | You know, there's there's basically callbacks for what happens when you click on a button where you just do standard Python things.

00:35:52.000 | And to actually render it, you just use widgets and you can lay it out using standard boxes and whatever.

00:36:00.000 | So it's this idea of creating applications inside notebooks is like it's really underused,

00:36:08.000 | but it's super neat because it lets you create tools for your fellow practitioners, for your fellow experimenters.

00:36:16.000 | Right. And you could definitely envisage taking this a lot further.

00:36:20.000 | In fact, by the time you're watching this on the MOOC, you'll probably find that there's a whole lot more buttons here because we've already got a long list of to do's that we're going to add.

00:36:29.000 | To this particular thing.

00:36:34.000 | So so that's it. So I think like.

00:36:38.000 | I'd love for you to have a think about now that you know it's possible to write applications in your notebook, what are you going to write?

00:36:45.000 | And if you Google for eye pie widgets, you can learn about the little gooey framework to find out what kind of widgets you can create and what they look like and how they work and so forth.

00:37:01.000 | And you'll find it's you know, it's actually a pretty, you know, complete gooey programming environment you can play with.

00:37:09.000 | And this will all work nicely with your models and so forth.

00:37:13.000 | It's not a great way to productionize an application because it is sitting inside a notebook.

00:37:19.000 | This is really for things which are going to help other practitioners, other experimentalists and so forth.

00:37:27.000 | For productionizing things, you need to actually build a production web app, which we'll look at next.

00:37:36.000 | OK, so after you have cleaned up your noisy images, you can then retrain your model and hopefully you'll find it's a little bit more accurate.

00:37:47.000 | One thing you might be interested to discover when you do this is it actually doesn't matter most of the time very much.

00:37:54.000 | Now on the whole, these models are pretty good at dealing with moderate amounts of noisy data.

00:38:03.000 | The problem would occur is if your data was not randomly noisy, but biased noisy.

00:38:09.000 | So I guess the main thing I'm saying is if you go through this process of cleaning up your data and then you rerun your model and find it's like 0.001% better, that's normal.

00:38:19.000 | OK, that's fine, but it's still a good idea just to make sure that you don't have too much noise in your data in case it is biased.

00:38:27.000 | So at this point, we're ready to put our model in production.

00:38:32.000 | And this is where I hear a lot of people ask me about which mega Google Facebook highly distributed serving system they should use

00:38:46.000 | and how do they use a thousand GPUs at the same time and whatever else.

00:38:50.000 | For the vast, vast, vast majority of things that you all do, you will want to actually run in production on a CPU, not a GPU.

00:39:00.000 | Why is that? Because a GPU is good at doing lots of things at the same time.

00:39:05.000 | But unless you have a very busy website, it's pretty unlikely that you're going to have 64 images to classify at the same time to put into a batch into a GPU.

00:39:15.000 | And if you did, you've got to deal with all that queuing and running it all together.

00:39:19.000 | All of your users have to wait until that batch has got filled up and run. It's a whole lot of hassle, right?

00:39:25.000 | And then if you want to scale that, there's another whole lot of hassle.

00:39:28.000 | It's much easier if you just grab one thing, throw it at a CPU to get it done, and it comes back again.

00:39:36.000 | So yes, it's going to take, you know, maybe 10 or 20 times longer, right? So maybe it'll take 0.2 seconds rather than 0.01 seconds.

00:39:46.000 | That's about the kind of times we're talking about. But it's so easy to scale, right?

00:39:51.000 | You can chuck it on any standard serving infrastructure. It's going to be cheap as hell.

00:39:56.000 | You can horizontally scale it really easily, right?

00:39:59.000 | So most people I know who are running apps that aren't kind of at Google scale based on deep learning are using CPUs.

00:40:06.000 | And the term we use is inference, right?

00:40:08.000 | So when you're not training a model, but you've got a trained model and you're getting to predict things, we call that inference.

00:40:15.000 | That's why we say here you probably want to use CPU for inference.

00:40:22.000 | So at inference time, you've got your pre-trained model, you saved those weights, and how are you going to use them to create something like Simon Willison's Cougar Detector?

00:40:33.000 | Well, first thing you're going to need to know is what were the classes that you trained with, right?

00:40:39.000 | You need to not know, not just what are they, but what were the order, okay?

00:40:43.000 | So you will actually need to, like, serialize that or just type them in or in some way make sure you've got exactly the same classes that you trained with.

00:40:54.000 | If you don't have a GPU on your server, it will use the CPU automatically.

00:41:00.000 | If you want to test, if you have a GPU machine and you want to test using a CPU, you can just uncomment this line.

00:41:07.000 | And that tells FastAI that you want to use CPU by passing it back to PyTorch.

00:41:14.000 | So here's an example. We don't have a Cougar Detector, we have a Teddy Bear Detector, and my daughter Claire is about to decide whether to cuddle this friend, okay?

00:41:25.000 | So what she does is she takes Daddy's Deep Learning model and she gets a picture of this, and here is a picture that she's uploaded to the web app, okay?

00:41:34.000 | And here is a picture of the potentially cuddlesome object, and so we're going to store that in a very record image.

00:41:41.000 | So OpenImage is how you open an image in FastAI, funnily enough.

00:41:47.000 | Here is that list of classes that we saved earlier.

00:41:51.000 | And so as per usual, we created a data bunch, but this time we're not going to create a data bunch from a folder full of images.

00:41:59.000 | We're going to create a special kind of data bunch, which is one that's going to grab one single image at a time.

00:42:06.000 | So we're not actually passing it any data.

00:42:09.000 | The only reason we pass it a path is so that it knows where to load our model from, right?

00:42:14.000 | That's just the path, that's the folder that the model is going to be in.

00:42:17.000 | But what we do need to do is that we need to pass it the same information that we trained with, so the same transforms, the same size, the same normalization.

00:42:26.000 | This is all stuff we'll learn more about, but just make sure it's the same stuff that you used before.

00:42:30.000 | And so now you've got a data bunch that actually doesn't have any data in it at all.

00:42:35.000 | It's just something that knows how to transform a new image in the same way that you trained with, so that you can now do inference.

00:42:43.000 | So you can now create a CNN with this kind of fake data bunch.

00:42:47.000 | And again, you would use exactly the same model that you trained with.

00:42:52.000 | You can now load in those saved weights, okay?

00:42:55.000 | And so this is the stuff that you would do once, just once when your web app's starting up, okay?

00:43:00.000 | And it takes, you know, 0.1 of a second to run this code.

00:43:04.000 | And then you just go learn.predict image.

00:43:08.000 | And it's lucky we did that because it is not a teddy bear.

00:43:12.000 | This is actually a black bear.

00:43:14.000 | So thankfully, due to this excellent deep learning model, my daughter will avoid having a very embarrassing black bear cut-all incident.

00:43:26.000 | So what does this look like in production?

00:43:29.000 | Well, I took Simon Willison's code and shamelessly stole it, made it probably a little bit worse.

00:43:37.000 | But basically it's going to look something like this.

00:43:40.000 | So now Simon used a really cool web app toolkit called Starlet.

00:43:44.000 | If you've ever used Flask, this will look extremely similar, but it's kind of a more modern approach.

00:43:51.000 | And by modern, what I really mean is that you can use a weight.

00:43:56.000 | This basically means that you can wait for something that takes a while, such as grabbing some data, without using up a process.

00:44:05.000 | So for things like I want to get a prediction or I want to load up some data or whatever,

00:44:10.000 | it's really great to be able to use this modern Python 3 asynchronous stuff.

00:44:15.000 | So Starlet would come highly recommended for creating your web app.

00:44:20.000 | And so, yeah, you just create a route as per usual in a web app.

00:44:25.000 | And in that, you say this is async to ensure that it doesn't steal the process while it's waiting for things.

00:44:35.000 | You open your image, you call .predict, and you return that response.

00:44:41.000 | And then you can use whatever, JavaScript client or whatever to show it.

00:44:47.000 | And that's it. That's basically the main contents of your web app.

00:44:52.000 | So give it a go, right?

00:44:57.000 | This week, even if you've never created a web application before,

00:45:02.000 | there's a lot of nice little tutorials online and kind of starter code.

00:45:08.000 | If in doubt, why don't you try Starlet?

00:45:11.000 | There's a free hosting that you can use.

00:45:15.000 | There's one called Python Anywhere, for example.

00:45:19.000 | The one that Simon's used, we'll mention that on the forum.

00:45:22.000 | It's something you can basically package it up as a Docker thing and shoot it off,

00:45:26.000 | then it'll serve it up for you.

00:45:28.000 | So it doesn't even need to cost you any money.

00:45:30.000 | And so all these classifiers that you're creating, you can turn them into web applications.

00:45:36.000 | So I'll be really interested to see what you're able to make of that.

00:45:41.000 | That'll be really fun.

00:45:44.000 | Okay, so let's take a break. We'll come back at 7.35.

00:45:51.000 | See you then.

00:45:55.000 | Okay.

00:45:59.000 | So let's move on.

00:46:07.000 | So I mentioned that most of the time, the kind of rules of thumb I've shown you will probably work.

00:46:17.000 | And if you look at the Share Your Work thread, you'll find most of the time,

00:46:21.000 | people are posting things saying, "I downloaded these images. I tried this thing.

00:46:26.000 | It works much better than I expected." Well, that's cool.

00:46:29.000 | And then like 1 out of 20 says like, "Ah, I had a problem."

00:46:35.000 | So let's have a talk about what happens when you have a problem.

00:46:38.000 | And this is where we're going to start getting into a little bit of theory,

00:46:41.000 | because in order to understand why we have these problems and how we fix them,

00:46:45.000 | it really helps to know a little bit about what's going on.

00:46:47.000 | So first of all, let's look at examples of some problems.

00:46:51.000 | The problems basically will be either your learning rate is too high or low,

00:46:57.000 | or your number of epochs is too high or low.

00:47:01.000 | So we're going to learn about what those mean and why they matter.

00:47:05.000 | But first of all, because we're experimentalists, let's try them.

00:47:09.000 | So let's go with our teddy bear detector and let's make our learning rate really high.

00:47:16.000 | The default learning rate is 0.003. It works most of the time.

00:47:21.000 | So what if we try a learning rate of 0.5? That's huge.

00:47:25.000 | So what happens? Our validation loss gets pretty damn high.

00:47:31.000 | Remember, this is normally something that's underneath 1.

00:47:35.000 | So if you see your validation loss, do that.

00:47:38.000 | Before we even learn what validation loss is, just know this.

00:47:42.000 | If it does that, your learning rate is too high. That's all you need to know.

00:47:46.000 | Make it lower. It doesn't matter how many epochs you do.

00:47:50.000 | And if this happens, there's no way to undo this.

00:47:54.000 | So you have to go back and create your neural net again

00:47:57.000 | and fit from scratch with a lower learning rate.

00:48:00.000 | So that's learning rate too high.

00:48:02.000 | Learning rate too low, what if we use a learning rate not of 0.003, but

00:48:09.000 | 1e_neg_5, so 0.00001, right?

00:48:15.000 | So this is just, I've just copied and pasted what happened when we trained before

00:48:20.000 | with our default learning rate. And within one epoch we were down to a

00:48:25.000 | 2 or 3% error rate. With this really low learning rate,

00:48:29.000 | our error rate does get better, but very, very slowly.

00:48:35.000 | And you can plot it. If you go to learn.recorder is an object which is going to

00:48:41.000 | keep track of lots of things happening while you train.

00:48:44.000 | You can call plot_losses to plot out the validation and training loss.

00:48:49.000 | And you can just see them just gradually going down so slow.

00:48:54.000 | So if you see that happening, then you have a learning rate which is too small.

00:49:00.000 | So bump it up by 10 or bump it up by 100 and try again.

00:49:06.000 | The other thing you'll see if your learning rate is too small is that your

00:49:10.000 | training loss will be higher than your validation loss.

00:49:16.000 | You never want a model where your training loss is higher than your validation loss.

00:49:23.000 | That always means you haven't fitted enough, which means either your

00:49:28.000 | learning rate is too low or your number of epochs is too low.

00:49:32.000 | So if you have a model like that, train it some more or train it with a higher

00:49:38.000 | learning rate. OK? Too few epochs. So what if we train for just one epoch?

00:49:49.000 | Our error rate is certainly better than random, 5%, but look at this.

00:49:54.000 | The difference between training loss and validation loss, the training loss,

00:49:57.000 | is much higher than the validation loss.

00:50:00.000 | So too few epochs and too low a learning rate look very similar.

00:50:06.000 | And so you can just try running more epochs.

00:50:09.000 | And if it's taking forever, you can try a higher learning rate.

00:50:12.000 | Or if you try a higher learning rate and the loss goes off to 100,000 million,

00:50:17.000 | then put it back to where it was and try a few more epochs.

00:50:20.000 | That's the balance. That's basically all you care about 99% of the time.

00:50:25.000 | And this is only the 1 in 20 times that the defaults don't work for you.

00:50:30.000 | OK. Too many epochs. We're going to be talking more about this.

00:50:34.000 | Create something called overfitting.

00:50:36.000 | If you train for too long, as we're going to learn about,

00:50:39.000 | it will learn to recognize your particular teddy bears, but not teddy bears in general.

00:50:45.000 | Here's the thing. Despite what you may have heard, it's very hard to overfit

00:50:50.000 | with deep learning. So we were trying today to show you an example of overfitting

00:50:55.000 | and I turned off everything.

00:51:00.000 | We're going to learn all about these terms soon. I turned off all the data

00:51:04.000 | augmentation. I turned off dropout. I turned off weight decay.

00:51:09.000 | I tried to make it overfit as much as I can. I trained it on a small-ish learning rate.

00:51:13.000 | I trained it for a really long time. And like, maybe I started to get it to overfit.

00:51:20.000 | Maybe.

00:51:24.000 | So the only thing that tells you that you're overfitting is that the error

00:51:29.000 | rate improves for a while and then starts getting worse again.

00:51:35.000 | You will see a lot of people, even people that claim to understand machine

00:51:41.000 | learning, tell you that if your training loss is lower than your validation loss,

00:51:47.000 | then you are overfitting. As you will learn today in more detail and during the

00:51:52.000 | rest of the course, that is absolutely not true. Any model that is trained

00:51:56.000 | correctly will always have train loss lower than validation loss.

00:52:00.000 | That is not a sign of overfitting. That is not a sign you've done something wrong.

00:52:03.000 | That is a sign you have done something right.

00:52:07.000 | The sign that you are overfitting is that your error starts getting worse.

00:52:12.000 | Because that's what you care about, right? You want your model to have a low error.

00:52:16.000 | So as long as your training and your model error is improving, you are not overfitting.

00:52:21.000 | How could you be? Okay. So there's basically the four possible,

00:52:26.000 | the main four things that can go wrong. There are some other details that we will

00:52:30.000 | learn about during the rest of this course. But honestly, if you stopped

00:52:34.000 | listening now, please don't. That would be embarrassing.

00:52:37.000 | And you're just like, okay, I'm going to go and download images.

00:52:41.000 | I'm going to create CNNs with ResNet 34 or ResNet 50.

00:52:45.000 | I'm going to make sure that my learning rate and number of epochs is okay.

00:52:48.000 | And then I'm going to chuck them up in a Starlet web API.

00:52:53.000 | Most of the time you're done, okay, at least for computer vision.

00:52:57.000 | Hopefully you'll stick around because you want to learn about NLP

00:53:01.000 | and collaborative filtering and tabular data and segmentation

00:53:05.000 | and stuff like that as well.

00:53:11.000 | Let's now understand what's actually going on. What does it mean?

00:53:16.000 | Loss mean? What does an epoch mean? What does learning rate mean?

00:53:20.000 | Because for you to really understand these ideas, you need to know

00:53:24.000 | what's going on. And so we're going to go all the way to the other side.

00:53:28.000 | Rather than creating a state-of-the-art Cougar detector,

00:53:33.000 | we're going to go back and create the simplest possible linear model.

00:53:38.000 | So we're going to actually start seeing a little bit of math.

00:53:50.000 | But don't be turned off. It's okay. We're going to do a little bit of math

00:53:55.000 | but it's going to be totally fine even if math is not your thing.

00:53:58.000 | Because the first thing we're going to realize is that when we see a picture

00:54:02.000 | like this number 8, it's actually just a bunch of numbers.

00:54:07.000 | It's a matrix of numbers. For this grayscale one, it's a matrix of numbers.

00:54:12.000 | If it was a color image, it would have a third dimension.

00:54:18.000 | So when you add an extra dimension, we call it a tensor. Rather than a matrix,

00:54:22.000 | it would be a 3D tensor of numbers, red, green and blue.

00:54:28.000 | So when we created that heavy-bear detector,

00:54:33.000 | what we actually did was we created a mathematical function that took the numbers

00:54:38.000 | from the images of the teddy bears, and the mathematical function converted

00:54:43.000 | those numbers into, in our case, three numbers.

00:54:48.000 | A number for the probability that it's a teddy, a probability that it's a grisly,

00:54:52.000 | and a probability that it's a black bear.

00:54:54.000 | In this case, there's some hypothetical function that's taking the pixels

00:54:58.000 | and presenting a handwritten digit and returning 10 numbers,

00:55:03.000 | the probability for each possible outcome, the numbers from 0 to 9.

00:55:09.000 | And so what you'll often see in our code,

00:55:14.000 | in other deep learning code, is that you'll find this bunch of probabilities

00:55:20.000 | and then you'll find something called max or argmax attached to it,

00:55:25.000 | the function called. And so what that function is doing is it's saying,

00:55:28.000 | find the highest number, the highest probability, and tell me what the index is.

00:55:34.000 | So np.argmax or torch.argmax of this array would return this number here.

00:55:42.000 | Okay, we return index 8. Does that make sense?

00:55:46.000 | In fact, let's try it.

00:55:50.000 | So we know that the function to predict something is called learn.predict.

00:56:01.000 | Okay, so we can chuck two question marks before it or after it to get the source code.

00:56:08.000 | And here it is, right?

00:56:10.000 | pred=res_result.argmax.

00:56:15.000 | And then what is the class? We just pass that into the classes array.

00:56:19.000 | So like you should find that the source code in the FastAI library can both kind of

00:56:26.000 | strengthen your understanding of the concepts and make sure that you know what's going on

00:56:30.000 | and really help you here.

00:56:33.000 | You've got a question. Come on over.

00:56:38.000 | Alright, so can we have a definition of the error rate being discussed

00:56:42.000 | and how it is calculated? I assume it's cross-validation error.

00:56:48.000 | Sure. So one way to answer the question of how is error rate calculated

00:56:53.000 | would be to type error rate, question mark, and look at the source code.

00:56:59.000 | And it is 1 - accuracy. Fair enough.

00:57:03.000 | And so then a question might be what is accuracy?

00:57:08.000 | Accuracy, question mark.

00:57:11.000 | It is argmax. So we now know that means find out which particular thing it is

00:57:16.000 | and then look at how often that equals the target.

00:57:20.000 | So in other words, the actual value and take the mean.

00:57:24.000 | So that's basically what it is.

00:57:26.000 | And so then the question is, OK, what is that being applied to?

00:57:30.000 | And always in FastAI, metrics, so these things that we pass in, we call them metrics,

00:57:38.000 | are always going to be applied to the validation set.

00:57:42.000 | OK? So any time you put a metric here, it'll be applied to the validation set

00:57:47.000 | because that's your best practice, right? That's what you always want to do,

00:57:51.000 | is make sure that you're checking your performance on data that your model hasn't seen.

00:57:57.000 | And we'll be learning more about the validation set shortly.

00:58:01.000 | Remember, you can also type doc.

00:58:06.000 | If the source code is not what you want, which I will not be,

00:58:10.000 | you actually want the documentation.

00:58:12.000 | That will both give you a summary of the types in and out of the function

00:58:18.000 | and a link to the full documentation where you can find out all about

00:58:25.000 | how metrics work and what other metrics there are and so forth.

00:58:30.000 | And generally speaking, you'll also find links to more information

00:58:36.000 | where, for example, you will find complete runs through and sample code

00:58:40.000 | and so forth showing you how to use all these things.

00:58:42.000 | So don't forget that the doc function is your friend, OK?

00:58:47.000 | And also in the documentation, both in the doc function and in the documentation,

00:58:53.000 | you will see a source link. This is like question mark, question mark.

00:58:57.000 | But what the source link does is it takes you into the exact line of code in GitHub

00:59:04.000 | so you can see exactly how that's implemented and what else is around it.

00:59:07.000 | So lots of good stuff there.

00:59:12.000 | Why were you using 3s for your learning rates earlier

00:59:15.000 | with 3eNEG 5 and 3eNEG 4?

00:59:22.000 | We found that 3eNEG 3 is just a really good default learning rate.

00:59:29.000 | Most of the time, for your initial fine-tuning before you unfreeze.

00:59:36.000 | And then I tend to kind of just multiply from there.

00:59:40.000 | So I generally find then that the next stage I will pick 10 times lower than that

00:59:45.000 | for the second part of the slice and whatever the LRFinder found

00:59:49.000 | for the first part of the slice.

00:59:52.000 | The second part of the slice doesn't come from the LRFinder.

00:59:55.000 | It's just a rule of thumb which is like 10 times less than your first part

00:59:59.000 | which defaults to 3eNEG 3.

01:00:02.000 | And then the first part of the slice is what comes out of the LRFinder.

01:00:06.000 | And we'll be learning a lot more about these learning rate details

01:00:09.000 | both today and in the coming lessons.

01:00:12.000 | But yeah, for now all you need to remember is that in your basic approach looked like this.

01:00:18.000 | It was learn.fit 1 cycle, some number of epochs I often pick 4

01:00:26.000 | and some learning rate which defaults to 3eNEG 3.

01:00:32.000 | I'll just type it out fully so you can see.

01:00:34.000 | And then we do that for a bit and then we unfreeze it.

01:00:39.000 | And then we learn some more.

01:00:42.000 | And so this is the bit where I just take whatever I did last time

01:00:46.000 | and divide it by 10.

01:00:49.000 | And then I also write like that and then I have to put one more number in here

01:00:57.000 | and that's the number that I get from the LRFinder.

01:00:59.000 | The bit where it's got the strongest slope.

01:01:01.000 | So that's kind of the, kind of, don't have to think about it,

01:01:05.000 | don't really have to know what's going on, rule of thumb that works most of the time.

01:01:12.000 | But let's now dig in and actually understand it more completely.

01:01:18.000 | So we're going to create this mathematical function

01:01:20.000 | that takes the numbers that represent the pixels

01:01:22.000 | and spits out probabilities for each possible plus.

01:01:26.000 | And by the way, a lot of the stuff that we're using here,

01:01:30.000 | we are stealing from other people who are awesome

01:01:32.000 | and so we are putting their details here.

01:01:35.000 | So like please check out their work because they've got great work

01:01:39.000 | that we are highlighting in our course.

01:01:43.000 | I really like this idea of this little animated gif of the numbers.

01:01:47.000 | So thank you to Adam Guykey for creating that.

01:01:50.000 | And I guess that was probably on Quora by the looks of this medium.

01:01:55.000 | Oh yes it was too, that terrific medium post, I remember.

01:01:59.000 | In fact a whole series of medium posts.

01:02:03.000 | So let's look and see how we create one of these functions.

01:02:14.000 | And let's start with the simplest function I know,

01:02:21.000 | y = ax + b. That's a line, right?

01:02:33.000 | That's a line and the gradient of the line is here

01:02:39.000 | and the intercept of the line is here.

01:02:43.000 | So hopefully when we said that you need to know high school math

01:02:46.000 | to do this course, these are the things we're assuming that you remember.

01:02:50.000 | If we do kind of mention some math thing which I'm assuming you remember

01:02:54.000 | and you don't remember it, don't freak out, right?

01:02:57.000 | Happens to all of us.

01:03:00.000 | Khan Academy is actually terrific, it's not just for school kids.

01:03:04.000 | Go to Khan Academy, find the concept that you need a refresher on

01:03:07.000 | and he explains things really well.

01:03:10.000 | So I strongly recommend checking that out.

01:03:15.000 | Now remember I'm just a philosophy student, right?

01:03:18.000 | All the time I'm trying to either remind myself about something

01:03:21.000 | or I never learnt something and so we have the whole internet to teach us these things.

01:03:26.000 | So I'm going to rewrite this slightly.

01:03:30.000 | y = a1 x + a2.

01:03:39.000 | So let's just replace b with a2 and just give it a different name.

01:03:43.000 | So there's another way of saying the same thing.

01:03:47.000 | And another way of saying that would be if I could multiply a2 by the number 1.

01:03:54.000 | This still is the same thing.

01:03:58.000 | And so now at this point I'm actually going to say let's not put the number 1 there

01:04:03.000 | but let's put an x1 here and an x2 here and I'll say x2 = 1.

01:04:13.000 | So far this is pretty early high school math.

01:04:17.000 | This is multiplying by 1 which I think we can handle.

01:04:21.000 | So these two are equivalent with a bit of renaming.

01:04:26.000 | Now in machine learning we don't just have one equation.

01:04:32.000 | We've got lots, right?

01:04:34.000 | So if we've got some data that represents the temperature versus the number of ice creams sold

01:04:50.000 | then we kind of have lots of dots.

01:04:55.000 | And so each one of those dots we might hypothesize is based on this formula.

01:05:01.000 | y = a1 x1 + a2 x2.

01:05:05.000 | And so basically there's lots of, so this is our y, this is our x.

01:05:10.000 | There's lots of values of y so we can stick a little i here.

01:05:14.000 | And there's lots of values of x so we can stick a little x here.

01:05:19.000 | So the way we kind of do that is a lot like numpy indexing.

01:05:22.000 | But rather than things in square brackets or pytorch indexing, rather than things in square brackets

01:05:27.000 | we kind of put them down here in the subscript of our equation.

01:05:36.000 | So this is now saying there's actually lots of these different yi's based on lots of different xi1 and xi2.

01:05:45.000 | But notice there's still only one of each of these.

01:05:49.000 | So these things here are called the coefficients or the parameters.

01:05:56.000 | So this is our linear equation and we're going to say that every xi2 is equal to 1.

01:06:07.000 | Why did I do it that way? Because I want to do linear algebra.

01:06:13.000 | Why do I want to do it in linear algebra?

01:06:15.000 | Well one reason is because Rachel teaches the world's best linear algebra course.

01:06:20.000 | So if you're interested check out computational linear algebra for coders.

01:06:24.000 | So it's a good opportunity for me to throw in a pitch for this free course which we make no money but never mind.

01:06:32.000 | But more to the point right now it's going to make life much easier because I hate writing loops.

01:06:40.000 | I hate writing code. I just want the computer to do everything for me.

01:06:45.000 | And anytime you see like these little i subscripts that sounds like you're going to have to do loops and all kinds of stuff.

01:06:50.000 | But what you might remember from school is that when you've got like two things being multiplied together,

01:06:57.000 | two things being multiplied together and then they get added up, that's called a dot product.

01:07:05.000 | And then if you do that for lots and lots of different numbers i, then that's called a matrix product.

01:07:14.000 | So in fact this whole thing can be written like this.

01:07:17.000 | Rather than lots of different yi's we can say there's one vector called y which is equal to one matrix called x times one vector called a.

01:07:33.000 | Now at this point I know a lot of you don't remember that. So that's fine. We have a picture to show you.

01:07:43.000 | I don't know who created this. So now I do. Somebody called Andre Stouts created this fantastic thing called matrix multiplication dot xyz.

01:07:51.000 | And here we have a matrix by a vector and we're going to do a matrix vector product. Go.

01:08:02.000 | That times that times that plus plus plus plus. That times that times that plus plus plus plus. That times that times that plus plus plus plus. Finished.

01:08:11.000 | That is what matrix vector multiplication does. In other words, it's just that. Except his version is much less messy.

01:08:24.000 | OK. So let's. This is actually an excellent spot to have a little break and find out what questions we have coming through our students. What are they asking Rachel?

01:08:34.000 | OK. When generating new image data sets how do you know how many images are enough? What are ways to measure enough?

01:08:43.000 | Yeah. That's a great question. So another possible problem you have is you don't have enough data.

01:08:49.000 | How do you know if you don't have enough data? Because you found a good learning rate because if you make it higher than it goes off into massive losses, if you make it lower, it goes really slowly.

01:09:00.000 | So you've got a good learning rate. And then you train for such a long time that your error starts getting worse.

01:09:08.000 | OK. So you know that you've trained for long enough and you're still not happy with the accuracy. It's not good enough for the teddy bear cuddling level of safety you want.

01:09:19.000 | So if that happens, there's a number of things you can do and we'll learn about some of them during or learn pretty much all of them during this course.

01:09:27.000 | But one of the easiest ones is get more data. If you get more data, then you can train for longer, get a higher accuracy, lower error rate without overfitting.

01:09:40.000 | Unfortunately, there's no shortcut. I wish there was. I wish there was somewhere to know ahead of time how much data you need.

01:09:46.000 | But I will say this, most of the time you need less data than you think. So organizations very commonly spend too much time gathering data, getting more data than it turned out they actually needed.

01:09:56.000 | So get a small amount first and see how you go.

01:10:00.000 | What do you do if you have unbalanced classes such as 200 grizzlies and 50 teddies?

01:10:07.000 | Nothing. Try it. It works. A lot of people ask this question about how do I deal with unbalanced data.

01:10:14.000 | I've done lots of analysis with unbalanced data over the last couple of years and I just can't make it not work. It always works.

01:10:22.000 | So there's actually a paper that said if you want to get it slightly better, then the best thing to do is to take that uncommon class and just make a few copies of it.

01:10:35.000 | That's called oversampling. But I haven't found a situation in practice where I needed to do that. I've found it always just works fine for me.

01:10:47.000 | Once you unfreeze and retrain with one cycle again, if your training loss is still lower than your validation loss, likely underfitting, do you retrain it unfrozen again, which will technically be more than one cycle, or do you redo everything with a longer epoch for the cycle?

01:11:05.000 | Hey, you guys asked me that last week. My answer is still the same. I don't know. Either is fine. If you do another cycle, then it'll kind of maybe generalize a little bit better. If you start again, do twice as long. It's kind of annoying.

01:11:21.000 | Depends how patient you are. It won't make much difference. For me personally, I normally just train a few more cycles. But yeah, it doesn't make much difference most of the time.

01:11:38.000 | So showing the code sample where you were creating a CNN with ResNet 34 for the Grizzly/Teddy classifier, it says this requires ResNet 34, which I find surprising. I had assumed that the model created by .save, which is about 85 megabytes on disk, would be able to run without also needing a copy of ResNet 34.

01:12:07.000 | Yeah, and I understand. We're going to be learning all about this shortly.

01:12:18.000 | There's no copy of ResNet 34. ResNet 34 is actually what we call an architecture. We're going to be learning a lot about this. It's a functional form. Just like this is a linear functional form, it doesn't take up any room, it doesn't contain anything, it's just a function. ResNet 34 is just a function.

01:12:35.000 | It doesn't contain anything, it doesn't store anything. I think the confusion here is that we often use a pre-trained neural net that's being learnt on ImageNet. In this case, we don't need to use a pre-trained neural net.

01:12:51.000 | And actually, to entirely avoid that even getting created, you can actually pass pre-trained equals false, and that'll ensure that nothing even gets loaded, which will save you another .2 seconds, I guess.

01:13:11.000 | But we'll be learning a lot more about this, so don't worry if this is a bit unclear. But the basic idea is, this thing here is basically the equivalent of saying, is it a line, or is it a quadratic, or is it a reciprocal?

01:13:25.000 | This is just a function. This is the ResNet 34 function. It's a mathematical function. It doesn't take any storage, it doesn't have any numbers, it doesn't have to be loaded, as opposed to a pre-trained model.

01:13:39.000 | And so that's why when we did it at inference time, the thing that took space is this bit, which is where we load our parameters, which is basically saying, as we were able to find out, what are the values of A and B.

01:14:00.000 | We have to store those numbers. But for ResNet 34, you don't just store two numbers, you store a few millions, or a few tens of millions of numbers.

01:14:13.000 | So why did we do all this? Well, it's because I wanted to be able to write it out like this. And the nice thing of being able to write it out like this is that we can now do that in PyTorch with no loops, single line of code, and it's also going to run faster.

01:14:35.000 | PyTorch really doesn't like loops. It really wants you to send it a whole equation to do all at once, which means you really want to try and specify things in these kind of linear algebra ways.

01:14:47.000 | So let's go and take a look, because what we're going to try and do then is we're going to try and take this, we're going to call it an architecture. This is like the tiniest world's tiniest neural network. It's got two parameters, you know, A1 and A2.

01:15:03.000 | We're going to try and fit this architecture to some data. So let's jump into a notebook and generate some dots, right, and see if we can get it to fit a line somehow.

01:15:17.000 | And the somehow is going to be using something called SGD. What is SGD? Well, there's two types of SGD. The first one is where I said in Lesson 1, "Hey, you should all try building these models and try and come up with something cool."

01:15:35.000 | And you guys all experimented and found really good stuff. So that's where the S would be student. That would be student gradient descent. So that's version 1 of SGD. Version 2 of SGD, which is what we're going to talk about today, is where we're going to have a computer try lots of things and try and come up with a really good function.

01:15:52.000 | And that will be called "Stochastic Gradient Descent". So the other one that you hear a lot on Twitter is "Stochastic Grad Student Descent". So that's the other one that you hear.

01:16:07.000 | So we're going to jump into Lesson 2, SGD. And so we're going to kind of go bottom up rather than top down. We're going to create the simplest possible model we can, which is going to be a linear model.

01:16:23.000 | And the first thing that we need is we need some data. And so we're going to generate some data. The data we're going to generate looks like this. So this might represent temperature and this might represent number of ice creams we sell or something like that.

01:16:37.000 | But we're just going to create some synthetic data that we know is following a line. And so as we build this, we're actually going to learn a little bit about PyTorch as well.

01:16:48.000 | So basically the way we're going to generate this data is by creating some coefficients. A1 will be 3 and A2 will be 2. And we're going to create some,

01:17:03.000 | like we just looked at before, basically a column of numbers for our x's and a whole bunch of 1's. And then we're going to do this, x at a. What is x at a? x at a in Python means a matrix product between x and a.

01:17:22.000 | It actually is even more general than that. It can be a vector-vector product, a matrix-vector product, a vector-matrix product, or a matrix-matrix product.

01:17:29.000 | And then actually in PyTorch specifically, it can mean even more general things where we get into higher-rank tensors, which we will learn all about very soon.

01:17:39.000 | But this is basically the key thing that's going to go on in all of our deep learning. The vast majority of the time, our computers are going to be basically doing this.

01:17:49.000 | Multiplying numbers together and adding them up, which is a surprisingly useful thing to do.

01:17:58.000 | OK. So we basically are going to generate some data by creating a line and then we're going to add some random numbers to it. But let's go back and see how we created x and a.

01:18:09.000 | So I mentioned that we've basically got these two coefficients, 3 and 2. And you'll see that we've wrapped it in this function called Tensor.

01:18:19.000 | You might have heard this word "tensor" before. Who's heard the word "tensor" before? About two-thirds of you. OK.

01:18:26.000 | So it's one of these words that sounds scary. And apparently if you're a physicist, it actually is scary. But in the world of deep learning, it's actually not scary at all.

01:18:38.000 | Tensor means array. OK? It means array. Now specifically, it's an array of a regular shape. Right? So it's not an array where row one has two things and row three has three things and row four has one thing, what you call a jagged array.

01:18:52.000 | That's not a tensor. A tensor is any array which has a rectangular or cube or whatever, you know, a shape where every element, every row is the same length and then every column is the same length.

01:19:07.000 | A 4 by 3 matrix would be a tensor. A vector of length 4 would be a tensor. A 3D array of length 3 by 4 by 6 would be a tensor. That's all a tensor is. OK?

01:19:22.000 | And so we have these all the time. For example, an image is a three-dimensional tensor. It's got number of rows by number of columns by number of channels, normally red, green, blue.

01:19:38.000 | So for example, kind of a VGA texture would be 640 by 480 by 3. Or actually, we do things backwards. So when people talk about images, they normally go width by height. But when we talk mathematically, we always go number of rows by number of columns.

01:19:56.000 | So it'd actually be 480 by 640 by 3. That will catch you out. We don't say dimensions, though, with tensors. We use one of two words. We either say "rank" or "axis". "Rank" specifically means how many axes are there, how many dimensions are there.

01:20:15.000 | So an image is generally a rank 3 tensor. So what we've created here is a rank 1 tensor, or also known as a vector.

01:20:30.000 | But in math, people come up with slightly different words, or actually, no, they come up with very different words for slightly different concepts. Why is a one-dimensional array a vector and a two-dimensional array is a matrix, and then a three-dimensional array doesn't even have a name?

01:20:48.000 | Not really. It doesn't have a name. It doesn't make any sense. With computers, we try to have some simple, consistent naming conventions. They're all called tensors.

01:20:58.000 | Rank 1 tensor, rank 2 tensor, rank 3 tensor. You can certainly have a rank 4 tensor. If you've got 64 images, then that would be a rank 4 tensor of 64 by 480 by 640 by 3, for example.

01:21:13.000 | So tensors are very simple. They just mean arrays. And so in PyTorch, you say tensor, and you pass in some numbers, and you get back, if in this case, just a list, get back a vector.

01:21:28.000 | So this then represents our coefficients, the slope and the intercept of our line. And so because remember, we're not actually going to have a special case of Ax + b. Instead, we're going to say there's always this second x value, which is always 1. You can see it here, always 1, which allows us just to do a simple matrix vector product.

01:21:54.000 | OK, so that's a. And then we wanted to generate this x array of data, which is going to have, we're going to put random numbers in the first column and a whole bunch of ones in the second column.

01:22:08.000 | So to do that, we basically say to PyTorch, create a rank 2 tensor. Actually, no, sorry, let's start that again. We say to PyTorch that we want to create a tensor of n by 2.

01:22:29.000 | So since we passed in a total of two things, we get a rank 2 tensor, the number of rows will be n, and the number of columns will be 2. And in there, every single thing in it will be a 1. That's what torch.ones mean.

01:22:45.000 | And then, this is really important, you can index into that, just like you can index into a list in Python, but you can put a colon anywhere, and a colon means every single value on that axis, or every single value on that dimension.

01:23:04.000 | So this here means every single row. And then this here means column 0. So this is every row of column 0. I want you to grab a uniform random number.

01:23:19.000 | And here's another very important concept. In PyTorch, any time you've got a function that ends in an underscore, that means don't return to me that uniform random number, but replace whatever this is being called on with the result of this function.

01:23:36.000 | So this takes column 0, and replaces it with a uniform random number between -1 and 1. So there's a lot to unpack there, right? But the good news is, those two lines of code, plus this one, which we're coming to, cover 95% of what you need to know about PyTorch.

01:23:59.000 | How to create an array, how to change things in an array, and how to do matrix operations on an array. So there's a lot to unpack, but these small number of concepts are incredibly powerful.

01:24:13.000 | So I can now print out the first five rows. OK, so colon 5 is standard Python slicing syntax to say the first five rows. So here are the first five rows, two columns, looking like my random numbers, and my 1.

01:24:35.000 | So now I can do a matrix product of that x by my a, add in some random numbers to add a bit of noise, and then I can do a scatterplot. And I'm not really interested in my scatterplot in this column of 1's, right? They're just there to make my linear function more convenient.

01:24:56.000 | So I'm just going to plot my 0 index column against my y's, and there it is.

01:25:06.000 | PLT is what we universally use to refer to the plotting library matplotlib. And that's what most people use for most of their plotting in Python, in scientific Python, we use matplotlib.

01:25:22.000 | It's certainly a library you'll want to get familiar with, because being able to plot things is really important. There are lots of other plotting packages.

01:25:33.000 | Lots of them, the other packages are better at certain things than matplotlib, but like matplotlib can do everything reasonably well.

01:25:44.000 | Sometimes it's a little awkward, but for me, I do pretty much everything in matplotlib because there's really nothing it can't do, even though some libraries can do other things a little bit better or a little bit prettier.

01:25:59.000 | But it's really powerful, so once you know matplotlib, you can do everything. So here I'm asking matplotlib to give me a scatterplot with my x's against my y's, and there it is.

01:26:10.000 | So this is my dummy data representing like temperature and ice cream sales.

01:26:18.000 | So now what we're going to do is we're going to pretend we were given this data, and we don't know that the values of our coefficients are 3 and 2.

01:26:27.000 | So we're going to pretend that we never knew that we have to figure them out.

01:26:31.000 | So how would we figure them out? How would we draw a line to fit to this data? And why would that even be interesting?

01:26:40.000 | Well, we're going to look at more about why it's interesting in just a moment, but the basic idea is this.

01:26:46.000 | If we can find, this is going to be kind of perhaps really surprising, but if we can find a way to find those two parameters to fit that line to those, how many points were there, and was 100.

01:27:02.000 | If we can find a way to fit that line to those 100 points, we can also fit these arbitrary functions that convert from pixel values to probability.

01:27:14.000 | It'll turn out that these techniques that we're going to learn to find these two numbers works equally well for the 50 million numbers in ResNet 34.

01:27:27.000 | So we're actually going to use an almost identical approach.

01:27:30.000 | And this is the bit that I found in previous classes, people have the most trouble digesting.

01:27:38.000 | Like I often find, even after week four or week five, people will come up to me and say, "I don't get it. How do we actually train these models?"

01:27:46.000 | And I'll say, "It's SGD. It's that thing we throw in the notebook with the two numbers."

01:27:51.000 | It's like, "Yeah, but we're fitting a neural network."

01:27:54.000 | It's like, "I know, and we can't print the 50 million numbers anymore, but it is literally identically doing the same thing."

01:28:02.000 | And the reason this is hard to digest is that the human brain has a lot of trouble conceptualizing of what an equation with 50 million numbers looks like and can do.

01:28:13.000 | So you just kind of, for now, will have to take my word for it.

01:28:16.000 | It can do things like recognize teddy bits.

01:28:20.000 | You know, these functions turn out to be very powerful.

01:28:23.000 | And we're going to learn a little bit more in just a moment about how to make them extra powerful.

01:28:27.000 | But for now, this thing we're going to learn to fit these two numbers is the same thing that we've just been using to fit 50 million numbers.

01:28:36.000 | Okay. So we want to find what PyTorch calls parameters, or in statistics you'll often hear called coefficients, these values A1 and A2.

01:28:47.000 | We want to find these parameters such that the line that they create minimizes the error between that line and the points.

01:29:00.000 | So in other words, you know, if we created, you know, if the A1 and A2 we came up with resulted in this line, then we'd look and we'd see, like, how far away is that line from each point?

01:29:20.000 | And we'd say, oh, that's quite a long way. And so maybe there was some other A1 or A2 which resulted in this line.

01:29:30.000 | And they would say, like, oh, how far away is each of those points?

01:29:33.000 | And then eventually we come up with, we come up with this line and it's like, oh, in this case, each of those is actually very close.

01:29:47.000 | So you can see how in each case we can say, how far away is the line at each spot away from its point?

01:29:53.000 | And then we can take the average of all those and that's called the loss.

01:29:59.000 | That is the value of our loss. So you need some mathematical function that can basically say, how far away is this line from those points?

01:30:10.000 | For this kind of problem, which is called a regression problem, a problem where your dependent variable is continuous.

01:30:21.000 | So rather than being grizzlies or teddies, it's like some number between -1 and 6. This is called a regression problem.

01:30:29.000 | And for regression, the most common loss function is called mean squared error, which pretty much everybody calls MSE.

01:30:37.000 | You may also see RMSE, which is root mean squared error. And so the mean squared error is a loss.

01:30:43.000 | It's the difference between some prediction that you've made, which is the value of the line, and the actual number of ice cream sales.

01:30:54.000 | And so in the mathematics of this, people normally refer to the actual, they normally call it Y, and the prediction they normally call it Y hat, as in they write it.

01:31:06.000 | Like that.

01:31:10.000 | And so what I try to do, like when we're writing something like a mean squared error equation, there's no point writing ice cream here and temperature here, because we want it to apply to anything.

01:31:23.000 | So we tend to use these mathematical placeholders.

01:31:28.000 | So the value of mean squared error is simply the difference between those two squared.

01:31:35.000 | And then we can take the mean. Because remember, that is actually a vector, or what we now call it, a rank one tensor.

01:31:44.000 | And that is actually a rank one tensor. So it's the value of the number of ice cream sales at each place.

01:31:51.000 | So when we subtract one vector from another vector, and we're going to be learning a lot more about this,

01:31:57.000 | but it does something called element-wise arithmetic. In other words, it subtracts each one from each other.

01:32:03.000 | And so we end up with a vector of differences. And then if we take the square of that, it squares everything in that vector.

01:32:10.000 | And so then we can take the mean of that to find the average square of the differences between the actuals and the predicted.

01:32:19.000 | So if you're more comfortable with mathematical notation, what we just wrote there was the sum of (which way round do we do it?)

01:32:32.000 | y hat minus y squared over n. So that equation is the same as that equation.

01:32:47.000 | So one of the things I'll note here is I don't think this is more complicated or unwieldy than this.

01:33:00.000 | But the benefit of this is you can experiment with it. Once you've defined it, you can use it.

01:33:08.000 | You can send things into it and get stuff out of it and see how it works.

01:33:11.000 | So for me, most of the time I prefer to explain things with code rather than with math, because they're the same.

01:33:21.000 | In this case, at least, in all the cases we'll look at, they're exactly the same. They're just different notations for the same thing.

01:33:29.000 | But one of the notations is executable. It's something that you can experiment with. And one of them is abstract.

01:33:36.000 | So that's why I'm generally going to show code. So the good news is if you're a coder with not much of a math background,

01:33:46.000 | actually you do have a math background, because code is math. If you've got more of a math background and less of a code background,

01:33:54.000 | then actually a lot of the stuff that you learn from math is going to translate very directly into code.

01:33:59.000 | And now you can start to experiment really with your math.

01:34:03.000 | OK, so this is the loss function. This is something that tells us how good our line is.

01:34:08.000 | So now we have to kind of come up with what is the line that fits through here. Remember, we don't know. We're going to pretend we don't know.

01:34:17.000 | So what you actually have to do is you have to guess. You actually have to come up with a guess. What are the values of a1 and a2?

01:34:25.000 | So let's say we guessed that a1 and a2 are both 1. So this is our tensor.

01:34:31.000 | a is 1, 1. Right? So here is how we create that tensor. And I wanted to write it this way because you'll see this all the time.

01:34:42.000 | Like, written out, it should be 1.0. Oh, sorry, it looks like we're starting with -1. -1. Written out fully, it would be -1.0, 1.0. Like, that's written out fully.

01:34:56.000 | We can't write it without the point because that's now an int, not a floating point. So that's going to fit the dummy if you try to do calculations with that in neural nets.

01:35:08.000 | Right? I'm lazy. I'm far too lazy to type .0 every time. Python knows perfectly well that if you add a dot next to any of these numbers, then the whole thing is now floats.

01:35:21.000 | Right? So that's why you'll often see it written this way, particularly by lazy people like me.

01:35:27.000 | Okay. So a is a tensor. You can see it's floating point. You see, like, even PyTorch is lazy. They just put a dot. They don't bother with a zero. Right?

01:35:38.000 | But if you want to actually see exactly what it is, you can write .type. And you can see it's a float tensor. Okay?

01:35:51.000 | And so now we can calculate our predictions with this, like, random guess x at a, matrix product of x and a. And we can now calculate the mean squared error of our predictions and our actuals.

01:36:06.000 | And that's our loss. Okay? So for this regression, our loss is 8.9.

01:36:13.000 | And so we can now plot a scatter plot of x against y, and we can plot the scatter plot of x against y hat, our predictions, and there they are.

01:36:24.000 | Okay? So this is the 1 comma minus 1 line. Sorry, minus 1 comma 1 line, and here's our actuals. So that's not great. It's not surprising. It's just a guess.

01:36:35.000 | So SGD, or gradient descent more generally, and anybody who's done any engineering or probably computer science at school will have done plenty of this, like, Newton's method, whatever.

01:36:48.000 | It's all the stuff that you did at university. If you didn't, don't worry. We're going to learn it now. It's basically about taking this guess and trying to make it a little bit better.

01:36:58.000 | So how do we make it a little bit better? Well, there's only two numbers, right? And the two numbers are the intercept of that orange line and the gradient of that orange line.

01:37:12.000 | So what we're going to do with gradient descent is we're going to simply say, what if we change those two numbers a little bit?

01:37:19.000 | What if we made the intercept a little bit higher or a little bit lower?

01:37:27.000 | What if we made the gradient a little bit more positive or a little bit more negative?

01:37:35.000 | So there's like four possibilities. And then we can just calculate the loss for each of those four possibilities and see what works. Did lifting it up or down make it better?

01:37:46.000 | Did tilting it more positive or more negative make it better? And then all we do is we say, OK, well, whichever one of those made it better, that's what we're going to do.

01:37:57.000 | And that's it. But here's the cool thing for those of you that remember calculus. You don't actually have to move it up and down and round about.

01:38:07.000 | You can actually calculate the derivative. The derivative is the thing that tells you would moving it up or down make it better or would rotating it this way or that way make it better.

01:38:18.000 | So the good news is if you didn't do calculus or you don't remember calculus, I just told you everything you need to know about it, right? Which is that it tells you how changing one thing changes the function.

01:38:32.000 | That's what the derivative is. Kind of not quite strictly speaking, but close enough, also called the gradient.

01:38:39.000 | So the gradient or the derivative tells you how changing A1 up or down would change our MSE, how changing A2 up or down would change our MSE.

01:38:50.000 | And it does it more quickly. It does it more quickly than actually moving it up and down, right?

01:38:56.000 | So in school, unfortunately, they force us to sit there and calculate these derivatives by hand. We have computers. Computers can do that for us. We are not going to calculate them by hand.

01:39:12.000 | Instead, we're going to call .grad. On our computer, that will calculate the gradient for us. So here's what we're going to do. We're going to create a loop. We're going to loop through 100 times and we're going to call a function called update.

01:39:32.000 | That function is going to calculate y hat, our prediction. It is going to calculate loss, our mean squared error. From time to time, it will print that out so we can see how we're going.

01:39:48.000 | It will then calculate the gradient. And in PyTorch, calculating the gradient is done by using a method called backward.

01:39:56.000 | So you'll see something really interesting, which is mean squared error was just a simple standard mathematical function.

01:40:06.000 | PyTorch, for us, keeps track of how it was calculated and lets us calculate the derivative. So if you do a mathematical operation on a tensor in PyTorch, you can call backward to calculate the derivative.

01:40:21.000 | What happens to that derivative? It gets stuck inside an attribute called .grad. So I'm going to take my coefficients a, and I'm going to subtract from them my gradient.

01:40:36.000 | And there's an underscore here. Why? Because that's going to do it in place. So it's going to actually update those coefficients a to subtract the gradients from them.

01:40:50.000 | So why do we subtract? Because the gradient tells us if I move the whole thing downwards, the loss goes up. If I move the whole thing upwards, the loss goes down.

01:41:02.000 | So I want to do the opposite of the thing that makes it go up. Because we want our loss to be small. So that's why we have to subtract.

01:41:12.000 | And then there's something here called Lr. Lr is our learning rate. And so literally all it is, is the thing that we multiply by the gradient.

01:41:27.000 | Why is there any Lr at all? Let me show you why. Let's take a really simple example. A quadratic.

01:41:47.000 | And let's say your algorithm's job was to find where that quadratic was at its lowest point. And so how could it do this?

01:41:55.000 | Well, just like what we're doing now, the starting point would just be to pick some x value at random. And then pop up here to find out what the value of y is.

01:42:07.000 | And that's the starting point. And so then it can calculate the gradient. And the gradient is simply the slope.

01:42:14.000 | It tells you moving in which direction is going to make you go down. And so the gradient tells you you have to go this way.

01:42:23.000 | So if the gradient was really big, you might jump this way a very long way. So you might jump all the way over to here.

01:42:38.000 | Maybe even here. Right? And so if you jumped over to there, then that's actually not going to be very helpful.

01:42:48.000 | Because then you see, well, where does that take us to? Oh, it's now worse. Right? We jumped too far.

01:42:58.000 | So we don't want to jump too far. So maybe we should just jump a little bit. Maybe to here.

01:43:07.000 | And the good news is that is actually a little bit closer. And so then we'll just do another little jump.

01:43:12.000 | See what the gradient is into another little jump. That takes us to here. And another little jump. That takes us to here.

01:43:19.000 | Here. Here. Right? So in other words, we find our gradient to tell us kind of what direction to go and like, do we have to go a long way or not too far?

01:43:29.000 | But then we multiply it by some number less than one so we don't jump too far.

01:43:35.000 | And so hopefully at this point, this might be reminding you of something, which is what happened when our learning rate was too high.

01:43:48.000 | So do you see why that happened now? Our learning rate was too high, meant that we jumped all the way past the right answer further than we started with, and it got worse and worse and worse.

01:44:04.000 | So that's what a learning rate too high does.

01:44:11.000 | On the other hand, if our learning rate is too low, then you just take tiny little steps.

01:44:19.000 | And so eventually you're going to get there, but you're doing lots and lots of calculations along the way.

01:44:24.000 | So you really want to find something where it's kind of either like this, or maybe it's kind of a little bit backwards and forwards. Maybe it's kind of like this.

01:44:35.000 | Something like that. You know, you want something that kind of gets in there quickly, but not so quickly it jumps out and diverges, not so slowly that it takes lots of steps.

01:44:46.000 | So that's why we need a good learning rate. And so that's all it does. So if you look inside the source code of any deep learning library, you'll find this.

01:44:58.000 | You'll find something that just says coefficients dot subtract learning rate times gradient.

01:45:05.000 | And we'll learn about some easy but important optimizations we can do to make this go faster.

01:45:14.000 | But that's basically it. There's a couple of other little minor issues that we don't need to talk about now.

01:45:19.000 | One involving zeroing out the gradients, another involving making sure that you turn gradient calculation off when you do the SGD update.

01:45:28.000 | If you're interested, we can discuss them on the forum. Or you can do our introduction to machine learning course, which covers all the mechanics of this in more detail.

01:45:41.000 | But this is the basic idea. So if we run update 100 times, printing out the loss from time to time, you can see it starts at 8.9 and it goes down, down, down, down, down, down, down.

01:45:54.000 | And so we can then print out scatter plots. And there it is. That's it. Believe it or not, that's gradient descent.

01:46:03.000 | So we just need to start with a function that's a bit more complex than x at a.

01:46:12.000 | But as long as we have a function that can represent things like is this a teddy bear, we now have a way to fit it.

01:46:19.000 | Okay? And so let's now take a look at this as a picture, as an animation. And this is one of the nice things that you can do with...

01:46:32.000 | This is one of the nice things that you can do with Matplotlib is you can take any plot and turn it into an animation.

01:46:39.000 | And so you can now actually see it updating each step. So let's see what we did here. We simply said, as before, create a scatter plot.

01:46:50.000 | But then rather than having a loop, we used Matplotlib's func animation. So call 100 times this function.

01:47:01.000 | And this function just called that update that we created earlier and then updated the y data in our line.

01:47:09.000 | And so did that 100 times, waiting 20 milliseconds after each one. And there it is.

01:47:14.000 | So you might think that visualizing your algorithms with animations is some amazing and complex thing to do.

01:47:23.000 | But actually now you know it's 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 lines of code. Okay? So I think that is pretty damn cool.

01:47:35.000 | So that is SGD visualized. And so we can't visualize as conveniently what updating 50 million parameters in a ResNet 34 looks like.

01:47:46.000 | But it's basically doing the same thing. Okay? And so studying these simple versions is actually a great way to get an intuition.

01:47:52.000 | So you should try running this notebook with a really big learning rate, with a really small learning rate, and see what this animation looks like.

01:48:01.000 | Try to get a feel for it. Maybe you can even try a 3D plot. I haven't tried that yet, but I'm sure it would work fine too.

01:48:08.000 | So the only difference between stochastic gradient descent and this is something called many batches.

01:48:16.000 | You'll see what we did here was we calculated the value of the loss on the whole data set on every iteration.

01:48:23.000 | But if your data set is 1.5 million images in ImageNet, that's going to be really slow, right?

01:48:28.000 | Just to do a single update of your parameters, you've got to calculate the loss on 1.5 million images. You wouldn't want to do that.

01:48:37.000 | So what we do is we grab 64 images or so at a time, at random, and we calculate the loss on those 64 images, and we update our weights.

01:48:48.000 | And then we grab another 64 random images, and we update the weights.

01:48:52.000 | So in other words, the loop basically looks exactly the same, but at this point here, it would basically be y^2 and some random indexes here,

01:49:05.000 | and would basically do the same thing. Well, actually, sorry, it would be there, right?

01:49:16.000 | So some random indexes on our x and some random indexes on our y to do a mini-batch at a time. That would be the basic difference.

01:49:24.000 | And so once you add those, grab a random few points each time, those random few points accord your mini-batch,

01:49:33.000 | and that approach is called SGD, or Stochastic Gradient Descent.

01:49:39.000 | So there's quite a bit of vocab we've just covered, right? So let's just remind ourselves.

01:49:48.000 | The learning rate is a thing that we multiply our gradient by to decide how much to update the weights by.

01:49:56.000 | And epoch is one complete run through all of our data points, all of our images.

01:50:05.000 | So for the non-stochastic gradient descent we just did, every single loop we did the entire data set.

01:50:12.000 | But if you've got a data set with a thousand images, and your mini-batch size is 100, then it would take you 10 iterations to see every image once.

01:50:22.000 | So that would be one epoch. Epochs are important because if you do lots of epochs, then you're looking at your images lots of time.

01:50:32.000 | And so every time you see an image, there's a bigger chance of overfitting. So we generally don't want to do too many epochs.

01:50:41.000 | A mini-batch is just a random bunch of points that you use to update your weights.

01:50:47.000 | SGD is just gradient descent using mini-batches.

01:50:53.000 | Architecture and model kind of mean the same thing. In this case, architecture is y = x_a.

01:51:05.000 | The architecture is the mathematical function that you're fitting the parameters to.

01:51:10.000 | And we're going to learn today or next week what the mathematical function of things like ResNet 34 actually is.

01:51:20.000 | But it's basically pretty much what you've just seen. It's a bunch of matrix products.

01:51:26.000 | Parameters, also known as coefficients, also known as weights, are the numbers that you're updating.

01:51:35.000 | And then loss function is the thing that's telling you how far away or how close you are to the correct answer. Any questions?

01:51:45.000 | Alright, so these models, these predictors, these teddy bear classifiers are functions that take pixel values and return probability.

01:51:54.000 | They start with some functional form like y = x_a and they fit the parameters a using SGD to try and do the best to calculate your predictions.

01:52:10.000 | So far we've learned how to do regression, which is a single number.

01:52:16.000 | Next week we'll learn how to do the same thing for classification where we have multiple numbers.

01:52:21.000 | But it's basically the same.

01:52:25.000 | In the process we had to do some math. We had to do some linear algebra and we had to do some calculus.

01:52:34.000 | And a lot of people get a bit scared at that point and tell us, "I am not a math person."

01:52:41.000 | If that is you, that's totally okay, but you're wrong. You are a math person. In fact, it turns out that in the actual academic research around this,

01:52:53.000 | there are not math people and non-math people. It turns out to be entirely a result of culture and expectations.

01:53:02.000 | So you should check out Rachel's talk. There's No Such Thing as Not a Math Person, where she will introduce you to some of that academic research.

01:53:13.000 | And so if you think of yourself as not a math person, you should watch this so that you learn that you're wrong,

01:53:19.000 | that your thoughts are actually there because somebody has told you you're not a math person.

01:53:25.000 | But there's actually no academic research to suggest that there is such a thing.

01:53:30.000 | In fact, there are some cultures like Romania and China where the Not a Math Person concept never even appeared.

01:53:39.000 | It's almost unheard of in some cultures for somebody to say, "I'm not a math person," because they just never entered that cultural identity.

01:53:49.000 | So don't freak out if words like derivative and gradient and matrix product are things that you're kind of scared of.

01:53:59.000 | It's something you can learn. It's something you'll be okay with, okay?

01:54:05.000 | So the last thing that we're going to close with today...

01:54:11.000 | Oh, I just got a message from Simon Willison.

01:54:17.000 | Ah, Simon's telling me he's actually not that special. Lots of people won medals.

01:54:25.000 | That's the worst part about Simon, is not only is he really smart, he's also really modest, which I think is just awful.

01:54:34.000 | I mean, if you're going to be that smart, at least be a horrible human being and make it okay.

01:54:42.000 | Okay, so the last thing I want to close with is the idea of, and we're going to look at this more next week, underfitting and overfitting.

01:54:53.000 | We just fit a line to our data, but imagine that our data wasn't actually line-shaped, right?

01:55:01.000 | And so if we tried to fit something which was like constant plus constant times x, i.e. a line to it, then it's never going to fit very well, right?

01:55:10.000 | No matter how much we change these two coefficients, it's never going to get really close.

01:55:16.000 | On the other hand, we could fit some much bigger equation, so in this case it's a higher degree polynomial, with lots and lots of wiggly bits, like so, right?

01:55:25.000 | But if we did that, it's very unlikely we go and look at some other place to find out the temperature that it is and how much ice cream they're selling,

01:55:33.000 | and that we'll get a good result, because like the wiggles are far too wiggly.

01:55:38.000 | So this is called overfitting. We're looking for some mathematical function that fits just right, to stay with the teddy bear analogy.

01:55:49.000 | So you might think if you have a statistics background, the way to make things fit just right is to have exactly the right number of parameters.

01:55:59.000 | To use a mathematical function that doesn't have too many parameters in it.

01:56:03.000 | It turns out that's actually completely not the right way to think about it.

01:56:08.000 | There are other ways to make sure that we don't overfit, and in general this is called regularization.

01:56:14.000 | Regularization are all the techniques to make sure that when we train our model, that it's going to work not only well on the data it's seen, but on the data it hasn't seen yet.

01:56:28.000 | So the most important thing to know when you've trained a model is actually how well does it work on data that it hasn't been trained with.

01:56:38.000 | And so as we're going to learn a lot about next week, that's why we have this thing called a validation set.

01:56:44.000 | So what happens with a validation set is that we do our mini-batch STD training loop with one set of data, with one set of teddy bears, grizzlies, black bears.

01:56:58.000 | And then when we're done, we check the loss function and the accuracy to see how good is it on a bunch of images which were not included in the training.

01:57:07.000 | And so if we do that, then if we have something which is too wiggly, it'll tell us, "Oh, your loss function and your error is really bad."

01:57:15.000 | Because on the bears that it hasn't been trained with, the wiggly bits are in the wrong spot.

01:57:20.000 | Or if it was underfitting, it would also tell us that your validation set's really bad.

01:57:26.000 | So even for people that don't go through this course and don't learn about the details of deep learning, like if you've got managers or colleagues or whatever at work who are kind of wanting to learn about AI,

01:57:43.000 | the only thing that you really need to be teaching them is about the idea of a validation set.

01:57:47.000 | Because that's the thing they can then use to figure out, you know, if somebody's telling them snake oil or not, you know, to like hold back some data and then they get told like, "Oh, here's a model that we're going to roll out."

01:57:58.000 | And then you say, "Okay, fine. I'm just going to check it on this held-out data to see whether it generalizes."

01:58:04.000 | There's a lot of details to get right when you design your validation set.

01:58:09.000 | We will talk about them briefly next week, but a more full version would be in Rachel's piece on the Fast AI blog called "How and Why to Create a Good Validation Set."

01:58:22.000 | And this is also one of the things we go into in a lot of detail in the Intro to Machine Learning course.

01:58:27.000 | So we're going to try and give you enough to get by for this course, but it is certainly something that's worth deeper study as well.

01:58:35.000 | Any questions or comments before we wrap up?

01:58:39.000 | Okay, good.

01:58:40.000 | All right.

01:58:41.000 | Well, thanks, everybody.

01:58:42.000 | I hope you have a great time building your web applications.

01:58:44.000 | See you next week.

01:58:46.000 | (audience applauds)

Lesson 2: Deep Learning 2019 - Data cleaning and production; SGD from scratch

Chapters