Live coding 6

00:00:00.000 | Okay. Did anybody have any questions or anything? I'm sorry. I'm sorry. I'm sorry. I'm sorry.

00:00:29.000 | Questions or anything from last time or comments or anything else? How's your VIM going?

00:00:38.000 | I just had one, Jeremy. I wasn't really sure what happened when I looked. So I did the VIM and CTAGS thing in paper space.

00:00:50.000 | And as per the session, and then when I shut down the machine or the instance and went back into the same machine, it didn't seem to work.

00:01:02.000 | So I'm not sure why or if you want to go into that. Okay. Let's have a look. See if we can hear what might have happened.

00:01:18.000 | What happened when you tried to use it?

00:01:28.000 | I muted myself. So what happened was the member and I reinstalled member. And what else was this? And CTAGS again.

00:01:53.000 | And the CTAGS file had changed, but actually didn't have anything in it compared to how it was when it was originally created.

00:02:10.000 | So I don't know what happened. But then when I tried to re-index or recreate the index and that didn't seem to work either.

00:02:34.000 | Well, I mean, one obvious thing we should probably try to get going is being able to see the same, you know, binaries and stuff reinstalled.

00:02:49.000 | Having to reinstall CTAGS every time is certainly not ideal, is it? So let's try and figure out how to do that.

00:02:59.000 | I'll share my screen.

00:03:11.000 | All right. Can you see that okay?

00:03:20.000 | All right. Let's start this one. This is the one we created the other day.

00:03:41.000 | So the only things that are going to still be there next time we come back are things that are in /storage or things that are in /notebooks on the exact machine that we just started.

00:04:02.000 | And to remind you, we can make this more flexible by taking advantage of the pre-run.sh and the .basrc.local scripts that we can put in /storage.

00:04:18.000 | And we can use those to copy, link or move stuff out of those persistent places into our home directory if we want to.

00:04:28.000 | So that's kind of how we've been organizing things.

00:04:34.000 | And so when we do pip install with a --local flag, it installs things into our home directory, into a subfolder called .local. And so that we have moved into /storage and then similar linked back again in our script back to home.

00:04:52.000 | And so now anything that we install with pip install --user will be persistent across all of our sessions.

00:05:00.000 | >> Can I ask a question about that?

00:05:04.000 | So does that mean that if you, in the past, suggested that you can wipe your Mamba directory and use it to reinstall?

00:05:12.000 | Will the fact that you have the sim link to local on the storage drive, will that mean that those pip installed files will be persistent even if you wipe your Mamba drive?

00:05:23.000 | >> Yes, although that thing about wiping was not for paper space, because remember on paper space, the Mamba stuff is installed into the root directory, into the /opt/ or whatever.

00:05:33.000 | It's not in our home directory like usual. So you can't really wipe conda on paper space. On paper space, conda is always wiped clean because you start with a new server.

00:05:44.000 | And so the only stuff left will be stuff in .local.

00:05:50.000 | But, yeah, I don't know why you would, but you could do the same thing on your own machine.

00:05:54.000 | You could pip install --user and have a .local directory that, yeah, would be separate from your Mamba environments and stuff.

00:06:04.000 | >> Okay. I guess it's moved. Yeah, I hadn't thought of it that way. Okay, thanks.

00:06:08.000 | >> Yeah, that's fine. So I'm just having a look. So I think the way we -- yeah, the way we want to install stuff like c tags, which is not a Python thing, is using conda.

00:06:33.000 | And so we can install with Mamba, as we've done before. Now, by default, that's going to install it into the /opt/etc, which we will lose.

00:06:45.000 | But what we can do is Mamba comes with a flag we can use, which is minus P for prefix.

00:07:03.000 | And if we use the minus P flag, it will install stuff into a different directory.

00:07:11.000 | And so we should be able to use that to install stuff in a way that we can -- that we can reuse it.

00:07:20.000 | And I think this is going to be a good exercise in understanding paths and aliases and -- as with sim links, I should say, and all that kind of thing.

00:07:32.000 | So I think -- yeah, I think this will be an exercise worth doing. So I'm just loading up JupyterLab.

00:07:43.000 | Here we are. And so this is the same server, notebook server, that I used last time.

00:07:51.000 | And since /notebooks is persistent, that fastai folder that I -- sorry, I didn't sing with, that I get cloned is still there.

00:07:59.000 | All right. So if we get a terminal up -- and we could probably take better advantage of our space.

00:08:14.000 | So -- good.

00:08:19.000 | All right. And then Apple B or control P, B shows and hides that sidebar, which is quite useful.

00:08:29.000 | So what we could do is we could create a -- so I just typed CD, right, which is moving me into my home directory.

00:08:37.000 | And so we could create something to put other Conda stuff in here. We'll call it Conda.

00:08:45.000 | And I believe we could go member and then say a prefix.

00:08:51.000 | So our prefix is going to be our home directory's Conda directory that we just created.

00:08:56.000 | Install universal -C tags.

00:09:03.000 | So -- okay. So -- all right. So we also don't have member installed, which is a good point.

00:09:08.000 | So we'd like that to also be permanently available.

00:09:13.000 | So let's start by saying Conda install -P. Conda -P install member.

00:09:30.000 | Okay. That's probably going to be after the install command, is my guess.

00:09:47.000 | So while that's going on -- oh. Okay. And then because they're not using member forge, that means that the Conda forge channel is not available by default either.

00:10:04.000 | So if you use member forge, then it kind of assumes that you're always going to say -- that you're always saying -C Conda forge.

00:10:11.000 | But since we're -- and I think the plan is that paper space is going to hopefully switch to using member forge soon.

00:10:18.000 | But for now, we'll have to say -C Conda forge.

00:10:25.000 | All right. So while that's going on --

00:10:34.000 | let's also get our fastbook here, which actually makes me think about something.

00:10:46.000 | I just tried to reuse a command that should have been in my history, and it wasn't there.

00:10:53.000 | And I think the reason it wasn't there is that we -- oh, here we go. Let's say yes over here.

00:11:04.000 | Wait. Why is this installing Python, even? Yikes.

00:11:11.000 | Oh, okay. That's annoying. So member -- okay. So let's try to install it into the tilde/conda directory.

00:11:23.000 | And it's saying, oh, that directory doesn't even have Python in it. So I want to install Python and everything into there.

00:11:32.000 | That's annoying. Let's say no.

00:11:44.000 | Okay. I do have another plan up my sleeve, which is there is a version of member that is fully self-contained. Everything statically compiled into it called micro member.

00:12:02.000 | So we're just experimenting a bit here. Once we get this working, we will make this all easier.

00:12:10.000 | So instead of using member, we say /storage. So we can put this somewhere else in a moment. So let's see if that would work.

00:12:32.000 | Okay. That's also not working. Hmm. This is quite tricky, isn't it?

00:12:52.000 | So maybe we need to figure out what a minimum environment is because we don't want to use up lots of our storage directory with junk that we don't really need.

00:13:03.000 | So what I'm going to do is I'm going to install all this junk and then I'm going to see what that actually installs.

00:13:18.000 | And then we'll try to get rid of as much of that junk as possible.

00:13:24.000 | So I think that's my plan.

00:13:50.000 | Yeah, that's roughly what I was getting as well.

00:13:53.000 | So, okay, no channel specified. I think I might be getting closer though on this condor forge.

00:14:12.000 | Okay, that seems to be doing something.

00:14:18.000 | So we might be on the way to finding a solution.

00:14:25.000 | Okay, let's see what this condor directory. Well, it's got a lot of stuff in it.

00:14:30.000 | So I want to find, I don't want to like have stuff that takes up storage space and I don't need.

00:14:36.000 | So the first thing to find out is how much storage space does things take up?

00:14:39.000 | So D stands for disk usage, that shows you how much storage space things up minus S just gives you a summary per directory and H is as usual human eyes.

00:14:49.000 | So like show me megabytes and stuff like that. So it's easy to read. So this is going to show me the size of every directory.

00:14:57.000 | And if I want it for the whole directory, okay, so I've got 276 meg, which isn't terrible.

00:15:03.000 | But let's see. It is 38 meg, lib is 203 meg. So it's mainly in lib, right? So what I could do then is go lib slash to see which ones.

00:15:15.000 | Now that's quite hard to read. So when things are kind of hard to read like that, I want to find everything that's like over a megabyte or more.

00:15:21.000 | So in the terminal, one of the really nice things you can do is you can use the vertical bar called pipe.

00:15:28.000 | And what pipe does is it takes the output of the previous command and it sends it to another command.

00:15:34.000 | And so the thing that you're most likely to pipe to is grep, and grep searches and only prints things that match a pattern.

00:15:43.000 | I just want to find those things that are a megabyte or more. So there's a capital M for megabyte, and then there's going to be a space after it.

00:15:50.000 | Sort of just search for an M followed by a space.

00:15:56.000 | No, that didn't work. Maybe that's not a space and it's actually a tab. Yes. Okay.

00:16:02.000 | So here's all the stuff that's over a megabyte in lib.

00:16:11.000 | And we could do the same thing for the directories inside Conda.

00:16:21.000 | There they are.

00:16:24.000 | So of that 203 meg, it's mainly Python, which I'm sure we're not going to need.

00:16:32.000 | So we could RM minus RF Python and see what else we got here.

00:16:51.000 | Python.

00:17:12.000 | Now got 161 meg, still quite a bit.

00:17:29.000 | Still mainly in lib.

00:17:40.000 | So this giant thing here, lib ICU data.

00:17:44.000 | I don't know if we need that or not. So what we could do is just move it out of the way.

00:17:54.000 | So 71 and maybe just move it out of the way.

00:18:01.000 | And then let's see if BAMBA works.

00:18:06.000 | Oh, we have an install. So let's see if we can. Oh, so BAMBA. Okay.

00:18:11.000 | So we've installed C tags and MAMBA, but I can't run them.

00:18:20.000 | And oh, that's interesting. I can run the C tags.

00:18:24.000 | Why is that?

00:18:27.000 | Oh, okay. So it looks like paper space comes with an old copy of C tags.

00:18:37.000 | And this might be the issue that you were having with not being able to use it.

00:18:46.000 | So normally dash dash version should tell you the version of things. There we go.

00:18:54.000 | C tags from Emacs. Wow. Okay. So my guess is it's not going to work the same way as what we're used to.

00:19:00.000 | And that's probably why some people were having problems.

00:19:03.000 | So this is not finding the one we just installed and we're not finding MAMBA at all.

00:19:07.000 | And so to remind you, the reason for this is that the way that the computer finds things to run is it searches in our path.

00:19:16.000 | And the stuff that we installed is in the Tilda slash Conda slash bin directory is MAMBA.

00:19:24.000 | And here's our newer version of C tags. So we could run things manually by typing bin slash C tags minus minus version.

00:19:35.000 | Okay, so it doesn't need that. Let's move that back in.

00:19:39.000 | I see you data. So that needs to go into it.

00:19:45.000 | Okay. And yeah, so here's a current version.

00:19:50.000 | Five point nine compiled. Well, two days ago. So to make it so that we'll be able to see stuff that's in our Conda directory, we need to ensure that it's in the path.

00:20:04.000 | So to make sure that things are in the path when we run a bash terminal, we have to put them into our dot VIM dot bash RC dot local file.

00:20:16.000 | So that's going to be.

00:20:23.000 | Okay, so it looks like I don't currently have one.

00:20:28.000 | So let's create one dot bash RC dot local.

00:20:34.000 | All right.

00:20:36.000 | So.

00:20:39.000 | This is a file that needs to be run with bash.

00:20:43.000 | And what we're going to do is we're going to export sets of variable.

00:20:49.000 | And so the variable we need to set is path.

00:20:52.000 | And we want to set it equal to the Conda VIM directory in our home directory.

00:21:01.000 | And then we want to also have the contents of the existing path variable.

00:21:05.000 | So you have to put dollar path to say the contents of a variable.

00:21:12.000 | Okay, so now if we create a new terminal we can test with that worked by printing the contents of dollar path echo is how you print things.

00:21:24.000 | Okay.

00:21:26.000 | Let's see. So that did not work.

00:21:29.000 | Let's try running it manually and see if it works.

00:21:33.000 | So source is a way of running a script and storing all the variables in this shell.

00:21:41.000 | So if we say slash storage slash bash RC dot local.

00:21:51.000 | It is Rick kind of been okay so why was it not working.

00:22:02.000 | Oh, it's not bash RC dot local it's dot bash dot local. That's why.

00:22:07.000 | So we could move.

00:22:12.000 | So I don't have to type up bash RC dot local I can just type exclamation mark dollar because that means the last token from the last line, which in this case is dot bash RC dot local and move it to bash dot local.

00:22:27.000 | Let's try again.

00:22:39.000 | There it is. Okay, so now if I run C text.

00:22:46.000 | I've got the right version, and I should get a right Matt run member.

00:22:58.000 | Okay, so mamba doesn't like being run so that's fine.

00:23:02.000 | What we can do is we can instead use this micro member thing instead so let's move micro member into condor bin.

00:23:17.000 | And so you can get micro member.

00:23:40.000 | You can download it from here.

00:23:50.000 | Alright, so let's see how we're doing.

00:24:00.000 | Let's get rid of member.

00:24:08.000 | And let's see how much space we're now taking up.

00:24:14.000 | Not bad. Right, 175 megs so we've now got a, you know, place that we can install software like C tags into, and it will work. So, to make sure that that'll work next time we need to move that into our slash storage.

00:24:41.000 | And then of course we'll need to similar kit back again. So, let's edit our pre run but sh file.

00:24:56.000 | You definitely don't have to remove anything so it won't be there so we just sim link slash storage slash Honda back to the home directory.

00:25:13.000 | Okay.

00:25:18.000 | So there's one other thing I'd like which is my dot bash history file it's really nice to come in each time and have the same bash history file on a machine.

00:25:29.000 | In my opinion.

00:25:32.000 | So, let's move that also into slash storage, because that means that our control are an app arrow and stuff like that are going to always work.

00:25:56.000 | So that goes into slash storage.

00:26:04.000 | And so let's do that as well.

00:26:08.000 | So how am I going to do this in VM I want to copy two lines and paste them. So to copy two lines.

00:26:14.000 | To copy a line is shift y so to copy two lines I press to shift y to copy two lines, and then shift P to paste those two lines above.

00:26:24.000 | Okay. And then I can press shift W to move a word forwards and shift C to change the rest of the line so that deletes and then it puts it in insert mode, and I can type bash underscore history.

00:26:42.000 | Now here's where it gets fun is, I want to do exactly the same thing for this line is replace this the rest of this line is dot bash history.

00:26:52.000 | So to repeat myself it is press dot.

00:26:56.000 | Isn't that fun. So, all right, let's, and that's not a directory so it doesn't need to be recursive.

00:27:15.000 | Okay, so let's try this out, shall we, by spinning up a new machine.

00:27:31.000 | I guess I mean I what I mean is spin up an existing machine, the instance of it.

00:27:46.000 | So the reason we're spending quite a bit of time to kind of set up our paper space environment is like, because I think it's good.

00:27:55.000 | There are practical examples of using VM and using links and using scripts, you know, which hopefully you'll be to reuse these ideas, lots of times.

00:28:05.000 | So let's start that one.

00:28:09.000 | Okay, so I'm just going to start up another machine and see if that all works.

00:28:15.000 | Let me wait for it. So yeah, I have a feeling now, once you've installed universal C tags properly and it's in your path, just a case of checking the version.

00:28:29.000 | So let's create a new shell, because that one's.

00:28:34.000 | App arrow. Oh, this is still not giving me the right version that's interesting.

00:28:46.000 | Add up to the shell to the path.

00:28:51.000 | Route Conda bin. That's interesting.

00:28:58.000 | Conda in C tags.

00:29:03.000 | Oh, okay, it's not similarly back again because we didn't run that storage script because this is not a new machine. So for now I'm just going to run the storage script manually.

00:29:18.000 | Okay, and.

00:29:27.000 | Okay, still not right.

00:29:31.000 | Oh, then the thing that puts in the path is meant to be the dot bash dot local, right, which should have run.

00:29:41.000 | Just create a new shell just to see if something weird happened.

00:29:48.000 | Okay, so that's in route Conda bin. So now if I type C tags minus minus version with control R to find it quickly. Okay, it is working.

00:29:58.000 | So it's just a case of getting that running it again. Okay, so here's our new machine, which is now finished starting.

00:30:09.000 | So let's see if the same thing works on a brand new machine.

00:30:21.000 | Where is this take a long time to start open up to put a lab the first time for some reason.

00:30:42.000 | On each machine.

00:30:49.000 | It comes. Okay, so fingers crossed.

00:30:56.000 | See tags minus minus version.

00:31:00.000 | There it is. And in theory.

00:31:07.000 | Yeah, we've also got our history so our history is actually saved between machines, which is quite nice, right.

00:31:15.000 | I love it. So if I wanted to echo path again, I could type control our capital PA and it'll find it from our history.

00:31:21.000 | Even though we're in a totally separate machine.

00:31:26.000 | One thing weird is we're getting the path here is repeating itself.

00:31:31.000 | Not quite sure why that's happening.

00:31:44.000 | It doesn't really matter too much, to be honest, but I'm just curious about why /opt/conda/conda/vn is appearing.

00:31:53.000 | But it's not appearing twice.

00:31:56.000 | User. Oh, no, they're all unique. Don't mind me. Okay, I was imagining things.

00:32:02.000 | So on this computer, I don't really want all this stuff in my notebooks either. So let me RM minus RF everything.

00:32:16.000 | That's a really dangerous command.

00:32:19.000 | Oh, Hamilton. Hi, Hamilton.

00:32:21.000 | Yeah, I'm so sorry for being late. I forgot.

00:32:24.000 | I didn't even tell people you're coming so they don't know you're late so you could have stayed silent and not known.

00:32:31.000 | Oh, hopefully it's going to be joining most to all of our sessions from now. So do you want to quickly introduce yourself, Hamel?

00:32:38.000 | Yeah, yeah, I'm Hamel.

00:32:40.000 | I have, you know, work on fast AI a lot. I especially like contributing to all the dev tools, like nb dev and fast core and stuff like that.

00:32:51.000 | And in like two other tools that help people automate what people do.

00:32:58.000 | So yeah, Hamilton Hamels got background as a machine learning researcher and developer and also quite a bit of stuff with ML ops.

00:33:11.000 | And also training as a lawyer. So, you know, if anybody needs help with the law, he's the guy.

00:33:20.000 | That's what I tell people about that, Jeremy.

00:33:25.000 | All right. Nice to see you. So we're just setting up our paper space environment Hamel.

00:33:32.000 | And we've got to a point now where I just launched a new instance and everything is exactly how I want it straight away, which is really nice.

00:33:42.000 | We've got a way to install persistently Python modules. We can install persistently binaries.

00:33:51.000 | So I think that's a good place to start working through the book because we've got ourselves an environment.

00:33:59.000 | So that is step one. Is there anything else that people felt like we're kind of missing from the environment that they would really want?

00:34:06.000 | Or should we start working on chapter one of the book?

00:34:15.000 | Sounds like we're all happy. Okay, great.

00:34:17.000 | I'll stop this machine because that one's costing me money.

00:34:21.000 | So here we are in slash notebook. So the first thing we're going to need is we're going to need a copy of the book.

00:34:29.000 | So fast book is here so we can copy the SSH GitHub thingy and clone it into slash notebooks.

00:34:49.000 | Actually, wait, we're trying to do this properly, right?

00:34:51.000 | So if we're going to do it properly, what we should actually do is create our own fork of it so that we can make changes and save it back again.

00:35:03.000 | So actually rather so what I'm going to do is I'm going to click fork and that's going to create my own copy in JPHOO.

00:35:12.000 | Okay, so let's get rid of that copy in JPHOO. We can actually do this all from scratch.

00:35:21.000 | Okay, so try following along with these steps.

00:35:26.000 | Make sure it works for you. So I'm just going to delete my copy of this.

00:35:36.000 | JPHOO slash fast book so that we can start from scratch.

00:35:58.000 | Looks like I need to go get my phone, excuse me.

00:36:27.000 | That's annoying.

00:36:42.000 | Normally it lets me use GitHub on my phone.

00:36:48.000 | Alright, I'm just going to do this on my other computer rather than set this all up.

00:37:05.000 | This device option seem like it was promising to me.

00:37:13.000 | This device. Yeah, no, because like this is my new Mac that like, I don't even normally on my Windows machine I use Windows Hello to do feast recognition.

00:37:27.000 | And when I say do it on my phone, it's not doing it with the GitHub app on my phone which is what normally happens but instead my it's on saying on my phone I have to install a USB security key so I don't know.

00:37:39.000 | I could just do it over on my other computer to leave this repository.

00:37:46.000 | Don't worry about that another time.

00:37:49.000 | Okay, my other computer I should be able to use my security key. Yes, because it uses face recognition.

00:37:58.000 | Okay, great.

00:38:00.000 | I love Windows Hello. It's very handy.

00:38:02.000 | Okay, so now, that's all done. So right now. Great. So now I can.

00:38:11.000 | Okay, so now I can create my fork.

00:38:19.000 | Oops.

00:38:22.000 | There we go. So this is going to create a basically a copy of the repository.

00:38:32.000 | But it's a copy that is linked back to the fast as version of the repository.

00:38:38.000 | So, as if, if faster I makes changes, then I can click fetch upstream and it'll copy those changes into our version, which is nice, but it'll also keep our changes as well.

00:38:51.000 | As long as they don't conflict.

00:38:53.000 | So now, rather than cloning faster as version, we will clone our version, and we can save those changes back as we need to.

00:39:03.000 | Okay, so I got to get clone our fork.

00:39:10.000 | You can see here it's got our username, and then the repo name.

00:39:23.000 | I noticed that I'm inside slash notebooks. When I do that,

00:39:30.000 | which means that now we can see it.

00:39:34.000 | And here it is. So let's open chapter one and let's open chapter one and clean

00:39:50.000 | checking.

00:39:53.000 | I remember previously with get clone you've done depth of one to not clone the whole repo. Yeah.

00:40:01.000 | I wondered about doing that this time but because we want to probably save like commit things back to our fork of the repo, I decided not to go with depth one.

00:40:11.000 | This time, because we're not just reading it but we're actually changing it.

00:40:18.000 | I have a question.

00:40:20.000 | I have never done a fork and an emerge.

00:40:25.000 | And I wonder if you can do a sample just to test. If that's okay.

00:40:29.000 | Yeah, let's make a change and copy it back. So, let's just on the way to the master.

00:40:35.000 | Yes, yes, yes.

00:40:38.000 | Okay.

00:40:39.000 | So let's see if this first cell runs first of all.

00:40:50.000 | You'll see at the very top there's a Pippin store fastbook, but it checks if this is a bash line because it starts with exclamation mark and it's checking whether something called slash content exists that only exists on colab.

00:41:02.000 | So that will only run on colab.

00:41:05.000 | So it actually it's importing the fastbook, and then it's going to try to import it.

00:41:10.000 | Okay, great.

00:41:13.000 | And we could try running something.

00:41:15.000 | Okay, great.

00:41:17.000 | So now let's try pushing that back to our repo. So, if we now CD into fastbook.

00:41:27.000 | And we do a get status.

00:41:33.000 | Get status.

00:41:39.000 | It looks like you haven't saved that.

00:41:46.000 | Yeah, I thought I did, but apparently I didn't.

00:41:50.000 | Oh, because it's on a Mac and you have to press Command S and I'm just not used to using a Mac here.

00:41:57.000 | Okay, so it tells us that we've made a change.

00:42:00.000 | So, to remind you the kind of a shortcut for adding something to a commit and giving it a message is get minus a to add it and M to provide a message.

00:42:13.000 | And then you need to say what it was going to do. So I'm going to commit.

00:42:17.000 | And the message has to be in single quotes, so it knows that's all one string test, making a change.

00:42:27.000 | Okay. So here's another thing that we're going to want to add to our.

00:42:43.000 | Here's another thing we're going to want to add to our persistent stuff is who am I.

00:42:50.000 | So let's run the things it says to.

00:43:10.000 | Okay, and then what you'll find is that there's now a config file that's appeared here.

00:43:23.000 | That contains that information. So hopefully you won't be too surprised to hear that what I want to do is move that into slash storage.

00:43:34.000 | And then VIM slash storage.

00:43:38.000 | Pretty run.

00:43:42.000 | And Sim link it in a script.

00:44:01.000 | First time around we'll have to do that manually.

00:44:10.000 | Okay, and so I rerun the commit by typing control out a search backwards and type commit.

00:44:16.000 | Press enter.

00:44:18.000 | Oh, I'm in the wrong place to get back to where I was a CD hyphen.

00:44:25.000 | There we go. Okay, so that's committed it to my local.

00:44:30.000 | There we go. So if I now go back to.

00:44:48.000 | GitHub.

00:44:51.000 | You'll see that this branch is one commit ahead of faster your master.

00:44:54.000 | So I've made a commit that isn't part of faster.

00:45:02.000 | So, you know, by the way, sometimes like a lot of the time you create a fork and you make changes because you just want to make changes to your version.

00:45:10.000 | Sometimes you make a fork and make changes because you actually want to provide those changes back to the original project, you know, you fixed a bug of theirs or added a feature of theirs.

00:45:20.000 | And so if I wanted to send my changes back to fast AI, because I think they're an improvement to the book, then I can say contribute.

00:45:31.000 | Right.

00:45:32.000 | And then that'll say, oh, okay, you can create something called a pull request.

00:45:36.000 | And so the pull request, it's going to show you the changes, which is I've executed some cells I hadn't executed before.

00:45:42.000 | And I've added a cell where the source was one plus one and the result was two.

00:45:46.000 | And then I could create a pull request and that would send something back up to fast AI.

00:45:54.000 | And I'll show you what that would look like.

00:45:55.000 | Obviously, never do this unless you assure you want to, because that's just going to annoy the developers.

00:46:07.000 | So then what happens is that would appear on my copy of fastbook.

00:46:11.000 | It would appear in pull requests.

00:46:13.000 | And so here's an example.

00:46:16.000 | So when you send in a pull request, I'll then be able to see what files you changed using this thing called review and be I'll even be able to see the changes that were made to the notebooks in a graphical view.

00:46:30.000 | So I'll show you what that looks like.

00:46:33.000 | There we go.

00:46:42.000 | And they've provided a description.

00:46:45.000 | And so then if I wanted to add this back to fast AI, I click this button.

00:46:49.000 | So that's what people talk about when they talk about making a pull request.

00:46:53.000 | Anyway, that's not quite what we're doing.

00:46:55.000 | We're just keeping our own copy of it as we discussed.

00:46:59.000 | All right, so

00:47:05.000 | can we have a version of review and be in local machine to see?

00:47:12.000 | Yeah, absolutely. Review and be is a is a startup. They're very, very good.

00:47:16.000 | You can absolutely so you can totally add it to your own repos just by going to review and be.com.

00:47:23.000 | No, I mean, in my local machine.

00:47:27.000 | No, but you can use something quite similar called nb dime,

00:47:33.000 | which is actually so it doesn't really make sense to like have it on a local machine because like you don't pull requests to yourself on your machine.

00:47:42.000 | But what you do want to be able to see is to see differences.

00:47:47.000 | And so, for example, with mine, they've got nb dime installed.

00:47:52.000 | If I let's make a change. So let's delete this cell, save it.

00:47:59.000 | And so it's nb times already installed on paper space. So I can click here and I can see, okay, I've made this change as I've deleted one cell.

00:48:11.000 | All right, so let's try running.

00:48:16.000 | Yeah, it's happening. Okay. Well, this is running.

00:48:22.000 | Let's talk about what's happening here. So the first thing we've got here is a from blah import blah statement.

00:48:31.000 | So let's understand what's happening here. And to do that, let's CD into.

00:48:38.000 | The first day I repo. Okay, so here's the first day I repo and the first day I repo contains a folder.

00:48:47.000 | Called first day.

00:48:50.000 | And that contains the first day I library.

00:48:55.000 | And in the first day I library, there are some also as well as some Python files as some subfolders. Now in this case we were importing stuff from fastai.vision.

00:49:07.000 | And that tells us that there must be a directory called vision, and there is indeed is a directory called vision.

00:49:17.000 | And then in there, finally, we're importing from a module called all and that means that there must be a directory called all sorry a folder file called all pie and here it is all pie.

00:49:35.000 | So what happens when we say from blah import blah is it goes through all the dot separated components except for the last one and treats these as directories.

00:49:46.000 | Right. And so basically what it's doing is it's going to look for a file.

00:49:51.000 | Called.

00:49:54.000 | Called fastai.

00:49:58.000 | Slash vision.

00:50:00.000 | Slash all.

00:50:03.000 | That's how it gets translated.

00:50:06.000 | It's all very mechanical, you know, there's nothing magical weird about it.

00:50:13.000 | And so then if we look at that file.

00:50:18.000 | There it is. Right. And what does this file do. Well this file is just importing things.

00:50:24.000 | And so it's importing something from models basics blah blah blah. Now, when you're inside a library like this, you'll see sometimes it uses dot or dot dot prefixes.

00:50:39.000 | That's going to be replaced with a file called dot dot slash basic stop pie. This will be replaced with a file called dot slash augment pie.

00:50:49.000 | So we should be able to find if I go control split control SP a ug tab.

00:50:56.000 | Yep, there is indeed a augment. So I've got a split plane here. Here's my augment file.

00:51:03.000 | And so when it goes from dot augment import star, what does that do.

00:51:09.000 | If there's a special variable called done to all this when you got underscore underscore blah underscore underscore, we pronounce that dunder blah. So this is done to all.

00:51:20.000 | If there's a special variable called done to all that's a list of all of the symbols that it wants to bring in.

00:51:29.000 | Right. And so here's a list of all the symbols it wants to bring in. So this file, this this command here will bring in, for example, a symbol called rand transform into this file.

00:51:41.000 | Now, this file doesn't have something called done to all.

00:51:44.000 | And so if you don't have a file called done to all and every single thing that is imported or defined in this file will end up being exported from it.

00:51:54.000 | So since this line here imports something called rand transform and there's no dunder all here, that means importing all.py should also import rand transform.

00:52:05.000 | And so that means I should be able to type rand t tab.

00:52:12.000 | And there it is. There is a rand transform. And if I hit shift enter, you can see where it's come from.

00:52:18.000 | So it's come from fast fast.ai vision dot augment dot rand transform, which remember is translated to fast.ai slash vision slash augment slash sorry.

00:52:30.000 | OK, this is actually the name of a method. So this will become fast.ai dot vision fast.ai slash vision slash augment.py.

00:52:38.000 | And then we'll find the rand transform class or symbol inside there.

00:52:42.000 | And so let's find it. So if I click on rand transform.

00:52:47.000 | I'll show you a really nifty trick.

00:52:49.000 | If you select something in VIM and hit the asterisk key, shift eight, it will search for the next place that this word occurs.

00:52:59.000 | So if I press star asterisk, here's the next copy of rand transform.

00:53:05.000 | And so here's the definition of it.

00:53:08.000 | So that's what it does when you say from blah, import blah. If you say import star, then that will import everything that's exported.

00:53:17.000 | So everything in done to all or if there's no done to all, everything that's defined or imported.

00:53:23.000 | So that's why we now have a thing called rand transform available to us.

00:53:29.000 | Yeah, if you ever want to know where something's from, a lot of people are used to not using a wildcard import, so not using star, but instead listing specifically exactly all the symbols they import.

00:53:42.000 | And so somebody does that. You can scroll back to the top of the file and search through it and try to find it.

00:53:48.000 | But you don't have to. It's much easier than indeed with star necessary to simply type the name and press shift enter and you can find out where it's from.

00:53:58.000 | Or to get more info about it, do the same thing with a question mark.

00:54:02.000 | And as you can see, it will give you the signature, the doc string and the full path name of where it's from.

00:54:09.000 | And so this is one way if you want to go and look at the source code for this is I could copy that and type colon SP and paste it.

00:54:21.000 | And now I've got a third split and here's the, here's that file. Okay.

00:54:29.000 | But if I just want the source code for that one thing, I can just put two question marks.

00:54:35.000 | And as you can see, this gives me the source code for that thing.

00:54:41.000 | Okay, so that's what the first line does.

00:54:44.000 | First day I vision or import star. What is the next line do.

00:55:02.000 | So if I type doc on target or enough in a fast AI library, it'll tell me what this one didn't quite work correctly.

00:55:13.000 | It'll tell me all the information about it.

00:55:16.000 | For some reason, this one's not getting the usual documentation I would expect to see. That's okay.

00:55:22.000 | You can always click on source to get a link to where it's defined.

00:55:29.000 | So here it is on hard data.

00:55:34.000 | Or you can click on show in docs.

00:55:43.000 | This one works correctly. That's interesting. So for some reason, the help the documents not working right. I'll fix that.

00:55:49.000 | Yeah, so you can see here's the details here and you can see that there's, you know, it says here, for example, see URLs, you can click on these things to get more information about them.

00:56:03.000 | So there's lots of similar links, as you can see within the documentation.

00:56:09.000 | So entire data downloads and extracts URL by default into sub-directories of tilde slash doc fast AI. So we now know that means you're your home directory.

00:56:22.000 | And it returns a pass.

00:56:25.000 | Okay, so let's see if we can understand what this is doing.

00:56:33.000 | So what I generally like to do.

00:56:36.000 | To understand a cell is to run every single line separately. So the most important thing to know how to run every single line separately.

00:56:44.000 | Is to.

00:56:47.000 | Know that you can press control shift hyphen is control even on a Mac not command control shift hyphen to split into two cells at the cursor.

00:56:58.000 | The step one is to separate this out into separate cells.

00:57:01.000 | Okay.

00:57:03.000 | And so then, you know, run each one and see what happens. So after I run this one, I should be able to look at path.

00:57:12.000 | Okay, so it tells me here that.

00:57:17.000 | It's been stored in storage data, etc. Now that's good news because we know that means it's persistent.

00:57:25.000 | So if I create a new instance, whatever, I'm going to have this same thing now.

00:57:32.000 | Downside is if you have a free or cheap account, you don't have much space and you might not want all that space being taken up.

00:57:40.000 | So let's find out how much space is being taken up. So let's copy that.

00:57:44.000 | Head up over to our terminal.

00:57:49.000 | Get over there.

00:57:50.000 | And so remember, we can type to you -sh.

00:57:55.000 | 789 megs. That's pretty big. So you might not want that to be there.

00:58:02.000 | Which is fine. So you can just move it somewhere else, you know, put it in the home directory or something like that.

00:58:07.000 | So one interesting question is here is like according to the documentation, it was going to extract things by default to subdirectories of tilde slash dot fast AI, but that's not what happened.

00:58:18.000 | Why is that not that happened? Well, it says it's a wraparound fast download dot get so we should probably.

00:58:26.000 | Look up that documentation to find out what's going on.

00:58:33.000 | So here's fast download. And here's fast download dot get.

00:58:40.000 | OK, so this is pretty much what we saw rather than, you know, this is using D dot get it returns a path by default.

00:58:46.000 | It goes into base archive, which by default is this case dot fast download for fast.

00:58:52.000 | I might change it to fast AI.

00:58:55.000 | You can change them by passing fast download.

00:58:59.000 | OK. Oh, look at this. If there's a config file in the base directory, then they will be used for fast download.

00:59:05.000 | Now our base directory for fast AI is dot fast AI.

00:59:08.000 | So let's go and have a look in the dot fast AI directory.

00:59:13.000 | CD tilde slash dot fast AI.

00:59:17.000 | There we go. There is indeed a config dot any.

00:59:22.000 | So you can see that paper space is actually set things up for us so that by default, all of the archives models we create data we download is all going to be put in the persistent storage.

00:59:35.000 | So that's like that's a good thing, unless it's not what you want. Right.

00:59:43.000 | So I would say this is another of these things that we probably want to be able to adjust if we want to.

00:59:50.000 | So how about we move that?

00:59:53.000 | Into slash storage.

01:00:02.000 | And then as per usual.

01:00:06.000 | Simulink it back.

01:00:11.000 | So when we start the machine, I don't know if there's a folder there or not. So what we can do is we can say make them as P basically creates a folder and all of its subfolders, sorry, all its parent folders and doesn't complain if it's not already there.

01:00:28.000 | So I use make them as P to create a dot fast AI.

01:00:34.000 | And we will then.

01:00:38.000 | Remove a dot fast AI.

01:00:46.000 | Oh, now I'll show you a little trick.

01:00:50.000 | I want to fill out the word fast AI without typing the whole thing. If I hit control P, it will fill in the rest of the word P for previous it will fill in the rest of the last word that it can find that starts with those letters.

01:01:05.000 | So I want to remove fast AI slash config dot any.

01:01:11.000 | And then we will link that back again.

01:01:15.000 | So it's going to be in slash storage. So I'll hit control P to fill out slash storage and it's going to be called config dot any. So I'll press control P and actually, if you then hit control X, control P, it keeps filling in.

01:01:30.000 | The rest.

01:01:35.000 | All right, so now we're going to have.

01:01:38.000 | Config any file.

01:01:43.000 | Oh, except I don't want to put it in the home directory. I want to put it in the dot fast AI directory.

01:01:54.000 | And so we can test that copying it.

01:02:03.000 | Paste it.

01:02:06.000 | There we go. Okay, so we've now got a consistent config dot any. I think in my case.

01:02:13.000 | What I want to do, and this might not be bad for most people, is I probably don't want the archives to be stored in my storage directory because I'm not going to need them again once it's been on archived.

01:02:28.000 | And so that's CD to slash storage slash archive.

01:02:34.000 | Yeah, I don't want this there. So let's just remove it.

01:02:38.000 | Cool. Okay, so.

01:02:44.000 | This is now going to put by default stuff that I download will be in my storage or better use it anywhere. But, you know, I can always move it somewhere else if I want to.

01:02:57.000 | Okay.

01:02:59.000 | The next two lines kind of go together.

01:03:04.000 | Which is I've got to use image data loaders. So again, you know, before you use something, it's good to understand what it is.

01:03:15.000 | So.

01:03:26.000 | Show in docs.

01:03:30.000 | All right, image data loaders is a wrapper around several data letters with methods for computer vision problems.

01:03:39.000 | And what you're going to be using as one of the factory methods. So they're the things underneath.

01:03:44.000 | And all these factory methods except various things. So this tells me that there are various different ways of creating image data loaders, and they've got a consistent API.

01:03:54.000 | So that's good.

01:03:56.000 | In our case.

01:04:00.000 | We've got.

01:04:02.000 | If I hit shift tab, I can find out what the parameters are.

01:04:06.000 | So I've got the path.

01:04:08.000 | I've got a list of file names.

01:04:12.000 | I've got a labeling. Let's see. Oh, then I've got valid percent and seed.

01:04:18.000 | The valid percent and seed. So what are those mean?

01:04:24.000 | So we're using.

01:04:27.000 | valid percent is passed to image data loaders from path.

01:04:32.000 | Okay, so quite often, you know, we'll take a argument and just pass it to something else.

01:04:36.000 | So I have to click on here to find out what that does.

01:04:39.000 | Here we are. Validation set. Validation set is a random subset of size validation percent.

01:04:44.000 | Okay, no worries. Optionally created with seed reproducibility.

01:04:51.000 | Cool. And it's got a labeling function.

01:04:58.000 | So.

01:05:10.000 | A function that receives a string, which is a file name and outputs a label.

01:05:13.000 | Okay, so let's just have a look at some of these things. So the list of file names.

01:05:18.000 | So remember the second argument with a list of file names is this.

01:05:23.000 | So let's pull this out and create something called files.

01:05:28.000 | So if I press A for append above, I'll type files equals pasted here.

01:05:35.000 | And let's look at that.

01:05:38.000 | Okay, so files is a list of 7,390 items and it contains various paths in it.

01:05:56.000 | I'm going to remind myself that if you go path.base or base path, I'll show you what I'm doing in the tick.

01:06:08.000 | Let's get rid of all this prefix copy and paste.

01:06:20.000 | There we go. Yeah, so if you create a special variable called path base path,

01:06:24.000 | then it will delete that from the start of everything it prints out.

01:06:29.000 | So it just shows you the relative to here. It's a bit easier to read.

01:06:34.000 | So the 7,390 things with images slash blah, images slash blah.

01:06:40.000 | And it looks like it's going to be the name of the breed and then an underscore and then some consecutive number.

01:06:49.000 | Now, this looks a little bit different to what you might be used to seeing.

01:06:54.000 | Normally, if you look at something like a list, you expect it to look something more like this, right?

01:07:05.000 | This is what lists look like in Python.

01:07:08.000 | So to find out why that looks different, we can check out what the type of it is by typing type.

01:07:15.000 | And it turns out it's not a list. It's something of type capital L.

01:07:21.000 | That's a special kind of list, which has lots of convenience functions in it.

01:07:32.000 | We could just use question mark to find out a bit more about it.

01:07:37.000 | So here's the definition of where it's coming from.

01:07:39.000 | Here's the doc string from it, or we could type help L and not help doc.

01:07:51.000 | So help is useful as well.

01:07:54.000 | That's a built into Python.

01:07:55.000 | It shows you a list of all the stuff it can do.

01:07:58.000 | Actually, maybe that's useful.

01:08:00.000 | So as you can see, it's got lots of functionality in addition to everything that a list does.

01:08:05.000 | Or we could type doc showing docs.

01:08:11.000 | So you can see this stuff's not just for fast AI libraries.

01:08:13.000 | It's for all libraries created by fast AI look pretty similar.

01:08:18.000 | And so here's lots of information about L.

01:08:25.000 | Let's learn how to read this documentation, and then I think we'll stop.

01:08:31.000 | So L is like a list.

01:08:33.000 | Okay, so a list in Python is something that you create like this.

01:08:42.000 | Square brackets.

01:08:44.000 | And you can print them, for example, or you can index into them, or you can select multiple things from them.

01:08:53.000 | Everything up to, but not including element two.

01:08:57.000 | That's a list.

01:08:58.000 | Okay, and L is very similar.

01:09:04.000 | But it doesn't have any special syntax like Python does.

01:09:09.000 | You have to create it like this, but it basically looks much the same.

01:09:19.000 | As you can see.

01:09:23.000 | But it has quite a lot more functionality than a list.

01:09:27.000 | So I don't use a normal Python list that often, because, like, why use something which is less useful.

01:09:36.000 | So let's see, okay.

01:09:37.000 | It behaves like a list, so we know what a list looks like.

01:09:39.000 | So a drop-in replacement for a list.

01:09:41.000 | So when something's described as a drop-in replacement, anywhere that uses a list, you should be able to use this as well.

01:09:46.000 | So it's got a super set of the functionality.

01:09:50.000 | It's like NumPy.

01:09:52.000 | It supports advanced indexing.

01:09:55.000 | Okay, so what that means is that you should be able to select multiple things from a list at once.

01:10:04.000 | I'm just trying to remember exactly how we do that.

01:10:07.000 | I think you can go like this.

01:10:12.000 | Yes, okay, so I can select the zero thing and the second thing and one go.

01:10:16.000 | So that's an example.

01:10:18.000 | And as I mentioned, this is similar to what NumPy can do.

01:10:21.000 | So NumPy has a thing called an array.

01:10:35.000 | And so in NumPy, you can do the same kind of thing.

01:10:41.000 | But a regular list, remember, A is our regular list.

01:10:48.000 | Can't do that.

01:10:50.000 | So you can kind of think of a capital L object as being a bit like a hybrid between a NumPy array and a Python list.

01:11:02.000 | You could try to use a NumPy array for things that you would otherwise use an L for.

01:11:08.000 | But the problem is that a NumPy array kind of expects everything to be of the same type.

01:11:16.000 | So you have to be a bit careful.

01:11:24.000 | Sometimes it might do it for us. No, sometimes it can actually handle it for us.

01:11:27.000 | It's going to put them into something called an object.

01:11:30.000 | So, you know, I mean, actually, now I think about it, you probably could use an array quite often, but they do behave differently.

01:11:35.000 | Actually, I'll show you an example. So if we go B plus Hello, then that's what that does in an L.

01:11:52.000 | It adds Hello to the list.

01:11:54.000 | So notice that L's show you how many items are in it.

01:11:58.000 | And it also, by default, doesn't show you all of them, but it puts dot, dot, dot, which is both very convenient things.

01:12:04.000 | A Python list.

01:12:13.000 | I used control by mistake.

01:12:16.000 | Works the same way.

01:12:18.000 | A NumPy array.

01:12:24.000 | Doesn't work the same way.

01:12:27.000 | And that's because NumPy is designed mainly for math.

01:12:31.000 | So if we make all of these into numbers, then a NumPy array, we could do this.

01:12:42.000 | It adds element wise.

01:12:45.000 | OK, so, you know, if you're kind of trying to go deep and fully understand each line of code, this is the kind of experimentation that you can do.

01:12:57.000 | And hopefully what you can see here is that all of the information you need to fully understand all of these things is available to you in the documentation, which you can link to directly using the dot command and by experimenting.

01:13:16.000 | But it is like initially, it's a lot to learn.

01:13:22.000 | But the nice thing is that the things you're learning are very reusable, right, because all this stuff is used all over the place.

01:13:31.000 | All right, so I'm going to wrap it up. Does anybody have any questions or comments?

01:13:37.000 | This has been a bit less interactive than usual. So I apologize if that's because I'm talking too much, but I definitely want to hear your thoughts or questions.

01:13:53.000 | Nothing? That's fine. Thank you. Is that because it's too easy or too hard or totally obvious or

01:14:05.000 | I want to go away and think about it.

01:14:10.000 | For me, it was very good, Jeremy. OK, great. Right. Now I think it was I think it was perfect for me. Oh, awesome. Cool. Yes. OK.

01:14:19.000 | I haven't seen somebody called Jess in a stream before. Are you a new person here? I've just not noticed you.

01:14:26.000 | A new old person who just saw it pop up in the forum.

01:14:31.000 | Sure. How did I how did it not pop up for me sooner? But thank you for getting caught up. Where are you joining? Yeah, no problem.

01:14:39.000 | I am in the Seattle area of USA. Fantastic. Well, that's that's great. I think.

01:14:48.000 | Yeah. OK, well, that's a good time to finish. So hope to see you all tomorrow.

01:14:53.000 | See ya.

01:14:55.000 | Awesome. See everybody. Bye.

Live coding 6

Chapters