Welcome to the final lesson of this section of 2019. Although I guess it depends on the videos, what order we end up doing the extra ones. This is the final one that we're recording live here in San Francisco. Anyway, lesson 14. Lesson two of our special Swift episodes. This is what we'll be covering today.

I won't read through it all, but basically we're going to be filling in the gap between matrix multiplication and training ImageNet with all the tricks. And along the way, we're going to be seeing a bunch of interesting Swift features and actually seeing how they make our code cleaner, safer, faster.

I want to do a special shout out to a couple of our San Francisco study group members who have been particularly helpful over the last couple of weeks since I know nothing about Swift. It's been nice to have some folks who do, such as Alexis, who has been responsible actually for quite some of the most exciting material you're going to see today.

And he is the CTO at Topology Eyewear. So if you need glasses, you should definitely go there and get algorithmically designed glasses, literally. So that's pretty cool. So thanks, Alexis, for your help. And thanks also to Pedro, who has almost single-handedly created this fantastic package cache that we have so that in your Jupyter Notebooks, you can import all the other modules that we're using and other exported modules from the Notebooks, and it doesn't have to recompile at all.

And so that's really thanks to Pedro. And I actually am a customer of his as well, or at least I was when I used an iPhone. He's the developer of Camera Plus, which is the most popular camera application on the App Store, literally. And back when I used an iPhone, I loved that program.

So I'm sure version two is even better, but I haven't tried version two. So you can use his camera while looking through your Topology Eyewear glasses. All right. So thanks to both of you. And where we left off last week was that I made a grand claim -- well, I pointed out a couple of things.

I pointed out through this fantastic Halide video that actually running low-level kind of CUDA-kernel-y stuff fast is actually much harder than just running a bunch of for loops in order. And I showed you some stuff based on Halide, which showed here some ways that you can write it fast, and here's some ways you could do it quickly.

And then I made the bold claim that being able to do this on the GPU through Swift is where we're heading. And so to find out how that's going to happen, let's hear it directly from Chris. >> Sure. Thanks, Jeremy. So we will briefly talk about this. So we went through this video, and the author of Halide gave a great talk about how in image processing kernels, there's actually a lot of different ways to get the computer to run this, and they all have very different performance characteristics, and it's really hard to take even a two-dimensional blur and make it go fast.

But we're doing something even harder. We're not talking about two-dimensional images, we're talking about 5D matrices and tensors and lots of different operations that are composed together, and hundreds or thousands of ops, and trying to make that all go fast is really, really, really, really hard. So if you wanted to do that, what you'd do is you'd write a whole new compiler to do this, and it would take years and years of time.

But fortunately, there's a great team at Google called the XLA team that has done all this for us. And so what XLA is, is it's exactly one of those things. It's something that takes in this graph of tensor operations, so things like convolutions and matmoles and adds and things like that.

It does low-level optimizations to allocate buffers, to take these different kernels and fuse them together, and then it generates really high-performance code that runs on things like CPUs, GPUs, or TPUs, which are crazy-fast high-performance accelerators that Google has. And so XLA does all this stuff for us now, which is really exciting.

And if you take the running bachelor example that we left off with, and we were talking about, this is the graph that XLA will generate for you. And this is generated from Swift code, actually. And so you can see here what these darker boxes are, is they're fusion nodes, where it's taken a whole bunch of different operations, pushed them together, gotten rid of memory transfers, pushed all the loops together.

And the cool thing about this is, this is all existing shipping technology that TensorFlow has now. There's a big question, though, and a big gotcha, which is, this only works if you have graphs. And with TensorFlow 1, that was pretty straightforward, because TensorFlow 1 was all about graphs. Jeremy talked about the shipping, shipping, shipping, ship, ship, ship, shipping thingy, ship, ship, shipping, ship, ship, ship, I don't know.

My recursion's wrong. And so with TensorFlow 1, it was really natural. With TensorFlow 2, with PyTorch, there's a bigger problem, which is, with eager mode, you don't have graphs. That's the whole point, is you want to have step at a time, you run one op at a time, and so you don't get the notion of these things.

So what the entire world has figured out is that there's two basic approaches of getting graphs from eager mode. There's tracing, and there's different theories on tracing. There's staging and taking code and turning it into a graph algorithmically. And PyTorch and TensorFlow both have similar but different approaches to both of these different things.

The problem with these things is they all have really weird side effects, and they're very difficult to reason about. And so if Swift for TensorFlow is an airplane, we've taken off, and we're just coming off the runway, but we're still building all this stuff into Swift for TensorFlow as the plane is flying.

And so we don't actually have this today. The team was working on the demo, and it just didn't come together today. But this is really cool. And so one of the problems with tracing, for example, is that in PyTorch or in TensorFlow Python, when you trace, if you have control flow in your model, it will unroll the entire control flow.

And so if you have an RNN, for example, it will unroll the entire RNN and make one gigantic thing. And some control flow you want to ignore, some control flow you want to keep in the graph. And so having more control over that is something that we think is really important.

So, Chris, this nearly there is at the end of April. This video will be out somewhere around mid to late June. I suspect it will be up and running by then, and if it's not, you will personally go to the house of the person watching the video and fix it for them.

So here's the deal. In two, three months, so that's July, look on the TensorFlow main page. There should be a co-lab demo showing this. So we'll see how the future -- >> And there should be a notebook in the hair brain repo that will be called batch norm or something.

And we'll have an XLA version of this running. >> And so Swift also has this thing called graph program extraction. The basic idea here is where autograph and torch script are doing these things where they're kind of like Python but kind of not, and Jeremy was talking before about how you had a comment in the wrong place and torch script will fall over and it's not -- it kind of looks like Python but really, really is not.

With Swift, we have a compiled, reasonable language, and so we could just use compiler techniques to form a graph, pull it out for you. And so a lot of things that are very magic and very weird are just very natural and plug into the system. So I'm very excited about where all this comes.

But for right now, this doesn't exist. The airplane is being built. So one last thing that doesn't exist, because Jeremy wanted to talk about this, he's very excited, is there's this question about what does MLIR relate to XLA, what is all this stuff going on, what does this make sense for TensorFlow?

And the way I look at this is XLA is really good if you have -- if you want high performance with these common operators like matrix multiplication, convolution, things like that. These operators can be combined in lots of different ways. And so these are the primitives that a lot of deep learning is built out of.

And XLA is really awesome for high performance, particularly weird accelerators. But there's a catch with this, because one of the things that power deep learning is the ability to innovate in many of these ways. And so depth-wise convolutions came out, and suddenly with many fewer parameters, you can get really good accuracy wins, and you couldn't do that if you just had convolution.

Yeah. And like on the other hand, like depth-wise convolutions are a specific case of grouped convolutions. And the reason we haven't been talking about grouped convolutions in class is that so far no one's really got them running quickly. And so there's this whole thing that like somebody wrote a paper about three years ago, which basically says, hey, here's a way to get all the benefit of convolutions, but much, much faster.

And we're still -- you know, the practical deep learning for coders course still doesn't teach them, because they're still not practical, because no one's got them running quickly yet. And so we've been talking about this whole course. The goal with this whole platform is to make it an infinitely hackable platform.

And so if it's infinitely hackable down in convolution, or give up all performance around a CPU, well, that's not good enough. And so what MLIR is about is there's multiple different aspects of the project, but I think one Jeremy's most excited about is, what about custom ops, right? How can we make it so you don't bottom out at matmul in convolution, and so you get that hackability to invent the next great convolution, right?

So the cool thing about this is that this is a solved problem. The problem is all the problems -- all the solutions are in these weird systems that don't talk to each other, and they don't work well together, and they're solving different slices of it. So Halide, for example, is a really awesome system if you're looking for 2D image processing algorithms, right?

That doesn't really help us. Other people have built systems on top of Halide to try to adapt it, and things like that. But this is really not a perfect solution. There's other solutions, so PlatML was recently acquired by Intel, and they have a lot of really cool compiler technology that is kind of in their little space.

TVM is a really exciting project, also building on Halide, pulling it together with its own secret sauce of different things. And it's not just the compiler technology. It's also in each of these cases they've built some kind of domain-specific language to make it easier for you, the data scientist, to write what you want in a quick and easy way.

Right. And so -- and often what happens here is that each of these plug into the deep learning frameworks in different ways, right? And so what you end up having to do is you end up in a mode of saying, TVM's really good for this set of stuff. And Tensor Comprehensions, which is another cool research project, is good at these kinds of things.

And so I have to pick and choose the framework I want to use based on which one they happen to build into, which is not very -- >> And again, we don't teach this in practical deep learning for coders because it's not practical yet. You know, these things are generally research-quality code.

They generally don't integrate with things like PyTorch. They generally require lots of complex build steps. >> The compile time is often really slow. They work really great on the algorithm and the paper, but they kind of fall apart on things that aren't. All those kinds of problems. So our goal and our vision here with TensorFlow but with Swiffer TensorFlow also is to make it so that you can express things at the highest level of abstraction you can.

So if you have a batch norm layer, totally go for that batch norm layer. If that's what you want, use it, and you're good. If you want to implement your own running batch norm, you can do that in terms of mat mules and adds and things like that, fine.

If you want to sync down further, you can go down to one of these systems. If you want to go down further, you can write assembly code for your accelerator if that's the thing you're into. But you should be able to get all the way down and pick that level of abstraction that allows you to do what you want to do.

And so I just want to give Tensor comprehensions as one random example of how cool this can be. So this is taken straight out of their paper. This is not integrated. But Tensor comprehensions gives you what is basically like Einstein notation on total steroids. It's like insome. Yes, good point.

It's like insome, but taken to a crazy extreme level. And what Tensor comprehensions is doing is you write this very simple, this very simple code. It's admittedly kind of weird, and it has magic, and the syntax isn't the important thing. But you write pretty simple code, and then it does all this really hardcore compiler stuff.

So it starts out with your code, it then fuses the different loops, because these two things expand out to loops. It does inference on what are the ranges for all the loops and what the variables that you're indexing into the arrays do. Then fuse and tile these things. Fuse, tile, then sync the code to make it so the inner loops can be vectorized.

This is actually a particularly interesting example, because this thing here, gem, is a generalized matrix-matrix product. This is actually the thing on which large amounts of deep learning and linear algebra and stuff is based on. So a lot of the stuff we write ends up calling a gem. And the fact that you can write this thing into lines of code, if you look inside most linear algebra libraries, there will be hundreds or thousands of lines of code to implement something like this.

So the fact that you can do this so concisely is super cool. And so the idea that then we could do nice little tweaks on convolutions or whatever in similar amounts of code is something that I get very excited about. Yeah. Me too. And the other thing to consider with this is that, again, generating really good code for this is hard.

But once you make it so that you separate out the code that gets compiled from the algorithms that get applied to it, now you can do search over those algorithms. Now you can apply machine learning to the compiler itself. And now you can do some really cool things that open up new doors.

So I mean, that's actually really interesting because in the world of databases, which is a lot more mature than the world of deep learning, this is how it works, right? You have a DSL, normally called SQL, where you express what you want, not how to get there. And then there's a thing called a query analyzer or query compiler or query optimizer that figures out the best way to get there.

And it'll do crazy stuff like genetic algorithms and all kinds of heuristics. And so like what we're seeing here is we'll be able to do that for deep learning, our own DSLs and our own optimizers, not deep learning optimizers, but more like database optimizers. Yeah. So it's going to be really exciting.

So we're building all this. The ML Air part of this is longer time horizon. This is not going to be done by the time this video comes out. But this is all stuff that's getting built, and it's all open source, and it's super exciting. So to overall summarize all this TensorFlow infrastructure stuff, so TensorFlow is deeply investing in the fundamental parts of the system.

This includes the compiler stuff, also the runtime, op dispatch, the kernels themselves. There's tons and tons and tons of stuff, and it's all super exciting. So let's stop talking about the future. Yeah. I mean, that's kind of boring. Like what can we do today? Yeah, this is very exciting, Chris, that sometime in the next year or two, there'll be these really fast things.

But I actually know about some really fast languages right now. Really? Yeah. They're called C, C++ and Swift. Seriously? Oh. Yeah. Let me show you what I mean. It's actually languages that we can make run really fast right now. And it's quite amazing, actually, how easy we can make this.

Like when you say to an average data scientist, hey, you can now integrate C libraries, their response is not likely to be oh, awesome, right? Because data scientists don't generally work at the level of C libraries. But data scientists work in some domain, right? You work in neuroradiology, image acquisition, or you work in astrophysics or whatever.

And in your domain, there will be many C libraries that do the exact thing that you want to do at great speed, right? And currently, you can only access the ones that have been wrapped in Python, and you can only access the bits that have been wrapped in Python.

What if you could actually access the entire world of software that's been written in C, which is what most software has been written in, and it's easy enough that, you know, an average data scientist can do it. So here's what it looks like, right? Let's say we want to do audio processing, okay?

And so for audio processing, I'm thinking like, oh, how do I start doing audio processing and in my quick look around, I couldn't see much in Swift that works on Linux for audio processing. >> So you write an MP3 decoder from scratch, right? >> Yeah, I thought about doing an MP3 decoder from scratch, but then I -- >> That's a single processing.

>> I figured, like, people have MP3 decoders already. What are they doing? And I looked it up on the Internet, and it turns out there's lots of C libraries that do it. And one popular one, apparently, is called SOX, right? And I'm a data scientist, I'm not an audio processing person, so this is my process last week was, like, C, library, MP3, decode, and it says use SOX.

So look at this. I've got something here that says import SOX. And then it says, in it's SOX, and then it says read SOX audio. Where did this come from? Well, this comes from a library. Here it is, sound exchange. This is what C library home pages tend to look like, they tend to be very 90s.

And basically, I looked at the documentation, and C library documentation tends to be less than obvious to kind of see what's going on, but, you know, you just kind of have to learn to read it, just like you learn to read Python documentation. So basically it says you have to use this header, and then these are the various functions you can call.

There's something called edit, and there's something called open. So here's what I did. I jumped into VIM, and I created a directory, and I called it Swift SOX. And in that directory, I created a few things. I created a file called package.swift, and this is the thing that defines a Swift package.

A Swift package is something that you can import, and you can actually type Swift package in it, and it will kind of create this skeleton for you. Personally, my approach to wrapping a new C library is to always copy an existing C library folder that I've created, and then just change the name, because every one of them has the same three files, right?

So this is file number one, you have to give it a name, and then you have to say what's the name of the library in C. And in SOX, the name of the library is SOX. Part two is you have to create a file called sources SOX module.moduleMap, and it contains always these exact lines of code, again, where you just change the word SOX, and the word SOX, and the word SOX.

So it's not rocket science. >> So what this is doing is this is saying that you want to call it SOX and Swift. They called it SOXU.H for some reason. >> Well, I actually call it SOXU.H, which we'll see in a moment. And then, but the library isn't -- it gets linked in by libsox.

>> Yeah, exactly. >> So all these things in C can be different. >> Yeah. So, you know, most of the time, we can make them look the same. And so then the final third file that you have to create is the .H file. And so you put that in sources, SOX, and I call it SOXU umbrella header.H, and that contains one line of code, which is the header file, which as you saw from the documentation, you just copy and paste it from there.

So once you add these three files, you can then do that, okay? And so now I can import SOX. And now this thing, this C function is available to Swift, right? And so this is kind of wild, right? Because suddenly, like a lot of what this impractical deep learning for coders course is about is like opening doors that weren't available to us as data scientists before and thinking what happens if you go through that door, right?

So what happens if you go through the door where suddenly all of the World C libraries are available to you? What can you do in your domain that nobody was doing before? Because there wasn't any Python libraries like that. So I -- and so what I tend to do is write little Swift functions that wrap the C functions to make them look nice.

So here's init SOX, which checks for the value I'm told the docs say to check for. And SOX open read, for some reason you have to pass nil, nil, nil, so I just wrap that. And so now I can say read SOX audio. And so that's going to return some kind of structure.

And so you have to read the documentation to find out what it is or copy and paste somebody else's code. Very often the thing that's returned to you is going to be a C pointer. And that's no problem. Swift is perfectly happy with pointers. You just say point E to grab the thing that it's pointing at.

And according to the documentation, there's going to be something called signal, which is going to contain things like sample rate, precision, channels, and length. And so I can -- let's run those two. So I can run that, and I can see I've opened an audio file with a C library without any extra stuff.

>> One of the things you can do is you can type SOX tab, and now here's all the stuff that's coming in from that header file. >> That's wild. Yeah. Super cool. So now I can go ahead and read that. And this is kind of somewhat similar to Python.

In Python you can open C libraries in theory and work with them. But I don't do it, almost never do it, because I find that when I try to -- the thing you get back are these C structures and pointers, which I can't work with in Python in a convenient way.

Or if I do use things like PyBind11, which is something that helps with that, then I have to create all these make scripts and compile processes, and I just don't bother, right? None of us bother. But in Swift, it's totally fine. And then the nice thing is we can bring Python and C and Swift together by typing import Python.

>> Be unholy marriage. >> Yeah. Now we can just say -- we can take our C array and say make NumPy array and plot it, right? So we're really bringing it all together now. And we can even use the ipython.display, and we can hear some audio. >> Hi, everyone.

I'm Chris. >> Hi, everyone. Hi, Chris. >> Hi, Jeremy. >> Thank you, Jeremy. >> My pleasure. All right. So -- >> Why did I say this to this again? >> Okay. So this is pretty great, right? We've got Swift libraries, C libraries, Python libraries. We can bring them all together.

We can do stuff that our peers aren't doing yet. But what I want to know, Chris, is how the hell is this even possible? >> Wow. Okay. Your guy likes to look under the covers or under the hood. Where's the -- cool. So let's talk about this. C is really a very simple language.

So it should be no problem to do this, right? So C is two things, actually. It's really important. I think you were just talking about why it's actually very useful. There's tons of code available in C. A lot of that C is really useful. But C is actually a terrible, crazy, gross language on its own right.

C has all these horrible things in it, like pointers, that are horribly unsafe. >> And we have a question. >> Oh. Let's do it. >> Is it possible to achieve similar results in Python using something like scytheon? >> Yeah. Absolutely. So scytheon is a Python-like language which compiles to C.

And I would generally rather write scytheon than C for integrating C with Python. You still kind of -- it's actually easier in a Jupyter notebook because you can just say percent, percent scytheon and kind of integrate it. But as soon as you want to start shipping a module with that thing, which presumably is the purpose is you want to share it, you then have to deal with, like, build scripts and stuff like that.

So scytheon has done an amazing job of kind of making it as easy as possible. But I personally have tried to do quite a lot with scytheon in the last few months and ended up swearing off it because it's just still not convenient enough. I can't quite use a normal debugger or a normal profiler and just ship the code directly.

And it's still -- yeah, it's great for Python if that's what you're working with. But it's nowhere near as convenient. I've created Swift C libraries, I created a Swift C library within a week of starting to use Swift. It was just very natural. >> Cool. And so the thing I want to underscore here is that C is actually really complicated.

C has macros. It's got this preprocessor thing going on. It's got bit fields and unions and it's weird notion of what verbs are. It's got volatiles. It's got all this crazy stuff that the grammar is context sensitive and gross. And so it's just actually really hard to deal with.

Does it sound like somebody who's been through the process of writing a C compiler and came out the other side? >> Well, so the only thing worse than C is C++. And it has this like dual side of it. It's both more gross and huge and it's also more important in some ways.

And so Swift doesn't integrate with C++ today, but we want to be able to. We want to be able to provide the same level of integration that you just saw with C and C++. But how are we going to do that? Well, Swift loves C APIs like Jeremy was just saying.

And so we love C APIs because we want you to be able to directly access all this cool functionality that exists in the world. And so the way it works as you just saw is we take the C ideas and remap them into Swift. And so because of that, because they're native pure Swift things, that's where you get the debugger integration.

That's where you get code completion. That's where you get all the things you expect to work in Swift talking to dusty deck old grody C code from the 80s or wherever you got it from, whatever epoch. And so we also don't want to have wrappers or overhead because that's totally not what Swift's about.

So Jeremy showed you that usually when you import the C API into Swift, it looks like a C API. And so you could, but the nice thing about that is that you can build the APIs you want to wrap it and you can build your abstractions and make that all good in Swift.

So one of the ways this happens is that inside the Swift compiler, it can actually read C header files. And so we don't have a great way to plug this into workbooks quite yet, but Swift can actually take a C header file like math.h, which has macros. Here's M under bar E because M under bar E is a good way to name E apparently.

Here's the old school square root. Here's the sine and cos function, which of course it returns sine and cosine in through pointers because C doesn't have tuples. And so when you import all that stuff into Swift, you get M under bar E as a double that you can get.

You have square root and you can totally call square root. You have sine and cos. You get this unsafe mutable pointer double thing, which we'll talk about later. Similarly, like malloc, free, realloc, all this stuff exists. And so just to show you how crazy this is, let's see if we can do the side by side thing.

Can you make it do side by side? Is that a challenge? Yes. My window skills are dusty. Check it out. Okay. Beautiful. So what we have here is we have the original header file, math.h on the left. If you look at this, you'll see lots of horrible things in C that everybody forgets about because you never write C like this, but this is what C looks like when you're talking about libraries.

We've got a whole bunch of if defs. We've got macros. We've got like crazy macros. We've got conditionally enabled things. We've got these things are also macros. We've got inline functions. We've got tons and tons and tons of stuff. We've got comments. We've got structures like exception, of course, that's an exception, right?

So when you import this into Swift, this is what the Swift compiler sees. You see something that looks very similar, but this is all Swift syntax. So you see you get the header, the comments, you get all the same functions. You get all of the -- like here's your munderbar e.

And you get your structures as well. This all comes right in. And this is why Swift can see it. Now how does this work? That's the big magic question. So if you want to get this to work, what you can do is you can build into the Swift compiler.

We can write a C parser, right, and we can implement a C preprocessor, and we can implement all the weird rules in C. Someday we can extend it and write C++ as well, and we can build this library so the Swift compiler knows how to parse C code, and a C++ compiler is pretty easy to write, so we can hack that on a weekend.

Or, yay, good news, like we've already done this many years ago, it's called Clang. So what Clang is, it's a C++ compiler. Oh, this is getting even -- just to -- even more talk about how horrible C is, you actually get -- you get inline functions. Inline functions, the insane thing about inline functions is that they don't exist anywhere in a program unless you use them.

Right? They get inlined. And so if you want to be able to call this function from C, you actually have to code gen, you have to be able to parse that, code gen, understand what unions are now, understand all of this crazy stuff just so you can get the sign bit out of a float.

C also has things like function pointers and macros and tons of other stuff, it's just madness. And so the right way to do this is to build a C compiler. And the C compiler we have is called Clang. And so what ends up happening is that when Jeremy says import socks, Swift goes and says, ha ha, what's a socks?

Oh, it's a module. Okay, what is a module? Oh, it's C. Oh, it's got a header file. Fire up Clang. Go parse that header file. Go parse all the things the header file pulls in. That's what an umbrella header is. And go pull the entire universe of C together into that module.

And then build what's called syntax trees to represent all the C stuff. Well, now we've got a very perfect, pristine C view of the world the exact same way a C compiler does. And so what we can do then is we can build this integration between Clang and Swift where when you say give me malloc or give me socks in it, Swift says, whoa, what is that?

Hey, Clang, do you know what this is? And Clang says, oh, yeah, I know what socks in it is. It's this weird function. It takes all these pointers and blah, blah, blah, blah. And Swift says, okay, cool. I will remap your pointers into my unsafe pointer. I will remap your int into my int32 because the languages are a little bit different.

And so that remapping happens. And then when you call that inline function, Swift doesn't want to know how unions work. That's crazy. So what it doesn't say is it says, hey, Clang, you know how to do all this stuff. You know how to code generate all these things. And they both talked to the LVM compiler that we were talking about last time.

And so they actually talked to each other. They share the code. Clang does all that heavy lifting. And now it's both correct. And it just works. Two things we like. And so these two things plug together really well. And now Swift can talk directly to C APIs. It's very nice.

If you want to geek out about this, there's a whole talk that's like a half hour, an hour long talking about how all this stuff works at a lower level. We will add that link to the lesson notes. Yeah. So let's jump back to your example. Okay. So one of the reasons I'm really interested in this description, Chris, is that it's kind of all about one of the reasons I really wanted to work with you, apart from the fact that you're very amusing and entertaining, is that -- Amusing to laugh at.

Yeah, absolutely. Is that this idea of what you did with Clang and Swift is like the kind of stuff that we're going to be seeing is what's happened with like how differentiation is getting added to Swift. And like this idea of like being able to pull on this entire compiler infrastructure, as you'll see, is actually going to allow us to do some similarly exciting and surprisingly amazing things in deep learning work.

And I'll say this is all simple now, but actually getting these two massive systems talk to each other was kind of heroic. And it was -- getting Python integrated was comparatively easy because Python is super dynamic and C is not dynamic. Yeah. And one thing I'll say about C libraries is each time I come across a C library, many of them have used these weird edge case things Chris described in surprising ways.

And so I just wanted to point at a couple of pointers as to how you can deal with these weird edge cases. So when I started looking at kind of how do I create my own version of tf.data, I need to be able to read JPEG files, I need to be able to do image processing, I was interested in trying this library called vips.

And vips is a really interesting C library for image processing. And so I started looking at the -- at bringing in the C library. And so I started in exactly the way you've seen already. So let's do that. So you'll find just like we have a Swift socks in the repo, there's also a Swift vips.

And we'll start -- we'll start seeing some pretty familiar things. There's the same package.swift that you've seen before, but now it's got some extra lines we'll describe in a moment. There's the sources, vips, module map with the exact three lines of code that you've seen before. There's the sources, vips, some header, I call it a different name in this case, which has the one line of code which is connecting to the header.

After you've done that, you can now import vips done. But it turns out that the vips documentation says that they actually added the ability to handle optional positional arguments. >> In C. >> In C. So it turns out that you can do that in C, even though it's not officially part of C, by using something called "bargs," which is basically in C.

You can say the number of arguments that go here is kind of not defined ahead of time, and you can use something I've never heard of before called a sentinel, and basically you end up with stuff which looks like this. You end up with stuff which looks like this, where you basically say I want to do a resize and it has some arguments that are specified, like horizontal scale, and by default it makes the aspect ratio the same.

But if you want to also change the aspect ratio and have a vertical scale, you literally write the string vscale, and that says, oh, the next argument is the vertical scale. And if you want to use some different interpolation kernel, you pass the word kernel, you say there's some different interpolation kernel.

Now this is tricky because for all the magic that Swift does do, it doesn't currently know how to deal with vargs and sentinels. It's just an edge case of the C world that Swift hasn't handled yet. >> I think this might be the last edge case. >> Okay. >> Jeremy has this amazing thing to find the breaking point of anything.

>> That's what I do. >> Yes. >> But no problem, right? The trick is to provide a header file where the things that Swift needs to call look like the things that they expect. So in this case, you can see I've actually written my own C library, right? And so I added a C library by literally just putting into sources.

I just created another directory. And in there, I just dumped a C header file, right? And here's the amazing thing. As soon as I do that, I can now add that C library, not precompiled, but actual C code I've just written to my package.swift, and I can use that from Swift as well, right?

And so that means that I can wrap the VIPs weird varargs resize version with a non-varargs resize version where you always pass in vertical scale, for instance. And so now, I can just go ahead and say, VIPs load image. And then I can say VIPs get, and then I can pass that to Swift for TensorFlow in order to display it through matplotlib.

Now, there's a really interesting thing here, which is when you're working with C, you have to deal with C memory management. So Swift has this fantastic reference counting system, which nearly always handles memory for you. Every C library handles memory management differently. So we're about to talk about OpenCV, which actually has its own reference counting system, believe it or not.

But most of the time, the library will tell you, hey, this thing is going to allocate some memory, you have to free it later, right? And so here's a really cool trick. The VIPs get function says, hey, this memory, you're going to have to free it later. To free memory in C, you use the free function, because we can use C functions from Swift.

We can use the free function. And I need to make sure that we call it when we're all done. And there's a super cool thing in Swift called defer. And defer says, run this piece of code before you finish doing whatever we're doing, which in this case would be before we exit from this function.

>> Yeah, so if you throw an exception, if you return early, anything else, it will make sure to run that. >> Yeah. In this case, I probably didn't need defer, because there isn't exceptions being thrown or lots of different return places, but that's my habit, is that if I need to clean up memory, I just chuck it in a defer block, or at least that's one of the two methods that I use.

So that's that. So because I like finding the edges of things and then doing it anyway, the next thing I looked at, and this gives you a good sense of how much I hate tf.data, is I was trying to do anything I could to avoid tf.data, and so I thought, all right, let's try OpenCV.

And for those of you that have been around FastAI for a while, you'll remember OpenCV is what we used in FastAI 0.7. And I loved it because it was insanely, insanely fast. It's fast, reliable, high-quality code that covers a massive amount of computer vision. It's kind of like -- it's what everybody uses if they can.

And much to my sadness, we had to throw it out, because it just -- it hates Python multiprocessing so much. It just kept creating weird race conditions and crashes and stalls, like literally the same code on the same operating system on two different AWS servers that are meant to be the same spec, would give different results.

So that was sad. So I was kind of hopeful maybe it'll work in Swift, so we gave it a go. And unfortunately, since I last looked at it, they threw away their C API entirely, and they're now C++ only, and Chris just told you we can't use C++ from Swift.

But here's the good news. You can disguise it so Swift doesn't know that it's C++. And so the disguise needs to be a C header file that only contains C stuff, right? But what's in the C++ file behind the header file -- >> Can be anything. >> Can be anything at all.

>> Maybe Pascal. >> Clang those Pascal calling conventions, and Swift can call them. I didn't know that. >> Pascal strings, too. So here's Swift CV, and so Swift CV has a very familiar-looking package.swift that contains the stuff that we're used to, and it contains a very familiar-looking OpenCV4 module map.

Now, OpenCV actually has more than one library, so we just have to list all the libraries. It has a very familiar-looking -- actually, we don't even have a -- oh, sorry, that's right. So we didn't use the header file here, because we're actually going to do it all from our own custom C++/C code.

So I created a C OpenCV, and inside here, you'll find C++ code. And we actually largely stole this from the Go OpenCV wrapper, because Go also doesn't know how to talk to C++, but it does know how to talk to C, so that was a convenient way to kind of get started.

But you can see that, for example, we can't call new, because new is C++, but we can create a function called matnew that calls that, and then we can create a header that has mat new, and that's not C++, right? This is actually a plain C struct -- pointer to a struct, and so I can call that.

And so even generics, C++ generics, we can handle this way. So OpenCV actually has a full-on multidimensional generic array, like NumPy, with, like, matrix multiplication, all the stuff in it, and the way its generic stuff works is that you can ask for a particular pixel and you say, "What data type is it using C++ generics?" So we just create lots and lots of different versions, all the different generic versions, which in the header file look like C.

So once we've done all that, we can then say import SwiftCV and start using OpenCV stuff. So what does that look like? Well, now that we can use that, we can read an image, we can have a look at its size, we can get the underlying C pointer, and we can start doing -- yeah, and we can start doing timing things and kind of see, is it actually looking like it's going to be hopeful in terms of performance, and so forth.

I was very scared when I started seeing in Swift all these unsafe, mutable pointers and whatnot. >> They're designed to make you scared. It starts with unsafe. >> Fair enough. >> But this is C, right, and so C is inherently unsafe. >> Yeah. >> Swift's theory on that is that it does not prevent you from using it.

It just makes it so you know that you're in that world. >> But there's actually this great table I saw from Ray Wendelix, from Ray Wendelix website, and I've stolen it here. And basically what he pointed out is the names of all of these crazy pointer types actually have this very well-structured thing.

They all start with unsafe, they all end with pointer, and in the middle there's this little mini-language which is can you change them or not, are they typed or not, do we know the count of the number of things in there or not, and what type of thing do they point to if they're typed.

So once you kind of realize all of these names have that structure, suddenly things start seeming more reasonable again. >> We have two questions. >> All right, let's go with the two questions. >> One is, are the C libraries dynamically linked or statically linked or compiled from source to be available in Swift?

>> Sure. By default, if you import them, they are statically linked. And so they'll link in with the normal linker flags, and if the library is a .a file, then it will get statically linked directly into your Swift code. If it's a .so file, then you'll dynamically link it, but it's still linked to your executable.

All the linker stuff, so dlopen, is a C API. And so if you want to, you can dynamically load C libraries. You can look up their symbols dynamically. You can do all that kind of stuff, too. >> Another question is, how much C do you have to know to do all these C-related imports?

>> Almost none. So I don't really know any C at all, so I kind of randomly press buttons until things start working or copy and paste other people's C code. >> The Internet Stack Overflow has a lot of helpful stuff. >> Yeah. You need to know there's a thing called a header file, and that contains a list of the functions that you can call, and you need to know that you type #include, angle brackets, header file.

But you can just copy and paste the Swift socks library that I've already shown you, which has the three files already there. And so really, you don't need to know any C. You just need to replace the word "socks" with the name of the library you're using, and then you need to know -- you need to kind of work through the documentation that's in C, and that's the bit where it gets, like, you know -- I find the tab completion stuff is the best way to handle that, is like hit tab, and you say let x equal, and then you call some function, and then you say x.

and you see what's inside it, and things kind of start working. >> And for all the hard time you give socks as, you know, not a web design firm, it has a pretty well-structured API, and so if you have a well-structured API like this, then using it is pretty straightforward.

If you have something somebody hacked together, they didn't think about it, then it's probably going to be weird, and you may have to understand their API, and it may require you to understand a lot of C. But those are the APIs that you probably won't end up using, because if they haven't gone a lot of love to their API, people aren't using it, usually.

>> My impression is that almost all of the examples of the future power of Swift seem to rely not on the abstraction to higher levels, but on the diving into lower-level details. As a data scientist, I try to avoid doing this, I only go low if I know there's a big performance gain to be had.

>> So let me set my perspective as a data scientist, and maybe we can hear you all. >> Well, and I was just going to inject, we're starting at the bottom, so we'll be getting much higher levels soon. But there's a reason that I'm wanting to teach this stuff, which is that I actually think as data scientists, this is our opportunity to be far more awesome.

It's like being able to access, something I've noticed for the last 25 years is everybody I know in, I mean, it didn't used to be called data science, we used to call it industrial mathematics or whatever, operated within the world that was accessible to them, right? So at the moment, for example, there's a huge world of something called sparse convolutions that are, I know they're amazing, I've seen competition-winning solutions, they get state-of-the-art results.

There's two people in the world doing it, because it all requires custom CUDA kernels. For years, for decades, almost nobody was doing differential programming, because we had to calculate the derivatives by hand. So like, it's not just about, oh, I want an extra, it's absolutely not about, I want an extra 5% of performance, it's about being able to do whatever's in your head.

I used to be a management consultant, I'm afraid to say, and I didn't know how to program, and I knew Excel, and the day that I learned Visual Basic was like, oh, now I'm not limited to the things I can do in a spreadsheet, I can program. And then when I learned Delphi, it was like, oh, now I'm not limited to the things that I can program in a spreadsheet, I can do things through in my head.

So that's where I want us all to get to. Yeah. >> Hey, and some people are feeling overwhelmed with Swift, C, C++, Python, PyTorch, TensorFlow, Swift for TensorFlow, do we need to become experts on all these different languages? >> No. >> No, we don't, but can I show why this is super interesting?

Because this is like -- so let me show you why I started going down this path, right? Which is that I was using tf.data. And I found that it took me 33 seconds to iterate through ImageNet. And I know that in Python, we have a notebook which Sylvia created to compare, called timing, and the exact same thing takes 11.5 seconds.

And this is not an insignificant difference, waiting more than three times as long just to load the data is just not okay for me. So I thought, well, I bet OpenCV can do it fast. So I created this little OpenCV thing. And then I created a little test program.

So this is the entirety of my test program, right, which is something that downloads ImageNet and reads and resizes images, and does it with four threads. And so if you go Swift run -- sorry, Swift run -- okay, so when I run this, check this out, 7.2 seconds, right?

And so this was like half a day's work. And half a day's work, I have something that can give me an image processing pipeline that's even faster than PyTorch. And so it's not just like, oh, we can now do things a bit faster, but it's now like any time I get stuck that I can't do something, it's not in the library I want, or it's so slow as to be unusable, you know, this whole world's open.

So I'd say we don't really touch this stuff until you get to a point where you have no choice but to, and at that point you're just really glad it's there. >> Well, and to me, I think it's also -- the capability is important even if you don't do it.

So keep in mind, this is all code that's in a workbook. So you can get code in the workbook from anywhere, and now you can share that workbook, and you don't have to share this like tangled web of dependencies that have to go with the workbook. And so the fact that you can do this in Swift doesn't mean that you yourself have to write the code, but it means you can build on code that other people wrote.

And if you haven't seen Swift at all, if this is your first exposure to it, this is definitely not the place you start. Like the data APIs that we're about to look at would be a much more reasonable place to start. You've had a month or two months' worth of hacking with Swift time, and that's Jeremy month, so that's like a year for normal people.

So this being like super powerful and the ability to do this is, I think, really great, and I agree with you. >> Yeah, and I am totally not a C programmer at all, and it's -- honestly, it's been more like two weeks, because before that I was actually teaching a Python course, believe it or not.

But Silva has been doing this for a month. >> Yeah. So, I mean, so this is all -- it's all there, and I would definitely recommend ignoring all of this stuff, and we're about to start zooming up the levels of the stack. But the fact that it's there, I think, is reassuring, because one of the challenges that we have with Python is that you get this ceiling, and if you get up to the ceiling, then there's no going further without this crazy amount of complexity, and whether that be concurrency, or whether that be C APIs, or whether that be other things, that prevents the next steps and the next levels of innovation and the industry moving forward.

>> And this is meant to be giving you enough to go on with until a year's time course, as well. So like it's -- hopefully this is something where you can pick and choose which bits you want to dig into, and whichever bit you pick to dig into, we're showing you all the depth that you can dig into over the next 12 months.

So I was really excited to discover that we can use OpenCV, which is something I've wanted ever since we had to throw it away from fast AI, and so I thought, you know, what would it take to create a data blocks API with OpenCV? And thanks to Alexis Gallagher, who kind of gave us the great starting point to say, well, here is what a Swifty-style data blocks would look like, we were able to flesh it out into this complete thing.

And when Chris described Alexis to me as the world leader on value types, I was like, wait, I thought you created them. I thought, okay, I guess we can listen to Alexis's code for this. >> I will say I'm terrified about presenting those slides, because Alexis is sitting right there, and if you start scowling, then -- oh, no.

>> We have a handheld mic, come and correct us any time. So there's a thing here called OC data block generic, where you'll find that what we've actually done is we've got the entire data blocks API in this really interesting Swifty-style, and what you'll see is that when we compare it to the Python version, this is on every axis very significantly better.

So let's talk about some of the problems with the data block API in Python. I love the data block API, but lots of you have rightly complained that we have to run all the -- in a particular order, if we get the wrong order, we get inscrutable errors. We have to make sure that we have certain steps in there, if we miss a step, we get inscrutable errors.

It's difficult to deal with at that level, and then the code inside the data blocks API, I hate changing it now, because I find it too confusing to remember like why it all fits together. But check out this Swift code that does the same thing, right? So is that download?

This is just the same get files we've seen before. All we need to do is we say, you know what, if you're going to create some new data bunch, you need some way to get the data, you need some way -- and let's assume that they're just paths for now -- it is some way to turn all of those paths into items, so something like our item list in Python.

You need some way to split that between training and validation. And you need some way to label the items. So for example, for ImageNet, download calls that. And things that convert paths to images, so which grab all of the list of paths is that collect files. And then the thing that converts -- that decides whether they're training or validation is whether the parent.parent is trained or not.

And the thing that creates the label is the parent. And so, like, we can basically just define this one neat little package of information and we're done. And Swift will actually tell us if we forgot something. Or if one of the things that we provided, like, is training is meant to return true or false if it's training or validation.

If we return, like, accidentally return the words train instead, it'll complain and let us know. So, like, I just love this so much. But to understand what's going on here, we need to learn a bit more about how Swift works and this idea of protocols. >> Yeah. So this is something that is actually useful if you are doing deep learning stuff.

So let's talk about -- >> Sorry. >> Go for it. So let's talk about what protocols are in Swift. So we've seen structs. Right now we want to talk about what protocols are. And if you've worked in other languages, you may be familiar with things like interfaces in Java, abstract classes that are often used, advanced other weird languages have things called type classes.

And so all these things are related to protocols in Swift. And what protocols do is they're all about splitting the interface of a type from the implementation. And so we'll talk about layer later. Layer is a protocol. And it says that to use a layer, or rather to define a layer, you have this call.

So layers are callable, just like in PyTorch. And so then you can define a dense layer and say how to call a dense layer. You can define a conv2D and show how to implement a conv2D layer. And so there's a contract here between what a type is supposed to have.

All layers must be callable. And then the implementations, these are different. Right. So this is pretty straightforward stuff. Even that's quite a nice track. It's like in PyTorch, you kind of have to know that there's something called forward. And if you misspell it, or forget it to put it there, or put under call instead of forward, you get kind of weird and screwable errors.

Whereas with this approach, you get time completion from the signature. Yeah. And Swift will tell you if you get it wrong, it'll say this is what the function should have been called. Yeah. That's great. So what this is really doing is this is defining behaviors for groups of types.

And so we're saying a layer. Layer is like the commonality between a whole group of types that behave like a layer. And what does that behavior mean? Well, it's a list of what are called requirements. And so these are all the methods that the type has to have. This is the signatures of the types.

And these things often have invariants. And so one of the things that Swift has in its library is the notion of equatable. What is equatable? An equatable is any type that has this equals equals operator, right? And then it says what is equatability and all that kind of stuff.

Now the cool thing about this is that you can build up towers of types. And so you can say, well, equatable gets refined by an additive arithmetic. And an additive arithmetic is something that sports addition and subtraction. Then if you also have multiplication, you can be numeric. And if you also have negation, then it can be signed.

And then you can have integers, and you can have floating point. And now you can have all these different things that exist in the ecosystem of your program, and you can describe them. And so this is all very -- these things, these ways to reason about these groups of types give you the ability to get these abstractions that we all want.

And these types already exist in Swift. These all exist, and you can go see them in the standard library. And so why do you want this? Well, the cool thing about this is that now you can define behavior that applies to all of the members of that class. And so what we're doing here is we're saying not equal.

Well, all things that are equatable, and this T colon equatable says I work with any type that is equatable. What this is doing is this is defining a function, not equal, on any type that's equatable, and it takes two of these things and returns a bool. And we can implement not equal by just calling equals equals, which all equatable things are, and then inverting it.

So to be clear, what Chris just did here was he wrote one function that is now going to add behavior to every single thing that defines equals automatically, which is pretty magic. Just like everywhere, boom, one place, super, super abstract. But this also works for lots of other things.

This works for like absolute value. What does absolute value mean? Well, it needs any type that is signed and numeric and that's comparable. And how do you implement absolute value? Well, you compare the thing against zero. If it's less than zero, you negate it. Otherwise, you return it. Super simple.

But now everything that is a number that can be compared is now absible. Types, same thing. All these things work the same way. And so with dictionary, what you want is you want the keys in a dictionary all have to be hashable. This is how the dictionary does its efficient lookups and things like that.

The value can be anything, though. And so all these things kind of stack together and fit together like building blocks. One of the really cool things about this now is that we can start taking this further. So we talked about not equal building on top of equal equal. In the last lesson, we defined this is odd function.

We defined it on int. Well, because this protocol exists, we can actually add it as a method to all things that are binary integers. And so we can say, hey, put this on all binary integers and give them all an is odd method. And now I don't have to put is odd on int and int 16 and int 32 and the C weird things.

You can just do it in one place and now everybody gets this method. On layers. This is something that's closer to home. Here we can say, hey, I want a inferring from method that does some learning phase switching magic nonsense. But now because I put this on all layers, well, I can use it on my model because your model is a layer.

My dense layer, that's a layer. And so this thing allows this one simple idea of defining groups of types and then broadcasting behavior and functionality onto all of them at once is really powerful. Yeah, I mean, it's like Python's monkey patching, which we use all the time. But A, it's not this kind of hacky thing with weird undefined behavior sometime.

And B, we don't have to monkey patch lots and lots of different places to add functionality to lots of things. We just put it in one place and the functionality gets sucked in by everything that can use it. Yeah. And remember, extensions are really cool because they work even if you didn't define the type.

So what you can literally do is you can pull in some C library, not that we'd love C, but some C library and add things to its structs. I mean, this is it will have things automatically added to those tracks because it already supports those operations. Yeah. So all this stuff composes together in a really nice way, which allows you to do very powerful and very simple and beautiful things.

So mix-ins show up and you can control where they go. And so here's an example. This is something that Jeremy wrote. And so he defined his own protocol countable and he says, things are accountable if they have a variable named count. And the only thing I care about for countable things is I can read it.

I don't have to write it. That's what the get means. And so he says, array is countable. His OpenCV mat thingy is countable. Like all these... There's a number of pixels in it. Yeah. All these things are countable. And then Jeremy says, hey, well, take any sequence. Let's add a method or a property called total count to anything that's a sequence.

So a sequence is the same as Python's iterables. What do you think you can iterate through? Exactly. And so this is things like dictionaries and arrays and ranges and all these things are sequences. And it says, so long as the element is countable, so I have an array of countable things, then I can get a total count method or a property.

And the way this is implemented is it just says, hey, map over myself, get the count, get all the counts up together, and then I have a total count of all the things in my array. And now if I have an array of arrays, an array of mats, lazy mapped collection sequence-y thingy of mats, whatever it is, now I can just ask for its count or its total count.

Hey, Chris, this -- so this functionality you're describing is basically the same as what Haskell calls type classes. Yes. Is that right? Yeah. Is this kind of, like, stolen from Haskell? Sparred. I mean, so the interesting thing for me here is -- We let them play with it, too.

Well, the interesting thing -- the reason I ask is because, like, I've tried to use Haskell before many times and have always failed. I'm clearly not somebody who's smart enough to use Haskell. Yet I wrote the code that's on the screen right now, like, a couple of days ago.

And I didn't spend a moment even thinking about the fact I was writing the code. It was only, like, the next day that I looked back at this code, and I thought, like, wow, I just did something which no other language I've used both could do, and I was smart enough to do it.

Like, it kind of makes this what I think of as, like, super genius functionality available to normal people. Yeah, and so back at the very, very beginning of this, we talked about Swift's goal is to be able to take good ideas wherever they are and assemble them in a tasteful way, and then be not weird.

Being not weird is a pretty hard but important goal. So the way I look at programming languages is that programming languages in the world have these gems in them, and Haskell has a lot of gems, incidentally. It's a really cool functional language. It's very academic. It's super powerful in lots of ways.

But then it gets buried in weird syntax, and it's just purely functional. You have to be -- you know, it has a very opinionated worldview of how you're supposed to write code, and so it appeals to a group of people, which is great, but then it gets ignored by the masses.

And to me it's really sad that the great technology in programming languages that's been invented for decades and decades and decades gets forgotten just because it happened to be put in the wrong place. Yeah. It's not just that, but it's even the whole way things are described are all about -- Yes.

-- monoids and monads and whatever -- Existentials and things like that. Yeah, exactly. And so a lot of what Swift is trying to do is just trying to take those ideas, re-explain them in a way that actually makes sense, stack them up in a very nice, consistent way, and design it, right?

And so a lot of this was pull these things together and really polish and really push and, like, make sure that the core is really solid. Okay. We have a question. How does the Swift protocol approach avoid the inheritance tree hell problem in languages like C#, where you end up with enormous trees that are impossible to refactor?

Yes. And similarly, what are the top opinions around using the mix-in pattern, which has been found to be an anti-pattern in other contexts? Yeah. So the way that Swift influences is completely different than the way that subclasses work in C# or Java or other object-oriented languages. There, what you get is something called a Vtable.

And so your type has to have one set of mappings for all these different methods, and then you get very deep inheritance hierarchies. In Swift, you end up adding methods to int. Like, so we, on the last slide, added a method is odd to all the integers. Just don't have a Vtable.

That would be a very inefficient thing to do. And so the implementation is completely different. The trails are completely different. I will, at the end of this, I think in a couple of slides, have a good pointer that will give you a very nice deep dive on all that kind of stuff.

So also there's the binary method problem, and there's a whole bunch of other things that are very cleanly solved in Swift protocols. Okay. And then there was also a question. Out of curiosity, could you give an estimate of how long it would take someone to go from a fair level of knowledge in Python, TensorFlow deep learning, to start being able to be a competent contributor to Swift for TensorFlow?

Yeah. So we've designed Swift in general to be really easy to learn, and so that you can learn as you go. And this course is a little bit very -- it's very bottoms up, but a lot of Swift, just like Python, was designed to be taught. And what you start with when you go in from that perspective is you get a very top-down kind of perspective.

And what I would do is I would start with the Google for a Swift tour, and you get a very nice top-down view of the language, and it's very approachable. And like just pick something that like is in FastAI in some FastAI notebook now. We haven't implemented it yet, and pop it into a notebook, right?

And the first time you try to do that, you'll get all kinds of weird errors and obstructions, and you won't know what's going on, but after a few weeks -- That's on the forum, and that's what the community's about. Yeah, lots of help from the forum, and Chris and I are both on the forum, and there's SFTF teams on the forum.

We'll help you out, and in a few weeks' time, you'll be writing stuff from scratch and finding it a much smoother process. Yeah. So I want to address one weird thing here and give you something to think about, and you might wonder, okay, well, Jeremy wants to know all the countable things.

We have arrays and we have mat, and we have to say that they are countable. But the compiler knows that it's countable or not. If you try to make something countable and it doesn't have a count method, the compiler will complain to you. So why do we have to do this?

Well, let's talk about a different example, and the answer is that protocols are not just about methods -- and this is also related to the C# question -- but the behavior of those methods also matters. And so here we're going to switch domains and talk about shapes. And so I have shape -- all shapes have to have a draw method, right?

This is super easy. And what I can do is I can define an octagon and tell it how to draw. I can define a diamond, tell it how to draw using exactly the same stuff that we just saw before. Really easy. And the cool thing about this is now I can define a method, refresh, and now refresh, all it does is it clears the canvas and draws the shape.

And so all shapes will get a refresh method. So if you go do a tab completion on your octagon, it all just works. But what happens if you have something else with the draw method? So cowboys know how to draw. It's a very different notion of what drawing is, right?

You don't want cowboys to have a refresh method. It doesn't make sense to clear the screen and then pull out a gun. That's not what we're talking about here. And so the idea of protocols is really, again, to categorize and describe groups of types. And one of the things you'll see, which is kind of cool, is you can define a protocol with nothing in it.

So it's a protocol that doesn't require anything. And then you go say, I want that type, that type, that type, that type to be in this group. And now I will have a way to describe that group of types. So it can be totally random, whatever makes sense for you.

And then you can do reflection on it. You can do lots of different things that now apply to exactly that group of types. And I actually found, I still find that this kind of protocol-based programming approach is like the exact upside down opposite of how I've always thought about things.

It's kind of like you don't create something that contains these things, but you kind of like, I don't know, somehow shove things in. And the more I've looked at code that works this way, the more I realize it tends to be clearer and more concise. But I still find it a struggle because I just don't have that sense of this is how to go about creating these kinds of APIs.

And one of the things you'll notice is that we added this protocol to array in an extension. So unlike interfaces in a Java or C# type of language, we can take somebody else's type and then make it work with the protocol after the fact. And so I think that's a superpower here that allows you to work with these values in different ways.

So this is a super brief, high-level view of protocols. Protocols are really cool in Swift, and they draw in a lot of great work in the Haskell and other communities. There's a bunch of talks, and even Jeremy wrote a blog post that's really cool that talks about some of the fun things you can do.

- Won't extensions make code hard to read? Because once a functionality of a particular API or class is extended in this way, you won't know if the functionality is coming from the original class or from somewhere else. - Yeah, so that's something you let go of when you write Swift code.

And there's a couple of reasons for that, one of which is that you get good ID support. And so, again, we're kind of building some parts of this airplane as we fly it, but in Xcode, for example, you can click on a method and jump to the definition, right?

And so you can say, well, okay, here's a map on array. Where does map come from? Well, map isn't defined on array. Map filter reduced. Those aren't defined on array. Those are actually defined on sequence. And so all sequences have map filter reduced and a whole bunch of other stuff.

And so arrays are, of course, sequences, and so they get all that behavior. And so the fact that it's coming out of sequence as a Swift programmer, particularly when you're starting, doesn't really matter. It's just good functionality. - And actually, you know, we've had this same discussion around Python, which is like, oh, Jeremy imports star, and therefore I don't know where things come from, because the only way I used to know where things come from is because I looked at the top of a file and it would say, from blah, import foo.

And so I know foo comes from blah. And we had that whole discussion earlier lesson where I said, that's not how you figure out where things come from. You learn to use jump to symbol in your IDE, or you learn to use Jupyter Notebox ability to show you where things come from.

That's just the way to go. - Thank you. I feel that Scala is often a very nicely designed language that my knowledge doesn't lack in terms of the features I've seen so far in Swift. Is that true? And if so, is the choice of Swift more about JVM as opposed to non-JVM runtimes and compilers?

- Yeah, so Scala is a great language. Scala is one of the, my, the way we explain Scala is that they are very generous in the features they accept. They're undergoing a big redesign of the language to kind of cut it down and try to make the features more sensible and stack up nicely together.

Swift and Scala have a lot of similarities in some places and they diverge wildly in other places. - I mean, I would say there's a, you know, I feel like anybody doing this course understands the value of tasteful curation because PyTorch is very tastefully curated and TensorFlow might not be.

And so like using a tastefully curated, carefully put together API like Swift has and like PyTorch has, I think it makes life easier for us as data scientists and programmers. - Yeah, but I think the other point is also very good. So Scala is very strong in the JVM, Java, virtual machine ecosystem and it works very well with all the Java APIs and it's great in that space.

Swift is really great if you don't want to wait for JVM to start up so you can run a script, right? And so there's nice duels and they have different strengths and weaknesses in that sense. - Do we have time before our break that I can quickly show how this all goes together?

- I probably can't stop you even if I wanted to. - So just to come back to this HC, right, you can basically see what's happened here. We have to find this protocol saying these are the things that we want to have in a Databox API and then we said here is a specific example of a Databox API.

Now at this point we're missing one important thing which we've never actually created the bit that says this is how you open an image and resize it and stuff like that, right? So we just go through and we can say let's call .download, let's call .get items. We can create nice simple little functions now.

We don't have to create complex class hierarchies to say things like tell me about some sample and it prints it out, right? And we can create a single little function which creates a train and a valid. This is neat, right? This is something I really like about this style of programming is this is a named tuple.

And I really like this idea that we don't have to create our own struct in class all the time. It's kind of a very functional style of programming where you just say I'm just going to define my type as soon as I need it and this type is defined as being a thing with a train and a thing with a valid.

So as soon as I work brackets parentheses around this it's both a type and a thing now. And so now I can partition into train and valid and that's returned something where I can grab a random element from valid and a random element from train. We can create a processor, again it's just a protocol, right?

So remember a processor is a thing like for categories, creating a vocab of all of the possible categories and so a processor is something where there's some way to say like what is the vocab and if you have a vocab then process things from text into numbers or deprocess things from numbers into text.

And so we can now go ahead and create a category processor, right? So here's like grab all the unique values and here's label to int and here's int to label. So now that we have a category processor we can try using it to make sure that it looks sensible and we can now label and process our data.

So we first have to call label and then we have to call process. Given that we have to do those things in a row, rather than creating whole new API functions we can actually just use function composition. Now in PyTorch we've often used a thing called compose but actually it turns out to be much easier as you'll see if you don't create a function called compose but you actually create an operator.

And so here's an operator which we will call compose, right? Which is just defined as first call this function f and then call this function g on whatever the first thing you passed it is. So now we have to find a new function composition which first labels and then processes.

And so now here's something which does both and so we can map, right? So we don't have to create again all these classes and special purpose functions. We're just putting together function composition and map to label all of our training data and all of our validation data. And so then finally we can say well this is the final structure we want.

We want a training set, we want a validation set and let's again create our own little type in line, right? So that's an array of tuples. Yeah, so our training sets, an array of named tuples, a validation set is an array of named tuples and so we're gonna initialize it by passing both in.

And so this basically is now our data blocks API. There's a function called make split labeled data and we're just gonna pass in one of those configuration protocols we saw. So we're gonna be passing in the image net configuration protocol, the thing that conforms to that protocol. And we're gonna be passing in some processor, right, which is gonna be a category processor and it's gonna sort of call download, get the items, partition, map, label of, and then initialize the processor state and then do label of and then process is our processing function and then map that, right?

And so that's it. So now we can say to use this with OpenCV, we define how to open an image. There it is. We define how to convert BGR to RGB, cuz OpenCV uses BGR, that's how old it is. We define the thing that resizes to 224 by 224 with bilinear interpolation.

And so the process of opening an image is to open, then BGR to RGB, and then resize and we compose them all together, and that's it, right? So now that we've got that, we then need to convert it to a tensor. So the entire process is to go through all those transforms and then convert to a tensor.

And then, I'll skip over the bit that does the mini batches. There's a thing we've got to do the mini batches with that split label data we created, and we then just pass in the transforms that we want, and we're done, right? So the data blocks API in kind of functional-ish, protocol-ish, Swift, ends up being a lot less code to write and a lot easier for the end user.

Cuz now for the end user, there's a lot less they have to learn to use this data blocks API. It's really just like the normal kind of maps and function composition that hopefully they're familiar with as Swift programmers. So I'm really excited to see how this came out, because it solves the problems that I've been battling with for the last year with the Python data box API, and it's been really just a couple of days of work to get to this point.

>> And one of the things that this points to in Swift that is a big focus is on building APIs. And so, again, we've been talking about this idea of being able to take an API, use it without knowing how it works. It could be in C or Python or whatever, but it's about building these things that compose together and they fit together in very nice ways.

And with Swift, you get these clean abstractions. So once you pass in the right things, it works. You don't get the stack trace coming out of the middle of somebody else's library that now you have to figure out what you did somewhere along the way that caused it to break, at least not nearly as often.

>> So to see what this ends up looking like, I've created a package called data block. It contains two files in, it's got a package.swift and it's got a main.swift. And main.swift is that, right? So all that in the end to actually use it, that's how much code it is to use your data blocks API and grab all the batches.

So it comes out super pretty. So let's take a five-minute break and see you back here at 8.05. Okay, so we're gradually working our way back to what we briefly saw last week, notebook11, trading image net, and we're gradually making our way back up to hit that point again.

It's a bit of a slow process because along the way we've had to kind of invent, float, and learn about a new language and stuff like that. But we are actually finally up to zero to a fully connected model, believe it or not. And the nice thing is at this point, things are going to start looking more and more familiar.

One thing I will say though that can look quite unfamiliar is the amount of typing that you have to type with Swift, but there's actually a trick, which is you don't have to type all these types. >> You don't have to type types. >> What you can actually do is you can say, oh, here's the type I use all the time, tensor, bracket, float, and I don't like writing angle brackets either.

So let's just create a type alias called tf. And now I just use tf everywhere. Now to be clear, a lot of real Swift programmers in their production code might not like doing that a lot. And personally I do do that a lot, even not in notebooks. But you might want to be careful if you're doing actual Swift programming.

>> The way I would look at it is if you're building something for somebody else to use, if you're publishing an API, you probably don't want to do that. >> Yeah. >> But if you're hacking things together and you're playing and having fun, it's no problem at all. >> Yeah.

I mean, different strokes. I personally, I would say if I'm giving somebody something that's the whole thing, tensor floats, I would do it. But anyway, in a notebook, I definitely don't want to be typing that. So in a notebook, make it easier for your interactive programming by knowing about things like type alias.

>> Yeah. That's something we also want to make better just in general so that these things all just default to float. >> Yeah. >> You don't have to worry about it. >> That'll be nice. So then we can write a normalized function that looks exactly the same as our Python normalized function.

And we can use mean and standard deviation just like in Python. And we can define tests with asserts just like in Python. So this all looks identical. We can calculate n and m and c, the same constant, the variables that we used in Python in exactly the same way as Python.

We can create our weights and biases just like in Python, except there's a nice kind of rule of thumb in the Swift world, which is any time you have a function that's going to create some new thing for you, we always use the init constructor for that. So for example, generating random numbers and dumping them into a tensor, that's constructing a new tensor for you.

So it's actually -- you're actually calling tensorflow.init here. And so if you're trying to find where is it in an API that I get to create something in this way, you should generally look for the -- in the init section. So this is how you create random numbers in Swift for TensorFlow.

This is how you create tensor of zeros in Swift for TensorFlow. So here's our weights and biases. This is all the same stuff we just basically copied and pasted it from the PyTorch version with some very, very minor changes. We use a linear function, except rather than at, we use dot, because that's what they use in Swift for TensorFlow.

If you're on a Mac, that's option eight. If you're on anything else, it's compose key dot equals. And so now we can go ahead and calculate linear functions. We can calculate relu, exactly the same as PyTorch. We can do proper climbing in it, exactly like PyTorch. And so now we're at the point where we can define the forward pass of a model.

And this looks, basically, again, identical to PyTorch. A model can just be something that returns some value. So that -- the forward pass of our model really just builds on stuff that we already know about and it looks almost identical to PyTorch, as does a loss function, right? It looks a little bit different because it's not called squeeze.

It's called squeezing shape. It does other than that. Mean squared error is the same as PyTorch as well. And so now here's our entire forward pass. So hopefully that all looks very familiar. If it doesn't, go back to 02 in the Python notebooks. And actually this is one of the tricks, like this is why we've done it this way for you all, is that we have, like, literally kind of these parallel texts, you know, there's a Python version, there's a Swift version, so you can see how they translate and see exactly how you can go from one language and one framework to another.

That's all very well. But we also need to do a backward pass. So to do a backward pass, we can do it exactly the same way as, again, we did it in PyTorch. One trick we kind of -- Python hack we used in PyTorch. >> And so this is doing it the hard way.

This is doing it all manually. >> Okay. >> Because we have to build it. >> Doing it all manually, yep, because we have to build everything in Scratch. And the PyTorch version, we actually added a .grad attribute to all of our tenses. We're not allowed to just throw attributes in arbitrary places in Swift, so we have to define a class which has the actual value and the gradient.

But once we've done that, the rest of this looks exactly the same as the PyTorch version did. Here's our MSC grad, our ReLU grad. That's all exactly the same. In fact, you can compare here, right? Here's the Python version we created for LinGrad, here's the Swift version for LinGrad.

It's almost identical. So now that we've done all that, we can go ahead and do our entire forward and backward pass and we're good to go. But it could be so much better. >> Well, you skipped past the big flashing red lights that says don't do this. Did you miss that part?

>> Tell me about it. >> Oh, okay. So let's talk about this. We're defining a class and putting things in classes, and we haven't seen classes yet, at least not very much. >> That's true. Because before, we've used things that looked like classes, but they didn't say class on them.

They said struct on them. >> Yes. And so what is that? >> That's true. >> So let's play a little game, and so let's talk about this idea of values and references, because that's what struct versus class really means in Swift. A value is a struct thing, and a reference is a class thing.

So let's talk about Python. Here's some really simple Python code, and there's no tricks here. What we're doing is we're assigning four into A, we're copying A into B, we're incrementing A and printing them out. And so when you do this, you see that A gets incremented. Of course.

B does not. Of course. This all makes perfect sense. In Swift, you do the same thing, you get the same thing out. This is how math works, right? All very straightforward. Let's talk about arrays. So here I have an array or list in Python, and I put into X, and then I copy X and Y.

I add something to X, and it has it. I have to point with this. And then it has the extra item. That makes perfect sense, right? What happens to Y? What? What just happened here? I just added something to X, and now Y changed? Now what is going on here?

Well, we learn. We learn that there's this thing called a reference, and we learn that it does things like this, and we learn when it bites us. What happens in Swift? Well, Swift has arrays. It doesn't have lists the same way. And so here we have, again, this identical code except var.

We put one and a two into X, we copy X into Y, we add something to X, we print it out, we get the extra element. But Y is correct. What just happened? So this is something called value and reference semantics. And in Swift, arrays, dictionaries, tensors, like all these things have what's known as value semantics.

And let's dive in a little bit about what that is. So a value in something that has value semantics is a variable that -- sorry, this is self-referential. When you declare something in your code, you're declaring a name. And if it's a name for a value, that name stands for that value, right?

X stands for the array of elements that it contains. This is how math always works. This is what you expect out of basic integers. This is what you expect out of basic things that you interact with on a daily basis. Reference semantics are weird if you think about it.

So what we're doing is we're saying that X is a name for something else. And so we usually don't think about this until it comes around to bite us. And so this is kind of a problem. And let's dive in a little bit to understand why this causes problems.

So here's a function. It's a do thing. It's something that Jeremy wrote with a very descriptive name. And it takes T, and then it goes and updates this, and that's fine, right? It's super fast. Everything is good. You move on and put in a workbook, and then you build the next workbook.

Next workbook calls in a do thing, and you find out, oh, well, it changed the tensor I passed in, but I was using that tensor for something else. And now I've got a problem because it's changing a tensor that I want to use. And now I've got this bug.

I have to debug it. And I find out the do thing is causing the problem. And so what do I do? I go put a clone in there. I don't know who here adds clones in a principled way or who here-- I do use it in a principled way.

So what we do in fast AI is we kind of don't have clone. And then when things start breaking, I add more until things start breaking, and then we're done. That sounds great. Yeah. So there's a lot of clone in fast AI. And yeah. That's a good principle. I see what you're going for.

Possibly a few too many, or possibly a few too few. Well, so now think about this. What we have is we have a foot gun here in the first case. So something that's ready to explode if I use it wrong. Now I added clone. And so good news, it's cracked but slow.

So it's going to do that copy even if I don't need to, which is really sad. In Swift, things just work. You pass in a tensor. You can update it. You can return it. And it leaves the original one alone. Arguments in Swift actually even default to constants, which makes it so that you can't do that.

If you do actually want to modify something in the caller, you can do that too. You just have to be a little bit more explicit about it and use this thing called in out. And so now if you want to update the thing somebody passed to you, that's fine.

Just pass it in out and everything works fine. And on the call side, you pass it with this ampersand thing so that they know that it can change. Now, what is going on here? So this is good math. This is like the correct behavior. But how does this work?

Well, when we talk about names, we're talking about values. And so here I have a struct. This is a valuey thing. And so I say it has two fields, real and imaginary. And I define an instance of my complex number here named X. And so this is saying I have X and it's a name for the value that has one and two in it.

And so I introduce Y. Y is another notational instance of this struct. And so it also has a one and a two. And if I go and I copy it, then I get another copy. And if I change one, then I update just Ys. This is, again, the way things should work.

And so this works with structs, this works with tuples, this works with arrays and dictionaries and all that kind of stuff. How do references work? Well, references, the name here. I have a class and the class has a string and it has an integer. And so somewhere in memory, there is a string and there is an array and they're stuck together, just like with the struct.

But now when I say X, X is actually a reference or a pointer or an indirection. The reason for that is because you wrote class instead of struct. So by writing class, you're saying when you create one of these things, please create a reference, not a value. Yes, that's exactly right.

And now what happens with references is you now get copies of that reference. And so when I copy X into Y, just like in PyTorch or Python, I have another reference or another pointer to the same data. And so that's why when you go and you update it, so I'm going to go change the array through Y, it's also going to change the data that you see through X.

And so in Swift, you have a choice. And so you can declare things as classes and classes are good for certain things and they're important and valuable and you can subclass them and classes are nice in various ways. But you have a choice and a lot of things that you've seen are all defined as structs because they have much more predictable behavior and they stack up more correctly.

So in this case, you know, I was trying to literally duplicate a Python/PyTorch API. And so I just found I wasn't able to unless I used class. Yes. And then you kind of said, okay, well, that's how you do it. Yeah, and we'll get back to auto diff in a second.

But don't do it that way. Yeah. And so you can absolutely do that. And again, when you're learning Swift, it's fine, just reach for the things that are familiar and then you can learn as you go. That's perfectly acceptable. But here we're trying to talk about things Swift is doing to help save you and make your code more efficient and things like that.

And I still reach for class a lot. But then every time a real Swift programmer takes my thing that had class and replaces it with something more Swift-y, it ends up being shorter and easier to understand. And so I agree, go for it, get things working with class. But when it becomes time, start to work with this and look at it and figure out how it works.

Now, there's one thing that's really weird here. And if you remember last time, the first thing I told you about was var and let, right? And what is going on here? This does not make any sense. We've got Y and now we are updating, if this thing will go away, we are updating a thing in Y even though Y is a constant.

And what does that even mean? Well, the reason here is that the thing that is constant, this constant is this reference. And so we've made a new copy of the reference, but we're allowed to copy the thing it points to because we're not changing X or Y itself. So this doesn't make sense, none of this makes sense, but how does let and var work?

Well, this is a thing that comes back to the mutation model in Swift. And I'll go through this pretty quickly. This is not something you have to know. But let's say I have a complex number and it's a struct and I say, hey, this thing is a constant. I want to go change it, right?

That's not supposed to work. What happens? Well, if you try to do that, Swift will tell you, ha ha, you can't do that. You can't use plus equals on a real that's in a C because C1 is a let. And Swift is helpful. And so it tries to lead you to solving a problem that says, hey, by the way, if you want to fix this, you want to make it go away, just change let to var and then everything is good.

That's totally fine. Now, okay, fine. Well, maybe I really do want to change it. And so what I'm going to do is I'm going to get a little bit trickier and I'm going to find this extension. I'm going to add a method increment to my complex number. I'm going to increment inside the method and then call the method.

Can I get away with that? Well, you know, these things may be in different files. The compiler may only be able to see one or the other. And so if you run this, it has no idea whether increment is going to change that thing, right? And so what the compiler does is says, ah, well, you can't implement real inside of this increment method either because it says self is immutable.

And it says Mark method mutating to make self mutable. Now the thing to think about in methods, both in Python, but also in Swift is that they have a self and in Python, you have to declare it. Swift has it too. It just, it's just not making you write it all the time because that would be annoying.

And so when you declare a method on a struct, what you do is you're getting self and it's a copy of the struct. Okay. Now what this is saying is this is saying that, hey, you're actually changing self.real. Self is constant. And so you can't do that here, but what you can do is you can mark it mutating.

And so what that looks like is that says we can mark this function as mutating. And what that does is it says our self is now one of these in out things, the in out thing that allows us to change it in the color. And because it's now mutating, it's totally fine to change it.

That's no big deal. The compiler leads you to this and shows you what to do, but now we come back to this problem over here. We say, well, we have a constant. We're calling increment. How does that work? Well, it still doesn't. The compiler will tell you, hey, you can't do that.

You can't mutate C1. And now it knows the increment can change it. And so it says really, really, really, if you want to do this, go mark C1 as a var. And Jeremy would say, just mark everything as a var because that's how he is. And so the nice thing about this, though, is it all stacks up nicely and it all works.

And this is what allows -- this is kind of the underlying mechanics that allow the value stuff to work. Now, you may be wondering, how is this efficient? So we were talking about in the PyTorch world, you end up copying all the time, even if you don't end up needing it.

In Swift, we don't want to do all those copies. And so on the other hand, we don't want to be, like, always copying. So where do the copies go and how does that work? So if you're using arrays, or arrays of arrays of arrays of dictionaries of arrays of super nested things, what ends up happening is arrays are struct, you might be surprised.

And inside of that struct, it has a pointer or a reference. And so the elements of an array are actually implemented with the class. And so what I have here is I have A1, which is some array, and I copied it to A2, and I copied it to A3, I copied it to A4 because I'm passing it all around, I'm just passing this array around, no big deal, and what happens is I'm just copying this reference and it happens to be buried inside of a struct.

And so this passing around arrays, full value semantics, super cheap, no problem, it's not copying any data, it's just passing the pointer around, right, just like you do in C or even in Python. The magic happens when you go and you say, okay, well, I've now got A4, and so all these things are all sharing this thing, I'm going to add one element to A4, well, what happens?

Well, first thing that happens is append is a mutating method, and so it says, hey, I'm this thing called a copy-on-write type, and so I want to check to see if I'm the only user of this data. And it turns out no, lots of other things are pointing to our data here, and so lazily, because it's shared, I'll make a copy of this array.

And so I only get a copy of the data if it's shared and if it changes. >> So that should be one, two, three, 22? >> Yeah, that should be one, two, three, 92. I am buggier than Swift. Now the interesting thing about this is because of the way this all works out is if you go and you change A4 again, it goes and just updates it in place, there's no extra copy.

And so the cool thing about this is that you get exactly the right number of copies and it just works, you as a programmer don't have to think about this. This is one of the things that Swift is just, like, subtracting from your consciousness of the things that you have to worry about, which is really nice.

And so a really nice aspect of this is that you get algebra, like, values work the way values are supposed to work, you get super high performance, we get to use more emojis, which I always appreciate. If you want to learn more about this, because this is also a really cool, deep topic that you can geek out about, particularly if you've done object-oriented programming before, there's a lot that's really nice about this, there's a video you can see more.

So let's go back to that auto-diff thing, and let's actually talk about auto-diff from a different perspective. So this is the auto-diff system implemented the same way as the manually done PyTorch version, and we didn't like it because it was using references. Let's implement, again, the very low-level manual way in Swift, but before we do, let's talk about where we want to get to.

So Swift has built-in, and Swift for TensorFlow has built-in automatic differentiation for your code. So you don't have to write gradients manually, you don't have to worry about all this stuff. And the way it works is really simple. There are functions like gradient, and you call gradient, and you pass it a closure, and you say, what is the gradient of x times x?

And it gives you a new function that computes the gradient of x times x, and here we're just calling that function on a bunch of numbers that we're striding over and printing them out, and it just gives you this gradient of this random little function we wrote. Now, one of the interesting things about this is, I wrote this out, it takes just doubles or floats or other things like that.

Auto-diff in Swift works on any differentiable type, anything that's continuous, anything that's not like integers, anything that has a gradient. And so -- >> So you can't do this in just a library. This has to be built into the language itself, because you're actually -- you're just -- you're literally compiling something that's multiplying doubles together, and it has to figure out how to get gradients out of that.

>> Yeah. You can do things as a library, and that's what PyTorch and other frameworks do in Python. >> Well, PyTorch -- >> But it doesn't work the same way at all. >> And PyTorch will not do that on doubles. PyTorch -- >> Oh, yes. That is true. >> Why is it using that on tenses?

>> Yes. Yes. And so this doesn't just work on doubles. If you want to define quaternions or other cool numeric scientific-y things that are continuous, those are differentiable, too, and that all stacks out and works. So the -- there's a bunch of cool stuff that works this way. You can define a function.

You can get the gradient at some point with the function. You can pass enclosures. Like, all this stuff is really nice. Instead of talking about that, we're going to do the -- from the bottom-up thing. And so I'm going to pretend I understand calculus for a minute, which is sad.

So if you think about what differentiation is, computing the derivative of a function, there's two basic things you have to do. You have to know the axioms of the universe, like, what does -- what is the derivative of plus or multiply or sine or cosine or tensor or matmul.

Then you have to compose these things together, and the way you compose them together is this thing called the chain rule, and this is something that I relearned, sadly, over the last couple of weeks. >> But we did in the Python part of this course. >> Yes. >> And we wrote it a different way.

We had to y dx equals to y du du dx. >> Yeah, apparently there's some ancient feud between the people who invented calculus independently, and they could not agree on notation. So what this is saying is this is saying, if you want the derivative of f calling g, the derivative of f calling g is the derivative of f applied to the forward version of g multiplied by the derivative of g.

And this is important because this is actually computing the forward version of g in order to get the derivative of this, and so -- >> Which we kind of hid away in our dy du du dx version. >> Do you want to make the final motion? >> Oh, sure.

I don't know how to do it on your machine. Oh, there you go. So how are we going to do this? Well, what we're going to do is we're going to look at defining the forward function of this, and so we'll use the mean squared error as the example function.

This is a little bit more complicated than I want, and so what I'm going to do is I'm going to actually just look at this piece here, and so I'm going to define this function MSE inner, and all it is is it's the dot squared dot mean. So it's conceptually this thing, MSE inner, that just gets the square of x and then does the mean just because that's simpler, and then we'll come back to MSE at the end.

And so in order to understand what's going on, I'm going to find this little helper function called trace, and all trace does is it -- you can put it in your function, and it uses this little magic thingy called pound function, and when you call trace, it just prints out the function that it's called from.

And so here we call foo, and it prints out, hey, I'm in foo AB, and I'm in bar X, and so we'll use that to understand what's happening in these cells. So here I can define, just like you did in the PyTorch version, the forward and the derivative versions of these things, and so X times X is the forward, the gradient version is two times X.

X dot mean is the forward, this weird thing of doing a divide is apparently the gradient of mean, and I checked it, it apparently works, I don't know why. So then when you define the forward function of this MSE inner function, it's just saying give me the square and take the mean, super simple, and then we can use the chain rule, and this is literally where we use the chain rule to say, okay, we want the gradient of one function on another function, just like the syntax shows, and the way we do that is we get the gradient of mean and pass it to the inner thing and multiply it by the gradient of the other thing.

So this is really literally the math interpretation of this stuff. And given that we have this, we can now wrap it up into more functions, and we can say, let's compute the forward and the backwards version of MSE, we just call the forward version, we call the backward version, and then we can run on some example data, one, two, three, four.

>> And just to be clear, the upside down triangle thing is not an operator here, it's just using inner code as part of the name of that function. >> That's a gradient delta symbol thingy, I found that on Wikipedia. So when you run this, what you'll see is it computes the forward version of the saying, it runs square and then it runs mean, and then it runs square again, and then it runs the backward version of mean and square, and this makes sense given the chain rule, right?

We have to recompute the forward version of square to do this, and for this simple example, that's fine, square is just one multiply, but consider it might be a multiply of megabytes worth of stuff. It's not necessarily cheap, and when you start composing these things together, this recomputation can really come back to bite you.

So let's look at what we can do to factor that out. So there's this pattern called chainers, and what we call the value and chainer pattern. And what we want to do is we want to find each of these functions, like square or mean or your model, as one function that returns two things.

And so what we're going to do is we're going to look at the other version of calculus's form of this, and so when you say that the derivative of x squared is 2x, you actually have to move the dx over with it, and this matters because the functions we just defined are actually only, those are only valid if you're looking at a given point, that they're not valid if you compose it with another function.

This is just another way of writing the chain rule. It's the exact same thing, and so we're going to call this the gradient chain, and all it is is an extra multiply. And Chris, I just need to warn you, in one of the earlier courses, I got my upside-down triangles mixed up as you just did, so the other way round is delta, and this one is called nabla, and I only know that because I got in trouble for screwing it up from everywhere last time.

Okay. Thank you, Jeremy, for saving me. So all this is, is the same thing we saw before, it just has an extra multiplication there because that's what the chain rule apparently really says. So what we can do now is, now that we have this, we can actually define this value with chain function, and check this out.

What this is doing is it's wrapping up both of these things into one thing. So here we're returning the value, when you call this, we're also returning this chain function. Can you just explain this TF arrow, TF, how do I read that, TF arrow, TF? So what this is doing is this is saying we're defining a function, square WVC, it takes X, it returns a tuple, we know what tuples are.

These are fancy tuples, like you were showing before, where the two things are labeled. So there's a value member of the tuple, and there's a chain label of the tuple. The value is just a tensor float. The chain is actually going to be a closure. And so this says it is a closure that takes a tensor of float and returns a tensor of float.

So that's just a way of defining a type in Swift where the type is itself a function. And so what square VWC is going to be is it's going to be two things, it's the forward thing, the multiple X times X, and that's the backwards thing, the thing we showed just up above that does DDX times two times X.

And the forward thing is the actual value of the forward thing, the backward thing is a function that will calculate the backward thing. And the chain here is returning a closure, and so it's not actually doing that computation. So we can do the same thing with mean, and there is the same computation.

And so now what this is doing is it's a little abstraction that allows us to pull together the forward function and the backward function into one little unit. And the reason why this is interesting is we can start composing these things. And so this MSE inner thing that we were talking about before, which is mean followed by square, or square followed by mean, we can define, we just call square VWC and then we pass the value that it returns into the mean VWC.

And then the result of calling this thing is mean.value, and the derivative is those two chains stuck together. And so if we run this, now we get this really interesting behavior where when we call it, we're only calling the forward functions once and the backward function once as well.

And we also get the ability to separate this out. And so here what we're doing is we're calling the VWC for the whole computation, which gives us the two things. And here we're using the value. So we got the forward version of the value. And if that's all we want, that's cool, we can stop there.

But we don't. We want the backward version too. And so here we call it what we call the chain function to get that derivative. And so that's what gives us both the ability to get the forward and the backward separate, which we need, but also it makes it so we're not getting the re-computation because we're reusing the same values within these closures.

So given that we have these like infinitesimally tiny little things, let's talk about applying this pattern. I'll go pretty quickly because the details aren't really important. So ReLU is just max with zero. And so we're using the same thing as ReLU grad from before. Here's the LIN grad using the PyTorch style of doing this.

And so all we're doing is we're pulling together the forward computation in the value thing here. And then we're doing this backward computation here. And we're doing this with closures. So can I just talk about this difference because it's really interesting because this is the version that Silva and I wrote when we just pushed it over from PyTorch.

And we actually did the same thing that Chris just did, which is we avoided calculating the forward pass twice. And the way we did it was to cache away in in.grad and out.grad the intermediate values so that we could then use them again later without recalculating them. Now what Chris is showing you here is doing the exact same thing but in a much more automated way, right?

It's a very mechanical process. Yeah. We're having to kind of use this kind of very heuristic, hacky, one at a time approach of saying what do I need at each point, let's save it away in something or give it a name and then we'll use it again later. It's kind of interesting.

And also without any mutation, this functional approach is basically saying let's package of everything we need and hand it over to everything that needs it. And so that way we never had to say what are we going to need for later. It just works. You'll see all the steps are here out times blah dot transposed, out times blah dot transposed, right?

But we never had to think about what to cache away. And so this is not something I would want to write ever again, manually, personally. But the advantage of this is it's really mechanical and it's very structured. And so when you write MSE, the full MSE, what we can do is we can say, well, it's that subtraction, then it's that dot mean dot squared, and then on the backwards pass we have to undo the squeeze and the subtraction thingy.

And so it's very mechanical how it plugs together. Now we can write that forward and backward function and it looks very similar to what the manual version of the PyTorch thing looked like where you're calling these functions and then in the backward version you start out with one because the gradient of the loss with respect to itself is one, which now I understand, thanks to Jeremy.

And then they chain it all together and you get the gradients. And through all of this work, again, what we've ended up with is we've gotten the forward and backwards pass, we get the gradients of the thing, and now we can do optimizers and apply the updates. Now the -- >> I've got this mention like what Chris was saying about this one thing here and so forth.

For Chris and I, we took a really long time to get to this point and we found it extremely difficult and at every point up until the point where it was done, we were totally sure we weren't smart enough to do it. And so like, please don't worry that there's a lot here and that you might be feeling the same way Chris and I did, but yeah, you'll get there, right?

>> This was a harrowing journey. >> Yeah. It's okay if this seems tricky, but just go through each step one at a time. So again, this is talking about the low-level math-y stuff that underlies calculus. And so the cool thing about this, though, from the Swift perspective is this is mechanical.

And compilers are good at mechanical things. And so one of the things that we've talked about a lot in this course is the idea of their primitives, they're the atoms of the universe and then they're the things you build out of them. And so the atoms of the universe for tensor, the atoms of the universe for float, we've seen, right?

And so we've seen multiply and we've seen add on floats. Well, if you look at the primitives of the universe for tensor, they're just methods and they call the raw ops that we showed you last time, right? And so if you go look at the TensorFlow APIs, what you'll see is those atoms have this thing that Swift calls them VJP's for weird reasons.

This defines exactly the mechanical thingy that we showed you. And so the atoms know what their derivatives are and the compiler doesn't have to know about the atoms, but that means that if you want to, you can introduce new atoms. That's fine. The payoff of this now, though, is you don't have to deal with any of this stuff.

So that's the upshot. What I can do is I can define a function. So here's MSE inner and it just does dot squared dot mean. And I say make it differentiable and I can actually get that weird thing, that chainer thing directly out of it and I can get direct low-level access if for some reason I ever wanted to.

Generally you don't and that's why you say give me the gradient or give me the value and the gradient. And so this stuff just works. And the cool thing about this is this all stacks up from very simple things and it composes in a very nice way. And if you want to, you can now go hack up the internals of the system and play around with the guts and it's exposed and open for you.

But if you're like me, at least, you would stay away from it and just write maps. Well, I mean, sometimes we do need it, right? So you'll remember when we did the heat maps, right, those heat maps, we actually had to dive into the registering a backward callback in PyTorch and grab the gradients and then use those in our calculations and so there's plenty of stuff we come across where you actually need to come up with this.

Yeah, and there are some really cool things you can do too. So now we ended up with a model and so this is something that I had never got around to doing with Fixme. So here's our forward function. Here we're implementing it with matmoles and with the lint function, the rel use and things like that.

The bad thing about defining a forward function like this is you get tons of arguments to your function. And so some of these arguments are things that you want to feed into the model. Some of these things are parameters and so as a refactoring, what we can do is we can introduce a struct, you might be surprised, that puts all of our parameters into it.

So here we have my model and we're saying it is differentiable and what differentiable means is it has a whole bunch of floating point stuff in it and I want to get the gradient with respect to all of these. So now I can shove all those arguments into the struct, it gives me a nice capsule to deal with and now I can use the forward function on my model.

I can declare it as a method. This is starting to look nicer, this is more familiar and I can just do math and I can use w1 and b1 and these are just values defined on our struct. Now I can get the gradient with respect to the whole model and our loss.

And all of this is building up on top of all those different primitives that we saw before that we, and the chain rule and all these things, that now we can say hey, give us the gradient of the model with respect to x-train and y-train and we get all the gradients of w1, b1, w2, b2 and all this stuff works.

You can see it all calling the little functions that we wrote and it's all pretty fast. Now again, like we were just talking about, this is not something you should do for matmul or convolution, but there are reasons why this is cool and so there are good reasons and there are annoying reasons, I guess.

So sometimes the gradients you get out of any auto-diff system will be slow because you do a ton of computation and it turns out the gradient ends up being more complicated and sometimes you want to do an approximation. And so it's actually really nice that you can say hey, here's the forward version of this big complicated computation, I'm going to have an approximation that just runs faster.

Sometimes you'll get numerical instabilities in your gradients and so you can define, again, a different implementation of the backwards pass, which can be useful for exotic cases. There are some people on the far research side of things that want to use learning and things like that to learn gradients, which is cool.

And so having the system where everything is just simple and composes but is hackable is really nice. There are also always going to be limitations of the system. Now one of the limitations that we currently have today, which will hopefully be fixed by the time the video comes out, is we don't support control flow and auto-diff.

And so if you do an if or a loop like an RNN, auto-diff will say I don't support that yet. But that's okay because you can do it yourself. So we'll go see an example of that in 11. There we go. And so what we have implemented here, and we'll talk about layers more in a second, is we have this thing called switchable layer.

And what switchable layer is, is it's just a layer that allows us to have a Boolean toggle to turn it on and off. And the on and off needs an if. And so Swift auto-diff doesn't currently support if. And so when we define the forward function, it's super easy.

We just check to see if it's on, and if so, we run the forward, otherwise we don't. Because it doesn't support that control flow yet, we have to write the backwards pass manually. And we can do that using exactly all the stuff that we just showed. We implement the value, and we implement the chainer thing.

And we can implement it by returning the right magic set of closures and stuff like that. And so it sucks that Swift doesn't support this yet. But it's an infinitely hackable system. And so for this or anything else, you can go and customize it to your heart's content. Yeah.

And I mean, one of the key things here is that Chris was talking about kind of the atoms. And at the moment, the atoms is TensorFlow, which is way too big an atom. It's a very large atom. But at the point when we're kind of in MLIR world, the atoms are the things going on inside your little kernel DSL that you've written.

And so this ability to actually differentiate on float directly suddenly becomes super important. Because it means that like, I mean, for decades, people weren't doing much researchy stuff with deep learning. And one of the reasons was that none of us could be bothered implementing a accelerated version of every damned, you know, CUDA operation that we needed to do the backward pass of and do the calculus, blah, blah, blah.

Nowadays, we only work with a subset of things that like PyTorch and stuff already supports. So at the point where-- so this is the thing about why we're doing this stuff with Swift now is that this is the foundations of something that in the next year or two will give us all the way down infinitely hackable, fully differentiable system.

Can we jump to layer really quick? So we've talked about MatMul, we've talked about Autodiff. Now let's talk about other stuff. So layers are now super easy, it just uses all the same stuff you've seen. And so if you go look at layer, it's a protocol, just like we were talking before.

And layers are differentiable, like they contain bags of parameters, just like we just saw. The requirement inside of a layer is you have to have a call. So layers in Swift are callable, just like you'd expect. And they have-- they work with any type that's an input or output.

And what layer says is the input and output types just have to be differentiable. And so layer itself is really simple. Yeah. And so underneath here, you can see us defining a few different layers. So for example, here is the definition of a dense layer. And so then now that we've got our layers and we've got our forward pass, that's enough to actually allow us to do many batch training.

And I'm not going to go through all this in any detail, other than just to point out that you can see here is defining a model. And it's just a layer, because it's just a differentiable thing that has a call function. And you can call the model layer. We can define log softmax.

We can define negative log likelihood, log sum X. Once we've done all that, we're allowed to use the Swift for TS version, because we've done it ourselves. And at that point, we can create a training loop. So we all define accuracy, just like we did in PyTorch, set up our mini-batch, just like we did in PyTorch.

And at this point, we can create a training loop. So we just go through and grab our X and Y and update all of our things. You'll notice that there's no torch.nograd here, and that's because in Swift, you opt into gradients, not out of gradients. So you wrap the stuff that wants gradients inside value with gradient.

And there we go. So we've got a training loop. Now one really cool thing is that all of these things end up packaged up together, thanks to the layer protocol, into a thing called variables. Which layer is differentiable? Differentiable is also a protocol. Protocols have lots of cool stuff on them.

So thanks to that, we don't have to write anything else. We can just say model.variables minus equals LR times grad, and it just works. Thanks to the magic of protocol extensions, our model got that for free, because we said it's a layer. Okay, so I think that's about all we wanted to show there.

So now that we've got that, we're actually allowed to use optimizers, so we can just use that instead. And that gives us a standard training loop, which we can use. And then on top of that, we can add callbacks, which I won't go into the details, but you can check it out in 0.4.

And you will find that, let's find them, here we go. We'll find a letter class, which has the same callbacks that we're used to. And then, eventually, we'll get to the point where we've actually written a stateful optimizer with hyperparameters, again, just like we saw in PyTorch. And most of this will now look very familiar.

We won't look at dictionaries now, but they're almost identical to PyTorch dictionaries, and we use them in almost the same way. So you see, we've got states and steppers and stats, just like in PyTorch. And so, eventually, you'll see we have things like the lamb optimizer written in Swift, which is pretty great.

And it's the same amount of code. And things like square derivatives, we can use our nice little Unicode to make them a bit easier to see. And so now we have a function created SGD optimizer, a function to create an atom optimizer. We have a function to do one-cycle scheduling.

And thanks to Matplotlib, we can check that it all works. >> Yep. >> So it's all there. >> So this is really the power of the abstraction, coming back to one of the earlier questions of earlier today we started in C. And we were talking about very abstract things like protocols and how things fit together.

But when you get those basic things -- and this is one of the reasons why learning Swift goes pretty quickly -- you get the basic idea, and now it applies everywhere. Yeah, and here we are doing mix-up. And so now we're in 10. And here we are doing label smoothing.

And to say it's really very similar-looking code to what we have in PyTorch. So then by the time we get to 11, other than this hacky workaround for the fact we can't do control flow differentiation yet, coming very soon, our XResNet, as you've seen, looks very similar, and we can train it in the usual way.

And there we go. So we've kind of started with nothing. And Chris spent a couple of decades for us, first of all building a compiler framework and then a compiler and then a C compiler and then a C++ compiler and then a new language and then a compiler for the language.

And then we came in and -- >> Let me correct you on one minor detail here. >> Some people helped you? >> I did not build all this stuff. Amazing people that I got to work with built all of this stuff. And likewise, all of these workbooks were built by amazing people that we were lucky enough to work with.

>> Yeah, absolutely. So that's what happened. And then let's look at -- so it's kind of like, thanks to all that work, we then got to a point where, 18 months ago, you and I met, you just joined Google, we were at the TensorFlow symposium, and I said, what are you doing here?

I thought, you're a compiler guy, and he said, oh, well, now I'm going to be a deep learning guy. >> Well, deep learning sounds really cool. >> Yeah. >> He hadn't told me it was uncool yet. >> Yeah. So then I complained about how terrible everything was, and Chris said -- so Chris said, I've got to create a new framework.

I was like, we need a lot more of a new framework, you know, I described the problems that we've talked about with, like, where Python's up to, and Chris said, well, I might actually be creating a new language for deep learning, which I was very excited about because I'm totally not happy with the current languages we have for deep learning.

So then 12 months ago, I guess we started asking this question of, like, what if high-level API design actually influenced the creation of a differentiable programming language? What would that mean? >> And so to me, one of the dreams is when you connect the building of a thing with teaching of a thing with using a thing in reality.

And one of the beautiful things about FastAI is pulling together, both building the framework, teaching the framework, and doing research with the framework. >> Yeah. So next time we caught up, I said, maybe we should try writing FastAI in Swift. And you're like, we could do that, I guess.

I was like, great. >> Well, so I think the one thing before this, I'm like, hey, Jeremy, it's starting to work. >> Yeah. >> And he says, oh, cool, can we ship it yet? I'm like, it's starting to work. >> And it's a high-level API. >> Yes. >> So that's the course where we teach people to use this thing that doesn't exist yet.

>> And I think I said naively, I like deadlines. Deadlines are a good thing. They force progress to happen. >> So then one month ago, we created a GitHub repo. And we put a notebook in it. And we got the last TensorFlow Dev Summit. We sat in a room with the Swift for TensorFlow team.

And we wrote first line of the first notebook. And you told your team, hey, we're going to write all of the Python book from scratch. And they basically said, what have you gotten us into? And I think we've learned a lot. >> Yeah. So, I mean, to me, the question is still this, which is, what if high-level API design was able to influence the creation of a differentiable programming language?

And I guess we started answering that question. >> Yeah. I don't think we're there yet. I mean, I think that what we've learned even over the last month is that there's still a really long way to go. And I think this is the kind of thing that really benefits from different kinds of people and perspectives and a different set of challenges.

And just today and yesterday working on data blocks, a breakthrough happened where there's an entirely new way to reimagine it as this functional composition that solves a lot of problems. And a lot of those kinds of breakthroughs, I think, are still just waiting to happen. >> I mean, it's been an interesting process for me, Chris, because we decided to go back and redo the Python library from scratch.

And as we did it, we were thinking, like, what would this look like when we get to Swift? And so even as we did the Python library, we created the idea of stateful optimizers. We created the new callbacks version too. So that was interesting. But that's also been interesting, I've seen, as an outsider from a distance, that Swift syntax seems to be changing thanks to some of this.

>> Yeah, absolutely. So there are new features in Swift, including callables. That's a thing that exists because of Swift for DensorFlow. The Python interoperability, believe it or not, we drove that because it's really important for what we're doing. There's a bunch of stuff like that that's already being driven by this project, and I think there's going to be more.

And so, like, making it so float can default away to nothing. That's really important. We have to do that. And otherwise, it wouldn't have been a priority. >> So I mean, so it's still really, really early days. And I think the question, in my mind, is now, like, what will happen when data scientists in lots of different domains have access to an infinitely hackable, differentiable language along with the world of all of the C libraries, you know, like, what do we end up with?

Because we kind of -- we're starting from very little in terms of ecosystem, right? But, like, there are things in Swift -- we haven't covered, for example, something called K paths, but there's this thing called K paths, which might let us write, like, little query language DSLs in Swift with type safety.

>> Yeah, give me all the parameters out of this thing and let me do something interesting to them. >> Yeah. >> It's really cool. >> And so, you know, I guess at this point, I'm kind of saying that people is like, pick some piece of this that might be interesting in your domain.

And over the next 12 to 24 months, explore with us, so that, you know, as Chris said, putting this airplane together whilst it's flying, you know, by the time it's -- actually, all the pieces are together, you'll have your domain-specific pieces together, and I think it will be super exciting.

>> And one of the things that's also really important about this project is it's not cast in concrete. So we can and we will change it to make it great. And to me, we're very much in the phase of let's focus on making the basic ingredients that everybody puts things together, like, let's talk about the core of layer is.

Let's talk about what data blocks should be. Let's talk about what all these basic things are. Let's not mess with float anymore. Let's go up a few levels. >> While it's done? >> Yeah, let's -- we can consider float down. But let's actually really focus on getting these right so that then we can build amazing things on top of them.

And to me, the thing I'm looking forward to is just innovation. Innovation happens when you make things that were previously hard accessible to more people, and that's what I would just love to see. >> So the thing I keep hearing is, how do I get involved? So like, I think there's many places you can get involved, but like, to me, the best way to get involved is by trying to start using little bits of this in work that you're doing or utilities you're building or hobbies you have, you know, just try -- you know, it's not so much how do I add some new custom derivative thing into Swift and TensorFlow, but it's like, you know, implement some notebook that didn't exist before or take some Python library that you've liked using and try and create a Swift version.

>> Try something or write a blog post. So one of the things when Swift first came up is that a lot of people were blogging about their experiences and what they learned and what they liked and what they didn't like, and that's an amazing communication channel because the team listens to that, and that's a huge feedback loop because we can see somebody was struggling about it, and even over the last couple of weeks, when Jeremy complains about something, we're like, oh, that is really hard.

Maybe we should fix that, and we do change it, and then progress happens, right? >> Yeah. >> And so we want that feedback loop in blogs and other kinds of -- >> Yeah, it's a very receptive community, very receptive team, for sure. Were there any highlight questions that you wanted to ask before we wrapped up, Rachel?

Really? Okay. Well, I mean, let me say this, Chris. It's been an absolute honor and absolute pleasure to get to work with you and with your team. It's like a dream come true for me and to see what is being built here, and you're always super humble about your influence, but, I mean, you've been such an extraordinary influence in all the things that you've helped make happen, and I'm super thrilled for our little community that you've -- let us piggyback on yours a little bit.

Thank you so much for this. >> Oh, and from my perspective, as a tool builder, tool builders exist because of users, and I want to build a beautiful thing, and I think everybody working on the project wants to build something that is really beautiful, really profound, that enables people to do things they've never done before.

I'm really excited to see that. >> I think we're already seeing that starting to happen, so thank you so much, and thanks everybody for joining us. >> Thanks for having us. >> See you on the forums. (audience applauding)