Okay so I had an idea. I thought maybe what we could do is actually put together a few videos and what we'll do in the next few videos is build out a full Q&A project. And this is something that I've actually been wanting to do for quite a while now and I think it will actually be really cool.

I was thinking through it this morning and I thought okay today we're going to start it. So what you can see on the screen right now is the github repo for this project which is completely empty almost. There's a readme, a gitignore and this one architecture drawing that I set up literally you can see four minutes ago.

So I'm going to show you that and I'm going to explain what we're actually going to go through and I think we're going to cover a lot of different things. So I think that is one of the reasons why I think this is such a cool project. So at the moment this is like the basic architecture of what we would need to build a Q&A model and the end goal is to have a front end which looks kind of like this.

So we'll have like a search bar here and we'll have some visual up here. It's going to be a little bit better than the sigma and I'm going to show you what I already have and maybe we can use that or maybe we'll do something different I don't know.

And what we're going to be able to do is ask a question here and we're going to be able to answer the question based on stoic philosophy books. So I haven't really read any of these, I've read like little bits but they're pretty interesting and I think quite unique.

So as far as I know there's definitely not anything like this out there at the moment where you ask a question and you get the answer back from some ancient stoic philosophy book. And there's only really two books that I've thought of so far which is Meditations by Marcus Aurelius and Letters from a Stoic by Seneca.

And the good thing with both of these is that we can find both of them online for free so we can use Python requests to get these. So I'll just kind of put a little list here of what I think we're going to need. So the first one is actually extracting and downloading this data so we'll be using requests for that.

And then once we have actually got that data we need to pre-process it and when we're pre-processing it I think that will just be a case of using Regex more than anything else but I'm not sure yet so let's see. So after we pre-process it and that's when we get into this stuff over here.

So this whole sort of stack that you can see without the API. So this is a typical, it's called a reader or retriever reader and what we do is we use this up here, this is our database, it's a document store, Elasticsearch document store and what we're going to do is feed all of these.

Sorry it's getting a little bit messy with the color so let me change it. So what we're going to do is take these and we're going to feed them into our document store and once we have that what we want to do is build this retriever reader stack and it will allow us to query the retriever down here and what the retriever will do is send that query to Elasticsearch here which is what you can see happening there.

And returning from that we'll get so many different contexts. So all of the text from meditations and letters from Stoic, we'll split them up by maybe paragraph and saw them in here and what these contexts will be are the most relevant paragraphs. And once we've done that, this retriever will pass on the context to our reader model down here and what the reader model will do is say it's given a long sentence like this or paragraph, it will say okay the actual answer that you want is actually only these three words here and it will return those three words and what we want to do is return those three words in our answer back to our API.

But alongside the answer we're also going to include the full context here as well. So we get a few things back and I think that that's like going to be the more machine learning side of it but obviously we need to support all the machine learning side and I mean the very first part of that that you can obviously see here is the API.

So let me so let me write down so we have the the ML part and for that we're going to be using something called haystack. And once we get out of that part we move on, I'm just going to use a different color here, move on to our API.

The API we'll just we'll use probably fast API to set that up. Then once we set that up we go on to our front end part and the front end I don't I'm not a front end developer I just mainly use Python but I do know Angular a little bit.

So what we're going to do is build all of this part using Angular. So this will be our Angular front end and that's essentially everything we'll be covering but there's you know there's quite a lot in here. In particular as well what I've missed is alongside you know we have our reader model down here but what I want to try and do is rather than just taking the reader model from Hugging Face Transformers as we normally would, I want to actually try training it and for that we need to use something called MLM which is mass language modeling.

So we would need to train a BERT model using MLM or fine-tune the BERT model I should say on the data from our books up here and then we'd also want to train it so that it performs Q&A and for that we need to use the squad data set, probably squad anyway.

So you know there's quite a lot that I think we would have to do to build this and I think it'd be pretty interesting. So that is what we're going to be covering in sort of the next few videos and the one other final little thing. Okay so over here we have the Marcus Aurelius Sit-Man and I thought maybe something like this would be cool.

I don't know this is something I drew ages ago and this is Marcus Aurelius and I think something like this, maybe this or something like it would be pretty cool to just have in the middle of the web page and underneath we have a search bar and keep it pretty simple.

So I think that's everything really for the plan and I think the first thing we're going to do in the next video is actually sell requests and download that data and maybe pre-process it as well or they might be two videos. So that's everything for this video. I hope that you're as excited about this as I am because I'm really looking forward to actually building all of this.

I think it'll be super cool and I mean ideally at the end it's one it's going to look cool and two we're going to learn like a huge amount of stuff if you even put all this together there's so many different things that you need to know in order to make everything work.

So it should be really cool and I'm looking forward to getting started with it. So I will see you in the next video where we'll actually download the the data and pre-process it.