Stanford XCS224U: Natural Language Understanding I Lit Review Overview I Spring 2023

00:00:00.000 | Welcome, everyone.

00:00:01.000 | This short screencast is an overview of the Lit Review.

00:00:04.000 | We've reached the project phase of this course,

00:00:07.000 | and I think the Lit Review is a really crucial element there.

00:00:10.000 | It's where you begin to build an intellectual foundation

00:00:13.000 | for your project.

00:00:14.000 | Fundamentally, for me, the rationale behind the Lit Review

00:00:18.000 | is about productive dialogue.

00:00:20.000 | We want you to enter into dialogue with your teammates

00:00:23.000 | and with yourself and with your mentors

00:00:25.000 | and with your colleagues.

00:00:27.000 | With your teammates and with yourself and with your mentor

00:00:30.000 | from the teaching team about what you're

00:00:32.000 | going to try to accomplish in the final project.

00:00:35.000 | So you should pose questions.

00:00:37.000 | You should identify obstacles and propose initial workarounds.

00:00:41.000 | You should find data sets and models and architectures

00:00:44.000 | and everything else, and you should think carefully

00:00:47.000 | about your resources.

00:00:49.000 | The idea here is to gather intel from the literature

00:00:52.000 | about what people are doing in your space

00:00:55.000 | and use that information to begin

00:00:57.000 | to carve out an original vision for your own final projects.

00:01:02.000 | The specific requirements are listed at the course website.

00:01:06.000 | You can follow the link at the top of the slide here.

00:01:09.000 | Let me just offer you a brief rundown.

00:01:11.000 | First, it's a roughly six-page document,

00:01:13.000 | and we say eight pages is the max so you don't go overboard,

00:01:16.000 | and that does not include the obligatory references section.

00:01:20.000 | We have a template that you can use.

00:01:22.000 | It's a LaTeX template based in the ACL format.

00:01:25.000 | You're not required to use it, but you might as well,

00:01:28.000 | and you'll begin to get used to dealing with that ACL format,

00:01:31.000 | which is required for the final paper.

00:01:35.000 | Groups of one should review five papers.

00:01:38.000 | Groups of two should review seven,

00:01:40.000 | and groups of three should review nine.

00:01:43.000 | I think you can hear in there a small incentive

00:01:45.000 | for doing group work, which we think is productive in general.

00:01:50.000 | The ideal is to have the same topic for your lit review

00:01:53.000 | and final project.

00:01:54.000 | Obviously, that's the most efficient,

00:01:56.000 | but we do grant that sometimes people finish their lit review,

00:02:00.000 | and that leads them to the realization that they don't want to work

00:02:03.000 | in that area anymore.

00:02:04.000 | That is perfectly fine, but then you should negotiate that

00:02:07.000 | with your mentor and with the members of your team

00:02:10.000 | to make sure you can have your actual project converge

00:02:13.000 | in the time available, which is always too short.

00:02:17.000 | Then we come to the major things to include,

00:02:20.000 | and you might as well use these phrases as section headings

00:02:23.000 | in the document to help your mentor understand

00:02:25.000 | what you're trying to do.

00:02:27.000 | General problem task definition, this is absolutely crucial.

00:02:30.000 | We want to know what kind of questions you're going to be asking,

00:02:33.000 | and you might begin to guide us toward things

00:02:35.000 | that will be crucial for your project.

00:02:38.000 | Then we want concise summaries of the articles.

00:02:41.000 | I will say we're not really looking for you

00:02:43.000 | to just summarize the content.

00:02:45.000 | The ideal summary is going to raise questions

00:02:48.000 | and identify issues that will be useful to you

00:02:52.000 | in thinking about the project work itself.

00:02:54.000 | Relatedly, you should compare and contrast these articles.

00:02:58.000 | How do they differ in terms of models and data

00:03:01.000 | and fundamental results?

00:03:03.000 | Because that too could help point the way toward space

00:03:06.000 | for an original contribution.

00:03:09.000 | Then really importantly, let's think about future work.

00:03:12.000 | What are you going to do next?

00:03:14.000 | How might this all come together into a project?

00:03:17.000 | It's never too early to begin that creative process,

00:03:20.000 | and the more you can do under this heading,

00:03:22.000 | the more productive your dialogue with your mentor will be,

00:03:25.000 | and then I think you'll be able to go farther.

00:03:28.000 | Then finally, so that you get in the habit of this

00:03:31.000 | and so that we can identify what literature items you're reviewing,

00:03:34.000 | we want an obligatory references section.

00:03:38.000 | In terms of actually conducting lit review searches,

00:03:42.000 | I do have some tips that I have found really productive over the years.

00:03:46.000 | Here's the kind of cycle that I still use regularly.

00:03:50.000 | Search with some keywords in the ACL anthology,

00:03:53.000 | Google Scholar, or Semantic Scholar.

00:03:56.000 | Because it's NLP, I recommend starting with the ACL anthology.

00:03:59.000 | A wonderful aspect of the NLP community is that it is

00:04:02.000 | very well organized when it comes to its literature.

00:04:05.000 | Essentially, all of the work that's published by the ACL

00:04:09.000 | is accumulated into this anthology, which has good Google search,

00:04:13.000 | good bib entries, abstracts, links to the papers, you name it.

00:04:18.000 | So that's a good first stop, especially if you're working on a core topic in NLP.

00:04:23.000 | But don't limit yourself to that.

00:04:25.000 | You should branch out also and check Google Scholar and Semantic Scholar.

00:04:29.000 | In this course, we really value interdisciplinary work,

00:04:32.000 | and so you want to connect with other literatures besides NLP in many cases.

00:04:37.000 | Next step, download relevant and/or highly cited results

00:04:42.000 | and check out their abstracts and related work sections.

00:04:45.000 | These are heuristics.

00:04:46.000 | Relevant will be delivered by the search engine according to your keywords,

00:04:50.000 | and then highly cited is just a good heuristic

00:04:53.000 | for finding things that have been influential.

00:04:55.000 | You shouldn't depend on it, but there's no doubt that it's a useful piece of information.

00:05:00.000 | When you do this, you're seeking out key questions and techniques

00:05:04.000 | and also other highly cited papers.

00:05:06.000 | You should not read entire papers at this point.

00:05:09.000 | That is tremendously inefficient.

00:05:11.000 | There are too many papers out there, so you need to use your time wisely.

00:05:15.000 | So the idea here is to get a feel for these papers

00:05:18.000 | and also get a sense for what else they are citing.

00:05:23.000 | Download the papers that you see prominently in the related work section

00:05:26.000 | and kind of add those to the set that you downloaded as part of your core search.

00:05:32.000 | And then return to step one with some new keywords

00:05:36.000 | that you gathered as part of the searching that you did.

00:05:39.000 | And you should keep going on that loop and break out of it

00:05:43.000 | when you have a sense for what you're doing and what others have done in the area.

00:05:47.000 | You'll start to iterate around in a few papers that you think are clearly important.

00:05:52.000 | Maybe some new directions will be suggested by the other papers that are in your set.

00:05:56.000 | Now you've got the basis for thinking about a selection for the lit review.

00:06:01.000 | At that stage, you select some core papers from that downloaded set.

00:06:05.000 | And finally, you read those deeply and you cover those in the lit review.

00:06:09.000 | Notice you do that only at the final stage so that you can learn as much as you can

00:06:13.000 | by kind of surveying widely in a lightweight way.

00:06:17.000 | And then you go deep once you have a sense for where to invest.

00:06:22.000 | This is sort of amusing, a plagiarism policy.

00:06:25.000 | It's especially meta-feeling for us because, after all, we study large language models.

00:06:30.000 | And there's a growing concern in academia that these language models will make it harder for us

00:06:35.000 | to assess student writing.

00:06:37.000 | Let's try to embrace this a little bit.

00:06:40.000 | So I did do a search based on how the Electra paper relates to the Transformers paper.

00:06:45.000 | This is with GPD 4, and I will confess to you that what came back looks awfully useful to me.

00:06:50.000 | This does look like kind of raw information that you could use to inform a lit review.

00:06:56.000 | So that's all to the good.

00:06:59.000 | Make sure you know the course policy, though.

00:07:01.000 | It's linked here.

00:07:02.000 | And what it essentially says is there's no rule against using an AI assistant like GPT-4

00:07:07.000 | to help you with your lit review.

00:07:10.000 | But all output from the model needs to be quoted.

00:07:13.000 | That's per the policy.

00:07:14.000 | You treat it like any other resource.

00:07:16.000 | And, of course, assignments that are just quotations from any resource are not going to do well

00:07:22.000 | in terms of evaluation.

00:07:25.000 | Assignments with substantial overlap in prose will be scrutinized for plagiarism.

00:07:30.000 | And what that means is that if two groups used a similar prompt and got back similar results

00:07:36.000 | and included them in the lit review unquoted, they'd probably get nabbed for plagiarism,

00:07:40.000 | not because they used a model, but because the two assignments look too much alike.

00:07:45.000 | And at that point, it's not the language model that we're implicating here,

00:07:49.000 | but rather the standard sort of thing that we see when we worry about plagiarism.

00:07:54.000 | So I would suggest using these assistants not to produce raw prose for you,

00:07:59.000 | but rather to help you figure out what's in the literature.

00:08:02.000 | And you'll want to be skeptical consumers because while I think this is a pretty good description

00:08:07.000 | of Elektra and Transformers, I haven't thoroughly audited it,

00:08:11.000 | and I wouldn't include it anywhere, even paraphrased by me,

00:08:15.000 | until I had given it a thorough audit to make sure that it was all factually correct.

00:08:20.000 | Because that's the ultimate thing that we're looking for,

00:08:22.000 | never mind where all this prose came from.

00:08:25.000 | But the idea here is that there's obviously value to these things.

00:08:28.000 | They could supercharge certain aspects of research, so we don't want to ban them.

00:08:32.000 | After all, we think they're really interesting artifacts.

00:08:34.000 | That's why we're in this course.

00:08:36.000 | But we want to use them with caution, and we want to make sure that they don't end up

00:08:41.000 | kind of producing really bad scholarship.

00:08:43.000 | That is the fundamental thing that we're watching out for.

00:08:48.000 | That's it for the lit review.

00:08:50.000 | It's worth thinking ahead to the next document,

00:08:52.000 | which is a bit more unusual in the context of this course.

00:08:55.000 | That's the experiment protocol.

00:08:57.000 | This is a short, structured report designed to help you establish your core experimental framework.

00:09:04.000 | And the required sections are listed here, hypotheses, data, metrics, models,

00:09:09.000 | general reasoning, summary of progress, and references.

00:09:12.000 | You can see that that's kind of the raw materials for a project in this space.

00:09:17.000 | And we're trying to look to see whether anything is missing,

00:09:20.000 | and whether there are any other obstacles that would prevent your project

00:09:24.000 | from converging in the time available.

00:09:27.000 | The idea is clarity around project goals, identification of obstacles, and project risks.

00:09:33.000 | So you should be erring on the side of disclosing too much in the interest of making sure

00:09:38.000 | we overcome all the obstacles and fill in all the gaps so that the project succeeds in the end.

00:09:44.000 | That's for the protocol, but you might as well be thinking along these lines for the lit review.

00:09:48.000 | It is never too early to begin brainstorming about exactly what the final project is going to look like.

00:09:55.000 | Even at the lit review stage, you can start to get a feel for what hypotheses are interesting,

00:10:00.000 | what techniques you want to try, and so forth and so on.

00:10:03.000 | So the earlier the better. That's what all of these preliminary project assignments are about.

00:10:08.000 | Thank you.

00:10:09.000 | [BLANK_AUDIO]

Stanford XCS224U: Natural Language Understanding I Lit Review Overview I Spring 2023

Chapters