Stanford XCS224U: NLU I Presenting Your Research, Part 1: Your Papers I Spring 2023
00:00:00.000 | Welcome everyone.
00:00:06.000 | This screencast kicks off
00:00:07.300 | our series on presenting your research.
00:00:09.520 | We're going to try to cover the full lifecycle of
00:00:11.900 | a project in the field from
00:00:13.480 | work you do for a course like this,
00:00:15.660 | on up to the day when you might be on stage at
00:00:18.000 | a top workshop or conference
00:00:19.560 | giving a talk about your research.
00:00:21.940 | Let's dive in. I just wanted to start with
00:00:24.240 | some practical notes about your papers for this course.
00:00:27.960 | Here are some links you might find useful.
00:00:30.120 | The links go to the website as well as to
00:00:32.480 | the projects page in the course code repository.
00:00:35.320 | I think I would single out that projects page
00:00:37.840 | as potentially especially useful.
00:00:40.000 | It's got FAQs about projects,
00:00:42.460 | it's got advice about the individual project components,
00:00:45.540 | as well as advice about publishing in the field in general.
00:00:49.240 | Also, I'm proud to say it now has
00:00:51.380 | an extensive list of published papers
00:00:54.280 | that began their lives as work for this course.
00:00:57.600 | I'm really proud of that list and I hope you find
00:00:59.720 | it inspiring to check that work out.
00:01:02.960 | This is a reminder about
00:01:06.080 | our overall perspective on project work for this course.
00:01:09.360 | I talked about this at length in
00:01:11.060 | our methods and metrics screencast,
00:01:13.120 | but I think it bears repeating
00:01:14.600 | because it is so fundamental.
00:01:16.680 | We will never evaluate a project for
00:01:19.200 | this course based on how good the results are.
00:01:22.140 | We recognize that there is a bias towards
00:01:25.080 | so-called positive results in
00:01:26.720 | the scientific literature in general and
00:01:28.760 | a bias away from so-called negative results.
00:01:31.680 | I think that's unfortunate, that bias.
00:01:33.960 | I feel like we're making progress as
00:01:35.860 | a scientific community in terms of getting
00:01:37.780 | people to value negative results,
00:01:40.480 | but it will be a long journey.
00:01:42.440 | For this course though, we're
00:01:44.000 | freed from all of those biases.
00:01:46.320 | We're not subject to any of
00:01:47.760 | the constraints that would motivate that,
00:01:49.560 | and so we can do the right and good thing of
00:01:52.040 | valuing positive results, negative results,
00:01:54.920 | and everything in between.
00:01:56.880 | Fundamentally, we're going to evaluate
00:01:59.440 | your work based on the appropriateness of your metrics,
00:02:02.600 | the strength of your methods,
00:02:04.200 | and the extent to which the paper is open and
00:02:06.600 | clear-sighted about the limitations of its findings.
00:02:10.040 | That really reflects our values.
00:02:12.480 | This does mean that if your paper
00:02:14.520 | reports top results on some leaderboard,
00:02:17.120 | but has chosen strange metrics and feels unmotivated,
00:02:20.360 | you will not necessarily do well on the final paper.
00:02:24.280 | Conversely, and this is more important,
00:02:26.480 | if you tried something really creative and
00:02:28.520 | ambitious and it didn't pan
00:02:30.600 | out in terms of results on some leaderboard,
00:02:33.080 | that matters hardly at all.
00:02:35.080 | You might have a really informative negative result on
00:02:38.280 | your hand and the whole scientific community
00:02:40.940 | would benefit from seeing it.
00:02:42.880 | There we're going to look to the strength of
00:02:44.680 | your methods and the evidence that you've
00:02:46.240 | got and that will carry the day.
00:02:49.840 | Papers for this course have
00:02:52.960 | a few special sections that come at the end.
00:02:56.020 | I thought I would just review those and talk
00:02:58.060 | in particular about the motivations.
00:03:00.080 | Let's start with the known project limitation section.
00:03:03.760 | The prompt here, imagine that your reader is
00:03:06.400 | a well-intentioned NLP practitioner who is seeking to
00:03:10.040 | make use of your data, models,
00:03:12.300 | or findings as part of a separate scholarly's project,
00:03:15.560 | deployed system, or some other real-world intervention.
00:03:19.420 | Have that person in mind and ask,
00:03:21.240 | what should such a person know about your work?
00:03:24.880 | Things you might think about,
00:03:26.600 | benefits and risks of the work,
00:03:28.680 | cost to your participants,
00:03:30.560 | to society, to the planet, and so forth.
00:03:33.360 | Responsible use of your data, models, and findings.
00:03:37.760 | You might be able to think of other things
00:03:39.640 | that should fall under this heading.
00:03:41.460 | I want to emphasize that I have asked you to have in
00:03:44.280 | mind a well-intentioned NLP practitioner.
00:03:46.900 | I think it's very hard to think through how to
00:03:49.640 | reach someone who is going to be a bad actor and try
00:03:52.080 | to apply your ideas for
00:03:53.760 | evil purposes or use them in some problematic way.
00:03:56.800 | Set that thing aside and just focused on the person who is
00:04:00.320 | trying to build productively
00:04:02.400 | on your ideas to do something good in the world.
00:04:05.040 | That person might have the best intentions,
00:04:07.840 | but not really appreciate where
00:04:09.360 | the limitations of your ideas lie.
00:04:11.560 | This is an opportunity to communicate
00:04:13.800 | directly with that person about the limitations.
00:04:16.760 | In doing that, I think you could save them a lot of
00:04:19.440 | grief, you could save their users a lot of grief,
00:04:21.760 | and ultimately, this seems like
00:04:23.400 | a really important thing for us to be doing in
00:04:26.200 | this era when our research can have such wide-ranging impacts.
00:04:30.600 | In this spirit, if you get really into this,
00:04:33.540 | you could think about doing things like data sheets,
00:04:36.320 | model cards, and impact statements.
00:04:38.900 | These are more extensive structured documents that again,
00:04:42.400 | help you with disclosures mainly to well-intentioned users.
00:04:46.840 | I didn't insist on them for
00:04:48.480 | this coursework because it is a lot of work,
00:04:50.480 | but if you think about releasing data and
00:04:52.440 | models out into the wider world,
00:04:54.680 | I think it would be great to confront all the issues
00:04:57.840 | that these structured documents ask you to confront.
00:05:01.720 | We also require an authorship statement.
00:05:05.880 | Again, this is about our scientific perspective,
00:05:08.680 | it is not about evaluation.
00:05:11.040 | Fundamentally, this statement should explain how
00:05:13.680 | the individual authors contributed to the project.
00:05:16.800 | You can include whatever information
00:05:19.000 | you deem important to convey.
00:05:21.040 | If you would like some examples,
00:05:23.040 | I recommend this document here,
00:05:24.960 | which is publication guidelines for PNAS.
00:05:27.720 | It includes some tips on good authorship statements.
00:05:31.120 | The rationale again is scientific.
00:05:33.700 | We think this is an important aspect of scholarship in general,
00:05:37.080 | especially in this era when we have large team papers.
00:05:41.260 | This is not about evaluation and it is not meant to be punitive.
00:05:45.960 | Only in extreme cases and after discussion with
00:05:49.000 | the entire team would we consider
00:05:51.560 | giving separate grades to team members based on this statement.
00:05:54.720 | It's not about grading,
00:05:56.200 | this is about how we publish and how we take credit for
00:05:59.800 | our ideas and how we explain
00:06:01.400 | the contributions of individual scientists.
00:06:05.000 | [BLANK_AUDIO]