Stanford XCS224U: NLU I Presenting Your Research, Part 3: NLP Conference Submission I Spring 2023

00:00:00.000 | Welcome back everyone.

00:00:06.160 | This is part three in our series on presenting your research.

00:00:09.200 | We're going to talk about the process of publishing an NLP,

00:00:11.960 | which really comes down to the NLP conference submission process,

00:00:16.000 | given that so many of the top venues in the field are actually conferences.

00:00:20.660 | I'll say at the start that I think there's a lot to like about the publishing process in NLP,

00:00:24.880 | but it also has its truly bewildering aspects.

00:00:28.200 | So I'm going to try to be a kind of trusted guide for you through this often strange process.

00:00:33.700 | Let's begin with one of the most unusual things of all, the ACL anonymity period.

00:00:39.960 | The ACL conferences have adopted a uniform policy that submitted papers cannot be uploaded to repositories like archive,

00:00:47.620 | or made public in any way, starting one month from the submission deadline and extending through the time when decisions go out.

00:00:55.580 | For specific conferences, check their sites for the precise date when this embargo goes into effect.

00:01:00.820 | And I'd go one step further. I would say, check the precise hour when the policy goes into effect,

00:01:06.400 | and make sure you know exactly what they mean by posting time.

00:01:10.580 | Is it the time that you submitted or the time it appeared?

00:01:13.580 | As you get very close to this deadline, you might really care about these fine details.

00:01:19.540 | The policy is an attempt to balance the benefits of free and fast distribution of new ideas and new results against the benefits of double blind peer review.

00:01:29.540 | I think the scenario that guides this is the one where you imagine a reviewer gets their assignments and they check out the paper titles,

00:01:36.700 | and then they look over at their Twitter window and they learn from some tweets exactly who wrote one of their papers and what institution it comes from.

00:01:45.360 | And then they feel conflicted as reviewers.

00:01:48.200 | We want to prevent that from happening while balancing this idea of free expression of ideas.

00:01:54.200 | For more on the policy and its rationale, I refer you to the page that I've linked at the bottom of this slide.

00:02:01.000 | Let me editorialize for a minute. I think that the ACL anonymity period idea was a noble experiment, but it's a failed experiment.

00:02:09.600 | We should move on from it. There are a few reasons I feel this way.

00:02:13.740 | First, you have a lot of unproductive discussions around exactly what it means to have the period kick in and whether you met it

00:02:20.940 | and what you can do during that embargo period in terms of talks and so forth.

00:02:25.600 | That's just a messy process. I'm more worried about the fact that if you post a paper and discover it has a mistake,

00:02:32.600 | you can't update the paper until the embargo period ends.

00:02:35.940 | And that's just straightforwardly unproductive because now we have known false results out there that can't be updated,

00:02:42.360 | when in fact we should want to correct the record as fast as possible.

00:02:46.640 | And the third thing is that I think it is just fed into the hype cycle.

00:02:50.740 | What people do now to avoid the embargo is publish very quickly on archive to meet the deadline, even if the ideas are half-baked.

00:02:59.340 | So it's a lot of flag planting and it's leading people to feel ever more frantic about these deadlines,

00:03:04.800 | when what we want instead is a more deliberative pace for the field.

00:03:12.400 | With the anonymity period out of the way, let's suppose that you are now going to take the act of submitting.

00:03:17.980 | You submit your paper along with area keywords that help determine which committee gets your paper.

00:03:23.840 | This will depend very heavily on the conference and the year. In terms of picking keywords,

00:03:28.140 | it would be great if you found an expert who could consult with you a little bit on which ones to pick

00:03:33.780 | so that you get the optimal reviewers for your paper.

00:03:38.140 | Increasingly, you also need to fill out very long and complicated checklists for various things the community cares about.

00:03:45.240 | Try to find an expert to help with this. These are often very extensive checklists with complicated questions.

00:03:52.140 | If you fully invested in them, you could spend 12 hours filling out the form.

00:03:56.540 | It might be that no one is really going to study the form. That might be a misallocation of your own time and energy.

00:04:02.180 | So you could have the expert tell you which questions are likely to matter

00:04:05.900 | and which ones can be filled out with sort of basic pointers into the content of the paper.

00:04:12.040 | Reviewers, once the submissions are all in, scan a long list of titles and abstracts and then bid on which ones they want to do.

00:04:19.640 | At this stage, the title is probably the primary factor in bidding decisions.

00:04:23.500 | The reason I say that is that as a reviewer, you might be facing a list of 500 submissions.

00:04:29.640 | It's impossible for you to read all the abstracts in the time allotted.

00:04:33.880 | And so you have to go by the titles.

00:04:35.540 | And what that means for you as an author is that the title might be a primary mechanism

00:04:39.900 | for you to connect with the reviewers that you're looking for. So think carefully about the title.

00:04:45.740 | The program chairs assign reviewers their papers, presumably based in large part on their bids,

00:04:50.880 | although having bid on lots of conferences, I have very little evidence that my bids are actually shaping which things get assigned to me.

00:04:58.300 | But I still do my bidding. Reviewers read the papers, write comments and supply ratings.

00:05:04.180 | I'll give you a glimpse of what that's like in a second. Authors are allowed to respond briefly to the reviews.

00:05:10.800 | This is the so-called author response period. It's a very tight time window in which you have a very constrained amount of text to use to respond to the reviews,

00:05:20.240 | object to mistakes, answer questions and things like that.

00:05:24.880 | Then the program or area chair stimulates discussion among the reviewers about conflicts between the reviewers or conflicts with the author response.

00:05:33.500 | And in general, the idea is that this discussion will get clarity on whether the paper should be accepted.

00:05:39.940 | And then finally, the program committee does some magic to arrive at the final program based on all of this input.

00:05:47.580 | You might get a meta review that provides some insight into the final decision making.

00:05:51.740 | But I can more or less guarantee for you that in the end, the decision making might be pretty opaque, especially in the final stages.

00:06:00.440 | That's what we have to live with, I guess, when we enter into the sort of lottery like NLP conference set up for reviewing.

00:06:09.380 | Here's a look at the form that reviewers are typically dealing with. You know, these used to be based mainly in structured data.

00:06:15.800 | Now it's more like structured text. The first box will ask, what is the paper about? What contributions does it make?

00:06:22.700 | And what are the main strengths and weaknesses? This is a great question because I think it's trying to check in with a reviewer and make sure they actually know what the paper is about and what its contributions are.

00:06:34.380 | After that come the important pieces. Reasons to accept. Reasons to reject.

00:06:39.780 | For better or worse, both of these will be required boxes and reviewers will think of something to type in both.

00:06:46.380 | I resist this, particularly for reasons to reject. I have often typed into that box.

00:06:51.180 | I have no reasons to reject for papers I simply wanted to support, but I had to enter some text in order to get the form to be accepted.

00:06:59.600 | Alas. Maybe a box for questions and additional feedback for the authors.

00:07:04.820 | This is where the reviewers might pose direct questions that they're hoping will get resolved as part of the author response period.

00:07:11.280 | Missing references. This can be kind of touchy. You want to cite all the work that you can discover and people feel really slighted when things are missing.

00:07:19.620 | And that can often lead you to get some points off with reviewers.

00:07:23.680 | Typos, grammar, style and presentation improvements. You would hope that this was kind of secondary.

00:07:29.060 | And then crucially the ratings. An overall recommendation, which is maybe the primary data point that shapes what happens to your paper in the submission process.

00:07:38.360 | And then reviewer confidence, which you would hope was kind of calibrating on how much the recommendation should contribute to the overall judgment.

00:07:47.540 | And then there'll be a text box for confidential information.

00:07:50.240 | You'll also have such a text box as part of your author response.

00:07:53.880 | And this is your chance to communicate directly with the leadership of the conference about problems that might have arisen.

00:08:00.160 | And reviewers have the same option.

00:08:03.720 | Author responses. A complicated and often bewildering part of this.

00:08:08.680 | Many of these conferences allow authors to submit short responses to the reviews.

00:08:13.820 | This is often done on a tight time window and you have a very limited number of characters you can enter.

00:08:19.520 | Overall, this is a rather uncertain business, but I think I can offer some advice.

00:08:24.800 | First, many people are cynical about author responses since reviewers rarely change their scores afterwards.

00:08:31.440 | I understand that, but my overall perspective is a bit more optimistic.

00:08:36.480 | Even given one, though, item two, it's bad in terms of signaling not to submit a response at all.

00:08:42.480 | It sort of signals to the group that you don't care.

00:08:45.200 | I realize this sounds kind of dismal because I'm saying you have to submit something, otherwise you'll get in trouble.

00:08:51.280 | But whatever you do submit might not have an impact.

00:08:54.600 | I think there is unfortunately some truth to that.

00:08:56.680 | But as I said, I'm optimistic in particular for conferences that have area chairs who are tasked with stimulating discussion and writing meta reviews for a small number of papers.

00:09:07.240 | The author response might have a major impact.

00:09:09.600 | I say this as someone who has played the area chair role many times.

00:09:13.160 | I find the author response invaluable because I have the reviews, I have the paper, and I have now the author response.

00:09:20.640 | And I can balance them against each other and use that to stimulate discussion and get some clarity on what I should recommend for the paper as a whole.

00:09:29.440 | So I always benefit and I think I'm not alone.

00:09:32.440 | And I hope that gives you some motivation for thinking about investing in the author response.

00:09:39.040 | NLP conferences, as usual, have some complex rules about this, about what you can and can't say in your author response.

00:09:47.240 | Read the instructions.

00:09:48.360 | If you have questions about what you can do in a particular case, you should again seek out an expert.

00:09:53.760 | I do feel it's unfortunate that I have to at so many points here recommend that you find an expert because it sounds like a conspiracy of insiders.

00:10:02.040 | But that is the sad state of affairs.

00:10:04.280 | The number one thing that comes to mind here is that often there's a rule against reporting new results.

00:10:10.640 | I'm not sure why that rule exists.

00:10:12.120 | As an area chair, I want as much information as I can get about these papers, including new results if they're available.

00:10:18.080 | But they're often forbidden.

00:10:19.240 | But then the question is, what counts as a new result?

00:10:22.560 | And for that, you might want your expert to at least offer you some guidance.

00:10:28.160 | Always be polite.

00:10:29.600 | It can be hard, but it pays to be polite.

00:10:32.080 | Be firm and direct, but do that strategically to signal what you feel most strongly about.

00:10:38.360 | Never, ever, ever write things like, "Your inattentiveness is embarrassing.

00:10:43.160 | Section 6 does what you say we didn't do."

00:10:46.480 | You can write that out once if it would be cathartic, but then delete it and replace it with something like, "Thank you.

00:10:53.000 | The information you're requesting is in Section 6.

00:10:55.840 | We will make this more prominent in our revision."

00:10:58.760 | You're not giving any ground.

00:11:00.080 | You're making it clear what you had achieved, but you're also being polite and productive.

00:11:04.280 | And with luck, everyone feels validated by this process, and your scores get boosted a little bit.

00:11:12.920 | Presentation types and venues.

00:11:15.040 | There are kind of two dimensions to this.

00:11:16.520 | First, for the type, you have oral versus poster.

00:11:20.080 | Nobody knows how that's assigned.

00:11:21.880 | The conference people decide somehow, and you're told you're doing a poster or an oral.

00:11:26.640 | I don't think this is so consequential at this point because what matters is the papers.

00:11:31.680 | For presentation venue, you have workshop versus main conference, and that is an important distinction.

00:11:36.840 | There are tradeoffs here.

00:11:38.360 | Workshops tend to be less prestigious and selective, but the bright side is that you get to connect with a community of like-minded people.

00:11:47.360 | You might add a workshop, interact with people who become lifelong collaborators.

00:11:52.880 | On the flip side, the main conference will be highly selective and very prestigious, but it will be absolutely overwhelming.

00:12:00.400 | The number of people will be enormous.

00:12:02.400 | It will be in some big conference hotel.

00:12:04.640 | You might have real trouble connecting with people and finding people who work on what you do.

00:12:09.240 | Your presentation might get lost in the mix, all of that stuff.

00:12:13.080 | So you have to decide on what experience you're looking for and what your goals are.

00:12:17.920 | For relevant conferences, I've listed out a whole bunch of them here.

00:12:21.160 | The left column is the ACL conferences.

00:12:24.320 | The top three, ACL, NACL, and EMNLP, are, I think, by consensus, the most prestigious in the field.

00:12:32.120 | Just below them are AACL and EACL, but I think we're nearing the point where those are just in the top mix,

00:12:40.040 | and people are just looking for the next conference to submit to as the year goes around.

00:12:45.880 | Below those are Connell and Koehling, which are great.

00:12:50.200 | I think they're just second tier, but lots of important work appears there.

00:12:53.760 | I've published there.

00:12:54.760 | I actually think they're wonderful venues, and they give you a balance between a workshop and a main conference.

00:12:59.880 | And then, in terms of prestige, below them is workshops, but I do feel that that's a different enough dimension to all this

00:13:05.920 | that you should think about them, especially if you're thinking about a first venue for publishing.

00:13:11.680 | In the middle, I have conferences devoted to kind of the web and knowledge bases and artificial intelligence,

00:13:18.120 | cognitive science, linguistics.

00:13:20.000 | These are all venues that are great and that will welcome NLP-oriented submissions,

00:13:26.080 | and I think the most prestigious ones are kind of at the top there right now.

00:13:30.560 | And then, on the right, I have what I think are, by consensus, the top three machine learning conferences,

00:13:35.880 | and all of them are really welcoming of NLP research.

00:13:39.240 | You just might want to orient it toward a slightly broader audience,

00:13:42.960 | but those are certainly good homes for research you might be doing.

00:13:47.800 | Here is my personal assessment of NLP reviewing, another bit of editorializing.

00:13:53.520 | First, I would say that the focus on conference papers has been good for the field.

00:13:57.200 | It fits with and encourages a rapid pace, and even though I worry about the relentless pace of research,

00:14:03.160 | I do appreciate that things move fast and it's a very lively area intellectually.

00:14:09.400 | Before about 2010, the reviewing in the field was admirably good and rigorous in comparison with other fields.

00:14:15.800 | I say this as someone who is coming from linguistics and cognitive science.

00:14:19.800 | I was deeply impressed by the investment reviewers would make in these submissions,

00:14:24.720 | in that pre-deep learning era especially.

00:14:28.760 | Lately, though, the growth of the field has reduced the general quality of reviewing.

00:14:32.920 | I think, by consensus, we all see that this is happening.

00:14:36.560 | The field is still grappling with this.

00:14:38.120 | It was inevitable, as you get larger and larger numbers of submissions and reviewers,

00:14:43.400 | that the quality would go down as we move from a kind of trusted group of experts

00:14:47.800 | out into a much more diffuse and diverse population.

00:14:51.720 | So that is all feeling like this has more of a lottery aspect to it when you submit to these conferences.

00:14:58.120 | This next point, I wanted to just issue a warning and kind of connect with you on this.

00:15:03.080 | Reviewers are occasionally incredibly mean.

00:15:07.240 | You need to kind of desensitize yourself to this,

00:15:10.480 | and one way to do that would be to share your reviews and your experiences with an experienced NL peer.

00:15:15.480 | Most likely, that person will say, "I'm really sorry that you had to get this feedback.

00:15:20.400 | Do not take it personally. We've all seen this."

00:15:23.280 | I worry about this point.

00:15:24.720 | Reviewers can be mean enough that I worry it could turn people off of the field entirely.

00:15:29.600 | It could send a signal, especially to people new to the field, that they don't belong,

00:15:33.960 | and I think that is tragic.

00:15:36.640 | Don't do this if you're asked to review.

00:15:38.960 | Be kind.

00:15:39.800 | Imagine what it would be like to be on the receiving end,

00:15:43.040 | and then if you do receive this very mean feedback, don't take it personally.

00:15:47.720 | It's really hard for me to say this because the whole point is that you've really invested in your research,

00:15:52.680 | and so to say that you shouldn't take it personally feels like a conflict on that point,

00:15:57.800 | but you have to kind of get to that point.

00:15:59.640 | But please, please, please, if someone is mean to you, just try to move on from it and keep contributing.

00:16:06.440 | It's more about the reviewers than it's about you.

00:16:10.440 | This is more mundane.

00:16:11.480 | Forcing every paper to be four or eight pages that is short and long submission is not good,

00:16:17.520 | but luckily the issue is being addressed productively with more use of supplementary materials.

00:16:22.680 | I love that we're moving into a mode of science where the papers are short and the appendices run to 30 pages

00:16:28.720 | so that if you need information about the details, you yourself can do a deep dive on the appendix.

00:16:34.480 | But if you're just trying to understand the ideas and contribution, you can read a short paper.

00:16:41.120 | The biggest failing of all of this is that there is no chance for authors to appeal to an editor and interact with that editor.

00:16:48.040 | That's a missing piece that journals provide and to good effect.

00:16:52.160 | There is one area for this.

00:16:53.400 | Transactions of the ACL is a journal that follows the standard ACL conference model fairly closely,

00:16:59.680 | but allows for journal-style interaction with the editor.

00:17:02.560 | I've been heavily involved with TACL over the years as an action editor and as a reviewer,

00:17:07.480 | and I think it is a shining example of what reviewing in the field could be if we scaled the model up to the level of our conferences.

00:17:18.920 | On titles, this is a smaller point.

00:17:21.560 | I'm using this paper.

00:17:23.480 | I'm using titles in scientific journals and article citation, which you will notice is not a jokey paper title.

00:17:29.640 | Jokey paper titles are somewhat risky.

00:17:32.720 | You should calibrate the title to the scope of your contribution,

00:17:36.520 | and you should consider especially the reviewers you're likely to attract as they are scanning that long list of papers to bid on.

00:17:45.000 | Avoid special fonts and formatting because they just won't travel with your paper,

00:17:49.280 | and so it can get kind of awkward, although I do have a soft spot for some of the emojis people have put in their titles recently.

00:17:57.480 | Abstracts, this is also important.

00:17:59.520 | This will create a first impression for your work.

00:18:02.120 | Here is a kind of template for thinking about these documents.

00:18:05.520 | The opening is a broad overview, a glimpse at the central problem.

00:18:10.000 | The middle takes concepts mentioned in the opening and elaborates on them,

00:18:14.160 | probably by connecting with specific experiments and results from the paper.

00:18:17.640 | This is the meat of it.

00:18:19.640 | The close establishes links between your proposal and broader theoretical concerns

00:18:24.760 | so that the reviewer has an answer to the question, "Does the abstract offer a substantive and original proposal?"

00:18:32.280 | So that's a way to think about this here.

00:18:33.880 | It isn't kind of abstract.

00:18:35.560 | This opening sentence situates you, dear reader.

00:18:39.480 | "Our approach seeks to address the following central issue," then you spell out the techniques we used.

00:18:45.520 | "Our experiments are these," and you give details,

00:18:47.880 | and then, "Overall, we find that our approach has the following properties and the significance of this is."

00:18:53.800 | If you can conceptually fill this out with details from your project, I think you're in a good place,

00:18:58.880 | and then you could decide to get a little creative if you want to.

00:19:03.280 | Style sheets or, on avoiding desk rejects, this is mundane but important,

00:19:08.480 | pay attention to the details of the style sheet and other requirements included in the call for papers.

00:19:13.680 | They change these details all the time, almost like they're running some kind of experiment on us,

00:19:19.280 | and you don't want to get rejected for a small infraction, right?

00:19:22.960 | If you break one rule, you could get the dreaded desk reject,

00:19:26.240 | which would be a tragic but short way to end your conference submission process for this round.

00:19:31.600 | Try to avoid that.

00:19:33.520 | And then, finally, the camera-ready version.

00:19:35.600 | This is such a comical phrase.

00:19:37.200 | It's so weird that camera-ready survives.

00:19:39.840 | This is like talking about dialing your phone and so forth.

00:19:43.200 | What this refers to is antiquated technology.

00:19:46.960 | They used to literally take a picture of your paper, and that would be the publication version.

00:19:51.520 | That was the camera-ready part.

00:19:53.520 | Now it just sticks around as a weird idiom.

00:19:56.400 | For most NLP conferences, you get an additional page on acceptance,

00:19:59.920 | presumably to respond to requests made by reviewers,

00:20:02.960 | though in practice you can use the space however you like.

00:20:06.000 | Do write by your reviewers if they had concerns.

00:20:08.960 | It's great to address them, but it is not a contract that you have with the reviewers.

00:20:13.200 | Ultimately, at this stage, you should do what you feel is right for your paper.

00:20:17.520 | You have this extra page.

00:20:19.200 | Use it wisely.

00:20:20.320 | Use it thoughtfully.

00:20:21.280 | In general, the extra page is probably for fixing things that were overly terse

00:20:26.640 | because you compressed what was a long paper down to a short one,

00:20:29.760 | but it's great if you have space to express some new ideas and so forth.

00:20:34.560 | However, err on the side of saving brand new results for subsequent publications.

00:20:40.720 | Your paper is already crammed enough, I'm guessing,

00:20:44.080 | and so you don't want to add more things in just because you've discovered them.

00:20:47.120 | It might be a good idea to publish the current paper

00:20:50.080 | and then begin thinking about the next contribution that you'll make.

00:20:54.560 | Thank you.

00:20:55.060 | [BLANK_AUDIO]

Stanford XCS224U: NLU I Presenting Your Research, Part 3: NLP Conference Submission I Spring 2023

Chapters