Welcome back everyone. This is part three in our series on presenting your research. We're going to talk about the process of publishing an NLP, which really comes down to the NLP conference submission process, given that so many of the top venues in the field are actually conferences. I'll say at the start that I think there's a lot to like about the publishing process in NLP, but it also has its truly bewildering aspects.

So I'm going to try to be a kind of trusted guide for you through this often strange process. Let's begin with one of the most unusual things of all, the ACL anonymity period. The ACL conferences have adopted a uniform policy that submitted papers cannot be uploaded to repositories like archive, or made public in any way, starting one month from the submission deadline and extending through the time when decisions go out.

For specific conferences, check their sites for the precise date when this embargo goes into effect. And I'd go one step further. I would say, check the precise hour when the policy goes into effect, and make sure you know exactly what they mean by posting time. Is it the time that you submitted or the time it appeared?

As you get very close to this deadline, you might really care about these fine details. The policy is an attempt to balance the benefits of free and fast distribution of new ideas and new results against the benefits of double blind peer review. I think the scenario that guides this is the one where you imagine a reviewer gets their assignments and they check out the paper titles, and then they look over at their Twitter window and they learn from some tweets exactly who wrote one of their papers and what institution it comes from.

And then they feel conflicted as reviewers. We want to prevent that from happening while balancing this idea of free expression of ideas. For more on the policy and its rationale, I refer you to the page that I've linked at the bottom of this slide. Let me editorialize for a minute.

I think that the ACL anonymity period idea was a noble experiment, but it's a failed experiment. We should move on from it. There are a few reasons I feel this way. First, you have a lot of unproductive discussions around exactly what it means to have the period kick in and whether you met it and what you can do during that embargo period in terms of talks and so forth.

That's just a messy process. I'm more worried about the fact that if you post a paper and discover it has a mistake, you can't update the paper until the embargo period ends. And that's just straightforwardly unproductive because now we have known false results out there that can't be updated, when in fact we should want to correct the record as fast as possible.

And the third thing is that I think it is just fed into the hype cycle. What people do now to avoid the embargo is publish very quickly on archive to meet the deadline, even if the ideas are half-baked. So it's a lot of flag planting and it's leading people to feel ever more frantic about these deadlines, when what we want instead is a more deliberative pace for the field.

With the anonymity period out of the way, let's suppose that you are now going to take the act of submitting. You submit your paper along with area keywords that help determine which committee gets your paper. This will depend very heavily on the conference and the year. In terms of picking keywords, it would be great if you found an expert who could consult with you a little bit on which ones to pick so that you get the optimal reviewers for your paper.

Increasingly, you also need to fill out very long and complicated checklists for various things the community cares about. Try to find an expert to help with this. These are often very extensive checklists with complicated questions. If you fully invested in them, you could spend 12 hours filling out the form.

It might be that no one is really going to study the form. That might be a misallocation of your own time and energy. So you could have the expert tell you which questions are likely to matter and which ones can be filled out with sort of basic pointers into the content of the paper.

Reviewers, once the submissions are all in, scan a long list of titles and abstracts and then bid on which ones they want to do. At this stage, the title is probably the primary factor in bidding decisions. The reason I say that is that as a reviewer, you might be facing a list of 500 submissions.

It's impossible for you to read all the abstracts in the time allotted. And so you have to go by the titles. And what that means for you as an author is that the title might be a primary mechanism for you to connect with the reviewers that you're looking for.

So think carefully about the title. The program chairs assign reviewers their papers, presumably based in large part on their bids, although having bid on lots of conferences, I have very little evidence that my bids are actually shaping which things get assigned to me. But I still do my bidding.

Reviewers read the papers, write comments and supply ratings. I'll give you a glimpse of what that's like in a second. Authors are allowed to respond briefly to the reviews. This is the so-called author response period. It's a very tight time window in which you have a very constrained amount of text to use to respond to the reviews, object to mistakes, answer questions and things like that.

Then the program or area chair stimulates discussion among the reviewers about conflicts between the reviewers or conflicts with the author response. And in general, the idea is that this discussion will get clarity on whether the paper should be accepted. And then finally, the program committee does some magic to arrive at the final program based on all of this input.

You might get a meta review that provides some insight into the final decision making. But I can more or less guarantee for you that in the end, the decision making might be pretty opaque, especially in the final stages. That's what we have to live with, I guess, when we enter into the sort of lottery like NLP conference set up for reviewing.

Here's a look at the form that reviewers are typically dealing with. You know, these used to be based mainly in structured data. Now it's more like structured text. The first box will ask, what is the paper about? What contributions does it make? And what are the main strengths and weaknesses?

This is a great question because I think it's trying to check in with a reviewer and make sure they actually know what the paper is about and what its contributions are. After that come the important pieces. Reasons to accept. Reasons to reject. For better or worse, both of these will be required boxes and reviewers will think of something to type in both.

I resist this, particularly for reasons to reject. I have often typed into that box. I have no reasons to reject for papers I simply wanted to support, but I had to enter some text in order to get the form to be accepted. Alas. Maybe a box for questions and additional feedback for the authors.

This is where the reviewers might pose direct questions that they're hoping will get resolved as part of the author response period. Missing references. This can be kind of touchy. You want to cite all the work that you can discover and people feel really slighted when things are missing. And that can often lead you to get some points off with reviewers.

Typos, grammar, style and presentation improvements. You would hope that this was kind of secondary. And then crucially the ratings. An overall recommendation, which is maybe the primary data point that shapes what happens to your paper in the submission process. And then reviewer confidence, which you would hope was kind of calibrating on how much the recommendation should contribute to the overall judgment.

And then there'll be a text box for confidential information. You'll also have such a text box as part of your author response. And this is your chance to communicate directly with the leadership of the conference about problems that might have arisen. And reviewers have the same option. Author responses.

A complicated and often bewildering part of this. Many of these conferences allow authors to submit short responses to the reviews. This is often done on a tight time window and you have a very limited number of characters you can enter. Overall, this is a rather uncertain business, but I think I can offer some advice.

First, many people are cynical about author responses since reviewers rarely change their scores afterwards. I understand that, but my overall perspective is a bit more optimistic. Even given one, though, item two, it's bad in terms of signaling not to submit a response at all. It sort of signals to the group that you don't care.

I realize this sounds kind of dismal because I'm saying you have to submit something, otherwise you'll get in trouble. But whatever you do submit might not have an impact. I think there is unfortunately some truth to that. But as I said, I'm optimistic in particular for conferences that have area chairs who are tasked with stimulating discussion and writing meta reviews for a small number of papers.

The author response might have a major impact. I say this as someone who has played the area chair role many times. I find the author response invaluable because I have the reviews, I have the paper, and I have now the author response. And I can balance them against each other and use that to stimulate discussion and get some clarity on what I should recommend for the paper as a whole.

So I always benefit and I think I'm not alone. And I hope that gives you some motivation for thinking about investing in the author response. NLP conferences, as usual, have some complex rules about this, about what you can and can't say in your author response. Read the instructions. If you have questions about what you can do in a particular case, you should again seek out an expert.

I do feel it's unfortunate that I have to at so many points here recommend that you find an expert because it sounds like a conspiracy of insiders. But that is the sad state of affairs. The number one thing that comes to mind here is that often there's a rule against reporting new results.

I'm not sure why that rule exists. As an area chair, I want as much information as I can get about these papers, including new results if they're available. But they're often forbidden. But then the question is, what counts as a new result? And for that, you might want your expert to at least offer you some guidance.

Always be polite. It can be hard, but it pays to be polite. Be firm and direct, but do that strategically to signal what you feel most strongly about. Never, ever, ever write things like, "Your inattentiveness is embarrassing. Section 6 does what you say we didn't do." You can write that out once if it would be cathartic, but then delete it and replace it with something like, "Thank you.

The information you're requesting is in Section 6. We will make this more prominent in our revision." You're not giving any ground. You're making it clear what you had achieved, but you're also being polite and productive. And with luck, everyone feels validated by this process, and your scores get boosted a little bit.

Presentation types and venues. There are kind of two dimensions to this. First, for the type, you have oral versus poster. Nobody knows how that's assigned. The conference people decide somehow, and you're told you're doing a poster or an oral. I don't think this is so consequential at this point because what matters is the papers.

For presentation venue, you have workshop versus main conference, and that is an important distinction. There are tradeoffs here. Workshops tend to be less prestigious and selective, but the bright side is that you get to connect with a community of like-minded people. You might add a workshop, interact with people who become lifelong collaborators.

On the flip side, the main conference will be highly selective and very prestigious, but it will be absolutely overwhelming. The number of people will be enormous. It will be in some big conference hotel. You might have real trouble connecting with people and finding people who work on what you do.

Your presentation might get lost in the mix, all of that stuff. So you have to decide on what experience you're looking for and what your goals are. For relevant conferences, I've listed out a whole bunch of them here. The left column is the ACL conferences. The top three, ACL, NACL, and EMNLP, are, I think, by consensus, the most prestigious in the field.

Just below them are AACL and EACL, but I think we're nearing the point where those are just in the top mix, and people are just looking for the next conference to submit to as the year goes around. Below those are Connell and Koehling, which are great. I think they're just second tier, but lots of important work appears there.

I've published there. I actually think they're wonderful venues, and they give you a balance between a workshop and a main conference. And then, in terms of prestige, below them is workshops, but I do feel that that's a different enough dimension to all this that you should think about them, especially if you're thinking about a first venue for publishing.

In the middle, I have conferences devoted to kind of the web and knowledge bases and artificial intelligence, cognitive science, linguistics. These are all venues that are great and that will welcome NLP-oriented submissions, and I think the most prestigious ones are kind of at the top there right now. And then, on the right, I have what I think are, by consensus, the top three machine learning conferences, and all of them are really welcoming of NLP research.

You just might want to orient it toward a slightly broader audience, but those are certainly good homes for research you might be doing. Here is my personal assessment of NLP reviewing, another bit of editorializing. First, I would say that the focus on conference papers has been good for the field.

It fits with and encourages a rapid pace, and even though I worry about the relentless pace of research, I do appreciate that things move fast and it's a very lively area intellectually. Before about 2010, the reviewing in the field was admirably good and rigorous in comparison with other fields.

I say this as someone who is coming from linguistics and cognitive science. I was deeply impressed by the investment reviewers would make in these submissions, in that pre-deep learning era especially. Lately, though, the growth of the field has reduced the general quality of reviewing. I think, by consensus, we all see that this is happening.

The field is still grappling with this. It was inevitable, as you get larger and larger numbers of submissions and reviewers, that the quality would go down as we move from a kind of trusted group of experts out into a much more diffuse and diverse population. So that is all feeling like this has more of a lottery aspect to it when you submit to these conferences.

This next point, I wanted to just issue a warning and kind of connect with you on this. Reviewers are occasionally incredibly mean. You need to kind of desensitize yourself to this, and one way to do that would be to share your reviews and your experiences with an experienced NL peer.

Most likely, that person will say, "I'm really sorry that you had to get this feedback. Do not take it personally. We've all seen this." I worry about this point. Reviewers can be mean enough that I worry it could turn people off of the field entirely. It could send a signal, especially to people new to the field, that they don't belong, and I think that is tragic.

Don't do this if you're asked to review. Be kind. Imagine what it would be like to be on the receiving end, and then if you do receive this very mean feedback, don't take it personally. It's really hard for me to say this because the whole point is that you've really invested in your research, and so to say that you shouldn't take it personally feels like a conflict on that point, but you have to kind of get to that point.

But please, please, please, if someone is mean to you, just try to move on from it and keep contributing. It's more about the reviewers than it's about you. This is more mundane. Forcing every paper to be four or eight pages that is short and long submission is not good, but luckily the issue is being addressed productively with more use of supplementary materials.

I love that we're moving into a mode of science where the papers are short and the appendices run to 30 pages so that if you need information about the details, you yourself can do a deep dive on the appendix. But if you're just trying to understand the ideas and contribution, you can read a short paper.

The biggest failing of all of this is that there is no chance for authors to appeal to an editor and interact with that editor. That's a missing piece that journals provide and to good effect. There is one area for this. Transactions of the ACL is a journal that follows the standard ACL conference model fairly closely, but allows for journal-style interaction with the editor.

I've been heavily involved with TACL over the years as an action editor and as a reviewer, and I think it is a shining example of what reviewing in the field could be if we scaled the model up to the level of our conferences. On titles, this is a smaller point.

I'm using this paper. I'm using titles in scientific journals and article citation, which you will notice is not a jokey paper title. Jokey paper titles are somewhat risky. You should calibrate the title to the scope of your contribution, and you should consider especially the reviewers you're likely to attract as they are scanning that long list of papers to bid on.

Avoid special fonts and formatting because they just won't travel with your paper, and so it can get kind of awkward, although I do have a soft spot for some of the emojis people have put in their titles recently. Abstracts, this is also important. This will create a first impression for your work.

Here is a kind of template for thinking about these documents. The opening is a broad overview, a glimpse at the central problem. The middle takes concepts mentioned in the opening and elaborates on them, probably by connecting with specific experiments and results from the paper. This is the meat of it.

The close establishes links between your proposal and broader theoretical concerns so that the reviewer has an answer to the question, "Does the abstract offer a substantive and original proposal?" So that's a way to think about this here. It isn't kind of abstract. This opening sentence situates you, dear reader.

"Our approach seeks to address the following central issue," then you spell out the techniques we used. "Our experiments are these," and you give details, and then, "Overall, we find that our approach has the following properties and the significance of this is." If you can conceptually fill this out with details from your project, I think you're in a good place, and then you could decide to get a little creative if you want to.

Style sheets or, on avoiding desk rejects, this is mundane but important, pay attention to the details of the style sheet and other requirements included in the call for papers. They change these details all the time, almost like they're running some kind of experiment on us, and you don't want to get rejected for a small infraction, right?

If you break one rule, you could get the dreaded desk reject, which would be a tragic but short way to end your conference submission process for this round. Try to avoid that. And then, finally, the camera-ready version. This is such a comical phrase. It's so weird that camera-ready survives.

This is like talking about dialing your phone and so forth. What this refers to is antiquated technology. They used to literally take a picture of your paper, and that would be the publication version. That was the camera-ready part. Now it just sticks around as a weird idiom. For most NLP conferences, you get an additional page on acceptance, presumably to respond to requests made by reviewers, though in practice you can use the space however you like.

Do write by your reviewers if they had concerns. It's great to address them, but it is not a contract that you have with the reviewers. Ultimately, at this stage, you should do what you feel is right for your paper. You have this extra page. Use it wisely. Use it thoughtfully.

In general, the extra page is probably for fixing things that were overly terse because you compressed what was a long paper down to a short one, but it's great if you have space to express some new ideas and so forth. However, err on the side of saving brand new results for subsequent publications.

Your paper is already crammed enough, I'm guessing, and so you don't want to add more things in just because you've discovered them. It might be a good idea to publish the current paper and then begin thinking about the next contribution that you'll make. Thank you.