Month: April 2019

Given the rise of AI, Natural Language Processing has become increasingly popular and almost all recent conferences have reported a record breaking number of submissions. Yet, never in the history of ACL have we seen such a dramatic growth: within just a single year, we have gone from 1544 submissions to 2906! This is illustrated in the following graph that shows the growth of ACL over the past 20 years in terms of the number of submissions, reviewers and (Senior) Area Chairs.

Review of such a large number of submissions requires a large, well-organised Program Committee. Extending the ACL 2018 practice, we created a structure similar to the conferences that have a Senior Program Committee alongside the Program Committee. For the Senior PC, we recruited a relatively large number of Senior Area Chairs (46, 2-4 to head each area) and Area Chairs (184, 3-15 per area). We also differentiated between their roles so that SACs assign papers to ACs and reviewers and make recommendations for their area, while ACs each manage a smaller set of papers within the area, lead discussions with reviewers, write meta-reviews and make initial recommendations. This structure also helps to compensate for the problem that our rapidly growing field is suffering from: the lack of experienced reviewers. As ACs focus on a smaller number of papers, they can pay more attention to the review process. As for reviewers, we simply have many of them this year: 2281 (ACL 2018 had 1610).

With such a huge number of submissions, every step of conference organisation (from the initial checking of submissions to decision making) takes longer than before. Knowing the timeline would be extremely tight, we looked into ways of improving efficiency. We wanted to improve efficiency in ways that would optimise the experience for authors and PC members. In particular, we reduced the number of deadlines requiring a short turn-around of 3 days (or less). Such deadlines at best are stressful for all, but often work poorly, given the diversity of work and life situations in the community (i.e. the great variation in times / days when people are actually available for conference-related work).

We implemented the following changes:

• We dropped the paper bidding phase. This phase can take several days of time, and given the large number of submissions, reviewers find it increasingly time consuming. However, the time considerations aside, we were also worried about the impact of reviewers choosing their favourite papers for review, as opposed to choosing to review papers that they are qualified to review (for an interesting blog post on the topic, see https://naacl2018.wordpress.com/2018/01/28/a-review-of-reviewer-assignment-methods/). Our plan was to rely on the Toronto Paper Matching System (TPMS) in allocating papers to reviewers. Unfortunately, this system didn’t prove as useful as we had hoped for (it requires more extensive reviewer profiles for optimal performance than what we had available) and the work had to rely largely on the manual effort. Our fantastic SACs did an outstanding job here, but this is clearly a task that needs better automated support.

• Like NAACL 2019, we didn’t have an author response phase this year. Originally introduced as an improvement to the review process, author response has proven time-consuming (taking not only authors but also reviewers and chairs time) and not hugely impactful on a larger scale. For example, the following paper (due to appear in NAACL 2019) summarises relevant data from ACL 2018:

Does My Rebuttal Matter? Insights from a Major NLP Conference
Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych and Yusuke Miyao

So, instead of author response, we decided to invest in promoting discussion within the PC,   and on ensuring that discussions, papers and reviews have the full attention of ACs.

• Finally, in contrast with the elaborate review forms of some recent conferences, we adopted much simpler, streamlined review form, adapted from EMNLP 2018 (many thanks to Julia Hockenmaier, David Chiang and Junichi Tsujii!). While encouraging thorough review, this form is less laborious for reviewers and more focused on highlighting the key points for decision making.

However, even with our time saving measures the conference schedule is still too tight, not only for us PC chairs but also for (S)ACs and reviewers. Interestingly, although ACL has grown significantly over the past 20 years, the schedule remains almost the same as it was back in 1999. In particular, the time between the submission deadline and the notification of acceptance is exactly the same (2 months) as it was in 1999, although the number of submissions has increased tenfold and the size and the complexity of the PC even more. It may be time to adopt the practice of related conferences (e.g., IJCAI, NeurIPS, SIGIR) and extend the schedule to allow for 3-4 months for this process. This could be critical for maintaining the quality of reviewing as the conference grows further.

The conference schedule is also impacted by the schedules for other conferences (e.g. this year NAACL and EMNLP-IJCNLP) and the ACL Guidelines and desire for preprints. We made a concerted effort with other conferences to avoid overlap in the review period (which otherwise shortened the available time for each conference). Overlapping review periods will either result in unhappy authors (when they cannot submit to all conferences) or chaos for PC chairs who struggle to manage multiple submissions and large numbers of withdrawn papers. Also reviewers may be less likely to review for multiple conferences at the same time. Even with no overlap this year, there is still a desire from authors for a longer period in between conferences, so that they can revise and resubmit papers based on the feedback from previous conferences, and rejection from one conference does not leave enough time for people to submit before the anonymity period of the next conference begins. In general, not just the dates of the conferences but also the number and scheduling of deadlines should be given community-wide attention going forward.

ACL 2019 received as many as 2906 submissions by the submission deadline. This constitutes more than a 75% increase over ACL 2018 and is an all-time record for ACL-related conferences! The huge logistics involved in handling these submissions explains our long silence. However, we can now finally give you some basic statistics:

• Out of the 2906 submissions, 120 submissions have been withdrawn by authors and 92 have been desk-rejected due to issues such as dual submissions, plagiarism or submissions not conforming to the submission guidelines. (These numbers are likely to still change over the coming weeks as papers undergo review).

• The resulting 2694 valid submissions, including 1609 long and 1085 short papers, have been sent to review.

• Each paper has been assigned to one of 22 areas for review. Each area is headed by 2-4 Senior Area Chairs (SACs) who are in charge of the overall review process within their area. They are assisted by 3-15 Area Chairs (ACs) who look after a subset of the papers (15 on average). The different areas have 59-319 reviewers, depending on the number of submissions. In total, our Programme Committee includes 2256 people: 46 SACs, 184 ACs and 2026 reviewers (1903 are currently involved in reviewing).

• The following table shows, for each area, the number of submissions (long, short and total) that are currently undergoing review.

Our 3 largest areas in terms of submissions are the same as in ACL 2018:

• Information Extraction and Text Mining (9.2% of all valid submissions vs. 11.5% in ACL 2018 – note that the percentages are not fully comparable because this year’s conference features an additional area, Applications)
• Machine Learning (8.2% vs. 7.4% in ACL 2018)
• Machine Translation (7.7% vs. 8.3% in ACL 2018)

Also Dialogue and Interactive systems are among the top 5 areas in both conferences. However, Document Analysis, which was the 4th largest area last year, ranks only the 16th this year, while Generation (which ranked the 14th last year with 59 submissions) is ranked now the 5th with 156 submissions (the increase in submissions is much larger here than our overall growth rate!). Another surprise is Linguistic Theories, Cognitive Modeling and Psycholinguistics, which clearly grew in popularity: 24 submissions last year, 60 this year.

Submissions remain still relatively evenly distributed across the different areas (see the below pie chart) in comparison with e.g. in ACL 2017 where IE was clearly dominating (23.4% of submissions).

Looking at the long and short papers, 60% of our submissions are long while in ACL 2018 66% were. So short papers are more popular this year. Looking the individual areas,

• short papers are clearly more popular than long ones in only one of the areas: Applications.
• the two types of paper are almost equally popular in Machine Translation and Tagging, Chunking, Syntax, Parsing,
• areas that have the clearest preference for long papers (over 65% of submissions are long) include Machine Learning, Vision, Robotics Multimodal Grounding, and Speech, and Dialogue and Interactive Systems.

Finally, regarding the geographical distribution of papers, we received papers from 61 countries. Considering the country of the corresponding author only (which is clearly a simplification), we looked at which countries produced most submissions. The first chart below shows the results for the top 20 countries, and the second zooms into the top 20 countries with fewer than 140 submissions.

As expected, we have the US and China in the lead. The UK and Germany rank third and fourth with 129 and 126 submissions, respectively, and are closely followed by Japan (120 submissions).

The chart differentiates between the long and short paper submissions. It shows that the top 5 countries, apart from Japan, have a clear preference for long papers. There is some variation among them, e.g. China produces relatively more long paper submissions (69% of all submissions) than the US (60% of all submissions). The countries among the top 20 where short paper submissions are more popular than long ones include Japan, India, Taiwan, Denmark and Spain. The countries that have the strongest preference for long papers are Singapore (84% of all submissions are long) and Israel (73%).

Let us know if you would like us to compute additional statistics.