The ACL survey on reviewing was conducted from May 6 – June 5, 2019 and received 422  responses.  The survey was advertised twice on the ACL membership email distribution list as well as on social media.  The  ACL PC Chairs and the  ACL Exec invites the membership to discuss these results both in the comments features of this blog and during a town hall session in the last 45 minutes of the Business meeting at ACL 2019 in Florence (Tuesday, July 30, 17:30-19:00, Cavagnilia room).  That portion of the meeting will  include a presentation of a summary of these results, a panel of discussants, and an open mic for discussion from the membership.

This blog post contains just the executive summary. The full results, including charts and categorized comments, are in this pdf document.

This discussion is complementary to one that took place two years ago on the ACL 2017 PC Chairs blog.  The topics are different than that one, and we would like the comments to stay focused to what was asked in this survey.

Background

Purpose: The purpose of this survey was to inform ACL leadership and the membership about the membership’s opinions about several aspects of the reviewing process, based on members’ experience as authors, reviewers, area chairs (AC), or program chairs (PC). The results of this survey will help guide decisions about these policies in the future. 

Structure: The survey consisted of 26 questions organized as (i) respondents’ experience as reviewers, authors, area chairs, and program co-chairs, and then opinions on: (ii) author response, (iii) author discussion, (iv) meta-review, (v) structured review forms, (vi) review transparency, (vii) acceptance rates, (viii) timing of conference dates and review release, (iv) public review, (x) open comments, and finally, (xi) demographics.  Participants were allowed to skip questions, so results are reported in percentages throughout. An option of “No opinion / not sure’’ was offered on most questions.

Discussion:  The ACL Exec invites the membership to discuss these results both in the comments features of this blog and during an open session in the Business meeting at ACL 2019 in Florence.

Summary of Results

Demographics: The vast majority of participants (95%) are either current or previous members of the ACL, and as far as we can tell from available membership statistics, the sample appears to be representative with respect to demographic factors like geographical affiliation, gender, role and academic background. 

The vast majority of participants have submitted an *ACL paper for review at least once (98%) or at least twice (95%) over the last 10 years.  However, there were a significant number of respondents with little experience reviewing (13% never, 8% one time, 8% two times). Furthermore, most respondents have not had first-hand experience with higher levels of reviewing.  74% have not been an AC, and only 16% have been an AC two or more times in the last ten years. The respondent pool consisted of 7% of people who have been program co-chairs for *ACL in that time period. Thus, most respondents are relatively inexperienced in the reviewing process.

Opinions: This is a brief summary; details and charts appear in the full documents that are linked below.

  • Author Response: A significant majority of survey respondents were in favor of author response as a whole, with 61% in favor compared to 27% opposed. Very few were strongly opposed.  Preference for author response was negatively correlated with reviewer experience.

 

  • Author Discussion: Opinions were nearly evenly split on author discussion, with 37% in favor, 32% opposed, and 30% with no opinion.  However, those who had experienced author discussion were more in favor of it than those who had not experienced it.  Those with experience as an Area Chair were more opposed.

 

  • Meta-Reviews: There was strong support for meta-reviews (only 9% opposed), with the majority (73%) agreeing that it is ok to have them only for borderline cases.  However, former PC chairs were split 50%-35% on the latter question. 

 

  • Review Form Structure: A significant majority supported (65%) a minimal amount of structure in review forms, although a minority (24%) preferred more structure. There was a clear trend that more experienced reviewers preferred less structured review forms.

 

  • Review Transparency:  A clear majority (54%) preferred that reviews be released only to authors, but a notable minority (36%) preferred public release of reviews. Those with no experience reviewing more often in favor of public release of reviews, while those with some experience reviewing were more often opposed. Public release of reviews was one area where there was a significant gap between genders, with only 26% of female respondents preferring reviews be publicly released, compared with about 40% for those who were male or preferred not to state. On the other hand, there were no regional differences in preferring public release, but a significant 20% of researchers from the Asia/Pacific region preferred that reviews not be released even to authors. Finally, those with program chair experience were even more strongly against public release of reviews, with only 14% agreeing. 18% were opposed to even releasing reviews to authors.

 

  • Post Review Discussion Transparency: There was significant disagreement in whether post-review discussion should be released to authors, with about half of those who responded with an opinion (47%) preferring that it not, and the remaining half split between release to authors  (24%) and public release (20%). As seen in similar questions, those with greater amounts of review experience preferred that post-review discussion be kept private, while those who had never reviewed strongly preferred that discussion be available to authors or the public.

 

  • Meta-Review Transparency:  There was strong support in releasing meta-reviews to authors, with 83% agreeing. 33% were also in favor of releasing to the public. Similar trends were seen as above for reviewer experience.

 

  • Acceptance Rates: A near majority (47%) were in favor of keeping the status quo with respect to acceptance rates, while 32% were in favor of increasing, 9% were in favor of decreasing. A large majority of respondents (68%) preferred that acceptance rates not be decided before-hand, but rather post-hoc based on quality of papers. There was strong overall support (83%) for the status quo for conference publications remaining selective.

 

  • Timing of Review Release: The great majority said that the timing between review release and the next conference deadlines was at least somewhat important, (83%-14%). People would prefer at least 2-3 weeks, and many (39%) said at least a month was preferable.  Female respondents slightly preferred having more time between review release and the next deadline, and those who preferred not to state their gender particularly preferred longer reviewing cycles.

 

  • Public Review: Opinions about public review were mixed and tended to be strong, with 42% opposed and 32% in favor.  Of these, 37% held strong opinions. Those who had experience participating in conferences with public review tended to be in favor 50%-30%, while those who did not have such experience were opposed 27%-47%.  In general, support for public review tended to be inversely correlated with reviewing experience. Female respondents were less likely to support public review than male respondents, 19%-55% vs. 36%-39% respectively. However, female respondents were only half as likely as male respondents to have participated in a conference with public review, and the effects of gender and experience with public review may be conflated in these results.

 

Summary of Open Comments:  This is a summary of the main comments raised in the open comments section, as well as selected ideas put forward;  the interested reader can review the full, organized list of comments:

 

  • Author Response: Positive: ACs/PCs find it useful; reviewers find it useful; authors find it useful for clarifications, to repair poor reviews, and for fairness.   Neutral: perhaps should be used only in some cases or only sent to ACs; need better indication of when response considered. Negative: too much effort for too little effect; a stressful time-sink with little feedback; too much time for reviewers and authors with little ultimate payoff; reviewer discussion is more valuable; ACL author response not as thorough/long as other conferences/journals.

 

  • Author Discussion: Positive: If set up more like a journal model, then a good idea; the OpenReview style of discussion is a good model.  Neutral: a good idea, but only with a mechanism for conditional acceptance; would need to reduce the number of papers per reviewer; maybe for limited cases only.  Negative: too much work for reviewers/ACs/authors; not enough time between conferences; likely to be as ineffective as regular author response; only works for those with flexible schedules. Idea: a revise-and-resubmit model for the next *ACL conference where reviewers have access to prior reviews.

 

  • Meta-Reviews: Positive: Helpful to authors and PCs when reviews conflict; ensures ACs review all reviews; imposes discipline on the reviewing process; catches errors. Negative: more work for AC and benefit not clear. Idea: occasional message from the PC explaining decision.

 

  • Review Form Structure: Positive: Minimal structure (over no structure) is useful as an AC; minimal structure a good middle ground for experienced reviewers; ACL 2019’s form received many positive comments; can reduce bias and increase fairness and clarity.  Negative: frequent changes in format across conferences; too much structure increases reviewer workload and can reduce review coherence (NAACL 2018 in particular); minimal word counts are not a good idea.

 

  • Transparency (all three types): Positive:  Comments are similar to those about public review, summarized below. Negative: Public release of review discussion may make frank discussion difficult/unintentionally reveal reviewers’ identities.

 

  • Acceptance Rates / Conference Selectivity: Pro higher acceptance rates: Promote journals more and at the same time make conferences less selective; have different levels of selectivity within conferences; low acceptance rates lead to inefficiencies as papers are resubmitted multiple times, and quality problems since good papers are rejected and there is variance in quality.  Neutral: current acceptance rates seem to be striking a good balance; the medical system is not a good one; our rate is good; other fields’ are too low. Con higher acceptance rates: being selective at conferences is an important quality control; important for jobs; impractical logistically. Idea: focus higher acceptance rates on specific special topics.

 

  • Timing of Review Release: Having more time encourages meaningful revisions; it is important to demotivate resubmissions “as is”; reviewers need time to rest between reviewing assignments; timing of major conferences needs to be better spaced out.  Negative: people should not feel they have to submit to every conference. Ideas: allow authors to see reviews before final accept/reject decision to give them more time to revise work; do not allow people to resubmit unaltered papers; prior reviews should travel with resubmitted papers.

 

  • Public Review: Positive: incentivizes better reviewing; a way to recruit reviewers; speeds up research; might improve review quality.   Neutral: it is essential to retain double-blind reviewing. Negative: may hurt diversity; may discourage participation; may hurt early careers; may be a popularity contest; no filter on quality of comments; lack of quality on posted; unreviewed papers; too easy to game; too time-consuming. Ideas: support public discussion after publication; try public review once as an experiment.

 

  • General Comments: Concerns about reviewer preparation, review quality, and reviewing load;  concerns about diversity of acceptable content; comments on the bidding process; concerns about over-publishing (just noticeable difference). Idea: two week review cycle (suggested by Omer Levy).

 

Implications for Policy

 

These results can be divided into three categories, broadly.  The first is in terms of guidance for conference planning for ACL Exec and Chapter Boards.   There is general agreement about keeping the status quo in terms of acceptance rates, keeping conference presentations selective, retaining gaps in time between conferences to allow for resubmission.  

The second is to help inform Program Co-Chairs and conference organizing committees terms of details for reviewing procedure in terms of author response, meta-reviews, and their transparency.  Program co-chairs can take this information into account when making decisions about reviewing, and the ACL Exec can use this information to help prioritize changes to reviewing software.

The third category is the potential for more radical changes to *ACL reviewing procedures.  In particular, some people are interested in open review. This survey has canvassed the views of the membership about this topic.  Currently, the views seem to be split, which suggests that any path to adoption most likely would need to first take place in some kind of laboratory setting, such as workshop reviewing or the like.  Additionally, some respondents have suggested other changes to the reviewing process that were not explicitly asked about but which may be a useful way forward. These include: requiring authors to include prior reviews when re-submitting papers;  allow authors to see reviews before final decisions are made to allow more time for improving their work, and providing a method for public discussion of papers after acceptance and publication.

This survey does not address other pressing issues relating to reviewing that remain before *ACL, including the need for better automation of detection of reviewing matches, recruiting of reviewers, detection of conflicts of interest, and improvement of reviewing quality.

 

Definitions

The text of the original survey is in this pdf document.

Terms were defined as followed in the survey.

Author Response: In recent years, some *ACL conferences have had author responses, which provide a chance for authors to respond to reviewers to answer questions and provide clarifications before final decisions are made.

Author Discussion: A more comprehensive version of author discussion, some conferences have a discussion period where authors can interact with reviewers over an extended period of time. After initial reviews are released, authors may respond to the reviews point-by-point, and then the reviewers or ACs can ask additional follow-up questions or clarifications until the author discussion period is over. All this can be done in an anonymous fashion, preserving double-blind review.

Meta-Review: A Meta-review is a review performed by ACs after the review process completes that summarizes the views of the reviewers, and also explains the reasoning of the ACs regarding why they reached their final decision. These meta-reviews potentially make the reasoning about why decisions were made more clear. 

Structured Review Form:  Review forms require various levels of structure ranging from a simple score and free-form text box with which to enter reviews, to multiple text boxes requiring information about different aspects of the review.

Transparency: The decision about by whom information should be seen: program chairs and reviewers only, the authors of the papers, the public at large.

Acceptance Rates: Currently, acceptance rates for the major *ACL conferences are around 20-25%, and papers must be accepted via a competitive review process in order to be presented.  (Demonstrations, posters and some other methods of presenting materials tend to have a higher acceptance rate, but no questions were asked about these.)

Conference Selectivity: In other fields (e.g. medicine), most submitted papers or abstracts are given presentations at conferences, and other measures are used to indicate relative quality of papers (e.g. journal publications).  This question asked if conference publications should remain selective, not be selective with journals taking the role, or not be selective but with public reviews released. 

Timing of Release: When several conferences are held in sequence back-to-back, it is sometimes the case that reviews are released only shortly before the next major conference submission deadline. Questions about timing refer to this limited time span for revisions before the next opportunity to submit.

Public Review: Some conferences have introduced a review mechanism called public review in which, in addition to a program committee, the public is allowed to view and publicly post comments about papers during the review period.  Although in some conferences the authors’ names are publicly exposed and the reviews remain even for rejected papers, it is possible to change the format and anonymize the submissions and remove rejected papers after the review period.

This survey and report were prepared primarily by Graham Neubig and Marti Hearst in June-July 2019.