Peer Review Guidelines

Based on “ACL’23 Peer Review Form” by Jordan Boyd-Graber, Naoaki Okazaki, Anna Rogers

Modified by Yuki Arase, Baotian Hu, Wei Lu

1. In-Depth Review

The answers to the following questions are mandatory, and will be shared with both the committee and the authors.

Core Review: What is this paper about, what contributions does it make, and what are the main strengths and weaknesses?

Please describe what problem or question this paper addresses, and the main contributions that it makes towards a solution or answer. Please also include the main strengths and weaknesses of this paper and the work it describes.

The following kinds of contributions are all welcome at IJCNLP-AACL: computationally-aided linguistic analysis, NLP engineering experiment, reproduction study, new data resources (particularly for low-resource languages), approaches for data- and compute efficiency, position papers, surveys, publicly available software and pre-trained models.

Reasons to accept

What would be the main benefits to the NLP community if this paper were to be presented at the conference?

Reasons to reject

What would be the main risks of having this paper presented at the conference (other than lack of space to present better papers)?

Please follow the ACL 2023 conference policies on what should not be considered weaknesses. The authors and meta-reviewers will be aware of these guidelines.

2. Questions and Additional Feedback for the Author(s) The answers to the following questions are optional. They will be shared with both the committee and the authors, but are primarily for the authors.

The following review elements are optional. They will be shared with both the committee and the authors, but are primarily for the authors.

Missing References

Please list any references that should be included in the bibliography or need to be discussed in more depth. If you believe that this work is not novel, misses previous work or baselines, give the full citation(s) below. Remember that contemporaneous work (in the three months before the deadline) is not required to be cited and should not be held against the authors.

Typos, Grammar, Style, and Presentation Improvements

Please list any typographical or grammatical errors, as well as any stylistic issues that should be improved. In addition, if there is anything in the paper that you found difficult to follow, please suggest how it could be better organized, motivated, or explained. Be sure to include line numbers for easy reference.

Additional Suggestions for the Author(s)

Other than the points mentioned above, please give any additional feedback to the authors that you feel could help them improve the work or its presentation in the paper. Include any points that the authors could address in a revised version (either for the conference or elsewhere), as well as suggestions for changes to the organization of the paper.

3. Overall Recommendation

Experiment Results

To what extent do the experimental results in this paper support the research claims?
0 (N/A), 1 (not at all) — 5 (full support)

If the paper is not an empirical paper, please indicate 0. Otherwise, use a numerical score to indicate the extent to which the reported experimental results support the research claims made by the paper.

Overall Recommendation

Should this paper be accepted to IJCNLP-AACL 2023?

In making your overall recommendation, please take into account all of the paper’s strengths and weaknesses, the paper’s appropriateness for the conference, as well as its clarity and originality. Acceptable long paper submissions must describe substantial, original, and completed work on empirical NLP (e.g., model design and implementation, corpus construction/annotation, evaluation methodologies). Acceptable short submissions include: small, focused contributions; works in progress; negative results and opinion pieces; and interesting application notes.

Please adhere to the score definitions below when scoring papers.

  • 5 = Transformative: This paper is likely to change our field. It should be considered for a best paper award.
  • 4.5 = Exciting: It changed my thinking on this topic. I would fight for it to be accepted.
  • 4 = Strong: I learned a lot from it. I would like to see it accepted.
  • 3.5 = Leaning positive: It can be accepted more or less in its current form. However, the work it describes is not particularly exciting and/or inspiring, so it will not be a big loss if people don’t see it in this conference.
  • 2.5 = Leaning negative: There are key weaknesses (e.g., I didn’t learn much from it, evaluation is not convincing, it describes incremental work). I believe it can significantly benefit from another round of revision. I am ambivalent, but slightly leaning towards rejection.
  • 2 = Mediocre: I would rather not see it in the conference.
  • 1.5 = Weak: I am pretty confident that it should be rejected.
  • 1 = Poor: I would fight to have it rejected.

Please note that we have removed the score 3, as we encourage reviewers to provide more informative decisions regarding papers, with their explicit opinions.

Reviewer Confidence

How confident are you in your assessment of this paper?

  • 5 = Positive that my evaluation is correct. I read the paper very carefully and I am very familiar with related work.
  • 4 = Quite sure. I tried to check the important points carefully. It’s unlikely, though conceivable, that I missed something that should affect my ratings.
  • 3 = Pretty sure, but there’s a chance I missed something. Although I have a good feel for this area in general, I did not carefully check the paper’s details, e.g., the math, experimental design, or novelty.
  • 2 = Willing to defend my evaluation, but it is fairly likely that I missed some details, didn’t understand some central points, or can’t be sure about the novelty of the work.
  • 1 = Not my area, or paper was hard for me to understand. My evaluation is just an educated guess.

4. Confidential Information The answers to the following questions will be shared with the committee only, not the authors.

Recommendation for Presentation Type

We have fewer slots for oral presentations (talks) than for posters, and want to make sure that the most appropriate papers get selected for talks. Note that the published proceedings will make no distinction between papers presented orally and those presented as posters.
Would this paper make for a better oral or poster presentation?

Ethical Concerns

Do you have any ethical concerns that Area Chairs/PC Chairs should be aware of? If so, please select “Yes” and provide more comments in the “confidential comments” box below. We also encourage you to flag this concern to the authors in the weakness section above

Recommendation for Best Paper Award

Do you think this paper should be considered for a Best Paper Award? There will be separate Best Paper Awards for long and for short papers.

Recommendation for Resource Award

Do you think this paper should be considered for a Resource Award? These are for papers that announce, describe, and share a fascinating, valuable, or potentially field-changing new resource (e.g., a dataset or knowledge graph).

Recommendation for Social Impact Award

Do you think this paper should be considered for a Social Impact Award? These awards are for papers that have the potential for significant positive societal impact.

Justification for Award Recommendations

Please describe briefly why you think this paper should receive an award. Your comments will not be shared with the authors, but if the paper receives an award, it is possible that some of your comments may be made public (but remain anonymous) in the award citation.

Confidential Comments to the Area Chairs/PC chairs

Is there anything you want to say solely to the committee?
For example, a very strong (negative) opinion on the paper, which might offend the authors in some way, or something which would expose your identity to the authors.