Writing Useful Reviews

One of the biggest frustrations as a research is getting a “bad” review for a paper.  I do not mean “bad” a review that recommended rejection for a conference or journal.  I also do not mean “bad” as in a review that looked unfavorably on my work, or had highly pointed criticisms.  I mean a bad review where it is not clear the reviewer read the paper at any depth or provided useful objective feedback that can help improve my work.  These are often cases where the reviewer put minimal work into the review or has a dogmatic view of the area.  While such reviews are often attached to rejections, I have experienced bad reviewing with papers that have been accepted, leading to papers on my CV of which I am less proud.   Research put considerable time and effort into the work and authoring papers.  Further, there is often value and useful insights in many of the papers that are not accepted to a given conference or journal.  Bad reviewing is disrespectful and unfair to authors, in addition to wasting their time.  More importantly, bad reviewing spreads dissension among authors, who may themselves choose to become bad reviewers.
I believe the essence of good reviewing is to provide clear feedback such that a paper can be accepted (even with minor revisions), or revised and improved for acceptance at future conferences.  While this feedback includes a judgment on acceptance, improving the quality of work in the community through constructive feedback is valued over reviewing for purely quality control.  As such, I often find a typical review of good quality has at least the following three elements/paragraphs:

1) Summarize of the paper’s major contributions (3-10 sentences). Provide the best good-faith summarization of the paper’s claims, methods, and results.  This paragraph establishes the reviewer’s understanding of the work that will both support further comments/criticisms and allow misunderstandings to be identified.  Do not comment or opine on the value of the work in this paragraph.

2) Summarize the reviewer’s opinion of the paper (3-10 sentences).  Identify the strengths and weaknesses of the paper with respect to criteria such as: relevance, conceptual novelty, technical soundness, quality of evaluation, clarity in composition and organization, etc.  Also, specify to relative confidence in the review, or aspects of the review.  For the weak aspects of the paper, provide high-level suggestions that could improve the paper to the level of acceptability.  Do not be snarky or dismissive.  This is not about finding faults.  It is about improving the quality of the work or finding new opportunities for research.

3) Detailed comments (as many paragraphs as necessary).  Address any specific points, positive or negative, in detail.  I typically do this as a bulleted list of paragraphs.  Each paragraph goes into detail about various points mentioned in the review summary and beyond, such as identifying spelling/grammatical errors and pointers to relevant related work.

I think of writing a review as crafting an argument or essay about the validity or invalidity of a paper.  Point (1) establishes the foundational premises based on your understanding of the paper.  Point (2) is your conclusion or thesis statement about the validity/invalidity of the paper.  Points (3)+ build your argument to the the conclusion.

Further there are a number good reviewing practices that can be helpful to authors:

• If you believe the paper is not sufficiently novel, this claim needs to be backed up with citations to at least 3 papers (ideally authored by different research groups).  If you cannot think of 3 other related papers, then maybe there is room in the area for new work.  Similarly, if you believe a paper is subsumed by another work, this should be said explicitly.
• If the paper suffers from many grammatical and spelling issues, point out and correct a reasonable subset of these examples from the paper, and suggest further proofreading.
• Find and point out all forward references of terms in the paper and undefined acronyms.

By cjenkins in News on January 17, 2012