A Short Guide to Re-evaluating Each Paper

Personal.ShortGuideReEvaluate History

Hide minor edits - Show changes to markup

Added lines 1-47:

(:title A Short Guide to Re-evaluating Each Paper :)

As AC and 2nd AC, you are now reconsidering reviews and rebuttals. This is the time where you should be willing to change your opinion (if warranted). You should pay special attention to the following points, as these will likely require special scrutiny and discussion during the committee meeting.

Rebuttals:

  • Read them carefully, and be ready to re-appraise your opinion.
  • Authors sometimes suggest revisions. This is ok if revisions are modest. Papers will be conditionally accepted, so you do have opportunities to check that changes were indeed done. However, checking is not the same as going through a 2nd referee round, which will not be done. Thus you have judge what is 'reasonable' revision that can be done within this scope.

Divergence: The AC and 2nd AC should take special care in cases where there is:

  • Large divergence in scores between referees
  • AC scores are mis-aligned with reviewer scores (e.g., lower or higher than all reviewer scores). This is not to say that ACs are limited to the 'average' score of reviewers, but that these papers deserve a strong second look.

Notes vs Papers: A Note needs to be judged as a Note, not as a 10 page paper. Notes are historically judged too harshly.

  • See http://www.chi2009.org/Authors/CallForPapers/PapersVsNotes.html
  • Some reviewers of notes have expectations that are unrealistic for a 4-page paper
  • To summarize, a Note is a much more focused and succinct contribution to research and is likely to have a smaller – yet still significant – scope of contribution. Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue. However, Notes should still be judged with high expectations: they are not posters or reports of works-in-progress.

Keep criticisms in perspective

  • All papers have problems at some level. Some are minor, and these should be treated as such.
  • Highly rated papers with many criticisms are sometimes judged too harshly. Some reviewers score papers highly and say its good, but then spend lots of time critiquing it rather than praising it. This is often a problem with how a referee report is written, vs. a problem with the paper. Thus always take into account the referee's overall appraisal in terms of interpreting the criticisms. If a referee liked the paper, don't overly devalue it just because the referee spent their time listing areas of improvement.

Related work

  • Authors are sometimes over-penalized for missing references.
  • Given the limited paper length, the paper should only be penalized if the missing references are crucial ones, e.g., that previously reported similar contributions that the author is claiming is novel.
  • Also, one should be reasonable about missing references to work far outside the normal CHI literature: you will have to make a judgment call in cases like that.
  • Remember: Adding related work is easy to do in a revision.

Evaluate what was done, not what someone else would have done

  • Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly.
  • This could indicate 'school of thought' differences (see below).

Schools of thought differences

  • Referees may have quite different perspectives and background that make excessively hard on authors. Watch out for this: they may be spotted as rants or tirades or suggestions that the approach taken by authors is completely wrong.
  • You may want to discount referee reports that exhibit this behavior.

Sub-committee fit.

  • See the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html,
  • Important: You should not judge a paper by how well it fits the subcommittee theme. Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.

Contribution type fit.

  • See the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html
  • As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type. However, the submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that.
  • When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI? This is important.
  • The contribution type and its associated questions are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.
  • Look carefully at reviews that have a divergence between the contribution score and the overall score.