MoreOnTheMetaReview

Subject: ACM CHI SCs/ACs: More on the Meta-review Sent Oct 30, 2008

Dear ACs;

As you know, the review process has changed this year to include sub-committees and contribution types, and to consider notes and papers together. Consequently, in your role as AC, there are a few things you have to watch for (beyond the normal things you do) when reading and evaluation reviews, and when writing your meta-review. Here is a summary..

1. Notes vs Papers.


The two are different, as explained in http://www.chi2009.org/Authors/CallForPapers/PapersVsNotes.html. For notes, what you have to watch for in reviews are:

2. Contribution types.


The purpose of contribution types is to try to break referee tendencies to favor 'formula' papers over those that make contributions outside of the formula. Authors indicated a contribution type. As part of a referee's judgment of whether the paper makes a strong contribution, he/she should have considered the questions about contributions as specified in the description of each contribution type on http://www.chi2009.org/Authors/CallForPapers/SelectingAType.html. What is important about these questions is that they should have pushed the referee to think about the contribution in ways that differ from what is normally perceive as the CHI formula paper.

However, these questions are not a checklist. The submission(s) may not cleanly fall within a single contribution type, and the paper should not be penalized for that. When this happens, the primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI?

This is important. The contribution type and its associated questions are a guide to extend (not narrow) one's thinking about what makes a good contribution; they are not a strict criteria by which one should measure acceptance. They are there to help one think more liberally about the paperís contribution. If the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly. See: http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html. When reading reviews and writing your meta-review, ask yourself if the referee has fallen back into the trap of favoring 'normal' papers and/or if they used the contribution type to restrict rather than extend their judgement of the contribution. If this is the case, you may want to contact the reviewer and/or start a discussion of it.

3. Sub-committees. --- Authors submit their papers to subcommittees. However, neither you nor reviewers should not judge a paper by how well it fits the subcommittee theme. That is, sub-committees are there only for administrative purposes: to ease the handling the large number of submissions, and to better assign good referees / ACs to papers. They are NOT an attempt to split the conference into themes. See: http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html

Another note will follow specifying thresholds for papers that will be discussed at the meeting.

Saul Greenberg and Scott Hudson
ACM Papers Chairs