More On The Meta Review

Subject: ACM CHI SCs/ACs: More on the Meta-review Sent Oct 30, 2008

Dear ACs;

As you know, the review process has changed this year to include sub-committees and contribution types, and to consider notes and papers together. Consequently, in your role as AC, there are a few things you have to watch for (beyond the normal things you do) when reading and evaluation reviews, and when writing your meta-review. Here is a summary..

1. Notes vs Papers.

The two are different, as explained in For notes, what you have to watch for in reviews are:

  • a note must be considered a worthy (albeit quite focused) contribution in of itself reporting mature and solid work. A note is NOT a short paper reporting work in progress. Some referees don't understand this difference, and may relax their reviewing standards as a result.
  • reviewers should respect that 4 pages is just 4 pages, which places severe limits on what can be placed into it. For example, the call for notes specifically states that "CHI Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue." Some reviewers ask for things that just cannot be fit into 4 pages. Of course, if the authors should have written a 10 page paper instead of a note, that is a different matter.

2. Contribution types.

The purpose of contribution types is to try to break referee tendencies to favor 'formula' papers over those that make contributions outside of the formula. Authors indicated a contribution type. As part of a referee's judgment of whether the paper makes a strong contribution, he/she should have considered the questions about contributions as specified in the description of each contribution type on What is important about these questions is that they should have pushed the referee to think about the contribution in ways that differ from what is normally perceive as the CHI formula paper.

However, these questions are not a checklist. The submission(s) may not cleanly fall within a single contribution type, and the paper should not be penalized for that. When this happens, the primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI?

This is important. The contribution type and its associated questions are a guide to extend (not narrow) one's thinking about what makes a good contribution; they are not a strict criteria by which one should measure acceptance. They are there to help one think more liberally about the paperís contribution. If the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly. See: When reading reviews and writing your meta-review, ask yourself if the referee has fallen back into the trap of favoring 'normal' papers and/or if they used the contribution type to restrict rather than extend their judgement of the contribution. If this is the case, you may want to contact the reviewer and/or start a discussion of it.

3. Sub-committees. --- Authors submit their papers to subcommittees. However, neither you nor reviewers should not judge a paper by how well it fits the subcommittee theme. That is, sub-committees are there only for administrative purposes: to ease the handling the large number of submissions, and to better assign good referees / ACs to papers. They are NOT an attempt to split the conference into themes. See:

Another note will follow specifying thresholds for papers that will be discussed at the meeting.

Saul Greenberg and Scott Hudson
ACM Papers Chairs