After Reviews Are In

Subject: ACM CHI: What Associate Chairs do After the Reviews are in Dear Associate Chairs;

The deadline for reviews are today, Wednesday October 29. This note outlines what your next set of duties are, with critical dates included. While its long, you should read all of it. Of course, the first few items are the most critical, and must be attended to right away.

Immediately:


Book your flight and reserve the hotel if you haven't already done so! See http://pages.cpsc.ucalgary.ca/~saul/wiki/uploads/Personal/SC_travel_final.pdf

Wed 10/29: Reviews due back from reviewers
Wed 10/29: Track missing reviews & quickly resolve


There are only ~3 days of extra time after the deadline for reviews to allow you to round up final missing reviewers from reviewers who have let you down, or reviewers who have (in your opinion) supplied a completely inadequate review. You may even have to find last minute replacements for non-responding reviewers. Finding these last minute people (who must turn around a review in a few days) will require begging, arm-twisting, calling in favours, etc. However, we really need 3 external reviews for every paper going into rebuttal on 11/4. But some ground rules:

  • Avoid asking other ACs to do last minute reviewes. Like you, they have enough to do.
  • Your meta-review and the 2nd AC comments that may be added at a later date do not count towards the three reviewers.

Wed 10/29: Initiate reviewer discussion as needed
Wed 10/29 - Nov 3: Write meta-reviews


Once reviews for a paper are available you can write a meta-review and initiate a discussion between reviewers if warranted.

Discussion: If there are significant disagreements between reviewers, you may wish to initiate a discussion among them to see if this can assist you in resolving these. As well, if there are significant differences in reviewer scoring, you may want to ask them if they can reach a concensus by adjusting their scores. However, reviewers should not feel pressured into changing their scores or opinions if they do not want to.

The meta-review: Meta-reviews are primarily a summary description of the external reviews indicating points where the reviews agree and disagree. If reviews disagree you may wish to indicate which conflicting arguments you feel are more persuasive or points should be weighted more heavily (but remember that the reviewers will see your meta-review also). The score you give for the meta-review should normally reflect the underlying reviews. As a guide you should likely start with the average of the review scores and then think about what adjustments from that might be justified based on resolving conflicts and weighting of the importance of particular points raised. We are asking you to use your expert judgment here, but you should not be completely substituting your opinion for that of your reviewers. If you do want to add your own personal review/opinion, then identify that as a separate section within the meta-review, i.e., beginning with 'My personal opinion'.

Some tips for writing a meta-review. Authors have put quite a bit of work in it, so having feedback done in a constructive and positive voice will be better received than one that says only negative things. That is, always find something positive to say about the paper, even if you are going to recommend rejection. One method is to start your meta-review by summarizing what authors liked, before you get into reasons why a paper may be rejected.

A note about expertise. There is a tendancy for experts to under-rate themselves (i.e., they may say 'knowledgeable' when you know they are expert), and for novices to over-rate themselves. Consider your personal knowledge of reviewers in judging how much weight to put on their reviews as well as their self-rated report. For the author's sake, if a reviewer under-reports their expertise, you may include a note that says 'even though reviewer X rated themselves as knowledgeable, they are actually an expert in the field'.

Mon 11/3: Hard deadline for all reviews (including meta-, replacement, and additional reviews)
Tues 11/4: Reviews and meta-reviews released to authors for rebuttal


All reviews and meta-reviews (with scores) will be released to authors on this date to help them prepare a rebuttal. Rebuttals allow authors to counter factual errors in the reviews, so it's important that the authors are able to see all the external reviews at this point. Note that after this point, since the authors will have seen your meta-review it is probably best to make all changes as additions to the review. If you end up adjusting the score on the meta-review after this point (e.g., based on the author's rebuttal) you should remember that the author knows this is a change and add an explicit explanation of why this was done to the meta-review.

Tues 11/4: Preliminary "discuss" status determined (based on score cut offs and AC input)
Wed 11/5: SCs assign "discuss" papers to 2ACs for review


Just after releasing papers to authors for rebuttal, the papers chairs will determine cutoff lines for papers which will be by default accepted or rejected without discussion at the PC meeting. Cutoffs will be based on score average and variance. In addition you may request discussion for papers which wouldn't normally receive it if you feel this is justified. All papers which are to be discussed at the PC meeting need to have an additional AC present who has also reviewed the paper. A secondary AC for all these papers will be assigned by your subcommittee chair at this point. 2AC reviews are full reviews much like external reviews.

  • We will send you a note in a few days with these cutoffs.
  • Your Sub-Committee Chair will then assign further papers to you as a 2AC

Mon 11/10: Author rebuttals due
Tues 11/11: Initiate reviewer discussion as needed
Tues 11/11: Review rebuttals, possibly move papers to "discuss" and recruit 2AC reviewer


You should look at all rebuttals. If you find that the rebuttal raises important points you may wish to change previously non-discussed paper to a discussed paper. In that case you should recruit a secondary AC for the paper. (In general, any paper discussed at the PC meeting needs to have at least two people in the room who have read it -- it is your responsibility to ensure this is the case). If there are points in the rebuttal that need discussion you may also initiate additional discussion among the reviewers at this point.

Mon 11/26: 2AC reviews due
Mon 12/1: Updated meta-reviews and final "discuss" status due


Based on the newly available 2AC review, the rebuttal, and any additional discussion, you can make additional changes (most likely additions) to your meta-reviews. Again, if you end up changing scores based on these updates it is important that you explicitly explain this in the meta-review. The final status of whether the paper is to be discussed at the PC meeting should be established at this point. This will happen either via the automatic cutoff points or your explicit request to discuss a paper (as marked on the meta-review form).

Thu 12/4: PC meeting in Boston


Additional details about the conduct of the PC meeting will be provided separately. Please plan to arrive Wednesday afternoon or evening so we can start work first thing Thursday morning. Work will continue until 1:30pm Friday. Please don't plan a departure that would require you to leave the meeting early (Friday night hotel costs will be covered where necessary due to travel arrangements).

Fri 12/5: Post-meeting finalization of feedback to authors (done in Boston if possible)
Tue 12/8: Materials certified by ACs as ready to go back to authors
Wed 12/9: Reviews and decisions returned to authors


After the meeting you will need to make final updates to the feedback that will be returned to the authors and mark in the system when this is ready to be released to the authors. This needs to happen fairly quickly to meet our schedule, so you probably want to try to have most or all of this done before you leave Boston. At this stage it is important that you provide information to the authors that enables them to understand why the decision that was made was made. For conditionally accepted papers you need to provide clear feedback about the expectations of the committee about what changes may be needed in the paper.

Scott Hudson and Saul Greenberg, papers co-chairs
papers@chi2009.org