New Duties Criteria Second AC

Subject: Your next set of duties (UPDATED 11/7)

Hello ACs.

Synopsis of this note:

  • reminder of final reviews plus handling of rebuttals and discussion
  • determining auto-rejected papers
  • your role as 2nd AC

Current status

  • You should be collecting final reviews and completing your meta-review (deadline Nov 9) in preparation for the rebuttal process that opens on Nov 10.
  • You should initiate reviewer discussion as needed
  • When rebuttals have been received (due Nov 16), you should modify your meta-review and/or re-open the discussion as needed.

Next steps: Determining auto-rejected papers

Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to work with your SCs to determine which papers will be tentatively 'auto-rejected', i.e., rejected outright. While borderline papers may change, getting a preliminary list to your Sub-Committee Chair will help him/her start the 2nd AC assignment. You can update this list after the rebuttal period, and the Sub-Committee Chair can notify 2nd ACs accordingly. This task needs to be done soon, as 2nd ACs need time to read and comment on these papers.

  1. Score <= 2.7, little divergence - standard deviation <= 1 OR standard deviation > 1 but no single score is above 3.0 (auto-reject)
    • These papers are rejected outright.
    • No 2nd AC will be assigned, nor will the paper be discussed at the committee meeting.
    • The AC can elevate a paper out of this category (i.e., to have a 2nd AC assigned to the paper and to have it discussed). This should be reserved for situations when they believe that there is a realistic chance that the reject decision should be reconsidered. However, this should be done well before the PC meeting, as a 2nd AC will have to be assigned.
    • ACs should NOT auto-reject a paper if its score is > 2.7, and/or its standard deviation > 1, no matter what their personal opinion is.

A full list of other criteria (assigned after the rebuttal period but before the PC meeting) will be sent to you later. If you want to preview them now, look at http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/CriteriaSecondACs

Providing advice on a 2nd AC

For all papers that are not auto-rejected and are not conflicts for your SC (they will be sending you a list of these), recommend two other ACs on your sub-committee who would be appropriate to handle it as a 2nd AC. Then send this list to your SC. Your SC will use your recommendation to make the actual assignment. However, they may not always take your advice, as your SC has to balance workload amongst all ACs. Some things to keep in mind:

  • Do not recommend 2nd ACs outside the committee. The logistics of dealing with this are just too difficult
  • If a particular 2nd AC is really needed due to particular expertise, let your SC know.
  • Do not send information about papers on the SCs conflict list as this would improperly break your anonymity (these will be handled separately by the papers co-chairs) instead, send this information to the papers co-chairs at papers@chi2009.org (if one of us has an obvious conflict with the paper, send it to the other co-chair only.)

The easiest way to do the above is to send your SC a note listing the (non-conflicted) paper numbers you are responsible for, whether it is in the auto-reject category or not, and the 2nd ACs you recommend e.g.,

Paper 111111 Auto reject
Paper 121111 Discuss. 2nd ACs: Joe Bloggs or Shirley Cohen)
Paper 131111 Auto-reject
Paper 121111 Discuss. 2nd ACs: Lucy Brown - I really need her expertise on this one. ...

Your job as 2nd AC

You will receive your 2nd AC assignment via PCS. Recognize that some of your assignments, especially ones just above the auto-reject threshold, may change their status as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. You may even get additional assignments. However, you will likely be managing most of the ones on this preliminary list.

You should start looking at these papers as they arrive. Form your own preliminary opinion of it, which you can enter in the Comments to the Committee section of the review form. Closer to the PC meeting, you can finalize your opinion.

While you can write this preliminary opinion as a review, you can compose it in any way you wish; reviewers should not see this unless you do decide to add it (or a revised version of it) to explain a decision at the PC meeting.

IMPORTANT: Again, put your thoughts in the Comments to the Committee section. Don't put it in the review section, as authors may see this during the rebuttal period).

During the PC meeting, your primary job will be to work with the AC to come to a decision about the paper. If both your decisions are the same and if they don't diverge hugely from referee decisions, then the discussion (led by the primary AC) should be very brief (more on this in another note). If you disagree, you will both present your position at the PC meeting, and likely bring in another AC to look at the paper and discuss it further.

Note that we are following a 'conditional acceptance' practice this year. If a paper is generally sound but requires a modest revision (and we do mean modest) that you can request, this will ensure that the authors have to follow your advice. If they don't we can still reject the paper.