Final Marching Orders for ACs and 2nd ACs

This is a long note, but it should be read carefully. It contains your final 'marching orders' that detail your duties up to and including the PC meeting.

What to do now.

All:

  • 2nd ACs are now being assigned. Your sub-committee chair will tell you when it is complete.

ACs and 2nd ACs

  • The rebuttals are now all in.
  • Read or Reread the reviews and the rebuttals carefully. See A Short Guide to Reevaluating Papers.
  • Invite reviewers to read the rebuttal if you feel it is warranted, and to discuss the paper using the PCS discussion facility. You may not want to bother with this if the paper is a clear accept or reject.
  • Change your opinion and scoring if needed.
  • Deadline. Nominate papers for Best Paper. On your review form, you will see a "Best Paper Nomination" area. Check the box, and add a paragraph in the space provided describing why its worthy. Note that nominations are typically (but not always chosen from auto-accept category.

For ACs

  • Put in a paragraph in the 'Associate Chair Additional Comments' box on the review form that tentatively stakes out your (possibly revised) opinion.
  • You will have opportunity to edit this again at or immediately after the PC meeting.
  • The 2nd AC will be able to see this.
  • Be polite and compassionate in this: make it clear that you have read the rebuttal, and that you have heard what the author has to say.
  • If you want to elevate a paper with a score < 2.7 and a std deviation < 1 to the discuss category, please check the "To be Discussed at the PC Meeting" box on the paper's review form and tell your Sub-Committee Chair as he/she will have to assign a 2nd AC to it.

For 2nd ACs

  • Deadline: Add a score for each of your assignments by November 30, evening. This will be used to rank-order the papers.
  • You do NOT have to write a full review, but you should write an opinion.
  • Put this opinion in the 'Comments to the Committee': only the committee will see it.
  • You should add a score to the paper. The score should reflect your opinion as well as your reading of all referee reports and the rebuttal.
  • You may be asked to transform this opinion into a review at the committee meeting, if it is warranted.

Balance your efforts.

  • See The Criteria section below.
  • Using the criteria below, put more effort into papers that will likely require more discussion, e.g., papers with middle scores, or with divergent scores, or where there is disagreement between AC and 2nd AC are likely the ones that will be discussed most. Your opinion should be more detailed in these cases.

The Criteria.

These are the various criteria for papers. The most important ones in the list below are the reject and accept outright. Other criteria are there to help us focus discussion in the PC meeting.

  1. Score <= 2.7, little divergence - standard deviation <= 1 OR standard deviation > 1 but no single score is above 3.0 (auto-reject)
    • These papers are rejected outright.
    • No 2nd AC will be assigned, nor will the paper be discussed at the committee meeting.
    • The AC can elevate a paper in this category to the category 2 discussion level for situations when they believe that there is a realistic chance that the reject decision should be reconsidered. However, this should be done well before the PC meeting, as a 2nd AC will have to be assigned.
  2. Score <= 2.7, significant divergence (tentative reject)
    • A 2nd AC will be assigned.
    • If both ACs can agree ahead of time that the paper should be rejected, then the ‘discussion’ will be limited to report that outcome at the meeting, otherwise it will be discussed in greater detail.
  3. 2.7 < Score < 4, (uncertain outcome)
    • A 2nd AC will be assigned.
    • If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the tentative decision, followed by discussion as warranted.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  4. 4 <= Score < 4.3 (tentative accept)
    • A 2nd AC will be assigned.
    • If the ACs reach consensus to accept, a quick summary of the issues will be presented at the meeting along with the decision.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  5. 4.3 <= Score <= 5 (accept outright)
    • A 2nd AC will be assigned
    • If the two ACs agrees with accept, this paper will be auto-accepted.
    • If they do not agree, this paper will be scrutinized carefully, with other ACs brought into the process. Rejecting a paper with a high score should be reserved for only the most extreme cases.
    • Discussion will usually be limited to a summary, unless issues are flagged.

The meeting process in brief

The PC Meeting process will cycle between the top- and bottom-rated papers. Doing so will help calibrate decisions. That is, we will have a session starting with the 'best' papers. After a break, we will restart at the bottom, and continue this cycle until we meet in the middle. The process each paper will typically follow is:

  • paper number is announced
  • ACs / SCs in conflict will leave the room
  • The decision is reported by the AC with little or no discussion if:
    • the paper has generally low scores (but still above the auto-reject threshold)
    • the AC/2nd AC agree to a decision
    • the variance of opinion between reviewers is modest
  • If the decision is positive, a brief summary of the paper is reported.
  • If there is disagreement between ACs / 2nd ACs, or if there is high variance, or if any AC wants to discuss issues, then a summary of issues will be presented
    • Depending on the situation, another AC will be assigned to look at the paper and join the discussion (off-line).
    • The above continues until a decision can be reached by the ACs involved. Additional ACs may be brought in as needed.
  • ACs should be prepared to discuss who the reviewers are and why they chose them.

Note that we are following a 'conditional acceptance' practice this year. If a paper is generally sound but requires a modest revision (and we do mean modest) that you can request, this will ensure that the authors have to follow your advice. If they don't we can still reject the paper.