CriteriaSecondACs

Sent Nov 6. Subject: ACM CHI: Your next duties as Sub-Committee Chair

Hello Sub-committee Chairs There is lots of detail in this note, but it is important – please read carefully.

Synopsis of this note:

Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to work with your ACs to determine which papers will likely be rejected outright according to the criteria below, and assign all others to 2nd ACs in your sub-committee that best match the remaining papers. This should be done soon, as 2nd ACs need time to read and comment on these papers. However, this will be complicated somewhat by the fact that some papers may drift across the reject outright/discuss line due to additional reviews, rebuttals, etc. Read on!

Assigning 2nd ACs

  • First send a list of all the papers in your sub-committee for which you have conflicts to all your ACs. This will allow them to avoid sending you information about those papers and hence breaking their anonymity.
  • I have asked your ACs to look at the paper scores via the criteria below, and to send you a tentative list of what papers will likely be auto-rejected. See http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/NewDutiesCriteriaSecondAC for what was sent to them.
  • All papers that are not auto-rejected will need a 2nd AC. I have also asked your ACs to recommend one or two 2nd ACs for those papers, and you may want to discuss which are the best choices. However, it is your job to balance the AC load and to do the actual assignment. You can assign them via PCS, simply by adding them as a reviewer to the paper.
  • The list you receive from ACs should look something like:
    • Paper 111111 Auto reject
    • Paper 121111 Discuss. 2nd ACs: Joe Bloggs or Shirley Cohen)
    • Paper 131111 Auto-reject
    • Paper 121111 Discuss. 2nd ACs: Lucy Brown - I really need her expertise on this one.
    • ...
  • You can then instruct the 2nd ACs that they can start looking at their assignments. Warn them that the ratings of some papers may change: as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. Still, 2nd ACs can read the papers and form their own preliminary opinion of it, which they can track in the Comments to the committee section of the review form.
  • Immediately after setting up your 2nd AC assignments send the papers co-chairs your list of conflicts along with a list indicating how many 2AC assignments were made to each of your sub-committee members. Assignment of 2ACs will then be handled by the paper co-chairs.

We will be sending a note to ACs about these responsibilities.

The Criteria.

These are the various criteria for papers. The most important one in the list below is the reject outright one,as 2nd AC assignment depends on this. Other criteria are there to help us focus discussion in the PC meeting.

  1. Score <= 2.7, little divergence - standard deviation <= 1 OR standard deviation > 1 but no single score is above 3.0 (auto-reject)
    • These papers are rejected outright.
    • No 2nd AC will be assigned, nor will the paper be discussed at the committee meeting.
    • The AC can elevate a paper in this category to the category 2 discussion level for situations when they believe that there is a realistic chance that the reject decision should be reconsidered. However, this should be done well before the PC meeting, as a 2nd AC will have to be assigned.
  2. Score <- 2.7, significant divergence (tentative reject)
    • A 2nd AC will be assigned.
    • If both ACs can agree ahead of time that the paper should be rejected, then the ‘discussion’ will be limited to report that outcome at the meeting, otherwise it will be discussed in greater detail.
  3. 2.7 < Score < 4, (uncertain outcome)
    • A 2nd AC will be assigned.
    • If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the tentative decision, followed by discussion as warranted.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  4. 4 <= Score < 4.3 (tentative accept)
    • A 2nd AC will be assigned.
    • If the ACs reach consensus to accept, a quick summary of the issues will be presented at the meeting along with the decision.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  5. 4.3 <= Score <= 5 (accept outright)
    • A 2nd AC will be assigned
    • If the two ACs agrees with accept, this paper will be auto-accepted.
    • If they do not agree, this paper will be scrutinized carefully, with other ACs brought into the process. Rejecting a paper with a high score should be reserved for only the most extreme cases.
    • Discussion will usually be limited to a summary, unless issues are flagged.

The meeting process in brief

The PC Meeting process will cycle between the top- and bottom-rated papers. Doing so will help calibrate decisions. That is, we will have a session starting with the 'best' papers. After a break, we will restart at the bottom, and continue this cycle until we meet in the middle. The process each paper will typically follow is:

  • paper number is announced
  • ACs / SCs in conflict will leave the room
  • The decision is reported by the AC with little or no discussion if:
    • the paper has generally low scores (but still above the auto-reject threshold)
    • the AC/2nd AC agree to a decision
    • the variance of opinion between reviewers is modest
  • If the decision is positive, a brief summary of the paper is reported.
  • If there is disagreement between ACs / 2nd ACs, or if there is high variance, or if any AC wants to discuss issues, then a summary of issues will be presented
    • Depending on the situation, another AC will be assigned to look at the paper and join the discussion (off-line).
    • The above continues until a decision can be reached by the ACs involved. Additional ACs may be brought in as needed.

Note that we are following a 'conditional acceptance' practice this year. If a paper is generally sound but requires a modest revision (and we do mean modest) that you can request, this will ensure that the authors have to follow your advice. If they don't we can still reject the paper.