Criteria Second A Cs

Personal.CriteriaSecondACs History

Show minor edits - Show changes to output

November 07, 2008, at 09:48 PM by 67.165.66.127 -
Added line 15:
** First send a list of all the papers in your sub-committee for which you have conflicts to all your ACs. This will allow them to avoid sending you information about those papers and hence breaking their anonymity.
Changed lines 25-26 from:
to:
** Immediately after setting up your 2nd AC assignments send the papers co-chairs your list of conflicts along with a list indicating how many 2AC assignments were made to each of your sub-committee members. Assignment of 2ACs will then be handled by the paper co-chairs.
Changed line 31 from:
# Score <= 2.7, little divergence - standard deviation <= 1 (auto-reject)
to:
# Score <= 2.7, little divergence - standard deviation <= 1 OR standard deviation > 1 but no single score is above 3.0 (auto-reject)
Added lines 1-3:
Sent Nov 6.
Subject: ACM CHI: Your next duties as Sub-Committee Chair
Changed lines 15-19 from:
*** Paper 111111 Auto reject \\
*** Paper 121111 Discuss. 2nd ACs: Joe Bloggs or Shirley Cohen)\\
*** Paper 131111 Auto-reject\\
*** Paper 121111 Discuss. 2nd ACs: Lucy Brown - I really need her expertise on this one. \\
...
to:
*** Paper 111111 Auto reject
*** Paper 121111 Discuss. 2nd ACs: Joe Bloggs or Shirley Cohen)
*** Paper 131111 Auto-reject
*** Paper 121111 Discuss. 2nd ACs: Lucy Brown - I really need her expertise on this one.
*** ...
Changed lines 15-18 from:
Paper 111111 Auto reject \\
Paper 121111 Discuss. 2nd ACs: Joe Bloggs or Shirley Cohen)\\
Paper 131111 Auto-reject\\
Paper 121111 Discuss. 2nd ACs: Lucy Brown - I really need her expertise on this one. \\
to:
*** Paper 111111 Auto reject \\
*** Paper 121111 Discuss. 2nd ACs: Joe Bloggs or Shirley Cohen)\\
*** Paper 131111 Auto-reject\\
*** Paper 121111 Discuss. 2nd ACs: Lucy Brown - I really need her expertise on this one. \\
Changed lines 13-14 from:
** All papers that are not auto-rejected will need a 2nd AC. I have also asked your ACs to recommend one or two 2nd ACs for those papers, and you may want to discuss which are the best choices. However, it is your job to balance the AC load and to do the actual assignment. You can assign them via PCS, simply by adding them as a reviewer to the paper. T** The list you receive from ACs should look something like:
to:
** All papers that are not auto-rejected will need a 2nd AC. I have also asked your ACs to recommend one or two 2nd ACs for those papers, and you may want to discuss which are the best choices. However, it is your job to balance the AC load and to do the actual assignment. You can assign them via PCS, simply by adding them as a reviewer to the paper.
** The list you receive from ACs should look something like:
Changed lines 12-13 from:
** Immediately ask your ACs to look at the paper scores via the criteria below, and to send you a tentative list of what papers will likely be auto-rejected, auto-accepted, or something inbetween. The 'auto-reject' category is the most important, as those will not have a 2nd AC.
** All papers that are not auto-rejected will need a 2nd AC. For those papers, you may want to discuss the best choices with the primary AC, but it is your job to balance the AC load and to do the actual assignment. You can assign them via PCS, simply by adding them as a reviewer to the paper.
to:
** I have asked your ACs to look at the paper scores via the criteria below, and to send you a tentative list of what papers will likely be auto-rejected. See http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/NewDutiesCriteriaSecondAC for what was sent to them.
** All papers that are
not auto-rejected will need a 2nd AC. I have also asked your ACs to recommend one or two 2nd ACs for those papers, and you may want to discuss which are the best choices. However, it is your job to balance the AC load and to do the actual assignment. You can assign them via PCS, simply by adding them as a reviewer to the paper. T** The list you receive from ACs should look something like:
Paper 111111 Auto reject \\
Paper 121111 Discuss. 2nd ACs: Joe Bloggs or Shirley Cohen)\\
Paper 131111 Auto-reject\\
Paper 121111 Discuss. 2nd ACs: Lucy Brown - I really need her expertise on this one. \\
...
Changed line 23 from:
** These papers are rejected outright, unless the AC opts to elevate it to the category 2 discussion level. This should only be reserved for situations when there is a realistic chance that the discussion may lead to the reject decision being reconsidered.
to:
** These papers are rejected outright.
Changed lines 25-26 from:
to:
** The AC can elevate a paper in this category to the category 2 discussion level for situations when they believe that there is a realistic chance that the reject decision should be reconsidered. However, this should be done well before the PC meeting, as a 2nd AC will have to be assigned.
Changed lines 20-21 from:
First, let’s discuss the various criteria for papers. The most important one in the list below is the reject outright one; the other criteria are there to help focus discussion in the PC meeting.
to:
These are the various criteria for papers. The most important one in the list below is the reject outright one,as 2nd AC assignment depends on this. Other criteria are there to help us focus discussion in the PC meeting.
Changed lines 13-14 from:
** All papers that are not auto-rejected will need a 2nd AC. For those papers, start matching them to your ACs, ideally balancing expertise and load across them. You can assign them via PCS. You can then instruct the 2nd ACs that they can start looking at their assignments. Warn them that the ratings of some papers may change as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. Still, 2nd ACs can read the papers and form their own preliminary opinion of it which they can track int the Comments to the committee section of the review form.
to:
** All papers that are not auto-rejected will need a 2nd AC. For those papers, you may want to discuss the best choices with the primary AC, but it is your job to balance the AC load and to do the actual assignment. You can assign them via PCS, simply by adding them as a reviewer to the paper.
** You can then instruct the 2nd ACs that they can start looking at their assignments. Warn them that the ratings of some papers may change: as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. Still, 2nd ACs can read the papers and
form their own preliminary opinion of it, which they can track in the Comments to the committee section of the review form.

We will be sending a note to ACs about these responsibilities
.
Changed line 11 from:
!! Assigning 2nd ACs for ‘discuss papers’
to:
!! Assigning 2nd ACs
Changed line 12 from:
** Immediately ask your ACs to look at the paper scores via the criteria below, and to send you a tentative list of what papers will likely be auto-rejected, auto-accepted, or something inbetween.
to:
** Immediately ask your ACs to look at the paper scores via the criteria below, and to send you a tentative list of what papers will likely be auto-rejected, auto-accepted, or something inbetween. The 'auto-reject' category is the most important, as those will not have a 2nd AC.
Added line 3:
Changed lines 9-10 from:
Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to determine which papers will likely be rejected outright according to the criteria below, and assign all others to 2nd ACs in your sub-committee that best match the remaining papers. This should be done soon, as 2nd ACs need time to read and comment on these papers. However, this will be complicated somewhat by the fact that some papers may drift across the reject outright/discuss line due to additional reviews, rebuttals, etc. Read on!
to:
Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to work with your ACs to determine which papers will likely be rejected outright according to the criteria below, and assign all others to 2nd ACs in your sub-committee that best match the remaining papers. This should be done soon, as 2nd ACs need time to read and comment on these papers. However, this will be complicated somewhat by the fact that some papers may drift across the reject outright/discuss line due to additional reviews, rebuttals, etc. Read on!
Changed line 31 from:
# 4 <= Score < 4.5 (tentative accept)
to:
# 4 <= Score < 4.3 (tentative accept)
Changed line 36 from:
# 4.3 < Score <= 5 (accept outright)
to:
# 4.3 <= Score <= 5 (accept outright)
Changed line 36 from:
# 4.5 < Score <= 5 (accept outright)
to:
# 4.3 < Score <= 5 (accept outright)
Changed line 33 from:
** If the ACs can reach consensus, a quick summary of the issues will be presented at the meeting along with the decision.
to:
** If the ACs reach consensus to accept, a quick summary of the issues will be presented at the meeting along with the decision.
Changed line 28 from:
** If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the decision, followed by discussion as warranted.
to:
** If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the tentative decision, followed by discussion as warranted.
Changed line 28 from:
** If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the decision. Cases where scoring seems at odds with the recommendation will be discussed in more detail (e.g., papers with higher scores that are rejected, and papers with lower scores that are accepted).
to:
** If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the decision, followed by discussion as warranted.
Changed line 28 from:
** If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the decision. Cases where scoring seems adds odds with the recommendation will be discussed in more detail (e.g., papers with higher scores that are rejected, and papers with lower scores that are accepted).
to:
** If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the decision. Cases where scoring seems at odds with the recommendation will be discussed in more detail (e.g., papers with higher scores that are rejected, and papers with lower scores that are accepted).
Changed line 28 from:
** If the ACs can reach consensus, a quick summary of the issues will be presented at the meeting along with the decision.
to:
** If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the decision. Cases where scoring seems adds odds with the recommendation will be discussed in more detail (e.g., papers with higher scores that are rejected, and papers with lower scores that are accepted).
Deleted line 26:
** Score > 2.7 but likely < 4.0, and/or papers with high variance
Changed line 27 from:
- Score > 2.7 but likely < 4.0, and/or papers with high variance
to:
** Score > 2.7 but likely < 4.0, and/or papers with high variance
Changed lines 8-9 from:
Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to estimate which papers will be rejected outright according to the criteria below, and assign all others to 2nd ACs in your sub-committee that best match the remaining papers. This should be done soon, as 2nd ACs need time to read and comment on these papers. However, this will be complicated somewhat by the fact that some papers may drift across the reject outright/discuss line due to additional reviews, rebuttals, etc. Read on!
to:
Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to determine which papers will likely be rejected outright according to the criteria below, and assign all others to 2nd ACs in your sub-committee that best match the remaining papers. This should be done soon, as 2nd ACs need time to read and comment on these papers. However, this will be complicated somewhat by the fact that some papers may drift across the reject outright/discuss line due to additional reviews, rebuttals, etc. Read on!
Added lines 56-58:
Note that we are following a 'conditional acceptance' practice this year. If a paper is generally sound but requires a modest revision (and we do mean modest) that you can request, this will ensure that the authors have to follow your advice. If they don't we can still reject the paper.
Changed lines 44-55 from:
The PC Meeting process will cycle between the top- and bottom-rated papers. Doing so will help calibrate decisions. That is, we will have a session starting with the 'best' papers. After a break, we will restart at the bottom, and continue this cycle until we meet in the middle.
to:
The PC Meeting process will cycle between the top- and bottom-rated papers. Doing so will help calibrate decisions. That is, we will have a session starting with the 'best' papers. After a break, we will restart at the bottom, and continue this cycle until we meet in the middle. The process each paper will typically follow is:
** paper number is announced
** ACs / SCs in conflict will leave the room
** The decision is reported by the AC with little or no discussion if:
*** the paper has generally low scores (but still above the auto-reject threshold)
*** the AC/2nd AC agree to a decision
*** the variance of opinion between reviewers is modest
** If the decision is positive, a brief summary of the paper is reported.
** If there is disagreement between ACs / 2nd ACs, or if there is high variance, or if any AC wants to discuss issues, then a summary of issues will be presented
*** Depending on the situation, another AC will be assigned to look at the paper and join the discussion (off-line).
*** The above continues until a decision can be reached by the ACs involved. Additional ACs may be brought in as needed.
Added line 4:
* assigning 2nd ACs for ‘discuss papers’
Deleted line 5:
* assigning 2nd ACs for ‘discuss papers’
Changed lines 10-11 from:
The Criteria.
-------------
to:
!! Assigning 2nd ACs for ‘discuss papers’
** Immediately ask your ACs to look at the paper scores via the criteria below, and to send you a tentative list of what papers will likely be auto
-rejected, auto-accepted, or something inbetween.
** All papers that are not auto
-rejected will need a 2nd AC. For those papers, start matching them to your ACs, ideally balancing expertise and load across them. You can assign them via PCS. You can then instruct the 2nd ACs that they can start looking at their assignments. Warn them that the ratings of some papers may change as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. Still, 2nd ACs can read the papers and form their own preliminary opinion of it which they can track int the Comments to the committee section of the review form.

!! The Criteria.
Changed lines 38-40 from:
** If there is little divergence (<1) and the AC recommends acceptance, no 2nd AC will be assigned unless it has been flagged
** Otherwise
, a 2nd AC will be assigned.
** If the two
ACs agrees with accept, this paper will be auto-accepted. Otherwise another AC will be brought into the process.
to:
** A 2nd AC will be assigned
** If
the two ACs agrees with accept, this paper will be auto-accepted.
** If they do not agree
, this paper will be scrutinized carefully, with other ACs brought into the process. Rejecting a paper with a high score should be reserved for only the most extreme cases.
Added lines 42-45:

!! The meeting process in brief
The PC Meeting process will cycle between the top- and bottom-rated papers. Doing so will help calibrate decisions. That is, we will have a session starting with the 'best' papers. After a break, we will restart at the bottom, and continue this cycle until we meet in the middle.
Added lines 1-38:
Hello Sub-committee Chairs
There is lots of detail in this note, but it is important – please read carefully.
Synopsis of this note:
* cut-offs for papers that will not be discussed.
* assigning 2nd ACs for ‘discuss papers’
* rough outline of the PC Meeting Process

Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to estimate which papers will be rejected outright according to the criteria below, and assign all others to 2nd ACs in your sub-committee that best match the remaining papers. This should be done soon, as 2nd ACs need time to read and comment on these papers. However, this will be complicated somewhat by the fact that some papers may drift across the reject outright/discuss line due to additional reviews, rebuttals, etc. Read on!

The Criteria.
-------------
First, let’s discuss the various criteria for papers. The most important one in the list below is the reject outright one; the other criteria are there to help focus discussion in the PC meeting.

# Score <= 2.7, little divergence - standard deviation <= 1 (auto-reject)
** These papers are rejected outright, unless the AC opts to elevate it to the category 2 discussion level. This should only be reserved for situations when there is a realistic chance that the discussion may lead to the reject decision being reconsidered.
** No 2nd AC will be assigned, nor will the paper be discussed at the committee meeting.

# Score <- 2.7, significant divergence (tentative reject)
** A 2nd AC will be assigned.
** If both ACs can agree ahead of time that the paper should be rejected, then the ‘discussion’ will be limited to report that outcome at the meeting, otherwise it will be discussed in greater detail.

# 2.7 < Score < 4, (uncertain outcome)
- Score > 2.7 but likely < 4.0, and/or papers with high variance
** A 2nd AC will be assigned.
** If the ACs can reach consensus, a quick summary of the issues will be presented at the meeting along with the decision.
** If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.

# 4 <= Score < 4.5 (tentative accept)
** A 2nd AC will be assigned.
** If the ACs can reach consensus, a quick summary of the issues will be presented at the meeting along with the decision.
** If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.

# 4.5 < Score <= 5 (accept outright)
** If there is little divergence (<1) and the AC recommends acceptance, no 2nd AC will be assigned unless it has been flagged
** Otherwise, a 2nd AC will be assigned.
** If the two ACs agrees with accept, this paper will be auto-accepted. Otherwise another AC will be brought into the process.
** Discussion will usually be limited to a summary, unless issues are flagged.