New Duties Criteria Second AC

Personal.NewDutiesCriteriaSecondAC History

Hide minor edits - Show changes to markup

November 07, 2008, at 09:35 PM by 67.165.66.127 -
Changed lines 1-3 from:

Subject: Your next set of duties

to:

Subject: Your next set of duties (UPDATED 11/7)

Changed lines 31-32 from:

For all papers that are not auto-rejected, recommend two other ACs on your sub-committee who would be appropriate to handle it as a 2nd AC. Then send this list to your SC. Your SC will use your recommendation to make the actual assignment. However, they may not always take your advice, as your SC has to balance workload amongst all ACs. Some things to keep in mind:

  • Do not recommend 2nd ACs outside the committee. The logistics of dealing with this are just to difficult
to:

For all papers that are not auto-rejected and are not conflicts for your SC (they will be sending you a list of these), recommend two other ACs on your sub-committee who would be appropriate to handle it as a 2nd AC. Then send this list to your SC. Your SC will use your recommendation to make the actual assignment. However, they may not always take your advice, as your SC has to balance workload amongst all ACs. Some things to keep in mind:

  • Do not recommend 2nd ACs outside the committee. The logistics of dealing with this are just too difficult
Changed lines 34-36 from:

The easiest way to do the above is to send your SC a note listing the paper numbers you are responsible for, whether it is in the auto-reject category or not, and the 2nd ACs you recommend e.g.,

to:
  • Do not send information about papers on the SCs conflict list as this would improperly break your anonymity (these will be handled separately by the papers co-chairs) instead, send this information to the papers co-chairs at papers@chi2009.org (if one of us has an obvious conflict with the paper, send it to the other co-chair only.)

The easiest way to do the above is to send your SC a note listing the (non-conflicted) paper numbers you are responsible for, whether it is in the auto-reject category or not, and the 2nd ACs you recommend e.g.,

Changed line 20 from:
  1. Score <= 2.7, little divergence - standard deviation <= 1 (auto-reject)
to:
  1. Score <= 2.7, little divergence - standard deviation <= 1 OR standard deviation > 1 but no single score is above 3.0 (auto-reject)
Changed lines 14-15 from:
  • When rebuttals have been received, you should modify your meta-review and/or re-open the discussion as needed.
to:
  • When rebuttals have been received (due Nov 16), you should modify your meta-review and/or re-open the discussion as needed.
Changed lines 14-15 from:
  • When rebuttal due date has been received (Nov 10), you should modify your meta-review and/or re-open the discussion as needed.
to:
  • When rebuttals have been received, you should modify your meta-review and/or re-open the discussion as needed.
Changed lines 45-48 from:

You will receive your 2nd AC assignment via PCS. Recognize that some of your assignments, especially ones just above the auto-reject threshold, may change their status as the ratings of some papers may change as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. However, you will likely be managing most of them.

You can start looking at these papers. Form your own preliminary opinion of it, which you can enter in the Comments to the Committee section of the review form. (Don't put it in the review section, as authors may see this during the rebuttal period). Closer to the PC meeting, you can finalize your opinion.

to:

You will receive your 2nd AC assignment via PCS. Recognize that some of your assignments, especially ones just above the auto-reject threshold, may change their status as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. You may even get additional assignments. However, you will likely be managing most of the ones on this preliminary list.

You should start looking at these papers as they arrive. Form your own preliminary opinion of it, which you can enter in the Comments to the Committee section of the review form. Closer to the PC meeting, you can finalize your opinion.

While you can write this preliminary opinion as a review, you can compose it in any way you wish; reviewers should not see this unless you do decide to add it (or a revised version of it) to explain a decision at the PC meeting.

IMPORTANT: Again, put your thoughts in the Comments to the Committee section. Don't put it in the review section, as authors may see this during the rebuttal period).

Changed lines 38-40 from:

Paper 121111 Discuss. 2nd AC possibilities: Joe Bloggs or Shirley Cohen)
Paper 131111 Auto-reject Paper 121111 Discuss. 2nd AC possibilities: Lucy Brown - I really need her expertise on this one.

to:

Paper 121111 Discuss. 2nd ACs: Joe Bloggs or Shirley Cohen)
Paper 131111 Auto-reject
Paper 121111 Discuss. 2nd ACs: Lucy Brown - I really need her expertise on this one.

Added lines 1-3:

Subject: Your next set of duties

Deleted lines 5-8:

Here are the next steps in your duty assignments.

There is lots of detail in this note, but it is important – please read carefully.

Changed lines 7-23 from:
  • assigning 2nd ACs for ‘discuss papers’
  • cut-offs for papers that will not be discussed.
  • rough outline of the PC Meeting Process

Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to work with your ACs to determine which papers will likely be rejected outright according to the criteria below, and assign all others to 2nd ACs in your sub-committee that best match the remaining papers. This should be done soon, as 2nd ACs need time to read and comment on these papers. However, this will be complicated somewhat by the fact that some papers may drift across the reject outright/discuss line due to additional reviews, rebuttals, etc. Read on!

Assigning 2nd ACs

  • Immediately ask your ACs to look at the paper scores via the criteria below, and to send you a tentative list of what papers will likely be auto-rejected, auto-accepted, or something inbetween. The 'auto-reject' category is the most important, as those will not have a 2nd AC.
  • All papers that are not auto-rejected will need a 2nd AC. For those papers, you may want to discuss the best choices with the primary AC, but it is your job to balance the AC load and to do the actual assignment. You can assign them via PCS, simply by adding them as a reviewer to the paper.
  • You can then instruct the 2nd ACs that they can start looking at their assignments. Warn them that the ratings of some papers may change: as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. Still, 2nd ACs can read the papers and form their own preliminary opinion of it, which they can track in the Comments to the committee section of the review form.

We will be sending a note to ACs about these responsibilities.

The Criteria.

These are the various criteria for papers. The most important one in the list below is the reject outright one,as 2nd AC assignment depends on this. Other criteria are there to help us focus discussion in the PC meeting.

to:
  • reminder of final reviews plus handling of rebuttals and discussion
  • determining auto-rejected papers
  • your role as 2nd AC

Current status

  • You should be collecting final reviews and completing your meta-review (deadline Nov 9) in preparation for the rebuttal process that opens on Nov 10.
  • You should initiate reviewer discussion as needed
  • When rebuttal due date has been received (Nov 10), you should modify your meta-review and/or re-open the discussion as needed.

Next steps: Determining auto-rejected papers

Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to work with your SCs to determine which papers will be tentatively 'auto-rejected', i.e., rejected outright. While borderline papers may change, getting a preliminary list to your Sub-Committee Chair will help him/her start the 2nd AC assignment. You can update this list after the rebuttal period, and the Sub-Committee Chair can notify 2nd ACs accordingly. This task needs to be done soon, as 2nd ACs need time to read and comment on these papers.

Changed lines 23-57 from:
  • The AC can elevate a paper in this category to the category 2 discussion level for situations when they believe that there is a realistic chance that the reject decision should be reconsidered. However, this should be done well before the PC meeting, as a 2nd AC will have to be assigned.
  1. Score <- 2.7, significant divergence (tentative reject)
    • A 2nd AC will be assigned.
    • If both ACs can agree ahead of time that the paper should be rejected, then the ‘discussion’ will be limited to report that outcome at the meeting, otherwise it will be discussed in greater detail.
  2. 2.7 < Score < 4, (uncertain outcome)
    • A 2nd AC will be assigned.
    • If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the tentative decision, followed by discussion as warranted.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  3. 4 <= Score < 4.3 (tentative accept)
    • A 2nd AC will be assigned.
    • If the ACs reach consensus to accept, a quick summary of the issues will be presented at the meeting along with the decision.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  4. 4.3 <= Score <= 5 (accept outright)
    • A 2nd AC will be assigned
    • If the two ACs agrees with accept, this paper will be auto-accepted.
    • If they do not agree, this paper will be scrutinized carefully, with other ACs brought into the process. Rejecting a paper with a high score should be reserved for only the most extreme cases.
    • Discussion will usually be limited to a summary, unless issues are flagged.

The meeting process in brief

The PC Meeting process will cycle between the top- and bottom-rated papers. Doing so will help calibrate decisions. That is, we will have a session starting with the 'best' papers. After a break, we will restart at the bottom, and continue this cycle until we meet in the middle. The process each paper will typically follow is:

  • paper number is announced
  • ACs / SCs in conflict will leave the room
  • The decision is reported by the AC with little or no discussion if:
    • the paper has generally low scores (but still above the auto-reject threshold)
    • the AC/2nd AC agree to a decision
    • the variance of opinion between reviewers is modest
  • If the decision is positive, a brief summary of the paper is reported.
  • If there is disagreement between ACs / 2nd ACs, or if there is high variance, or if any AC wants to discuss issues, then a summary of issues will be presented
    • Depending on the situation, another AC will be assigned to look at the paper and join the discussion (off-line).
    • The above continues until a decision can be reached by the ACs involved. Additional ACs may be brought in as needed.
to:
  • The AC can elevate a paper out of this category (i.e., to have a 2nd AC assigned to the paper and to have it discussed). This should be reserved for situations when they believe that there is a realistic chance that the reject decision should be reconsidered. However, this should be done well before the PC meeting, as a 2nd AC will have to be assigned.
  • ACs should NOT auto-reject a paper if its score is > 2.7, and/or its standard deviation > 1, no matter what their personal opinion is.

A full list of other criteria (assigned after the rebuttal period but before the PC meeting) will be sent to you later. If you want to preview them now, look at http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/CriteriaSecondACs

Providing advice on a 2nd AC

For all papers that are not auto-rejected, recommend two other ACs on your sub-committee who would be appropriate to handle it as a 2nd AC. Then send this list to your SC. Your SC will use your recommendation to make the actual assignment. However, they may not always take your advice, as your SC has to balance workload amongst all ACs. Some things to keep in mind:

  • Do not recommend 2nd ACs outside the committee. The logistics of dealing with this are just to difficult
  • If a particular 2nd AC is really needed due to particular expertise, let your SC know.

The easiest way to do the above is to send your SC a note listing the paper numbers you are responsible for, whether it is in the auto-reject category or not, and the 2nd ACs you recommend e.g.,

Paper 111111 Auto reject
Paper 121111 Discuss. 2nd AC possibilities: Joe Bloggs or Shirley Cohen)
Paper 131111 Auto-reject Paper 121111 Discuss. 2nd AC possibilities: Lucy Brown - I really need her expertise on this one. ...

Your job as 2nd AC

You will receive your 2nd AC assignment via PCS. Recognize that some of your assignments, especially ones just above the auto-reject threshold, may change their status as the ratings of some papers may change as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. However, you will likely be managing most of them.

You can start looking at these papers. Form your own preliminary opinion of it, which you can enter in the Comments to the Committee section of the review form. (Don't put it in the review section, as authors may see this during the rebuttal period). Closer to the PC meeting, you can finalize your opinion.

During the PC meeting, your primary job will be to work with the AC to come to a decision about the paper. If both your decisions are the same and if they don't diverge hugely from referee decisions, then the discussion (led by the primary AC) should be very brief (more on this in another note). If you disagree, you will both present your position at the PC meeting, and likely bring in another AC to look at the paper and discuss it further.

Added lines 1-65:

Hello ACs.

Here are the next steps in your duty assignments.

There is lots of detail in this note, but it is important – please read carefully.

Synopsis of this note:

  • assigning 2nd ACs for ‘discuss papers’
  • cut-offs for papers that will not be discussed.
  • rough outline of the PC Meeting Process

Because of the large volume of papers submitted, we will be rejecting a large number of papers outright; these will not be assigned to 2nd ACs, nor will they be discussed at the PC meeting. Thus your next major duty will be to work with your ACs to determine which papers will likely be rejected outright according to the criteria below, and assign all others to 2nd ACs in your sub-committee that best match the remaining papers. This should be done soon, as 2nd ACs need time to read and comment on these papers. However, this will be complicated somewhat by the fact that some papers may drift across the reject outright/discuss line due to additional reviews, rebuttals, etc. Read on!

Assigning 2nd ACs

  • Immediately ask your ACs to look at the paper scores via the criteria below, and to send you a tentative list of what papers will likely be auto-rejected, auto-accepted, or something inbetween. The 'auto-reject' category is the most important, as those will not have a 2nd AC.
  • All papers that are not auto-rejected will need a 2nd AC. For those papers, you may want to discuss the best choices with the primary AC, but it is your job to balance the AC load and to do the actual assignment. You can assign them via PCS, simply by adding them as a reviewer to the paper.
  • You can then instruct the 2nd ACs that they can start looking at their assignments. Warn them that the ratings of some papers may change: as rebuttals are considered, if reviewers change their comments / scores, or as a result of any discussion. Still, 2nd ACs can read the papers and form their own preliminary opinion of it, which they can track in the Comments to the committee section of the review form.

We will be sending a note to ACs about these responsibilities.

The Criteria.

These are the various criteria for papers. The most important one in the list below is the reject outright one,as 2nd AC assignment depends on this. Other criteria are there to help us focus discussion in the PC meeting.

  1. Score <= 2.7, little divergence - standard deviation <= 1 (auto-reject)
    • These papers are rejected outright.
    • No 2nd AC will be assigned, nor will the paper be discussed at the committee meeting.
    • The AC can elevate a paper in this category to the category 2 discussion level for situations when they believe that there is a realistic chance that the reject decision should be reconsidered. However, this should be done well before the PC meeting, as a 2nd AC will have to be assigned.
  2. Score <- 2.7, significant divergence (tentative reject)
    • A 2nd AC will be assigned.
    • If both ACs can agree ahead of time that the paper should be rejected, then the ‘discussion’ will be limited to report that outcome at the meeting, otherwise it will be discussed in greater detail.
  3. 2.7 < Score < 4, (uncertain outcome)
    • A 2nd AC will be assigned.
    • If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the tentative decision, followed by discussion as warranted.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  4. 4 <= Score < 4.3 (tentative accept)
    • A 2nd AC will be assigned.
    • If the ACs reach consensus to accept, a quick summary of the issues will be presented at the meeting along with the decision.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  5. 4.3 <= Score <= 5 (accept outright)
    • A 2nd AC will be assigned
    • If the two ACs agrees with accept, this paper will be auto-accepted.
    • If they do not agree, this paper will be scrutinized carefully, with other ACs brought into the process. Rejecting a paper with a high score should be reserved for only the most extreme cases.
    • Discussion will usually be limited to a summary, unless issues are flagged.

The meeting process in brief

The PC Meeting process will cycle between the top- and bottom-rated papers. Doing so will help calibrate decisions. That is, we will have a session starting with the 'best' papers. After a break, we will restart at the bottom, and continue this cycle until we meet in the middle. The process each paper will typically follow is:

  • paper number is announced
  • ACs / SCs in conflict will leave the room
  • The decision is reported by the AC with little or no discussion if:
    • the paper has generally low scores (but still above the auto-reject threshold)
    • the AC/2nd AC agree to a decision
    • the variance of opinion between reviewers is modest
  • If the decision is positive, a brief summary of the paper is reported.
  • If there is disagreement between ACs / 2nd ACs, or if there is high variance, or if any AC wants to discuss issues, then a summary of issues will be presented
    • Depending on the situation, another AC will be assigned to look at the paper and join the discussion (off-line).
    • The above continues until a decision can be reached by the ACs involved. Additional ACs may be brought in as needed.

Note that we are following a 'conditional acceptance' practice this year. If a paper is generally sound but requires a modest revision (and we do mean modest) that you can request, this will ensure that the authors have to follow your advice. If they don't we can still reject the paper.