Final Marching Orders for ACs and 2nd ACs

Personal.FinalMarchingOrdersACs History

Hide minor edits - Show changes to markup

Changed line 12 from:
  • Invite reviewers to read the rebuttal if you feel it is warranted, and to discuss the paper using the PCS discussion facility.
to:
  • Invite reviewers to read the rebuttal if you feel it is warranted, and to discuss the paper using the PCS discussion facility. You may not want to bother with this if the paper is a clear accept or reject.
Added line 12:
  • Invite reviewers to read the rebuttal if you feel it is warranted, and to discuss the paper using the PCS discussion facility.
Changed lines 1-2 from:

This is a long letter, but it should be read carefully. It contains your final 'marching orders' that detail your duties up to and including the PC meeting.

to:

(:title Final Marching Orders for ACs and 2nd ACs :) This is a long note, but it should be read carefully. It contains your final 'marching orders' that detail your duties up to and including the PC meeting.

Changed line 10 from:
  • Read or Reread the reviews and the rebuttals carefully (see below).
to:
Deleted lines 78-124:

A Short Guide to Re-evaluating Each Paper.

As AC and 2nd AC, you are now reconsidering reviews and rebuttals. This is the time where you should be willing to change your opinion (if warranted). You should pay special attention to the following points, as these will likely require special scrutiny and discussion during the committee meeting.

Rebuttals:

  • Read them carefully, and be ready to re-appraise your opinion.
  • Authors sometimes suggest revisions. This is ok if revisions are modest. Papers will be conditionally accepted, so you do have opportunities to check that changes were indeed done. However, checking is not the same as going through a 2nd referee round, which will not be done. Thus you have judge what is 'reasonable' revision that can be done within this scope.

Divergence: The AC and 2nd AC should take special care in cases where there is:

  • Large divergence in scores between referees
  • AC scores are mis-aligned with reviewer scores (e.g., lower or higher than all reviewer scores). This is not to say that ACs are limited to the 'average' score of reviewers, but that these papers deserve a strong second look.

Notes vs Papers: A Note needs to be judged as a Note, not as a 10 page paper. Notes are historically judged too harshly.

  • See http://www.chi2009.org/Authors/CallForPapers/PapersVsNotes.html
  • Some reviewers of notes have expectations that are unrealistic for a 4-page paper
  • To summarize, a Note is a much more focused and succinct contribution to research and is likely to have a smaller – yet still significant – scope of contribution. Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue. However, Notes should still be judged with high expectations: they are not posters or reports of works-in-progress.

Keep criticisms in perspective

  • All papers have problems at some level. Some are minor, and these should be treated as such.
  • Highly rated papers with many criticisms are sometimes judged too harshly. Some reviewers score papers highly and say its good, but then spend lots of time critiquing it rather than praising it. This is often a problem with how a referee report is written, vs. a problem with the paper. Thus always take into account the referee's overall appraisal in terms of interpreting the criticisms. If a referee liked the paper, don't overly devalue it just because the referee spent their time listing areas of improvement.

Related work

  • Authors are sometimes over-penalized for missing references.
  • Given the limited paper length, the paper should only be penalized if the missing references are crucial ones, e.g., that previously reported similar contributions that the author is claiming is novel.
  • Also, one should be reasonable about missing references to work far outside the normal CHI literature: you will have to make a judgment call in cases like that.
  • Remember: Adding related work is easy to do in a revision.

Evaluate what was done, not what someone else would have done

  • Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly.
  • This could indicate 'school of thought' differences (see below).

Schools of thought differences

  • Referees may have quite different perspectives and background that make excessively hard on authors. Watch out for this: they may be spotted as rants or tirades or suggestions that the approach taken by authors is completely wrong.
  • You may want to discount referee reports that exhibit this behavior.

Sub-committee fit.

  • See the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html,
  • Important: You should not judge a paper by how well it fits the subcommittee theme. Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.

Contribution type fit.

  • See the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html
  • As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type. However, the submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that.
  • When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI? This is important.
  • The contribution type and its associated questions are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.
  • Look carefully at reviews that have a divergence between the contribution score and the overall score.
Changed lines 12-13 from:
  • Nominate papers for Best Paper. On your review form, you will see a "Best Paper Nomination" area. Check the box, and add a paragraph in the space provided describing why its worthy. Note that nominations are typically (but not always chosen from auto-accept category.
to:
  • Deadline. Nominate papers for Best Paper. On your review form, you will see a "Best Paper Nomination" area. Check the box, and add a paragraph in the space provided describing why its worthy. Note that nominations are typically (but not always chosen from auto-accept category.
Changed line 22 from:
  • ''Deadline: Add a score for each of your assignments by November 30, evening. This will be used to rank-order the papers.
to:
  • Deadline: Add a score for each of your assignments by November 30, evening. This will be used to rank-order the papers.
Changed lines 12-13 from:
  • Nominate papers for Best Paper. On your review form, you will see a "Best Paper Nomination" area. Check the box, and add a paragraph in the space provided describing why its worthy. Note that nominations are typically (but not always_ chosen from auto-accept category.
to:
  • Nominate papers for Best Paper. On your review form, you will see a "Best Paper Nomination" area. Check the box, and add a paragraph in the space provided describing why its worthy. Note that nominations are typically (but not always chosen from auto-accept category.
Changed lines 12-13 from:
to:
  • Nominate papers for Best Paper. On your review form, you will see a "Best Paper Nomination" area. Check the box, and add a paragraph in the space provided describing why its worthy. Note that nominations are typically (but not always_ chosen from auto-accept category.
Changed line 8 from:

ACs and 2nd AC

to:

ACs and 2nd ACs

Added line 21:
  • ''Deadline: Add a score for each of your assignments by November 30, evening. This will be used to rank-order the papers.
Changed lines 114-116 from:
  • you should not judge a paper by how well it fits the subcommittee theme.
  • Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.
to:
  • Important: You should not judge a paper by how well it fits the subcommittee theme. Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.
Changed lines 121-122 from:
to:
  • Look carefully at reviews that have a divergence between the contribution score and the overall score.
Changed lines 96-98 from:
  • Highly rated papers with many criticisms are sometimes judged incorrectly. Some reviewers score papers highly and say its good, but then spend lots of time critiquing it rather than praising it. This is often a problem with how a referee report is written, vs. a problem with the paper. Thus always take into account the referee's overall appraisal in terms of interpreting the criticisms. If a referee liked the paper, don't overly devalue it just because the referee spent their time listing areas of improvement.
to:
  • Highly rated papers with many criticisms are sometimes judged too harshly. Some reviewers score papers highly and say its good, but then spend lots of time critiquing it rather than praising it. This is often a problem with how a referee report is written, vs. a problem with the paper. Thus always take into account the referee's overall appraisal in terms of interpreting the criticisms. If a referee liked the paper, don't overly devalue it just because the referee spent their time listing areas of improvement.
Changed lines 94-100 from:

Minor vs. Major Points

  • Keep criticisms in perspective. All papers have problems at some level. Some are minor, and these should be treated as such.

Highly rated papers with many criticisms.

  • Some reviewers score papers highly and say its good, but then spend lots of time critiquing it rather than praising it. This is often a problem with how a referee report is written, vs. a problem with the paper.
  • Always take into account the referee's overall appraisal in terms of interpreting the criticisms. That is, if they like the paper, don't overly devalue it just because the referee spent their time listing areas of improvement.
to:

Keep criticisms in perspective

  • All papers have problems at some level. Some are minor, and these should be treated as such.
  • Highly rated papers with many criticisms are sometimes judged incorrectly. Some reviewers score papers highly and say its good, but then spend lots of time critiquing it rather than praising it. This is often a problem with how a referee report is written, vs. a problem with the paper. Thus always take into account the referee's overall appraisal in terms of interpreting the criticisms. If a referee liked the paper, don't overly devalue it just because the referee spent their time listing areas of improvement.
Changed lines 93-94 from:
to:

Minor vs. Major Points

  • Keep criticisms in perspective. All papers have problems at some level. Some are minor, and these should be treated as such.

Highly rated papers with many criticisms.

  • Some reviewers score papers highly and say its good, but then spend lots of time critiquing it rather than praising it. This is often a problem with how a referee report is written, vs. a problem with the paper.
  • Always take into account the referee's overall appraisal in terms of interpreting the criticisms. That is, if they like the paper, don't overly devalue it just because the referee spent their time listing areas of improvement.
Changed lines 99-100 from:
to:
  • Remember: Adding related work is easy to do in a revision.
Changed lines 102-104 from:
  • Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly. * * This could indicate 'school of thought' differences (see below).
  • Evaluate what the authors did vs. what a referee would have done.
to:
  • Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly.
  • This could indicate 'school of thought' differences (see below).
Changed lines 18-19 from:
  • If you want to elevate a paper with a score < 2.7 and a std deviation < 1 to the discuss category, please check the "To be Discussed at the PC Meeting" box on the paper's review form.
to:
  • If you want to elevate a paper with a score < 2.7 and a std deviation < 1 to the discuss category, please check the "To be Discussed at the PC Meeting" box on the paper's review form and tell your Sub-Committee Chair as he/she will have to assign a 2nd AC to it.
Changed line 21 from:
  • You do NOT have to write a review, but you should write an opinion.
to:
  • You do NOT have to write a full review, but you should write an opinion.
Changed lines 72-73 from:
to:
  • ACs should be prepared to discuss who the reviewers are and why they chose them.
Changed line 40 from:
  1. Score <- 2.7, significant divergence (tentative reject)
to:
  1. Score <= 2.7, significant divergence (tentative reject)
Changed line 5 from:

All:

to:

All:

Changed line 8 from:

ACs and 2nd AC

to:

ACs and 2nd AC

Changed line 13 from:

For ACs

to:

For ACs

Changed line 20 from:

For 2nd ACs

to:

For 2nd ACs

Changed line 26 from:

Balance your efforts.

to:

Balance your efforts.

Changed line 35 from:
  1. Score <= 2.7, little divergence - standard deviation <= 1 OR standard deviation > 1 but no single score is above 3.0 (auto-reject)
to:
  1. Score <= 2.7, little divergence - standard deviation <= 1 OR standard deviation > 1 but no single score is above 3.0 (auto-reject)
Changed line 40 from:
  1. Score <- 2.7, significant divergence (tentative reject)
to:
  1. Score <- 2.7, significant divergence (tentative reject)
Changed line 44 from:
  1. 2.7 < Score < 4, (uncertain outcome)
to:
  1. 2.7 < Score < 4, (uncertain outcome)
Changed line 49 from:
  1. 4 <= Score < 4.3 (tentative accept)
to:
  1. 4 <= Score < 4.3 (tentative accept)
Changed line 54 from:
  1. 4.3 <= Score <= 5 (accept outright)
to:
  1. 4.3 <= Score <= 5 (accept outright)
Changed line 80 from:

Rebuttals:

to:

Rebuttals:

Changed line 84 from:

Divergence: The AC and 2nd AC should take special care in cases where there is:

to:

Divergence: The AC and 2nd AC should take special care in cases where there is:

Changed line 88 from:

Notes vs Papers: A Note needs to be judged as a Note, not as a 10 page paper. Notes are historically judged too harshly.

to:

Notes vs Papers: A Note needs to be judged as a Note, not as a 10 page paper. Notes are historically judged too harshly.

Changed line 94 from:

Related work

to:

Related work

Changed line 99 from:

Evaluate what was done, not what someone else would have done

to:

Evaluate what was done, not what someone else would have done

Changed line 103 from:

Schools of thought

to:

Schools of thought differences

Changed line 107 from:

Sub-committee fit.

to:

Sub-committee fit.

Changed line 112 from:

Contribution type fit.

to:

Contribution type fit.

Changed lines 76-85 from:

A Second Look: A Short Guide to Re-evaluating Each Paper. ======= A Second Look: A Short Guide to Re-evaluating Each Paper. >>>>>>>


to:

A Short Guide to Re-evaluating Each Paper.

Changed lines 3-5 from:

What to do now.


to:

What to do now.

Changed lines 54-55 from:

<<<<<<<

to:
Changed lines 28-30 from:
  • Low-scoring and high-scoring papers with non-divergent reviews will likely need a modest check from you. If you agree with the opinion, then just say so.
  • Papers with middle scores, and / or with divergent scores are likely the ones that will be discussed most. Your opinion should be more detailed in these cases.
to:
  • See The Criteria section below.
  • Using the criteria below, put more effort into papers that will likely require more discussion, e.g., papers with middle scores, or with divergent scores, or where there is disagreement between AC and 2nd AC are likely the ones that will be discussed most. Your opinion should be more detailed in these cases.

The Criteria.

These are the various criteria for papers. The most important ones in the list below are the reject and accept outright. Other criteria are there to help us focus discussion in the PC meeting.

  1. Score <= 2.7, little divergence - standard deviation <= 1 OR standard deviation > 1 but no single score is above 3.0 (auto-reject)
    • These papers are rejected outright.
    • No 2nd AC will be assigned, nor will the paper be discussed at the committee meeting.
    • The AC can elevate a paper in this category to the category 2 discussion level for situations when they believe that there is a realistic chance that the reject decision should be reconsidered. However, this should be done well before the PC meeting, as a 2nd AC will have to be assigned.
  2. Score <- 2.7, significant divergence (tentative reject)
    • A 2nd AC will be assigned.
    • If both ACs can agree ahead of time that the paper should be rejected, then the ‘discussion’ will be limited to report that outcome at the meeting, otherwise it will be discussed in greater detail.
  3. 2.7 < Score < 4, (uncertain outcome)
    • A 2nd AC will be assigned.
    • If the ACs can reach consensus, a summary of the issues will be presented at the meeting along with the tentative decision, followed by discussion as warranted.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
  4. 4 <= Score < 4.3 (tentative accept)
    • A 2nd AC will be assigned.
    • If the ACs reach consensus to accept, a quick summary of the issues will be presented at the meeting along with the decision.
    • If the ACs cannot reach consensus, or if there is wide divergence (>=1), a quick summary of the issues will be presented with the likelihood that at least one other AC will be solicited to read the paper and inform the decision process.
Added lines 55-81:

<<<<<<<

  1. 4.3 <= Score <= 5 (accept outright)
    • A 2nd AC will be assigned
    • If the two ACs agrees with accept, this paper will be auto-accepted.
    • If they do not agree, this paper will be scrutinized carefully, with other ACs brought into the process. Rejecting a paper with a high score should be reserved for only the most extreme cases.
    • Discussion will usually be limited to a summary, unless issues are flagged.

The meeting process in brief

The PC Meeting process will cycle between the top- and bottom-rated papers. Doing so will help calibrate decisions. That is, we will have a session starting with the 'best' papers. After a break, we will restart at the bottom, and continue this cycle until we meet in the middle. The process each paper will typically follow is:

  • paper number is announced
  • ACs / SCs in conflict will leave the room
  • The decision is reported by the AC with little or no discussion if:
    • the paper has generally low scores (but still above the auto-reject threshold)
    • the AC/2nd AC agree to a decision
    • the variance of opinion between reviewers is modest
  • If the decision is positive, a brief summary of the paper is reported.
  • If there is disagreement between ACs / 2nd ACs, or if there is high variance, or if any AC wants to discuss issues, then a summary of issues will be presented
    • Depending on the situation, another AC will be assigned to look at the paper and join the discussion (off-line).
    • The above continues until a decision can be reached by the ACs involved. Additional ACs may be brought in as needed.

Note that we are following a 'conditional acceptance' practice this year. If a paper is generally sound but requires a modest revision (and we do mean modest) that you can request, this will ensure that the authors have to follow your advice. If they don't we can still reject the paper.

Added lines 83-85:

======= A Second Look: A Short Guide to Re-evaluating Each Paper. >>>>>>>

Changed line 32 from:

A second look: a short guide to re-evaluating each paper.

to:

A Second Look: A Short Guide to Re-evaluating Each Paper.

Deleted line 34:
Changed lines 7-8 from:

- 2nd ACs are now being assigned. Your sub-committee chair will tell you when it is complete.

to:
  • 2nd ACs are now being assigned. Your sub-committee chair will tell you when it is complete.
Changed lines 10-13 from:

- The rebuttals are now all in. - Read or Reread the reviews and the rebuttals carefully (see below). - Change your opinion and scoring if needed.

to:
  • The rebuttals are now all in.
  • Read or Reread the reviews and the rebuttals carefully (see below).
  • Change your opinion and scoring if needed.
Changed lines 15-20 from:

- Put in a paragraph in the 'Associate Chair Additional Comments' box on the review form that tentatively stakes out your (possibly revised) opinion. - You will have opportunity to edit this again at or immediately after the PC meeting. - The 2nd AC will be able to see this. - Be polite and compassionate in this: make it clear that you have read the rebuttal, and that you have heard what the author has to say. - If you want to elevate a paper with a score < 2.7 and a std deviation < 1 to the discuss category, please check the "To be Discussed at the PC Meeting" box on the paper's review form.

to:
  • Put in a paragraph in the 'Associate Chair Additional Comments' box on the review form that tentatively stakes out your (possibly revised) opinion.
  • You will have opportunity to edit this again at or immediately after the PC meeting.
  • The 2nd AC will be able to see this.
  • Be polite and compassionate in this: make it clear that you have read the rebuttal, and that you have heard what the author has to say.
  • If you want to elevate a paper with a score < 2.7 and a std deviation < 1 to the discuss category, please check the "To be Discussed at the PC Meeting" box on the paper's review form.
Changed lines 22-26 from:

- You do NOT have to write a review, but you should write an opinion. - Put this opinion in the 'Comments to the Committee': only the committee will see it. - You should add a score to the paper. The score should reflect your opinion as well as your reading of all referee reports and the rebuttal. - You may be asked to transform this opinion into a review at the committee meeting, if it is warranted.

to:
  • You do NOT have to write a review, but you should write an opinion.
  • Put this opinion in the 'Comments to the Committee': only the committee will see it.
  • You should add a score to the paper. The score should reflect your opinion as well as your reading of all referee reports and the rebuttal.
  • You may be asked to transform this opinion into a review at the committee meeting, if it is warranted.
Changed lines 28-30 from:

- Low-scoring and high-scoring papers with non-divergent reviews will likely need a modest check from you. If you agree with the opinion, then just say so. - Papers with middle scores, and / or with divergent scores are likely the ones that will be discussed most. Your opinion should be more detailed in these cases.

to:
  • Low-scoring and high-scoring papers with non-divergent reviews will likely need a modest check from you. If you agree with the opinion, then just say so.
  • Papers with middle scores, and / or with divergent scores are likely the ones that will be discussed most. Your opinion should be more detailed in these cases.
Deleted lines 31-33:

Things for ACs and SCs to watch out for.


Changed lines 33-35 from:
to:

Changed lines 39-51 from:

-Read them carefully, and be ready to re-appraise your opinion. -Authors sometimes suggest revisions. This is ok if revisions are modest. Papers will be conditionally accepted, so you do have opportunities to check that changes were indeed done. However, checking is not the same as going through a 2nd referee round, which will not be done. Thus you have judge what is 'reasonable' revision that can be done within this scope.

Divergence: The AC and 2nd AC should take special care in cases where there is: - Large divergence in scores between referees - AC scores are mis-aligned with reviewer scores (e.g., lower or higher than all reviewer scores). This is not to say that ACs are limited to the 'average' score of reviewers, but that these papers deserve a strong second look.

Notes vs Papers A Note needs to be judged as a Note, not as a 10 page paper. Notes are historically judged too harshly. - See http://www.chi2009.org/Authors/CallForPapers/PapersVsNotes.html - Some reviewers of notes have expectations that are unrealistic for a 4-page paper - To summarize, a Note is a much more focused and succinct contribution to research and is likely to have a smaller – yet still significant – scope of contribution. Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue. However, Notes should still be judged with high expectations: they are not posters or reports of works-in-progress.

to:
  • Read them carefully, and be ready to re-appraise your opinion.
  • Authors sometimes suggest revisions. This is ok if revisions are modest. Papers will be conditionally accepted, so you do have opportunities to check that changes were indeed done. However, checking is not the same as going through a 2nd referee round, which will not be done. Thus you have judge what is 'reasonable' revision that can be done within this scope.

Divergence: The AC and 2nd AC should take special care in cases where there is:

  • Large divergence in scores between referees
  • AC scores are mis-aligned with reviewer scores (e.g., lower or higher than all reviewer scores). This is not to say that ACs are limited to the 'average' score of reviewers, but that these papers deserve a strong second look.

Notes vs Papers: A Note needs to be judged as a Note, not as a 10 page paper. Notes are historically judged too harshly.

  • See http://www.chi2009.org/Authors/CallForPapers/PapersVsNotes.html
  • Some reviewers of notes have expectations that are unrealistic for a 4-page paper
  • To summarize, a Note is a much more focused and succinct contribution to research and is likely to have a smaller – yet still significant – scope of contribution. Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue. However, Notes should still be judged with high expectations: they are not posters or reports of works-in-progress.
Changed lines 53-54 from:

Authors are sometimes over-penalized for missing references. Given the limited paper length, the paper should only be penalized if the missing references are crucial ones, e.g., that previously reported similar contributions that the author is claiming is novel. Also, one should be reasonable, e.g., to a missing reference to work far outside the normal CHI literature: you will have to make a judgment call in cases like that.

to:
  • Authors are sometimes over-penalized for missing references.
  • Given the limited paper length, the paper should only be penalized if the missing references are crucial ones, e.g., that previously reported similar contributions that the author is claiming is novel.
  • Also, one should be reasonable about missing references to work far outside the normal CHI literature: you will have to make a judgment call in cases like that.
Changed lines 58-59 from:

Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly. This could indicate 'school of thought' differences as there are many ways to tackle a problem. Evaluate what the authors did vs. what a referee would have done.

to:
  • Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly. * * This could indicate 'school of thought' differences (see below).
  • Evaluate what the authors did vs. what a referee would have done.
Changed lines 62-64 from:

- Referees may have quite different perspectives and background that make excessively hard on authors. Watch out for this: they may be spotted as rants or tirades or suggestions that the approach taken by authors is completely wrong. - You may want to discount referee reports that exhibit this behavior.

to:
  • Referees may have quite different perspectives and background that make excessively hard on authors. Watch out for this: they may be spotted as rants or tirades or suggestions that the approach taken by authors is completely wrong.
  • You may want to discount referee reports that exhibit this behavior.
Changed lines 66-67 from:

As mentioned in the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html, you should not judge a paper by how well it fits the subcommittee theme. Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.

to:
  • See the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html,
  • you should not judge a paper by how well it fits the subcommittee theme.
  • Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.
Changed lines 71-73 from:

Also mentioned in the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html : As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type. However, the submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that. When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI? This is important. The contribution type and its associated questions are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.

to:
  • See the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html
  • As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type. However, the submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that.
  • When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI? This is important.
  • The contribution type and its associated questions are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.
Added lines 6-8:

All: - 2nd ACs are now being assigned. Your sub-committee chair will tell you when it is complete.

Changed line 15 from:

- Put in a paragraph at the beginning of your meta-review that tentatively stakes out your (possibly revised) opinion.

to:

- Put in a paragraph in the 'Associate Chair Additional Comments' box on the review form that tentatively stakes out your (possibly revised) opinion.

Changed lines 18-19 from:

- Be polite and compassionate in this: make it clear that you have read the rebuttal, and that you have heard what the author has to say.

to:

- Be polite and compassionate in this: make it clear that you have read the rebuttal, and that you have heard what the author has to say. - If you want to elevate a paper with a score < 2.7 and a std deviation < 1 to the discuss category, please check the "To be Discussed at the PC Meeting" box on the paper's review form.

Changed lines 22-30 from:

- You do NOT have to write a review, but you should write an opinion. If you put it in the 'For the Committee Box',

to:

- You do NOT have to write a review, but you should write an opinion. - Put this opinion in the 'Comments to the Committee': only the committee will see it. - You should add a score to the paper. The score should reflect your opinion as well as your reading of all referee reports and the rebuttal. - You may be asked to transform this opinion into a review at the committee meeting, if it is warranted.

Balance your efforts. - Low-scoring and high-scoring papers with non-divergent reviews will likely need a modest check from you. If you agree with the opinion, then just say so. - Papers with middle scores, and / or with divergent scores are likely the ones that will be discussed most. Your opinion should be more detailed in these cases.

Added lines 1-19:

This is a long letter, but it should be read carefully. It contains your final 'marching orders' that detail your duties up to and including the PC meeting.

What to do now.


ACs and 2nd AC - The rebuttals are now all in. - Read or Reread the reviews and the rebuttals carefully (see below). - Change your opinion and scoring if needed.

For ACs - Put in a paragraph at the beginning of your meta-review that tentatively stakes out your (possibly revised) opinion. - You will have opportunity to edit this again at or immediately after the PC meeting. - The 2nd AC will be able to see this. - Be polite and compassionate in this: make it clear that you have read the rebuttal, and that you have heard what the author has to say.

For 2nd ACs - You do NOT have to write a review, but you should write an opinion. If you put it in the 'For the Committee Box',

Changed lines 23-24 from:

As you reconsider reviews and rebuttals, pay special attention to the following points, as these will likely require special scrutiny and discussion during the committee meeting.

to:

A second look: a short guide to re-evaluating each paper.

As AC and 2nd AC, you are now reconsidering reviews and rebuttals. This is the time where you should be willing to change your opinion (if warranted). You should pay special attention to the following points, as these will likely require special scrutiny and discussion during the committee meeting.

Changed lines 29-30 from:

-Authors sometimes suggest revisions. This is ok, especially if they are modest. Papers will be conditionally accepted, so you do have opportunities to check that changes were indeed done. However, checking is not the same as refereeing. Thus you have judge what is 'reasonable' revision that can be done within this scope.

to:

-Authors sometimes suggest revisions. This is ok if revisions are modest. Papers will be conditionally accepted, so you do have opportunities to check that changes were indeed done. However, checking is not the same as going through a 2nd referee round, which will not be done. Thus you have judge what is 'reasonable' revision that can be done within this scope.

Changed line 32 from:

The 2nd AC should take special care in cases where there is:

to:

The AC and 2nd AC should take special care in cases where there is:

Changed line 37 from:

A note needs to be judges as a note, not as a 10 page paper. Notes are sometimes judged too harshly as a result.

to:

A Note needs to be judged as a Note, not as a 10 page paper. Notes are historically judged too harshly.

Changed lines 39-41 from:

- To summarize, a Note is a much more focused and succinct contribution to research and is likely to have a smaller – yet still significant – scope of contribution. Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue. - Some reviewers of notes have expectations that are unrealistic for a 4-page paper.

to:

- Some reviewers of notes have expectations that are unrealistic for a 4-page paper - To summarize, a Note is a much more focused and succinct contribution to research and is likely to have a smaller – yet still significant – scope of contribution. Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue. However, Notes should still be judged with high expectations: they are not posters or reports of works-in-progress.

Changed lines 32-33 from:

As mentioned in the reviewing guide, you should not judge a paper by how well it fits the subcommittee theme. Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.

to:

As mentioned in the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html, you should not judge a paper by how well it fits the subcommittee theme. Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.

Changed lines 35-37 from:

Also mentioned in the reviewing guide: As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type. However, the submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that. When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI? This is important. The contribution type and its associated questions are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.

to:

Also mentioned in the reviewing guide http://www.chi2009.org/Authors/Guides/ArchiveWhatsChanged.html : As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type. However, the submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that. When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI? This is important. The contribution type and its associated questions are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.

Changed lines 24-25 from:

Evaluated what was done, not what the reviewer would have done Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly. This is often unfair, as there are many ways to tackle a problem. Evaluate what was done in the paper vs. what wasn't done.

to:

Evaluate what was done, not what someone else would have done Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly. This could indicate 'school of thought' differences as there are many ways to tackle a problem. Evaluate what the authors did vs. what a referee would have done.

Schools of thought - Referees may have quite different perspectives and background that make excessively hard on authors. Watch out for this: they may be spotted as rants or tirades or suggestions that the approach taken by authors is completely wrong. - You may want to discount referee reports that exhibit this behavior.

Sub-committee fit. As mentioned in the reviewing guide, you should not judge a paper by how well it fits the subcommittee theme. Sub-committees are strictly an administrative entity for matching papers to referees and for handling large numbers.

Contribution type fit. Also mentioned in the reviewing guide: As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type. However, the submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that. When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI? This is important. The contribution type and its associated questions are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.

Changed lines 2-3 from:

to:

Changed lines 13-14 from:

- AC score diverges with the referee score (e.g., lower or higher than all reviewer scores):

to:

- AC scores are mis-aligned with reviewer scores (e.g., lower or higher than all reviewer scores). This is not to say that ACs are limited to the 'average' score of reviewers, but that these papers deserve a strong second look.

Changed lines 21-25 from:

Over-reaction to small issues - Missing references: Referees often point to a missing reference and sometimes devalue the paper for missing it. In many cases, these references can be easily included in a minor revision. - Missing references to work done outside of CHI: Referees sometimes point to key papers well outside of the normal CHI literature that parallels the reported work. You will have to make a judgment call here: reinvention across fields is common.

to:

Related work Authors are sometimes over-penalized for missing references. Given the limited paper length, the paper should only be penalized if the missing references are crucial ones, e.g., that previously reported similar contributions that the author is claiming is novel. Also, one should be reasonable, e.g., to a missing reference to work far outside the normal CHI literature: you will have to make a judgment call in cases like that.

Evaluated what was done, not what the reviewer would have done Referees sometimes argue that the paper should have been done a different way, and penalize the paper accordingly. This is often unfair, as there are many ways to tackle a problem. Evaluate what was done in the paper vs. what wasn't done.

Added lines 1-24:

Things for ACs and SCs to watch out for.


As you reconsider reviews and rebuttals, pay special attention to the following points, as these will likely require special scrutiny and discussion during the committee meeting.

Rebuttals: -Read them carefully, and be ready to re-appraise your opinion. -Authors sometimes suggest revisions. This is ok, especially if they are modest. Papers will be conditionally accepted, so you do have opportunities to check that changes were indeed done. However, checking is not the same as refereeing. Thus you have judge what is 'reasonable' revision that can be done within this scope.

Divergence: The 2nd AC should take special care in cases where there is: - Large divergence in scores between referees - AC score diverges with the referee score (e.g., lower or higher than all reviewer scores):

Notes vs Papers A note needs to be judges as a note, not as a 10 page paper. Notes are sometimes judged too harshly as a result. - See http://www.chi2009.org/Authors/CallForPapers/PapersVsNotes.html - To summarize, a Note is a much more focused and succinct contribution to research and is likely to have a smaller – yet still significant – scope of contribution. Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue. - Some reviewers of notes have expectations that are unrealistic for a 4-page paper.

Over-reaction to small issues - Missing references: Referees often point to a missing reference and sometimes devalue the paper for missing it. In many cases, these references can be easily included in a minor revision. - Missing references to work done outside of CHI: Referees sometimes point to key papers well outside of the normal CHI literature that parallels the reported work. You will have to make a judgment call here: reinvention across fields is common.