ACM CHI PC Meeting Process Description

Personal.PCMeetingProcessDescription History

Show minor edits - Show changes to output

Changed lines 103-105 from:
To help later session creation sub-committees are asked to provide clusters of topically related papers.

During the decision meeting:
to:
To help later session creation sub-committees are asked to provide clusters of topically related papers. During the decision meeting:
Changed line 59 from:
* SC announces title, authors and institutions
to:
* SC announces title, authors, institutions, Note vs. Paper type, and Contribution Type.
Added line 62:
* AC quickly names reviewers and indicates why they were selected (optionally omitted to save time)
Changed lines 1-2 from:
(:title ACM CHI PC Meeting Process Description
to:
(:title ACM CHI PC Meeting Process Description :)
Added lines 1-2:
(:title ACM CHI PC Meeting Process Description
Changed lines 108-111 from:
to:
!!If your SC or AC duties are done early
Some sub-committee groups are handling smaller sets of papers than others, and thus may be done earlier. If this happens, you may be invited to serve as a reader for papers being discussed by other groups OR sit in on the deliberation.
Changed line 56 from:
Discuss papers have 4.3 > score > 2.7 OR it was an auto-reject paper marked for discussion or has a high variance > 1.0. Expectations on the amount of discussion varies with the actual score (see below).
to:
Discuss papers have 4.3 > score > 2.7 OR it was an auto-reject paper marked for discussion or has a high variance > 1.0. Expectations on the amount of discussion varies with the actual score.
Changed line 56 from:
Discuss papers have 4.3 > score > 2.7 OR is auto-reject paper marked for discussion, or with a high variance > 1.0. Expectations on the amount of discussion varies with the actual score (see below).
to:
Discuss papers have 4.3 > score > 2.7 OR it was an auto-reject paper marked for discussion or has a high variance > 1.0. Expectations on the amount of discussion varies with the actual score (see below).
Changed lines 53-54 from:
* Everything in-between will require good discussions before a decision is reached.
to:
* The decision on papers with in-between scores will be based on a good discussion between its readers
Changed lines 53-54 from:
* Everything in-between will require discussion.
to:
* Everything in-between will require good discussions before a decision is reached.
Changed lines 49-54 from:
!!Expectations
Highly ranked papers will likely be accepted.
Low-ranked papers will likely be rejected.
Everything in-between will require discussion.

!!!The process
for each ''discuss'' paper
to:
!! How Individual Papers are Handled in the SC Meeting
In general, expect the following:
* Highly ranked papers will likely be accepted .
* Low-ranked papers will likely be rejected.
* Everything in-between will require discussion.

!!!The Process
for each ''discuss'' paper
Changed line 41 from:
* All others: see 'Discuss process' below
to:
* All others: see 'Process for each Discuss paper' below
Changed line 58 from:
* People in conflict MUST leave the room (see below)
to:
* People in conflict MUST leave the room (see conflicts of interest below)
Changed lines 63-64 from:
->AC & 2AC indicate if paper should be considered for best paper nomination discussion later
-> (all papers > 4.3 are automatically considered in the best paper nomination discussion later).
to:
->AC & 2AC indicate if paper should be considered for best paper nomination; brief discussion if it is nominated
Changed line 68 from:
* For full discussion papers
to:
* All others (full discussion)
Changed lines 78-84 from:
Conflicts of interests
* Same institution as a co-author. This includes separate labs of same company (e.g., IBM TJ Watson and IBM Almaden), but not different campuses in a university system (e.g., not UC Berkeley and UC San Diego).
* Current or recent (last year, optionally longer) co-author or close collaborator with a co-author.
* Recently (last year) left a conflicting institution, or currently being considered for employment there.
* Strong financial ties that might be seen by others as a CoI (i.e., major funding from an institution).
* Advisor or advisee of co-author (optionally expires after 4 years)
* Any conflict (possibly unstated) which you feel might affect your ability to remain unbiased, or which you feel someone else might reasonably see as affecting your objectivity.
to:
Added lines 89-97:
!!! Conflicts of interests
You need to leave the room if you have a conflict, as defined below.
* Same institution as a co-author. This includes separate labs of same company (e.g., IBM TJ Watson and IBM Almaden), but not different campuses in a university system (e.g., not UC Berkeley and UC San Diego).
* Current or recent (last year, optionally longer) co-author or close collaborator with a co-author.
* Recently (last year) left a conflicting institution, or currently being considered for employment there.
* Strong financial ties that might be seen by others as a CoI (i.e., major direct funding from an institution).
* Advisor or advisee of co-author (optionally expires after 4 years)
* Any conflict (possibly unstated) which you feel might affect your ability to remain unbiased, or which you feel someone else might reasonably see as affecting your objectivity.
Changed line 55 from:
Discuss papers have 4.3 > score > 2.7 OR is auto-reject paper marked for discussion, or with a high variance > 1.0.
to:
Discuss papers have 4.3 > score > 2.7 OR is auto-reject paper marked for discussion, or with a high variance > 1.0. Expectations on the amount of discussion varies with the actual score (see below).
Changed lines 56-59 from:
# SC announces title, authors and institutions
# SC projects submission onto the display (optional)
# People in conflict MUST leave the room (see below)
# If paper is likely-accept (>= 4.0)
to:
* SC announces title, authors and institutions
* SC projects submission onto the display (optional)
* People in conflict MUST leave the room (see below)
* If paper is likely-accept (>= 4.0)
Changed lines 65-66 from:
# If paper is likely-reject (< 3.0)
to:
* If paper is likely-reject (< 3.0)
Changed lines 69-70 from:
# For full discussion papers
to:
* For full discussion papers
Changed line 75 from:
Notes about paper discussions:
to:
!!!Notes about paper discussions:
Changed line 59 from:
## If paper is likely-accept (>= 4.0)
to:
# If paper is likely-accept (>= 4.0)
Changed line 66 from:
## If paper is likely-reject (< 3.0)
to:
# If paper is likely-reject (< 3.0)
Changed line 71 from:
## For full discussion papers
to:
# For full discussion papers
Changed lines 56-62 from:
* SC announces title, authors and institutions

* SC projects submission onto the display (optional)

* People in conflict MUST leave the room (authors, collaborators, institution)

* If paper is likely-accept (>= 4.0)
to:
# SC announces title, authors and institutions
# SC projects submission onto the display (optional)
# People in conflict MUST leave the room (see below)
## If paper is likely-accept (>= 4.0)
Changed line 66 from:
* If paper is likely-reject (< 3.0)
to:
## If paper is likely-reject (< 3.0)
Changed line 71 from:
* For full discussion papers
to:
## For full discussion papers
Changed lines 54-55 from:
!!!The process for each ''discuss'' paper (score < 4.3 and > 2.7 and not marked for discussion):
to:
!!!The process for each ''discuss'' paper
Discuss papers have 4.3 > score > 2.7 OR is auto-reject paper marked for discussion, or with a high variance > 1.0.
Changed line 54 from:
!!The process for each ''discuss'' paper (papers scoring < 4.3 and > 2.7 and not marked for discussion):
to:
!!!The process for each ''discuss'' paper (score < 4.3 and > 2.7 and not marked for discussion):
Changed lines 21-23 from:
These meetings are chaired by the sub-committee chair in a breakout room.

The
following will be supplied:
to:
These meetings are chaired by the sub-committee chair in a breakout room. The following materials will be supplied.
Changed lines 17-18 from:
A final report will be provided on what has been accepted and rejected, and we will discuss suggestions for process improvements.
to:
* SCs: final report on what has been accepted, rejected, and nominated for best paper
* All:
discuss suggestions for process improvements.
Changed lines 2-3 from:
As seen in the [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/PCMeetingSchedule|agenda]], as needed we will have several short plenary meetings where all sub-committees are present in a single room. These plenary meetings will work more or less as follows.
to:
As seen in the [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/PCMeetingSchedule|agenda]], we will have several short plenary meetings where all sub-committees are present in a single room. These plenary meetings will work more or less as follows.
Changed lines 40-41 from:
At this point, you will start discussing each paper. Start with the '''Likely-Accepts''' and work your way down.
to:
At this point, you will start discussing each paper starting from the top scoring ones.
* Auto-accept >= 4
.3: each AC briefly summarizes the paper content (no discussion needed)
* All others: see 'Discuss process' below
* Best paper nominees: Discuss as they come up
Changed lines 11-13 from:
* Saul and Scott will provide higher-level comments on any issues or process adjustments they have seen.
* Sub-committee chairs (that's you) will report on progress accepting/rejecting papers, and describe any issues that have arisen.
* If needed, particular papers with very specific concerns requiring global concensus can be discussed at these meetings (although those in conflict will have to leave).
to:
* Saul and Scott: provides higher-level comments on any issues or process adjustments they have seen.
* Sub-committee chairs (that's you): report on progress accepting/rejecting papers, and describe any issues that have arisen.
* All: If needed, particular papers with very specific concerns requiring global concensus can be discussed at these meetings (although those in conflict will have to leave).
Changed lines 36-37 from:
** ACs describe what auto-reject papers need to be discussed.
to:
** ACs verify auto-accept/auto-reject papers
** ACs list what auto-reject papers need to be discussed (or controversial auto-accepts)
** ACs provide initial list of best paper nominees (details to be discussed when the paper is discussed)
Added line 54:
Added line 56:
Added line 58:
Added line 65:
Added line 70:
Changed lines 14-15 from:
* On Friday, each group will summarize the papers being considered for best paper.
to:
* Late on Thursday, each group will summarize a subset of best paper nominations, which will be finalized on Friday.
Added lines 74-80:
Conflicts of interests
* Same institution as a co-author. This includes separate labs of same company (e.g., IBM TJ Watson and IBM Almaden), but not different campuses in a university system (e.g., not UC Berkeley and UC San Diego).
* Current or recent (last year, optionally longer) co-author or close collaborator with a co-author.
* Recently (last year) left a conflicting institution, or currently being considered for employment there.
* Strong financial ties that might be seen by others as a CoI (i.e., major funding from an institution).
* Advisor or advisee of co-author (optionally expires after 4 years)
* Any conflict (possibly unstated) which you feel might affect your ability to remain unbiased, or which you feel someone else might reasonably see as affecting your objectivity.
November 27, 2008, at 06:40 PM by 67.165.66.127 -
Changed lines 2-3 from:
As seen in the [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/PCMeetingSchedule|agenda]], we will have several short plenary meetings where all sub-committees are present in a single room. These plenary meetings will work more or less as follows.
to:
As seen in the [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/PCMeetingSchedule|agenda]], as needed we will have several short plenary meetings where all sub-committees are present in a single room. These plenary meetings will work more or less as follows.
Added lines 8-10:
For each of the remaining potential plenary sessions, a quick meeting of all the sub-committee chairs will be held to uncover issues, address questions, and determine if a full plenary session is needed, or continued work in sub-committees will be most productive.

For plenary sessions:
Changed lines 14-15 from:
* Depending on the time available, each group will summarize the papers being considered for best paper.
to:
* On Friday, each group will summarize the papers being considered for best paper.
Changed lines 17-18 from:
A final report will be provided on what has been accepted and rejected.
to:
A final report will be provided on what has been accepted and rejected, and we will discuss suggestions for process improvements.
Changed lines 23-24 from:
* Paper copies for all ACS of two spread sheets, one ordered by paper number, the other by score. These will reflect the PCS system score ranking for papers handled by your group as of Monday night.
* Paper copies for SCs of above, but with AC names included (to be kept confidential by ACs)
to:
* Paper copies for all ACs of two spread sheets, one ordered by paper number, the other by score. These will reflect the PCS system score ranking for papers handled by your group as of Monday night.
Changed lines 26-28 from:
* Office supplies (flip chart, marking pens, tape, postits)
* Two student volunteers (shared by all)
to:
* Office supplies (flip chart, marking pens, tape, post-its)
* Two student volunteers (shared across all sub-committees)
Changed lines 38-39 from:
At this point, you will start discussing each paper. Start with the '''Auto Accepts''' and work your way down.
to:
At this point, you will start discussing each paper. Start with the '''Likely-Accepts''' and work your way down.
Changed lines 42-43 from:
At the third meeting, reverse the order, and so on.
to:
At the third meeting, reverse the order, and so on.
(Reversals may also occur earlier if needed to establish calibration within the sub-committee.)
Changed line 50 from:
!!The process for each paper:
to:
!!The process for each ''discuss'' paper (papers scoring < 4.3 and > 2.7 and not marked for discussion):
Changed lines 54-70 from:
# AC / 2nd AC present tentative decision
# If unanimous accept
** AC summarizes
paper, strengths and issues.
# If unanimous reject and score < 3.0 little divergence,
** accept decision as
is; discussion not needed (or at discretion of AC)
#
if not unanimous or if score > 3.0
** AC summarizes paper, strengths and issues,

** 2nd AC summarizes his/her perspective.
** The paper may be tabled so 1st and 2nd AC can discuss it.
** Decision presented at next opportunity
.
** If the paper is still unresolved, another
AC should be assigned to it and it is tabled; the discussion continues off-line until consensus is reached or until issues need to be discussed by the entire committee

# Special case: Auto
-accepts
** Contribution summarized by AC
** AC and 2nd AC state whether this should be considered for best
paper. Record for later discussion.
** Acceptance should
be discussed ONLY if there is a truly major issue raised. If the committee wants to reject an auto-accept paper, the paper must be discussed in the next plenary session.
to:
# If paper is likely-accept (>= 4.0)
->AC presents very short description of
paper (2 sentences max)
->AC & 2AC present tentative decision.

-> If both agree to accept, paper is accepted, otherwise discuss like other papers
->AC & 2AC indicate
if paper should be considered for best paper nomination discussion later
->
(all papers > 4.3 are automatically considered in the best paper nomination discussion later).
# If paper is likely-reject (< 3
.0)
->AC presents very short description of paper (2 sentences max)
->
AC & 2AC present tentative decision.
-> If both agree to reject, paper
is rejected, otherwise discuss like other papers
# For full discussion papers
->AC summarizes paper, strengths and issues,
->2nd AC summarizes his/her perspective.
->Other committee members ask questions and raise issues (that AC and 2AC, having read the
paper can presumably best address)
->Paper may
be accepted, rejected, or deferred (likely with additional reader assigned) based on committee consensus (or difficult papers late in the process, a vote of the committee)

Notes about paper discussions:
* Papers scoring in the likely-accept range need a significant justification, likely with an additional reading, to be rejected
. This justification must be throughly discussed in a fashion that ensures it is calibrated with other decisions, and must be clearly conveyed back to the authors in comments added to the reviews.
* In general, higher scoring papers should be deferred rather than rejected until the committee has seen enough papers to be calibrated
Changed lines 85-86 from:
This should begin as papers are accepted in a sub-committee
to:
To help later session creation sub-committees are asked to provide clusters of topically related papers.
Changed line 88 from:
* ACs can group like accepted papers together as they are being decided on.
to:
* ACs can group like accepted papers together as they are being decided on, placing results on flip charts. (SCs should bring these flip charts to the first session formation meetings on Friday afternoon).
Changed lines 90-99 from:
* Papers need not be grouped exactly into sessions: e.g 4 related full papers on a topic instead of 3.

During the session meeting on Friday afternoon
* SCs can then bring these flip charts to the meeting on Friday afternoon
and re-sort as needed (e.g., to balance numbers, themes) with other SC input.
* Along with our ‘schedule’ of panels and other constraints, we will work on session timing i.e., which sessions are best across from each other, author conflicts, etc
.
to:
* Papers need not be grouped exactly into sessions: e.g 2 or 4 related full papers on a topic instead of 3.
* These clusters will only be a partial guide to final session creation -- additional mixing between sub-committees will be encouraged, and many other constraints (such as timing and author conflicts) will also be brought into the process.
Changed lines 69-73 from:
*** mark accept on the spreadsheet and by typing 'A' into the PCS field
*** mark the paper number onto a master paper flipchart page (just in case of electrical failure)
*** tape the summary sheet onto flipchart
*** idle ACs should try to group these into potential sessions (see below).
to:
* mark accept on the spreadsheet and by typing 'A' into the PCS field
* mark the paper number onto a master paper flipchart page (just in case of electrical failure)
* tape the summary sheet onto flipchart
* idle ACs should try to group these into potential sessions (see below).
Changed lines 75-77 from:
*** mark reject on the spreadsheet and by typing 'R' into the PCS field
*** ACs should try to revise meta-review to reflect the decision and the reasons for it, if needed.
to:
* mark reject on the spreadsheet and by typing 'R' into the PCS field
* ACs should try to revise meta-review to reflect the decision and the reasons for it, if needed.
Deleted lines 37-39:
'''Task 3. Keep going down the list.
Discuss each paper and issues (see below).
Changed lines 40-41 from:
Discuss each paper and issues.
to:
At the third meeting, reverse the order, and so on.
Changed lines 20-22 from:
* Copies for all ACS of two spread sheets, one ordered by paper number, the other by score. These will reflect the PCS system score ranking for papers handled by your group as of Monday night.
* Boardroom-style table layout
* Projector
(but no computer: SCs need to bring their own)
to:
* Paper copies for all ACS of two spread sheets, one ordered by paper number, the other by score. These will reflect the PCS system score ranking for papers handled by your group as of Monday night.
* Paper copies for SCs of above, but with AC names included (to be kept confidential by ACs)
* Boardroom-style table layout of room
* Projector and screen
(but no computer: SCs need to bring their own)
Deleted line 15:
Changed lines 18-35 from:
<<<<<<<


=======

SCs should prepare an excel spreadsheet of the papers, ranked by score before the meeting and have it on their computer or dongle.
* Logon to PCS,
* Go to the submissions page.
* Select the menu item in the View pulldown menu (top left) called "PC - <your committee name>".
* Click the Download link (top row) to download this view as a cvs file that can be opened by most spreadsheet packages.
* Use the sort feature of your spreadsheet to sort it by Score, with high score on top.
* Insert a line between the major categories, e.g.
** auto-accept
** auto-reject, etc.
* mark all papers auto-rejected as such.
* Remove unneeded columns - (see the PCS view for the columns you will want to keep).

>>>>>>>
to:
Changed lines 19-23 from:
to:
<<<<<<<


=======
Added line 36:
>>>>>>>
Added lines 19-31:

SCs should prepare an excel spreadsheet of the papers, ranked by score before the meeting and have it on their computer or dongle.
* Logon to PCS,
* Go to the submissions page.
* Select the menu item in the View pulldown menu (top left) called "PC - <your committee name>".
* Click the Download link (top row) to download this view as a cvs file that can be opened by most spreadsheet packages.
* Use the sort feature of your spreadsheet to sort it by Score, with high score on top.
* Insert a line between the major categories, e.g.
** auto-accept
** auto-reject, etc.
* mark all papers auto-rejected as such.
* Remove unneeded columns - (see the PCS view for the columns you will want to keep).
Changed lines 27-28 from:
*
* SC: get master spreadsheet projected
to:
* SC: project master spreadsheet onto computer
Changed lines 19-25 from:
to:
The following will be supplied:
* Copies for all ACS of two spread sheets, one ordered by paper number, the other by score. These will reflect the PCS system score ranking for papers handled by your group as of Monday night.
* Boardroom-style table layout
* Projector (but no computer: SCs need to bring their own)
* Office supplies (flip chart, marking pens, tape, postits)
* Two student volunteers (shared by all)
Changed line 27 from:
* Supplied: copies for all of two spread sheets, one ordered by paper number, the other by score
to:
*
Changed lines 2-3 from:
As seen in the [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/PCMeetingSchedule|agenda]], we will have several short plenary meetings where all sub-committees are present. These plenary meetings will work more or less as follows.
to:
As seen in the [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/PCMeetingSchedule|agenda]], we will have several short plenary meetings where all sub-committees are present in a single room. These plenary meetings will work more or less as follows.
Changed lines 18-19 from:
These meetings are chaired by the sub-committee chair.
to:
These meetings are chaired by the sub-committee chair in a breakout room.
Changed lines 13-15 from:
!!!Closing plenary: A final report of what has been accepted and rejected.
to:
!!!Closing plenary:
A final report will be provided on what has been accepted and rejected.
Changed lines 11-12 from:
* Depending on time, each group will summarize the papers being considered for best paper.
to:
* Depending on the time available, each group will summarize the papers being considered for best paper.
Changed lines 5-6 from:
Saul and Scott will give an overview of the process, give some summary statistics, and describe what each AC and 2nd AC should be doing. Any outstanding issues will be presented.
to:
Saul and Scott will give an overview of the process described here, give some summary statistics, and describe what each AC and 2nd AC should be doing. Any outstanding issues will be presented.
Changed lines 2-3 from:
As seen in the agenda, we will have several short plenary meetings where all sub-committees are present. These plenary meetings will work more or less as follows.
to:
As seen in the [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/PCMeetingSchedule|agenda]], we will have several short plenary meetings where all sub-committees are present. These plenary meetings will work more or less as follows.
Changed lines 44-48 from:
* SC announces title, authors and institutions
* SC projects submission onto the display (optional)
* People in conflict MUST leave the room (authors, collaborators, institution)
* AC / 2nd AC present tentative decision
* If unanimous accept
to:
# SC announces title, authors and institutions
# SC projects submission onto the display (optional)
# People in conflict MUST leave the room (authors, collaborators, institution)
# AC / 2nd AC present tentative decision
# If unanimous accept
Changed line 50 from:
* If unanimous reject and score < 3.0 little divergence,
to:
# If unanimous reject and score < 3.0 little divergence,
Changed line 52 from:
* if not unanimous or if score > 3.0
to:
# if not unanimous or if score > 3.0
Changed line 59 from:
* Special case: Auto-accepts
to:
# Special case: Auto-accepts
Changed line 34 from:
!!Subsequent sub-committee meetings
to:
!!!Subsequent sub-committee meetings
Added lines 38-42:
!!Expectations
Highly ranked papers will likely be accepted.
Low-ranked papers will likely be rejected.
Everything in-between will require discussion.
Deleted lines 0-3:
!Discussion of the PC Process
\\
Deleted line 12:
Deleted line 13:
Deleted line 14:
Added line 19:
Changed lines 29-39 from:
'''Task 1. Update spreadsheet.'''
* ACs supply any changes to scores
* ACs describe what auto-reject papers need to be discussed.

'''Task 2. Auto Accepts'''
* Start at Auto accept papers
* Contribution summarized by AC
* AC
and 2nd AC state whether this should be considered for best paper. Record for later discussion.
* Acceptance should be discussed ONLY if there is a truly major issue raised.
to:
* Update the master spreadsheet:
** ACs supply any changes to scores
** ACs describe what auto-reject papers need to be discussed.

At this point, you will start discussing each paper. Start with the '''Auto Accepts''' and work your way down.
Added lines 58-62:
* Special case: Auto-accepts
** Contribution summarized by AC
** AC and 2nd AC state whether this should be considered for best paper. Record for later discussion.
** Acceptance should be discussed ONLY if there is a truly major issue raised. If the committee wants to reject an auto-accept paper, the paper must be discussed in the next plenary session.
Changed lines 4-6 from:
!! The Overall Process

!
!!Plenary Sessions
to:
!!Plenary Sessions
Changed lines 8-38 from:
# '''First plenary session''': Saul and Scott will give an overview of the process, give some summary statistics, and describe what each AC and 2nd AC should be doing. Any outstanding issues will be presented.

# '''Other plenary sessions''':
**
Saul and Scott will provide higher-level comments on any issues or process adjustments they have seen.

** Sub-committee chairs (that's you) will report on progress accepting/rejecting papers, and describe any issues that have arisen.

** If needed, particular papers with very specific concerns requiring global concensus can be discussed at these meetings (although those in conflict will have to leave).

** Depending on time, each group will summarize the papers being considered for best paper.

# '''Closing plenary''': A final report of what has been accepted and rejected.

!!!Sub-committee meetings
# '''First sub-committee meeting
** Supplied: copies for all of two spread sheets, one ordered by paper number, the other
by score
** SC: get master spreadsheet projected
** All: Set up PCs, log onto PCS
** Quick round-table introductions by SC/ACs (name, where from, specialization)
**
SC introduces overall task (discuss spread sheet, indicate auto-accept/reject categories, etc.) and how ACs will be using it

** Task 1. Update spreadsheet.
*** ACs supply any changes to
scores
*** ACs describe what auto-reject papers need to be discussed.

** Task 2. Auto Accepts
** Start at Auto accept papers
*** Contribution summarized by AC
*** AC and 2nd AC state whether this should be considered for best paper. Record for later discussion.
*** Acceptance should be discussed ONLY if there is a truly major issue raised.
to:
!!!First plenary session
Saul and Scott will give an overview of the process, give some summary statistics, and describe what each AC and 2nd AC should be doing. Any outstanding issues will be presented.

!!!Other plenary sessions
* Saul and Scott will provide higher-level comments on any issues or process adjustments they have seen.

* Sub-committee chairs (that's you) will report on progress accepting/rejecting papers, and describe any issues that have arisen.

* If needed, particular papers with very specific concerns requiring global concensus can be discussed at these meetings (although those in conflict will have to leave).

* Depending on time, each group will summarize the papers being considered for best paper.

!!!Closing plenary: A final report of what has been accepted and rejected.

!!Sub-committee meetings
These meetings are chaired by the sub-committee chair.

!!!First sub-committee meeting
* Supplied: copies for all of two spread sheets, one ordered
by paper number, the other by score
* SC: get master spreadsheet projected
* All: Set up PCs, log onto PCS
* Quick round-table introductions by
SC/ACs (name, where from, specialization)
* SC introduces overall task (discuss spread sheet, indicate auto-accept/reject categories, etc.) and how ACs will be using it

'''Task 1. Update spreadsheet.'''
* ACs supply any changes to
scores
* ACs describe what auto-reject papers need to be discussed.

'''Task 2. Auto Accepts'''
* Start at Auto accept papers
* Contribution summarized by AC
* AC and 2nd AC state whether this should be considered for best paper. Record for later discussion.
* Acceptance should be discussed ONLY if there is a truly major issue raised.

'''Task 3. Keep going down the list.
Discuss each paper and issues (see below).

!!Subsequent sub-committee meetings
Reverse the order, i.e., for the 2nd meeting, start at the bottom of the list.
Discuss each paper and issues
.
Changed lines 22-34 from:
!!!Sub-committee meeting
# '''First sub-committee meeting.
, and longer sub-committee groupsThe finalize lists (auto-accepts / auto rejects that the AC wants to discuss)
** Start at top scoring papers.
** Auto accept papers
*** contribution summarized by AC
*** judge for best paper
** See process above for other papers.
* As papers are accepted, they should be pasted on the walls
*
ACs who are idle should try to group these into sessions.


!!For
each paper:
to:
!!!Sub-committee meetings
# '''First sub-committee meeting
** Supplied: copies for all of two spread sheets, one ordered by paper number, the other by score
** SC: get master spreadsheet projected
** All: Set up PCs, log onto PCS
** Quick round-table introductions by SC/ACs (name, where from, specialization)
** SC introduces overall task (discuss spread sheet, indicate auto-accept/reject categories, etc.) and how
ACs will be using it

** Task 1. Update spreadsheet.
*** ACs supply any changes to scores
*** ACs describe what auto-reject papers need to be discussed.

** Task 2. Auto Accepts
** Start at Auto accept papers
*** Contribution summarized by AC
*** AC and 2nd AC state whether this should be considered for best paper. Record for later discussion.
*** Acceptance should be discussed ONLY if there is a truly major issue raised.

!!The process for
each paper:
Changed line 43 from:
* People in conflict leave the room
to:
* People in conflict MUST leave the room (authors, collaborators, institution)
Changed lines 45-47 from:
* If unanimous accept, AC summarizes paper, strengths and issues.
* If unanimous reject and score < 3.0 little divergence, accept decision as is
* if not unanimous, AC summarizes paper, strengths and issues, 2nd AC summarizes his/her perspective.
to:
* If unanimous accept
** AC summarizes paper, strengths and issues.
* If unanimous reject and score < 3.0 little divergence,
**
accept decision as is; discussion not needed (or at discretion of AC)
* if not unanimous or if score > 3.0
** AC summarizes paper, strengths and issues,
**
2nd AC summarizes his/her perspective.
Changed lines 54-58 from:
** If still unresolved, paper is tabled again, another AC should be assigned to it and the discussion continues off-line until consensus is reached or until issues need to be discussed.

!!Marching Orders
# Go over sheet originally given out
to:
** If the paper is still unresolved, another AC should be assigned to it and it is tabled; the discussion continues off-line until consensus is reached or until issues need to be discussed by the entire committee

!!! When a paper is accepted
*** mark accept on the spreadsheet and by typing 'A' into the PCS field
*** mark the paper number onto a master paper flipchart page (just in case of electrical failure)
*** tape the summary sheet onto flipchart
*** idle ACs should try to group these into potential sessions (see below).

!!! When a paper is rejected
*** mark reject on the spreadsheet and by typing 'R' into the PCS field
*** ACs should try to revise meta-review to reflect the decision and the reasons for it, if needed.
Changed lines 67-75 from:
* This should begin as papers are accepted in a sub-committee, i.e., we post paper #s and titles on stickies, and people can group like ones together as they are being decided on. ACs and SCs can fine-tune these as the meeting progresses. Maybe SVs can print off title/abstract/authors as the day goes on, so we see a full description.
* SCs can then bring these to the meeting on Friday afternoon and re-sort as needed (e.g., to balance numbers, themes) with other SCs input.
* If we have a ‘schedule’ of panels and other constraints, we can also figure out timing i.e.
, which sessions are best across from each other, author conflicts, etc.
* Emphasize that all papers need not be grouped into sessions (it is ok to have a separate list of oddball papers)
. Also SC’s should have the understanding that these are just tentative groupings that are subject to refinement as the scheduling process moves forward. For example, if there are 4 related papers on a topic, they should be grouped together rather than having a session of 3 and one paper in the oddball pile.

Saul
to:
This should begin as papers are accepted in a sub-committee

During the decision meeting:
* ACs can group like accepted papers together as they are being decided on.
* ACs and SCs can fine-tune these as the meeting progresses.
* Papers need not be grouped exactly into sessions: e.g 4 related full papers
on a topic instead of 3.

During the session meeting on Friday afternoon
* SCs can then bring these flip charts to the meeting on Friday afternoon and re-sort as needed (e
.g., to balance numbers, themes) with other SC input.
* Along with our ‘schedule’ of panels and other constraints
, we will work on session timing i.e., which sessions are best across from each other, author conflicts, etc.

Changed lines 12-15 from:
** Sub-committee chairs (that's you) will report on where you are so far in terms of accepting/rejecting papers, and will describe any issues that have arisen.
to:
** Saul and Scott will provide higher-level comments on any issues or process adjustments they have seen.

** Sub-committee chairs (that's you) will report on progress accepting/rejecting papers, and describe any issues that have arisen.
Changed line 17 from:
** Saul and Scott will also provide higher-level comments on any issues or process adjustments they have seen.
to:
Added line 19:
Changed lines 2-3 from:
to:
\\
Changed lines 8-10 from:
# '''First plenary session''': Saul and Scott will give an overview of the process, give some summary statistics, and describe what each AC and 2nd AC should be doing. Issues will be presented.
to:
# '''First plenary session''': Saul and Scott will give an overview of the process, give some summary statistics, and describe what each AC and 2nd AC should be doing. Any outstanding issues will be presented.
Changed line 13 from:
** If needed, particular papers with very specific concerns can be discussed at these meeting (although those in conflict will have to leave).
to:
** If needed, particular papers with very specific concerns requiring global concensus can be discussed at these meetings (although those in conflict will have to leave).
Changed lines 3-4 from:
!!Overall
** finalize lists (auto-accepts / auto rejects that the AC wants to discuss)
to:
!! The Overall Process

!!!Plenary Sessions
As seen in the agenda, we will have several short plenary meetings where all sub-committees are present. These plenary meetings will work more or less as follows.
# '''First plenary session''': Saul and Scott will give an overview of the process, give some summary statistics, and describe what each AC and 2nd AC should be doing. Issues will be presented.
# '''Other plenary sessions''':
** Sub-committee chairs (that's you) will report on where you are so far in terms of accepting/rejecting papers, and will describe any issues that have arisen.
** If needed, particular papers with very specific concerns can be discussed at these meeting (although those in conflict will have to leave).
** Saul and Scott will also provide higher-level comments on any issues or process adjustments they have seen.
** Depending on time, each group will summarize the papers being considered for best paper.
# '''Closing plenary''': A final report of what has been accepted and rejected.

!!!Sub-committee meeting
# '''First sub-committee meeting.
, and longer sub-committee groupsThe
finalize lists (auto-accepts / auto rejects that the AC wants to discuss)
Added lines 3-13:
!!Overall
** finalize lists (auto-accepts / auto rejects that the AC wants to discuss)
** Start at top scoring papers.
** Auto accept papers
*** contribution summarized by AC
*** judge for best paper
** See process above for other papers.
* As papers are accepted, they should be pasted on the walls
* ACs who are idle should try to group these into sessions.
Changed lines 24-38 from:
** If still unresolved, paper is tabled again, another AC should be assigned to it and the discussion continues off-line until consensus is reached or until issues need to be discussed.
to:
** If still unresolved, paper is tabled again, another AC should be assigned to it and the discussion continues off-line until consensus is reached or until issues need to be discussed.

!!Marching Orders
# Go over sheet originally given out

!!Session Creation
* This should begin as papers are accepted in a sub-committee, i.e., we post paper #s and titles on stickies, and people can group like ones together as they are being decided on. ACs and SCs can fine-tune these as the meeting progresses. Maybe SVs can print off title/abstract/authors as the day goes on, so we see a full description.
* SCs can then bring these to the meeting on Friday afternoon and re-sort as needed (e.g., to balance numbers, themes) with other SCs input.
* If we have a ‘schedule’ of panels and other constraints, we can also figure out timing i.e., which sessions are best across from each other, author conflicts, etc.
* Emphasize that all papers need not be grouped into sessions (it is ok to have a separate list of oddball papers). Also SC’s should have the understanding that these are just tentative groupings that are subject to refinement as the scheduling process moves forward. For example, if there are 4 related papers on a topic, they should be grouped together rather than having a session of 3 and one paper in the oddball pile.

Saul

Added lines 1-13:
!Discussion of the PC Process

!!For each paper:
* SC announces title, authors and institutions
* SC projects submission onto the display (optional)
* People in conflict leave the room
* AC / 2nd AC present tentative decision
* If unanimous accept, AC summarizes paper, strengths and issues.
* If unanimous reject and score < 3.0 little divergence, accept decision as is
* if not unanimous, AC summarizes paper, strengths and issues, 2nd AC summarizes his/her perspective.
** The paper may be tabled so 1st and 2nd AC can discuss it.
** Decision presented at next opportunity.
** If still unresolved, paper is tabled again, another AC should be assigned to it and the discussion continues off-line until consensus is reached or until issues need to be discussed.