ACM CHI PC Meeting Process Description

Plenary Sessions

As seen in the agenda, we will have several short plenary meetings where all sub-committees are present in a single room. These plenary meetings will work more or less as follows.

First plenary session

Saul and Scott will give an overview of the process described here, give some summary statistics, and describe what each AC and 2nd AC should be doing. Any outstanding issues will be presented.

Other plenary sessions

For each of the remaining potential plenary sessions, a quick meeting of all the sub-committee chairs will be held to uncover issues, address questions, and determine if a full plenary session is needed, or continued work in sub-committees will be most productive.

For plenary sessions:

  • Saul and Scott: provides higher-level comments on any issues or process adjustments they have seen.
  • Sub-committee chairs (that's you): report on progress accepting/rejecting papers, and describe any issues that have arisen.
  • All: If needed, particular papers with very specific concerns requiring global concensus can be discussed at these meetings (although those in conflict will have to leave).
  • Late on Thursday, each group will summarize a subset of best paper nominations, which will be finalized on Friday.

Closing plenary:

  • SCs: final report on what has been accepted, rejected, and nominated for best paper
  • All: discuss suggestions for process improvements.

Sub-committee meetings

These meetings are chaired by the sub-committee chair in a breakout room. The following materials will be supplied.

  • Paper copies for all ACs of two spread sheets, one ordered by paper number, the other by score. These will reflect the PCS system score ranking for papers handled by your group as of Monday night.
  • Boardroom-style table layout of room
  • Projector and screen (but no computer: SCs need to bring their own)
  • Office supplies (flip chart, marking pens, tape, post-its)
  • Two student volunteers (shared across all sub-committees)

First sub-committee meeting

  • SC: project master spreadsheet onto computer
  • All: Set up PCs, log onto PCS
  • Quick round-table introductions by SC/ACs (name, where from, specialization)
  • SC introduces overall task (discuss spread sheet, indicate auto-accept/reject categories, etc.) and how ACs will be using it
  • Update the master spreadsheet:
    • ACs supply any changes to scores
    • ACs verify auto-accept/auto-reject papers
    • ACs list what auto-reject papers need to be discussed (or controversial auto-accepts)
    • ACs provide initial list of best paper nominees (details to be discussed when the paper is discussed)

At this point, you will start discussing each paper starting from the top scoring ones.

  • Auto-accept >= 4.3: each AC briefly summarizes the paper content (no discussion needed)
  • All others: see 'Process for each Discuss paper' below
  • Best paper nominees: Discuss as they come up

Subsequent sub-committee meetings

Reverse the order, i.e., for the 2nd meeting, start at the bottom of the list. At the third meeting, reverse the order, and so on. (Reversals may also occur earlier if needed to establish calibration within the sub-committee.)

How Individual Papers are Handled in the SC Meeting

In general, expect the following:

  • Highly ranked papers will likely be accepted .
  • Low-ranked papers will likely be rejected.
  • The decision on papers with in-between scores will be based on a good discussion between its readers

The Process for each discuss paper

Discuss papers have 4.3 > score > 2.7 OR it was an auto-reject paper marked for discussion or has a high variance > 1.0. Expectations on the amount of discussion varies with the actual score.

  • SC announces title, authors, institutions, Note vs. Paper type, and Contribution Type.
  • SC projects submission onto the display (optional)
  • People in conflict MUST leave the room (see conflicts of interest below)
  • AC quickly names reviewers and indicates why they were selected (optionally omitted to save time)
  • If paper is likely-accept (>= 4.0)
AC presents very short description of paper (2 sentences max)
AC & 2AC present tentative decision.
If both agree to accept, paper is accepted, otherwise discuss like other papers
AC & 2AC indicate if paper should be considered for best paper nomination; brief discussion if it is nominated
  • If paper is likely-reject (< 3.0)
AC presents very short description of paper (2 sentences max)
AC & 2AC present tentative decision.
If both agree to reject, paper is rejected, otherwise discuss like other papers
  • All others (full discussion)
AC summarizes paper, strengths and issues,
2nd AC summarizes his/her perspective.
Other committee members ask questions and raise issues (that AC and 2AC, having read the paper can presumably best address)
Paper may be accepted, rejected, or deferred (likely with additional reader assigned) based on committee consensus (or difficult papers late in the process, a vote of the committee)

Notes about paper discussions:

  • Papers scoring in the likely-accept range need a significant justification, likely with an additional reading, to be rejected. This justification must be throughly discussed in a fashion that ensures it is calibrated with other decisions, and must be clearly conveyed back to the authors in comments added to the reviews.
  • In general, higher scoring papers should be deferred rather than rejected until the committee has seen enough papers to be calibrated

When a paper is accepted

  • mark accept on the spreadsheet and by typing 'A' into the PCS field
  • mark the paper number onto a master paper flipchart page (just in case of electrical failure)
  • tape the summary sheet onto flipchart
  • idle ACs should try to group these into potential sessions (see below).

When a paper is rejected

  • mark reject on the spreadsheet and by typing 'R' into the PCS field
  • ACs should try to revise meta-review to reflect the decision and the reasons for it, if needed.

Conflicts of interests

You need to leave the room if you have a conflict, as defined below.

  • Same institution as a co-author. This includes separate labs of same company (e.g., IBM TJ Watson and IBM Almaden), but not different campuses in a university system (e.g., not UC Berkeley and UC San Diego).
  • Current or recent (last year, optionally longer) co-author or close collaborator with a co-author.
  • Recently (last year) left a conflicting institution, or currently being considered for employment there.
  • Strong financial ties that might be seen by others as a CoI (i.e., major direct funding from an institution).
  • Advisor or advisee of co-author (optionally expires after 4 years)
  • Any conflict (possibly unstated) which you feel might affect your ability to remain unbiased, or which you feel someone else might reasonably see as affecting your objectivity.

Session Creation

To help later session creation sub-committees are asked to provide clusters of topically related papers. During the decision meeting:

  • ACs can group like accepted papers together as they are being decided on, placing results on flip charts. (SCs should bring these flip charts to the first session formation meetings on Friday afternoon).
  • ACs and SCs can fine-tune these as the meeting progresses.
  • Papers need not be grouped exactly into sessions: e.g 2 or 4 related full papers on a topic instead of 3.
  • These clusters will only be a partial guide to final session creation -- additional mixing between sub-committees will be encouraged, and many other constraints (such as timing and author conflicts) will also be brought into the process.

If your SC or AC duties are done early

Some sub-committee groups are handling smaller sets of papers than others, and thus may be done earlier. If this happens, you may be invited to serve as a reader for papers being discussed by other groups OR sit in on the deliberation.