CHI2009Report

Inconsistencies

The note guidelines (http://www.chi2009.org/Authors/CallForPapers/PapersVsNotes.html) state that "CHI Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue."

Yet the review template asks the following questions regarding contribution for NOTES as well as papers.

     * Does the paper provide context, where it clearly reviews what is already known and what limitations exist in knowledge about this artifact or technique?
     * Does the paper motivate a real problem that is worth solving?
     * Does the paper describe the artifact or technique in sufficient detail for others to replicate it?
     * Does the paper include a rigorous and convincing validation of the artifact/technique, where it clearly shows that incremental gains not only exist, but that the gains are of practical significance?

According to the guidelines, three of these (1, 3, and 4) are less important for notes (full review, detail, and rigor). There's a risk these statements will steer note reviewers in the wrong direction.

It may be too late to take action, but it'd be nice if the wording under "Contribution" in the review template could be updated to read something like the following:

      1. Does the paper motivate a real problem that is worth solving?
      2. Does the paper provide context, where it clearly reviews what is already known and what limitations exist in knowledge about this artifact or technique?
      3. Does the paper describe the artifact or technique in sufficient detail for others to replicate it?
      4. Does the paper include a rigorous and convincing validation of the artifact/technique, where it clearly shows that incremental gains not only exist, but that the gains are of practical significance

PCS Issues

Separation of papers chairs, sub-committee chairs, ACs, and reviewers

Currently, PCS only recognizes the roles of chairs, ACs, and reviewers. It has no notion of a sub-committee chair. This lead to several issues, with workarounds indicated in the sub-points

Recommendation. PCS should be expanded to recognize the SC role. In particular, SC roles should be similar to chairs except that they can only see papers and reviews directly submitted to their committee.

Reviewer Database

Its out of date, not used very well, and we have to constantly tell people that the numbers mean nothing. It is also inconsistent with ACs, who are already volunteering to handle papers.

Recommendation. Not sure, but the whole idea of the reviewer database needs to be rethought.

Notes vs papers

Notes and Papers are currently not distinguished by PCS, although people can tell the different between them by their length. Still this is an inconvenience.

Recommendation. PCS should be modified so that authors can specify if a submission is a note or a paper. This information should appear as part of the description of each submission, and views could be based around sorting / filtering notes vs. papers.

Reviewers being registered several times under different names/emails in PCS

Sometimes people are in PCS under several identities. This is likely because, over the years, they had forgotten that they had registered previously, and then register again. They may have variations in their name, and / or different emails. This leads to problems. Typically, an AC may request a reviewer (under one name/email), and then the reviewer goes onto PCS using another name/email account and sees no reviews assigned to them. In other cases, we had different ACs assign papers to one person using different accounts.

Selecting ACs / Reserve ACs

This proved a surprising headache. Issues we came across are listed below.

Problems in submissions

After submissions were closed, we typically fixed small problems ourselves. For example, we would download a PDF, edit it using Adobe Writer to cut out author names (in case of anonymity) and then upload it back. This seemed more effective and faster than having the author do it (it usually took about 2-5 minutes per paper).

Anonymity

As usual, some people forgot to anonymize thier paper at the most basic level, i.e. ,names and affiliation listed on the 1st page of the paper, or identifying information in the Acknowledgements.

Recommendation.

"Have you anonymized your paper? (see xx/xx for details if you are not sure)?

Wrong format

Some people used the CHI format for posters (i.e., the landscape mode).

Recommendation: we need to check the instructions page in both places to indicate what those formats are for. In general we passed on those papers to review as the authors actually short-changed themselves on space, so its not really cheating. We suspect that reviewers will be much harsher on these papers.

Over length papers

Some authors were overlenght, such as:

Papers with missing / extra bits

Some papers (a few) had problems in terms of missing substance or adding substance. The question is, do we tell authors about these cases and give them opportunity to fix it? Two examples: