CHI 2009 Report

Inconsistencies

The note guidelines (http://www.chi2009.org/Authors/CallForPapers/PapersVsNotes.html) state that "CHI Notes are not expected to include a discussion of related work that is as broad and complete as that of a submission to the Papers venue."

Yet the review template asks the following questions regarding contribution for NOTES as well as papers.

     * Does the paper provide context, where it clearly reviews what is already known and what limitations exist in knowledge about this artifact or technique?
     * Does the paper motivate a real problem that is worth solving?
     * Does the paper describe the artifact or technique in sufficient detail for others to replicate it?
     * Does the paper include a rigorous and convincing validation of the artifact/technique, where it clearly shows that incremental gains not only exist, but that the gains are of practical significance?

According to the guidelines, three of these (1, 3, and 4) are less important for notes (full review, detail, and rigor). There's a risk these statements will steer note reviewers in the wrong direction.

It may be too late to take action, but it'd be nice if the wording under "Contribution" in the review template could be updated to read something like the following:

      1. Does the paper motivate a real problem that is worth solving?
      2. Does the paper provide context, where it clearly reviews what is already known and what limitations exist in knowledge about this artifact or technique?
      3. Does the paper describe the artifact or technique in sufficient detail for others to replicate it?
      4. Does the paper include a rigorous and convincing validation of the artifact/technique, where it clearly shows that incremental gains not only exist, but that the gains are of practical significance

PCS Issues

Separation of papers chairs, sub-committee chairs, ACs, and reviewers

Currently, PCS only recognizes the roles of chairs, ACs, and reviewers. It has no notion of a sub-committee chair. This lead to several issues, with workarounds indicated in the sub-points

  • SCs could see all submissions, not just the ones handled by their own committee.
    • workarounds as indicated below
  • SCs needed to create their own tailored views that just showed the submissions handled by their own committee
    • We gave instructions to SCs on how to create their own views, and we tried to supply a view that they could use to them
  • SCs had to look at all papers and declare conflicts, rather than just the ones handeld by their own committee
    • We gave SCs a fairly complicated set of instructions of how to search for conflicts and how to deal with them. This is by far the biggest issue.

Recommendation. PCS should be expanded to recognize the SC role. In particular, SC roles should be similar to chairs except that they can only see papers and reviews directly submitted to their committee.

Reviewer Database

Its out of date, not used very well, and we have to constantly tell people that the numbers mean nothing. It is also inconsistent with ACs, who are already volunteering to handle papers.

Recommendation. Not sure, but the whole idea of the reviewer database needs to be rethought.

Notes vs papers

Notes and Papers are currently not distinguished by PCS, although people can tell the different between them by their length. Still this is an inconvenience.

  • Knowing papers vs notes helps the SC and AC assign a more balanced load to ACs and Reviewers
  • Sorting papers vs notes helps SCs and ACs deal with them somewhat separately when assigning them

Recommendation. PCS should be modified so that authors can specify if a submission is a note or a paper. This information should appear as part of the description of each submission, and views could be based around sorting / filtering notes vs. papers.

Reviewers being registered several times under different names/emails in PCS

Sometimes people are in PCS under several identities. This is likely because, over the years, they had forgotten that they had registered previously, and then register again. They may have variations in their name, and / or different emails. This leads to problems. Typically, an AC may request a reviewer (under one name/email), and then the reviewer goes onto PCS using another name/email account and sees no reviews assigned to them. In other cases, we had different ACs assign papers to one person using different accounts.

  • Recommendation 1. If PCS can be modified to check to see if a person has multiple accounts, then that person should be 'warned' about this and given the option and instructions on how to merge their accounts.
  • Recommendation 2. Currently, when this problem arises (usually flagged by an email message), one of the papers chairs goes into PCS, searches for multiple accounts for that person, merges them (using the PCS merge opton) under what they believe is the last best email address, and informs the person. This doesn't take too long, and seemed to work (although it does takes time).

Selecting ACs / Reserve ACs

This proved a surprising headache. Issues we came across are listed below.

  • Some SCs waited quite a while before finalizing their ACs, and especially their reserve ACs. The worse case was one day before the submission deadling. Yet this information is critical to submittiers. As well, this demanded constant update to the web page describing ACs, as they tended to come in a piecemeal fashion.
    • Recommendation: A firm deadline should be placed on this.
  • Some ACs volunteered for more than one committee. This led to obvious conflicts, as an AC could not serve on two committees at the same time.
    • Recommendation: better instructions. An online registration system for the AC role may help, but this could be too hard to implement.

Problems in submissions

After submissions were closed, we typically fixed small problems ourselves. For example, we would download a PDF, edit it using Adobe Writer to cut out author names (in case of anonymity) and then upload it back. This seemed more effective and faster than having the author do it (it usually took about 2-5 minutes per paper).

Anonymity

As usual, some people forgot to anonymize thier paper at the most basic level, i.e. ,names and affiliation listed on the 1st page of the paper, or identifying information in the Acknowledgements.

Recommendation.

  • Include a checkbox on the submission form that specifically says:

"Have you anonymized your paper? (see xx/xx for details if you are not sure)?

  • include advice that while a space for acknowledgements should be included, text should not be supplied.

Wrong format

Some people used the CHI format for posters (i.e., the landscape mode).

Recommendation: we need to check the instructions page in both places to indicate what those formats are for. In general we passed on those papers to review as the authors actually short-changed themselves on space, so its not really cheating. We suspect that reviewers will be much harsher on these papers.

Over length papers

Some authors were overlenght, such as:

  • small format cheats: reducing spacing, eliminating title block / copyright block, etc.
    • Recommendation: In general we let this pass as long as it wasn't too excessive.
  • a line or two over on page 11.
    • Recommendation: In general, we let this pass as its not a big deal in the grand scheme of things, and that it could easily be a result of problems in print drivers (i.e., its 10 pages in Word, but 10+ a few lines when its seen in PDF)
  • a page over. We saw this when an author puts references on an 11th page (did they really think this didn't count?), and when an author put an appendix in (one of several extra pages!).
    • Recommendation: Clarify in page length that 10 pages are inclusive of everything. We passed these onto referees anyways, but are still not sure if that was the right thing to do.
  • a page over, but its blank or it has a note to ACs/Referees (e.g., about prior submission history, or relations to other papers).
    • Recommendation. This is fine. The blank page is usually a PDF generation artifact. The note should be fine as its not part of the publication, and there is no other place to put it (assuming the note is worth putting in).

Papers with missing / extra bits

Some papers (a few) had problems in terms of missing substance or adding substance. The question is, do we tell authors about these cases and give them opportunity to fix it? Two examples:

  • A 3.5 page paper refers to a figure that is not included in it. Yet there is space for that figure...
  • A paper includes in the 'extra material' box (where one normally posts a video figure) a table. It is not referred to in the text.
    • Recommendation. We did not inform authors or give them opportunity to change it, as this opens the door to us somehow judging 'drafts' vs. 'final' submissions. We took the stance that the author is responsible for their paper, and understands the 'as is' way they are refereed. We told ACs who raised this issue that reviewers could flag problems, and then make their own recommendation of the paper. However, if the problem was clearly a technical one (e.g., a PDF problem), then we would likely make an exception to this.