CHI 2009 Criteria Proposed

Back to CHI 2009

See http://www.chi2009.org/Authors/CallForPapers/SelectingAType.html for how it now looks in the advanced call.

Quick list (this is on the web page now)


<Introductory text - see web page> In all cases, a CHI Paper or Note must break new ground and make an original research contribution. Further details of each contribution type will be provided in the final call for papers.

  1. Development or Refinement of an Interface Artifact and Techniques concerns the on-going development of new interface artifacts or the refinement of knowledge of interface artifacts and techniques already well-known to the HCI community.
  2. Understanding Users contributes an analysis of people, their behaviors, their situations and context, their perceptual, cognitive, and behavioral capabilities, and/or their interaction with their artifacts.
  3. Systems, Tools, Architectures and Infrastructure submissions contribute the underlying software and technology associated with a novel interactive application, user interface feature, user interface design, or development tool.
  4. Methodology contributes a novel method or method variation that helps us design an HCI artifact or evaluate an HCI artifact or setting. That is, it improves how we do what we do.
  5. Theory submissions contribute principles, concepts, or models on which work in HCI might be based.
  6. Innovation, Creativity and Vision contributes a significant innovation, vision or design concept in how a user interface might appear, behave or be used, or a new invention that significantly enables new styles of user interfaces.
  7. Experience submissions contribute discussions of how HCI methods, theory, or tools were applied to the design or development of an HCI artifact.
  8. Opinions contribute provocative essays. They present the author's well-supported opinion about a topic of significant interest to a relatively broad segment of the CHI community.

Suggested detailed list

In all cases, a CHI Paper or Note must break new ground and make an original research contribution. Further details of each contribution type will be provided in the final call for papers.

  1. Development or Refinement of an Interface Artifact and Techniques concerns the on-going development of new interface artifacts or the refinement of knowledge of interface artifacts and techniques already well-known to the HCI community. Your paper will likely include answers to the following questions:
    • Do you provide context, where you clearly review what is already known and what limitations exist in knowledge about this artifact or technique?
    • Do you motivate a real problem that is worth solving? For example, do you justify the beneficial value of this artifact or technique not only in isolation but within the global context of its expected uses?
    • Do you describe the artifact or technique in sufficient detail for others to replicate it?
    • Do you include a rigourous and convincing validation of the artifact/technique (e.g., empirical study, usability study, field study as appropriate), where you clearly show that incremental gains not only exist, but that the gains are of practical significance?
  2. Understanding Users contributes an analysis of people, their behaviors, their situations and context, their perceptual, cognitive, and behavioral capabilities, and/or their interaction with their artifacts. Such an understanding forms the background of many HCI and software engineering disciplines, where it serves to inspire and ground new design directions or where it forms the core of a requirements analysis. Your paper will likely include answers to the following questions:
    • Do you clearly describe the methodology and mechanics behind the analysis (e.g., ethnographic, field investigation, interviews, contextual inquiry).
    • Do you clearly describe what new understandings were gained?
    • Do you show how the value of this new understanding is relevant to HCI, i.e., where it increases our understanding of human behavior as related to HCI, or how it adds insight into the organization of social settings, or by showing implications for design, or how it causes us to reconsider prior perspectives of human behavior and contexts within HCI, or by helping us analyze other relevant contexts?
  3. Systems, Tools, Architectures and Infrastructure submissions contribute the underlying software and technology associated with a novel interactive application, user interface feature, user interface design, or development tool. Your paper will likely include answers to the following questions:
    • Do you show that the idea is original, of interest, and valuable to a reasonably broad subset of the CHI community?
    • Is the idea placed in context, where you detail the expected situation, tasks and users?
    • Do you validate the idea, where validation can be by sound argumentation, reflection of experiences, user studies, or other evidence proving its value?
    • Are you extremely clear about what has and has not been implemented?
    • Do you provide sufficient information for an experienced researcher to implement a similar system?
    • Can the system work within its proposed settings, or are there significant obstacles to its use?
    • Do you validate the idea? For example, do you demonstrate that it makes new or more functional interfaces practical or the creation of existing interface techniques dramatically easier? Or do you demonstrate performance gains that clearly impact the user experience? Or do you reflect on experiences using the contribution to create interfaces or provide other evidence proving its value?
  4. Methodology contributes a novel method or method variation that helps us design an HCI artifact or evaluate an HCI artifact or setting. That is, it improves how we do what we do. The method may be intended for use in research or development settings (or both), but the paper should be clear about the intended audience. Your paper will likely include answers to the following questions:
    • Is the methodology novel?
    • Do you show how the methodology builds on or contrasts with related work?
    • Does the methodology have relevance and importance to HCI practice and/or research?
    • Is the methodology described in sufficient detail so that others can use it?
    • Has the value of the methodology been demonstrated, e.g., by showing how it helps product design, development, and use, or by demonstrating its use in a particular setting, or by contrasting the results obtained with a competing methodology, or by showing how its use improved the process it was intended to address?
  5. Theory submissions contribute principles, concepts, or models on which work in HCI might be based. Your paper will likely include answers to the following questions:
    • Is the theory novel or a significant variation of an existing theory?
    • Is the theory understandable by a general HCI practitioner as well as an expert?
    • Do you show how the theory is situated with respect to other theories, and how it builds upon them?
    • Does the theory have relevance and importance to HCI practice and/or research?
    • Has the value of the theory been demonstrated, e.g., by showing how it helps describe, predict and/or influence the design of an interactive application or setting?
    • Does the theory recast existing work in other areas in a way that provides greater value to an HCI practitioner / researcher?
  6. Innovation, Creativity and Vision contributes a significant innovation, vision or design concept in how a user interface might appear, behave or be used, or a new invention that significantly enables new styles of user interfaces. While thought pieces are possible, proof of concept implementations and demonstrations will be favoured. Your paper will likely include answers to the following questions:
    • Do you clearly detail the idea?
    • Is the idea novel or a significant variation of an existing idea?
    • Do you convincingly describe the potential significance and impact of this work, where you clearly argue about its significance, how it enables a new class of user interfaces or user experiences, and how it can affect future generations of users?
    • Do you suggest how the innovation, vision or design might be adopted in practice, e.g., by reflection, by probes, or by case studies?
    • Do you stimulate thought on new ways to interact with technology, or how new cultural meanings will be applied to the technology we build?
  7. Experience submissions contribute discussions of how HCI methods, theory, or tools were applied to the design or development of an HCI artifact. They are judged on the value of the experience to practitioners as well as to researchers. Your paper will likely include answers to the following questions:
    • Is the experience and its context clearly described?
    • Is it significant? That is, will an experienced person doing HCI work in a development organization learn something new from this paper that they can apply to their practice?
    • If you are describing something that has been done before, do you show that it has significant added value, e.g., a combination of methods that is worth more than the individual ones, or ironing out some practical problems in an academic method?
    • If you are describing something new or a variation of an existing technique, do you show that it adds significant value, preferably by real and convincing uses of it?
  8. Opinions contribute provocative essays. They present the author's well-supported opinion about a topic of significant interest to a relatively broad segment of the CHI community. They have well supported arguments, including consideration of other perspectives, and/or data from research or practice, if applicable. They are expected to have a stimulating effect on the CHI community. Your paper will likely include answers to the following questions:
    • Do you clearly show that this is a topic of highly significant interest to a relatively broad segment of the CHI audience?
    • Is it provocative?
    • Will it have high impact to HCI? That is, will it stimulate significant discussion and perhaps changes in our practices?
    • Is the opinion well founded? That is, is it well argued? Is it well supported by facts? Does it consider the range of current research and practice experience of our community?
    • Does it incorporate other opinions and consider them in a balanced fashion?

Old FULL Descriptions.

Opinions

Description

Opinion Papers are provocative essays. They present the author's well-supported opinion about a topic of significant interest to a relatively broad segment of the CHI community. They have well supported arguments (including consideration of other perspectives, and/or data from research or practice, if applicable), and be expected to have a stimulating effect on the CHI community. Review criteria include the expected impact, the quality of the argumentation, and the experience (research or practice) used to support the opinion. Such papers are a challenge to write well, but when a good one comes along, it is often the most discussed paper at the conference. Authors of opinion papers are urged to contact one of the Papers Co-Chairs in advance of submitting a paper, to get feedback on their idea, since CHI rarely accepts opinion papers. Particularly high standards will be expected of an Opinion Paper -- it will be harder to get an Opinion Paper accepted than any other type of submission.

Review Criteria

  • Does the paper cover a topic of significant interest to a relatively broad segment of the CHI audience?
  • Is it provocative?
  • Will it have high impact to HCI? That is, will it stimulate significant discussion and perhaps changes in our practices?
  • Is the opinion well founded? That is, is it well argued? Is it well supported by facts? Does it consider the range of current research and practice experience of our community?
  • Does it incorporate other opinions and consider them in a balanced fashion, or is it a rant?

Rationale

This criteria rewritten from the CHI '97 description matches what Dan suggested as a 'Letters to the Community'. My edits just merge the 'short' and 'long' description into a single paragraph, and I added the review criteria. I am not sure if I like the phrase 'and the experience (research or practice) used to support the opinion'.

Refinement of Artifacts and Techniques

Description

Refinement of Artifacts and Techniques Papers concern the on-going refinement of knowledge of interface artifacts and techniques already well-known to the HCI community. A good Incremental Artifacts and Techniques paper will clearly review what is already known, as well as what limitations exist in this knowledge, i.e., is there a problem worth solving? It will present the refinement within this context. It will strongly motivate the beneficial value of this refinement within the 'ecology' of the interface, i.e., where the refinement not only has value in isolation but in its interplay with other expected interface techniques. It will always include a validation of the new artifact or technique. This validation will usually be a rigorous analysis (e.g., empirical study, usability study, field study) that clearly shows that incremental gains not only exist, but that the gains are of practical significance.

Review Criteria

  • Does the paper clearly review what is already known?
  • Does it motivate a real problem that is worth solving?
  • Is the new interface artifact or technique strongly motivated within this context?
  • Does it consider how the refinement would interoperate with other aspects of the interface (i.e., the interface ecology)?
  • Does the paper include a rigourous and convincing validation of the artifact/technique, where is shows that gains are real and of practical significance?

Rationale

The basic idea of the above was capture in the CHI 97 'Empirical Papers' category, which takes about "the collection and interpretation of data concerning the design or use of an HCI artifact." However, that emphasises the method rather than the problem. I have tried to recast this as above, where incremental techniques need to be well substantiated. I am not happy with the words as currently written, but I think this suffices to show the gist of this category. I would probably want to expand it beyond 'technique' and 'artifact' to include user interfaces in general. This category captures (to some extent) Scott's category of Evaluation of Technology, the CHI 2009 Planning Committee category of 'Normative' (refine what we have), and Dan Olson;s Usable Techniques.

Systems, Tools, Architectures and Infrastructure Papers

Description

Systems papers describe the software and technology associated with a novel interactive application, user interface feature, user interface design or development tool. The system idea must be original, and of interest and valuable to a reasonably broad subset of the CHI community. The system idea should be placed in context, by detailing the expected situation, tasks and users. It should be based on sound reasoning. Validation can be by sound argumentation, reflection of experiences, or other evidence proving its value. Systems papers should be extremely clear about what has and has not been implemented. They should also provide enough information that an experienced researcher could implement a similar system. For development tools, it should clearly show how they simplify or improve the process of creating, building or maintaining new user interfaces. Authors should be clear to what extent the system has been implemented. Authors are encouraged to develop a coordinated demonstration or video submission of the system for CHI 97.

Review Criteria (see http://icie.cs.byu.edu/paperPDFs/EvaluatingSystems.pdf )

  • Is the system idea original? Has the problem been previously solved? Is it a generalizable solution?
  • Is it of interest and valuable to the community?
  • Is the system idea situated in its context of use and in the related work?
  • Has the system been implemented, and is enough detail provided so that an expert can reproduce the idea?
  • Can the system work within its proposed settings, or are there significant obstacles to it?
  • If it is a development tool: Does it simply the development process compared to other methods? Does it empower new design participants? Does it simplify interconnection with other existing methods?
  • Can it scale?

Rationale (paraphrase of Dan Olson paper)

Computer Science contributes greatly to UI development, and are forces for change. This category considers contributions from a systems perspective, where the system shows its value as an enabling technology.

Theory Papers

Description

Theory Papers describe principles, concepts, or models on which work in HCI might be based. Because CHI draws from many theoretical traditions, your paper has to make sense both to those who are highly knowledgeable about your traditions and to those who want to begin using your theoretical constructs and predictions in their own work. Authors of theoretical papers are expected to position their ideas within a broad context of HCI frameworks and theories. Review criteria include the originality or soundness of the analysis provided as well as the relevance of the theoretical content to HCI practice and/or research. A strong theory paper will demonstrate the value of the theory by showing how it helps describe, predict and/or influence the design of an interactive application. The paper should be very clear about how the theory builds on or contrasts with related work (particularly work in this area that leads to different conclusions than your work). You will also need to make clear the relevance of your results to research and practice in HCI; don't assume that the connections will be obvious to the reviewers.

Review Criteria

  • Is the theory novel?
  • Is the theory understandable by a general HCI practitioner as well as an expert?
  • Is the theory situated with respect to other theories?
  • Does the theory have relevance and importance to HCI practice and/or research?
  • Has the value of the theory been demonstrated?
  • Does the theory recast work in other areas in a way that provides greater value to an HCI practitioner / researcher?

Rationale

Theory is important to HCI. A good theory grounds our thinking, provides a framework that helps us analyze what we see, and can even predict human behavior when using our interactive systems.

Methodology Papers

Description

Methodology Papers describe a novel method that helps us design an HCI artifact or evaluate an HCI artifact or setting. That it, it improves how we do what we do. The method may be intended for use in research or development settings (or both), but the paper should be clear about the intended audience. Review criteria include the originality or soundness of the analysis provided as well as the relevance of the methodology to HCI practice and/or research. A strong methodology paper will describe the methodology in sufficient detail (or will have pointers to that detail) so that others can replicate and use it. The value of the methodology will be demonstrated by showing how it helps product design, development, and use, e.g., by demonstrating its use in a particular setting, or by contrasting the results obtained with a competing methodology, or by showing how its use improved the process it was intended to address. The paper should be very clear about how the methodology builds on or contrasts with related work. You will also need to make clear the relevance of your methodology to research and practice in HCI; don't assume that the connections will be obvious to the reviewers.

Review Criteria

  • Is the methodology novel?
  • Is the methodology understandable by a general HCI practitioner as well as an expert?
  • Is the methodology situated with respect to other methodology?
  • Does the methodology have relevance and importance to HCI practice and/or research?
  • Has the value of the methodology been demonstrated?
  • Is the methodology described in sufficient detail so that others can use it?

Rationale

Methodologies are critical to much of what we do: how we do our design, how we iterate over designs, how we deploy them, how we test them.

Understanding Users Papers

Description

Understanding Users Papers contribute an analysis of people, their behaviours, their situations and context, and their interaction with their artifacts. Such an understanding forms the background of many HCI and software engineering disciplines, where it serves to inspire new design directions or where it forms the core of a requirements analysis. A paper in this category will clearly describe the methodology and mechanics behind the analysis (e.g., ethnographic, field investigation, interviews, contextual inquiry). It will clearly describe what new understandings were gained. The value of this new understanding must be relevant to HCI. It will show how it contributes to a theory that increases our understanding of human behavior as related to HCI, or how it adds insite into the organization of social settings, or by showing implications to design, or how it causes us to reconsider prior perspectives of human behavior and contexts within HCI, or by helping us analyze other relevant contexts. Because CHI draws from many methodological traditions, your paper has to make sense both to those who are highly knowledgeable about your methodology and to those who want to begin using your methodology in their own work. Authors of methodology papers are expected to position their methodologies within a broad context of HCI methodologies.

Review Criteria

(see http://www.ics.uci.edu/~jpd/classes/readings/Dourish-Implications.pdf )

  • To be added

Rationale

Understanding users is a critical part of what we do in HCI; some would argue it is the most important aspect. Contributions that help us understand users have to be considered worthy in their own right, i.e., independent of the production of a technology.

Breakthrough, Creativity and Vision Papers

Description

Breakthrough, Creativity and Vision Papers contribute a significant innovation, vision or design concept in how a user interface might appear, behave or be used, or a new invention that significantly enables new styles of user interfaces. Papers are judged by the potential significance and impact of the reported work. A good paper will detail the idea, will clearly argue about its significance, and will describe as much as reasonable about how it can affect our future generations of users. While thought pieces are possible, proof of concept implementations and demonstrations will be favoured. The paper will suggest how the innovation, vision or design might be adopted, e.g., by reflection, by probes, or by case studies. A good paper will stimulate thought on new ways to interact with technology, or how new cultural meanings will be applied to the technology we build.

Review Criteria

To be added

Rationale

We want to make sure that breakthrough speculative ideas are allowed..

Experience Papers

Description

Experience papers describe how HCI methods, theory, or tools were applied to the design or development of an HCI artifact. Experience Papers are judged on the value of the experience to practitioners -- will someone doing HCI work in a development organization learn something from this paper that they can apply to their practice? If you are describing something that has been done before, you need to have significant added value -- e.g., a combination of methods that is worth more than the individual ones, or ironing out some practical problems in an academic method. If you are describing something new or a variation of an existing technique, you need to also show that it adds significant value, preferably by real and convincing uses of it. It is important to focus on and draw out the information useful to practitioners, rather than just describing your experience.

Review Criteria

To be added

Rationale

This is really another recast of the Methodology section , but geared to practitioners.