Meta Review Scores

Dear all ACs;

(SCs... you should watch out for this if you see it in your own review of all scores handled by your committee)

To add to your flood of email during this busy week while you are gathering the final reviews and composing your meta-review, here are some more hints that will help make this a consistent process.

As you are constructing your meta-review and providing it with an AC score, here are a few things to watch out for (based on some things I currently see across all papers).

First, your meta-review score should reflect where the paper fits in the 1 – 5 range. It should NOT be a binary choice, i.e., don’t do:

  • 1 if your advice is to reject the paper
  • 5 if your advice is to accept the paper

The reason you should choose an appropriate value in the range is:

  • It should reflect as much as possible the scores of referees
  • Authors are really insulted if reviewer scores are (say) a 3.5 average but they get a 1 from the AC; they also think that the AC killed the paper regardless of reviewer comments
  • It strongly changes how the overall average score is seen by the committee on the discussion list, as it pulls the ‘averaged’ score either too high or too low.

Second, your score should, as much as reasonably possible, reflect the comments of referees as well as your own. This doesn’t mean it should be the average, but it does mean that you have to temper your personal score by considering referee scores as well. Here are some samples of where things are likely unbalanced:

  • If your score is outside of the range of referee scores, you are probably pushing your own opinion too much. For example, if a paper is rated 3.5, 4.0, 4.5 and you give it a 3, this seems quite odd. Reviewers can rightly say “it looks like the AC killed my paper without considering referee comments. Similarly, if you gave it a 5, this looks like you didn’t take referee concerns to heart (although the authors likely wouldn’t complain).
  • Similarly, if your score is quite different from the average score but within range (perhaps due to disagreement by authors), take a careful look at whether your score really does reflect the diversity of opinion vs. your sole opinion.

The reasons for being careful are similar to those stated in the first point.

Remember that you can always raise a paper for discussion regardless of the score. However, the score as it appears – especially the AC score – will influence how the 2nd AC and the committee view the paper. If you really do want to put your personal score in it, put it in text, and flag it, i.e., “I scored to the papers 3.5 to reflect the diversity of reviews. However, my personal scoring would have been less – perhaps a 2.5 – due to my personal concerns as detailed below”.

Saul Greenberg and Scott Hudson,
Papers co-chairs