How open should reviews be in CompSci?

A short collection of different review styles and their pros and cons
published:
by Harshvardhan J. Pandit
academia publications reaction

This post is a reaction for ISWC2019's decision to use double-blind for their research track

This post will be updated as I'm corrected, criticised, made aware of more content. You can help.

blind process

double-blind

pros

  • Since no identifies are involved, there are less chances of a bias due to known authors or institutions (in theory).
  • Reviewers can review work by people they have previously worked with (provided they cannot discern identity from the work itself).
  • Since reviewers identities are not known, authors cannot hold grudges or bias (or favouritism) against specific individuals based on their reviewing. e.g. Gender-bias or Racial-bias
  • If rebuttal is used, the authors can be more critical of the review without worrying about personal repurcussions

cons

  • Is it possible to truly hide my identity if my work is in a sufficiently niche area? e.g. I work on GDPR and Semantic Web
  • Is it possible to effectively hide that I have authored the past work re-used in the paper? e.g. If I use A, B, and C (references) all authored by one person, will it not be evident that I must be one of those authors?
  • For resources such as source code, I have to create a non-identifying copy for the paper. e.g. I usually put my name and email in files I author (in the header section), or put the URL for the repository such as Github, or put names in comments regarding more information or Issues or TODOs. If I supply this code with the paper, I have to find and scrub such references.
  • For information hosted on institution infrastructure, I have to find alternative means of hosting. e.g. If I have documentation and manuals associated with my work on my research group's website, I have to find and segregate relevant sections, and host them elsewhere where they cannot be used to identify me.
  • For demos and executables hosted on institution infrastructure, I have to find alternative means of hosting. e.g. If a particular service runs on university VMs, which is part of a demo presented in the paper, then I have arrange alternative non-identifiable hosting solutions (such as AWS) to host the demo. Additionally, I have to scrub any identifying information from the demos such as names, contact, university, funding.
  • Iterational updates are difficult to anonymise e.g. presenting v2 of some work is unlikely to be authored by someone not involved in the previous work

single-blind

pros

  • Authors don't know who the reviewers are, so cannot hold bias or grudge against them
  • Anonymisation of author identity is not required, which requires less changes or work for the authors to prepare their work for presentations. e.g. software code can have identifying information
  • Reviewers can be critical of the work without worrying about personal repurcusions. e.g. Grudge against a reviewer because they did not accept the paper
  • Reviewers can look up previous work by the author which can assist in evaluating current work

cons

  • Reviewers know who the authors are, so can be biased or influenced. e.g. one of the authors is a well-known or popular persona in the field, and so adds an 'invisible pressure' to automatically accept their work. Can also be specified as "identity validates work".

open (no-blind)

pros

  • Reviews are transparent which can encourage more quality in terms of the review itself e.g. being specific and not vague in referencing issues, or being more civil in language

cons

  • Reviewers may not feel comfortable being more critical of the work because their identities will be known
  • Reviewers may not feel comfortable being more critical of authors well-known or popular in their fields
  • Authors may hold a grudge or bias against reviewers e.g. based on gender or race

other concerns

published reviews

pros

  • Reviews can be assessed by all parties, including non-authors, which motivates more quality in reviews.
  • Corrections can be found and made in the future.

cons

  • Authors and Reviewers may not feel comfortable sharing more critical reviews.

licensed reviews

pros

  • Others can reuse or adapt existing reviews in other reviews e.g. re-using someone critical argument's of an approach in an review

cons

  • Authors and reviewers may not be aware of the best license suited for reviews e.g. CC-by-ND may prevent some re-use of reviews

forum-style reviews

One or more reviewers discuss in a forum-style about the work, and collaboratively or individually submit reviews.

pros

  • Discussion between reviewers can lead to better reviews in terms of clarifications and justifications

cons

  • Reviewers may not be comfortable discussing their opinions with others, especially if identities are known

linked reviews

Reviews are associated with specific versions of a manuscript, and corrections to the manuscript address which issues in reviews have been address i.e. reviews evolve as the manuscript evolves.

pros

  • Reviews are always relevant to the specific version of the manuscript
  • Progress of work can be assessed more systematically

cons

  • Results in more work for the authors and reviewers, or editors in terms of maintaining information