How open should reviews be in CompSci?
> main site > research > blog
description: A short collection of different review styles and their pros and cons
tags: academia; publications; reaction;
This post is a reaction for ISWC2019's decision to use double-blind for their research track
This post will be updated as I'm corrected, criticised, made aware of more content. You can help.
- Since no identifies are involved, there are less chances of a bias due to known authors or institutions (in theory).
- Reviewers can review work by people they have previously worked with (provided they cannot discern identity from the work itself).
- Since reviewers identities are not known, authors cannot hold grudges or bias (or favouritism) against specific individuals based on their reviewing. e.g. Gender-bias or Racial-bias
- If rebuttal is used, the authors can be more critical of the review without worrying about personal repurcussions
- Is it possible to truly hide my identity if my work is in a sufficiently niche area? e.g. I work on GDPR and Semantic Web
- Is it possible to effectively hide that I have authored the past work re-used in the paper? e.g. If I use A, B, and C (references) all authored by one person, will it not be evident that I must be one of those authors?
- For resources such as source code, I have to create a non-identifying copy for the paper. e.g. I usually put my name and email in files I author (in the header section), or put the URL for the repository such as Github, or put names in comments regarding more information or Issues or TODOs. If I supply this code with the paper, I have to find and scrub such references.
- For information hosted on institution infrastructure, I have to find alternative means of hosting. e.g. If I have documentation and manuals associated with my work on my research group's website, I have to find and segregate relevant sections, and host them elsewhere where they cannot be used to identify me.
- For demos and executables hosted on institution infrastructure, I have to find alternative means of hosting. e.g. If a particular service runs on university VMs, which is part of a demo presented in the paper, then I have arrange alternative non-identifiable hosting solutions (such as AWS) to host the demo. Additionally, I have to scrub any identifying information from the demos such as names, contact, university, funding.
- Iterational updates are difficult to anonymise e.g. presenting v2 of some work is unlikely to be authored by someone not involved in the previous work
- Authors don't know who the reviewers are, so cannot hold bias or grudge against them
- Anonymisation of author identity is not required, which requires less changes or work for the authors to prepare their work for presentations. e.g. software code can have identifying information
- Reviewers can be critical of the work without worrying about personal repurcusions. e.g. Grudge against a reviewer because they did not accept the paper
- Reviewers can look up previous work by the author which can assist in evaluating current work
- Reviewers know who the authors are, so can be biased or influenced. e.g. one of the authors is a well-known or popular persona in the field, and so adds an 'invisible pressure' to automatically accept their work. Can also be specified as "identity validates work".
- Reviews are transparent which can encourage more quality in terms of the review itself e.g. being specific and not vague in referencing issues, or being more civil in language
- Reviewers may not feel comfortable being more critical of the work because their identities will be known
- Reviewers may not feel comfortable being more critical of authors well-known or popular in their fields
- Authors may hold a grudge or bias against reviewers e.g. based on gender or race
- Reviews can be assessed by all parties, including non-authors, which motivates more quality in reviews.
- Corrections can be found and made in the future.
- Authors and Reviewers may not feel comfortable sharing more critical reviews.
- Others can reuse or adapt existing reviews in other reviews e.g. re-using someone critical argument's of an approach in an review
- Authors and reviewers may not be aware of the best license suited for reviews e.g. CC-by-ND may prevent some re-use of reviews
One or more reviewers discuss in a forum-style about the work, and collaboratively or individually submit reviews.
- Discussion between reviewers can lead to better reviews in terms of clarifications and justifications
- Reviewers may not be comfortable discussing their opinions with others, especially if identities are known
Reviews are associated with specific versions of a manuscript, and corrections to the manuscript address which issues in reviews have been address i.e. reviews evolve as the manuscript evolves.
- Reviews are always relevant to the specific version of the manuscript
- Progress of work can be assessed more systematically
- Results in more work for the authors and reviewers, or editors in terms of maintaining information