This document imagines a distributed review and curation ecosystem where the primary activity is the awarding of badges to papers. Badges in this ecosystem are a simple dual assertion: that a paper was reviewed, and that the review conforms to a specified review policy. A badge’s meaning can be almost anything and is up to the badge provider to define. They can be used to do everything from simply tag a paper as part of a collection (ie curation), subjectively judge a paper positively or negatively (ie peer review), certify it as meeting some kind of objective criteria (ie statistical review), and beyond.
Diagram
The goal of the ecosystem should be to build at least one complete stack (on the left) for every paper — even if the ultimate badge is just that the paper doesn’t need review, or that it has been added to a collection by the badge provider.
Features
Posting of review policies (aka criteria) by journals and third-party reviewers
Assertion and depositing of reviews of a given paper (or papers) by multiple parties
Verification of reviews by multiple verifiers
Depositing of badges — a signed assertion that a review conforms to a review policy — to platforms that verify and distribute the badge.
Requesting badges based on history of reviews from multiple parties
Curating papers by posting badges with self-contained reviews
Cross-platform discoverability of reviews given to papers
Cross-platform discoverability of badges given to papers
Use Cases
Traditional Publisher
In this case, a traditional publisher provides every part of the stack, including a review policy that is referenced within the review object itself. The benefit is that providing this is simple, and requires very little work on the part of the publisher — just a few additions to article metadata. However, it might be seen as less trustworthy by the community than a stack with multiple parties involved.
Audited Review
In this case, a traditional publisher provides a badge for one of their papers, references a standard review type, and submits the badge to a review auditing service that verifies the badge conforms to their standards. Essentially, an auditor is asserting that a review was done to their standards. This provides a layer of verification that may add to community trust compared to a fully self-provided stack.
Third Party Review Service
In this case, a third party — could be a review service or a society — provides a badge for a paper published on a pre-print server based on a review they conducted and submits it to a platform. This could be part of their editorial operation, or in response to a request from an author. In this example, the review policy is the standard one published by the third party that they use for reviews of this type, but it could be self-contained or they could reference a standard type published by a review auditor.
Iterative Review
In this case, a third party — could be a review service or a society — provides a badge for a paper published on a blog based on multiple reviews published across the web by different parties and submits it to a platform. Essentially, they’re saying that the combination of all the independent reviews meets their requirements for a badge.
Editorial Curation
In this case, a third party — it could be a society or publisher — provides a badge for a paper on a preprint server that names it as part of a collection and includes a self-asserted review policy and, if they’re providing editorial commentary, a link to a review on their site. Essentially, they’re saying that they have chosen to include this paper in a collection with other papers that have been given the same badge.
Review Discoverability
Anyone can ask any platform or auditor if they have badges for a given article (initially by DOI). This allows publishers to display third-party badges on their sites, for statistical analysis of badges, etc.
Actors
Author
Someone who writes a paper.
Paper Hoster
Someone who hosts a paper. This can be:
Journals/publishers
Pre-print servers
Platforms
Individuals (ie bloggers)
Reviewer
Someone who provides a review of a paper or multiple papers. This can be:
Individual, independent reviewers
Journals/Publishers
Third-party review services (ie societies)
Bots providing automated reviews (ie statistical)
Review Policy Maintainer
Someone who maintains a review policy that, if met, merits a badge. This can be:
Journals/publishers
Third-party review services (ie society)
Review auditors (ie PRT)
Badge Provider
Someone who provides a badge by asserting that a review (or multiple reviews) of a paper meets the requirements of a review policy. This can be:
Journals/publishers
Third-party review services (ie societies)
Review auditors (ie PRT)
Badge Verifier
Someone who verifies that a badge is valid and provides a list of badges others can query. This can be:
Review auditors (ie PRT)
Platforms (ie HHMI/F1000)
Journals/publishers
Self-verified (ie for curation, where verification isn’t needed)
Objects
Paper
A scholarly work in any format, published on the web.
Paper Hosters
Preprint servers
Journals/Publishers
Platforms
Individuals (ie bloggers)
Review Policy
A human- and machine-readable description of a review process published by a policy maintainer. Essentially, a specification for a type of review. Scales from simply “a review happened” to any number of arbitrary requirements (ie non-blind with at least 1 statistical reviewer).
Policy Maintainers
Review auditors (ie PRT)
Journals/Publishers
Third-party review services (ie societies)
Format: JSON/XML/YAML document compiled to HTML
Locations
Published to review audit sites (PRT)
Published to journal sites
Published to publisher sites
Deposited to repository (ie Crossref)
Directly asserted inside reviews
Directly asserted inside badges
Linked to inside reviews
Linked to in JATS/paper metadata
Linked to from article page on publisher site in HTML metadata
Attributes
Name
Maintainer
List of review requirements (ie blindness, number of reviewers, etc.)
List of possible outcomes
Review
A human- and machine-readable assertion that a review took place published by a reviewer. Scales from simply “a single review happened” to a full version history of a multi-step review process with reviewer contacts, comments, scores, etc.
Reviewers
Journals
Publishers
Third-party reviewer services (ie societies)
Individual reviewers
Bots providing automated reviews (ie statistical)
Format: JSON/XML/YAML document compiled to HTML
Locations
Linked to in JATS/paper metadata
Linked to from article page on publisher site in HTML metadata
Deposited to repository (ie Crossref)
Deposited to pre-print servers
Published to third-party review sites
Published to review & curation platforms (ie HHMI platform, F1000, etc.)
Directly asserted inside badges
Attributes
* = required
Paper (or papers) reviewed (link, doi, etc.)
Asserter (can be anonymous/pseudonymous, can be a single reviewer, or an organization)
Description
Asserted outcome
Review policy (link or self-contained)
List of review events (ie reviewers, blindness, comments left, etc.)
To be further specified and somewhat standardized — but it should be designed to accommodate something as simple as “3 anonymous reviewers showed up” or as complex as showing every change to a document during a review.
A simplified version could be posted publicly, while a more complex version could be kept behind a paywall, functioning as subscription content.
Badge
A digitally signed assertion provided by a badge provider and verified by a bade verifier that a review or collection of reviews conforms to a review map.
Badge Providers
Journals/publishers
Third-party review services (ie society)
Review auditors (ie PRT)
Badge Verifiers
Platforms
Review auditors (ie PRT)
Journals/publishers
Format: JSON/XML/YAML document
Locations
Journal & publisher sites, linked from paper metadata
Published to third-party review sites
Published to review & curation platforms (ie HHMI platform, F1000, etc.)
A site where badges are audited, verified, and curated. Badges contain links to papers and reviews, so hosting those objects locally are not necessary, but platforms can choose to host papers and reviews.
Auditor
Any service that verifies and makes badges discoverable, including platforms.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
