The SciScore core report is an automated assessment of a research paper’s methodologies and reporting that combines criteria from a variety of NIH-supported principles and guidelines, such as ARRIVE, CONSORT, and MDAR. It includes three tables and a reporting score. The report primarily covers rigor adherence and key resource identification to help promote reproducibility in life science research.
The reporting score - a score out of 10 - is a number researchers, journal editors, and funders can use to help them decide how rigorous and transparent a research manuscript is. The score is based on both rigor adherence and key resource identification within the materials and methods sections. A good score is essential to help ensure that interested parties have enough information to accurately judge the reproducibility of a research article. If your SciScore is not where you want it to be, don’t worry; improving your score can be as simple as adding an identifier. In general though, the more information, the better.
In the rigor table (Table 1), SciScore highlights sentences that include various elements of rigor as described by Hackam and Redelmeier in 2006, and by van der Warp and colleagues in 2010. SciScore was trained using sentences from thousands of published papers that were tagged by expert curators to indicate that the sentence described a rigor criterion such as blinding (either during the experiment or during data analysis). SciScore uses conditional logic when scoring your paper. This means that if, for instance, a cell line is detected, then SciScore ‘expects’ to find cell line authentication criteria within the paper. If a cell line is not detected, these criteria (cell line authentication and contamination) will not be included in scoring. In cases where a criterion is not detected by SciScore but is expected, the statement “not detected” is displayed. In cases where a criterion is neither detected nor expected, "not required" is shown. It is possible that a criterion is not necessary for a particular manuscript or that SciScore, an automated tool, makes a mistake. If SciScore makes substantial mistakes with your manuscript, please contact us to help us learn from our mistakes.
Please note there are currently some bugs impacting this table and our scoring. We intended to have all criteria in Table 1 impact scoring (either conditionally or otherwise), however, protocol identifiers, data information, and code information have not been implemented into our scoring algorithm as of yet. We are diligently working on this and apologize for any inconvenience this may cause.
The rigor items detected in this version of SciScore include the following:
Scoring for Rigor Table (5 total points):
The rigor table makes up 5 points of the total score. Those five points are split evenly among the expected rigor criteria. Scores are rounded to the nearest whole number. For each sentence that describes an expected rigor criterion, e.g. blinding, SciScore adds the fractional number of points for that criterion, and if it is unable to find a statement on blinding then this section is labeled "Not Detected" and receives a score of 0. To improve detection, please make sure that your language is clear and written in standard English.
Conditional criteria such as cell line authenticity are only included in the expected list if cell lines are detected in the Key Resources table (Table 2). Likewise, an IACUC statement is expected if an appropriate animal is detected in the Key Resources table. Currently, the field sample permit will be detected but never expected.
When organisms or human participants are detected, it is expected that blinding, group selection criteria such as randomization and inclusion/exclusion, attrition, and demographic information such as sex or gender will be present. Biological variables such as sex should inform subject and group selection.
SciScore attempts to classify papers based on the paper type to reduce the burden of requiring all criteria where it may be irrelevant, however, we tend to err on the side of caution, expecting criteria where SciScore is unsure. We do this because SciScore is primarily a tool that assists peer review by bringing attention to something that may have been omitted. Protocol, code, and data identifiers refer to persistent identifiers (either a DOI such as DOI:10.17504/protocols.io.9gbh3sn, a URL such as https://github.com/tophat, or an accession number in a repository such as GSE145917). SciScore will then try to authenticate these. For accession numbers, SciScore will check for the identifiers’ existence in their source database. For DOIs and URLs, SciScore will check to see if these resolve. Identifiers that are validated will be displayed in blue, while dead links will be shown in red such as DOI:10.17504/protocols.io.9gbh3snr. This is intended to quickly alert the author or reviewer to potential problems with a website or a typo in an accession number. SciScore does not check the relevance of the cited identifiers, only their existence. In rare cases, a typo may still result in a valid identifier. Consequently, we wish to remind users that SciScore is not a substitute for expert review. Rather, SciScore should be used in concert with reviewers for the best results.
How to get a better score on this section:
The key resources table (Table 2), contains:
RRIDs are unique identifiers for reagents and other resources that largely overlap with the resource types that have been labeled as particularly problematic by the National Institutes of Health in recent changes to their grant review criteria, please see "key biological resources", e.g., antibodies, cell lines and transgenic organisms. The RRID initiative is led by community repositories that provide persistent, unique identifiers to their resources, such as transgenic mice, salamanders, antibodies, cell lines, plasmids and software projects such as statistical software. RRIDs are described on the rrids.org website and in a primer by Bandrowski and Martone in 2016. RRIDs are unique numbers that resolve to a particular database record, for example, the RRID:CVCL_0063 resolves to this record for a cell line (Cellosaurus community repository).
How does it work: The information in the Cellosaurus database (https://web.expasy.org/cellosaurus/) is structured and curated by Cellosaurus staff, the authority for cell lines (all RRIDs have an authority specific for the resource type). If authors use this RRID, then SciScore will ask the database about that particular identifier. In cases where a RRID fails to resolve (i.e. database has no record of that identifier, most likely due to a typo), SciScore will display an “unresolved” error message in red. If an RRID was recently submitted to the authority by authors, it often takes a week or more to become available in the database, thus exercising caution in the interpretation of the SciScore report in cases of newly minted RRIDs is advisable.
Sentences that ‘should have RRIDs’ are detected by SciScore using patterns in sentences that are similar to how each resource is commonly described in published papers. A sentence that describes one or more antibodies may be detected by SciScore and this will be placed into the table without a corresponding RRID. SciScore will then attempt to find the name, catalog number, and vendor of the resource. In cases where the tool is relatively confident, it will suggest an RRID (this will contain the word “suggestion” and be in gray), as a courtesy. A link is provided, so authors can quickly verify whether the correct RRID was suggested.
Note of caution: Please verify all RRID suggestions, only the author can know whether suggestions are correct.
The Key Resources types detected in this version of SciScore include the following:
Scoring for Resources Table (5 total points):
The total for the entire Key Resources table is 5 points with scores rounded to the nearest whole number. Each resource that is detected in this section is included in the score. For each valid RRID detected with matching metadata (e.g. catalog number or name), full points are awarded. Because a single resource can often be described in a variety of ways, SciScore utilizes fuzzy matching to correctly link resources with their corresponding RRIDs. In cases where multiple resources and RRIDs are listed out in a single sentence, authors should verify that the resources and RRIDs are correctly matched as SciScore is not perfect. Partial points are awarded if SciScore detects resources where a suggestion can be made, or if an RRID does not resolve properly. Therefore, the way to maximize the points from this section is to add RRIDs and proper citations that include vendor names, catalog numbers, lot and version numbers into the methods section of the manuscript for every key resource used.
How to get a better score on this section:
The other entities table (Table 3) contains:
Sentences containing entities of interest are shown in the leftmost column, while the specific statistical tests and oligonucleotides detected are displayed in the right column. Again, none of the criteria in the “other entities table” impact the overall score.
Incorrect sentences: SciScore is a machine learning, text analysis tool, and it is therefore susceptible to making two types of errors: false positives and false negatives.
False negatives: The most common error occurs when the algorithm fails to detect a sentence that contains a rigor criterion or a resource, such as an antibody. False negatives generally occur either because the sentence is complex or in a less common syntax pattern. Generally, simple sentences in clear standard English are simpler to process and result in fewer false negatives. If a truly complex sentence structure is required to describe reagents, a table may help not only SciScore, but also human readers. If an RRID is detected in a sentence, SciScore will be triggered to take a look at the sentence, which may have been skipped otherwise.
False positives: This type of error occurs when SciScore falsely detects something including cases where a sentence does not contain an antibody, but the algorithm asserts that this sentence does have an antibody. If many resources are used and all have RRIDs, a single false positive will not reduce the score substantially, if at all. But if only 1-2 resources are used or if the false positive is in the cell line or organism category, it will trigger scoring for cell line authentication and other rigor criteria, which can reduce your SciScore needlessly. False positives are most often seen in the tools portion of table 2, as the algorithm detects company names, where it should not. We try to minimize these false positives using several strategies, however, they still occur in roughly 3-5% of cases. If this impacts your score, please contact our team (http://sciscore.com) and include the sentence where SciScore made the error. While we can't fix the score, SciScore can certainly learn from its mistakes for improved performance next time around.
As mentioned before, SciScore is not perfect. Below we have provided a list of problems we are working to fix. If you notice any other persistent problems, please contact us.