Cart

A grant application, submitted at the National Institute of Health (NIH) and other funding agencies, is reviewed by a panel of internal and external reviewers. These reviewers are seasoned scientists and experts in biomedical scientific disciples who judge the application on five critical parameters, namely;

  • Significance
  • Investigator(s)
  • Innovation
  • Approach
  • Environment

Based on these criteria, an application is rated on a scoring scale. The grant scoring system is a type of quality assurance that separates and filters high-quality applications from the low-quality ones. The applications obtaining good scores make it to the Second Level of the Review which denotes increased likelihood of grant funding.

Application scoring is an extremely important step in the sifting of grant proposals because the future of biomedical science pretty much relies on it. Not all applications are reliable, significant, meticulous and science-friendly and can/should not be proceeded with. The experts follow a strict set of rules while scoring the applications so that only the highest-quality applications carrying substantial research data, scientific impact, and potential to benefit the medical science are funded. (Eblen, Patel, Pearson, RoyChowdhury, and Wagner, 2016)

What is the NIH Scoring System?

The NIH scoring system is a 9-point rating scale where 1 = exceptional and 9 = poor. The score, which is always a whole number and never a decimal point, determines the Overall Impact and Criterion scores of the application. Most of the applications are scored anywhere between 3-7 with 5 being the average on the scale. A score of 5 is considered a medium impact score.

Reviewers weigh the application on the criteria mentioned above. However, it is important to understand the meaning and significance of each criterion in order to gauge the process.

  • Review Criteria

1.      Significance – The reviewers assess whether the proposed research addresses an important health challenge? Will it benefit the scientific knowledge and widen the biomedical research?

2.      Investigator(s) – Are the principal investigator (PI) and the collaborators suitable, qualified and experienced to conduct the research? What are their credentials and accomplishments? Do they require training?

3.      Innovation – Does the research have the capacity to shift the dynamics of existing clinical practice? How innovative are the concepts, approach, methodology, and instruments?

4.      Approach – Is the proposal backed by a substantial rationale? Is the research strategy capable of accomplishing the specific aims?

5.      Environment – Will the environment where research will be conducted, i.e., institution, clinic, laboratory, be conducive to scientific research?

These criteria determine the strength and weakness of the application. All categories are scored individually. At this point, all applicants are informed in writing of their scores. Then each reviewer gives a preliminary overall impact score for each application. This impact score determines the eligibility of the application for the review meeting held under the supervision of the Scientific Review Officer (SRO).

Final Overall Impact Score – At the scientific review group (SRG) meeting, each member will again give the final overall impact score after a careful assessment and determination of the application merit. Scores from all members will be added, and a mean will be taken. The mean is multiplied by 10. This is called the Final Impact Score. The final score is written on the face page of the application summary statement and reported to the applicant.

It is important that the applicant focuses on all these categories to increase the odds of success. However, no application can have exceptional scores in all categories. (Bhattacharya, et al., 2017). An application may have a great approach but lack experienced investigators; the other may have institutional support but lack innovation. Therefore aiming a score anywhere between 2-5 should be goals. (Kopstein, Janice, & Martin, 2010)

The 9-point rating scale below explains the scores:

ScoreDescriptionWhat it Means
1ExceptionalAbsolutely brilliant proposal. Essentially error-free
2OutstandingExtremely strong research proposal with very few, if any, weaknesses
3ExcellentVery strong save a few minor shortcomings
4Very GoodStrong but has more than one minor weakness
5GoodStrong but has at least one moderate weakness/shortcoming
6SatisfactoryModerately strong but has more than one moderate shortcoming
7FairPartially strong but has more than one major shortcoming
8MarginalMarginally strong but has numerous major shortcomings
9PoorPoor with numerous major shortcomings
  • Minor Shortcoming: A minor error – such as grammatical, formatting or punctuation mistake – that does not affect the Overall Impact Score and can be easily rectified.
  • Moderate Shortcoming: A mistake that is rectifiable but can hamper the Impact.
  • Major Shortcoming: A shortcoming that can seriously affect Impact and reduces chances of success.
  • Preliminary Scoring

Before the SRG meeting, every assigned reviewer is tasked to analyze, weigh and assess the application and give a preliminary score for each of the 5 review criteria. This is a tentative score that reflects the reviewer’s judgment as well as the scientific appropriateness of the proposal. The reviewer also assigns a preliminary Overall Impact Score for the application which is discussed in the combined meeting of all reviewers.

The preliminary score is enlisted in the Internet Assisted Review (IAR) website and written on the summary statement. The entered score is also printed and delivered on the meeting day for the grand discussion. The meeting commences giving each reviewer a chance to critique after which s/he submits the scores. Once submitted, the preliminary scores can be reversed or edited. The preliminary scores are then followed by a final score from each reviewer. Final scoring is made in private.

  • Criterion Scoring

For most of the applications, 5 review criteria are used, but some funding announcements may include more criteria. Criterion scores determine the strength or weakness of the application and are separate from the Overall Impact score. Submitting scores alone is discouraged; the scores are almost always accompanied by a written critique. Like Preliminary Scores, the criterion scores are also entered into the IAR site and later discussed in the meeting. If during the meeting, the discussant’s view about the criterion scores changes, s/he can EDIT the criterion score to match the written critique.

The criterion scores along with the critique are added to the summary statement of each application.

  • Impact Score

Impact scores refer to the numerical scores awarded by each eligible reviewer before and during the meeting. Impact score is the amalgam of criterion score and additional criteria. The additional criteria revolve around the safety, care, and protection of the human and animal subjects, biohazards and other specifics of the funding.

The impact score is based on a 1-9 scale where each number reflects a specific descriptor as discussed below.

ScoreImpactDescription
9

8

7

Low ImpactFair

Marginal

Poor

6

5

4

Moderate ImpactSatisfactory

Good

Very Good

3

2

1

High ImpactExcellent

Outstanding

Exceptional

The reviewers guide the score to full range of the scale to discriminate applications from each other. They enjoy the freedom to award the score of their choosing, i.e., they score what they deem best represents the value of the application. They are not bound or pressured to keep the score to the upper or lower limit (Brauer et al., 2017). Scores from all reviewers are discussed at the meeting before the final Overall Impact Score is calculated. The average of the sum is taken. The average falls anywhere between 10 – 90.

  • Non-Numeric Score

    • Applications that do not qualify for a discussion or review at the peer-review meeting are termed “Not Discussed.” These are considered less competitive and are neither given criterion nor impact score.
    • An application lacking significance or failing to meet the ethical standards will be termed Non-Recommended for Further Consideration (NRFC). The NRFC applications are neither scored nor make it to the Second Level of Review. They are not funded; however, they are discussed in the First Level of Review and are declared NRFC after the committee recommendation.
    • An application lacking substantial information or in any way involved in research misconduct is labeled Deferred. Deferred applications are not scored.
    • An application may be declared ‘Conflict’ if after reviewer scoring it has been found to have a conflict of interest.

References

    1. Eblen, M.K., Patel, K.C., Pearson, K. RoyChowdhury, D. & Wagner, R.M. (2016, June 1) How Criterion Scores Predict the Overall Impact Score and Funding Outcomes for National Institutes of Health Peer-Reviewed Applications. PLoS One, 11(6), http://dx.doi.org/10.1371/journal.pone.0155060
    2. Bhattacharya, A., Filut, A., Kaatz, A., Lee, Y., Magua, W., Potvien, A.,…Carnes, M. (2017, August 1). Analysis of NIH R01 Application Critiques, Impact and Criteria Scores: Does the Sex of the Principal Investigator Make a Difference? Acad Med, 91(8), 1080–1088. http://dx.doi.org/10.1097/ACM.0000000000001272
    3. Kopstein, A., Janice, J.M., & Martin, M.R. (2010, November 17). An Analysis of Preliminary and Post-Discussion Priority Scores for Grant Applications Peer Reviewed by the Center for Scientific Review at the NIH. PLoS One, 5(11), e13526. http://dx.doi.org/10.1371/journal.pone.0013526
    4. Brauer, M., Carnes, M., Ford, C.E., Kaatz, A., Nathan, M.J., Pier, E.L.,… Raclaw, J.R. (2017, February 14). ‘Your comments are meaner than your score’: score calibration talk influences intra- and inter-panel variability during scientific grant peer review. Res Eval, 26(1), 1–14. http://dx.doi.org/10.1093/reseval/rvw025

Author Shuhan He

More posts by Shuhan He