Cart

What is Peer-Review?

Peer-review is the critical evaluation of a grant application. It is the cornerstone in academic research; it ensures scientific credibility and ethical standards of research proposals. It ensures funded studies and research projects are in strict accordance with the line of the funder’s mission. It is applied at the beginning and the later stages of the project life-cycle to maintain uninterrupted quality output. It is one of the most efficient and effective ways to produce and promote high-quality research across all realms of biomedical sciences. The research emanating after the scrutiny of a peer-review is published in academic and scientific journals and plays an important role in navigating the science, practice and the policy.

Every year, the funders, i.e., the research councils and academic bodies invest in reviewers to scrutinize grant applications. The U.S. National Institute of Health (NIH), for instance, recruits about 2500 reviewers to evaluate grant applications. Per estimates, the NIH receives over 40,000 manuscripts each year. The NIH spends $14 million to bring the reviewers together thrice a year. The budget is allocated in their traveling, staying and honorarium. (Aaron et al., 1998)

Peer-review is the lifeline of scientific research. Without it, medical funding cannot be allocated. In 2009, peer-review accounted for more than 95% of research funding in the UK. (Ismail, Farrands, and Wooding, 2009).

How is Peer-Review Done?

The submitted grant application goes through a round of expert evaluation by internal and external reviewers, scored, and discussed by the review committee. The review committee –consisting of the funding agency’s members and internal reviewers – decides the fate of the application. The grant score, reviewers’ ratings, and committee discussions play a fundamental role in the acceptance or rejection of the application.

The review has two levels:

  • First Level of Review
  • Second Level of Review

Each level demands a detailed explanation.

1.     First Level of Review

The first level of review is conducted by a group called Scientific Review Group (SRG). It consists of veteran scientists and experts in the relevant scientific field. The meetings are held and administered in the NIH Institutes and Centers (ICs) or Center for Scientific Review (CSR).

The composition of SRG is as follows:

  • Scientific Review Officer: heads the SRG. S/he is a federal official working in the capacity of the NIH extramural staff scientist. The job of the SRO is to:
    • Ensure that all regulations, rules, and policies of the funding agency are followed throughout the review
    • Analyze each submitted grant application, its contents, documents, and requirements.
    • Provide fair treatment, attention, and value to every grant application in the first round of review.
    • Recruit the reviewers on the basis of their qualification, experience, and ability to conduct unbiased, quality and objective peer review.
    • Assign applications to each reviewer for analysis, feedback, critique, and scoring.
    • Administer the SRG meeting and oversee regulatory aspects.
    • Prepare a consolidated summary for all the reviewed applications to be discussed in the meeting.
  • Chair: moderates the meeting of the panelists discussing the scientific and technical worth of the applications.
  • Reviewers: consist of the panelists who review the grant applications. Grant applications are sent to the reviewers at least six weeks prior to the review meeting. Each reviewer reads the application and scores it as per the score review criteria (described below). The reviewer also writes a review of the assigned application and makes recommendations on the basis of scientific/technical assessment and numerical scores. If need be, the reviewer will also raise questions and concerns about the safety of human and animal subjects of the research proposal. These include women, children, minorities, animal welfare, etc. The allocated budget is also scrutinized in the first level of review. The reviewer makes recommendations about the propriety and rationale of the budget. The confidentiality of the peer-review is strictly maintained throughout the process.
  • NIH Staff: consist of federal officers who have valuable suggestions and inputs for the close peer-review meeting. These consist of federal officials and NIH institute or center staff. However, they are not the integral part of the SRG meeting and need advance approval from the SRO before joining the meeting.
  • Advice for Applications at First Level of Review: The peer-review process is confidential; the applicant is strictly advised not to contact or approach any of the NIH staff or the reviewer throughout the review. Doing so will result in dire consequences resulting in canceling of the license.

Other things included in the first level of review are:

  • Peer-Review Criteria
    • Overall Impact: The reviewers assess the application with the help of an overall impact score that determines its strength, potential, and power. The NIH follows a scored review criteria to judge the application; 5 parameters are assessed for each application. These are;
      • Significance
      • Investigator(s)
      • Innovation
      • Approach
      • Environment

Each criterion is assessed, investigated and scored. The score determines and predicts the overall likelihood of the application of obtaining the grant fund. These scores are discussed in detail below.

    • Additional Review Criteria: The application is further evaluating the scientific and technical merit of the application based on the following items:
      • Applications from organizations other than the U.S.’s
      • Agent
      • Plans and procedures of resource sharing
      • Verification of biological/chemical Resources
      • Budget

None of these items, however, are scored.

  • Scoring: Each application is scored on a scale of 1-9 where 1 denotes = ‘exceptional’ and 9 denotes = poor.

Scoring is initially done individually; each reviewer assigns a score to all five criteria for each application, i.e., Significance, Investigator(s), Innovation, Approach, and Environment. At the same time, the reviewer gives a preliminary overall impact score for the application. The preliminary scores determine the eligibility of the application for the meeting, i.e., whether the applicant possesses the merit to make it to the peer-review meeting discussion.

In the meeting, each member of the committee, as well as the assigned reviewer, again provide the final impact score. The final score of the application is determined by adding all the impact scores from each member and taking a mean. The mean is multiplied by 10 in a way that a scale (percentile) is developed, i.e., 10 (high impact) – 90 (low impact).

Applications that do not make it to the meeting are rejected (Not Discussed – ND) and marked “++” on the first page of the application, i.e., summary statement.

Rejected applications usually lack merit and ethics and are considered risky (pose a threat to humans or animals; select agents are hazardous) and are designated Not Recommended for Further Consideration (NRFC). The NRFC applications do not make it to the Second Level of Review.

All applicants are informed of the application status and scores by the end of the First Level of Review.

  • Appeals: The applicants unhappy with the impact score believe the review was flawed/biased/erroneous can appeal through the NIH Peer Review Appeal System for a revision.

2.     Second Level of Review

This review is performed by the Institute and Center (IC) or National Advisory Councils consisting of scientific and public officers possessing expertise in the field of health, medicine, disease, and clinical sciences. These scientists, recruited by the concerned IC, maybe public representatives or hail from extramural research centers. The board is approved by the Department of Health and Human Services.

How the Second Level of Review is Done

The grant application, after passing through the First Level of Review, enters the Second Level where the committee members examine it. During the second review, the following items are reviewed:

  • The overall impact scores from First Level Peer Review
  • Percentile Rankings
  • Summary Statements

All three aspects are reviewed and analyzed in light of the concerned Center’s policies, priorities, and aims. In addition, the Council members have the complete grant funding plan based on which they conduct a Special Council Review of the available applications. The applications that make it to round two also include those that require re-funding. These applications are mainly from well-funded recipients receiving at least $1 million from the NIH. The additional review of these applications is done to determine if more funds should be allocated.

The reviewers forward their feedback, advice, and recommendations of the applications to the Institute/Center Directors who make the final decision. The fate of a grant application is determined in the Second Level of Review. The applicant is sent the summary statement along with the score and application approval/rejection status.

Post-Review Process

A grant application meets its fate in two forms; either rejection or approval. Approval results in funding, a prestigious label and a pathway to independence. Rejection can be frustrating, but applicants are invited and encouraged to apply again after making up for the shortcomings in the first application and research plan.

Either way, the NIH guides applicants about steps to take.

When Your Application is Funded

Once funded, the NIH ICs announce the paylines to guide and facilitate the grantees. The payline depends upon the application scores; the more the applicant scores on the scale of 1-9, the more the funds. The payline varies according to the fiscal year when the application was under review. Determining the pay lines can be a bit confusing in the start and there is a delay of several months before you would be able to identify the payline for the given fiscal year.

The best source of information about the payline is the IC that is announcing it. The program official mentioned at the top of your application’s summary statement can be another source of information.

Once awarded, the grantee is invited to work closely with the following staff:

  • IC program official on scientific research
  • Grants Management Officer on administrative and budget issues

Both of these officers will cooperate with the grantee throughout the funding duration.

When Your Application is Rejected

It is not surprising or unusual for an application to get rejected. The NIH is literally inundated with the funding requests, and it is natural that not every application gets through. In fact, barely a few hundred applications manage to win the award. In 2017, the NIH received over 50,000 research proposal grant applications, of which only about 640, involving 11,000 principal investigators, secured funding (Lauer, 2018).

The applicant can either make changes in the submitted application and resubmit it or drop a fresh application. The NIH program official can be in contact for further help and guidance.

References

  1. Aaron, D.J., Acosta, B., LaPorte, R.E., Nishimura, R., & Sekikawa, A. (1998, September 26). Peer review of grant applications. The Lancet, 352(9133), https://doi.org/10.1016/S0140-6736(05)60103-1
  2. Ismail, S., Farrands, A., & Wooding S. (2009, July 16). Evaluating grant peer review in the health sciences a review of the literature. Santa Monica, CA: RAND. https://doi.org/10.7249/TR742
  3. nih.gov. (2019). [online] Available at: https://grants.nih.gov/grants/peer/guidelines_general/Review_Criteria_at_a_glance.pdf [Accessed 23 Apr. 2019].
  4. Lauer, M. (2018, March 7). FY 2017 By the Numbers [Blog post]. Retrieved from https://nexus.od.nih.gov/all/2018/03/07/fy-2017-by-the-numbers/

Author Conductscience

More posts by Conductscience