2. Florida Sea Grant Research in 2006-2007: Outlook and Digest of Review

Note: Florida Sea Grant has completed its review of proposed research for the 2006-2007 biennium. Next year 14 new projects will be supported, subject to award of the expected Federal budget. Below is a two-part, in-depth review of the recent proposal review process. We thank all faculty and collaborators for their participation. The next FSG statewide call for proposals will be issued in February 2007.

Florida Sea Grant Research in 2006-2007 (part 1): A Look Ahead

Two-year Florida Sea Grant projects to begin in 2006 fall into six categories, identified in Table 1. Out of 76 Statements of Interest submitted in spring, 14 full projects are scheduled for support at eight institutions. Statements of Interest were received from 15 of the 16 participating institutions in Florida Sea Grant and from three other universities or organizations in Florida.

Table 1. Subjects of Florida Sea Grant research proposed for 2006-2007 biennial core program.

The titles of the projects to be supported will be announced in February 2006, once the Sea Grant award is final.

Total funding for the Florida Sea Grant core program is anticipated to be approximately $2 million in FY 06-07. Based on national guidelines, about 50% is used to fund research. Of the remaining, Florida Sea Grant Extension receives about 33%, the Communications program 8% and 9% is used to run the management office. The $2 million does not include funds gained by faculty in various national competitions. For complete information on our funding from all sources, see our Performance Counts, Year 2004 report, section 3, program funding from all sources http://www.flseagrant. org/about_us/reports/annual_report/annual_report.htm.

The 76 Statements of Interest submitted reflected the priorities advertised in the February 2005 solicitation for project ideas. That solicitation was based on long-range planning that over several years has involved hundreds of faculty, agency scientists and administrators, business representatives, and citizens. (The FSG long-range plan also is at our website http://www.flseagrant.org/about_us/strategic/index.htm.

Florida Sea Grant Research in 2006-2007 (Part 2): How Proposals Were Reviewed

Proposal Review Process and Decision-making

After Statement of Interest review by six visiting out-of-state scientists - - assisted by mail by two "non-visiting" panelists and a host of individual readers for selected items - - the 76 original Statements (i.e., 3-page preliminary proposals) were reduced to 31 full narrative proposals. This occurred during April 2005.

Thirty-one full proposals were received and subsequently reviewed during August 2005. Each received at least three peer reviews (the average was 5.7 per project). Readers included six visiting Technical Panelists, two non-visiting panelists who read certain projects in disciplines not represented on the technial panel, and scores of mail peer reviews worldwide. Over a two-day period the Technical Panelists (again, all recognized experts from out-of-state) reviewed all external comments, their own reviews, and sometimes notes from FSG program management staff on program themes, extension aspects, etc. The National Sea Grant College Program monitor from the National Sea Grant Office was also in attendance to participate in and observe the process.

Peer Reviews

The job of each visiting Technical Panelist was to present to the group a synopsis of the proposal's intent and design, and then his or her own scientific review and the external written reviews. In most cases, a formally designated secondary reader on the panel added comments as well as individual reviews. The meeting spread over two long days, and discussion of each proposal normally took 15 to 20 minutes. Typically, proposals ranked as excellent by all sources required less discussion.

All readers were asked to comment on four criteria: (1) rationale, or importance of the issue, with quantification; (2) scientific merit; (3) innovativeness; and (4) investigator qualifications. The most useful review comments explain -- at length in a page or two -- why the proposed work has merit, or how its weaknesses might constructively be addressed. Finally, reviewers indicated a summary score of Excellent, Very Good, Good, Fair or Poor. The discussion of each proposal concluded with assignment of an overall rank from among the five choices just indicated and a written panel summary providing the strengths, weaknesses and recommendations for improvement. Each summary was prepared by the lead discussant and read to the entire panel for approval.

In theory, all 31 proposals could have been rated "Excellent" by the Panel. In fact, the mean score of all proposals fell in the Good to Very Good range. When the "grade-point average" (GPA) for all proposals is calculated, on a scale of 1.0 to 5.0 (i.e., Poor to Excellent) the average score is 3.6 (Figure 1, middle bar).

The merit of proposals is further indicated by the left-hand bar of Figure 1, in that for the proposals ultimately funded by FSG, a mean Panel ranking of between Very Good and Excellent was recorded, and the GPA was 4.3. Meanwhile declined proposals had a GPA of 3.0, falling as a group squarely on the Good ranking (Figure 1, right-hand bar).

Figure 1. Average Technical Review Panel ranking of proposals based on the following scale: Excellent, Very Good, Good, Fair, Poor.

As noted, the discussion and recommendations from the Panel included external written reviews. The distribution of all written reviews across the five rankings (on the Excellent-Poor scale; 5-1) is indicated in Figure 2. Again, funded proposals received a larger proportion of Excellent scores.

Figure 2. Average number of written peer reviewers ranking a proposal as Excellent, Very Good, Good, Fair, Poor.

It is important to note that after review of all proposals individually, the Panel and FSG managers re-visited the categories of rankings (Excellent, etc.) in order to develop a comparative sense of the scoring. This afforded an opportunity to examine how proposals fit into FSG theme areas or complemented one another.

Funding Decisions

It was clear when the Panel adjourned that there were more proposals worthy of funding than there was available funding. For three full days after the Panel dismissed, FSG program management compiled the review data in order to make decisions about funding. The guiding principle was to follow the Panel's recommendations. To aid in the decision process, a worksheet whereby numerical scores could be assigned to six review criteria -- namely project rationale (20%), scientific merit (40%), investigator qualifications (5%), external participants (15%), expected results (10%), links to other projects (10%) -- was employed, as has been done for several years. This worksheet was identical to the one used at the Statement of Interest level, thus each proposal writer had seen one prior to writing the full proposal. Assignments of scores reflected Technical Panel discussion (which sometimes brings out new information not recorded in written review sheets) and analysis by FSG staff. Scores of funded and declined proposals are summarized in Figure 3.

Figure 3. Average Florida Sea Grant score based on written peer reviews for Rationale (R), Scientific Merit (S), Participants/Applications (A), Expected Results (E), Links (L)
and Investigator Qualifications (I).

As clearly indicated by FSG in its call for proposals, student involvement is important. Indeed this factor was helpful in deciding between proposals that otherwise might be similar in overall score. It is significant that on average almost two students per project will be supported in 2006-2007 (Figure 4).

Figure 4. Average student involvement per proposal in number of students per project and months of involvement in accepted, all and declined.

Other factors that came out in discussion included the quality of progress reports if a proposal is building on a current Sea Grant project, if other states are contributing funds for their personnel, if a multi-state investigator/student team is proposed, and how unique the program area was to Sea Grant as compared to other funding sources and the strength of the outside partner's participation.

All faculty received a worksheet and the technical panel summary with evaluation comments and anonymous copies of all written peer reviews, to maximize feedback of information. Investigators for accepted proposals are requested to write rebuttal letters addressing review comments, and those letters are included as part of the package sent to the National Sea Grant Office.

Reviewers

One final analysis was also conducted of the reviewer database associated with the 31 proposals reviewed. It took contact with 253 potential reviewers to secure 3.7 reviews per project. Adding two written technical reviews from technical review panel members means 5.7 reviews per proposal were obtained. These data as well as the average per project for other measures are given in Table 2.

Table 2. Number of reviewers contacted, status of contact and reviews received for 31 proposals, 2006-07.

* Includes those who indicated it would be conflict of interest.
**Includes two written technical panel reviews per proposal.

The 253 reviewers contacted reside in 132 different academic units: universities, research laboratories, state or federal scientific agencies. Forty of the units (30%) contacted were outside the U.S. Of the total 253 reviewers contacted, 46 (18%) were in the units outside the U.S.