Transformative Research is Not Easily Predicted

Research-funding agencies that require scientists to declare at the proposal stage how their projects will be “transformative” may actually be hindering discovery, according to a study by Oregon State University ecologists.

The requirement can result in decreased funding for the “incremental” research that often paves the way for paradigm-shifting breakthroughs, the OSU scientists assert.

Their findings, as well as their recommendation for how to best foster transformative research, were published recently in Trends in Ecology and Evolution.

Sarah Gravem, postdoctoral scholar in integrative biology in Oregon State’s College of Science, was the lead author on the paper, titled “Transformative Research is Not Easily Predicted.”

Gravem, integrative biology professor Bruce Menge and the other collaborators note that the National Science Foundation, which funds roughly one-quarter of the federally supported research at U.S. colleges and universities, “has made the pursuit of transformative research a top priority by asking for a transformative research statement in every major research proposal solicited.”

The NSF defines transformative research as being “driven by ideas that have the potential to radically change our understanding of an important existing scientific or engineering concept or leading to the creation of a new paradigm … . Such research is also characterized by its challenge to current understanding or its pathway to new frontiers.”

Gravem says asking scientists to attempt to create new paradigms or fields in every proposal is unrealistic and potentially harmful.

The OSU scientists argue that a better approach, and one that was suggested more than a decade ago by the board that oversees the National Science Foundation, would be to create a funding subset: a separate NSF-wide program to solicit and support transformational research proposals.

“The board had been concerned that the U.S. was lagging behind other countries in scientific advances, concerned that creative and risky research was not getting funding,” Menge said. “It concluded that what the NSF should do is set aside some funds for risky research proposals, those defined by reviewers as they may or may not work, the chances are sort of slim, but they could turn out to be pretty cool.”

What the NSF did instead, Menge said, was require all proposals to show how the research being proposed would be transformative.

“Instructions to reviewers include the expectation that the reviewer will comment on how transformative the proposed research is,” Menge added.

The problem, the Oregon State collaborators say, is that it’s rarely possible to know at the proposal stage whether a project will turn out to be transformative; their assertion follows interviews and surveys of 78 highly cited ecologists who began with incremental goals and only later realized the transformative potential of their work.

“To start out with that transformative question is a backward way of thinking,” Gravem said. “Surely you have to think big to come up with big answers, and everyone is striving for that, but truly transformative research is an unobtainable standard to place on people at the proposal stage. Trying to make every project paradigm shifting can mean ignoring the incremental and basic science that eventually goes into shifting paradigms. It’s a detriment to ignore the building blocks in favor of the building.”

Gravem said the necessity of incremental research was also explained recently on Freakonomics Radio.

“Economist Ed Glaeser noted that Nobel Prizes are not typically given for single transformative research papers but are often given for a body of incremental research,” she said. “If transformations arise from incremental research, then the transformative criterion is redundant with the solicitation of incremental research. This is reflected by mixed evidence that soliciting transformative research led to increases in transformative outcomes compared with the typical model.”

Expanding fields of knowledge, adding to bodies of evidence, and comparing two fields that haven’t been compared before are the types of gains researchers can reasonably predict, Gravem added. Being asked to forecast how a project will turn out to be transformative puts “researchers in an awkward position that nobody likes.”

“We’re being forced to hype our work at the beginning of a proposal, which doesn’t do anything to help science or to help build trust in science,” Gravem said. “And it turns the funding process into an essay competition that favors people who take more liberty in predicting what their research might show.”

Menge notes that NSF’s plan all along was to reassess the transformative research statement requirement at some point, “and now is the time.”

“Research funding is effectively decreasing, but the demand for funding is increasing, so they look for ways to prune the field of who gets funded – I recognize that as a problem,” he said. “But making artificial hurdles is just wrong. Funding agencies should concentrate on the goals of the research rather than the unknowable outcome.”