Thursday, July 06, 2017

Scientists say "sloppy science" more serious than fraud

An article on Nature: INDEX reports on a recent survey of scientists: Cutting corners a bigger problem than research fraud. The subtitle says it all: Scientists are more concerned about the impact of sloppy science than outright scientific fraud.

The survey was published on BioMed Central.
Bouter, L.M., Tijdink, J., Axelsen, N., Martinson, B.C., and ter Riet, G. (2016) Ranking major and minor research misbehaviors: results from a survey among participants of four World Conferences on Research Integrity. Research Integrity and Peer Review, 1(1), 17. [doi: 10.1186/s41073-016-0024-5]

Abstract
Background


Codes of conduct mainly focus on research misconduct that takes the form of fabrication, falsification, and plagiarism. However, at the aggregate level, lesser forms of research misbehavior may be more important due to their much higher prevalence. Little is known about what the most frequent research misbehaviors are and what their impact is if they occur.

Methods

A survey was conducted among 1353 attendees of international research integrity conferences. They were asked to score 60 research misbehaviors according to their views on and perceptions of the frequency of occurrence, preventability, impact on truth (validity), and impact on trust between scientists on 5-point scales. We expressed the aggregate level impact as the product of frequency scores and truth, trust and preventability scores, respectively. We ranked misbehaviors based on mean scores. Additionally, relevant demographic and professional background information was collected from participants.

Results

Response was 17% of those who were sent the invitational email and 33% of those who opened it. The rankings suggest that selective reporting, selective citing, and flaws in quality assurance and mentoring are viewed as the major problems of modern research. The “deadly sins” of fabrication and falsification ranked highest on the impact on truth but low to moderate on aggregate level impact on truth, due to their low estimated frequency. Plagiarism is thought to be common but to have little impact on truth although it ranked high on aggregate level impact on trust.

Conclusions

We designed a comprehensive list of 60 major and minor research misbehaviors. Our respondents were much more concerned over sloppy science than about scientific fraud (FFP). In the fostering of responsible conduct of research, we recommend to develop interventions that actively discourage the high ranking misbehaviors from our study.
This is important because science journals and leading scientific societies tend to focus on fraud and retractions as the leading causes of mistrust in science. This survey suggests—correctly in my opinion—that the real problems are far more serious.

Here are the top five problems according to the survey results.
  • Selectively cite to enhance your own findings or convictions
  • Insufficiently supervise or mentor junior coworkers
  • Not publish a valid “negative” study
  • Demand or accept an authorship for which one does not qualify
  • Selectively cite to please editors, reviewers, or colleagues
If these are the real problems then that puts the leading journals in an awkward position. So far, they have put the blame on scientists and pretended that they are not part of the problem. But if those five problems are the real reasons why scientists and the general public are losing faith in science then the journals themselves are partly to blame (e.g. ENCODE, Arsenic-Life). Science journals need to improve peer review and they need to punish authors and reviewers who allow publication of sloppy science.


11 comments :

  1. This is fantastically well-timed, thank you Larry. The problem remains what can one do about it? Unlike refuting a valid idea, through the usual counter-argument channels of peer review, some mistakes are so awful, they can only get published by compromising the quality of publication. Luckily for the opportunist, who must be so ignorant that they do not see their own mistake, nor realise the meaning behind no one of repute wishing to touch their work with a proverbial barge pole, there are a bounty of predatory and hack journals out there who are only nominally associated with science. Alas, airing this crap is sufficient to do the damage, so how can it be undone? (Not asking for a friend). A case in point,

    http://www.nauka-a-religia.uz.zgora.pl/index.php/pl/czasopismo/46-fag-2015/921-fag-2015-art-05

    ReplyDelete
  2. I just recently published my first paper in over thirty years in a Springer journal. The peer review was thorough and constructive. The thing that surprised me the most was the number of unsolicited emails I received from other journals congratulating me on the paper and asking me to submit a paper for publication in their journal. I presume that the reputable journals do not do this, but the number of hack/predatory journals out there professing to be peer reviewed science journals is eye-opening. And this does not even include the journals, such as BioComplexity, that exist simply to advocate an ideology.

    Another negative aspect of the publishing process that you touched on, and that is as old as the hills, is the difficulty in publishing papers that obtain negative findings. I often wonder how much time and money is spent on repeating research that has already been done but never published because the findings were not as expected or inconclusive.

    ReplyDelete
  3. It is interesting to note that the Journal of Negative Results in BioMedicine is shutting down. They claim that their goal has been met because many other journals are now publishing negative results so JNRB is no longer needed.

    https://jnrbm.biomedcentral.com/

    ReplyDelete
  4. Speaking of "research misconduct that takes the form of fabrication, falsification":

    https://uncommondescent.com/intelligent-design/turkish-science-group-takes-teach-both-sides-approach-to-darwinism/

    In this case those who were responsible for having helped radicalize entire countries now civil war in Turkey are even proud of their involvement. How do you even estimate the damage? Keep a daily tally of how many academic teachers were wounded and killed in the countries that the DI serves?

    ReplyDelete
    Replies
    1. I wonder if DI etc realise that if Turkey is going to be teaching Intelligent Design it is not going to have anything to do with the Bible?

      Delete
    2. My best guess is it was an "enemy of my enemy is my friend" affair.

      The only thing I am confident of is that their "sloppy science" is now a national security threat to all nations of the world. An epic example of scientific irresponsibility. But (with all the US is automatically blamed for) thankfully spill the beans "News" lives in Canada.

      Delete
  5. Off-topic, but here is Dan Graur dropping a new article bomb:

    https://academic.oup.com/gbe/article/doi/10.1093/gbe/evx121/3952726/An-upper-limit-on-the-functional-fraction-of-the

    ReplyDelete
    Replies
    1. Related: https://www.sciencedaily.com/releases/2017/06/170629134244.htm

      The twist to this one is that all the still functional but kept under control regions can still be considered functional. I'm not sure how much it changes estimates, but our having hitchhikers from the past still functional in our genetic system makes it even harder to deny common ancestry.

      Delete
    2. "Mutational load considerations lead to the conclusion that the functional fraction within the human genome cannot exceed 25%, and is probably considerably lower." -D. Graur

      Human genome could be 25% functional?

      That's quite a change from 10% or less not that long ago...

      Good thing Larry didn't publish his book yet on human genome being 90% junk... ;-)

      Delete
    3. "cannot exceed 25%" does not equal "is 25%".

      Delete
  6. @Jass
    I think you missed the "and is probably considerably lower"
    The 25% number is a highest estimate... ;-)

    ReplyDelete