{"id":12863,"date":"2023-01-29T18:52:34","date_gmt":"2023-01-29T07:52:34","guid":{"rendered":"https:\/\/rationalemagazine.com\/?p=12863"},"modified":"2023-01-29T18:57:58","modified_gmt":"2023-01-29T07:57:58","slug":"success-without-substance","status":"publish","type":"post","link":"https:\/\/rationalemagazine.com\/index.php\/2023\/01\/29\/success-without-substance\/","title":{"rendered":"Success without substance"},"content":{"rendered":"

New Zealander Nigel Richards has won the French Scrabble championship twice. What\u2019s more remarkable than double wins is that Nigel\u00a0doesn\u2019t speak French<\/a>. He applied his prodigious brain to the task of memorising words from the French dictionary, bypassing the need for understanding.<\/p>\n

In 2022,\u00a0The Lancet<\/i>\u00a0medical journal\u00a0achieved a feat<\/u><\/a>\u00a0that has parallels with Nigel\u2019s Scrabble win. It achieved the highest \u2018impact factor\u2019 of any scientific journal in history. This number does not measure noble goals such as improvements in the length or quality of life. Instead, impact factor is a numbers game tangential to the purpose of medical research. Like Nigel\u2019s Scrabble game, it is success without substance.<\/p>\n

The impact factor is calculated using the total number of times scientists have referenced papers from the journal, so it measures attention, both good and bad. The average journal article in medicine gets 15 citations over 10 years. As an example of bad attention,\u00a0The Lancet<\/i>\u2019s famously discredited study that linked the MMR vaccine to autism has more than 1,800 citations.<\/p>\n

Citations are often inaccurate, with researchers referencing findings that were\u00a0not in the paper<\/u><\/a>, or misunderstanding the paper. Many citations are fleeting, with authors using long lists of citations as a way of demonstrating their in-depth knowledge of a field.<\/p>\n

The desire to win the impact factor game has predictably led to bad behaviour, with journals seeking to inflate their impact factor by\u00a0fiddling their data<\/u><\/a>. Some citations are even corrupt, with journals and reviewers\u00a0manipulating citations<\/u><\/a>\u00a0to make themselves look better.<\/p>\n

Real improvements are what journals should aspire to, but measuring how a journal has helped people\u2019s lives is extremely difficult. Instead, we are lumped with the impact factor that is simple to measure, but measures nothing of value.<\/p>\n

To win the impact factor game, journals need to publish papers that will garner a lot of attention. This creates an incentive for journals to publish headline-grabbing papers about new breakthroughs, while important papers debunking existing treatments can find it harder to get in a \u2018top\u2019 journal.<\/p>\n

Real improvements are what journals should aspire to, but measuring how a journal has helped people\u2019s lives is extremely difficult.<\/strong><\/p><\/blockquote>\n

This also creates an incentive for researchers to work on new and exciting breakthroughs. But there\u2019s a lot of much-needed research on relatively mundane parts of health. For example, governments could save an enormous amount of money if they stopped providing\u00a0treatments that have no scientific evidence<\/u><\/a>.<\/p>\n

One\u00a0headline-grabbing paper<\/u><\/a>\u00a0published by\u00a0The Lancet<\/i> has had 5,878 citations \u2013 the Scrabble equivalent of playing ‘quiz’ on a triple letter score. This paper grabbed the headlines because it was an early study of the risks of dying from COVID-19 from a hospital in Wuhan, China. But this paper includes a serious flaw. The calculations excluded patients who were still alive, creating an enormous potential for reverse-survivor bias<\/u><\/a>\u00a0to skew the results.<\/p>\n

For scientists, this flaw is easy to spot, so it\u2019s not clear why it wasn\u2019t spotted in the peer review process, where other experts check over the paper before publication. But\u00a0The Lancet<\/i>\u00a0has also missed other flaws, for example another\u00a0headline-grabbing COVID-19 paper<\/u><\/a>\u00a0<\/u><\/a>that needed readers to point out impossibilities in the data. Possibly the urgency of the pandemic meant that peer review was trumped by attention.<\/p>\n

It\u2019s not just journals that are distracted by flimsy numbers. Scientists are also vulnerable to citation competitions rather than competing to do the best science. Scientists can boost their citation scores by citing their own work. Bigger boosts come from working together, and\u00a0citation cartels<\/u><\/a>\u00a0have been discovered where groups of scientists make a pact to cite each other\u2019s papers.<\/p>\n

Universities are also prone to meaningless metrics, as they compete in international rankings tables, such as\u00a0QS<\/u><\/a>,\u00a0Times Higher Education<\/u><\/a>\u00a0and\u00a0ARWU<\/u><\/a>. The idea that any university could be summed up using a single number is something that primary school children could understand. Nevertheless universities \u2014 the pinnacle of education \u2014 are enslaved to these numbers. And what are the tables based on? Citations feature heavily, and other numbers that are easy to measure but do not measure the quality of education.<\/p>\n

But recently two heavyweight\u00a0US law schools have withdrawn<\/u><\/a>\u00a0from the biggest league table used in the US. This is a bold decision as league tables\u00a0can influence student numbers<\/u><\/a>\u00a0and higher education policy. The schools took a stand because the tables are \u201cusing a misguided formula that discourages law schools from doing what is best for legal education.\u201d<\/p>\n

Scientists, journals and universities have become slaves to misguided formulas based on meaningless data. Science is one of humanity\u2019s proudest achievements, but scientists are human and have become distracted by prestige games that do nothing to advance science.<\/p>\n

Now more than ever, science needs to be performing at its peak, as humanity deals with its biggest ever challenges. It\u2019s time to stop the counting games and work on what really counts.<\/p>\n

Originally published <\/i>by\u00a0<\/i>360info<\/i><\/a>. It is republished under Creative Commons<\/a>.<\/i><\/strong><\/p>\n

Photo by Brett Jordan<\/a> on Unsplash.<\/strong><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

New Zealander Nigel Richards has won the French Scrabble championship twice. What\u2019s more remarkable than double wins is that Nigel\u00a0doesn\u2019t<\/p>\n","protected":false},"author":623,"featured_media":12864,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[65],"tags":[543,371],"coauthors":[542],"class_list":["post-12863","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-science-health","tag-academia","tag-science"],"acf":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/posts\/12863","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/users\/623"}],"replies":[{"embeddable":true,"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/comments?post=12863"}],"version-history":[{"count":2,"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/posts\/12863\/revisions"}],"predecessor-version":[{"id":12866,"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/posts\/12863\/revisions\/12866"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/media\/12864"}],"wp:attachment":[{"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/media?parent=12863"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/categories?post=12863"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/tags?post=12863"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/rationalemagazine.com\/index.php\/wp-json\/wp\/v2\/coauthors?post=12863"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}