Bibliometric Analysis of Dubious Research

Abstract

Retracted papers are scientific or scholarly works officially withdrawn by the publisher or journal after their initial publication. The primary goal of retractions is to rectify the literature and alert readers about articles containing substantially flawed or erroneous content or data, or due to ethical concerns, rendering reported findings and conclusions unreliable. Retraction notices are typically issued for various reasons, including scientific misconduct, genuine mistakes, or problems with peer review. This chapter provides a systematic analysis of the dubious research identified in the Web of Science Core Collection. Bibliometric analysis was conducted on dubious research to assess the magnitude and influence of the questionable work on the pool of knowledge. The contingency matrix between countries and Web of Science categories of retracted papers reveals correlations between certain domains and the countries. To counter this growing tendency, a multi-pronged approach is essential. Robust policies, vigilant watchdogs, and targeted interventions by institutions are necessary to uphold the integrity of scholarly literature. Academia cannot afford to remain silent in the face of this threat to its credibility.

1. Introduction

The number of retractions in journals is on the rise, posing a significant challenge to scientific integrity [1]. Retraction notices are typically issued due to various factors, including scientific misconduct, genuine mistakes, and problems with peer review, among others. The primary purpose of retractions is to rectify the literature and apprise readers about articles with substantially flawed or erroneous content, and ethical concerns, rendering their findings and conclusions unreliable [2]. The increasing trend in retractions in recent years has raised concerns about the quality of the peer-review process.

Typically, a retraction comprises the original article and the retraction announcement, often accompanied by explanations [3]. Traditional print journals lack a mechanism to remove retracted articles, unlike online-only journals, which can eliminate such articles from all repositories. Eliminating all copies, especially those shared on personal websites, proves impractical many times. Consequently, there is an ongoing risk that certain readers may not identify a retracted article.

The dissemination of erroneous research findings, as exemplified by the chloroquine study, has far-reaching negative impacts on the scientific community, society, and the authors themselves, posing a threat to their professional standing [4]. When a mistake or misconduct is identified, publishing a retraction of the fraudulent article becomes essential [5]. This serves to rectify the literature, inform the scientific community, and underscore the commitment of the scientific community to correct its errors. The “publish or perish” culture jeopardizes the integrity and honesty of researchers. To ensure ethical practices in academic publishing and preserve the integrity of researchers and accessible knowledge, the Committee on Publication Ethics (COPE) has established comprehensive guidelines and measures. Included in these guidelines is the Code of Conduct, which strives to integrate ethical practices into the editorial culture. One notable measure is the promotion of error reporting by researchers and publishers to maintain journal reliability. Through this approach, COPE aims to tackle various aspects of scientific misconduct, including duplicate publication, plagiarism, falsification and fabrication, ethical concerns, authorship disputes, and fraudulent peer reviews. Furthermore, the COPE guidelines establish the criteria for ethical publication practices and offer guidance to editors, authors, and publishers [6].

Through proficient utilization of bibliometrics, the causes of retractions can be identified, trust can be restored, and the integrity of scientific knowledge can be protected. This endeavor goes beyond mere academic analysis; it is a mission to safeguard the very fabric of scientific progress.

The following provides operational definitions [7, 8] for the key reasons behind retracted papers as available in the academic literature:

Plagiarism: The author published work that included unacknowledged text, data, images, or ideas borrowed from other authors, violating academic integrity by not providing appropriate citations.

Duplicate publication: The author repeatedly failed to provide proper citations to their previous work when publishing their own texts, images, or data, presenting them as new content. Submitting the same work to multiple journals, i.e., accepted by a journal while still undergoing peer review by the initial journal, is considered a violation. In a similar case, five papers of an author were retracted, as the journals detected resemblances in titles, content, and data by individuals from distinct university affiliations [9, 10]. This behavior makes up a breach of professional ethics and the COPE code of conduct. According to COPE guidelines, submitting a paper implies that the work is neither published before nor being considered for publication elsewhere.

Falsification and fabrication: Falsification pertains to the deliberate manipulation or selective omission of the data by the author(s) to support their hypothesis. The act of fabrication involves the creation or alteration of data by author(s), which is then documented and disseminated in the results and later published.

Error: The scope of this category extends to encompass genuine mistakes or technical errors that arose during the publication process, legitimate errors made by authors, unavailability of raw data, inability to reproduce results, and errors in data that authors subsequently accepted with apologies in notices. Authors in coordination with the journals start voluntary withdrawals in this category. The term “heroic act” pertains to retractions that are requested by the authors themselves [11].

Misconduct: This terminology was specifically employed to refer to papers that were retracted because of issues pertaining to authorship, ethics, funding, or falsified peer reviews. The Medical Research Council (MRC) [12] and the US Office of Research Integrity (ORI) [2, 7] identify duplicate publication, plagiarism, falsification and fabrication, ethical issues, authorship issues, and fake peer reviews as forms of intentional misconduct.

Retraction Watch, a blog that has documented approximately 51k retracted papers, has observed a significant increase in retractions since its establishment 13 years ago, which is believed to be vastly underestimated [1, 13]. Confidence in researchers, universities, journals, and the scientific community can be undermined by unethical research. Intending to uphold the quality of accessible knowledge, the peer-review process, alongside the responsible and transparent correction of articles, serves this purpose [14].

2. Methodology

The data were accessed from the Web of Science Core Collection on January 20, 2024, using the following Boolean string in advanced search:

DT = (Expression of Concern OR Retracted Publication OR Retraction OR Withdrawn Publication OR Publication with Expression of Concern) AND PY = (1986–2023). All instances of retraction, including publications with expressions of concern and withdrawn, were included in this analysis. The time period for the publications was specified as 1986–2023.

For further investigation, connections between retractions and diverse bibliometric factors, including institutions, countries, and author/journal/publisher attributes, were explored using various tools, including bibliometrix, VoSviewer, Gephi, and CorText.

3. Results and discussion

In this section, a comprehensive analysis of the results obtained from an in-depth examination of retracted papers from Web of Science Core Collection database during 1986–2023 is presented. The analysis encompasses a detailed exploration of the descriptive characteristics, subject areas, journals, publishers, and the geographical distribution associated with these retracted publications.

3.1 Descriptive analysis

Understanding the landscape of retracted papers is crucial for the scholarly community as it provides insights into the factors contributing to retractions and their implications for scientific integrity. By delving into the descriptive analysis of retracted papers, the goal is to identify patterns and commonalities that can illuminate the underlying causes leading to retractions. This knowledge is essential for developing strategies to uphold the rigor and reliability of scientific research.

The information regarding editorial notices issued to dubious publications reveals that “Retraction” is the most prevalent notice, comprising 16,188 (55.33%) of the total count, while “Retracted Publication” accounts for 11,073 (37.85%). “Expression Of Concern” constitutes 1300 (4.44%), and “Publication with Expression Of Concern” accounts for 709 (2.42%) of the total count. Editorial notices for “Withdrawn Publication” and “Item Withdrawal” represent the smallest proportions, with 238 (0.81%) and 63 (0.22%), respectively. Consistency in the notices issued by journals or publishers may enhance the bibliometric data, aiding in identifying the real root cause of these retracted papers [7].

Figure 1 illustrates the temporal evolution of retractions from 1986 to 2023. The retraction count was relatively low between 1986 and 2007. There was a consistent rise from 2008 to 2015, followed by a period of exponential growth. In 2023, the number of retractions reached 4429, making up 0.15% of the total publications for that year alone.

Figure 1.

Temporal evolution of retracted papers from 1986 to 2023 according to Web of Science.

3.2 Subject areas

The examination of subject areas associated with retracted papers offers insights into the specific fields where retractions are more prevalent. Identifying these subject areas is essential for targeted interventions and improvements in research practices to mitigate the occurrence of retractions.

Despite being more prevalent in countries with a strong research background and reputable journals, retractions are not limited to specific fields of study and can be found in diverse areas such as medicine, engineering, and the social sciences [15]. Nonetheless, recent studies on retractions in the Web of Science database have revealed elevated rates in the domains of chemistry, biomedical sciences, and engineering [16, 17].

The word cloud displayed in Figure 2a visualizes the prevailing terms correlated with retracted papers in a range of academic disciplines. Each term’s frequency is depicted by its respective size. Terms like “oncology,” “biochemistry & molecular biology,” and “multidisciplinary sciences” are prominently featured, suggesting they are common topics associated with retracted papers. To categorize them into wider fields, an overlay map was created.

Figure 2.

(a) Word cloud of most frequent Web of Science subject categories, and (b) overlay map of retracted papers across Web of Science categories, grouped into five main disciplines: Biology & Medicine, Chemistry & Physics, Environmental Science & Technology, Psychology & Social Sciences, and Engineering & Mathematics.

Figure 2b displays a network map of a retracted paper across various Web of Science categories, grouped into five main disciplines. The field of Biology & Medicine is visually represented by the color red, while Chemistry & Physics are depicted in blue. Environmental Science & Technology is associated with the color yellow, Psychology & Social Sciences in green, and Engineering & Mathematics showcased in a purple-colored cluster. Nodes in each discipline represent sub-domains identified from journal listings, such as “Oncology” and “Genetics & heredity” within “Biology & Medicine.” Lines connecting nodes indicate relationships or overlaps between categories, with the size of each node corresponding to the frequency of retractions in that category.

Results indicate that the majority of retractions come from Biology & Medicine, Chemistry & Physics, and Engineering & Mathematics, while retractions in Environmental Science & Technology and Psychology & Social Sciences are relatively low in volume.

3.3 Journals

An analysis of the journals and publishers involved in disseminating retracted papers provides valuable information about the platforms through which potentially problematic research has been communicated. This scrutiny is essential for understanding the role of academic publishing in maintaining the quality and credibility of scientific knowledge.

When considering the impact factor, establishing a direct correlation between higher-impact-factor journals and an increased number of retractions was not feasible. A study by Rubbo et al. [15] demonstrated a direct relationship between the impact factor of journals and the occurrence of retractions. However, contrary to this, some studies have asserted that journals with a lower impact factor are more prone to retracting articles [18, 19].

Table 1 describes the count of retractions in different scientific journals. PLoS One has a significantly higher number of retractions, with 1056 counts (3.54%), while other journals such as J Biol Chem, J Cell Biochem, and RSC Adv have fewer retraction counts, with 425 (1.42%), 347 (1.16%), and 343 (1.15%) respectively.

Journals	Retractions count	Percent of total retractions (%)
PLoS One	1056	3.54
J Biol Chem	425	1.42
J Cell Biochem	347	1.16
RSC Adv	343	1.15
Biomed Res Int	325	1.09
Tumor Biol	293	0.98
Eur Rev Med Pharmacol Sci	283	0.95
Onco Targets Ther	265	0.89
Sci Rep	263	0.88
Appl Bionics Biomech	247	0.83

Table 1.

Retraction counts from various scientific journals.

The correlation between this expansion and the retraction of articles by journals does not necessarily imply a surge in scientific misconduct. A meticulous study emphasized the need to understand whether there has been an actual increase in misconduct or if the rise in cases is solely a result of enhanced detection tools [20].

Figure 3 illustrates the trend of retractions in five scientific journals from 1996 to 2023. These journals retracted most of their published papers at different times. The last decade witnessed significant retractions in PLoS One, the Journal of Biological Chemistry, the Journal of Cellular Biochemistry, RSC Advances, and Tumor Biology. PLoS One and the Journal of Cellular Biochemistry show alarming spikes in retractions in 2021 and 2022, respectively.

Figure 3.

Temporal evolution of retracted papers for five scientific journals from 1996 to 2023.

3.4 Publishers

Publishers serve as guardians of scholarly integrity, wielding significant influence in preventing and addressing retractions. Through stringent manuscript vetting, plagiarism detection tools, and clear ethics guidelines, they establish a foundation for trustworthy academic content. Editorial boards, appointed by publishers, play a pivotal role in upholding ethical standards within journals.

Publishers also contribute to fostering a culture of research integrity by providing educational resources for authors and actively monitoring post-publication discussions. The role of readers in developing a research culture cannot be undermined. For example, in the case of 33 papers, readers assumed the role of whistleblowers, identifying cases of misconduct and bringing them to the attention of the journal editor [21].

Table 2 presents the number of retractions and the corresponding percentage by various publishers. According to the data, Elsevier has the highest retraction count at 5529, which makes up 18.9% of the total retractions, followed by Springer Nature and Wiley with 4299 (14.7%) and 2440 (8.3%) retractions, respectively.

Publisher	Retractions count	Percent of total retractions (%)
Elsevier	5529	18.9
Springer Nature	4299	14.7
Wiley	2440	8.3
Hindawi Publishing Group	1598	5.5
Taylor & Francis	1577	5.4

Table 2.

Retractions by five publishers and the corresponding percentage of total retractions by each publisher.

3.5 Countries

The geographical distribution of retracted papers offers a global perspective on the prevalence of retractions across different regions. Recognizing patterns in the distribution of retractions can inform efforts to address region-specific challenges and promote international collaborations for research integrity.

Figure 4 shows the temporal evolution of retracted papers for different countries, normalized at each period between 1996 and 2023. Each country is represented by a unique color, and the magnitude of each color corresponds to the percentage of retractions from that country at any specific point in time. Notable temporal fluctuations/bumps are observed in countries such as the USA, Germany, Japan, the UK, Canada, Italy, India, China, Iran, Egypt, and South Korea. For instance, in the 1990s and 2000s, the USA had a relatively high volume of retractions compared to other countries. However, in the early 2010s, the volume of retractions in China began to increase, surpassing that of the USA by the mid-2010s. The percentage of retractions from most of the developed countries saw a decline after 2015, whereas China, Iran, and India witnessed an increase during that period.

Figure 4.

Temporal evolution of retracted papers for different countries from 1996 to 2023.

Further investigation is required to explore this trend, which may be influenced by a range of factors including research metrics and roles within the ranking system.

Table 3 presents a compilation of leading countries with retractions, encompassing the overall document count, publications from a single country (SCP), publications involving multiple countries (MCP), the proportion of retractions attributed to each country in relation to the global total, and the MCP ratio determined by dividing MCP by the total documents from that country. China leads in terms of total retractions and SCP, although its MCP ratio is relatively lower compared to other countries. Interestingly, the USA is second, and the list includes scientifically strong countries such as Germany, Italy, the UK, Japan, and Korea.

Country	Documents	Proportion	SCP	MCP	MCP ratio
China	7783	0.267	7016	767	0.099
USA	2396	0.082	1819	577	0.241
India	1249	0.043	1104	145	0.116
Iran	912	0.031	723	189	0.207
Japan	888	0.031	791	97	0.109
UK	508	0.017	384	124	0.244
Korea	451	0.015	365	86	0.191
Germany	381	0.013	285	96	0.252
Italy	360	0.012	255	105	0.292

Table 3.

Countries wise share of retractions.

Based on the research conducted by Ribeiro and Vasconcelos [22], an analysis of retracted articles documented in Retraction Watch from 2013 to 2015 revealed that 85% of these articles were associated with 15 distinct countries. It can be inferred that this problem exhibits uneven distribution across different countries.

Figure 5 demonstrates co-authorship patterns among authors from various countries. The figure displays the co-authorship network of authors’ affiliated countries, with a threshold of 50 documents receiving at least 100 citations. Using VOSviewer and Gephi applications, the analysis was conducted for the period 1986–2023. The clusters (modularity class) of authors’ affiliated countries can be determined by the color of the nodes, visually conveying the size and prominence of the countries within the network. The strength of the links or arrows signifies co-authorships. Notably, China, positioned at the center of the network, was found to collaborate strongly with the USA for dubious research. The data confirms earlier research findings, indicating that predominant retractions originate from countries with a notable research legacy, including the United States and China.

Figure 5.

Co-authorship of authors from various countries.

Figure 6 depicts the co-authorship network of authors’ affiliating institutions from 1986 to 2023, with a threshold of 75 documents and at least 100 citations. The color of the nodes represents the clusters (modularity class) corresponding to the authors’ affiliations. The varying size of the nodes visually conveys the degree or prominence of the institutions within the network, while the thickness of the lines or arrows indicates the strength of co-authorships. Islamic Azad University and the University of Tehran, along with Qingdao University and Shandong University, demonstrated the highest level of collaboration in dubious research.

Figure 6.

Co-authorship of authors-affiliated institutions.

Liu and Lei [23] conducted a thorough bibliometric analysis examining the factors contributing to the rise in retractions in the Middle East over the past two decades. The study identified plagiarism, duplicate publishing, fabricated peer review, approval from institutions/researchers, and fraud as the leading factors behind retractions. Instances of plagiarism and duplicate publications often originate from countries that have recently entered the research field and are associated with journals of low impact factor.

The study conducted by Brainard [20] demonstrates a correlation between implementing comprehensive research misconduct policies in countries or institutions and a decrease in the number of retractions.

Figure 7 shows the contingency matrix of countries and Web of Science categories for retracted papers. The y-axis lists various science categories, including “multidisciplinary sciences,” “biochemistry & molecular biology,” “cell biology,” “oncology,” “medicine, research & experimental,” “pharmacology & pharmacy,” “biotechnology & applied microbiology,” “materials science, multidisciplinary,” and “chemistry, multidisciplinary,” etc. Significant retractions from the USA are highly correlated with “multidisciplinary sciences,” whereas Iranian retractions are strongly correlated with “materials science, multidisciplinary.” Retractions from India fall within the domain of “engineering, electrical & electronic,” while Egyptian retractions show high deviation in “pharmacology & pharmacy.”

Figure 7.

Contingency matrix between countries and Web of Science categories of retracted papers.

4. Conclusions

In concluding this analysis of retracted publications accessed from the Web of science Core Collection database for the period of 1986–2023, it becomes evident that understanding the intricate dynamics of retractions is paramount for sustaining the credibility of scholarly literature. The preceding sections have unveiled patterns in the descriptive analysis, subject areas, journals, publishers, and the global distribution of retractions, shedding light on the multifaceted challenges faced by the academic community. Such as

1. Rising trend of retractions: Between 2000 and 2019, a 15-fold increase has been witnessed. The overall number of dubious publications has increased exponentially, especially after 2015.

2. Focus areas and reasons: Retractions are not limited to specific fields. Biology & Medicine, Chemistry & Physics, and Engineering & Mathematics have higher retraction rates.

3. Journal and publisher landscape: PLoS One leads in retractions as a journal, whereas major publishers like Elsevier, Springer Nature, and Wiley are the main publishers with retractions.

4. Geographical distribution: The temporal distribution fluctuates across countries with a notable research legacy, such as the USA, Germany, Japan, the UK, Canada, Italy, and China. China leads in total retractions and single-country publications, but its multiple-country publication ratio is lower than other countries. Strong collaboration for dubious research exists between certain institutions, such as Islamic Azad University and the University of Tehran. International collaboration in dubious research also exists.

5. Recommendations

There is a pressing need for targeted interventions to address this growing issue. The primary onus, however, lies with publishers to enact measures aimed at averting retractions and upholding the trustworthiness and credibility of scholarly work. Publishers should:

1. Implement rigorous manuscript vetting processes to ensure that submitted articles meet high standards of quality and accuracy.

2. Establish clear ethics guidelines for authors, reviewers, and editors.

3. Use plagiarism detection tools to detect any instances of plagiarism in submitted articles.

4. Provide training and education for authors, reviewers, and editors on best practices in academic publishing in order to ensure they understand their responsibilities and avoid common mistakes.

5. Implement post-publication review processes to identify any errors or issues in published articles before they become widespread.

6. Strive for transparency in the publishing process by providing clear and detailed information about the review process, the authors, and the funding sources for the research.

7. Encourage error reporting by researchers and publishers to maintain journal reliability.

Apart from these, it is imperative for institutions to establish rigorous policies and procedures to handle instances of research misconduct. They should also appoint vigilant watchdogs to monitor the research process and ensure that all ethical guidelines are being followed.

Acknowledgments

We gratefully acknowledge the financial support provided by Northumbria University for this research project.

Conflict of interest

The authors declare no conflict of interest.