New ‘diversity index’ in medical research enhances health outcomes.

Impact of Diversity on Evaluating Medical Research

Scientific journals and research papers are typically evaluated based on their “impact factor,” which measures how frequently a paper is cited by other papers. However, a recent study conducted by MIT and other institutions argues that this traditional metric fails to accurately capture the true impact of medical papers on health outcomes, especially for patients in low- or middle-income countries.

In order to better assess the impact of medical papers on health, the research team proposes a new metric called the “diversity factor.” This metric takes into account various factors such as the diversity of authors (in terms of gender and geographic location), the diversity of patients studied, and the interdisciplinary nature of the research team. The researchers evaluated over 100,000 medical papers published in the last two decades and found that most of them scored poorly on the diversity factor.

“The medical knowledge system is controlled by a very noninclusive group of academics, and it’s not diverse at all.”

– Leo Anthony Celi, senior research scientist at MIT’s Institute for Medical Engineering and Science, physician at Beth Israel Deaconess Medical Center, associate professor at Harvard Medical School, and co-author of the study

The researchers hope that their findings will spark further discussions on how to evaluate medical papers in a way that contributes positively to health outcomes for diverse populations, instead of primarily benefiting the groups that have traditionally dominated medical research.

“Blind Spots” in Medical Knowledge

The motivation behind developing the diversity factor index was to address the lack of diversity in the authors of prominent medical publications. The majority of these authors hail from affluent nations, particularly the United States, and are disproportionately white and male.

This lack of diversity in the medical knowledge system has led to a situation where 80 percent of publications originate from only 20 percent of countries. Consequently, the treatment guidelines for various diseases, such as diabetes, hypertension, and cancer, are primarily informed by trials and studies conducted in these privileged nations.

In order to quantify the extent of this problem, the researchers devised an index that incorporates multiple factors. Firstly, they assessed the diversity of authors, considering whether they belonged to a high-income or a low-/middle-income country. The researchers also used an algorithm to determine the gender distribution among authors. This gender diversity is crucial as it fosters inclusion and ensures that different groups are adequately represented in medical studies.

According to Celi, the lack of diversity in the authors of a study results in shared blind spots among them, limiting their ability to fully understand and address the unique complexities of a particular disease. Therefore, cognitive diversity, rooted in varied lived experiences, is essential for advancing medical research.

In addition to author diversity, the index also takes into account the diversity of the departments to which authors are affiliated. Studies that involve collaboration across different disciplines are given higher scores. For instance, a medical study involving healthcare professionals, engineers, and scientists would be rated more favorably than one limited to only a single field.

Lastly, the researchers proposed incorporating the diversity of patient traits as another metric. This includes factors such as sex, race-ethnicity, language, age, and geography. However, due to limited access to data from non-open access papers, the researchers were unable to analyze this aspect extensively in their study.

Celi emphasizes that the metrics they developed should be viewed as a starting point for measuring the impact of medical research on health outcomes in a more equitable manner. He hopes that the broader community will engage in the conversation and contribute to refining these metrics.

Tracking Diversity

Using the OpenAlex database, which aggregates information from various academic article databases, the researchers extracted metadata on approximately 130,000 medical papers published between 2000 and August 2022 from over 7,500 journals.

The findings confirmed the researchers’ expectations, revealing consistent underrepresentation of female authors across all journals. Though male authors still outnumber their female counterparts, the representation of female authors has been gradually increasing. In renowned journals, the ratio of female to male authors per publication ranged from 0.30 to 0.42 in 2021.

Additionally, the study identified a lack of geographic diversity among authors. In 2021, high-income countries accounted for over 5 million authors, upper-middle-income countries contributed approximately 1.5 million authors, lower-middle-income countries had around 470,000 authors, while low-income countries had just over 27,000 authors.

The researchers noted that open-access journals generally performed better on the diversity index compared to non-open-access journals. Among the top 25 journals, both open-access and non-open-access, based on impact factor, the former consistently exhibited higher proportions of female authors as well as authors from low and middle-income countries.

Various entities, including academic institutions, journals, and funding agencies, hold the responsibility to foster diversity in medical publications. Celi suggests that funding agencies, like the National Institutes of Health, can play a significant role in promoting diversity by implementing initiatives that require research teams to be led by principal investigators from under-funded institutions or those with a history of serving minority communities.

Celi concludes by emphasizing the collective accountability of funding agencies, universities, journals, and the media in addressing the existing problems and working towards operationalizing their mission and vision in a more inclusive manner.

Source