Impact Measurements

Citation politics

The responsible use of metrics in research assessment and responsible citation practices acknowledges bias, power, and privilege in academic scholarship. In addition, there are huge differences in publication and citation practices between fields, there are issues with author names and disambiguation, there is no way to know why a specific paper has been cited, and metrics can be manipulated or gamed.

Responsible use of metrics in research assessment

Before using metrics to evaluate an individual:

• Leiden manifesto for research Metrics: 10 principles to guide research evaluation
• DORA Resource Library: Research assessment toolkit for institutions
• The metric tide: review of metrics in research assessment (UK Research and Innovation)

Responsible citation practices

Look critically at bibliographies:

• Citational politics training module from CLEAR
• An article passes the Gray test if it cites and discusses the scholarship of at least two women and two non-white people.
• Gender Balance Assessment Tool
• Is there a Citation Diversity Statement?
  • Zurn, P., Bassett, D. S., & Rust, N. C. (2020). The Citation Diversity Statement: A Practice of Transparency, A Way of Life. Trends in Cognitive Sciences, 24(9), 669-672. https://doi.org/10.1016/j.tics.2020.06.009

Readings

Racial bias

• Chakravartty, P., Kuo, R., Grubbs, V., & McIlwain, C. (2018). #CommunicationSoWhite. Journal of Communication, 68(2), 254-266. https://doi.org/10.1093/joc/jqy003
• Escobar Jones, Cherice, Reid, Gwendolynne, Poe, Mya. “Leading American medical journal continues to omit Black research, reinforcing a legacy of racism in medical knowledge”
• The politics of citation: Is the peer review process biased against Indigenous academics? CBC Radio (2018)

Gendered citation practices

• Dion, M. L., Sumner, J. L., & Mitchell, S. M. (2018). Gendered Citation Patterns across Political Science and Social Science Methodology Fields. Political Analysis, 26(3), 312-327. https://doi.org/10.1017/pan.2018.12
• Dworkin, J. D., Linn, K. A., Teich, E. G., Zurn, P., Shinohara, R. T., & Bassett, D. S. (2020). The extent and drivers of gender imbalance in neuroscience reference lists. Nature Neuroscience, 23(8), 918-926. https://doi.org/10.1038/s41593-020-0658-y
• Ross, M. B., Glennon, B. M., Murciano-Goroff, R., Berkes, E. G., Weinberg, B. A., & Lane, J. I. (2022). Women are credited less in science than men. Nature, 608(7921), 135-145. https://doi.org/10.1038/s41586-022-04966-w
• Teich, E. G., Kim, J. Z., Lynn, C. W., Simon, S. C., Klishin, A. A., Szymula, K. P., Srivastava, P., Bassett, L. C., Zurn, P., Dworkin, J. D., & Bassett, D. S. (2022). Citation inequity and gendered citation practices in contemporary physics. Nature Physics, 18(10), 1161-1170. https://doi.org/10.1038/s41567-022-01770-1

Gender in publication patterns

• Mihaljević-Brandt, H., Santamaría, L., & Tullney, M. (2016). The effect of gender in the publication patterns in mathematics. PLoS One, 11(10), e0165367. https://doi.org/10.1371/journal.pone.0165367
• Day, A., Corbett, P., & Boyle, J. (2020). Is there a gender gap in chemical sciences scholarly communication? Chemical Science, 11(8), 2277-2301. https://doi.org/10.1039/C9SC04090K

Gender makeup of editors

• Liu, F., Holme, P., Chiesa, M., AlShebli, B., & Rahwan, T. (2023). Gender inequality and self-publication are common among academic editors. Nature Human Behaviour, 7(3), 353-364. https://doi.org/10.1038/s41562-022-01498-1

Status bias

• Huber, J., Inoua, S., Kerschbamer, R., König-Kersting, C., Palan, S., & Smith, V. L. (2022). Nobel and novice: Author prominence affects peer review. Proceedings of the National Academy of Sciences, 119(41), e2205779119. https://doi.org/doi:10.1073/pnas.2205779119
• Merton, R. K. (1968). The Matthew Effect in Science. Science, 159(3810), 56-63. https://doi.org/10.1126/science.159.3810.56
• Teixeira da Silva, J. A. (2021). The Matthew effect impacts science and academic publishing by preferentially amplifying citations, metrics and status. Scientometrics, 126(6), 5373-5377. https://doi.org/10.1007/s11192-021-03967-2
• Tomkins, A., Zhang, M., & Heavlin, W. D. (2017). Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48), 12708-12713. https://doi.org/10.1073/pnas.1707323114

Geographic bias

• Kahalon, R., Klein, V., Ksenofontov, I., Ullrich, J., & Wright, S. C. (2022). Mentioning the Sample’s Country in the Article’s Title Leads to Bias in Research Evaluation. Social Psychological and Personality Science, 13(2), 352-361. https://doi.org/10.1177/19485506211024036
• Skopec, M., Issa, H., Reed, J., & Harris, M. (2020). The role of geographic bias in knowledge diffusion: a systematic review and narrative synthesis. Research Integrity and Peer Review, 5(1), 2. https://doi.org/10.1186/s41073-019-0088-0
• Tijssen, R. (2007). Africa’s contribution to the worldwide research literature: New analytical perspectives, trends, and performance indicators. Scientometrics, 71(2), 303-327. https://doi.org/10.1007/s11192-007-1658-3