Preprints and Armchair Epidemiologists
by Anh-Khoi Trinh, Science & Policy Exchange
COVID-19 has been on everyone’s minds recently, especially within the scientific community. The recent weeks have seen a surge in preprint publications about the coronavirus on repositories such as arXiv, medRxiv, and bioRxiv, following an exponential growth eerily similar to the spread of the virus itself. So much so that designated quick links were created to quickly access these publications.
On the one hand, this highlights the incredible multidisciplinary, global, and collaborative effort of the scientific community to address this exceptional crisis. On the other, these unvetted publications have also exposed the inadequacies of open-science and the preprint culture.
This was especially well-described by u/VeryLittle in this Reddit post where they pointed out that many of the recent preprints were lacking in scientific rigor. This is, of course, not a new observation. What is new however is that a large portion of these preprints is written by armchair epidemiologists with no background in epidemiology.
“Why would you ask an epidemiology question to a physicist?” I thought. Unfortunately, I cannot answer this question. I can only tell you that people are asking physicists, mathematicians, and computer scientists, and that, to my bewilderment, such a paper had been picked up by a news outlet in another country where it then percolated through social media across their local communities.
This preprint in question has already been criticized at length and debunked on social media, and therefore I will refrain from identifying it. However, u/VeryLittle’s arguments bear repeating.
- Any publication outside of your field of expertise will add little value to your professional portfolio.
- The lay public can easily latch onto your findings without understanding the subtleties of the preprint culture or the assumptions in your model.
- Any model that you could cook up would be a first-order approximation in comparison to models of actual epidemiologists.
- This isn’t an exotic material or a new subatomic particle; people’s lives are at stake and the consequences of your mistakes are severe:
- In the best-case scenario, you could sully your own reputation in the eyes of the public and that of the academic community.
- In the worst-case, your article could influence the public discourse to act against actual evidence-based decision-making.
I tried reading some of these papers myself but couldn’t always assess the validity of their claims, and that’s precisely the point. This is not the time for any physicist, mathematician or computer scientist to try their hands at epidemiology by publishing their results as preprints. Yes, we all use math. Yes, we all know how to code and fit data. Yes, we’ve seen an exponential curve before. And yes, we know how to solve PDEs and hence how to program SIR curves. Some of us might know how to implement machine learning algorithms. Yet, most of us are unable to accurately evaluate the parameters of these models, nor are we properly trained to interpret the results. Above all, most of us aren’t trained to provide policy recommendations to governing institutions.
To be clear, mathematical epidemiology is a profession. Popular math Youtubers Numberphile and 3Blue1Brown have done an excellent job of describing the underlying mathematics of infectious disease outbreaks. However, if you compare their models to those described by Dr. Robin Thomson in his online public forum at the Oxford Mathematical Institute, it is clear that there are many subtleties that only expert epidemiologists are equipped to properly consider. Let us not forget that these scientific findings serve to inform expeditious decision-making in a time of crisis; overlooking such crucial subtleties could lead policy-makers to adopt dreadful policies.
The current publication climate places the status of preprints in a precarious situation. Preprints have allowed for anyone with internet access to obtain to-be-peer-reviewed scientific papers without being paywalled by scientific journals. They’ve further enabled scientists to receive faster feedback from their peers before embarking upon the long-drawn-out road of academic publishing. In recent weeks, however, we have witnessed how such a system allows for poor academic integrity to dilute the voices of experts. There may reach a point where these preprint repositories are forced to take action, by elevating their editorial standards or introducing an extended review process, thereby negating the aforementioned benefits. However, a much simpler and sustainable solution is for the academic community to elevate its publishing standards and to acknowledge the limitations of its expertise — this is a small price to pay for a democratized scientific publishing culture.
In the age of preprints, we must recognize that non-experts may read our papers. Therefore, to those using preprint services and writing about COVID-19, here are a few tips:
- Include a lay summary (like this)
- Clearly delineate your assumptions
- Highlight the limitations of your model
- Explicitly state the impact of your study to both the scientific community, and to potential stakeholders.
Although this article has so far targeted scientists with a background in quantitative sciences, this PSA really extends to anyone that has the potential to disseminate inaccurate scientific evidence. This includes bioRxiv and medRxiv users, physicians, journalists, influencers, and every other person who uses social media: (1) don’t be an armchair expert (2) check your sources before sharing.
According to our epidemiological study, the number of people saying they know what should have been done against the pandemic skyrocketed!
While this article was initially motivated by u/VeryLittle’s Reddit post, I must disagree with its title. I believe that not publishing your preprints is in fact the least you can do. The best thing you can do is to promote the voices of actual experts and to support your local communities. Below is a shortlist of reliable Canadian resources compiled by Science & Policy Exchange. This list includes academic & governmental institutions, scientists, journalists and science communicators. The full list can be found here, and a Twitter thread contextualizing some of these players can be found here.
Theresa Tham (Government), Patty Hajdu (Government), Mona Nemer (Government), Rémi Quirion (Government), David Fisman (Researcher), Tara Moriarty (Researcher), Jennifer Robson (Researcher), Andre Picard (Journalist), Valérie Borde (Journalist), Curtis Kim (Data scientist), Samantha Yammine (Science communicator), Christopher Labos (Physician & Science communicator).
Despite clamors from the scientific community imploring governments to take action on climate change, plastic pollution, data privacy, and many other issues, policy-makers have largely disregarded our concerns in favor of economic prosperity and capital gain. We are now in a position to show the world how science can serve to the betterment of society. Let’s make the best out of this.
A special thank you to the volunteer scientists at Science & Policy Exchange for compiling the list of Canadian resources, and Maia Dakessian, Irene Kaloyannis, and Jessica Bou Nassar for providing editorial feedback.
@AnhKhoiTrinh is a doctoral candidate in physics at McGill University and VP Internal of Science & Policy Exchange.
