Updated to correct typo in copy
A personal, non-peer-reviewed study claiming to reveal the global COVID-19 vaccine rollout has increased deaths and cases of the disease has been cited in some social media posts as comprehensive analysis. However, Reuters presented the piece to data experts, who highlighted several major limitations.
One tweet with over 7,000 likes described the analysis as “exhaustive” while Infowars, a conspiracy-theory website and channel, highlighted it in a video segment (here) .
He compared data from a control group of four countries with low vaccination rates – Burkina Faso, Chad, the Democratic Republic of Congo (DRC) and South Sudan – to others where vaccine rollouts were far more widespread.
These control group countries “best represent a ‘natural’ progression of the virus with limited vaccine intervention,” he said.
Beattie claimed his research indicated that the global COVID-19 vaccine rollout had “a strong and significant propensity” to increase deaths and cases of the disease. He added that this finding “should be highly worrisome for policy makers”.
Beattie later told Reuters in an email he had written the paper due to his “personal interest in this topic” and had “no assistance or affiliation from my university”.
“This work is my honest attempt to understand broadly what the effect of vaccine administration is as an instituted public policy and whether it has helped countries lower cases or deaths from COVID,” he said.
“I have published this as a preprint on the ResearchGate server in order to show my results, debate them, and hopefully receive constructive criticism on where this methodology can be improved.”
Reuters presented the piece to three experts, all of whom pointed to several issues with the study’s methodology and conclusion, and highlighted how it contradicted results of other widely accepted research.
CRITIQUE FROM EXPERTS
“Overall, I have very low confidence in the results of this analysis,” said Elizabeth Williamson, an associate professor in the Department of Medical Statistics at the London School of Hygiene and Tropical Medicine (LSHTM) – adding that the paper took “a very naive approach to an incredibly complex question without discussing the plausibility of the very strong assumptions being made.”
“The study chose to use four countries in Africa as the ‘comparison’, i.e., these countries were used to represent what would have happened in other countries (e.g., Japan, U.S.) had they not introduced the vaccine.
“Whether the synthetic comparison approach used could ever give valid results for COVID vaccine efficacy is not certain in this approach, like all causal methods, relies on untestable assumptions.”
She added: “What I am certain about is that using these four countries as the comparison with every other country is very unlikely to give a sensible answer.”
Agreeing with the critique, Williamson’s colleague, Professor Linda Sharples, the head of LSHTM’s Department of Medical Statistics, said the paper had drawn together “aggregate data from a wide range of very different countries”.
She told Reuters: “First, these countries vary widely in the age distribution of their citizens, geographic, economic and other population characteristics, not to mention their different strategies for testing for COVID and different vaccination programs.
“Combining data in this case is fraught with difficulty and likely to be misleading.”
Also referring to untestable assumptions, she added: “Obviously, we don’t know what would have happened if we had done something different to what we actually did. If we have a lot of detailed information about each individual, then we can lessen the influence of these very strong assumptions, but we can’t make their effect disappear completely.”
Reuters also spoke to John Whittaker, the director of the MRC Biostatistics Unit (BSU) and professor of Biostatistics at the University of Cambridge, who echoed the comments from Williamson and Sharples.
“We need these controls to be ‘exchangeable’ with the [vaccine] intervention period/countries, i.e., to believe they are similar enough to give reasonable predictions of what would have happened without intervention (the counterfactual). I don’t believe either assumption here.”
Commenting specifically on the choice of the four control countries, Whittaker said this “amounts to saying that vaccines were harmful in France because Chad reported very low rates of deaths and cases over the entire pandemic period”.
All three experts told Reuters that conclusions would be better drawn from looking at evidence from randomised trials, as well as other data and analysis.
“We have a lot of data now from randomised trials and associated studies that means we don’t have to rely on these methods, and these trials give us a different, more evidence-based picture,” Sharples said, summing up the consensus.
“From these, I have confidence in the effectiveness of vaccination.”
Beattie told Reuters he understood the limitations of the control countries he had chosen and acknowledged that “an astute commentator” had highlighted that Burkina Faso, Chad, the DRC and South Sudan had “low average-aged populations and are thereby less affected by COVID-19 and thus may not make good controls” – which, he conceded, “is possible”.
“The countries chosen for this report ultimately provided the lowest standards of deviation and the highest predictive confidence with the data available,” Beattie concluded.
In response, Whittaker said: “There are many, many reasons why these countries are different to, for example, the UK, with its national centralised testing system, but reporting differences are certainly one and age structure another – those four countries have reported very low rates through the pandemic.
“This just isn’t a valid comparator group, especially as it seems to have been selected because it had low reported COVID rates.”
Williamson added: “The argument appears to be that any other set of countries from these data would have been a worse choice of comparison; this is very different from claiming that these four countries provide reasonable comparison.
“I suspect there is no control population (group of countries) within these data that would fulfil the necessary assumptions. In a situation where it is not possible to perform a valid causal comparison, the correct choice is not to do one at all.”
Misleading. The conclusions in the non-peer-reviewed paper written in a personal capacity have been disputed by medical data experts.
This article was produced by the Reuters Fact Check team. Read more about our fact-checking work here .
Our Standards: The Thomson Reuters Trust Principles.