On 2025-04-10 20:11:03, user Jeffrey_S_Morris wrote:
This study's conclusion of -27% negative effectiveness does not seem to be supported by the study, given they did not account for testing bias, which happens to also be 27%, with vaccinated testing on average a 27% higher rate than unvaccinated.
To their credit, the authors acknowledged this in the following plot:<br />
https://uploads.disquscdn.c... <br />
Here it can be seen in my replotting of their Figure 1a scatterplot on the log y axis (after extracting the data by applying AI tool to their scatterplot image), with the 27% increase being the (geometric) mean testing rate (vaccinated/unvaccinated) over the days they plotted.
https://uploads.disquscdn.c...
Incidentally, taking simple means (or fitting linear regressions) for a sample of ratios is not good statistical practice since the <1 and >1 parts are asymmetric, so instead the geometric mean (averaging on the log scale) should be used. For example, if one day is 4x higher for vaccine and one day 4x lower, they should average to be equivalent. The average on the raw scale (4 + 0.25)/2 = 2.125 would imply a mean 2.125x increase, which is incorrect, while a geometric mean (averaging on the log scale and then exponentiating) would get the correct result. 2^{log_2(4)+log_2(0.25)} = 2^(2 + -2) = 2^0 = 1. That is why I used geometric mean in the plot above and plot in the log scale, and think the authors should do the same in their paper.
While they acknolwedge the increased testing rate, the text of the paper dismisses it as a potential source of bias by claiming the test positivity rate is equivalent in vaccinated and unvaccinated. I agree with their logic that if test positivity were identical in vaccinated and unvaccinated, then the 27% higher testing rate could simply be a result of a 27% higher infection rate, and not from testing bias.
However, the analysis they present to support this assumption is not justified and seems flawed. They perform a linear regression of the ratio of testing positivity (vaccinated/unvaccinated) by day over time, and because the confidence bands intersect zero they conclude the test positivity is no different between vaccinated and unvaccinated, and thus the difference in testing rate is not a bias, but from the negative effectiveness that they conclude is true.<br />
https://uploads.disquscdn.c... <br />
However, this analysis is problematic for numerous reasons:<br />
1. It is not clear why a regression over time should be done to answer this question, and not clear why one would assume any time trend is strictly linear. It would make much better sense to compute a (geometric) mean over time, or if wanting to model time trends to use a smooth nonparametric function.<br />
2. Computing means or modeling time trends on ratios should not be done on the raw scale, but the log scale, for the reasons discussed above.
Plotting these numbers on the log scale (again, after using AI tool to extract it from their scatterplot image in the paper), I computed the geometric mean test positivity, and find it to be 0.80, meaning the "average" test positivity over time is 20% lower in vaccinated than unvaccinated, certainly not the same.
https://uploads.disquscdn.c... <br />
This lower test positivity is obscured in their original plot on the raw scale, since the ratios <1 got compressed and ratios>1 expanded.
If you have a situation with vaccinated having 1.27x the testing rate and 0.80x the test positivity, this would correspond to an infection rate that is 1.27 x 0.80 = 1.016x higher infection rate. This would correspond no difference in infection rate, certainly not a 27% increased infection rate in vaccinated.
While not a formal analysis, this demonstrates that vaccinated having a 27% higher testing rate along with a 20% lower test positivity rate could result in a 27% higher rate of confirmed flu infections even if the infection rate was equivalent between vaccinated and unvaccinated.
In that case. the 1.27x increased testing rate would be a testing bias that produces a spurious 1.27x confirmed infection rate even if the infection rate were not higher in the vaccinated.
Based on this, one cannot tell from the study whether the 1.27x increased rate of confirmed flu infections is from negative effectiveness (as claimed), or from the testing bias (which is not adjusted for in the analysis).
The authors cannot rule out the possibility that their results are caused by the testing bias, which is not accounted for in their analysis.
Thus, I don't think the conclusion of -27% VE is valid.
At most, they could say there is no evidence of any vaccine effectiveness vs. infection, but cannot conclude a significant negative effectiveness because of failure to account for the testing bias.
Of course, there are designs to adjust for this testing bias -- test negative designs -- but the authors eschew this design, seemingly because it gives odds ratios rather than relative rates which they express concern that they are not as intuitive to grasp.
To me, that seems like a relatively minor issue relative to testing bias of sufficient magnitude to drive spurious results.
If I were reviewing this paper, I'd require them to adjust for the testing bias, and ideally perform a test negative design, even if considered a secondary analysis.
Of course test negative designs have their own limitations and potential biases, but at least considering it as a secondary analysis would be useful to see if they obtain equivalent results using that design and, if not, should raise questions on whether they should boldly conclude negative effectiveness in this study, or instead more carefully conclude a lack of evidence of vaccine effectiveness in their cohort.
These concerns are also summarized in an http://x.com thread