Viagra, Melanoma And The Problem With Big Data
This week saw the climax (or, for those who like to demonize the global pharmaceutical industry, perhaps the anti-climax) of a year-long saga linking an increased risk of melanoma with drugs such as sildenafil (better known as Viagra) used to treat erectile dysfunction.
The story began in 2014 when a team of Boston scientists observed higher rates of melanoma among men taking Viagra. Both the original paper in JAMA Internal Medicine, and the accompanying editorial, encouraged caution: the observational nature of the study design made it impossible to know whether the drugs increased the risk of melanoma (a causal effect) or were merely associated with it (a correlation).
But expressing an abundance of caution about the interpretation of the data in the primary scientific literature unsurprisingly did not prevent a great deal of concern among users of the drugs in question. While doctors couldn’t conclude the link was causal, equally they could not reassure patients that the association was benign either. Some men undoubtedly stopped taking the drug just in case the risks were real.
This is not the first time patients have been left with such decisions, and it most certainly will not be the last. Vast datasets collected by US insurance companies have already thrown up a number of such “candidate associations” between individual drugs and particular side-effects (such as this study mining hundreds of thousands of insurance claim records to reveal an association between use of SSRIs, such as Prozac, and bone fractures). And the developed world is putting in place ever more sophisticated data collection processes embedded in the healthcare system (such as centralized electronic medical records).
Because of the huge numbers of patients in these datasets, it is possible to see even tiny associations with statistical confidence (associations that would have disappeared into the noise in even the largest pre-approval clinical trials). Many of these datasets already dwarf the 20,000 or so patient records that led to the original observation linking Viagra and melanoma. With a million patients, even a 1% increase in risk could be reliably identified.
What happened next was a case-study in epidemiological sleuthing: Stacy Loeb and her colleagues at New York University looked carefully at another dataset, this time from Sweden. Interestingly, the same association was observed a second time (effectively eliminating the possibility that the original observation was a statistical fluke). But looking more closely, they were able to conclude that the association was more than likely correlation and not causation.
Most tellingly, there was no link between the extent of exposure to the drugs (either as a result of using an erectile dysfunction drug with a longer half-life or by simply using the drugs more often) and the risk of melanoma. If the drug actually caused melanoma, then you would expect to see a typical “dose-response” relationship (the more you took, the greater your risk).
In their paper, published this week in JAMA, Loeb concluded that the new data “raises questions about whether this association is causal.”
For those Viagra users who don’t make their living reading scientific and medical literature (surely the vast majority), that hardly sounds like a clean bill of health for the drug. But that’s simply because of the impossibility of proving a negative. In reality, given Loeb’s analysis, the chances that drugs to treat erectile dysfunction really do promote melanoma is very small indeed. But proof of that absolute safety will never be forthcoming.
This story tells us more about the power (and danger) of so-called “big data” in healthcare than it does about the safety of Viagra.
Treating erectile dysfunction undoubtedly improves the lives of those unfortunate enough to suffer it. Equally, though, you can live without such treatment. But other “associations” emerging from the vast, real-world datasets relate to drugs that confer life-saving benefits. The conservative option – halting treatment – while the association is investigated further is rarely a viable option. Doctors representatives implore patients not to stop their medication without proper consultation.
The effects on the companies who make the drugs can also be substantial, if less emotive. If the threat of patent litigation (as wielded most recently by hedge fund boss Kyle Bass) can wipe billions from corporate valuations, the possibility of a previously unanticipated risk associated with a blockbuster drug could be catastrophic – even if the observed association was later shown likely to be correlation rather than cause.
Phrases from every day life like “there is no smoke without fire” ensure that the public perception of a drug can be tarnished by such claims, whatever the merits of the epidemiological analysis underpinning them. The impossibility of subsequently proving the association is NOT causal only worsens the impact of the original observation. We can quickly find ourselves in the situation where the simple observation itself is robust, but what it actually means remains shrouded in complexity.
Its not hard to imagine, a decade from now, a parade of mass tort litigations stemming from these real-world observations, hinging on jury deliberations about what constitutes proof of causation. Allowing ourselves to reach that position could deal a lethal blow to healthcare innovation.
The solution does not lie in limiting the collection of big data in healthcare. Quite the reverse: the ability to spot the unexpected signal without a pre-existing hypothesis is tremendously powerful. But what we need is a better framework for interpreting the observations that undoubtedly will emerge, ever more frequently.
For a start, querying broad datasets without an a priori hypothesis will always find apparently significant associations, simply as a factor of the number of possible associations being queried. This is called “over-fitting” by statisticians, and plagues big data enterprises in any field. The real signals will be genuinely indistinguishable from a larger number of chance associations.
Over-fitting is easy to solve by replication in a different dataset or prospective collection of new data. If the observation keeps turning up, as it did when Loeb looked in the Swedish Viagra users, its probably a real signal. But establishing causality is a much harder prospect.
An early attempt to set out “rules” for assessing the likelihood of causality, was made by Sir Austen Bradford Hill in 1965, long before the advent of “big data”. Although these criteria have frequently been refined, including by DrugBaron, the central principle remains: you cannot extract evidence for causality from observational data, no matter what you do. You can only tilt the balance of probabilities one way or the other through careful analysis. Indeed, the dose-response relationship Loeb used to support her conclusions is one of Bradford Hill’s original criteria from his seminal paper fifty years ago.
The only proof of causality can come from prospective intervention studies. In the case of drug side-effects that would mean more trials, larger than any conducted prior to approval, and of greater concern some people would have to deliberately take more or less of the drug under the instruction of the experimenter, risking more side-effects if the association was causal, or loss of the benefit of the drug (or indeed both).
Even if this were ethically desirable, it would be hugely costly and in any case unlikely to deliver a clear answer for many years, even decades, after the original association were described. In the intervening period, patients would still be left with the unenviable decision of what to do with the knowledge of the observed association.
Perhaps a more viable solution would be to set up some binding guidelines in advance as to the evidence-base required to report an association extracted from these large datasets. Had the original report of the association between Viagra use and melanoma (or, indeed, the report linking SSRIs and bone fracture) been obliged to include replication from a second dataset and an analysis of the dose-response relationship prior to publication then a lot of unnecessary worry could have been avoided. And for drugs with life-saving benefits, the prospect of real harm to patients would be largely avoided.
Its worth noting that the behavior of the scientists, and indeed the majority of journalists covering the Viagra story, has been very responsible. The risks were kept clearly in context. But we know from episodes such as the establishment of an “anti-vaxxer” movement on the basis of no real data, that such responsibility is not always evident from either the originating scientists or the journalists who cover their work.
Before the flow of hard-to-interpret associations observed in big healthcare datasets turns from a trickle into an avalanche, as it surely will, the scientific community and drug regulators need to get their house in order and decide on a framework for dealing with them. It would be far more powerful to set the ground rules for publication and interpretation ahead of the curve.
As my mother always said to me, “a little knowledge is a dangerous thing”. She may not have been thinking about observational associations derived from big datasets, but her admonition applies here, perhaps more than anywhere. A lot of data can still yield only a little knowledge.
Source: Forbes
