Blinded Wheat Challenge
The beauty of the scientific method is that it offers us effective tools to minimize this kind of bias. This is probably its main advantage over more subjective forms of inquiry**. One of the most effective tools in the scientific method's toolbox is a control. This is a measurement that's used to establish a baseline for comparison with the intervention, which is what you're interested in. Without a control measurement, the intervention measurement is typically meaningless. For example, if we give 100 people pills that cure belly button lint, we have to give a different group placebo (sugar) pills. Only the comparison between drug and placebo groups can tell us if the drug worked, because maybe the changing seasons, regular doctor's visits, or having your belly button examined once a week affects the likelihood of lint.
Another tool is called blinding. This is where the patient, and often the doctor and investigators, don't know which pills are placebo and which are drug. This minimizes bias on the part of the patient, and sometimes the doctor and investigators. If the patient knew he were receiving drug rather than placebo, that could influence the outcome. Likewise, investigators who aren't blinded while they're collecting data can unconsciously (or consciously) influence it.
Back to diet. I want to know if I react to wheat. I've been gluten-free for about a month. But if I eat a slice of bread, how can I be sure I'm not experiencing symptoms because I think I should? How about blinding and a non-gluten control?
Procedure for a Blinded Wheat Challenge
1. Find a friend who can help you.
2. Buy a loaf of wheat bread and a loaf of gluten-free bread.
3. Have your friend choose one of the loaves without telling you which he/she chose.
4. Have your friend take 1-3 slices, blend them with water in a blender until smooth. This is to eliminate differences in consistency that could allow you to determine what you're eating. Don't watch your friend do this-- you might recognize the loaf.
5. Pinch your nose and drink the "bread smoothie" (yum!). This is so that you can't identify the bread by taste. Rinse your mouth with water before releasing your nose. Record how you feel in the next few hours and days.
6. Wait a week. This is called a "washout period". Repeat the experiment with the second loaf, attempting to keep everything else about the experiment as similar as possible.
7. Compare how you felt each time. Have your friend "unblind" you by telling you which bread you ate on each day. If you experienced symptoms during the wheat challenge but not the control challenge, you may be sensitive to wheat.
If you want to take this to the next level of scientific rigor, repeat the procedure several times to see if the result is consistent. The larger the effect, the fewer times you need to repeat it to be confident in the result.
* Although it can also be disastrous. People who get into the most trouble are "extreme thinkers" who have a tendency to take an idea too far, e.g., avoid all animal foods, avoid all carbohydrate, avoid all fat, run two marathons a week, etc.
** More subjective forms of inquiry have their own advantages.
Blinded Wheat Challenge
The beauty of the scientific method is that it offers us effective tools to minimize this kind of bias. This is probably its main advantage over more subjective forms of inquiry**. One of the most effective tools in the scientific method's toolbox is a control. This is a measurement that's used to establish a baseline for comparison with the intervention, which is what you're interested in. Without a control measurement, the intervention measurement is typically meaningless. For example, if we give 100 people pills that cure belly button lint, we have to give a different group placebo (sugar) pills. Only the comparison between drug and placebo groups can tell us if the drug worked, because maybe the changing seasons, regular doctor's visits, or having your belly button examined once a week affects the likelihood of lint.
Another tool is called blinding. This is where the patient, and often the doctor and investigators, don't know which pills are placebo and which are drug. This minimizes bias on the part of the patient, and sometimes the doctor and investigators. If the patient knew he were receiving drug rather than placebo, that could influence the outcome. Likewise, investigators who aren't blinded while they're collecting data can unconsciously (or consciously) influence it.
Back to diet. I want to know if I react to wheat. I've been gluten-free for about a month. But if I eat a slice of bread, how can I be sure I'm not experiencing symptoms because I think I should? How about blinding and a non-gluten control?
Procedure for a Blinded Wheat Challenge
1. Find a friend who can help you.
2. Buy a loaf of wheat bread and a loaf of gluten-free bread.
3. Have your friend choose one of the loaves without telling you which he/she chose.
4. Have your friend take 1-3 slices, blend them with water in a blender until smooth. This is to eliminate differences in consistency that could allow you to determine what you're eating. Don't watch your friend do this-- you might recognize the loaf.
5. Pinch your nose and drink the "bread smoothie" (yum!). This is so that you can't identify the bread by taste. Rinse your mouth with water before releasing your nose. Record how you feel in the next few hours and days.
6. Wait a week. This is called a "washout period". Repeat the experiment with the second loaf, attempting to keep everything else about the experiment as similar as possible.
7. Compare how you felt each time. Have your friend "unblind" you by telling you which bread you ate on each day. If you experienced symptoms during the wheat challenge but not the control challenge, you may be sensitive to wheat.
If you want to take this to the next level of scientific rigor, repeat the procedure several times to see if the result is consistent. The larger the effect, the fewer times you need to repeat it to be confident in the result.
* Although it can also be disastrous. People who get into the most trouble are "extreme thinkers" who have a tendency to take an idea too far, e.g., avoid all animal foods, avoid all carbohydrate, avoid all fat, run two marathons a week, etc.
** More subjective forms of inquiry have their own advantages.
St Jude Medical Settles Again
Another Legal Settlement by St Jude Medical
St. Jude Medical Inc. agreed to pay $16 million to settle a U.S. government probe of claims the company paid kickbacks to doctors who implanted its heart devices in patients.
The accord resolves a five-year investigation of St. Jude’s marketing practices for defibrillators and pacemakers.
This was not even the first recent settlement by this particular company:
In June, St. Jude agreed to pay the federal government $3.7 million to resolve a separate whistleblower case over claims that it made illegal payments to hospitals in Kentucky and Ohio that used the company’s heart devices.
Our relevant post about that previous settlement was here.
Kickbacks, Manipulated Research
The $16 million settlement stemmed from a case filed by Charles Donigian, a former St. Jude technician from St. Louis, who accused the company of using kickbacks to market products.
The kickbacks, which ranged as high as $2,000 per patient, came in the form of 'sham fees' for phony clinical-research studies on the devices, Donigian said in his suit.
There was slightly more detail in a report in TheHeart.org:
According to the DoJ, the company used postmarketing studies and a registry as vehicles to reward physicians participating in those studies to implant the devices. 'In each case, St Jude paid each participating physician a fee that ranged up to $2000 per patient,' a statement from US Attorney Carmen M Ortiz notes. 'The United States alleges that St Jude solicited physicians for the studies in order to retain their business and/or convert their business from a competitor's product.'
The statement also observes: 'Although St Jude collected data and information from participating physicians, it knowingly and intentionally used the studies and registry as a means of increasing its device sales by paying certain physicians to select St Jude pacemakers and ICDs for their patients.'
Summary
It all is getting so old, isn't it? Yet another pharmaceutical, biotechnology or device company settles a case alleging payments to physicians to induce them to use its products. The twist here is that the form of the payment allegedly also manipulated the clinical research process.
There have been other cases of manipulation or suppression of research about cardiac devices. For example, see our recent post about how Guidant, now a subsidiary of Boston Scientific, was put on probation for its actions in another such case.
The allegations amounted to serious external threats to physicians' professionalism: payments to promote use of health care products whether or not their implantation was in patients' best interests, and manipulation of clinical research that could have further corrupted the clinical evidence base, and presumably could have broken the trust of clinical research subjects. Left unsaid in the brief articles is how willingly the physicians gave in to these threats to their professionalism for the dollars involved.
Yet despite the apparently corrosive quality of the bad behavior, the penalties to the company were trivial. Per St Jude Medical's most recent financial profile, its yearly revenue was over $4.5 billion. Two settlements totaling less than $20 million amount to a relatively small cost of doing business. And note that St Jude Medical is not admitting it did anything wrong. Per TheHeart.org, its statement was:
We are pleased to have reached a settlement agreement with the DoJ that fully resolves the postmarket-study matter in Boston. The company maintains that its postmarket studies and registries are legitimate clinical studies designed to gather important scientific data, and St Jude Medical does not admit liability or wrongdoing by entering into this agreement. The company entered into a settlement agreement to avoid the potential costs and risks associated with litigation. This settlement brings the previously reported postmarket-study investigation to a close.
So it all gets so tiresome. The continuing march of legal settlements like this do provide evidence of how professionalism and evidence-based medical practice are under threat. However, as we have said ad infinitum, such settlements are likely to do nothing to alleviate that threat.
To repeat the conclusion of our last post about St Jude Medical, the usual sorts of legal settlements we have described do not seem to be an effective way to deter future unethical behavior. Even large fines (and the one described above would be peanuts to a large health care corporation) can be regarded just as a cost of doing business. Furthermore, the fine's impact may be diffused over the whole company, and ultimately comes out of the pockets of stockholders, employees, and customers alike. It provides no negative incentives for those who authorized, directed, or implemented the behavior in question. My refrain has been: we will not deter unethical behavior by health care organizations until the people who authorize, direct or implement bad behavior fear some meaningfully negative consequences. Real health care reform needs to make health care leaders accountable, and especially accountable for the bad behavior that helped make them rich.
Is the Executive Branch/HHS Trying to Put Lipstick on a Pig Regarding Health IT?
The data withholding is reminiscent of the data manipulation practiced in the pharmaceutical industry.
Legislative branch, take note:
By Joseph Conn
Modern Healthcare.com
Last week, Dr. David Blumenthal announced the results of two surveys funded by the Office of the National Coordinator for Health Information Technology on hospital and physician participation in the federal electronic health-record incentive programs.
But Dr. B left out a few numbers in going over the results of the survey of office-based physicians conducted by the National Center for Health Statistics. On request, the ONC and NCHS released those missing numbers.
Docs were asked: "Are there plans to apply for Medicare or Medicaid incentive payments for meaningful use of health IT?" Blumenthal reported on their answers in part, noting that 41.1% indicated "yes" and 14% said "no." However, a 44.9% plurality, which he did not mention, chose "uncertain whether we will apply." [Nothing to see here ... move along - ed.]
Those who answered "yes" were asked a follow-up question: "What year do you expect to apply for the meaningful-use payments?" Of the 41.1% of docs who indicated they were sure they would apply (a figure Blumenthal released), nearly one in five (19.7%, a number he didn’t mention) were unsure as to when. [Must be the Luddite faction - ed.]
Additionally, the NCHS asked physicians, "Which incentive payment do you plan to apply for?" Their responses to this question also weren’t mentioned by Blumenthal last week. Not surprisingly, given current eligibility thresholds, 65.1% selected Medicare and just 6.8% chose Medicaid, but again there was considerable uncertainty, with 28.2% choosing "unknown" or leaving the choice blank. [This finding suggests rank confusion to me more than anything else. In case nobody noticed, physicians are rather busy these days taking care of patients, and don't have time to find out what's in lengthy government documents - ed.]
Read the whole article.
It seems the executive is simply determined to push this technology onto a significantly skeptical physician community - skeptical of the incentives, of MU, of government intentions, and/or of the technology itself.
Of course, perhaps they feel that such data on a beneficent technology that will absolutely, positively benefit medicine and patients needs to be withheld - for the greater good, of course, which some people have to suffer to achieve (see the post "MAUDE and HIT Risks: Mother Mary, What in God's Name is Going on Here?"). ONC on healthcare IT in recent months:
http://healthcarereform.nejm.org/?p=3732&query=home
... The widespread use of electronic health records (EHRs) in the United States is inevitable. EHRs will improve caregivers’ decisions and patients’ outcomes. Once patients experience the benefits of this technology, they will demand nothing less from their providers. Hundreds of thousands of physicians have already seen these benefits in their clinical practice. (Also see http://hcrenewal.blogspot.com/2010/07/new-england-journal-on-meaningful-use.html).
and
http://www.massdevice.com/news/blumenthal-evidence-adverse-events-with-emrs-anecdotal-and-fragmented
... [Blumenthal's] department is confident that its mission remains unchanged in trying to push all healthcare establishments to adopt EMRs as a standard practice. "The [ONC] committee [investigating FDA reports of HIT endangement] said that nothing it had found would give them any pause that a policy of introducing EMR's could impede patient safety," he said. (Also see http://hcrenewal.blogspot.com/2010/05/david-blumenthal-on-health-it-safety.html).
A good number of physicians still appear to possess critical thinking skills.
Perhaps the new Congress (i.e., legislative branch) can benefit from physician-style skepticism about health IT as well.
-- SS
The Impact of eHealth on the Quality and Safety of Health Care: You Heard Much of This First On Healthcare Renewal ...
Black AD, Car J, Pagliari C, Anandan C, Cresswell K, et al. (2011) The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview. PLoS Med 8(1): e1000387. doi:10.1371/journal.pmed.1000387.
A text version can be seen at this link and a PDF downloaded at this link.
I will reproduce only the abstract here, and then make a few comments (over and above the hundreds of posts on these issues I've authored at this blog since 2004):
The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview.
Ashly D. Black1, Josip Car1, Claudia Pagliari2, Chantelle Anandan2, Kathrin Cresswell2, Tomislav Bokun1, Brian McKinstry2, Rob Procter3, Azeem Majeed4, Aziz Sheikh2*
1 eHealth Unit, Department of Primary Care and Public Health, Imperial College London, London, United Kingdom, 2 eHealth Research Group, Centre for Population Health Sciences, The University of Edinburgh, Edinburgh, United Kingdom, 3 National Centre for e-Social Science, University of Manchester, Manchester, United Kingdom, 4 Department of Primary Care and Public Health, Imperial College London, London, United Kingdom
Abstract
Background
There is considerable international interest in exploiting the potential of digital solutions to enhance the quality and safety of health care. Implementations of transformative eHealth technologies are underway globally, often at very considerable cost. [I believe it would be more accurate to write "always at considerable cost, with generous profits being made by the IT merchants, implementers and consultants"- ed.] In order to assess the impact of eHealth solutions on the quality and safety of health care, and to inform policy decisions on eHealth deployments, we undertook a systematic review of systematic reviews assessing the effectiveness and consequences of various eHealth technologies on the quality and safety of care.
Methods and Findings
We developed novel search strategies, conceptual maps of health care quality, safety, and eHealth interventions, and then systematically identified, scrutinised, and synthesised the systematic review literature. Major biomedical databases were searched to identify systematic reviews published between 1997 and 2010. Related theoretical, methodological, and technical material was also reviewed. We identified 53 systematic reviews that focused on assessing the impact of eHealth interventions on the quality and/or safety of health care and 55 supplementary systematic reviews providing relevant supportive information. This systematic review literature was found to be generally of substandard quality with regards to methodology, reporting, and utility. We thematically categorised eHealth technologies into three main areas: (1) storing, managing, and transmission of data; (2) clinical decision support; and (3) facilitating care from a distance. We found that despite support from policymakers, there was relatively little empirical evidence to substantiate many of the claims made in relation to these technologies. Whether the success of those relatively few solutions identified to improve quality and safety would continue if these were deployed beyond the contexts in which they were originally developed, has yet to be established. Importantly, best practice guidelines in effective development and deployment strategies are lacking.
Conclusions
There is a large gap between the postulated and empirically demonstrated benefits of eHealth technologies. [You've heard that here before - ed.]
In addition, there is a lack of robust research on the risks [I believe it would be more accurate to say there's been a suppression of research and/or of publicity on the risks - ed.] of implementing these technologies and their cost-effectiveness has yet to be demonstrated [in other words, healthcare IT is an experimental technology. You've heard that here before, too - ed.], despite being frequently promoted by policymakers and “techno-enthusiasts” as if this was a given.
[In other words, there's an irrational - or contrived - exuberance. You've heard that here before, too - ed.]
In the light of the paucity of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, it is vital that future eHealth technologies are evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle. [You've heard that here before, too - ed.]
Such evaluation should be characterised by careful attention to socio-technical factors to maximise the likelihood of successful implementation and adoption.
[You've heard that here before, too, as well as at my site "Contemporary Issues in Medical Informatics: Common Examples of Healthcare Information Technology Difficulties", as a matter of fact previously entitled "Sociotechnologic Issues in Medical Informatics...", started in 1999 - ed.]
I can add these additional comments:
- The findings of this study were of no surprise to me or likely to readers of this blog. I've been writing on these issues since the 1990's, and on this blog since 2004.
- In the main article itself it is stated "... there remains a disparity between the evidence-based principles that underpin health care generally and the political, pragmatic, and commercial drivers of decision making in the commissioning of eHealth tools and services." In other words, the rigor of medical science itself is lacking in healthcare IT. I've written that repeatedly here and elsewhere. I ask again: how can IT "transform" ot "revolutionize" healthcare when the culture of IT itself is in need of reform?
- Also stated in the article's body was this: "Risks largely went ignored apart from anecdotal evidence of time-costs associated with recording of data due to both end-user skill and the inflexibility of structured data, increased costs of EHRs, and a decrease in patient-centeredness within the consultation." See my prior post "MAUDE and HIT Risks: Mother Mary, What in God's Name is Going on Here?" for what I find as deeply concerning ("hair-raising") revelations regarding risk that you should think about next time you find yourself in a computerized hospital or doctor's office.
- Also in the article body: "Our major finding from reviewing the literature is that empirical evidence for the beneficial impact of most eHealth technologies is often absent or, at best, only modest. While absence of evidence does not equate with evidence of ineffectiveness, reports of negative consequences indicate that evaluation of risks – anticipated or otherwise – is essential. Clinical informatics should be no less concerned with safety and efficacy than the pharmaceutical industry. [Having worked in both sectors, hospital and pharma , you've read that here before - ed.] Given this, there is a pressing need for further evaluations before substantial sums of money are committed [as you've read here in numerous posts, that's BEFORE, NOT DURING OR AFTER - ed.] to large-scale national deployments under the auspices of improving health care quality and/or safety." Are governments listening, I ask?
- Study methodology and source material limitations were noted. However, also noted was this: "Our greatest cause for concern was the weakness of the evidence base itself. A strong evidence base is characterised by quantity, quality, and consistency. Unfortunately, we found that the eHealth evidence base falls short in all of these respects. In addition, relative to the number of eHealth implementations that have taken place, the number of evaluations is comparatively small." In my view, national re-engineering programs involving a large portion of the economy and a critical societal function should not be undertaken when the evidence base is weak and the number of evaluations is small. Further, national leaders should not be making statements of certainty such as made by the HHS ONC office as I wrote about at "Science or Politics? The New England Journal and The 'Meaningful Use' Regulation for Electronic Health Records" and at "Blumenthal on health IT safety: nothing to see here, move along."
- This article might be compared to the Dec. 2009 Milbank Quarterly article "Tensions and Paradoxes in Electronic Patient Record Research: A Systematic Literature Review Using the Meta-narrative Method" by Greenhalgh, Potts, Wong, Bark and Swinglehurst at University College London. I wrote about that article at this link. It was also a meta-analysis; its major conclusion was that "the tensions and paradoxes revealed in this study extend and challenge previous reviews and suggest that the evidence base for some EPR programs is more limited than is often assumed."
Read the entire PLoS article at the links above.