Friday, February 05, 2010

Efficacy of Interactive Metronome treatment study (Cosper, 2009): An example of a "non-example" of IM efficacy research

As a member of the Interactive Metronome Scientific Advisory Board I am often asked to review published research studies that have investigated the IM method.

Recently, a investigation of the effectiveness IM (which is a mental timing-based intervnetion) with children with attention deficit disorder and developmental coordination disorders was published in the International Journal of Rehabilitation Research. The citation and abstract for the article are reproduced below.

  • Cosper, S., Lee, G., Peters, S & Bishop, E. (2009).  Interactive Metronome training in children with attention deficit and developmental coordination disorders, Internal Journal of Rehabilitation Research, 32 (4), 331-336.  (click here to view)

Abstract: The objective of this study was to examine the efficacy of Interactive Metronome (Interactive Metronome, Sunrise, Florida, USA) training in a group of children with mixed attentional and motor coordination disorders to further explore which subcomponents of attentional control and motor functioning the training influences. Twelve children who had been diagnosed with attention deficit hyperactivity disorder, in conjunction with either developmental coordination disorder (n=10) or pervasive developmental disorder (n=2), underwent 15 1-h sessions of Interactive Metronome training over a 15-week period. Each child was assessed before and after the treatment using measures of attention, coordination, and motor control to determine the efficacy of training on these cognitive and behavioral realms. As a group, the children made significant improvements in complex visual choice reaction time and visuomotor control after the training. There were, however, no significant changes in sustained attention or inhibitory control over inappropriate motor responses after treatment. These results suggest Interactive Metronome training may address deficits in visuomotor control and speed, but appears to have little effect on sustained attention or motor inhibition.

After reading thie manuscript, I decided to make this formal blog post in an effort to educate readers on a few critical issues related to intervention research. After a thorough read of the study, and despite the fact that it reported positive findings in the area of motor coordination improvement (but not in attention), it is my professional opinion that there are enough major methodological flaws which make the study an invalid study of the effectiveness of the IM treatment program.

Below are the major concerns I have about the study. I will only focus on my major concerns and will not highlight other minor concerns I noted during my review of manuscript.

Treatment integrity/fidelity/validity:  The most serious (fatal) flaw  is the lack of treatment integrity or validity. As described in the methods section, the treatment consisted of one hour sessions (per week) of IM treatment over a period of 15 weeks. To readers unfamiliar with the IM program, the standard treatment protocol is for three hourly sessions per week.  The use of only one hourly session per week (versus the recommended three hourly sessions per week) is a clear example of poor treatment integrity as defined below.

Treatment integrity (also known as treatment fidelity) refers to the degree to which intervention is implemented as intended. Interpretations of the obtained results require some assurance that treatment was carried our as it was designed or, in other words, that treatment was implemented with high levels of integrity. Compromised integrity has serious implications for inferences drawn about the relationship between treatment and outcome. Indeed, the obtained results are related not to the intended intervention (i.e., treatment as designed) but to the implemented intervention (i.e., treatment as delivered by treatment agents). [Click here for more information]

This violation of the IM treatment protocol would be analogous to the following:

  • Being prescribed three weekly sessions of physical therapy for back pain but only doing one session a week.
  • Being prescribed 300 mg/day of some form of medication to treat a condition, but only taking 100 mg/day.
  • Having a reading disabled student who is recommended to receive three hourly sessions of reading-focused special education services a week only receive one hourly session per week.

The bottom line is that this this study should not be considered a valid study of the efficacy of the IM treatment program as it was not implemented as per the prescribed IM treatment protocol (poor treatment integrity/fidelity).  This major methodological flaw in itself disqualifies this study as a valid study of the IM treatment program.

Confounding of IM treatment with pharmacological treatments. The sample consisted of 12 subjects with various diagnoses, but whom all shared a diagnosis of ADHD.  As described by the authors, half of the subjects were taking some form of medication during the study. More importantly, "no medication changes occurred over the course of the study". This introduces a potential intervention confound. It is very possible that the medications received by half of the sample were already impacting the ability of these children to attend and concentrate at their maximal levels. As a result, no non-pharmacological intervention would have a chance to prove its effectiveness as these six subjects were already being "treated " via medication (and might already be near or at their optimal level of functioning). When it can be ethically designed, the efficacy of nondrug (or new drug) effectiveness is typically investigated by having subjects on a  current medication regime discontinue their medication for a rescribed period of time to establish a nondrug baseline performance level. Then the new treatment (or new drug) is introduced and performance measured to evaluate effectiveness. This was not done in the current study for half of the subjects.   It can be easily argued that  at least half of the subjects were already receiving treatment or intervention (medication) which confoundes the ability to detect effectiveness for any new treatment (e.g.,  IM). 

Statistical and design analysis: It is well known in the research methodology literature that the simple analysis of change scores (from a pre- post-test design) is a less than optimal research design. The most critical issue is the fact that the reliability of the change or difference score is a function of the reliability of the pre-and post test scores and the correlation between the two. The reliability of change scores is always much lower than the reliability of individual measures. In addition, possible differences in the subjects as they entered the study should be controlled for in the analysis via some form of statistical control (e.g., analysis of covariance, repeated measures designs, etc.). Simple pre-/post-test change score research designs suffer from serious methodological flaws.  This has been recognized for decades.  The results are of questionable value. Click here for additional background information.  Furthermore, the study does not include a control group or the random assignment of subjects to treatment and control groups.

Incomplete literature review:  In the introduction the authors state that "there are only two peer-reviewed studies comparing pretreatment and posttreatment effects that include a control group, and one of these found that the Interactive Metronome improved golf swings in healthy normal individuals."  The authors failed to mention one of the largest and best designed IM efficacy studies (random assigment of subjects to treatment and control groups;  analysis that controlled for initial pret-test scores) by Taub et al.  (2007).  Furthermore, the above quote conveys the implicit assumption that a study of the effectiveness of IM on golf swings is not relevant to the current research. This reflects a narrow understanding of the potential causal mechanisms of the IM program. As discussed in the Taub et al article, and in a recent review of the efficacy of various brain rhythm treatments, a review of theoretical and empirical research (from a diverse range of disciplines) suggests the hypothesis that the effectiveness of IM is due to its impact on a "domain-general" cognitive mechanism. If a treatment programs efficacy is believed to be based on the modification of a domain general cognitive/brain-based mechanism, one would expect it to have generalized treatment effects across a wide variety of human performance domains. This is discussed in greater detail in the Taub et al paper. [Conflict of interest note - I am a coauthor on this paper].

In conclusion, it is my professional opinion that the Cosper et al (2009) IM efficacy study suffers from major methodological flaws which make the study an invalid study of the effectiveness of the IM treatment program.

Technorati Tags: , , , , , , , , , ,


4 comments:

Anonymous said...

What is the difference between IM and Cogmed? Which is better, or which would you choose for whom?

Traveller said...

a number of practioners are only prescribing 1 hour (or even less per week). I think it is insurance coverage that is driving this. In a small sample of 3, folks who take insurance are doing the shorter sessions

Tina said...

I too would like to hear from the author of this page as to the difference between Cogmed and Im. We have to choose one treatment for our child.

Kevin said...

Sorry for delay in responding. I think IM has greater potential, but currently CogMed has the largest empirical collection of studies to support it's efficacy. In time I hope that IM and COGMED will be compared in the same studies, but until then COGMED has more empirical support. Conflict of interest---I am a research advisor for IM