On Shock

I'm mostly caught up on the newsfeeds and blogs that I follow. This is after effectively missing three days, and dedicating most of two days specifically to catching up.

Somehow, I think I try to follow too much.

Writing on the articles I've been planning to review will, however, have to wait... as I literally can't find my copies of them. I'd print out replacements, but my printer's broken, and I find it difficult to review an article while needing to switch between windows to read it and write about it.

While looking, however, I found my printed copy of what is easily the worst article I've ever read. In other words, it's the closest thing I have to a prototype of "bad science" in the peer-reviewed literature.

And no, papers in Medical Hypotheses and the like don't count. They're not peer-reviewed.

Anyway, that paper is a "wonderful" little piece (notice the sarcasm) by van Oorsow, Israel, von Heyn, & Duker (2008) entitled "Side effects of contingent shock treatment".

I originally came across this paper around a year ago. I summarily ignored it.

I came back to it, however, earlier this (calendar) year. As part of one of my classes, I was required to do a brief review of the literature on one topic and summarize the methodology of a few papers on it. I chose contingent electrical stimulation -- not because I liked the topic, but rather because I viewed it as an excuse to familiarize myself with the literature on it... and thereby improve my ability to counter that literature.

My paper's section on this steaming pile of crap reads:

Van Oorsouw et al. (2008) attempted to evaluate the side effects of the use of contingent shock to treat the "severe problem behaviors" (p. 513) of nine students at the Judge Rotenberg Center (JRC) in Canton, MA who showed "severe forms of SIB and aggressive behavior" (p. 514). The shock devices used were JRC's graduated electronic decelerator model one (GED-1).

A nonconcurrent quasi-multiple baseline across participants design was used. The length of baseline in each subject was the difference in time between their arrival to JRC and court approval for the use of contingent skin shock. During baseline conditions, DRO procedures, a token system, a response cost system, and contingent restraint were in effect. These procedures were also in effect during treatment conditions (van Oorsow et al., 2008).

During treatment conditions, skin shock was delivered contingent on SIB and/or aggressive behaviors. Subsequently, a staff person or teacher who administered the shock would explain the contingency and prompt the student to continue what they were doing prior to the shock (if needed). Following this, the electrodes would be rotated slightly and the time of administration would be recorded (van Oorsow et al.,
2008).

Four categories of target behaviors were also recorded: positive verbal and nonverbal utterances (PVNU), negative verbal and nonverbal utterances (NVNU), socially appropriate behaviors (SAB), and off task behaviors (OT). Van Oorsow et al. (2008) defined these behavior categories only by example.

Data was recorded by 10s partial interval recording based on 10-min videotapes of randomly chosen points of time in the participants' day. All recordings were selected so that teachers and staff were not applying reinforcing contingencies within them. Average IOA was 95.48%. Data was analyzed by visual analysis by seven clinicians. Mean agreement for change across clinicians was 78% for PVNU graphs, 92% for NVNU graphs, 91% for SAB graphs, and 89% for OT graphs. Agreement ranged from 57% to 100%. All participants were evaluated as having either improved or not changed in each category. Group analyses were not conducted (van Oorsow et al.,
2008).

Data on one participant ("S.S.") was not taken in regard to SAB because she had "severe mental retardation, and who, by definition, failed to show any form of SAB" (van Oorsow et al., 2008, p. 522). Despite this assertion, S.S. is listed as a girl with moderate mental retardation and autism within the study's demographic table. Van Oorsow et al. do not provide support for this assertion.

Additionally, this design is not appropriate for a full analysis of side effects, as it only allowed for the possibility of four specific negative effects (i.e. lowered PVNU, increased NVNU, reduced SAB, and increased OT) and only included nine participants. As it is common practice to report effects that occur in less than one percent of the treatment population as side effects of a treatment and to take them into account when planning a course of treatment, the number of participants was too small to allow a comprehensive evaluation of side effects. Additionally, it is common practice to take into account a wide variety of effects, including those that are only directly available as private events (c.f. Eli Lilly and Company, 2006). As such, the study fails to provide sufficient support for van Oorsouw et al.'s (2008) assertion that "the objection that CS should not be used due to associated negative side effects, fails to be inconsistent with the results of this study" (p. 523).

This, of course, is only a tiny fraction of the problems with the paper. For one thing, I have reason to suspect that they misrepresented their procedure in the article. A careful reading of the article and comparison to publically available documentation will indicate that standard procedure at JRC involves a change of setting and activity schedule upon receipt of court approval for the use of shock, something that the article did not report. Assuming that this was the case... well, let's just say that the ethics of their procedure were even shittier than my summary suggests.

And, of course, since it was a purely methodological paper, I didn't even begin to directly touch on the ethical problems with what they did. Suffice it to say that they were many and manifold.

For instance, one of the subjects was an eleven year old boy whose only diagnoses was oppositional defiant disorder and "mild" mental retardation. Another was a nineteen-year-old woman who had a diagnosis of post-traumatic stress disorder.

Words fail me here. The horses and deer are prancing.

Oh, and there are the usual randomization issues, selectivity bias... yeah, all of those. Poor, poor study on so many levels it's not funny.

References:

Eli Lilly and Company (2006). Prescribing Information (Prozac Label, 2006). Retrieved February 28, 2009, from http://www.fda.gov/cder/foi/label/2006/018936s076lbl.pdf

van Oorsouw, W., Israel, M., von Heyn, R., & Duker, P. (2008). Side effects of contingent shock treatment. Research in Developmental Disabilities, 29, 513-523.