Did Hattie get his statistics wrong?

[Update 11 February 2012: After I wrote this post, both John Hattie and Arne Kåre Topphol have commented on the debate. You can read more about this here.]

For reasons unfathomable to the average Norwegian, the entire rest of the Global community has heretofore failed to recognize Norway as the undisputed apogee of the civilized world, and has thus made only feeble and inconsequential attempts to learn to speak and understand our language. Hence, considering the following topic as having some interest beyond the confines of Norwegiandom, this post will be written in English.

By now, it is probably impossible to underestimate the impact John Hattie, and specifically his 2009 publication Visible learning: a synthesis of over 800 meta-analyses relating to achievement, has had on policy-makers, researchers and academics, and educators, concerned with improving school systems and best teacher practices across the globe. References to Hattie’s effect study are prolific in teacher training curricula, in white papers, and in the general public discussion on education (at least as seen from Norway, and the teacher-training course at the University of Oslo, and my own admittedly subjective assessment of it all).

Now, however, an article has emerged, titled “Can we trust the use of statistics in educational research” (“Kan vi stole på statistikkbruken i utdanningsforskinga”) (Topphol, 2011), published in the Norwegian Journal of Pedagogy (Norsk pedagogisk tidsskrift), questioning Hattie’s use of one of the statistical measures in Visible learning, namely the Common Language Effect size (CLE).

(Unfortunately, the article is in Norwegian only; you must be connected to the university server to gain access.) 

First introduced by McGraw and Wong (1992), the CLE is, as I understand it, an effect measure intended to be more intuitively understandable for the average non-expert-statistician reader. In short, the CLE size is the probability that one randomly selected sample from one distribution (e.g. of scores on a test) will be greater or smaller than one randomly selected sample from another distribution. This definition is consistent with Hattie’s own explanation of this measure (2009, p. 9), and also with the accompanying example, first used by McGraw and Wong, and revisited by Hattie in Visible learning:

Consider a difference in male (5’10”/177,5 cm) and female (5’4”/162,5 cm) average height. The effect size measured in Cohen’s d (for an explanation of this measure, I must refer to Hattie, or to the work of Jacob Cohen himself) is 2.0, considered very large, and the calculated CLE will be 92 per cent. The CLE percentage then says that, if you pick a random man and a random woman from the two respective populations, there is a 92 per cent chance that the man would be taller than the woman. Thus, in 92 out of 100 couples selected for blind dates, the man would be taller than the woman. (The “effect” in this case would be something like the effect of gender on average height.) Note that it does not say anything about how much taller – only the probability of a difference in heights in favor of one of the populations compared with the other, when you compare randomly selected samples from the two populations.

(And if we considered the bizzarro-world alternative that men and women had the exact same average height, what would the probability be of picking out a random sample from each group where the man was taller than the woman? Think for a second, and click “Les resten av dette innlegget -->” for the answer.)

(If men and women had the exact same average size, in a random selection from each group there would be a 50 per cent chance for the man to be the tallest – and a 50 per cent chance for the woman to be tallest (this very pedagogical instructional method is inspired by this excellent introduction to statistics).)

Thus, any CLE size has to be a percentage between 0 and 100. A positive effect would give a percentage between 51 and 100; a negative would give a percentage between 0 and 49 per cent; and no effect, i.e. no difference between populations, would give a CLE of 50 per cent.

However, when Hattie goes on to explain this measure further (2009, p. 9), he gives the following example: “Consider the [Cohen’s] d = 0.29 from introducing homework […]. The CLE is 21 per cent so that in 21 times out of 100, introducing homework into schools will make a positive difference, or 21 per cent of students will gain in achievement compared to those not having homework.”

This does not make sense, for several reasons: First, if introducing homework has an effect, albeit small, the CLE must be a number in the range between 51 and 100 per cent. A CLE of 21 per cent signifies a negative effect: if you have two groups, one given homework and the other not, and there is (only) a 21 per cent chance that a randomly chosen pupil from the group doing homework has a better outcome than a random pupil from the group not doing homework, homework clearly has a negative effect (and is probably better avoided altogether).

Second, if the effect of introducing homework in schools is positive (as Hattie already has stated, given the d=0.29), it will have that positive effect every time, i.e. 100 out of 100 times,¹ – and not, as Hattie says, “if you take two classes, the one using homework will be more effective 21 out of a 100 times” (ibid.).

And third, Hattie draws a conclusion on a group level based on an individually measured effect size: Albeit the middle values of class (i.e. group) scores will be identical to the average of individual scores, the standard deviation of class scores will be much smaller, and thus the CLE will increase considerably (Topphol, 2011, p. 469–70) (If you are interested in the math behind this argument, please refer to the article – or send me an email, and I will give you the underlying calculations.)

Throughout the book, Hattie continues to use the CLE measure indicating effect sizes for all the factors he considers. Remarkably, Hattie’s CLEs ranges from -49 per cent to 219 per cent (cf. e.g. Hattie, 2009, p. 263ff), which, as seen from the above discussion, are well outside the range of possible measures of CLE. Moreover, the concrete ways in which Hattie uses the CLE brings about some strange reasoning. For example, on page 42, in a discussion on the effect of prior achievement, Hattie says that

the overall effect size of [Cohen’s d=]0.67 is among the highest effect sizes in this synthesis of meta-analysis, although the common language estimate [CLE] should remind us that, on average, prior achievement will lead to gains in achievement on 48 per cent of the occasions, although there is much that is unexplained beyond prior achievement (100-48=52 per cent that is unexplained) and so there is much that schools can influence beyond what the student brings from prior experiences and attainments.

Again, (i) a percentage of 48 would actually indicate a negative effect; (ii) any effect would be an effect always and not only in a selection of cases;¹ and (iii) it makes no sense to deduct 48 from 100 if you operate with a scale from -49 to +219.

This is remarkable in its own right. To my knowledge, the other means Hattie employs to present effect sizes have not been questioned; but then again, for the averagely informed stakeholder in educational questions, the level of statistical knowledge needed to do that is rare.

However, and much more importantly – and especially considering that Visible learning has been the focus of immense attention the last couple of years, without any of this having been noticed – in my opinion, this opens up for at least two much more pressing conclusions beyond the question of use or misuse of statistics in Visible learning:

Most people supposed to consider and sometimes act upon statistic-based research have an insufficient knowledge of statistics; and most people do not actually sit down and read the literature they use and refer to (reading here meaning something beyond the superficial skimming of abstracts, summaries, tables, and conclusions). Lots of people write, but do we read? Not so much.   

As to the first point, the conclusion in Topphol’s article – and one which I fully support – is a call for more knowledge of statistical methodology among publishers and peer-reviewers; and a suggestion that researchers do not go beyond the limitations of their statistical competency when doing research and presenting their material. For my own account, being a student at a teacher-training program, and at the receiving end of much of the research and subsequent literature relevant to our object of study, I would add that quantitative methodology should be given a much more prominent place in at least our curriculum, and probably in similar curricula in general.

As to the second point, my humble suggestion is this: we should introduce a blanket ban on all educational research for the foreseeable future – that is, until we have actually read, understood, and if deemed necessary implemented into our professional practices and educational curricula, all the educational research and literature that have piled up so far.

And in the meantime, we could redirect the immense amount of freed-up resources to the things we all already know that we actually need and want: More supplementary education and training for teachers; better school buildings and school infrastructure; early intervention for struggling pupils; eradication of bullying; qualified teachers for all pupils; qualified substitutes, and a budget to get them; and so on (and you can add to this list at your own discretion. You know perfectly well what should be included).

Finally, a disclaimer of sorts: I am by absolutely no standards an accomplished statistician. If you see anything in this post that is wrong, please let me know, and I will adjust it accordingly.

¹ Not considering the variance: in a sample with a large dispersion, some individual samples may of course come out with a negative effect.

Cited sources

Hattie, J. (2009) Visible learning: a synthesis of over 800 meta-analyses relating to achievement. London, Routledge.

McGraw, K. O. & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin 111, 2. p. 361–5.

Topphol, A. K. (2011) Kan vi stole på statistikkbruken i utdanningsforskinga? Norsk pedagogisk tidsskrift, Vol. 6/2011. p. 460–71.