Tuesday, April 07, 2009

What if you are really, really average?

Some time back, I wrote about the yawning chasm that exists between Managers' and Executives' perceptions of themselves and the somewhat less forgiving reality that statistical analysis most inconsiderately insists upon inflicting on them. That post generated a lot of comments and emails and much laughter whenever I present the points I made in it to a live audience.

So imagine my delight when I came across Bad Science by Dr Ben Goldacre - courtesy of my splendid mother. Not only is this book important in its clear, calm debunking of much of the twaddle we are forced to read, see and listen to courtesy of today's audience-hungry media, but it's also a rollicking good read, peppered throughout with hilarious asides.

In the chapter on health scares, Dr Goldacre discusses how the Evening Standard, the Mirror, the Sun and the News of the World had all quoted results generated by a Dr Malyszewicz who was running a 'microbiology laboratory' on kitchen worktops in a shed in his garden. These newspapers referenced Dr Malyszewicz as "a respected MRSA specialist" "the UK's top MRSA expert" and a "microbiologist" – none of which he was. He had no microbiology qualification whatsoever, and his PhD was granted as a result of a non-accredited correspondence course.

Nevertheless, these august organs continued to peddle the findings that Dr Malyszewicz's 'lab' generated, despite the fact that no other accredited laboratory in the entire UK agreed with his findings. Two senior consultant microbiologists wrote to the Standard, then another colleague wrote to the paper and got this response:
"We stand by the accuracy and the integrity of our articles. The research was carried out by a competent person using current testing media. Chris Malyszewicz ... is a fully-trained microbiologist with eighteen years' experience ... We believe the test media used ... were sufficient to detect the presence of pathogenic type MRSA."
Pause for a moment and let that sink in. Read it again, please do. Here's Dr Goldacre's take on what that stunning piece of correspondence means:
"What you are seeing here is a tabloid journalist telling a department of world-class research microbiologists that they are mistaken about microbiology. This is an excellent example of a phenomenon described in one of my favourite psychology papers – 'Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments', by Justin Kruger and David Dunning."
And he's right - it's a great paper. Kruger and Dunning's concluding remarks say it all:
"We propose that those with limited knowledge in a domain suffer a dual burden: Not only do they reach mistaken conclusions and make regrettable errors, but their incompetence robs them of the ability to realize it."
The graphs in the study cracked me up:
X axis is the four actual percentiles, Y axis is the percentage score for self-assessment on the test areas – logic, grammar and humour.
This lovely, shiny red line is the Median score of 50%, below which common sense, reason and the laws of nature dictate that half of the test subjects must reside. But, funnily enough, this cohort didn't think so ...
Nope - the boys and girls of Cornell all rated themselves a bit stronger than average.
And then we pump in the Actual test scores. Oh look! 50% below the median – who woulda thunk it?
What's interesting here is that the above-average performers in the third quartile were pretty much on the money with their perceptions and the truly gifted tended to underestimate their abilities. But look at that gap between perception and reality for the bottom two quartiles! Perhaps this explains the perplexed expressions and blank faces on the bankers around the table at recent Congressional, Parliamentary and Dail inquiries? "But I know I'm right! I know I am! If they had just let it ride a little longer, the market would have come up ..."

Great book, highly recommended.

H/T: Mum

Related posts:
What if you are average?
Admitting ignorance

4 comments:

Anonymous said...

Finally, some quality research that backs up my years of real world experience, mostly with sales people. If you ask a sales force, to rate their own performance, every single one will be above average. By far.
thanks to your splendid mum, too.
best,
GL HOFFMAN
www.whatwoulddadsay.com

HR jobs said...

Ive been reading Ben Goldacre's columns in The Guardian but never picked up the book thanks for the recommendation. I agree with most said here in your post have seen it in day to day life, somehow statistics just proves the point better. On a vaguely related subject, have you read ‘How Mumbo-jumbo Conquered the World’ by Francis Wheen? I read it while back and remember being both mightily entertained and hugely impressed. Always a good combination.

Rowan Manahan said...

GL - Likewise. I have encountered this mode of thought more times than I care to remember, so it was very interesting to come across some robust academic analysis of the phenomenon.

HRJ - I loved Wheen's book and have passed copies on to many starry-eyed friends. SHAM by Steve Salerno is excellent too.

Derek Cassidy said...

If you haven't read Karl Popper, perhaps you could try Michael Shermer's ~"Why people believe weird things" (Jared Diamond and S.J Gould endorsed) He's also a major Ted conference speaker on pseudoscience and it's consequences. Amazon anybody?