The Poli Sci Bitches love a good Twitterstorm, and while we know that, as Twitterstorms go, yesterday’s discussion of this New York Times article on laptops in the classroom was pretty tame, it really resonated with us. Which has led us to our first edition of This Week in Bad Journalism!
While there are others who can break down flawed statistics in journal articles better than we can (we only teach intro methods, after all), there are countless examples of bad research journalism that go viral on our Facebook and Twitter feeds every day. Usually what happens is: an journalist looking to make a click-bait headline or to prove a point will link to a study without really reading it. People share it without clicking on the study at all. Sometimes without even reading the article. Then all of a sudden you have everyone saying that giving blow jobs cures depression.
So we’re taking it upon ourselves to find and point out examples of bad research journalism, starting with the NYT article.
The Twitterverse* did an excellent job at capturing exactly why the Leave Your Laptops article was so flawed. Our biggest beef with the article was this: it’s easy to read an abstract, find a statement that supports your point of view, and link to a study to prove your point. It’s a lot trickier to actually read the study and see if it says what you think it says.
You see, the author quickly runs through reasons why laptops in classrooms are bad, one of which is that they reduce exam scores by 18%. Eighteen percent! Wow! That’s a big effect! Clearly, if removing laptops from classrooms can improve my students’ performance by almost two letter grades, I should look into it.
We aren’t going to beat a dead horse with the West Point study’s flaws (we really don’t think it’s a bad study). Here are our bullet point highlights for why it doesn’t necessarily support the NYT author’s point:
- The authors use information about groups (a laptop-permitted or -prohibited class) to make claims about individual student performance, what we call an ecological fallacy.
- The authors’ model showed that permitting or prohibiting laptops in the classroom had an effect only on the multiple choice and short answer portions, but there was no significant effect on the essay portion.
- The effect of the treatment is relatively small.
- While we acknowledge that saying, “This isn’t generalizable” is what every political scientist says when he or she is woken from an accidental slumber during any research presentation, we’re going to say it here too. This isn’t generalizable.
But our problem isn’t with the study itself (and we loved the excellent Twitter conversations about its flaws and its strengths!), it’s with the op-ed that used it in a fly-by to support a total laptop ban.
Click-bait headlines are the hill that the Poli Sci Bitches are going to die on. So, stay tuned for the next issue of This Week in Bad Journalism!
*Trying to give credit where credit is due, because these ideas were certainly not all original to us: Joshua Eyler @joshua_r_eyler, Kevin Gannon @TheTattooedProf, Daniel Franke @danfranke79, and many others. Please let us know if you’re among those we should credit here!