Last week the journalism world was buzzing about — guess who? — Jonah Lehrer. Yes, again. We knew about the science writer’s self-plagiarism and Bob-Dylan-quote fabrication. Last week a New York Magazine exposé by Boris Kachka claimed that Lehrer also deliberately misrepresented other people’s ideas.
Kachka’s piece led to some fascinating discussions about whether it’s possible to tell a science story that’s both riveting and fully accurate. Science journalist Carl Zimmer, for example, wrote a thoughtful, inspiring post about the messiness of science. All of the commentary left me wanting to hear more details from the scientists in Lehrer’s stories. Had they been misrepresented? If so, how? Were they upset? Did they complain?
Kachka and Zimmer zeroed in on a 2010 story about the scientific method that Lehrer wrote for the New Yorker. The story’s premise is clear from the title (“The Truth Wears Off”), the subtitle (“Is there something wrong with the scientific method?”), the nutgraf (“It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable.”), and the last few lines (“Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.”).
This article works well in Kachka’s piece for two reasons. First, it has a dash of got-cha! Schadenfreude: Well well well, fabricator Jonah Lehrer doesn’t believe in truth! The second reason, and the one I care about, is that the article was vetted by the New Yorker‘s famous fact-checkers. It has no tweaked quotes, no plagiarism, no obviously wrong statistics or data points — in other words, none of the easy mistakes that all of us make, occasionally, and might be forgiven, occasionally. Lehrer’s offense in this piece, at least according to Kachka, is worse than any of that. Lehrer describes an idea — The scientific method is broken — as if many scientists support it. Do they?
The scientists quoted in the piece are interested in the ‘decline effect’, the phenomenon that robust scientific results often can’t be replicated. It’s a fascinating topic, with enormous implications, and worthy of attention by the New Yorker. But is the decline effect, as the article claims, a rebuke of the scientific method?
On Friday I wrote to seven of the scientists quoted in the article and asked what they thought of it. Six responded, with the earnestness and precision that I’ve come to expect from scientists. The short story is: Almost everybody was satisfied with Lehrer’s specific descriptions of their work, but some were annoyed by his spin on it.
Some excerpts (with spelling and grammar cleaned up):
Rich Palmer: I had no problem with Lehrer’s portrayal of my own results…I actually thought he did an excellent job there. I could take issue with his subtitle “Is there something wrong with the scientific method?” (which so offended Kachka), because the problem isn’t with the scientific method in the strict philosophical sense of “observation –> hypothesis –> test” or “one chooses between alternative hypotheses based on new data”. The problem arises because the ‘scientific method’ is implemented by humans, and humans are inherently fallible.
Part of my email had read: “Was the piece, in the end, true?” Palmer wrote:
There is no truth (except in mathematics). All other so-called scientific conclusions are necessarily false to some degree (i.e., include some uncertainty). I think journalists and bloggists need to make peace with this.
…If there’s a lesson here, it’s about a widespread human failing. Most people would rather some other clever person distill down all the complex details into a good story for them, preferably in excellent prose. But those distilled stories should never be treated as a substitute for original research results. If anyone really wants ‘the truth’, they’re going to have to slog through an awful lot of turgid and arcane original research and draw their own conclusion.
Leigh Simmons: I actually thought that piece was rather good. And he quoted me accurately.
John Ioannidis: I think that Lehrer’s write-up was overall pretty good. Retrospectively, perhaps the one aspect that seemed a bit off was the tone, but this is not an unusual problem in journalistic writings (unfortunately it is very common even in purely scientific writings).
In a follow-up message, I asked Ioannidis if he agreed with the somewhat mystical argument that the scientific method doesn’t work. He replied:
Of course, I certainly think the scientific method does work (!). It is primarily an issue of optimizing its efficiency and making sure it is applied the way it should be applied. Retrospectively, perhaps you are right that there was a bit of mysticism in the ending of that piece, but I thought it was primarily a figurative mode of writing that you would expect from New Yorker rather than from a Science or Cell or JAMA article.
John Crabbe: Well, I disagree with the conclusion that science is broken. But it was, after all, an opinion piece. I felt that Lehrer’s portrayal of our study was partially accurate, but also misleading.
…Lay science is a very tough game, I think. You have to simplify, and you have to engage. This leads sometimes to provocative lines of argument. While I am not at all opposed to this, I do think he went quite a ways beyond the data in his NY piece. I wouldn’t characterize it as scientific misconduct…He didn’t invent anything, just spun complex data.
Jonathan Schooler: Lehrer appropriately portrayed both my research and the discussions that I had with him.
…Lehrer’s conclusion that “When the experiments are done, we still have to choose what to believe” suggests that in the end science is simply reduced to opinion. Although I do not personally concur with this view, I do not think that in stating this perspective Lehrer was misrepresenting the scientific views that he queried. Lehrer ended with his own editorializing on the subject and while I do not personally endorse his final opinion, he never suggests that I nor other scientists necessarily subscribe to his view.
Michael Jennions: I think his portrayal of my work was perfectly fair…One of the charges levelled against Lehrer with respect to his article is that he implied that science might be broken. The last paragraph in particular annoys many scientists…Perhaps Lehrer should have emphasised that these are not mainstream views?
…This leads to the more general problem. If scientist A says X occurs, and scientist B says X does not occur – what should the journalist do? Is their job to be balanced and present both cases, or to work out where the balance of evidence lies and reach a conclusion? This is an age-old question that all journalists struggle with, and there are rarely simple answers.
—
That age-old question is a good one. Would it have been preferable for Lehrer to cover the controversy — that is, to not only explain the decline effect and its various explanations, but to point out to his readers that some of the explanations are more widely accepted than others? This would have made for a more complicated narrative and, perhaps, a less popular one. If he had pitched the story that way, would his editors have bought it?
I also asked the scientists whether they were upset about the article, and if so, whether they did anything about it. Some, like Schooler, were happy: “I was not upset, to the contrary this article accelerated discussion of these important issues.” Ioannidis simply shrugged: “This issue certainly did not reach the level of annoying me to the point that I would think a rebuttal would be due.” Crabbe, though, was pretty miffed. He wrote Lehrer an email, received a reply he wasn’t satisfied with, and ultimately wrote a formal complaint letter to the magazine. Here’s part of the letter that resonates strongly with me:
“To me, a working scientist, your article was troubling not because of the solid points scattered throughout (Science is difficult, measurement itself is difficult, publication bias is a problem, etc.). Rather, the message to a non-scientist (the large majority of New Yorker readers) is, science is a useless exercise. Nothing can ever be learned from it because no single experiment can establish ‘truth.'”
I happen to agree with Crabbe’s assessment of the New Yorker story. But the thesis isn’t the story’s biggest problem. The biggest problem is that Lehrer presented an argument that is not supported by the vast majority of scientists, and never let his readers know just how far out of the mainstream he was taking them. I think it was misleading, and perhaps dangerously so.
Hearing from these scientists made me wonder, somewhat despondently, whether science journalism is a useless exercise. Here is Lehrer, one of the best science writers I ever read, publishing in the most elite magazine with the help of the smartest editors and most rigorous fact-checkers. And still, still, the story isn’t true. Why should the rest of us bother, and why should scientists give us their time in our attempts?
In another email, Crabbe gave me a damn good answer. “Science journalism is crucial for the maintenance of a funding base for science,” he wrote. “Why should people pay for stuff they don’t understand at all!” There’s also the broader aim of helping people better understand the natural world. Lofty, maybe, but worthwhile goals.
And yeah, it’s hard, but then so is science. I suppose, like these scientists trying to learn from their failed experiments, we science journalists just have to keep trying to root out the truth.
*
I invite Lehrer and all of the scientists mentioned here to leave a comment if they’d like to say more — I only quoted parts of their responses, after all. I especially hope they pipe up if I’ve misrepresented their views, which would be so deliciously meta, wouldn’t it?
A big thanks to fellow science writer Kelly Rae Chi, whose Twitter messages inspired this post.
“Here is Lehrer, one of the best science writers I ever read, publishing in the most elite magazine with the help of the smartest editors and most rigorous fact-checkers. And still, still, the story isn’t true.”
There’s increasing evidence that papers published in top tier journals are more likely to be wrong than in other journals. I wonder what a similar study of science journalism would produce? That is, it could be that the pressure of publishing in an elite magazine that was partly responsible for why it was untrue in the first place. As you said, would they have bought the piece if he pitched it as a fringe idea, and one that would cover it that way?
I don’t know if that helps or heightens your worry, but it would get it to make more sense.
That’s an interesting idea, Ben — and kind of depressing if you continue the analogy. Just as a Nature paper can make a postdoc’s career…
I think at the end of the day all journalism is just one (sometimes two or three or four) person’s accounting of what happened. It can be extremely accurate when it is only dealing with an event, or facts, but in the realm of ideas any piece like this is basically an essay. You’re trusting the writer to give you a good perspective. Very frequently we trust people we shouldn’t trust. Sometimes that charismatic guy you just can’t stop listening to is only going to get you in trouble.
Well put. And matches with another point a couple of the scientists said — science journalism is most accurate in news stories about a single study.
Interesting that some of the scientists thought single-study news stories are the most accurate. But that, too, is obviously problematic for science journalists thinking about the meaning of it all, considering all the ways that single-study stories are so often wrong, or skewed in an unhelpful direction, or covering topics that in the longer-term are not that valuable to readers.
Yeah, and actually, one of them said that the reason they were more accurate is because journalists rely on the press releases for those (eye-yi-yi!)… which actually reminds me of another point I meant to say in the piece:
I DO NOT think the goal of the journalist should be to make a source happy.
The problem is the way the story was sold and spun, not its raw facts. I think the New Yorker’s editors let their publication down by accepting the story as it appeared, and not fighting back on the way Lehrer framed the issue. (This is not the kind of thing that rigorous fact-checkers would pick up, so I would blame the editors, not them).
You say that the New Yorker is ‘the most elite magazine with the help of the smartest editors and most rigorous fact-checkers. But perhaps when it comes to *science*, the New Yorker may not be the most elite magazine, and it may not have the smartest *science* editors.
so lemme guess, the most elite magazine for science writing is….Nature? 🙂
(Can’t say I disagree, really, I heart you guys…)
I am not sure that the fact that papers published in top journals are more likely to be wrong is a finding that should be too depressing. There is an easy way to be right, just replicate something that has been replicated many many times, with a very small twist. The smaller the twist, the more likely it is to be right. This kind of finding is more likely to be published in a lower tier journal.
The stuff published in top tier journals is selected for its newness and impact, not necessarily for its careful adherence to what we already know is true.
If something could have a huge impact, but gets a 50/50 chance of being true, why not publish it in Science or Nature? This will reach the most number of other scientists, and give them a quick chance at trying it out.
Certainly that can have perverse consequences, and if something only has a 10% chance of being true, then maybe we shouldn’t waste everyone’s time. But I don’t see how it is a problem within science that top tier journals publish relatively more “wrong” (or let’s say, “tentative”) findings than other journals.
But the economic incentives are totally against just doing coverage of single, interesting studies. This was a great deal of science coverage a decade ago, now it often seems mechanically written — and that’s because both publications and individual writers have found that larger, scope-ier, think-ier pieces are better for their brands.
Cedar’s point is the least cynical take I have ever seen on this, and I find myself warmed and chastened by it. Seriously. Everything I say on the internet sounds sarcastic somehow, but I appreciate his point.
That said, there is a problem when people feel like *in order to have a career* they *must* publish in elite journals that make novelty and impact such high priorities.
The problem is that you don’t get to publish in these places just because you are very smart and diligent and creative. You must also be lucky to have an exciting result. You must also be willing to spend a lot of time crafting a narrative that makes the novelty and impact of your work obvious enough to the editors that they will send it out for review.
It’s not that storytelling doesn’t have an important place in “objective” fields like science and journalism. But if we do our work in desperate fear that the wolves will be at our doors unless we can place something in elite publications, we will become a culture that understands how to pitch attention-grabbing narratives better than we understand the world around us.
I remember being bothered by that article when it came out.
I’m kind of shocked that the scientists seemed so indifferent to Leher’s spin and what readers might come away with.
Crabbe said it well: “Rather, the message to a non-scientist (the large majority of New Yorker readers) is, science is a useless exercise.”
Yep. That was the take home message. It’s an attention grabbing narrative –at least if it’s coming from a publication like the New Yorker. Of course if you read that on the blog page of a creationist, you wouldn’t look at it twice.
Re indifference: Scientists have low expectations of the press, especially when it comes to setting the context of their work, representing uncertainty and keeping reported claims within the realm of fact. They are often astonished when a reporter gets it right. This is hardly new, I suspect, but more scientists have been able to give voice to their dissent in recent years; a good trend, but one the press can’t solely rely on as a corrective. Anyway, as Ed Yong notes, the reporters and scientists concerned about this are already doing their best. The problem children are down at the pub, ignoring it all.
I loved Carl’s post. The ever-failing nature of science writing is one reason I tack Beckett to my cubicle wall: “Ever tried. Ever failed. No matter. Try again. Fail again. Fail better.”
When writers don’t respect this tension, it whispers out of their prose in a low drone. It can be hard to hear in the online din, and they can fool so many for so long — especially with long-form editors versed in science a vanishing breed — but the work, in the end, won’t last.
The New Yorker piece is certainly the *least of* the problems I have with JL’s work. The primary flaw was, as I think you pointed out well, that it left a major philosophical bombshell unexamined. It needed a “To be continued…”.
I don’t particularly agree with the nihilism apparent in some of the comments. Science journalism will, like science, always have flaws, and, like science, the flaws are part of the process. This meta-discussion itself is useful self-examination.
Also, there are so many good science writers now. As MSM’s cut major departments, bloggers and other indies have really been filling the gap well, and allowing for a good conversational flow.
I read the New Yorker piece, and I didn’t at all think that the message was “science is meaningless.” Yes, the headline tried to grab readers with a provocative question. But the final lines weren’t telling readers how to think about all of science. They were a few meditative/lyrical lines that came after a careful description of a very disquieting study—one in which somebody had designed a study whose entire point was to replicate basic results in different labs and had failed to do so. So, yes, Lehrer accentuated that sense of vertigo: it’s sometimes extremely hard to get the noise out even if you are a very careful scientist. But to say that the article was peddling mysticism is silly. Some scientist types are so terrified of the Postmodern Threat that the lightest gesture in that direction freaks them out, but most readers aren’t like that. Are there a lot of blog posts out there from people who read the piece, and concluded that science Doesn’t Work At All? I doubt it. In other words, the damage allegedly done to readers’ brains was not done. Readers probably got the message that was intended: the science is so hard to do with precision that, at times, it can be very very hard to get at the truth.
Thanks for this thoughtful and sharply felt piece. This overlaps with so many of my concerns about Jonah’s work and what Seth Mnookin called the ‘summer of our discontent.’ I’m struggling with these right now as I prepare for a talk I’m giving tomorrow in Wisconsin about how to write about science in an honest way. This helps clarify some things.
The most encouraging thing amidst this? The presence of not just the Carl Zimmers who cut to the chase (while examining their own work), but of younger writers like Virginia Hughes who are struggling with these issues so ardently.
Embrace the mess. Fail better. We can do this.
As a lay reader who is 62 years old with an engineering background, please keep up the effort to explain to us lay people the wondrous science discoveries. We want to know and we appreciate your best efforts.
Carl Sagan would often tell this story about a 19th century physicist Robert Wood who when asked to give a witty anecdote about the difference between physics and metaphysics said it lies in the laboratory. Both priests and scientists have great ideas but the priests have no way of determining good from the bad save their own fallible minds. I think for a long time journalism lacked a working laboratory and few incentives to replicate the work of others. I think journalism could be improved by discussing how the ideas the journalist is presenting could be tested and found false. Blogs and the internet has solve the replication problem. How do we avoid fooling ourselves? Equal parts skepticism, imagination and laboratory.
I very much enjoyed Lehrer’s post, while not at all believing that there is something wrong with science as such. I also did not think that he was foisting his ideas on the scientists whose work he cited. I thought, on the contrary, that he was very much trying to claim credit for originating his critique.
Virginia Hughes, if you don’t like Lehrer’s interpretation of these result, what’s your explanation for them?
But Dr. Palmer, is there a scientific method at all without humans. Does it exist as an independent entity?
John,
I think the scientists gave plenty of plausible explanations, the most plausible being publication bias. They’re all mentioned in Lehrer’s article, which is why it was all the more puzzling that he gave it a mystical slant at the beginning and end.