I finally finished Anna Karenina, which means I now understand how gentleman farmer Konstantin Levin felt when, after spending far too much time thinking about farming, he finally just grabs a scythe and starts mowing hay, real hay!
That, gentle reader, is how I felt last week when I talked to Leif Nelson, senior author of the recent PNAS study “Empirical audit and review and assessment of evidentiary value in research on the psychological consequences of scarcity.”
Some background: A few years back I wrote about a popular concept in psychology and economics called scarcity mindset. In its original form, scarcity mindset refers to the idea that poverty alters how people allocate their attention, making them worse at certain kinds of decision-making (and thus more susceptible to traps like high-interest payday loans).
Early experiments looking at the psychological impacts of scarcity focused on actual financial hardship. In one, behavioral scientist Sendhil Mullainathan and economist Eldar Shafir asked people who lived below the poverty line to think about a hypothetical car repair that cost $1,500, and found that it temporarily impaired their performance on an IQ test as much as losing a night of sleep.
The field has since expanded to include many different types of scarcity, including perceived scarcity, lack of time, social status, etc. It’s a fascinating area and the sources I interviewed for my story, including Shafir, are highly-regarded and articulate. As a freelance journalist, stressed about money and deadlines, I found their arguments that even a perceived lack of time and resources could be taking a toll on my cognitive function easy to accept. I delivered the story my editor had asked for and she liked it, barely touching the copy before posting it. Once it was published, though, I realized that I hadn’t addressed the most important question: Is “scarcity mindset” real?
Rather than carefully evaluating the best and weakest evidence for scarcity mindset, to see which claims were solid and which weren’t, I’d taken its existence for granted. This bothered me at the time and continued to worry me later. One of my fears, as a reporter who often covers the brain and people, is that I’m reinforcing false beliefs about people –beliefs that shape how we think about and treat each other. What if scarcity mindset turns out to be just another conceptual earworm that has wriggled its way into the popular consciousness, but turns out to be wrong?
These are the questions that gnaw at me late at night, when I’m listening to Maggie Gygyllenhaal reading Anna Karenina aloud on Audible. So when a nice PIO from UC Berkeley asked if I’d be interested in writing about a new study published this month in PNAS that “raises significant questions about the psychology of scarcity—or the idea that a lack of resources can lead to reduced cognitive ability, impulsiveness, and other behavior changes,” I thought, “At last!” Like Konstantin Levin, I was ready to scythe my way to revelation.
Nelson, who teaches at Berkeley’s business school, is known for his research on p-hacking—the biased selection of study results that favor one’s hypothesis, say, by only reporting positive results or throwing out data that contradict the idea you favor. Although the word “hack” implies malicious intent, most people don’t do it on purpose, Nelson says. It’s just our natural self-serving response to ambiguity. Psychology is full of p-hacking, which means that a lot of studies that appear convincing at first–more than half, according to this big 2015 study — fall apart when independent groups attempt to replicate them.
Nelson and his colleagues are developing a method of sorting out which claims are well-supported by evidence, and which are likely to be nonsense. It’s called an empirical audit and it takes a lot of work. In the new PNAS study, he and his colleague, Don Moore, asked PhD students in a course they co-teach to choose an area of psychology to delve into — the students, not the professors, decided on scarcity — Nelson says. Then they tried to narrow their analysis by selecting only studies that cited a particularly influential Science study by Mullainathan, Shafir, and behavioral scientist Anuj Shah. From that pool they randomly selected 20 studies, and each student tried to replicate one of them. For practical reasons, they replicated only online studies, which made it easier to study the large numbers of people needed for statistically meaningful results.
Just four of the 20 studies they tried to replicate worked out. Obviously, that does not mean that all scarcity psychology is bunk, by a long shot; they didn’t try to replicate any of the original field studies, for example, and there are plenty of good reasons why replication efforts could have failed. But it does suggest that we should be wary of claims based on online studies of scarcity “priming” — when researchers ask people to imagine not having something they need, or to recall a time of scarcity in their lives. This audit suggests that just thinking about scarcity — as opposed to actually experiencing it — doesn’t change our cognitive abilities much, Nelson says. Nor, he adds, is cultivating “an abundance mindset” likely to make a dent in systemic societal problems that stem from a real lack of resources, like poverty.
One study that used a scarcity prime did perform well in the audit. When researchers asked people to think about a time when they lacked resources, they consistently reported higher levels of back pain. Could this have anything to do with the fact that I almost always blow out my lower back before a major deadline? Or the excruciating pain shooting down my neck and into my arm as I race to finish writing this post? Impossible to say, Nelson says, “but for some reason, that finding replicates perfectly.”