Torturing the Numbers

Post your research in political economy, share ideas and workshop your empirical work.

Moderator: sanha926

Torturing the Numbers

Postby blairfix » Thu Sep 24, 2015 4:34 pm

The quantitative social sciences are in crisis. Published quantitative works in the social sciences almost always find "significant" results ... that is .. results that are less than 5% likely to occur by chance alone.

How can it be that social scientists are "correct" so frequently, when the vast majority of theories in the natural sciences are proven wrong?

One element is certainly publication bias ... scientists are unlikely to publish results that disproved their theory.

A more insidious element, however, is documented in an excellent article by Joseph P. Simmons.

Simmons_2011_False-Positive Psychology.pdf
(221.12 KiB) Downloaded 158 times


He calls it the "degrees of freedom" problem. The economist Coase called it torturing the data until it confesses.

The problem is that their are a bewildering number of arbitrary choices that go into any experiment (or observational study). Often times, social scientists make these decisions based on what gives the best results. The problem is that the larger the degree of freedom in the experiment, the more likely one will find spurious results.

By cherry picking which data is used, one can almost always find "significant" results. For instance, by using sleight of hand, Simmons is able to demonstrate that listening to The Beatles song "When I'm 64" actually made people feel younger (he uses data from a real experiment).

The solution is to make all decisions prior to conducting one's analysis (and documenting all decisions clearly).

Psychology is currently in upheaval over the realization that many key studies are actually garbage that cannot be reproduced. I suspect that many econometric findings are also garbage ... the result of cherry picking data.
blairfix
 
Posts: 88
Joined: Fri Dec 20, 2013 3:45 pm

Re: Torturing the Numbers

Postby dtcochrane » Mon Oct 26, 2015 5:44 pm

Explain to me why it is wrong to torture the numbers.

If it warrants a confession that furthers our understanding, what is objectionable to the torture?
dtcochrane
 
Posts: 160
Joined: Mon Dec 08, 2008 9:23 pm

Re: Torturing the Numbers

Postby blairfix » Mon Oct 26, 2015 8:42 pm

It depends what we mean by "torture". For instance, say we are looking for a correlation between two quantities. Say we have 10,000 data points in each series, and that the data is generated randomly, so should have no correlation. If we carry out a regression on all 10,000 points, will will indeed find no correlation. But what if we start looking for subsets of this data for which there is a correlation. Even though the underlying data is completely random (so we should expect no correlation) it will be possible to choose a subset of the data such that we do find a correlation.

This correlation will not "reveal" something of importance. It will be completely spurious. It will be quite easy, for example, to choose 20 data points from the 10,000 such that the correlation is quite high. Any conclusions drawn from this correlation will be totally false. We have "tortured" the numbers into submission.

So the lesson is that we have to have a reason for choosing a small sample size. There may be legitimate reasons for choosing only a subset of the data ... but these reasons must be spelled out and the result must be reproducible. Most importantly, the reason cannot be "if I choose this subset, I get a better correlation".
blairfix
 
Posts: 88
Joined: Fri Dec 20, 2013 3:45 pm

Re: Torturing the Numbers

Postby dtcochrane » Tue Oct 27, 2015 2:46 pm

It would seem that this is where the public availability of datasets is important. If we maintain the metaphor, if the dataset is publicly available then others can go and find that although tortured, the numbers have not submitted. They remain just as defiant, ready to name those who seek to oppress them.

However, you raise an important clarification that I think needs to be amplified more generally: the importance of the qualitative 'story' accompanying quantitative analysis. As you say, reasons must be "spelled out." Okay, there is a correlation. We all know it does not imply causation, so tell me what the meaning of this correlation is. This relates to something economist Dierdre McCloskey identified as one of the major sins of economics: the confusion of statistical significance with economic significance.
dtcochrane
 
Posts: 160
Joined: Mon Dec 08, 2008 9:23 pm

Re: Torturing the Numbers

Postby blairfix » Tue Oct 27, 2015 8:31 pm

Interesting that you mention McCloskey. I just finished The Cult of Statistical Significance ... a very interesting read.
blairfix
 
Posts: 88
Joined: Fri Dec 20, 2013 3:45 pm

Re: Torturing the Numbers

Postby dtcochrane » Wed Oct 28, 2015 6:23 am

blairfix wrote: ... a very interesting read.


That seems like deliberate phrasing. Care to elaborate?
dtcochrane
 
Posts: 160
Joined: Mon Dec 08, 2008 9:23 pm

Re: Torturing the Numbers

Postby joefrancis » Thu Oct 29, 2015 7:27 am

The good news is that I think the problem is starting to be recognised in economics as well:

http://www.businessinsider.com.au/federal-reserve-paper-on-the-replicability-of-economic-studies-2015-10

The bad news is that it might just put economists off doing more quantitative research...
joefrancis
 
Posts: 81
Joined: Thu Jan 08, 2009 3:30 am

Re: Torturing the Numbers

Postby blairfix » Thu Oct 29, 2015 9:45 am

Thanks for this link, Troy. The replicability crisis has been big news in psychology ... not surprising that economics is even worse. The replicability rate they find in the article is almost comically low!
blairfix
 
Posts: 88
Joined: Fri Dec 20, 2013 3:45 pm

Re: Torturing the Numbers

Postby blairfix » Thu Oct 29, 2015 10:43 am

Thanks for this link, Troy. The replicability crisis has been big news in psychology ... not surprising that economics is even worse. The replicability rate they find in the article is almost comically low!
blairfix
 
Posts: 88
Joined: Fri Dec 20, 2013 3:45 pm

Re: Torturing the Numbers

Postby dtcochrane » Thu Oct 29, 2015 2:23 pm

Although I'd love to take the credit, that link was posted by Mr. Joe Francis.

Happy to see you around these parts, Joe.
dtcochrane
 
Posts: 160
Joined: Mon Dec 08, 2008 9:23 pm

Next

Return to Research

Who is online

Users browsing this forum: No registered users and 1 guest

cron