Fanelli Considered Harmful
Regarding
Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences:
Fanelli reports:
If the hierarchy hypothesis is correct, then researchers in “softer” sciences should have fewer constraints to their conscious and unconscious biases, and therefore report more positive outcomes. . . . the odds of reporting a positive result were around 5 times higher among papers in the disciplines of Psychology and Psychiatry and Economics and Business compared to Space Science, 2.3 times higher in the domain of social sciences compared to the physical sciences, and 3.4 times higher in studies applying behavioural and social methodologies on people compared to physical and chemical studies on non-biological material.
Which is to say, there is evidence for the hierarchy hypothesis.
However, since an analysis of the process of science is itself a social science– and obviously one that would have to come in lower on the “Hierarchy of the Sciences,” by virtue of its greater level of abstraction– we’d expect it to be even more prone to unjustified positive results as the result of workers in the field having “fewer constraints to their conscious and unconscious biases”. That Fanelli reports a positive outcome thus raises the unfortunate possibility that if Fanelli’s proposed mechanism is correct, it must logically be the result of an unjustified positive, and thus is incorrect.
This would seem to be a paradox. Fortunately, it is one that is easily resolved. The argument in question is being advanced in this blog post, which by attempting to consider the validity of Fanelli’s work must inevitably be considered even lower on the “Hierarchy of the Sciences” than Fanelli. Ergo, every conclusion suggested here will be even more prone to error than Fanelli’s, as I have even “fewer constraints” on my “conscious and unconscious biases” than Fanelli does. Thus, that this blog post argues Fanelli’s work cannot be trusted on paradox-avoidance grounds implies that actually, it can be.
But there is a bigger issue at stake here. This blog post is not unique; all discourse that takes Fanelli’s work into consideration is subject to the same source of error. Attempting to learn from what Fanelli teaches puts one’s subsequent discourse down into the error-prone realms. One may think one is thinking about (for example) the genetics of fruit flies, but since one is considering what one is saying by light of Fanelli’s insights, one is actually thinking about genes and fruit flies and the possibility of one’s “conscious and unconscious biases” resulting in one reporting a positive result without justification. Subsequent statements thus span multiple levels of the “Hierarchy of the Sciences”, and so will be subject to the elevated levels of unjustified positive results.
Thus, using a not uncommon working definition of “stupidity” as “a tendency to say things that are incorrect”, we must therefore conclude that if Fanelli is correct, reading Fanelli’s paper makes the victim stupider. The more one considers the issues and meta-issues it raises, the less trustworthy one’s judgement becomes.
As this is obviously exactly what we wish to happen, I suggest that reading Fanelli’s paper be made mandatory.