Russ Roberts on the Difficulties of Definitive Tests

Russ doesn't use that language, but that is what is going on when he asks what examples readers can provide where economists have had to overturn their theoretical propositions due to a statistical analysis.

This is not the first time Russ has made this argument.  in an earlier post, he actually pointed out the difficulty of getting statistical tests to provide unambiguous results in the field of complext phenomena.  A very Hayekian argument no doubt, and one that can see its most articulate statements in Hayek's Studies in Philosophy, Politics, and Economics, and New Studies in Philosophy, Politics, Economics, and the History of Ideas.

But this remains an argument that obviously most readers who fashion themselves "scientific" simply cannot bring themsleves to understand.* It is too bad because parading as "scientific" for pretending to achieve something which in fact cannot be achieved in practice, results in little more than pseudo-science and doesn't advance knowledge in the discipline.  There is a lack of self-knowledge of the limits and achievements of the discipline.  In fact, it is this sort of scientistic posing that discredits a discipline when it is pushed to engage in practices which are inappropriate for that discipline, and clouds the real contributions that in fact can be made by a discipline of complex phenomena, such as economics.  Nassim Taleb makes this sort of argument in The Black Swan, in his critique of contemporary economics, but it was also made by scholars such as Mises and Hayek --- and also I should add Buchanan, Coase, Knight, etc.

Does the fact that statisical tests cannot provide unambiguous refutations of economic theory imply that we learn nothing from statistical analysis?  I would argue definitely NO, we can learn a lot from statistical analysis.   But what we cannot do is "test" theories with statistical tests.  Does it also mean that we cannot refute economic theories?  Again, I would argue NO.  Refutations of "theoretical" propositions result from demonstrations of logical error, and one can also demonstrate the irrelevance of a logical argument to a contemporary problem due to the inapplicability of the theory to the situation to be examined because one or more of the various subsidiary assumptions that make up the network of statements that constitute a theoretical construction might not hold.

-----------

*See also here.  But more nuanced thinkers such as Vernon Smith understand the issues involved and the relationship between theory and empirics.  The critical points in Russ's questions are to note his use of the word "causation" and his focus on "economic theory".  It is relevant, I think, that Tyler finds most of the examples in macroeconomics, rather than in microeconomics.

Comments

[Devin Snead]

What was shocking to me about Russ Robert's post was that he once asked Ian Ayers to name a study that "ended a controversy" and he named the Levitt and Donahue study on abortion and crime rates.

[Redland Jack]

I've never been a fan of econometrics as an academic 'tool'. It takes a theory to defeat a theory. (Now that I think about it, I probably 'stole' that idea from somebody, but I've held it for so long now, that I feel like I came up with it...)

I do think that econometrics is a very useful tool for businesses (or, I suppose, governments).

[Adam]

The responses have exposed the very lack of understanding you describe, Professor Boettke.

Tyler Cowen simply provided a laundry list as answer, while for economics Caplan names Behavioral economics.

Both of them seem to naively think that the things they list are somehow uncontroversial, or that it was the sophisticated empirical work that persuaded the people that now believe them. Professor Roberts' question goes right over their heads.

[mgm]

Let me just add that the greatest thinker to write on this subject is G. L. S. Shackle. Taleb's thesis in the Black Swan is great because it comes directly from the work of Shackle, and he makes no attempt to conceal that fact. Shackle's books are referenced in The Black Swan, and in his podcast with Russ Roberts Taleb points out that Shackle has been his biggest influence, but Roberts seemed unwilling to question him further on that point! This is an interesting subject to study, but it is deeply philosophical. You can't just read Mises, Hayek and Vernon Smith and think that you understand the contested applicability of statistical analysis. You have to read what philosophers have to say about time and metaphysics, and this would require students to dip into the books of Bergson, Heidegger, and Spinoza. But who has time for that? Mises always fashioned himself a philosopher, but no honest scholar can propose to deal with metaphysical topics in under 3 pages, which is what he did in books like "Theory and History" and "The Ultimate Foundations of Economic Science." It is all very intimidating, and that is why I gave up on academics. It is very diffficult to understand a topic, because it requires so much reading. The short-cut is to buy into an ideology, and you can find that in Austrian economics. You can go back months in this blog, for example, and find that Pete Boettke basically just repeats the same things over and over again. That is not a bad thing, but it is an example of the benefits of ideology vis-a-vis real erudition.

[Zac Gochenour]

Prof Boettke writes, "But what we cannot do is "test" theories with statistical tests."

If the observed world does not match up with theory, then the theory is wrong. So yes, empirical study (including statistical tests) can "test" theory. The classic Austrian Business Cycle Theory, for instance, was tested against the evidence and is found to be false. I cannot understand why Austrians continue to say things like "you cannot test theory" instead of working on a viable synthesis supported by empirical evidence, (http://www.marginalrevolution.com/marginalrevolution/2005/02/austrian_econom.html)

Let me be fair: that empirical studies are rarely, if ever, conclusive is a point that should be well taken in the profession and cause economists to present their empirical findings with the appropriate degree of humility as well as shift their focus. In many cases, empirical studies are overvalued by many (particularly when the results support their prior beliefs), and they are often presented as far more conclusive than they are.

@Adam, it does not seem you actually read the responses from Cowen and Caplan.

As to the comment from Redland Jack that "it takes a theory to defeat a theory" - huh? It takes an example to defeat a theory. If you posit a theory of motion that says F=mv, I do not necessarily need to show you that F=ma to prove you are wrong, just that F!=mv. Or, if you say all penguins are red and I show you a black penguin, I have disproved your penguin theory, I do not necessarily need to say all penguins are black. For any phenomena of interest, ideally a correct theoretical model is developed at some point, but that's something different. And once you have two competing theories, which one "defeats" the other is the one that is confirmed by observing the world (empirical study).

[Lee Kelly]

But what we cannot do is "test" theories with statistical tests. - Peter Boettke
A theory is only testable if you decide to make it testable. In other words, there is always a way to re-interpret recalcitrant evidence. Each individual must decide for themselves when to categorise something as "refuted", because logic isn't going to force them into anything. That's how we learn.

If the mere logical possibility of some other true explanation were enough to prevent testing, then there would be no science at all, never mind economics.

But yeah, statistics is mostly nonsense, in my opinion. The problem is not the mathematics, or even the data, but more often the statisticians understanding of his role, and interpretation of his results.

[Lee Kelly]

Zac,

if you say all penguins are red and I show you a black penguin, I have disproved your penguin theory, I do not necessarily need to say all penguins are black.

A magician once showed me how to make a rabbit disappear inside a hat, so I guess that disproves my theory that all rabbits cannot disappear inside a hat, doesn't it? Every observation is theory-laden, and not just a little bit theory-laden, but infinitely theory-laden.

It always takes a theory to refute a theory.

[Roger Koppl]

"parading as "scientific" for pretending to achieve something which in fact cannot be achieved in practice, results in little more than pseudo-science"

Right! That's one reason I think computable economics has such potential. It makes it harder to avoid the fact of our deep ignorance and the implications of that fact. Epistemic humility is not in vogue, but if you take algorithmic information theory seriously, you end up in epistemic humility. I wish more mathematical economists would have a serious look at it.

[Nikolaj]

Zac,

do you think it's possible to test the hypothesis that with growing supply, ceteris paribus, diminishes the price of commodity. Or that increased minimal wage, ceteris paribus increases unemployment?

If not, please do not talk that economic hypothesis can be empirically tested? Or do you think that some hypothesis can and some cannot be tested that way? Where is the line?

[Peter Boettke]

Zac,

May I suggest that you read Leland Yeager's discussion of Tyler's Risk and Business Cycles, and/or if you can track it down Kevin Hoover's reaction to the empirical arguments in the book.

Tyler's book has some very imporatant aspects of it, but the evidence provided is not as decisive as he wants it to be, and the argument is not as definitive as he wants it to be.

This is not to say Austrian theory is right because one of the critics of the Austrian theory is not as persuasive in his criticism as he (and some others) claim.

But it does mean that it is not unreasonable to hold ABCT after Cowen.

How can this be?

Well because the issue of theory construction and testing is complicated and interpreting counter evidence is not straightforward.

I don't know how much you have read in the philosophy of science, but the arguments in the literature are a bit more cautious about hypothesis testing and the act of refutation than you seem to suggest in your reply.

Pete

[Arare Litus]

"Does the fact that statisical tests cannot provide unambiguous refutations of economic theory imply that we learn nothing from statistical analysis?"

Is this not a "straw man"? Other than mathematical proofs, or other logical deductions, which single small study can be unambiguous? And even there, the starting point - the broad assumptions and axioms one starts with - must somehow defended, and can be flawed (i.e. ambiguous) and for more complex systems the starting assumptions will likely be arguable.

The question seems to be one of compelling argument, and diminishing returns on finding increasingly subtle effects with increasingly "high powered" techniques.

In a sense one *can* make unambiguous refutations with pure empirics - if one cannot show significant empirical effects, this refutes the theory: not in principle, but in *magnitude/importance* - one can rank the magnitude of the effect, and drop some theoretical effects as being of little measurable consequence.

As an example "Friedman's law" is a rule of thumb that discusses significant gains from competition, stating that ~2X gains are to be expected from private alternatives. That is a large and compelling effect, one that should really be at the fore front of discussions of economics - I stumbled on this "law" well after becoming interested in economics, and I find this strange. This suggests that the logical argument and mechanism behind competition is a very important effect, one that should be considered deeply. If instead one is looking at minimum wage laws the evidence is much more poor - one can always make arguments that in principle this is important, but we can say fairly accurately that *given the structure of our current society* this is a small effect. The focus then should be on the effects the swamp out this principle, where one expects the problems to become large (i.e. a minimum wage set at 1/2 of the top wage in a nation would have huge effects, and few would argue with that - but where is the cutoff?), etc.

The question is simply one of compelling argument, weak effects showing that the "force" of the theoretical principle being investigated is blunted by other effects, or perhaps is weak to start with. Statisitical arguments can "calibrate" the significance of a theoretical principle.

As a final note - **no single piece of evidence, or single study**, ever is compelling enough to demonstrate the vitality or papacy of a theory. The theory will be slightly modified, or the range slightly reduced, in order to address very compelling counter evidence, but rarely would one expect to see a collapse of a theory in the face of some evidence. This points to another way that statistical studies are quite useful - if the measured effect is weak, and yet someone finds that compelling evidence, I have information on that person: they are somewhat intellectually dishonest and do not confront their biases.

[Greg Ransom]

Samuelson admits that his understanding in the 1940s and 1950s and 1960s of how econometrics would work to confirm theories and decide issues in economics was a complete bust -- he even admitted that his whole picture of how economics as a science was to work rested on this understanding.

It's remarkeable that more economists aren't aware of this self recognized failure of the 1940-1960 picture of "economics science" -- a picture which still controls most editorial decisions at the "top" economic "science" journals -- an which still controls most of graduate education and the tenure process.

[Greg Ransom]

In 1975 Friedman was still lecturing Hayek about how statistical tests were going to the thing which decided questions about economic issues both in policy and in theory.

... and he was still utterly failing to have any understanding of Hayek's explanatory strategy and scientific program.

When will the Hayek correspondence be published. He's been dead for more than a decade and a half and this correspondence is publicly available at Stanford.

It seems a shame that this stuff is locked way from anyone who can't afford a plane ticket and hotel expenses.

[TGGP]

Rather than looking for "conclusive" tests it might be better to consider them evidence which (combined with priors) a Bayesian comes up with updated posteriors.

mgm, are you Matthew Mueller? It's a terrible shame your blog is gone, I try to link to old posts all the time before remembering it isn't there.

[Peter Boettke]

TGGP,

To get an idea of the issues I am trying to raise related to the Duhem-Quine thesis, look at Giorgio Fagiolo1, Alessio Moneta,and Paul Windrum, "A Critical Guide to Empirical Validation of Agent-Based Models in Economics: Methodologies, Procedures, and Open Problems," Computational Economics, 30 (3) October 2007: 195-226.

Also look at my entry on Ludwig von Mises in the Handbook of Economic Methodology, where I suggest a reconstruction of Mises's argument in light of post-positivist philosophy of science, including Duhem-Quine. An earlier attempt at methodological reconstruction that is notable is Rizzo and his examination of Lakatos and Mises.

The traditional argument of Bayesian updating doesn't quite work.

I also recommend to everyone the book by Frydman and Goldberg, Imperfect Knowledge Economics (Princeton, 2007), and in particular their critique of rational expectations economics.

Pete

P.S.: Matt --- why do you insist that I am repeating and that it is ideology that enables this? First, I am not sure I agree with your empirical claim about repeating, but lets say it is true. So, second, why is it ideology and not limited cognitive skills that prevent me from coming up with the novel insights that you (I assume) are in possession of? Can you tell me where my biggest gaps in understand are so I might learn --- I mean that in all sincerity.

P.S.S.: Greg --- Hayek archives are now (if I am right) at Duke as well on micro-film and part of the new Center Caldwell has established there. I am anxious to see this correspondence as well, the hints we have seen imply it would entail a critique of the enterprise of macroeconomics.

[Lee Kelly]

The Duhem-Quine thesis is the nub of the problem. Figuring out how knowledge can be without it is the solution. And probability, Bayesian or otherwise, doesn't matter to that answer.

[Current]

Zac: "If you posit a theory of motion that says F=mv, I do not necessarily need to show you that F=ma to prove you are wrong, just that F!=mv."

As a physical scientist I can't help commenting on this.

The arguments the pre-newtonian physicists had over force are more complicated than that.

Any particular part of Aristotelian physics sits within the whole. When a physicist wrote that "F=mv" they intended to do something with the F. (I don't think the Aristotelians said precisely F=mv). Another equation followed on to use that F. It was at this sort of stage the earlier physicist went wrong, their derived variables were not universally useful.

Newtonian physics is similar. The achievement of Newton though was to make the derived variables that are much more useful. An understanding of force as F=ma is useful because it fits in with the other Newtonian laws. The equations for summing forces work.

Newton convinced people of the whole system. This was the only way that people could be convinced. If you drop any one of Newton's laws the others don't make sense.

So, this is nothing like as plain and obvious as you describe it to be.

[Current]

Matt Mueller - Can you put your blog back online?

Then we may understand why we need to read these philosophers.

[Greg Ransom]

I haven't read it all -- and haven't seen the new additions -- but the most interesting things I saw were there correspondence between Hayek & Mises and between Hayek & Popper, and the correspondence between a handful of leading economists in the 1930s on the technical problems with the "average period of production" construction -- a landmark discovery in the history of the micro/macro relationship. I think I posted about this stuff years ago on the Hayek-L email list.

Peter writes:

>>Hayek archives are now (if I am right) at Duke as well on micro-film and part of the new Center Caldwell has established there. I am anxious to see this correspondence as well, the hints we have seen imply it would entail a critique of the enterprise of macroeconomics.<<

[Peter Boettke]

Greg,

I have worked in the Hayek archives at Hoover three times, twice related to my 3 volume reference work on Hayek that was published in 1999. But I found the most exciting stuff to be the correspondence with Mises (a great letter from Mises to Hayek on the 1940 AEA meeting is extremely memorable). Also notable is Mises's first reaction to the free market activist in the US -- not favorable. But I also though the Machlup-Hayek file perhaps had the most stuff in it. The letters with Polanyi, especially over MPS are fascinating. Other great facts to discover in the archives include Hayek's course descriptions at Chicago --- first course he taught was a philosophy of science course.

Pete

[Greg Ransom]

Pete -- the Machlup file is interesting, the most interesting thing being the round-robin on the average period of production issue which Machlup organized.

[Greg Ransom]

Pete -- I've got copies of most of this correspondence in my personal files.

Most of the leading scientists at Chicago participated in Hayek's philosophy of science course.

Fermi, for example, gave a presentation. As did Milton Friedman.

[custom dissertation]

Excellent Blog every one can get lots of information for any topics from this blog nice work keep it up.

thanks

[Auto Masini Inchirieri]

not the first time when i hear this story and i incline indeed to say that is true