©Deirdre Nansen McCloskey | COPYRIGHTED MATERIAL
Coelho on Ziliak and McCloskey, The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives
Reviewed for EH.NET by Philip R.P. Coelho, Department of Economics, Ball State University.
Published by EH.NET (June 2008)
Filed under academic interests [statistical significance]
Stephen T. Ziliak and Deirdre N. McCloskey, The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Ann Arbor, MI: University of Michigan Press, 2008. xxiii + 287 pp. $25 (paperback), ISBN: 978-0-472-05007-9.
[T]his is an important work that deals with a major problem of statistical analysis in the social, medical and physical sciences. If you are not aware of the problem, you should be. If you are aware of the problem, this book is a good compendium of the problem, real-world issues, and the historical milieu in which the cult of significance evolved.Ziliak and McCloskey have written a fine book of 24 chapters, a reader's guide, and preface. They write for an "implied" audience of "keeper[s] of numerical things" to persuade them that: "Statistical significance is not the same thing as scientific finding. R-squared, t-statistic, p-value, f-test, and all the more sophisticated versions of them ... are misleading at best" (p. xv). The authors have accomplished this and more in a well-researched, written and documented book. The authors start with a Contents section that contains a brief précis of each chapter's contents. The précis is an imaginative and highly useful resurrection of nineteenth- and early twentieth-century practice; a reader-friendly technique that can be usefully employed today. An examination of the Contents is revealing. Directly opposite the beginning of the Contents section is a photograph of William Sealy Gosset, the "Student" of the "Student t [commonly truncated to the t] distribution." In conjunction with an exposition of why statistical significance is very different from importance or scientific (economic/historic or whatever) significance, they have written a paean and brief biography of Gosset. I am convinced that Gosset was a noble and modest man, a great statistician and intellect who was shabbily treated by his supposed friend and colleague, R.A. Fisher. He has also been neglected by historians of science and statistics; Gosset deserves to be memorialized, and certainly warrants biographies. That being said, combining two fine books on disparate subjects (statistical methodology and historical biography) does not make an even better book. Ziliak and McCloskey emphatically make their argument against the use of statistical significance as a proxy for importance in Chapters 1 through 5. The basic difficulty with statistical significance is that it has been permeated with the mathematical ethos of certainty. A mathematical "proof" implies a truth (Gödel's Theorem is conventionally ignored) that is invulnerable to time, space, and reality; it is an abstraction that cannot be falsified using mathematical epistemology. Relevance, economic importance, and any metrics other than mathematics are beside the point. Scientific assertions should be confronted quantitatively with the world as it is or else the assertion is a philosophical or mathematical one, meritorious no doubt in its own terms but not scientific. ... The problem we are highlighting is that the so-called test of statistical significance does not in fact answer a quantitative, scientific question. Statistical significance is not a _scientific_ test. It is a philosophical, qualitative test. It does not ask how much. It asks "whether." Existence, the question of whether, is interesting. But it is not scientific (pp. 4-5). In the absence of some measure of how big an effect is, the existence of an effect reveals nothing of importance about the world of observational reality. Ziliak and McCloskey highlight the danger and corruption that flow from the overwhelming importance placed upon statistical significance (a measure of existence or lack thereof) by using the tragic example of Vioxx. Vioxx was a formulation developed by Merck designed to combat pain. In clinical trials Vioxx had about five times the number of fatalities as a generic version of a control drug (naproxen). Because the number of observations did not reach the appropriate size, the 5 to 1 ratio of excess fatalities caused by Vioxx was deemed statistically insignificant. (Merck may have reduced the actual number of fatalities by manipulating the data [p. 29].) Merck's ethics and the clinical/scientific studies of Vioxx that were sponsored by Merck have been sharply criticized by the scientific and journalistic establishments. (See the _Wall Street Journal_, April 16, 2008, p. B4) By simply discarding some fatalities (on dubious grounds) the 5 to 1 disadvantage in mortality became statistically insignificant in the submitted trials, and Vioxx was marketed. It was literally a fatal error that cost Merck billions of dollars and caused a number of needless deaths. In the absence of any measure for costs or benefits the standard use of an acceptance/rejection rate arbitrarily set at five percent is mindless and/or non-scientific. Five percent of a very large number (say the world's human population or the GDP of the United States) is still a large number; and conversely one hundred percent of a minuscule number is still minuscule. These are not Nobel Prize winning observations; regardless they are ignored by researchers in a depressingly large number of disciplines. Ziliak and McCloskey document (Chapters 5 through 16) the standard statistical conventions that predominate in publications in a number of journals and disciplines. The results do not inspire confidence in the scientific competence of the editors and practitioners. Typically overweening emphasis is placed on the existence of an effect (statistical significance) while the magnitude of the effect is either barely noticed or entirely ignored. ... [Despite] quibbles; this is an important work that deals with a major problem of statistical analysis in the social, medical and physical sciences. If you are not aware of the problem, you should be. If you are aware of the problem, this book is a good compendium of the problem, real-world issues, and the historical milieu in which the cult of significance evolved.
Philip R.P. Coelho is a professor of economics at Ball State University. He has written on and is continuing his study of long-run economic growth and the impact of parasitic diseases and biology upon economic growth, history and development. His papers have been published in the Journal of Economic History, the American Economic Review, Explorations in Economic History, Economic Inquiry, Southern Economic Journal, and other outlets.
Copyright (c) 2008 by EH.Net. All rights reserved. This work may be copied for non-profit educational uses if proper credit is given to the author and the list. For other permission, please contact the EH.Net Administrator (administrator at eh.net; Telephone: 513-529-2229). Published by EH.Net (June 2008). All EH.Net reviews are archived at http://www.eh.net/BookReview.
ORDER INFO In The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives (University of Michigan Press, 2007), McCloskey and Stephen Ziliak show that statistical signficance, though routine in economics, psychology, medicine, and increasingly in the courts, is nonsense when used alone (as it almost invariably is), and has been known to be nonsense in this partial use since it was invented — often by the very inventors, such as Edgeworth and Gosset (the "Student" of Student's-t).
Deirdre N. McCloskey is Distinguished Professor of Economics, History, English, and Communication at the University of Illinois at Chicago;
and Professor of Economic History, Gothenburg University, Sweden
Prudentia
Luis Abrigo, Editor
www.deirdremccloskey.org - website designed & built by Susan B. MacDonald - statcounter