“The Cult of Statistical Significance” by Stephan T. Ziliak and Deirdre N. McClosky

I am disappointed with this book. I agree almost entirely with their theses. They wrote too much like me when I am in a bad mood. I attended Berkeley when both Neymann and Blackwell were there. Significance tests were taught in stat courses but only with the requirement the student know the meaning of the thresholds. It was clear to the careful student that such tests were arbitrary and should not supplant the numeric values. Fischer was revered at Berkeley for his vast contributions to probability theory and evolution.

I think the book should have identified an audience. The current book is too much like a math book where all the equations were deleted at the last minute. There is much text that makes sense only in the context of formulas. Here are some relevant audiences:

On page 9 Savage is quoted who speaks of using statistics to choose alternative courses of action. This seems indeed to be the thrust of the book. I completely agree.

There is in this review a section titled: “Should Science Govern Choices?”. Well science should provide information to help us make decisions. Up or down information on a drug is insufficient guidance. That is the sort of guidance our laws require our regulators to make. The above review is the only one of about 10 that raises problems with the book.

I think the source of the impetus towards significance testing is “First know.”, then we will decide what to do. It is a philosophical stance — a logical imperative — perhaps a left brain thing. Our language does not provide a convenient way of reporting a statistical result other than saying that the statistics confirm that ... . Reporting the estimated size of the effect is good but one must also include the sampling error.
Science deals with non-numeric hypotheses. Not all science is numeric. Numbers are necessary intermediaries in just about all laboratory yield. Theories must be as simple as possible and while there are numbers in the formulae of our theories, there is non-numerical content too. When we say that the maximum speed of a particle is the speed of light, there are no error bars in the theory. There are error bars in laboratory yield. It is considered a failure of particle theory, by some, that the masses of particles are mere numbers, with error bars.

When we do an experiment to see if the mass of the neutrino is zero we are not estimating the mass of the neutrino. We have an a priori estimate of the probability that it is zero.

When we look for magnetic monopoles we are not trying to estimate their abundance. If there is in the universe even one monopole some of our theories are in deep trouble. Having looked at some 1035 particles and noticed than none were monopoles is an important result which increases some people’s subjective probability that the abundance is zero.

None of these points detracts from the thesis of the book—I quibble only with his characterization of science.

On page 144 Ziliak quotes Fisher deriding the idea that cost of measurement should be considered. Fisher wants to know whereas Gosset wants to decide what to do.

The notion that one might use statistics to decide what to do without first deciding ‘what is so’ seems to some as crude and uncivilized. It is indeed both, but yet the best that we can do with the raw statistical results. It is indeed a shortcut past epistemology which is in the spirit of biological development of epistemology outlined in the anthology “Evolution of Epistemology”. We were deciding what to do long before we began to think we knew what is so. Deciding what to do based on what is so opens up many new survival strategies. Sometimes the old ways are better however.

I think there is part of the brain, the ‘knower’, that takes umbrage at being left out of the loop. The reasons cited by Fischer and quoted in the book are just along these lines. Fischer demands first to know, then he will decide.