I enjoyed a lot reading a "discussion" prepared by Maury Goodman on the value of "confidence level", discovery thresholds, and what physicists believe or not. If you are a HEP physicist and you want to widen your horizons on the value of statistical claims in experimental results, you are bound to read it. But you might find it thought-provoking and enlightening even if you are a layman, provided you can use three neurons in a row.

A few random excerpts should convince you to read the whole piece:

Manjib:  No reasonable high energy physics will believe a two sigma effect.

Karana:  I’ve believed several two sigma effects.  I canname several (first SAGE indication of a solar neutrino deficit in Gallium, nutau appearance in Super-K, MINOS’ measurement of theta-13).  I believed the LMA before KamLAND when global fits favored the LMA by 2 sigma.

Manjib:  I calculate that the chance of that distribution is 6e-04, less than the claimed non-zero value.

Karana:  Your calculation is meaningless.  There are an infinite number of tests of the null hypothesis.  Your test is a-posteriori.   I could make a similar calculation about every right result

Karana:  Let’s say someone claims a three sigma effect.  It could be 1) something new; 2) an unlikely statistical fluctuation; 3) A systematic effect that was not taken into account, or 4) and most frequently, it wasn’t an (a-priori) three sigma effect.

Manjib:  People can handle that with trials factors.

Karana:  It is virtually impossible.   People make multiple plots and cuts, and they are usually correlated.  People often combine two uncorrelated probabilities P1 and P2 as the product P1*P2.  That is an unreasonable test of the null hypothesis.  A more reasonable one is to limit the product, and that probability is P1*P2 *(1-ln[P1*P2])