For the benefit of future classes, and of alumni who need reminders:

1.    Random does not mean haphazard or accidental. Asked what ‘random’ means, students reply ‘no pattern,’ or something of that nature. Your sample ain’t random just because you say it’s random! Show how you’ve taken pains to ensure the sample meets the true definition of randomness, namely that each member of the studied population has the same probability of being selected for the sample.

o   If the sample departs from randomness, show why the departure is small, and not material to answering the research question.

o   Never speak of a ‘random sample of experts’: You choose experts for their special expertise, not for their typicality. Expert panels must be chosen expertly, not randomly. Their responses may be subject to descriptive statistics, but never to inferential statistics.

2.    Specify the population. Oh, how students forget that statistical inference is reasoning from a sample to a specified population. As a late replacement for his dissertation examiner, I listened to a candidate’s oral defense. He showed a sophisticated analysis and a big n, and claimed support for his hypothesis at a high p level. I asked, What is the population? He replied, Huh?

o   A sample must be drawn from a population. If you can’t say what the population is, your sample is crap. Then the dissertation is crap.

3.    And specify the sampling frame and the sampling plan. If the population is ‘shoppers at Horton Plaza Mall in San Diego,’ the frame might be ‘people entering Horton Plaza through the south doors between 11a.m. and 6p.m. next Tuesday and next Saturday.’ The plan is ‘Interview every 25^th person to enter, with at least 300 interviews to be completed.’

o   You give your undergrad assistant your clipboarded questionnaire, along with the sampling plan. Face it: If the 25^th person is a smelly guy with a beer gut, your assistant will give him a pass and instead interview #26, who is an attractive member of the opposite sex. Thus violating randomness.

o   Solution: Send pairs of assistants. One will do the interviewing while the other enforces the sampling plan.

4.    Don’t confuse a census with a sample. If you measure every member of the population, you’ve done a census, not a sample. No statistical inference is needed, or appropriate. You’re not making inference from sample to population, because you’ve ‘sampled’ the entire population!

o   One student measured every country in his well-specified population of interest. He then performed statistical tests. ‘Why?’ I asked. ‘Well,’ he said, ‘it wouldn’t look like much of a dissertation if I didn’t have statistical tests.’ BRRRP! Buzzer! Wrong answer!

5.    Don’t forget nonresponse bias. You invited 1000 people (randomly selected from a specified population, natch) to your surveymonkey.com site. 700 of them completed the questionnaire. Congratulations, this is actually a very good response rate for management research. However… Why do you believe the responses of the 300 non-responders – if they had responded – would have been similar to the responses of the 700? A passable dissertation must include an answer to this. There are techniques for estimating nonresponse bias.

o   One student got a response rate like this, questioning businesspeople in his country about their interest in interacting with foreign businesspeople. Doesn’t it stand to reason, his examiners asked, that people not interested in foreigners would not be interested in a questionnaire about their interest in foreigners? Is this not a red flag for nonresponse bias?

o   The student got a ‘conditional pass’ and was required to come back later with written estimates of nonresponse bias and its impact on his results.

6.    Too many hypotheses? Testing many hypotheses on one too-small sample will almost guarantee at least one false positive or false negative. As a rule of thumb, you need a perfectly random sample of n=30 – with perfect controls on non-treatment effects – to get a good estimate of just one parameter.

o   If you must test multiple hypotheses on multiple quantities, scale up your n accordingly.

o   There’s no hard and fast rule. But you will need a bigger sample than you think you need.

7.    Assuming normality? Take care. There are two issues here: Normality of the quantity you’re measuring, and normality of measurement errors. There are tests for normality.

o   The world financial crash occurred (among other reasons) because experienced financial managers believed returns on assets would follow a Gaussian distribution. In fact, a longer-tailed distribution (e.g., a power law) was needed to catch the true probability of an extreme event of the kind that did, in fact, happen. For more on this, see The Black Swan.

o   Your research model is y = e^x + ε . You’re more comfortable with linear regression, so you transform it to log_e y = x + ε’ .  Regression depends on Gaussian-distributed error terms. Is it ε or ε’ that’s normally distributed? Or neither? (It almost surely won't be both.) Find out before you regress!

8.    Justify your use of a statistical test by understanding its mathematical foundation and its match to your research question. Students, and some published scholars, tend to wave away this step with a magic gesture, saying ‘So-and-so (1998) used this test in a similar situation.’ Citing a prior use of the test is not a categorical no-no. But it’s better to show you know why it was used.

o   I guarantee you, Professor So-and-so’s research question was quite different from yours. Just because his referees saw the logic of using the test doesn’t mean yours will.

9.    The matter of replicability. Is your 90% significance level ‘in principle’ only, or possibly relevant in practice? Compare your management research to that of a medical investigator in a mouse lab: That investigator knows mouse physiology will not suddenly change tomorrow. Tomorrow’s repetition of the mouse experiment will show a result much the same as today’s. The management researcher knows the business environment will change tomorrow. The prospect of replicating your management study on many independent samples under the same conditions is nil.

o   If an answer to your research question is likely to be very ephemeral, don’t do the study.

§ A possible exception: If you’re studying an important one-of-a-kind event, for example a nation’s adoption of the Euro, and its effect on consumer prices or attitudes.

o   If you think you’ve revealed a management principle of lasting value, say why you think so. Many extra points if you can convince the reader of the study’s practical replicability.

10.The dangers of SEM, multi-level models, etc. The worst reason in the world to do something is just because you can. These highly complex statistical procedures are possible only because of the power of today’s computers. That alone is no reason to use them; you’ll need a much better reason. SEM, factor analysis and the like require judgment on the researcher’s part. (They are not just plug-in formulas.) Can a novice researcher exercise that judgment?

o   You write a thesis to show you can do a supervised research project – not to display your virtuosity.

o   Using a technique without showing complete understanding of it (see #8 above) won’t persuade your examiners to award the degree you seek.

11.Non-sampling errors will almost always be much bigger than sampling errors. Professors teach statistical math because it’s easier than teaching research logic. Sampling error, or reliability, is the significance level or p-value of your test. Errors that you make in formulating an unambiguous research question and measurable hypotheses, framing your study, interpreting results, and so on, as well as errors described in #1 through #4 above, are non-sampling errors.

o   Avoiding them requires as much or more attention than doing the tests properly.

12.The p-value is not the probability of H_o being true. Or false. The hypothesis statement is a matter of fact, not of probability. Either it is true or it isn’t, out there in the real world. Makes no sense to speak of the probability of it being true. Though they shouldn’t have, the statistical forefathers used ‘p’ sometimes to denote a probability, and sometimes to denote a quantile of an error distribution. They thought you could keep the two straight. Don’t prove them wrong!

o   Ditto for the significance level α. α is not a probability – of the truth or of anything else.

Modern statistical inference is one of the top intellectual achievements of the 20^th century, and one of history’s greatest advances in applied epistemology. However, deciding a hypothesis at the 95% level (which is almost impossible in management studies anyway) only means that if you repeated the experiment 100 times on 100 independent samples, you would make the same decision approximately 95 times.

In other words, you still don’t know whether the hypothesis is true or not. All you have done is quantify your confidence in its truth or falsity. Interpret your data with the appropriate modesty and do not use the word ‘proved.’