Exact tests

An exact test is a nonparameteric test in which the significance levels are calculated without making any assumptions about the probability distributions that generated the observed data values.

For example, consider the two-sample t-test. Under the “null hypothesis” that there is no difference between the two samples, the probability is calculated by assuming that the data values come from Normal distributions with equal means and variances. This can be justified by the observation that, in practice, many samples do seem to come from Normal distributions. A more rigorous justification is based on the fact that the test assesses the difference between the means of the two samples, and means of samples from most distributions tend to become Normally distributed as the sample size becomes increasingly large. Statistically these distributions are said to be asymptotically Normally distributed. So the probability is based on the asymptotic properties of the test (although it usually works well with smaller samples too).

The exact alternative to the conventional t-test makes the assumption that the observed data are representative of the full population of possible data values, and calculates the significance level by considering all the possible ways in which the values could have been allocated to the two samples (including the allocation that actually occurred). The t-statistic is calculated for all of these possibilities, and the probability of the observed data is calculated by seeing where its t-statistic occurs within the full set of statistics. So, for example, it would be significant at the 1% level in a one-sided test if its statistic was in the largest 1% of the statistics.

Genstat can produce exact probabilities for most nonparametric tests, including the Mann-Whitney test, Wilcoxon test, binomial test, sign test, Poisson test, McNemar test, Cochran’s Q test, Kendall’s τ and Spearman’s rank correlation statistic, as well as Fisher’s exact test for counts in 2×2 tables (which was the original exact test). For some of the tests, it may not be feasible to calculate the exact probability with very large samples. So there the probability will be based on the asymptotic properties of the test as discussed earlier. However, these are exactly the situations where the asymptotic probabilities can be relied on.

Genstat can also do permutation tests for t-tests, analysis of variance, Steel’s test, regression analyses and the analysis of similarities. You specify how many random permutations to make and, if that is greater than the number that is possible for the data set, the exact test is done instead.

The relevant procedures are as follows: