Exact tests

An exact test is a nonparameteric test in which the significance levels are calculated without making any assumptions about the probability distributions that generated the observed data values.

For example, consider the two-sample t-test. Under the “null hypothesis” that there is no difference between the two samples, the probability is calculated by assuming that the data values come from Normal distributions with equal means and variances. This can be justified by the observation that, in practice, many samples do seem to come from Normal distributions. A more rigorous justification is based on the fact that the test assesses the difference between the means of the two samples, and means of samples from most distributions tend to become Normally distributed as the sample size becomes increasingly large. Statistically these distributions are said to be asymptotically Normally distributed. So the probability is based on the asymptotic properties of the test (although it usually works well with smaller samples too).

The exact alternative to the conventional t-test makes the assumption that the observed data are representative of the full population of possible data values, and calculates the significance level by considering all the possible ways in which the values could have been allocated to the two samples (including the allocation that actually occurred). The t-statistic is calculated for all of these possibilities, and the probability of the observed data is calculated by seeing where its t-statistic occurs within the full set of statistics. So, for example, it would be significant at the 1% level in a one-sided test if its statistic was in the largest 1% of the statistics.

Genstat can produce exact probabilities for most nonparametric tests, including the Mann-Whitney test, Wilcoxon test, binomial test, sign test, Poisson test, McNemar test, Cochran’s Q test, Kendall’s τ and Spearman’s rank correlation statistic, as well as Fisher’s exact test for counts in 2×2 tables (which was the original exact test). For some of the tests, it may not be feasible to calculate the exact probability with very large samples. So there the probability will be based on the asymptotic properties of the test as discussed earlier. However, these are exactly the situations where the asymptotic probabilities can be relied on.

Genstat can also do permutation tests for t-tests, analysis of variance, Steel’s test, regression analyses and the analysis of similarities. You specify how many random permutations to make and, if that is greater than the number that is possible for the data set, the exact test is done instead.

The relevant procedures are as follows:

    FEXACT2X2 does Fisher’s exact test for 2×2 tables
    MANNWHITNEY performs a Mann-Whitney U test
    PRMANNWHITNEYU calculates probabilities for the Mann-Whitney U statistic
    WILCOXON performs a Wilcoxon Matched-Pairs (Signed-Rank) test
    PRWILCOXON calculates probabilities for the Wilcoxon signed-rank statistic
    BNTEST calculates one- and two-sample binomial tests
    SBNTEST calculates the sample size for binomial tests
    PNTEST calculates one- and two-sample Poisson tests
    SIGNTEST performs a one or two sample sign test
    SSIGNTEST calculates the sample size for a sign test
    MCNEMAR performs McNemar’s test for the significance of changes
    SMCNEMAR calculates sample sizes for McNemar’s test
    QCOCHRAN performs Cochran’s Q test for differences between related-samples
    KTAU calculates Kendall’s rank correlation coefficient τ
    SPEARMAN calculates Spearman’s rank correlation coefficient
    PRSPEARMAN calculates probabilities for Spearman’s rank correlation coefficient
    TTEST performs a one- or two-sample t-test
    APERMTEST does random permutation and exact tests for analysis of variance
    CHIPERMTEST performs a random permutation test for a two-dimensional contingency table
    STEEL performs Steel’s many-one rank test
    RPERMTEST does random permutation and exact tests for regression or generalized-linear-model analyses
    ECANOSIM performs an analysis of similarities (ANOSIM)

Updated on December 1, 2017

Was this article helpful?