1. Home
  2. VPERMTEST procedure

VPERMTEST procedure

Does random permutation tests for the fixed effects in a REML analysis (R.W. Payne).


PRINT = string tokens Controls printed output (prwald, criticalwald, ownstatistics, monitoring); default prwa, crit
NTIMES = scalar Number of permutation samples to make; default 99
NRETRIES = scalar Maximum number of extra samples to take when some REML analyses fail to converge; default NTIMES

Model formula defining any blocking to consider during the randomization; default none

EXCLUDE = factors Factors in the block formula whose levels are not to be randomized
SEED = scalar Seed for random number generation; default 0 continues an existing sequence or, if none, selects a seed automatically
WMETHOD = string token Controls which Wald statistics are used (add, drop); default add
OWNMETHOD = string token Type of test required for own statistics (twosided, greaterthan, lessthan); default twos
CIPROBABILITY = scalar Probability level for the confidence interval for own statistics; default 0.95


SAVE = REML save structures Specifies the (REML) save structure of the original analysis; default * uses the SAVE structure from the most recent REML analysis
WALD = pointers Wald statistics saved in a pointer with a variate for each term
PRWALD = pointers Critical values for Wald statistics saved in a pointer with a scalar for each term
CRITICALWALD = pointers Saves a pointer with variates for the 5%, 1% and 0.1% significance levels containing the corresponding critical values for the fixed terms, obtained from the quantiles of the Wald statistics from the permuted data sets
NNOTCONVERGED = scalars Saves the number of permutations whose REML analysis failed to converge
OWNDATA = pointers Data required to calculate own statistics
OWNOBSERVEDVALUES = variates Saves observed values of the own statistics
OWNPROBABILITIES = variates Saves probabilities for the own statistics
OWNESTIMATES = variates Saves estimates for the own statistics
OWNSES = variates Saves standard errors for the own statistics
OWNLOWERCIS = variates Saves lower values of the confidence intervals for the own statistics
OWNUPPERCIS = variates Saves upper values of the confidence intervals for the own statistics
OWNSTATISTICS = pointers Saves the own statistics obtained from the permutation samples, in a pointer with a variate for each statistic


VPERMTEST performs a random permutation test for fixed effects in a REML analysis. The SAVE parameter can supply the save structure from the original analysis; if this is not set, the tests are done for the most recent REML analysis.

The test probabilities are calculated by taking the proportion of Wald statistics in the permutation samples that are larger than the observed Wald statistic of each fixed term. (As a result these should not suffer from the bias that is found in the probabilities for the Wald statistics themselves, which tend to be too low.) The WMETHOD option controls whether the Wald statistics are obtained from the table where terms are added sequentially (the default), or from the table where suitable terms are dropped from the full fixed model. Note that, if you use the table where terms are dropped, the only terms that can be tested are those that are not marginal to any other term in the fixed model: for example, the main effect A cannot be tested if the model contains an interaction, such as A.B.

The NTIMES option defines how many random permutations to perform; by default there are 99 (as well as the “null” permutation where the data keep their original order). The NRETRIES option specifies the maximum number of extra samples to take when some REML analyses fail to converge; the default is to use the same number as specified by NTIMES. The SEED option allows you to specify the seed to use for the random-number generator that is used to construct the permutation samples. The default, SEED=0, continues the sequence of random numbers from a previous generation or, if this is the first use of the generator in this run of Genstat, it initializes the seed automatically. If NTIMES exceed the maximum possible number of permutations for the data, an “exact” test is performed in which every permutation is used once. This is feasible only for small datasets. There are n! (n factorial) permutations of n units: 3!=6, 4!=24, 5!=120, 6!=720, 7!=5040, 8!=40320, and so on. The NNOTCONVERGED parameter can save the number of samples whose analyses did not converge, in a scalar.

If the data are from a designed experiment, you may need to use the BLOCKSTRUCTURE option to specify a block model to define how to do the randomization. The EXCLUDE option can then restrict the randomization so that one or more of the factors in the block model is not randomized. See the RANDOMIZE directive for further details.
Output is controlled by the PRINT option, with settings:

prwald to print the probabilities calculated from the distribution of the Wald statistics;
criticalwald to print a table giving estimated critical values for each of the Wald statistics, formed from the permutation samples;
ownstatistics to print the estimates, standard errors and confidence intervals for the own statistics, and
monitoring to monitor the progress of the test.

The Wald statistics from the permutation tests can be saved, in a pointer with a variate for each of the FIXEDTERMS, using the WALD parameter. The probabilities calculated from the tests can be saved, in a pointer with a scalar for each of the FIXEDTERMS, using the PRWALD parameter.

You can define your own statistics to be assessed by the test. They are calculated by a procedure _VPERMownstatistics, which is called by VPERMTEST following the REML analysis of each permutation sample. Its use is shown in the VPERMTEST example, which can be modified to calculate your own statistics instead. The information required by _VPERMownstatistics to do the calculations is supplied, in a pointer, by the OWNDATA parameter. The OWNMETHOD option specifies the type of test to be made. The default, twosided tests whether the statistics differ from zero. The greaterthan setting tests whether they are greater than zero, and the lessthan setting tests whether they are less than zero. Standard errors and confidence intervals are also calculated, The CIPROBABILITY option specifies the probability for the confidence intervals (default 0.95). The OWNOBSERVEDVALUES parameter can save a variate containing the values of the own statistics from the original data set. The OWNPROBABILITIES can save a variate containing the probabilities from the tests. The OWNESTIMATES can save a variate containing the bootstrap estimates of the statistics (calculated as the mean of the values obtained from the bootstrap samples) The OWNSES can save a variate containing standard errors of bootstrap estimates. The OWNLOWERCIS and OWNUPPERCIS parameters can save variates containing the lower and upper values, respectively, of the confidence intervals. Finally, the OWNSTATISTICS can save the values of the own statistics obtained from the bootstrap samples, in a pointer with a variate for each statistic.

The maximum number of iterations (MAXCYCLE) and number of blocks of internal memory to be (WORKSPACE) to be used in the REML analyses can be set by a call to the VAOPTIONS procedure before you use VPERMTEST.


See also

Commands for: REML analysis of linear mixed models.


CAPTION      'VPERMTEST example',!t('Split plot design, see the',\
             'Guide to Genstat, Part 2, Section 4.2.1.'); STYLE=meta,plain
SPLOAD       [PRINT=*] '%gendir%/data/Oats.gsh'
VCOMPONENTS  [FIXED=variety*nitrogen] RANDOM=blocks/wplots/subplots
REML         [PRINT=model,comp,Wald] yield

" Own statistics: test differences (in the full model) between variety 
  Marvellous and each of the other two varieties "
PROCEDURE    [WORDLENGTH=long] '_VPERMownstatistics'
  'DATA',      "(I: pointer) information required to calculate the statistics"\
  'STATISTICS';"(O: variate) estimated statistics"\
  MODE=p; TYPE='pointer','variate'
  " insert commands to calculate the statistics " 
  VKEEP      DATA[1]; MEANS=means
  VARIATE    vmeans; VALUES=means
  CALCULATE  STATISTICS = vmeans$[DATA[3]] - vmeans$[DATA[4]]
ENDPROCEDURE "_VPERMownstatistics"

TEXT         [VALUES='Marvellous - Victory','Marvellous - Golden rain'] Contrast
POINTER      [VALUES=variety,Contrast,!(3,3),!(1,2)] Owninfo
VPERMTEST    [PRINT=#,ownstatistics; NTIMES=99; SEED=251015] OWNDATA=Owninfo
Updated on June 28, 2022

Was this article helpful?