1. Home
  2. NNFIT directive

NNFIT directive

Fits a multi-layer perceptron neural network.

Options

PRINT = string tokens Controls fitted output (description, estimates, fittedvalues, summary); default desc, esti, summ
NHIDDEN = scalar Number of functions in the hidden layer; no default, must be set
HIDDENMETHOD = string token Type of activation function in the hidden layer (logistic, hyperbolictangent); default logi
OUTPUTMETHOD = string token Type of activation function in the output layer (linear, logistic, hyperbolictangent); default line
GAIN = scalar Multiplicative constant to use in the functions; default 1
NTRIES = scalar Number of times to search for a good initial starting point for the optimization; default 5
NSTARTITERATIONS = scalar Number of iterations to use to find a good starting point for the optimization; default 30
VALIDATIONOPTIONS = variate Variate containing three integers to control validation for early stopping; default * i.e. no early stopping; default !(10,4,16)
SEED = scalar Seed for random numbers to generate initial values for the free parameters; default 0
MAXCYCLE = scalar Maximum number of iterations of the conjugate-gradient algorithm; default 50

Parameters

Y = variates Response variates
X = pointers Input variates
YVALIDATION = variates Validation data for the dependent variates
XVALIDATION = pointers Validation data for the independent variates
FITTEDVALUES = variates Fitted values generated for each y-variate by the neural network
OBJECTIVE = scalars Value of the sum of squares objective function at the end of the optimization
NCOMPLETED = scalars Number of completed iterations of the conjugate-gradient algorithm
EXIT = scalars Saves the exit code
SAVE = pointers Saves details of the network and the estimated parameters

Description

A neural network is a method for describing a nonlinear relationship between a response variate supplied here by the Y parameter, and a set of input variates supplied here in a pointer by the X parameter. The type of neural network fitted by NNFIT is a fully-connected feed-forward multi-layer perceptron with a single hidden layer. This network starts with a row of nodes, one for each input variable (i.e. x-variate), which are all connected to every node in the hidden layer. The nodes in the hidden layer are then all connected to the output node in the final, output layer. The number of nodes in the hidden layer is specified by the NHIDDEN option.

The output value y is given by

y = ψ( Σk = 1…m wk φ( Σj = 1…d wjk xj – θ) – η)

where d is the number of input nodes (i.e. x-variates),
    m is the number of hidden nodes (NHIDDEN),
    xj is value of the jth x-variate,
    wjk are weight parameters in the connections between the nodes in the input and hidden layers,
    wk are weight parameters in the connections between the nodes in the hidden and output layer,
    θ is the threshold value subtracted at the hidden layer,
    η is the threshold value subtracted at the single node in the output layer,
    φ(.) is the activation function applied at the hidden layer,
    ψ(.) is the activation function applied at the output layer.

The activation functions for the hidden and outer layer are specified by the HIDDENMETHOD and OUTPUTMETHOD options, respectively, with settings:

    linear φ(z) = z (OUTPUTMETHOD only),
    logistic φ(z) = 1 / (1 + exp(-γz)),
    hyperbolictangent φ(z) = tanh(γz),

where the parameter γ is specified by the GAIN option; the default setting is logistic for HIDDENMETHOD, and linear for OUTPUTMETHOD.

Values for the free parameters in the multi-layer perceptron model are optimized by using a preconditioned, limited-memory quasi-Newton conjugate gradients method to minimize the objective (sum of squares) function equal to 0.5 times the average sum of squared deviation of the estimated y-values from the observed y-values.

Printed output is controlled by the PRINT option, with settings:

    description a description of the network (number of input variables, nodes etc.),
    estimates estimates of the free parameters,
    fittedvalues fitted values,
    summary summary (numbers of iterations, objective function etc.).

The NTRIES option defines the number of times to search for a good initial starting point for the optimization (default 5). The NSTARTITERATIONS option defines the number of iterations to use to find a good starting point for the optimization (default 30).

The SEED option supplies a seed for the random numbers to generate initial values for the free parameters. The default of zero continues the existing sequence of random numbers if any have already been used in the current Genstat job. If none have yet been used, Genstat picks a seed at random.

The MAXCYCLE option sets a limit on the number of iterations of the conjugate-gradient algorithm to use for the estimation (default 50).

To improve the accuracy of the neural-network approximations to new data records, it is usually desirable to stop the optimization before the value of the objective function reaches a global minimum on the training set. This method, which is known as early stopping, and can be performed by using a validation set of data records, specified by the YVALIDATION and XVALIDATION parameters. The optimization is then halted when the sum of squares error function achieves a minimum over the validation set of data records which has not been used to estimate the values of the free parameters in the model. The VALIDATIONOPTIONS option specifies a variate containing three integers to control validation for early stopping. The first integer defines the number of iterations of the optimizing function to complete before beginning validation; default 10. The second integer defines the number of iterations between consecutive validations; default 4. The third integer defines the number of iterations to continue validating beyond the current minimum of the objective function before stopping; default 16. This is to try to avoid the possibility of getting stuck at a local minimum. The variates in the XVALIDATION pointer must be in the same order as the corresponding variates in the X pointer.

The results of the fit, together with details about design of the neural network, can be saved using the SAVE parameter. This can then be used in the NNDISPLAY directive to display further output, or the NNPREDICT directive to form predictions.

Options: PRINT, NHIDDEN, HIDDENMETHOD, OUTPUTMETHOD, GAIN, NTRIES, NSTARTITERATIONS, VALIDATIONOPTIONS, SEED, MAXCYCLE.

Parameters: Y, X, YVALIDATION, XVALIDATION, FITTEDVALUES, OBJECTIVE, NCOMPLETED, EXIT, SAVE.

Method

NNFIT uses the function nagdmc_mlp from the Numerical Algorithms Group’s library of Data Mining Components (DMCs), which estimates the free parameters using a conjugate gradient method.

Action with RESTRICT

You can restrict the set of units used for the estimation by applying a restriction to the y-variate or any of the x-variates. If several of these are restricted, they must all be restricted to the same set of units. Similarly, you can restrict the set of units used for the validation by applying a restriction to the YVALIDATION variate or any of the XVALIDATION variates.

See also

Directives: NNDISPLAY, NNPREDICT, ASRULES, RBFIT.

Procedure: KNEARESTNEIGHBOURS.

Commands for: Data mining.

Example

" Example NNFI-1: Fitting a multi-layer perceptron neural network."

" This example fits a multi-layer perceptron neural network with five hidden
  layers, a hyperbolic activation function in the hidden layer and a linear 
  activation function in the output layer."

" The data are in a file called iris.GSH and contain the data from Fisher's
  Iris data set."

SPLOAD   [PRINT=*] '%GENDIR%/Data/iris.GSH' 
POINTER  [VALUES=Sepal_Length,Sepal_Width,Petal_Length,Petal_Width] Measures
CALC     yval = NEWLEVELS(Species)
NNFIT    [PRINT=description,estimates,summary; NHIDDEN=5;\
          HIDDENMETHOD=hyperbolictangent; OUTPUTMETHOD=linear; SEED=12]\ 
           Y=yval; X=Measures
Updated on March 7, 2019

Was this article helpful?