A Simple Neural Net Example

John F. Jarvis

The term "neural net" is more or less a misnomer, implying that some sort of cognitive process is being developed or used by a computing system. In reality, the backpropagation (BP) neural net is a well defined numerical calculation for classifying an input parameter set into two or more categories. The BP neural net can be given a simple geometric interpretation. Before it can be used to classify data, a neural net must be trained by presenting the training routine a series of points with values for each input parameter and the dependent variable. The training algorithm for the backpropagation neural net is given in the primary reference

Consider a BP neural net used to classify its input into two categories using two input units (parameters), the coordinates X and Y. Each hidden unit has the form:

F_k(X,Y) = A_kX + B_kY + C_k
Z_k = S(F_k(X,Y))

where A_k, B_k, and C_k are the values that are determined for each hidden unit by the training procedure. Each of the hidden units, distinguished by the index k, will have a unique set of values for A, B and C. The number of hidden units is also a choice that must be made when crafting a neural net to solve a particular problem. S(Z) is a bipolar sigmoidal function having its output limited to ±1 and a slope of 1 at Z=0. Each F_k(X,Y)=0 can be interpretted as an equation for a line in the plane. For any pair of values X and Y, F_k(X,Y) gives the signed distance from the line to the point X,Y. The effect of the sigmoidal function is to limit the magnitude of the output, thus distance, to 1. The output unit for three hidden units is:

W(X,Y) = DZ₀ + EZ₁ + FZ₂ + G
output = S(W(X,Y))

yields the output value of the neural net for the values of the input parameters X and Y. Writing this expression for a different number of hidden units is obvious. The training procedure also computes the values of the output unit constants D, E, F and G. The three hidden units (F_k(X,Y)) each define a line. A triangle, the simplest figure enclosing an area, is formed from three lines. The output unit combines the values of the three hidden units to define the final area defining the two categories.

The figures at the bottom of this page provide examples of using the two input unit back propagation neural net with 2, 3 and 4 hidden units to create a decision "surface" which in the plane is a curve. The two categories of points, asterisks, and squares, are generated using a random number generator and have no significance beyond a simple demonstration of the neural net technique. The asterisks (red) are uniform in a statistical sense over the entire area. The squares (green) are generated from three overlapping Gaussian distributions concentrated near the center of the plots. The solid curve (blue) enclosing most of the squares clearly shows a number of lines corresponding to the number of hidden units and the blending of them into a single boundary by the output unit. The data for each figure is the same. There are 500 data points in each category and in the following table the number of squares and asterisks correctly classified is given for the three hidden unit values. By design, it is impossible to correctly classify all of the points.

Hidden  Correct . . . . .  Total
 Units  Squares Asterisks     %
    2       479       362   84.1
    3       488       412   90.0
    4       493       488   91.1

Moving to three dimensions, the hidden units describe planes and give the distance of three dimensional points to the plane. The simplest solid object composed from planes is the four sided tetrahedron. Four hidden units are appropriate when there are three input units. While I can't visualize geometry in higher dimensions, the equations have the same form. This discussion provides the reason for using eleven hidden units in the IBB situation classification neural net. In practice the classification error changes, but only slightly, as the number of hidden units is increased or decreased.

The wikipedia contains a general discussion on neural networks that include extensive references and links. Many different kinds of neural net have been invented. There is no assurance that the back propagation net is the optimum method for the demonstration on this page or the Intentional Base on Balls classification task. The theory of this particular form of neural net used is covered in “Neural Networks”, Laurene Fausett, Prentice Hall, 1994, and other textbooks.