SIGMOID ACTIVATION FUNCTION

Whyuse the sigmoid function? Because it combines nearly linear behavior, curvilinear behavior, and nearly constant behavior, depending on the value of the input. Figure 7.3 shows the graph of the sigmoid function y = f (x) = 1/(1 + e?x ), for ?5 < x < 5 [although f (x) may theoretically take any real-valued input]. Through much of the center of the domain of the input x (e.g., ?1 < x < 1), the behavior of f (x) is nearly linear. As the input moves away from the center, f (x) becomes curvilinear. By the time the input reaches extreme values, f (x) becomes nearly constant.

Moderate increments in the value of x produce varying increments in the value of f (x), depending on the location of x. Near the center, moderate increments in the value of x produce moderate increments in the value of f (x); however, near the extremes, moderate increments in the value of x produce tiny increments in the value of f (x). The sigmoid function is sometimes called a squashing function, since it takes any real-valued input and returns an output bounded between zero and 1.

BACK-PROPAGATION

How does the neural network learn? Neural networks represent a supervised learning method, requiring a large training set of complete records, including the target variable. As each observation from the training set is processed through the network, an output value is produced from the output node (assuming that we have only one output node, as in Figure 7.2). This output value is then compared to the actual value of the target variable for this training set observation, and the error (actual ? output) is calculated. This prediction error is analogous to the residuals in regression models.

The problem is therefore to construct a set of model weights that will minimize the SSE. In this way, the weights are analogous to the parameters of a regression model. The true values for the weights that will minimize SSE are unknown, and our task is to estimate them, given the data. However, due to the nonlinear nature of the sigmoid functions permeating the network, there exists no closed-form solution for minimizing SSE as exists for least-squares regression.