VISUAL BASIC (VB.NET) NEURAL NETWORK MULTILAYER FEEDFORWARD BACKPROPAGATION ALGORITHM VERSION 1.1 - OPEN SOURCE CODE RELEASE
A Neural Network is a Network modeled after neurons in brain tissues and is a set of neurons in an input layer connected to one or more hidden layers of neurons which are in turn connected to an output layer of neurons.
The connections between the cells in the input layer and the hidden layer and the hidden layer and output layer are associated with weights.
For simplicity sake, each Neuron could be said to consist of an INPUT, TRAINING OUTPUT, TARGET OUTPUT, BIAS, ERROR, CONNECTING WEIGHT.
INPUT LAYER
Typically one or more neurons or cells exist in an input layer. The input layer could receive data normalized between 0 and 1 or -1 and 1. The input layer converts the data it receives to an OUTPUT using a Sigmoid Function or Squashing Function that generally maps any input to a value between 0 and 1 as an output.
Each cell in the INPUT LAYER is connected to all the cells in the HIDDEN LAYER by weighted connections. For modeling or simplicity purposes, the bias and weights in the INPUT LAYER are set to zero.
HIDDEN LAYER
There can be one or more HIDDEN LAYERS in a FeedForward Neural Network, but for most practical applications, the number of hidden layers is one (1).
Each cell in the HIDDEN LAYER is fully connected to each cell in the INPUT LAYER, next HIDDEN LAYER or OUTPUT LAYER and each of these connections are weighted.
The number of cells in a HIDDEN LAYER can be adjusted during the training of a Neural Network to avoid overfitting. The weights and bias in the hidden layer are initialized to random values between -1.0 and 1.0 or -0.5 and 0.5. These initial values are used to compute the OUTPUT from each cell in the HIDDEN LAYER which will be fed to the next HIDDEN LAYER or OUTPUT LAYER.
Assume that a cell in the hidden layer (C3) is connected to two cells in the input layer (C1 and C2)
If the input to cell C1 is 0.4 and the input to cell C2 is 0.8, then the outputs from cell C1 and C2 respectively are 0.4 and 0.8
Let's the connection between the cell in the hidden layer (C3) and the cell C1 in the input layer be represented as W31 and the connection between the cell C3 and C2 to be represented by W32
Since W31, W32 and the hidden cell bias B3 are initialized randomly, assume that W31 = -0.25, W32 = 0.46 and B3 = -0.25
The input to cell C3 = (OUTPUT from input cell C1 * W31) + (OUTPUT form cell C2 * W32) + (BIAS of cell C3)
= (0.4 * - 0.25) + (0.8 * 0.46) - 0.25
= 0.13
Each cell both in the HIDDEN LAYER and OUTPUT LAYER calculates it's OUTPUT using a mathematical Sigmoid / Activation / Squashing / Logistic Function that is represented by OUTPUT O = (1.0 / (1.0 + System.Math.Exp(-val))) in Visual Basic .NET (VB.NET) or C#.
Applying the Sigmoid Function or Logistic Function to 0.13 gives a value of 0.53245430638731872 as the OUTPUT of cell C3.
OUTPUT LAYER
Each cell in the OUTPUT LAYER is fully connected to each cell in the hidden layer and has a bias and weighted connections which are also initialized to random values.
Each cell in the OUTPUT LAYER receive inputs from every cell in the last hidden layer. The INPUT and OUTPUT of cells in the OUTER LAYER are calculated like that of cells in the HIDDEN LAYER.
FEED-FORWARD
The Neural Network Algorithm that I have been describing is a Multi-Layer Feed-Forward Back-Propagation Neural Network Algorithm.
After the weights and bias are initialized for all the weights in the Neural Network, the INPUTS and OUTPUTS are calculated for all the cells in the HIDDEN LAYER and OUTPUT LAYER in a process described as a Feed Forward. A network topology that includes 1 HIDDEN LAYER and 1 OUTPUT LAYER is a 2 LAYER Network. If it has N LAYERS in the HIDDEN LAYER then it is described as an (N + 1) LAYER Network where N representing the number of hidden layers in the Network.
After this process, the Neural Network compares the OUTPUT from each cell in the OUTPUT LAYER to the expected or training or target value to calculate the ERROR.
The ERROR in a cell in the OUTPUT LAYER = OUTPUT * (1 - OUTPUT) * (TARGET - OUTPUT) where OUTPUT is the actual OUTPUT from a cell and TARGET is the expected or training or target value.
The ERROR is used to update the weights and bias in the OUTPUT LAYER and the HIDDEN LAYER, where,
NEW BIAS VALUE = OLD BIAS VALUE + (OLD BIAS VALUE * ERROR * LEARNING RATE )
''VISUAL BASIC (VB.NET) NEURAL NETWORK MULTILAYER FEEDFORWARD BACKPROPAGATION ALGORITHM VERSION 1.0
''DATE : OCTOBER 25, 2004 10:40 PM
''AUTHOR : KINGSLEY TAGBO
''LICENSE : DISTRIBUTED UNDER GNU GENERAL PUBLIC LICENSE VERSION 2, JUNE 1991 (SEE WWW.FSF.ORG)
'' : YOU CAN ALSO READ THE LICENSE INCLUDED BELOW AFTER THE SOURCE CODE
''VERSION : 1.0
''COPYRIGHT : KINGSLEY TAGBO - HTTP://WWW.KDKEYS.NET
''RELEASE: : GOLD
SOURCE CODE DOCUMENTATION
Artificial MiultiLayer FeedForward Neural Network With BackPropagation Algorithm
rand field
Static variable for random numbers
input_count field
Static variable for number of neurons or cells in INPUT LAYER
hidden_count field
Static variable for number of neurons or cells in HIDDEN LAYER
output_count field
Static variable for number of neurons or cells in OUTPUT LAYER
iterations field
Static variable for the number of processing cycles of the Neural Network
learning_rate field
Represents the LEARNING RATE used in gradient descent to prevent weights from converging at sub-optimal solutions.
input_data field
A two dimensional array of input data from a training sample An INPUT represented as double [,] inputDataArray = new Double [,] {{0.20, 0.80}, {0.80, 0.4}} would represent 2 training instances for an INPUT LAYER with 2 NEURONS or CELLS
output_data field
A two dimensional array of output data from a training sample An OUTPUT represented as double [,] outputDataArray = new Double [,] {{0}, {1}} would represent 2 training instances for an OUTPUT LAYER with 1 NEURON or CELL
input field
A collection of neurons representing the input layer
hidden field
A collection of neurons representing the hidden layer
output field
A collection of neurons representing the output layer
Initialize(Double[0:,0:],Double[0:,0:],Int32) method
Initializes the LAYERS in the NEURAL NETWORK
Parameters
- inputData
- INPUTS data for the NEURONS in the INPUT LAYER
- outputData
- OUTPUT data for the NEURONS in the OUTPUT LAYER
- hidden_layer_count
- The number of neurons in the HIDDEN LAYER
FeedForward(Int32) method
Initializes the NEURAL NETWORK with training data input or a real world data input for classification. FeedForward feeds the INPUT to the INPUT LAYER neurons FeedForward feeds the OUTPUT from the INPUT LAYER to the HIDDEN LAYER FeedForward feeds the OUTPUT from the HIDDEN LAYER to the OUTPUT LAYER FeedForward uses the Sigmoid Function or Logistic Function to calculate the OUTPUT from the INPUT in the HIDDEN and OUTPUT LAYERS
Parameters
- sampleNumber
- A numeric ordered value representing the training or classification instance. If the dataset contains 10 instances or rows, the first row has a sampleNumber = 0 and the last row or instance has a sample number = 9
BackPropagate() method
Recalculates the BIAS and ERROR in the HIDDEN LAYER and OUTPUT LAYER. Adjustes the WEIGHTS between the OUTPUT LAYER and HIDDEN LAYER and between the HIDDEN LAYER and the INPUT LAYER using the derivative of the Sigmoid Function or Logistic Function
Main() method
Executes a Neural Network MultiLayer FeedForward BackPropagation Algorithm for Learning or Classification
Random property
Generates random double values between -1.0 and +1.0
NeuralNetwork.Neuron
A Neuron is the basic building block of a Neural Network
Neuron(Int32,Int32) constructor
Public constructor for initializing a neuron
Parameters
- layer
- Input Layer = 0, Hidden Layer = 1 and Output Layer = 2
- index
- A number assigned to a cell for ordering the cell or identifying it
player field
Internal storage for Neuron.Layer public property
pindex field
Internal storage for Neuron.Index public property
pinput field
Internal storage for Neuron.Input public property
poutput field
Internal storage for Neuron.Output public property
poutputTraining field
Internal storage for Neutron.OutputTraining public property
pweight field
Internal storage for Network.Weight public property
pbias field
Internal storage for Network.Bias public property
perror field
Internal storage for Network.Error public property
LogisticFunction(Double) method
Sigmoid Function or Logistic Function or Activation Function is applied to the INPUT to a NETWORK LAYER to get an OUTPUT
Parameters
- val
- The value to calculate a Sigmoid or Logistic Function for
Returns
System.Double datatype of the Logistic or Sigmoid Function result
Derivative of the Sigmoid Function or Logistic Function or Activation Function
Parameters
- val
- The value to calculate the derivative of the Sigmoid Function for
Returns
System.Double datatype of the numeric result
INPUT LAYER = 0, HIDDEN LAYER = 1, OUTPUT LAYER = 2
Index property
Identifies a Neuron or the position of a Neuron in a LAYER
Input property
Input data fed to the Neural Network
Output property
Calculated Ouput from the INPUT, HIDDEN or OUTPUT LAYER
OutputTraining property
Expected or Target or Learning output for a neutron in the OUPUT LAYER
Weight property
Storage for array of weights from OUTPUT to HIDDEN LAYER and from HIDDENLAYER TO INPUT LAYER. Each Neuron in the OUTPUT and HIDDEN LAYER is connected to an array of Neurons
Bias property
Varies the OUTPUT of a Neuronin a HIDDEN or OUTPUT LAYER
Error property
Error = (ACTUALOUTPUT * (1 - ACTUALOUTPUT) * (EXPECTEDOUTPUT - ACTUALOUTPUT))
NeuralNetwork.Neurons
A Neural Network Layer or collection of cells or neurons
Add(Neuron) method
Adds a neuron to a Neural Network Layer or collection of cells
Parameters
- newNeuron
- The neuron to add to a Neural Network Layer
Insert(Int32,Neuron) method
Adds a neuron to a Neural Network Layer at a specific position
Parameters
- index
- The position where a neuron will be inserted
- newNeuron
- The neuron to add to a Neural Network Layer
this[Int32] indexer
ReadOnly indexer - retrieves the neuron at a specific position orlocation in the Neural Network Layer or Neural Network Collection