Introduction
According to Haykin, S., Nigrin, A. and Zurada, artificial neural network is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It is a circuit composed of a very large number of simple processing elements that are neurally based. Each element operates only on local information. Furthermore each element operates asynchronously, thus there is no overall system clock.
In broad terms, Artificial Neural Networks (ANN) are data processing structures whose operation has been inspired by the behavior of biological structures present in human brain.
Understanding
The Analogy to the Brain
The most basic components of neural networks are modeled after the structure of the brain. Some neural network structures are not closely to the brain and some does not have a biological counterpart in the brain. However, neural networks have a strong similarity to the biological brain and therefore a great deal of the terminology is borrowed from neuroscience.
The Biological Neuron
The most basic element of the human brain is a specific type of cell, which provides us with the abilities to remember, think, and apply previous experiences to our every action. These cells are known as neurons, each of these neurons can connect with up to 200000 other neurons. The power of the brain comes from the numbers of these basic components and the multiple connections between them.
All natural neurons have four basic components, which are dendrites, soma, axon, and synapses. Basically, a biological neuron receives inputs from other sources, combines them in some way, performs a generally nonlinear operation on the result, and then output the final result. The figure below shows a simplified biological neuron and the relationship of its four components.
The Artificial Neuron
The basic unit of neural networks, the artificial neurons, simulates the four basic functions of natural neurons. Artificial neurons are much simpler than the biological neuron; the figure below shows the basics of an artificial neuron.
Note that various inputs to the network are represented by the mathematical symbol, x (n). Each of these inputs are multiplied by a connection weight, these weights are represented by w (n). In the simplest case, these products are simply summed, fed through a transfer function to generate a result, and then output.
Design Issues
What is required to create an ANN?
· A clear understanding of the problem
· Relevant data for training and testing
· Software tools to refine training and testing sets from raw data.
· An ANN development environment (e.g. NeuralWorks Explorer)
· Software tools to analyze the results
The developer must go through a period of trial and error in the design decisions before coming up with a satisfactory design. The design issues in neural networks are complex and are the major concerns of system developers.
Designing a neural network consists of:
· Arranging neurons in various layers.
· Deciding the type of connections among neurons for different layers, as well as among the neurons within a layer.
· Deciding the way a neuron receives input and produces output.
· Determining the strength of connection within the network by allowing the network learns the appropriate values of connection weights by using a training data set.
The process of designing a neural network is an iterative process; the figure below describes its basic steps.
Layers
As the figure above shows, the neurons are grouped into layers. The input layer consist of neurons that receive input form the external environment. The output layer consists of neurons that communicate the output of the system to the user or external environment. There are usually a number of hidden layers between these two layers; the figure above shows a simple structure with only one hidden layer.
When the input layer receives the input its neurons produce output, which becomes input to the other layers of the system. The process continues until a certain condition is satisfied or until the output layer is invoked and fires their output to the external environment.
To determine the number of hidden neurons the network should have to perform its best, one are often left out to the method trial and error. If you increase the hidden number of neurons too much you will get an over fit, that is the net will have problem to generalize. The training set of data will be memorized, making the network useless on new data sets.
Communication and types of connections
Neurons are connected via a network of paths carrying the output of one neuron as input to another neuron. These paths is normally unidirectional, there might however be a two-way connection between two neurons, because there may be another path in reverse direction. A neuron receives input from many neurons, but produces a single output, which is communicated to other neurons.
The neuron in a layer may communicate with each other, or they may not have any connections. The neurons of one layer are always connected to the neurons of at least another layer.
· Inter-layer connections
There are different types of connections used between layers; these connections between layers are called inter-layer connections.
I. Fully connected
Each neuron on the first layer is connected to every neuron on the second layer.
II. Partially connected
A neuron of the first layer does not have to be connected to all neurons on the second layer.
III. Feed forward
The neurons on the first layer send their output to the neurons on the second layer, but they do not receive any input back form the neurons on the second layer.
IV. Bi-directional
There is another set of connections carrying the output of the neurons of the second layer into the neurons of the first layer.
Feed forward and bi-directional connections could be fully- or partially connected.
V. Hierarchical
If a neural network has a hierarchical structure, the neurons of a lower layer may only communicate with neurons on the next level of layer.
VI. Resonance
The layers have bi-directional connections, and they can continue sending messages across the connections a number of times until a certain condition is achieved.
· Intra-layer connections
In more complex structures the neurons communicate among themselves within a layer, this is known as intra-layer connections. There are two types of intra-layer connections.
Recurrent
The neurons within a layer are fully- or partially connected to one another. After these neurons receive input form another layer, they communicate their outputs with one another a number of times before they are allowed to send their outputs to another layer. Generally some conditions among the neurons of the layer should be achieved before they communicate their outputs to another layer.
On-center/off surround
A neuron within a layer has excitatory connections to itself and its immediate neighbors, and has inhibitory connections to other neurons. One can imagine this type of connection as a competitive gang of neurons. Each gang excites itself and its gang members and inhibits all members of other gangs. After a few rounds of signal interchange, the neurons with an active output value will win, and is allowed to update its and its gang member’s weights. (There are two types of connections between two neurons, excitatory or inhibitory. In the excitatory connection, the output of one neuron increases the action potential of the neuron to which it is connected. When the connection type between two neurons is inhibitory, then the output of the neuron sending a message would reduce the activity or action potential of the receiving neuron. One causes the summing mechanism of the next neuron to add while the other causes it to subtract. One excites while the other inhibits.)
Learning
The brain basically learns from experience. Neural networks are sometimes called machine learning algorithms, because changing of its connection weights (training) causes the network to learn the solution to a problem. The strength of connection between the neurons is stored as a weight-value for the specific connection. The system learns new knowledge by adjusting these connection weights.
The learning ability of a neural network is determined by its architecture and by the algorithmic method chosen for training.
The training method usually consists of one of three schemes:
I. Unsupervised learning
The hidden neurons must find a way to organize themselves without help from the outside. In this approach, no sample outputs are provided to the network against which it can measure its predictive performance for a given vector of inputs. This is learning by doing.
II. Reinforcement learning
This method works on reinforcement from the outside. The connections among the neurons in the hidden layer are randomly arranged, then reshuffled as the network is told how close it is to solving the problem. Reinforcement learning is also called supervised learning, because it requires a teacher. The teacher may be a training set of data or an observer who grades the performance of the network results.
Both unsupervised and reinforcement suffer from relative slowness and inefficiency relying on a random shuffling to find the proper connection weights.
III. Back propagation
This method is proven highly successful in training of multilayered neural nets. The network is not just given reinforcement for how it is doing on a task. Information about errors is also filtered back through the system and is used to adjust the connections between the layers, thus improving performance. A form of supervised learning.
· Off-line / On-line
One can categorize the learning methods into yet another group, off-line or on-line. When the system uses input data to change its weights to learn the domain knowledge, the system could be in training mode or learning mode. When the system is being used as a decision aid to make recommendations, it is in the operation mode, this is also sometimes called recall.
o Off-line
In the off-line learning methods, once the systems enters into the operation mode, its weights are fixed and do not change any more. Most of the networks are of the off-line learning type.
o On-line
In on-line or real time learning, when the system is in operating mode (recall), it continues to learn while being used as a decision tool. This type of learning has a more complex design structure.
Learning laws
There are a variety of learning laws which are in common use. These laws are mathematical algorithms used to update the connection weights. Most of these laws are some sort of variation of the best known and oldest learning law, Hebb’s Rule. Man’s understanding of how neural processing actually works is very limited. Learning is certainly more complex than the simplification represented by the learning laws currently developed. Research into different learning functions continues as new ideas routinely show up in trade publications etc. A few of the major laws are given as an example below:
o Hebb’s Rule
The first and the best known learning rule was introduced by Donald Hebb. The description appeared in his book The organization of Behavior in 1949. This basic rule is: If a neuron receives an input from another neuron, and if both are highly active (mathematically have the same sign), the weight between the neurons should be strengthened.
o Hopfield Law
This law is similar to Hebb’s Rule with the exception that it specifies the magnitude of the strengthening or weakening. It states, "if the desired output and the input are both active or both inactive, increment the connection weight by the learning rate, otherwise decrement the weight by the learning rate." (Most learning functions have some provision for a learning rate, or a learning constant. Usually this term is positive and between zero and one.)
o The Delta Rule
The Delta Rule is a further variation of Hebb’s Rule, and it is one of the most commonly used. This rule is based on the idea of continuously modifying the strengths of the input connections to reduce the difference (the delta) between the desired output value and the actual output of a neuron. This rule changes the connection weights in the way that minimizes the mean squared error of the network. The error is back propagated into previous layers one layer at a time. The process of back-propagating the network errors continues until the first layer is reached. The network type called Feed forward, Back-propagation derives its name from this method of computing the error term.
This rule is also referred to as the Windrow-Hoff Learning Rule and the Least Mean Square Learning Rule.
o Kohonen’s Learning Law
This procedure, developed by Teuvo Kohonen, was inspired by learning in biological systems. In this procedure, the neurons compete for the opportunity to learn, or to update their weights. The processing neuron with the largest output is declared the winner and has the capability of inhibiting its competitors as well as exciting its neighbors. Only the winner is permitted output, and only the winner plus its neighbors are allowed to update their connection weights.
Applications of Neural Networks
Neural networks are performing successfully where other methods do not, recognizing and matching complicated, vague, or incomplete patterns. Neural networks have been applied in solving a wide variety of problems.
The most common use for neural networks is to project what will most likely happen. There are many areas where prediction can help in setting priorities. For example, the emergency room at a hospital can be a hectic place, to know who needs the most critical help can enable a more successful operation. Basically, all organizations must establish priorities, which govern the allocation of their resources. Neural networks have been used as a mechanism of knowledge acquisition for expert system in stock market forecasting with astonishingly accurate results. Neural networks have also been used for bankruptcy prediction for credit card institutions.
Although one may apply neural network systems for interpretation, prediction, diagnosis, planning, monitoring, debugging, repair, instruction, and control, the most successful applications of neural networks are in categorization and pattern recognition. Such a system classifies the object under investigation (e.g. an illness, a pattern, a picture, a chemical compound, a word, the financial profile of a customer) as one of numerous possible categories that, in return, may trigger the recommendation of an action (such as a treatment plan or a financial plan.
A company called Nestor, have used neural network for financial risk assessment for mortgage insurance decisions, categorizing the risk of loans as good or bad. Neural networks has also been applied to convert text to speech, NETtalk is one of the systems developed for this purpose. Image processing and pattern recognition form an important area of neural networks, probably one of the most actively research areas of neural networks.
An other of research for application of neural networks is character recognition and handwriting recognition. This area has use in banking, credit card processing and other financial services, where reading and correctly recognizing handwriting on documents is of crucial significance. The pattern recognition capability of neural networks has been used to read handwriting in processing checks, the amount must normally be entered into the system by a human. A system that could automate this task would expedite check processing and reduce errors. One such system has been developed by HNC (Hecht-Nielsen Co.) for BankTec.
A few of presently working ANN systems :
· Transport systems :
o Forecast weather NNICE,
o Damage recovery NASA
o
o Quince Vibration analysis for jet engines
· Consumer products :
o Microwave oven LogiCook
o LogiCook can deal with frozen, pre-heated and different sized portions of the same food. It is also capable of detecting the dangerous condition of liquids, superheating, and can issue stir commands as necessary
· Healthcare systems :
o GLADYS (
o Many other diagnosis systems
Future of ANN
The need of the day is to incorporate intelligence in the machines. We have made machines to work under human’s control. Now we want the machines to We want computer systems to determine, to classify, to predict, to recognize, just like humans do. There are lot many places where the pattern recognition, processing unpredictable inputs, and we require not only output, we require intelligent action.
This is where ANN comes into picture. If you think of a humanoid, you need a robot, and you need a brain that makes that robot work. For the brain part of that humanoid, what could be a better option than the ANN, as, ANN resembles analogy to human neural system itself. If we want a computer system, to decide what to do, and take actions accordingly, just like a human would do, the best approach is ANN. As we all know, the future is going to see sophisticated systems, that recognizes faces, predicts weather, or make a humanoid walk. And that, requires intelligence. So as on today, we can safely state that the future is going to see a lot of research on Neural Nets and the era of intelligent applications based on ANN.
TO DOWN LOAD REPORT AND PPT
