NEURAL NETWORKS AND DEEP LEARNING

Subject Code - CSE_ 3273

CREDITS – 03

Module -2

1
 Module -2
KERNEL METHODS & RADIAL-BASIS FUNCTION NETWORKS
AND SELF-ORGANIZING MAPS

Topic Page No.

Introduction 3
Cover’s Theorem on the Separability of Patterns 6
The Interpolation Problem 16
Radial-Basis-Function Networks 23
K-Means Clustering 30
Recursive Least-Squares Estimation of the Weight Vector 49
Hybrid Learning Procedure for RBF Networks 55
Feature-Mapping Models 58, 69
Self-Organizing Map 78
Properties of the Feature Map 86
Case Studies 97

Text Book 1: 5.1-5.7 Text Book 2: Chapter 15, relevant journals.

INTRODUCTION

Discussion on

➢ Cover’s theorem on the separability of patterns. Illustrated by revisiting the XOR

problem

➢ Radial-basis functions, setting the stage for the construction of radial-basis function (RBF)

➢ The K-means algorithm. Popular, algorithm for clustering, which is well suited for
training the hidden layer in an unsupervised manner

➢ K-means clustering algorithm to describe a recursive implementation of least-squares

estimation for training the output layer of the RBF network in a supervised manner

3
 INTRODUCTION

The supervised training of a neural network may be approached in several different ways.

The back-propagation learning algorithm for multilayer perceptrons, or application of a recursive

technique known in statistics as stochastic approximation.

In this chapter, we take a completely different approach. Specifically, we solve the problem of
classifying nonlinearly separable patterns by proceeding in a hybrid manner, involving two stages:

➢ The first stage transforms a given set of nonlinearly separable patterns into a new set for which,
under certain conditions, the likelihood of the transformed patterns becoming linearly separable
is high; the mathematical justification of this transformation is traced to an early paper by Cover
(1965).
➢ The second stage completes the solution to the prescribed classification problem by using least-
squares estimation.

4
 INTRODUCTION

Implementation of hybrid approach to pattern classification by using a radial-basis function

(RBF) network, the structure of which consists of only three layers:

➢ The input layer is made up of source nodes (sensory units) that connect the network to its
environment.
➢ The second layer, consisting of hidden units, applies a nonlinear transformation from the
input space to the hidden (feature) space. For most applications, the dimensionality of the
only hidden layer of the network is high; this layer is trained in an unsupervised manner
using stage 1 of the hybrid learning procedure.
➢ The output layer is linear, designed to supply the response of the network to the activation
pattern applied to the input layer; this layer is trained in a supervised manner using stage 2 of
the hybrid procedure.

5
COVER’S THEOREM ON THE SEPARABILITY OF PATTERNS

When a radial-basis function (RBF) network is used to perform a complex pattern classification
task, the problem is basically solved by first transforming it into a high dimensional space in a
nonlinear manner and then separating the classes in the output layer. The underlying justification is
found in Cover’s theorem on the separability of patterns, which, in qualitative terms, may be stated
as follows (Cover, 1965):

A complex pattern-classification problem, cast in a high-dimensional space nonlinearly,

is more likely to be linearly separable than in a low-dimensional space, provided that the
space is not densely populated.

Radial-basis function (RBF)- Multilayer perceptron 6

COVER’S THEOREM ON THE SEPARABILITY OF PATTERNS

7
Phi are hidden functions in hidden layers which separates class H1 and H2
COVER’S THEOREM ON THE SEPARABILITY OF PATTERNS

m0 is the dimensionality of input and mapping X into high dimensionality space m1.
8
COVER’S THEOREM ON THE SEPARABILITY OF PATTERNS

When linearly separable

Got good surface which linearly separates.

Reverse map of phi space into input space is no longer hyperplane , its is a hyper
surface.

Inverse image of this hyperplane separates input plane and it is a hyper surface. 9
COVER’S THEOREM ON THE SEPARABILITY OF PATTERNS

10
COVER’S THEOREM ON THE SEPARABILITY OF PATTERNS

11
COVER’S THEOREM ON THE SEPARABILITY OF PATTERNS

Cover considered, probability of belonging to class either H1 or H2 among all possible dichotomies. When
m increases probability of m increases hence better separability.
12
THE XOR PROBLEM

The four patterns of

the XOR problem

13
THE XOR PROBLEM

14
 THE XOR PROBLEM

In this example, there is no increase in the

dimensionality of the hidden space compared with
the input space. In other words, nonlinearity
exemplified by the use of Gaussian hidden functions
is sufficient to transform the XOR problem into a
linearly separable one. 15
THE INTERPOLATION PROBLEM

An interpolation problem is a mathematical problem that involves finding a

function that estimates the value of a dependent variable based on a set of
independent variables.

It's a type of estimation that's used in engineering and science

•The interpolation function passes through all the data points in the dataset.
•The function provides a smooth and continuous estimation.
•Interpolation is useful when reliable data points are available.

16
 THE INTERPOLATION PROBLEM

➢ The important point that emerges from Cover’s theorem on the separability of patterns is that
in solving a nonlinearly separable pattern-classification problem, there is usually practical
benefit to be gained by mapping the input space into a new space of high enough dimension.

➢ Basically, a nonlinear mapping is used to transform a nonlinearly separable classification

problem into a linearly separable one with high probability.

Interpolation is a process of determining the unknown values that lie in between the known data points. It is mostly
used to predict the unknown values.
17
THE INTERPOLATION PROBLEM

➢ Consider then a feedforward network with an input layer, a single hidden layer, and
an output layer consisting of a single unit.

➢ We have purposely chosen a single output unit to simplify the exposition without
loss of generality. The network is designed to perform a nonlinear mapping from
the input space to the hidden space, followed by a linear mapping from the hidden
space to the output space.

18
THE INTERPOLATION PROBLEM

Let mo denote the dimension of the input space

19
THE INTERPOLATION PROBLEM

For strict interpolation as specified here, the interpolating surface (i.e., function F) is constrained to pass through
all the training data points.

Mapping of N dimensional real numbers into 1 dimensional real numbers. 20

 THE INTERPOLATION PROBLEM

Taking different phi functions which are centered around different Xi. There are N such phi functions; hence
N such synaptic weights will be there in N dimensional hidden functions to single output space. 21
THE INTERPOLATION PROBLEM

22
 RADIAL-BASIS-FUNCTION NETWORKS

Envision a radial-basis-function (RBF) network in the form of a layered structure having three layers:.

23
RADIAL-BASIS-FUNCTION NETWORKS

24
RADIAL-BASIS-FUNCTION NETWORKS

25
RADIAL-BASIS-FUNCTION NETWORKS

26
 RADIAL-BASIS-FUNCTION NETWORKS

Having a hidden layer of the same size as the input layer could be wasteful of computational
resources, particularly when dealing with large training samples.

When the hidden layer of the RBF network is specified in the manner described in Eq. (5.20), we
find that any correlation existing between adjacent data points in the training sample is
correspondingly transplanted into adjacent units in the hidden layer. Stated in another way, there is
redundancy of neurons in the hidden layer when they are chosen in accordance with Eq. (5.20) by

Gaussian function as the radial-basis function

27
RADIAL-BASIS-FUNCTION NETWORKS

Unlike the case for a multilayer perceptron, the training of an RBF network does not involve
the back propagation of error signals.

Moreover, the approximating function realized by both of these two RBF structures has the
same mathematical form,

28
RADIAL-BASIS-FUNCTION NETWORKS

29
 K-MEANS CLUSTERING

In designing the RBF network of Fig. 5.4, a key issue that needs to be addressed is how to
compute the parameters of the Gaussian units that constitute the hidden layer by using unlabeled
data.

In other words, the computation is to be performed in an unsupervised manner. In this section, we

describe a solution to this problem that is rooted in clustering, by which we mean the following:

Clustering is a form of unsupervised learning whereby a set of observations (i.e., data points)
is partitioned into natural groupings or clusters of patterns in such a way that the measure of
similarity between any pair of observations assigned to each cluster minimizes a specified cost
function.

30
K-MEANS CLUSTERING

31
 K-MEANS CLUSTERING

To do this encoding, we need a measure of similarity between every pair of vectors xi and ,
which is denoted by d(xi, ).When the measure d(xi, ) is small enough, both xi and are assigned to
the same cluster; otherwise, they are assigned to different clusters.

32
K-MEANS CLUSTERING

Set of K clusters

33
K-MEANS CLUSTERING

34
K-MEANS CLUSTERING

Clustering problem as follows:

Given a set of N observations, find the encoder C that assigns these observations to the K
clusters in such a way that, within each cluster, the average measure of dissimilarity of the
assigned observations from the cluster mean is minimized.

35
K-MEANS CLUSTERING

With encoder C being unknown, how do we minimize the cost function J(C)? To
address this key question, we use an iterative descent algorithm, each iteration of
which involves a two-step optimization.

36
K-MEANS CLUSTERING

37
 K-MEANS CLUSTERING

of Cover’s theorem—that is, that the dimensionality of the hidden layer is high enough.

K-means algorithm is indeed computationally powerful enough to transform a set of

nonlinearly separable patterns into separable ones in accordance with this theorem.
38
K-MEANS CLUSTERING

Reference: Internet: [Link]

39
K-MEANS CLUSTERING

Reference: Internet: [Link] 40

K-MEANS CLUSTERING

Reference: Internet: [Link]

41
K-MEANS CLUSTERING

Calculate new centroids

Reference: Internet: [Link]

42
K-MEANS CLUSTERING

First cluster has one data point . Hence centroid remains same

Second cluster has 5 datapoints.

A1: (2,20)
B1:
(8+5+7+6+4=30/5=6
(4+8+5+4+9=30/5=6)

Reference: Internet: [Link]

43
K-MEANS CLUSTERING

Reference: Internet: [Link]

44
K-MEANS CLUSTERING

Reference: Internet: [Link]

45
K-MEANS CLUSTERING

C2 has changed from cluster 2 to 1. Hence not converged.

Calculate the new centroid.

Reference: Internet: [Link]
46
K-MEANS CLUSTERING

B1 has changed from cluster 2 to 1. Hence not converged.

Calculate the new centroid.

Reference: Internet: [Link]
47
K-MEANS CLUSTERING

No change. Hence converged.

Reference: Internet: [Link]

48
 RECURSIVE LEAST-SQUARES ESTIMATION OF THE WEIGHT VECTOR

R(n)-> Correlation matrix

W(n)->Weight vector
r(n)-> Desired response

The Least Square method is a mathematical technique that minimizes the sum of squared differences between
observed and predicted values 49
 RECURSIVE LEAST-SQUARES ESTIMATION OF THE WEIGHT VECTOR

Solve 5.30 to find

w(n)->Weight vector

resulting algorithm is called the recursive least-squares (RLS) algorithm.

50
 RECURSIVE LEAST-SQUARES ESTIMATION OF THE WEIGHT VECTOR

The RLS algorithm

Consider the correlation vector

51
 RECURSIVE LEAST-SQUARES ESTIMATION OF THE WEIGHT VECTOR

The RLS algorithm

52
RECURSIVE LEAST-SQUARES ESTIMATION OF THE WEIGHT VECTOR

53
RECURSIVE LEAST-SQUARES ESTIMATION OF THE WEIGHT VECTOR

g(n) is called the gain vector of the RLS algorithm

prior estimation error (n)

54
Hybrid Learning Procedure for RBF Networks

The K-means algorithm for training the hidden layer is applied first; it is then followed by the RLS
algorithm for training the output layer. Hereafter, we refer to this hybrid learning procedure as the
“Kmeans, RLS” algorithm, aimed at training an RBF network with the following composition:

55
Hybrid Learning Procedure for RBF Networks

56
Hybrid Learning Procedure for RBF Networks

An attractive feature of the “K-means, RLS” algorithm is its computational efficiency, which
follows from the fact that the K-means and RLS algorithms are both computationally efficient in
their own individual ways. The only questionable feature of the algorithm is the absence of an
overall optimality criterion that combines the training of the hidden and output layers, assuring
the whole system of optimality in some statistical sense. 57
 Self-Organizing Map

In unsupervised learning, the requirement is to discover significant patterns, or features, of the

input data through the use of unlabeled examples. That it to say, the network operates in accordance
with the rule:
Learn from examples without a teacher.

58
 Self-Organizing Map

The study of unsupervised learning may be pursued from two different perspectives:

(i) Self-organized learning, the formulation of which is motivated by neurobiological considerations.

In particular, the unsupervised-learning algorithm is supplied with a set of rules of local behavior, and
the requirement is to use the rules to compute an input–output mapping with desirable properties.
Here, the term “local” means that the adjustments applied to the synaptic weights of each neuron
in the network are confined to the immediate local neighborhood of the neuron. In this context,
the modeling of a neural network used for self-organized learning tends to follow neurobiological
structures, recognizing that network organization is fundamental to the brain.

(ii) Statistical learning theory, which is the approach that is traditionally pursued in machine learning.
The notion of locality of learning that is emphasized in neural networks plays a lesser role in machine
learning. Instead, in statistical learning theory, a much greater emphasis is placed on well-established
mathematical tools.
59
PRINCIPLES OF SELF-ORGANIZATION

Principle 1. Self-amplification
Modifications in the synaptic weights of a neuron tend to self-amplify in accordance with Hebb’s postulate
of learning, which is made possible by synaptic plasticity.

Principle 2. Competition
The limitation of available resources, in one form or another, leads to competition among the synapses of a
single neuron or an assembly of neurons, with the result that the most vigorously growing (i.e., fittest)
synapses or neurons, respectively, are selected at the expense of the others. This second principle is made
possible by synaptic plasticity (i.e., adjustability of a synaptic weight).

Principle 3. Cooperation
Modifications in synaptic weights at the neural level and in neurons at the network level tend to cooperate with
each other.

Principle 4. Structural Information

The underlying order and structure that exist in an input signal represent redundant information, which is
acquired by a self-organizing system in the form of knowledge.
60
 Self-Organizing Map
PRINCIPLES OF SELF-ORGANIZATION

Principle 1. Self-amplification

This first principle of self-organization states the following:

Modifications in the synaptic weights of a neuron tend to self-amplify in accordance with

Hebb’s postulate of learning, which is made possible by synaptic plasticity.

Self-amplification, or self-reinforcement, indicates modifications in the synaptic weights of the

neuron must be based on presynaptic and postsynaptic signals available at the local level.

Strong synapse leads to the coincidence of presynaptic and postsynaptic signals. In turn, the
synapse is increased in strength by such a coincidence. (Essence of Hebbian learning)
61
Self-Organizing Map

Hebb’s postulate of learning is the oldest and most famous of all learning rules; it is named in
honor of the neuropsychologist Hebb (1949). Hebb’s book The Organization of Behavior (1949)
states the following (p. 62):

When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes
part in firing it, some growth process or metabolic changes take place in one or both cells
such that A’s efficiency as one of the cells firing B is increased.

62
Self-Organizing Map

The statement on Hebb’s postulate of learning is made in a neurobiological context. We may

expand and rephrase it as a two-part rule (Stent, 1973; Changeux and Danchin, 1976):

➢ If two neurons on either side of a synapse (connection) are activated simultaneously (i.e.,
synchronously), then the strength of that synapse is selectively increased.

➢ If two neurons on either side of a synapse are activated asynchronolusly, then that synapse is
selectively weakened or eliminated.

Such a synapse is called a Hebbian synapse. Hebbian synapse as a synapse that uses a time-
dependent, highly local, and strongly interactive mechanism to increase synaptic efficiency as a
function of the correlation between the presynaptic and postsynaptic activities.

63
 Self-Organizing Map

Principle 2. Competition
This second principle of self-organization states the following:

The limitation of available resources, in one form or another, leads to competition among the
synapses of a single neuron or an assembly of neurons, with the result that the most vigorously
growing (i.e., fittest) synapses or neurons, respectively, are selected at the expense of the others.
This second principle is made possible by synaptic plasticity (i.e., adjustability of a synaptic
weight).

For a given single neuron to stabilize, for example, there must be competition among its synapses for
limited resources (e.g., energy) in such a way that the increase in strength of some synapses in the
neuron is compensated for by a decrease in strength in others. Accordingly, only the “successful”
synapses can grow in strength, while the less successful synapses tend to weaken and may eventually
disappear altogether.

64
 Self-Organizing Map

At the network level, a similar competitive process may prevail, by proceeding as follows (Rumelhart
and Zipser, 1985):
➢ To begin with, the neurons in the network are all the same, except for some randomly distributed
synaptic weights; hence, the neurons respond differently to a given set of input patterns.
➢ A specific limit is imposed on the “strength” (e.g., the sum of synaptic weights) of each neuron in
the network.
➢ The neurons compete with each other in accordance with a prescribed rule for the right to
respond to a given subset of inputs; consequently, only one output neuron, or one neuron per
group, is active at a time. The neuron that wins the competition is called a winner-takes-all neuron.
This competitive-learning process, the individual neurons of the network assume the role of feature
detectors for different classes of input patterns. Whereas in Hebbian learning, several output
neurons of a neural network may be active simultaneously, in competitive learning only a single
output neuron, or one output neuron per group, is active at any one time. It is this characteristic of
competitive learning that makes it highly suited to discovering statistically salient features that
could be used to classify a set of input patterns.
65
Self-Organizing Map

Self-organizing maps networks are based on competitive learning; the output neurons of the
network compete among themselves to be activated or fired, with the result that only one
output neuron, or one neuron per group, is on at any one time.

An output neuron that wins the competition is called a winner-takes-all neuron, or simply a
winning neuron.

One way of inducing a winner-takes-all competition among the output neurons is to use lateral
inhibitory connections (i.e., negative feedback paths) between them; such an idea was
originally proposed by Rosenblatt (1958).

66
Self-Organizing Map

In a self-organizing map, the neurons are placed at the nodes of a lattice that is usually one or two
dimensional. Higher-dimensional maps are also possible but not as common.

The neurons become selectively tuned to various input patterns (stimuli) or classes of input patterns
in the course of a competitive-learning process.

The locations of the neurons so tuned (i.e., the winning neurons) become ordered with respect to
each other in such a way that a meaningful coordinate system for different input features is created
over the lattice.

A self-organizing map is therefore characterized by the formation of a topographic map of the

input patterns, in which the spatial locations (i.e., coordinates) of the neurons in the lattice are
indicative of intrinsic statistical features contained in the input patterns—hence, the name
“self-organizing map.”

67
Self-Organizing Map

As a neural model, the self-organizing map provides a bridge between two levels of
adaptation:

➢ Adaptation rules formulated at the microscopic level of a single neuron.

➢ Formation of experientially better and physically accessible patterns of feature selectivity at the
microscopic level of neural layers.

The self-organizing map is inherently nonlinear.

The development of self-organizing maps as a neural model is motivated by a distinct feature of

the human brain:
The brain is organized in many places in such a way that different sensory inputs are
represented by topologically ordered computational maps.

68
TWO BASIC FEATURE-MAPPING MODELS

In particular, sensory inputs such as tactile (Kaas et al., 1983), visual (Hubel and Wiesel, 1962,
1977), and acoustic (Suga, 1985) inputs are mapped onto different areas of the cerebral cortex in
a topologically ordered manner. Thus, the computational map constitutes a basic building block
in the information-processing infrastructure of the nervous system.

69
 Self-Organizing Map
TWO BASIC FEATURE-MAPPING MODELS

Different sensory inputs (motor, somatosensory, visual, auditory, etc.) are mapped onto corresponding
areas of the cerebral cortex in an orderly fashion

Computational maps offer four properties (Knudsen et al., 1987; Durbin and Michison, 1990):

➢ In each map, neurons act in parallel and process pieces of information that are similar in
nature, but originate from different regions in the sensory input space.

➢ At each stage of representation, each incoming piece of information is kept in its proper context.

➢ Neurons dealing with closely related pieces of information are close together so that they can
interact via short synaptic connections.

➢ Contextual maps can be understood in terms of decision-reducing mappings from higher

dimensional parameter spaces onto the cortical surface.
70
Self-Organizing Map TWO BASIC FEATURE-MAPPING MODELS

Our interest lies in building artificial topographic maps that learn through self organization in a
neurobiologically inspired manner.

The spatial location of an output neuron in a topographic map corresponds to a particular

domain or feature of data drawn from the input space.

71
 Self-Organizing Map

TWO BASIC FEATURE-MAPPING MODELS

➢ Willshaw and von der Malsburg (1976)

➢ Kohonen (1982)

72
 Self-Organizing Map

TWO BASIC FEATURE-MAPPING MODELS

➢ Willshaw and von der Malsburg (1976)

Post synaptic neurons

Presynaptic neurons

73
 Self-Organizing Map

TWO BASIC FEATURE-MAPPING MODELS

➢ Willshaw and von der Malsburg (1976)

The model in Fig. 9.1a was originally proposed by Willshaw and von der Malsburg (1976) on
biological grounds to explain the problem of retinotopic mapping from the retina to the visual
cortex (in higher vertebrates).

One lattice represents presynaptic (input) neurons, and the other lattice represents postsynaptic
(output) neurons.

The postsynaptic lattice uses a short-range excitatory mechanism as well as a long-range

inhibitory mechanism. These two mechanisms are local in nature and critically important for self-
organization.
74
 Self-Organizing Map

TWO BASIC FEATURE-MAPPING MODELS

➢ Willshaw and von der Malsburg (1976)

The basic idea of the Willshaw–von der Malsburg model is for the geometric proximity of
presynaptic neurons to be coded in the form of correlations in their electrical activity, and to use
these correlations in the postsynaptic lattice so as to connect neighboring presynaptic neurons to
neighboring postsynaptic neurons.

A topologically ordered mapping is thereby produced through a process of self-organization.

Note, however, that the Willshaw–von der Malsburg model is specialized to mappings for which
the input dimension is the same as the output dimension.

75
 Self-Organizing Map

TWO BASIC FEATURE-MAPPING MODELS

➢ Kohonen (1982),

The model captures the essential features of computational maps in the brain and yet remains
computationally tractable. It appears that the Kohonen model is more general than the
Willshaw–von der Malsburg model in the sense that it is capable of performing data
compression (i.e., dimensionality reduction on the input).

76
 Self-Organizing Map

TWO BASIC FEATURE-MAPPING MODELS

➢ Kohonen (1982),

The Kohonen model belongs to the class of vector-coding algorithms. The model provides a
topological mapping that optimally places a fixed number of vectors (i.e., code words) into a higher-
dimensional input space, thereby facilitating data [Link] Kohonen model may therefore be
derived in two ways. First, we may use basic ideas of self-organization, motivated by neurobiological
considerations, to derive the model, which is the traditional approach (Kohonen, 1982, 1990, 1997).
Alternatively, we may use a vector quantization approach that uses a model involving an encoder and a
decoder; this way is motivated by considerations of communication theory (Luttrell, 1989b, 1991a).
77
SELF-ORGANIZING MAP

The principal goal of the self-organizing map (SOM) is to transform an incoming signal pattern of
arbitrary dimension into a one- or two-dimensional discrete map, and to perform this transformation
adaptively in a topologically ordered fashion. Figure 9.2 shows the schematic diagram of a two-
dimensional lattice of neurons commonly used as a discrete map.

78
SELF-ORGANIZING MAP

Each neuron in the lattice is fully connected to all the source nodes in the input layer. This network
represents a feedforward structure with a single computational layer consisting of neurons arranged in
rows and columns. A one-dimensional lattice is a special case of the configuration depicted in Fig. 9.2:
in this special case, the computational layer consists simply of a single column or row of neurons.

Each input pattern presented to the network typically consists of a localized region or “spot” of activity
against a quiet background. The location and nature of such a spot usually varies from one realization of
the input pattern to another. All the neurons in the network should therefore be exposed to a sufficient
number of different realizations of the input pattern in order to ensure that the self-organization process
has a chance to develop properly.
The algorithm responsible for the formation of the self-organizing map proceeds first by initializing the
synaptic weights in the network. This can be done by assigning them small values picked from a
random-number generator; in so doing, no prior order is imposed on the feature map.

79
SELF-ORGANIZING MAP

Once the network has been properly initialized, there are three essential processes involved in the
formation of the self-organizing map, as summarized here:

➢ Competition

➢ Cooperation

➢ Synaptic Adaptation

80
SELF-ORGANIZING MAP

Once the network has been properly initialized, there are three essential processes involved in the
formation of the self-organizing map, as summarized here:

➢ Competition

For each input pattern, the neurons in the network compute their respective values of a
discriminant function. This discriminant function provides the basis for competition among the
neurons. The particular neuron with the largest value of discriminant function is declared winner
of the competition

81
SELF-ORGANIZING MAP

Once the network has been properly initialized, there are three essential processes involved in the
formation of the self-organizing map, as summarized here:

➢ Cooperation.

The winning neuron determines the spatial location of a topological neighborhood of excited
neurons, thereby providing the basis for cooperation among such neighboring neurons.

82
SELF-ORGANIZING MAP

Once the network has been properly initialized, there are three essential processes involved in the
formation of the self-organizing map, as summarized here:

➢ Synaptic Adaptation.

This last mechanism enables the excited neurons to increase their individual values of the
discriminant function in relation to the input pattern through suitable adjustments applied to their
synaptic weights. The adjustments made are such that the response of the winning neuron to the
subsequent application of a similar input pattern is enhanced.

83
 SELF-ORGANIZING MAP

There are three basic steps involved in the application of the algorithm after initialization:
sampling, similarity matching, and [Link] three steps are repeated until formation of the
feature map has been [Link] algorithm is summarized as follows:

84
SELF-ORGANIZING MAP

85
PROPERTIES OF THE FEATURE MAP

86
PROPERTIES OF THE FEATURE MAP

87
PROPERTIES OF THE FEATURE MAP

88
PROPERTIES OF THE FEATURE MAP

89
PROPERTIES OF THE FEATURE MAP

90
PROPERTIES OF THE FEATURE MAP

91
PROPERTIES OF THE FEATURE MAP

92
PROPERTIES OF THE FEATURE MAP

93
PROPERTIES OF THE FEATURE MAP

94
PROPERTIES OF THE FEATURE MAP

95
PROPERTIES OF THE FEATURE MAP

input–output mapping corrupted by additive noise. In such a situation, principal-components

analysis works perfectly fine: It tells us that the best description of the “linear” distribution
in Fig. 9.7a is defined by a straight line (i.e., one-dimensional “hyperplane”) that passes
through the origin and runs parallel to the eigenvector associated with the largest eigenvalue
of the correlation matrix of the data. Consider next the situation described in Fig. 9.7b,
which is the result of a nonlinear input–output mapping corrupted by additive noise of zero
mean. In this second situation, it is impossible for a straight-line approximation computed
from principal-components analysis to provide an acceptable description of the data. On
the other hand, the use of a self-organizing map built on a one-dimensional lattice of neurons
is able to overcome this approximation problem by virtue of its topological-ordering
[Link] latter approximation, illustrated in Fig. 9.7b, works well only when the
dimensionality of the lattice matches the intrinsic dimensionality of the distribution.

96
CASE STUDIES- REFER JOURNALS

97