After quite a detailed look at supervised networks (see Perceptrons, Back-propagation and Associative Networks) we should look at a good example of unsupervised networks. The Kohonen Network is probably the best example, because it is quite simple yet introduces the concept of self-organization and unsupervised training easily.
There are many researchers who require biological plausibility in proposed neural network models. Indeed, they have a right to - since the aim of (most) networks is to emulate the brain. It is generally accepted that perceptrons, back-propagation and many other techniques are not biological plausible for several reasons I will not go into. With the demand for biological plausibility rising, the concept of self-organizing networks became a point of interest amoung researchers. Self-organizing networks could be both supervised or unsupervised, and have four additional properties:
There are quite a few types of self-organizing networks, like the Instar-Outstar network, the ART-series, and the Kohonen network. For purposes of simplicity, we will look at the Kohonen network. The term Kohonen network is a slightly misleading one, because the researcher Teuvo Kohonen in fact researched many kinds of network, but only a small number of these are called Kohonen networks. We will look at the idea of self-organizing maps, networks that attempt to 'map' their weights to a set of coordinates. The Kohonen network is an n-dimensional network, where n is the number of inputs. For simplicity we will look at a 2-dimensional networks. The schematic architecture of the network would look like this:
The above picture only shows a little bit of the network, but you can see how every neuron gets the same input, and there is one output per neuron. To help us visual the problem of mapping the weights to coordinates, imagine that all the weights of the network were initialized to a random value. The weights are then graphed on a standard Cartesian graph. Immediately adjacent (above, below, left, right) neurons are then connected via lines. Remember, these lines are merely schematic and do not represent anything in the net itself, there are no connections between neurons in the network. Now, the networks aim is to "expand" out to fill the graph paper in some ordered way. Nothing beats a diagram at this point - here are four screenshots from my Kohonen program at various stages of its training:
Note that the top right image seems to have less points - I can only attribute this to the network mapping multiple weights to the same coordinate to begin with, but they diversify as the network learns more. Now that you can visualize what the network is doing, let us looking at how it does it. The basic idea behind the Kohonen network is competitive learning. The neurons are presented with the inputs, which calculate their net (weighted sum) and neuron with the closest output magnitude is chosen to receive additional training. Training, though, does not just affect the one neuron but also its neighbours. So, how does one judge what the 'closest output magnitude' is? One way is to find the distance between the input and net of the neuron:
Notice that when applied to our 2-dimensional network, it reduces down to the standard Euclidean distance formula. So, if we want the output the closely represents the input pattern, it is the neuron with the smallest distance. Let us call the neuron with the least distance xd0. Now, remember that we change both the neuron and the neurons in its neighbourhood Nx. Nx is not constant, it can change from anything ranging between the entire network to just the 6 adjacent neurons. We will talk about the neighbourhood soon. Kohonen learning is very simple, following a familiar equation:
Where k is the learning coefficient. So all neurons is neighbourhood Nx to neuron xd0 have their weights adjusted. So how do we adjust k and Nx during training? This is an area of much research, but Kohonen has suggest splitting the training up into two phases. Phase 1 will reduce down the learning coefficient from 0.9 to 0.1 (or similar values), and the neighbourhood will reduce from half the diameter of the network down to the immediately surrounding cells (Nx = 1). Following that, Phase 2 will reduce the learning coefficient from perhaps 0.1 to 0.0 but over double or more the number of iterations in Phase 1. The neighbourhood value is fixed at 1. You can see that the two phases allow firstly the network to quickly 'fill out the space' with the second phase fine-tuning the network to a more accurate representation of the space. Refer back to the diagram, the bottom left picture actually shows the network right after Phase 1 has finishes, with the bottom right one after the second phase is complete. Interpreting Kohonen networks is quite easy since only one neuron will fire per input set after training. Therefore, it is a case of classifying the outputs. For example, if this neuron fires do this - or if this group of neurons fire do this etc. Kohonen networks have been successfully applied to speech recognition, since it is after cognitive networks that inspired self-organizing networks. Kohonen networks can also be well applied to gaming - by expanding the dimensionality (number of inputs) you can create much more complicated mappings, far beyond the redundant example explained above (and demonstrated in my program!). 
|
