Hi guys,

I've got an assignment for one of my classes to write a neural network for recognising patterns. The patterns are ascii files depicting images like a smiley face, or a stop sign. Everything has been going well (as far as I can tell) with the development, but I've hit a bit of a wall. I'm running the network with 12 patterns for training, with twelve outputs (one output for a pattern).

The problem I'm having is that after training, the network outputs the same output, regardless of what input is given. I'm sure the problem is something simple, but I have no idea where to start. I'm sure it's something that people have seen before, so I figured a fresh set of eyes might be good.

In case I'm being a little unclear, I'll give some examples. These are the outputs of the network after being trained with only 1 pattern, "Clock.txt". Correct recognition of Clock.txt should output 1 on output 0, and obviously 0 on everything else.

Test for Clock.txt (
Output 0: 0.996912
Output 1: 0.003909
Output 2: 0.002875
Output 3: 0.018395
Output 4: 0.007166
Output 5: 0.015837
Output 6: 0.018766
Output 7: 0.018764
Output 8: 0.018743
Output 9: 0.018719
Output 10: 0.018760
Output 11: 0.018642

But no matter what file you give it for testing, it will always output (nearly) 1 for output 0, and (nearly) 0 for every other output. An example here is the output for "Stop.txt".

Test for Stop.txt
Output 0: 0.998664
Output 1: 0.017184
Output 2: 0.000995
Output 3: 0.008642
Output 4: 0.013252
Output 5: 0.009622
Output 6: 0.010032
Output 7: 0.071634
Output 8: 0.005989
Output 9: 0.054927
Output 10: 0.008992
Output 11: 0.010151

Similarly, if I train it with all twelve patterns, here is the output for testing the same two files as above:

Test for Clock.txt
Output 0: 0.000359
Output 1: 0.001865
Output 2: 0.072090
Output 3: 0.001352
Output 4: 0.000179
Output 5: 0.004311
Output 6: 0.051663
Output 7: 0.001231
Output 8: 0.006009
Output 9: 0.003236
Output 10: 0.001374
Output 11: 0.001719

Test for Stop.txt
Output 0: 0.000064
Output 1: 0.006168
Output 2: 0.041046
Output 3: 0.000080
Output 4: 0.000400
Output 5: 0.003056
Output 6: 0.028817
Output 7: 0.002576
Output 8: 0.008774
Output 9: 0.001774
Output 10: 0.001356
Output 11: 0.000421

Then the network outputs (nearly) 0 for everything! If anyone needs me to post any of the code or anything, I'm more than happy too, but I'm not sure at this point what else someone may need, so I leave it to you to ask.

Thanks heaps for any help!

Probably worth mentioning, it's a feed forward network, using back-propagation with one hidden layer.

Hey again guys.

Just thought I'd post a follow up, I've solved this problem now. If anyone else comes across this post with the same problem and wants to know the solution, here it is.

The input layer was handling the input incorrectly. Simple as that. Other problems that probably factored into it were that I was calculating all the deltas, then correcting all the weights. The way I've done it now (that works) is to calculate the deltas for a layer, correct that layer, and repeat for each other layer. Hope I help for anyone that reads this!

I rate you up just for reporting back and posting your solution :)

I'll jump on this bandwagon ++ good sir for being new and coming back to post your solution instead of forgetting about the thread as many do.

Thanks guys! If anyone wants the code to look at, just ask.

Although now I've found another bug. Every run, there are a few patterns the network can't classify at all. It classifies about 6-8 out of 12 right on every run. I'm not sure if this is a bug though as I seem to remember my lecturer mentioning something about neural nets doing things like this sometimes.

Oh well, if I solve it I'll post back here!

Edit: No idea what the bug is yet, but I thought I'd post back to say that the RMS error is NEVER any lower than 0.2. If this means anything to anyone and they know where my problem might be, I'd love to hear from you!

[Edited by - NathanX on October 9, 2010 4:21:45 AM]

Problem solved!

When calculating the weight corrections for each Neuron, I was using the output of that Neuron, as opposed to the output of the corresponding Neuron from the previous layer. Really easy to miss!

My code has had that implemented for about a week and a half, and comparing it to the algorithms it looked like I'd done everything correctly. I only realised I had the bug when I sat down to work out the weight corrections on paper.

The RMS of errors is now 0.01, a much much nicer value.

Quote: Original post by NathanX
It classifies about 6-8 out of 12 right on every run. I'm not sure if this is a bug though as I seem to remember my lecturer mentioning something about neural nets doing things like this sometimes.

Yes, there is absolutely no guarantee that the particular neural net you have will be able to correctly classify each of the items you train it with. (Although your lecturer would presumably have set you a task with parameters where this is not the case, but it's still worth knowing.) Do a web search for the 'XOR problem' to show a very trivial example of this.