The great thing about this is being amongst experts, and also not being distracted from getting on with it.
I had a chat with a couple of experts on image classification and it was brought to my attention that it is pretty normal for neural nets to be trained on the training dataset multiple times - epochs. I was staggered! I tried it and the results improved!
That's the massive benefit of just talking to people!
I also found the time to finally measure the accuracy of the neural network predictions against the test set - previously I had been measuring the output error as the training proceeded.
The accuracy against the training set jumped to a fantastic 97% (and later to 97.5% with 50 epochs)
Overall the results are now in line with what we should expect from a neural network - see http://yann.lecun.com/exdb/mnist/ benchmarks for different methods.
