Inside the Mind of a Neural Network

Neural Network's Knowledge

Neural networks learn to solve problems in ways that are not entirely clear to us. The reason we use neural networks is that some problems are not easily reducible to simple solutions made up of a few short crisp rules. They're able to solve problems where we don't really know how to approach those problems, or what the key factors are in deciding what the answer should be.

In this sense, the "knowledge" that a neural network gains during training is somewhat mysterious and largely not understandable to us - the "mind" of a neural network.

Is there any way of seeing what this "mind" has become? It would be like seeing the dreams or hallucinations of a neural mind.

Backwards Query

Here's a crazy idea.

Normally we feed a question to a neural network's input nodes to see what the answer is popping out of the output nodes.

Why not feed answers into the output nodes - to see what questions pop out of the input nodes?
 

Since our network learns to recognise images of handwritten numbers - we can interpret the stuff that pops out of the input nodes as an image too, and hopefully an interesting image.

If we put signals that represent the label "5" into the output nodes, and get an image out of the input nodes, it makes sense that that the image is an insight into what the mind of the neural network thinks of "5".

Let's try it! The code to do this is on github:

• https://github.com/makeyourownneuralnetwork/makeyourownneuralnetwork/blob/master/part3_neural_network_mnist_backquery.ipynb

The code is really simple:

• We just propagate the signals backwards, using the link weights to moderate the signals as usual. Remember, these weights have already been refined through previous training - that is, the network has already "learned". 
• The inverse of the logistic sigmoid is used when the signal goes backwards across a node, and it happens to be called a logit function.
• There is an extra step where we rescale the signals at a layer before we use the logit function because they must be between 0 and 1 (not including 0 and 1). That is, logit(x) does not exist for x<=0 and x>=1.

The Label "0"

If we back query with the label "0" we get the following image popping out of the input nodes:

Well - that's interesting!

We have here a privileged insight into the mind of a neural network. As we'll see next, this is an insight into what and how it thinks.

How do we interpret this image?

• Well it is round, which makes sense because we are asking it what the ideal question image would be to get a perfect output signal for the label "0".
• There are light and dark and grey bits. 
• The dark bits show which parts of the input image, if marked by handwriting, strongly indicate the answer would be a "0".
• The light bits show which parts of the input image definitely need to be clear of any handwriting marks to indicate the answer would be a "0".
• The grey bits are indifferent.

So we have actually understood what the neural network has learned about classifying images as "0".

More Brain-Scans

Let's have a look at the images that emerge from back-querying the rest of the digits (click to enlarge)

Wow - again some really interesting images.

They're like ultrasound scans into the brain of the neural network.

Some notes about these images:

• The "7" is really clear. You can see the dark bits which, if marked in the query image, strongly suggest a a label "7". You can also see the additional "white" area which must be clear of any marking. Together these two characteristics indicate a "7".
• Same applies to the "3" - there area areas which, if marked, indicate a "3", and areas which must be clear too. The "5" is similarly apparent too.
• The "4" is interesting in that there is a shape which appears to have 4 quadrants, and excluded areas too. 
• The "8" is largely made up of white areas suggesting that an eight is characterised by markings kept out of the central regions.

Some of the images are very clear and definite but some aren't and this might suggest that there is more learning to be done. Without more learning, the confidence in the network answering with the label "8" woud be lower than for a really clear "0", "5" or a "7".

Having said that, the network performs really really well anyway  - at 97.5% accuracy over the test set of 10,000 images!

So there you have it - a brain scan into the mind of a neural network!