Comment by tossandthrow

I think the easiest way to think about this is in terms of natural numbers, ie. 1, 2, 3, 4.

When you only have a fixed width, ie. a static feed forward network, you have an upper limit to the data you can represent and compute on.

Eg. if the highest number you can represent is 1.000, then you will need a new NN if you want to do computations on 1.001.

... or use an inductive structure, like a recurrent neural network has.