Comment by WithinReason

Comment by WithinReason 2 days ago

Karpathy suggests the following error:

  def clipped_error(x): 
    return tf.select(tf.abs(x) < 1.0, 
                   0.5 * tf.square(x), 
                   tf.abs(x) - 0.5) # condition, true, false

Following the same principles that he outlines in this post, the "- 0.5" part is unnecessary since the gradient of 0.5 is 0, therefore -0.5 doesn't change the backpropagated gradient. In addition, a nicer formula that achieves the same goal as the above is √(x²+1)

macleginn 2 days ago

If we don't subtract from the second branch, there will be a discontinuity around x = 1, so the derivative will not be well-defined. Also the value of the loss will jump at this value, which will make it hard to inspect the errors, for one thing.

Reply View 2 replies

WithinReason 2 days ago

No, that's not how backprop works. There will be no discontinuity in a backpropagated gradient.

Reply View | 1 reply
- macleginn 2 days ago
  
  I did not say there will be a discontinuity in the gradient; I said that the modified loss function will not have a mathematically well-defined derivative because of the discontinuity in the function.
  
  Reply View | 0 replies

kingstnap 2 days ago

You do that to make things smoother when plotted. You could in theory add some crazy stairstep that adds a hundred to the middle part. It would make your loss curves spike and increase towards convergence but then those spikes are just visual artifacts from doing weird discontinuous nonsense with yoru loss.

Reply View 0 replies

slashdave 2 days ago

square roots are expensive

Reply View 1 reply

WithinReason 2 days ago

they are negligible, especially when the post was written when ops were not fused. The extra memory you need to store the extra tensors when you use the original version is more expensive

Reply View | 0 replies