Debugging Neural Network Gradients
Building LiveAPI, a tool for generating API docs from code. Fixed bug in automatic gradient calculation by accumulating gradients instead of replacing them.
Hi there! I'm Shrijith Venkatrama, founder of Hexmos. Right now, I’m building LiveAPI, a tool that makes generating API docs from your code ridiculously easy.
A Bug In Our Code
In the previous post, we got automatic gradient calculation going for the whole expression graph.
However, it has a tricky bug. Here's a sample program that invokes the bug:
a = Value(3.0, label='a')
b = a + a ; b.label = 'b'
b.backward()
draw_dot(b)
In the above, forward pass looks alright:
b = a + a = 3 + 3 = 6
But think about the backward pass:
b = a...