[DNN] Learning Rate Dacay

Learning rate decay

• Another way to speed up the learning algorithm is to slowly reduce the learning rate over time.

• If the learning rate is fixed over all the iterations, It might not converge to the cost minimum.
• If the learning rate is reduced, It is highly likely to converge.

※ Other learning rate decay methods