From b5085d20140eb4f492adaaefc1c307ad7ddcc197 Mon Sep 17 00:00:00 2001 From: "A. Unique TensorFlower" Date: Mon, 9 Mar 2020 10:50:11 -0700 Subject: [PATCH] Correct the formulation in the description of the centered rmsprop PiperOrigin-RevId: 299879992 Change-Id: I8e0ec675cc933a11ff8ac8a7e29ca0f4de030d36 --- tensorflow/python/training/rmsprop.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tensorflow/python/training/rmsprop.py b/tensorflow/python/training/rmsprop.py index 90ab83f1ad4..05f449153dc 100644 --- a/tensorflow/python/training/rmsprop.py +++ b/tensorflow/python/training/rmsprop.py @@ -30,7 +30,7 @@ This implementation of RMSProp uses plain momentum, not Nesterov momentum. The centered version additionally maintains a moving (discounted) average of the gradients, and uses that average to estimate the variance: -mean_grad = decay * mean_square{t-1} + (1-decay) * gradient +mean_grad = decay * mean_grad{t-1} + (1-decay) * gradient mean_square = decay * mean_square{t-1} + (1-decay) * gradient ** 2 mom = momentum * mom{t-1} + learning_rate * g_t / sqrt(mean_square - mean_grad**2 + epsilon)