experiments/STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Fix TPU embedding implementation initialization of proximal yogi optimizer.

Browse Source 
Similar to Adagrad, setting initialization to 0.1 (instead of 0.0) will avoid numerical instability issues.

PiperOrigin-RevId: 288327127
Change-Id: I66a89bbb4e18fe653479cb64003b7c42daba7158
This commit is contained in:
A. Unique TensorFlower 2020-01-06 10:29:49 -08:00 committed by TensorFlower Gardener
parent 28018f2cd7
commit 8033e41d7c
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
tensorflow/core/tpu/tpu_embedding_optimization_parameters_utils.cc
View File

@ -265,7 +265,7 @@ Status GetOptimizationAlgorithmStateVariables(
}
case OptimizationAlgorithm::kProximalYogi: {
state_variables->push_back(
MakeStandardStateVariableSpecification("v", 0.0));
MakeStandardStateVariableSpecification("v", 0.1));
state_variables->push_back(
MakeStandardStateVariableSpecification("m", 0.0));
break;