Starting the server before we (optionally) sleep for the case where no device_filters are set. Otherwise, the servers will wait to connect to each other before starting to train. We might as well start as soon as we can.

Change: 132376998
This commit is contained in:
A. Unique TensorFlower 2016-09-06 15:37:07 -08:00 committed by TensorFlower Gardener
parent cde9adb6d5
commit 0949ecee16

View File

@ -115,14 +115,18 @@ class Experiment(object):
The trained estimator.
"""
start = time.time()
# Start the server, if needed. It's important to start the server before
# we (optionally) sleep for the case where no device_filters are set.
# Otherwise, the servers will wait to connect to each other before starting
# to train. We might as well start as soon as we can.
if self._estimator.config.cluster_spec:
self._start_server()
if delay_secs is None:
task_id = self._estimator.config.task or 0
delay_secs = min(60, task_id * 5)
# Start the server, if needed.
if self._estimator.config.cluster_spec:
self._start_server()
if delay_secs:
elapsed_secs = time.time() - start
remaining = delay_secs - elapsed_secs