experiments/STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Comment on why NCCL can't be ordered in tf1.

Browse Source 
PiperOrigin-RevId: 355484203
Change-Id: I23606db9759b72355f58147cdf7ad61d3076641d
This commit is contained in:
Chenkai Kuang 2021-02-03 14:33:26 -08:00 committed by TensorFlower Gardener
parent a5e16cb4d1
commit 6ff048e951
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
tensorflow/python/distribute/cross_device_utils.py
View File

@ -286,6 +286,8 @@ class CollectiveReplicaLauncher(object):
return CollectiveReplicaLauncher._prefer_unique_instance_key return CollectiveReplicaLauncher._prefer_unique_instance_key
def _use_ordering_token(self): def _use_ordering_token(self):
# We rely on auto control dep to insert control edges between NCCL calls,
# but for tf1 graph mode auto control dep is not used.
if not ops.executing_eagerly_outside_functions(): if not ops.executing_eagerly_outside_functions():
return False return False
return CollectiveReplicaLauncher._prefer_ordering_token return CollectiveReplicaLauncher._prefer_ordering_token