experiments/STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

[ROCm] Adding no_rocm tag to a unit-test regression.

Browse Source 
The following commit to bump up the LLVM commit pointer, introduces a unit-test regression on the ROCm platform.

4de4c60972

There are changes other than LLVM commit pointer bump, in the above commit, but it is the change in LLVm version that seems to be causing the regression.

The failing unit-test is

```
//tensorflow/python/keras/optimizer_v2:adam_test_gpu                     FAILED in 13.7s
...

...
[ RUN      ] NonFusedAdamOptimizerTest.testSparse
2020-06-02 11:32:19.520540: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libhip_hcc.so
2020-06-02 11:32:19.661617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1562] Found device 0 with properties:
pciBusID: 0000:23:00.0 name: Vega 10 [Radeon Instinct MI25]     ROCm AMD GPU ISA: gfx900
...
...
2020-06-02 11:32:28.304094: I tensorflow/compiler/jit/xla_compilation_cache.cc:314] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
'+fp64-fp16-denormals' is not a recognized feature for this target (ignoring feature)
'-fp32-denormals' is not a recognized feature for this target (ignoring feature)
'+fp64-fp16-denormals' is not a recognized feature for this target (ignoring feature)
'-fp32-denormals' is not a recognized feature for this target (ignoring feature)
'+fp64-fp16-denormals' is not a recognized feature for this target (ignoring feature)
'-fp32-denormals' is not a recognized feature for this target (ignoring feature)
'+fp64-fp16-denormals' is not a recognized feature for this target (ignoring feature)
'-fp32-denormals' is not a recognized feature for this target (ignoring feature)
Memory access fault by GPU node-4 (Agent handle: 0x345e4b0) on address 0x7f4200258000. Reason: Page not present or supervisor privilege.
Fatal Python error: Aborted
...
```

This commit puts a no_rocm tag on that test to get the ROCm CSB to pass, while we root cause the regression and come up with a fix
This commit is contained in:
Deven Desai 2020-06-02 14:47:24 +00:00
parent 5c3cdff00b
commit 4cb805dba2
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
tensorflow/python/keras/optimizer_v2/BUILD
View File

@ -105,6 +105,7 @@ cuda_py_test(
size = "medium",
srcs = ["adam_test.py"],
shard_count = 4,
tags = ["no_rocm"],
deps = [
":optimizer_v2",
"//tensorflow/python:client_testlib",