experiments/STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Merge changes from github.

Browse Source 
END_PUBLIC

---
Commit 18f369271 authored by Yifei Feng<fengyifei2026@gmail.com>
Committed by gunan<gunan@google.com>:
Branch 167812735 (#12867)

* Internal cleanup

PiperOrigin-RevId: 167636242

* Move the Keras API to tf.keras.

PiperOrigin-RevId: 167638421

* Automated g4 rollback of changelist 167604306

PiperOrigin-RevId: 167639833

* Call HloComputation.Accept instead of HloInstruction.Accept to get all instructions profiled.

RELNOTES: n/a
PiperOrigin-RevId: 167640259

* Add fast math attributes to all generated methods when fast math enabled.

RELNOTES: n/a
PiperOrigin-RevId: 167646637

* Extended ScratchSpace to expose its underlying scratch tensor object.

PiperOrigin-RevId: 167649551

* Change zip(...)[1] to list(zip(...))[1], for python 3 compatibility.

PiperOrigin-RevId: 167654035

* Add scoped timer to log jit compile times.

RELNOTES: n/a
PiperOrigin-RevId: 167656720

* Verify that predictions are in the expected range for ops that use thresholds, e.g. tf.contrib.metrics.streaming_auc.

PiperOrigin-RevId: 167658134

* Internal change.

PiperOrigin-RevId: 167658401

* Fix list formatting

PiperOrigin-RevId: 167660250

* Enable java test.

PiperOrigin-RevId: 167660276

* Add shape functions on debug ops.

PiperOrigin-RevId: 167668811

* Increase session_bundle_test to a medium test.

PiperOrigin-RevId: 167672587

* Include layout of convolution input data in the op_profile.

PiperOrigin-RevId: 167680208

* Fix tf.sparse_add for SparseTensor with _ref typed values.

Example:
st = tf.SparseTensor(
    indices=[[1]], values=tf.Variable([1.0]), dense_shape=[1])
tf.sparse_add(st, st)
PiperOrigin-RevId: 167681121

* Fix conversion to explicit scalar broadcast

The dimensions field of a broadcast HLO op is meant to be populated with the
dimensions that are broadcasted, which in case of a scalar is the empty vector.
Generally, the rank of the operand of a broadcast op should always equal the
size of the dimensions vector.

PiperOrigin-RevId: 167686946

* Add 'unknown shape' shape functions on deprecated linalg ops.

PiperOrigin-RevId: 167719029

* Be more careful in IsInitalized, and log when it is called on an unknown
node_id.

PiperOrigin-RevId: 167722344

* tfdbg: Refactor graph-processing code out of debug_data.py

The basic idea is to separate the code in debug_data.py that handles graph structures into its own module (debug_graphs.py). This tackles an existing TODO item to simplify the code debug_data.DebugDumpDir.

In a later CL, code will be added to debug_graphs.DebugGraph to allow reconstruction of the original GraphDef, i.e., the GraphDef without the Copy* and Debug* nodes inserted by tfdbg. This will be useful for, among other things, the TensorBoard Debugger Plugin.

PiperOrigin-RevId: 167726113

* internal

PiperOrigin-RevId: 167727508

* Update MaxPoolV2Shape to support NCHV_VECT_C.

PiperOrigin-RevId: 167732437

* Deleting tf.contrib.learn.dnn benchmark tests.

PiperOrigin-RevId: 167741308

* Fix off-by-one documentation error.

sequence_lengths is the actual length of the sequence and therefor should not be used as zero-based indexing.
The code is correct but the documentation was misleading.

PiperOrigin-RevId: 167742082

* contrib summaries work in eager-graph mode (with defun)

As a side effect fix issues related to using eager-defined variables in graph
mode.

PiperOrigin-RevId: 167744121

* Fix minor documentation error in ZlibInputStream.

PiperOrigin-RevId: 167745218

* Sets the distributed training related properties of RunConfig based on TF_CONFIG.

PiperOrigin-RevId: 167752997

* Improved documentation about eval ops in EstimatorSpec.

PiperOrigin-RevId: 167753099

* Automated g4 rollback of changelist 156748870

PiperOrigin-RevId: 167753805

* Make cuda_solvers_gpu.cu.cc compile with nvcc8.

PiperOrigin-RevId: 167754383

* Add csv dataset example to get_started/regression.

PiperOrigin-RevId: 167754634

* Switches to OrderedDict to make the dictionary order deterministic so we have less randomness from graph building.

PiperOrigin-RevId: 167755072

* Add int8 version of fused_conv2d_bias_activation operator for the forward phase,
and support side_input and scaling parameters in float and int8 versions.

PiperOrigin-RevId: 167763219

* Make the text summary write no plugin data content

This is actually a safe removal because no logic makes use of the content of text plugin data.

PiperOrigin-RevId: 167763880

* Avoid unnecessary buffer allocations & deallocations

Before this change, when we reached the end of a file, we would
 (1) clear the existing buffer (which at large buffer sizes typically involved
     deallocating it).
 (2) reserve a buffer (which at large buffer sizes is non-trivial)
 (3) realize we had reached EoF, and therefore clear the buffer, deallocating
     it again.

With this change, whenever the buffered reader detects an EoF condition, we
remember it, so that we can short-circuit the above logic.

The above optimization results in a more than 25x performance improvement for
large buffers reading small files.

PiperOrigin-RevId: 167766751

* [TF:XLA] In Literal: correctly handle operands with zero elements in
Copy.

PiperOrigin-RevId: 167769308

* Reduce batch size for resampler backward pass test, to speed up test.

PiperOrigin-RevId: 167769539

* Remove `SimpleGraphExecutionState::costs_`, which is unused.

PiperOrigin-RevId: 167772120

* detecting cycles when users add a control edge to a graph

PiperOrigin-RevId: 167773598

* Make writer_test avoid setting content to a string

That content field of the PluginData proto is going to be converted into a bytes field, and setting it to a string makes the test fail. Furthermore, the purpose of this test is to make sure that correct data is written, so setting the name of the plugin suffices.

PiperOrigin-RevId: 167776457

* Propagate the original stack trace when exceptions caught be MonitoredSession
are re-raised.

PiperOrigin-RevId: 167781071

* Change trace.py to not access a graph as a default argument.

Checks for None and access via default graph inside the function.

PiperOrigin-RevId: 167788815

* Added custom metric support for tf.estimator.Estimator.

PiperOrigin-RevId: 167788891

* A eager Saver that allows restore on create.

PiperOrigin-RevId: 167789332

* Make content field of PluginData a bytes field

The content field had previously been a string field, which had been problematic because
string fields can only store UTF-8 strings.

This problem can manifest in various ways. For instance, take the precision-recall curve plugin. Its summary collects data that scales in size based on the number of thresholds. When the content field is a string, the summary logic serializes the relevant data proto just fine when we only have a few thresholds (about 100). However, for large numbers of thresholds (ie, around 200), the summary logic fails to serialize and throws a cryptic error.

ValueError: '\x10\xc8\x01' has type str, but isn't valid UTF-8 encoding. Non-UTF-8 strings must be converted to unicode objects before being added.

Changing the content field to a bytes field fixes this issue because bytes fields are not restricted to UTF-8 strings. I just happened to have needed a long enough string for the string to no longer be a valid UTF-8 one.

PiperOrigin-RevId: 167790594

* Temporarily disable tf_should_use wrapper, since it can cause
python Graph/Operation/Tensor memory leaks.

PiperOrigin-RevId: 167790657

* Ensure using "path" as a URI will keep working.

PiperOrigin-RevId: 167793848

* Fix typo in graph transforms error message

PiperOrigin-RevId: 167796563

* Merge changes from github.
END_PUBLIC

---
Commit 607816029 authored by Eugene Brevdo<ebrevdo@google.com>
Committed by TensorFlower Gardener<gardener@tensorflow.org>:
Extended ScratchSpace to expose its underlying scratch tensor object.

PiperOrigin-RevId: 167649551

---
Commit db43fe68e authored by A. Unique TensorFlower<gardener@tensorflow.org>
Committed by TensorFlower Gardener<gardener@tensorflow.org>:
Add fast math attributes to all generated methods when fast math enabled.

RELNOTES: n/a
PiperOrigin-RevId: 167646637

---
Commit aebe8cc6f authored by A. Unique TensorFlower<gardener@tensorflow.org>
Committed by TensorFlower Gardener<gardener@tensorflow.org>:
Call HloComputation.Accept instead of HloInstruction.Accept to get all instructions profiled.

RELNOTES: n/a
PiperOrigin-RevId: 167640259

---
Commit 0ab137cd8 authored by A. Unique TensorFlower<gardener@tensorflow.org>
Committed by TensorFlower Gardener<gardener@tensorflow.org>:
BEGIN_PUBLIC
BEGIN_PUBLIC
Automated g4 rollback of changelist 167604306

PiperOrigin-RevId: 167868631
This commit is contained in:
A. Unique TensorFlower 2017-09-07 09:20:48 -07:00 committed by TensorFlower Gardener
parent 82ae84bf18
commit f8da474255
11 changed files with 105 additions and 57 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

128
tensorflow/c/eager/c_api.h
View File

@ -20,6 +20,25 @@ limitations under the License.
#include "tensorflow/c/c_api.h"
// Macro to control visibility of exported symbols in the shared library (.so,
// .dylib, .dll).
// This duplicates the TF_EXPORT macro definition in
// tensorflow/core/platform/macros.h in order to keep this .h file independent
// of any other includes.$a
#ifdef SWIG
#define TF_CAPI_EXPORT
#else
#if defined(COMPILER_MSVC)
#ifdef TF_COMPILE_LIBRARY
#define TF_CAPI_EXPORT __declspec(dllexport)
#else
#define TF_CAPI_EXPORT __declspec(dllimport)
#endif // TF_COMPILE_LIBRARY
#else
#define TF_CAPI_EXPORT __attribute__((visibility("default")))
#endif // COMPILER_MSVC
#endif // SWIG
#ifdef __cplusplus
extern "C" {
#endif
@ -30,11 +49,11 @@ extern "C" {
// TODO(ashankar): Merge with TF_Session?
typedef struct TFE_Context TFE_Context;
extern TFE_Context* TFE_NewContext(const TF_SessionOptions* opts,
TF_Status* status);
extern void TFE_DeleteContext(TFE_Context* ctx, TF_Status* status);
extern TF_DeviceList* TFE_ContextListDevices(TFE_Context* ctx,
TF_Status* status);
TF_CAPI_EXPORT extern TFE_Context* TFE_NewContext(const TF_SessionOptions* opts,
TF_Status* status);
TF_CAPI_EXPORT extern void TFE_DeleteContext(TFE_Context* ctx, TF_Status* status);
TF_CAPI_EXPORT extern TF_DeviceList* TFE_ContextListDevices(TFE_Context* ctx,
TF_Status* status);
// A handle to a tensor on a device.
//
@ -43,14 +62,15 @@ extern TF_DeviceList* TFE_ContextListDevices(TFE_Context* ctx,
// placed in memory of different devices or remote address spaces.
typedef struct TFE_TensorHandle TFE_TensorHandle;
extern TFE_TensorHandle* TFE_NewTensorHandle(TF_Tensor* t, TF_Status* status);
extern void TFE_DeleteTensorHandle(TFE_TensorHandle* h);
extern TF_DataType TFE_TensorHandleDataType(TFE_TensorHandle* h);
extern int TFE_TensorHandleNumDims(TFE_TensorHandle* h);
extern int64_t TFE_TensorHandleDim(TFE_TensorHandle* h, int dim_index);
extern const char* TFE_TensorHandleDeviceName(TFE_TensorHandle* h);
extern TF_Tensor* TFE_TensorHandleResolve(TFE_TensorHandle* h,
TF_Status* status);
TF_CAPI_EXPORT extern TFE_TensorHandle* TFE_NewTensorHandle(TF_Tensor* t,
TF_Status* status);
TF_CAPI_EXPORT extern void TFE_DeleteTensorHandle(TFE_TensorHandle* h);
TF_CAPI_EXPORT extern TF_DataType TFE_TensorHandleDataType(TFE_TensorHandle* h);
TF_CAPI_EXPORT extern int TFE_TensorHandleNumDims(TFE_TensorHandle* h);
TF_CAPI_EXPORT extern int64_t TFE_TensorHandleDim(TFE_TensorHandle* h, int dim_index);
TF_CAPI_EXPORT extern const char* TFE_TensorHandleDeviceName(TFE_TensorHandle* h);
TF_CAPI_EXPORT extern TF_Tensor* TFE_TensorHandleResolve(TFE_TensorHandle* h,
TF_Status* status);
// Create a new TFE_TensorHandle with the same contents as 'h' but placed
// in the memory of the device name 'device_name'.
@ -58,10 +78,10 @@ extern TF_Tensor* TFE_TensorHandleResolve(TFE_TensorHandle* h,
// that shares the underlying buffer. Otherwise, it currently requires at least
// one of the source or destination devices to be CPU (i.e., for the source or
// destination tensor to be placed in host memory).
extern TFE_TensorHandle* TFE_TensorHandleCopyToDevice(TFE_TensorHandle* h,
TFE_Context* ctx,
const char* device_name,
TF_Status* status);
TF_CAPI_EXPORT extern TFE_TensorHandle* TFE_TensorHandleCopyToDevice(TFE_TensorHandle* h,
TFE_Context* ctx,
const char* device_name,
TF_Status* status);
// Description of the TensorFlow op to execute.
//
@ -76,49 +96,49 @@ extern TFE_TensorHandle* TFE_TensorHandleCopyToDevice(TFE_TensorHandle* h,
// the additional sanity checks there seem unnecessary;
typedef struct TFE_Op TFE_Op;
extern TFE_Op* TFE_NewOp(TFE_Context* ctx, const char* op_or_function_name,
TF_Status* status);
extern void TFE_DeleteOp(TFE_Op* op);
TF_CAPI_EXPORT extern TFE_Op* TFE_NewOp(TFE_Context* ctx, const char* op_or_function_name,
TF_Status* status);
TF_CAPI_EXPORT extern void TFE_DeleteOp(TFE_Op* op);
// TODO(ashankar): TFE_OpSetDevice and TFE_Execute should not have a TFE_Context
// parameter. Instead, the TFE_Context should be captured when creating the
// TFE_Op.
extern void TFE_OpSetDevice(TFE_Op* op, TFE_Context* ctx,
const char* device_name, TF_Status* status);
TF_CAPI_EXPORT extern void TFE_OpSetDevice(TFE_Op* op, TFE_Context* ctx,
const char* device_name, TF_Status* status);
extern void TFE_OpAddInput(TFE_Op* op, TFE_TensorHandle* h, TF_Status* status);
TF_CAPI_EXPORT extern void TFE_OpAddInput(TFE_Op* op, TFE_TensorHandle* h, TF_Status* status);
extern TF_AttrType TFE_OpGetAttrType(TFE_Op* op, const char* attr_name,
unsigned char* is_list, TF_Status* status);
TF_CAPI_EXPORT extern TF_AttrType TFE_OpGetAttrType(TFE_Op* op, const char* attr_name,
unsigned char* is_list, TF_Status* status);
extern void TFE_OpSetAttrString(TFE_Op* op, const char* attr_name,
const char* value);
extern void TFE_OpSetAttrInt(TFE_Op* op, const char* attr_name, int64_t value);
extern void TFE_OpSetAttrFloat(TFE_Op* op, const char* attr_name, float value);
extern void TFE_OpSetAttrBool(TFE_Op* op, const char* attr_name,
unsigned char value);
extern void TFE_OpSetAttrType(TFE_Op* op, const char* attr_name,
TF_DataType value);
TF_CAPI_EXPORT extern void TFE_OpSetAttrString(TFE_Op* op, const char* attr_name,
const char* value);
TF_CAPI_EXPORT extern void TFE_OpSetAttrInt(TFE_Op* op, const char* attr_name, int64_t value);
TF_CAPI_EXPORT extern void TFE_OpSetAttrFloat(TFE_Op* op, const char* attr_name, float value);
TF_CAPI_EXPORT extern void TFE_OpSetAttrBool(TFE_Op* op, const char* attr_name,
unsigned char value);
TF_CAPI_EXPORT extern void TFE_OpSetAttrType(TFE_Op* op, const char* attr_name,
TF_DataType value);
// If the number of dimensions is unknown, `num_dims` must be set to
// -1 and `dims` can be null. If a dimension is unknown, the
// corresponding entry in the `dims` array must be -1.
extern void TFE_OpSetAttrShape(TFE_Op* op, const char* attr_name,
const int64_t* dims, const int num_dims,
TF_Status* out_status);
TF_CAPI_EXPORT extern void TFE_OpSetAttrShape(TFE_Op* op, const char* attr_name,
const int64_t* dims, const int num_dims,
TF_Status* out_status);
extern void TFE_OpSetAttrStringList(TFE_Op* op, const char* attr_name,
const char** value, int num_values);
extern void TFE_OpSetAttrIntList(TFE_Op* op, const char* attr_name,
const int64_t* values, int num_values);
extern void TFE_OpSetAttrFloatList(TFE_Op* op, const char* attr_name,
const float* values, int num_values);
extern void TFE_OpSetAttrBoolList(TFE_Op* op, const char* attr_name,
const unsigned char* values, int num_values);
extern void TFE_OpSetAttrTypeList(TFE_Op* op, const char* attr_name,
const TF_DataType* values, int num_values);
extern void TFE_OpSetAttrShapeList(TFE_Op* op, const char* attr_name,
const int64_t** dims, const int* num_dims,
int num_values, TF_Status* out_status);
TF_CAPI_EXPORT extern void TFE_OpSetAttrStringList(TFE_Op* op, const char* attr_name,
const char** value, int num_values);
TF_CAPI_EXPORT extern void TFE_OpSetAttrIntList(TFE_Op* op, const char* attr_name,
const int64_t* values, int num_values);
TF_CAPI_EXPORT extern void TFE_OpSetAttrFloatList(TFE_Op* op, const char* attr_name,
const float* values, int num_values);
TF_CAPI_EXPORT extern void TFE_OpSetAttrBoolList(TFE_Op* op, const char* attr_name,
const unsigned char* values, int num_values);
TF_CAPI_EXPORT extern void TFE_OpSetAttrTypeList(TFE_Op* op, const char* attr_name,
const TF_DataType* values, int num_values);
TF_CAPI_EXPORT extern void TFE_OpSetAttrShapeList(TFE_Op* op, const char* attr_name,
const int64_t** dims, const int* num_dims,
int num_values, TF_Status* out_status);
// Execute the operation defined by 'op' and return handles to computed
// tensors in 'retvals'.
@ -128,14 +148,14 @@ extern void TFE_OpSetAttrShapeList(TFE_Op* op, const char* attr_name,
//
// On return, 'num_retvals' will be set to the actual number of outputs
// returned by the operation.
extern void TFE_Execute(TFE_Op* op, TFE_TensorHandle** retvals,
int* num_retvals, TF_Status* status);
TF_CAPI_EXPORT extern void TFE_Execute(TFE_Op* op, TFE_TensorHandle** retvals,
int* num_retvals, TF_Status* status);
// Add a function (serialized FunctionDef protocol buffer) to ctx so
// that it can be invoked using TFE_Execute.
extern void TFE_ContextAddFunctionDef(TFE_Context* ctx,
const char* serialized_function_def,
size_t size, TF_Status* status);
TF_CAPI_EXPORT extern void TFE_ContextAddFunctionDef(TFE_Context* ctx,
const char* serialized_function_def,
size_t size, TF_Status* status);
#ifdef __cplusplus
} /* end extern "C" */

1
tensorflow/c/version_script.lds
View File

@ -2,6 +2,7 @@ VERS_1.0 {
# Export symbols in c_api.h.
global:
TF_*;
TFE_*;
# Hide everything else.
local:

1
tensorflow/contrib/cmake/tf_tests.cmake
View File

@ -247,6 +247,7 @@ if (tensorflow_BUILD_PYTHON_TESTS)
# Broken tensorboard test due to cmake issues.
"${tensorflow_source_dir}/tensorflow/contrib/data/python/kernel_tests/dataset_constructor_op_test.py"
"${tensorflow_source_dir}/tensorflow/contrib/data/python/kernel_tests/iterator_ops_cluster_test.py" # Needs portpicker
"${tensorflow_source_dir}/tensorflow/contrib/data/python/kernel_tests/sloppy_transformation_dataset_op_test.py" # b/65430561
# tensor_forest tests (also note that we exclude the hybrid tests for now)
"${tensorflow_source_dir}/tensorflow/contrib/tensor_forest/python/kernel_tests/count_extremely_random_stats_op_test.py" # Results in wrong order.
"${tensorflow_source_dir}/tensorflow/contrib/tensor_forest/python/kernel_tests/sample_inputs_op_test.py" # Results in wrong order.

1
tensorflow/contrib/data/python/kernel_tests/map_dataset_op_test.py
View File

@ -20,6 +20,7 @@ from collections import namedtuple
import os
import threading
from collections import namedtuple
import numpy as np

1
tensorflow/core/kernels/parse_tensor_op.cc
View File

@ -22,6 +22,7 @@ limitations under the License.
#include "tensorflow/core/framework/tensor_shape.h"
#include "tensorflow/core/framework/types.h"
#include "tensorflow/core/lib/core/errors.h"
#include "tensorflow/core/framework/register_types.h"
namespace tensorflow {

19
tensorflow/core/ops/ops.pbtxt
View File

@ -15804,6 +15804,25 @@ op {
}
summary: "Transforms a serialized tensorflow.TensorProto proto into a Tensor."
}
op {
name: "SerializeTensor"
input_arg {
name: "tensor"
description: "A Tensor of type `T`."
type: "T"
}
output_arg {
name: "serialized"
description: "A serialized TensorProto proto of the input tensor."
type_attr: DT_STRING
}
attr {
name: "T"
type: "type"
description: "The type of the input tensor."
}
summary: "Transforms a Tensor into a serialized TensorProto proto."
}
op {
name: "Placeholder"
output_arg {

4
tensorflow/core/platform/default/logging.cc
View File

@ -83,11 +83,11 @@ void LogMessage::GenerateLogMessage() {
const size_t time_buffer_size = 30;
char time_buffer[time_buffer_size];
strftime(time_buffer, time_buffer_size, "%Y-%m-%d %H:%M:%S",
localtime(&now_seconds));
localtime(&now_seconds));
// TODO(jeff,sanjay): Replace this with something that logs through the env.
fprintf(stderr, "%s.%06d: %c %s:%d] %s\n", time_buffer, micros_remainder,
"IWEF"[severity_], fname_, line_, str().c_str());
"IWEF"[severity_], fname_, line_, str().c_str());
}
#endif

2
tensorflow/python/feature_column/feature_column_test.py
View File

@ -3209,7 +3209,7 @@ class IndicatorColumnTest(test.TestCase):
def test_transform_with_weighted_column(self):
# Github issue 12557
ids = fc.categorical_column_with_vocabulary_list(
key='ids', vocabulary_list=('a', 'b', 'c'))
key='ids', vocabulary_list=('a', 'b', 'c'))
weights = fc.weighted_categorical_column(ids, 'weights')
indicator = fc.indicator_column(weights)
features = {

1
tensorflow/tf_exported_symbols.lds
View File

@ -2,4 +2,5 @@
*perftools*gputools*
*tf_*
TF_*
TFE_*
*nsync_*

1
tensorflow/tf_version_script.lds
View File

@ -3,6 +3,7 @@ tensorflow {
*tensorflow*;
*perftools*gputools*;
TF_*;
TFE_*;
*nsync_*;
local:
*;

3
tensorflow/workspace.bzl
View File

@ -99,6 +99,9 @@ def _execute_and_check_ret_code(repo_ctx, cmd_and_args):
# Apply a patch_file to the repository root directory
# Runs 'patch -p1'
def _apply_patch(repo_ctx, patch_file):
if not repo_ctx.which("patch"):
fail("patch command is not found, please install it")
cmd = [
"patch", "-p1", "-d", repo_ctx.path("."), "-i", repo_ctx.path(patch_file)
]