diff --git a/tensorflow/python/autograph/g3doc/reference/control_flow.md b/tensorflow/python/autograph/g3doc/reference/control_flow.md new file mode 100644 index 00000000000..494e556c8c5 --- /dev/null +++ b/tensorflow/python/autograph/g3doc/reference/control_flow.md @@ -0,0 +1,517 @@ +# AutoGraph reference + +[Index](index.md) + +## Control flow + +AutoGraph rewrites all control flow statements with specialized AutoGraph +function calls. These function calls are capable of executing the corresponding +control flow statement using Python semantics for effects outside the Python +interpreter itself (see the [Introduction](intro.md)). + +### Dispatch rules + +Key Point: Only statements that are conditioned on, or iterate over, a +TensorFlow object such as `tf.Tensor`, are converted into TensorFlow ops. + +As described in the [Introduction](intro.md), AutoGraph aims to preserve the +semantics of valid Python code. If a control flow statement runs in graph +execution without raising an error, then AutoGraph will also execute it as +normal Python control flow. Statements which would normally raise an error, for +example because a `tf.Tensor` cannot be used as a `bool` in an `if` statement, +are converted to TensorFlow control flow ops. + +#### Analogy with compile-time constants and code optimization + +From the perspective of a TensorFlow graph, non-Tensor values, for example an +integer or a NumPy array, are _constants_: they do not change value while the +graph executes. + +For example, in the graph below, the condition is always `True` (it is +invariant): + +``` +x = 1 +y = tf.cond(x > 0, lambda: 3 * x, lambda 5 * x) +``` + +That is equivalent to the code below: + +``` +x = 1 +y = 3 * x +``` + +In the example above, we've optimized away the conditional on a constant +condition. The AutoGraph dispatch rules have the same effect: anything that is +not a TensorFlow object is a compile-time constant for TensorFlow, and can be +optimized away. For this reason, you can usually mix Python and TensorFlow +computation and it will transparently have the expected result even +when only some computations are executed in the graph. + + +Caution: The assumption of invariant code made above is not true if the +TensorFlow graph had callbacks into the Python code. If you modify data +from within a `tf.py_function`, then the code outside a `tf.py_function` +will have unpredictable behavior if it depends on the same data. + +For example, the `tf.cond` that runs as part of the `if` statement below will +miss the update made by `f`: + +``` +n = [10] +def f(): + n[0] = 20 + return 0 +tf.py_function(f, (), (tf.int32,)) +if tf.equal(n[0], 10): + tf.print('n is 10') +``` + +``` +n is 10 +``` + +### Compound symbols + +AutoGraph usually handles basic symbols: + +``` +if a < 0: + a = -a +``` + +``` +a = tf.cond(a < 0, lambda: -a, lambda: a) +``` + +But it can also handle complex symbols in many cases. For example, if we treat +`a.b` as a symbol in the code below, then we can use it as if it were a basic +symbol name: + +``` +if a.b < 0 + a.b = -a.b +``` + +``` +a.b = tf.cond(a.b < 0, lambda: -a.b, lambda: a.b) +``` + +This is useful in methods, which can operate on properties of `self`, as well as +working directly on more complex object structures or collections. + +Caution: There are certain [limitations](limitations.md) around using Python +collections and object mutation. When in doubt, place the values you work +with into local variables and operate on those. + +### Effects of the tracing process + +#### All Python code paths are executed during tracing + +When constructing a graph, TensorFlow _traces_ the code. The tracing of control +flow requires visiting _every possible code path_ (usually once). + +Note: In rare cases, the runtime may decide to trace some code paths several +times. For example, the condition of a `while` statement may be executed twice, +first with a temporary graph, to determine whether it evaluates to a +`tf.Tensor`, then if it is a `tf.Tensor`, it's executed a second time in the +proper graph. + +In other words, when tracing executes both branches of an if statement. +Similarly, the body of loops is executed once (even if the loop would otherwise +not iterate at all). + +This explains why inserting `print` statements in an `if` statement produces +this output: + +``` +print('before if') +if tf.constant(True): + print('true branch') +else: + print('false branch') +print('after if') +``` + +``` +before if +true branch +false branch +after if +``` + +Note: Control flow that is not executed as a TensorFlow graph is not traced. Its +body will execute as expected. + +Example of code that runs as regular Python code: + +``` +print('before if') +if True: # Condition not a Tensor, running normally + print('true branch') +else: + print('false branch') +print('after if') +``` + +``` +before if +true branch +after if +``` + +#### Python values modified in TensorFlow control flow become Tensors + +If a symbol is modified in a TensorFlow control flow statement, then it becomes +a `tf.Tensor`, even if it started off as a Python promitive value. + +For example, the conditional below will run as a `tf.cond` (its condition is a +`tf.Tensor`), which in turn will cause `i` to become a `tf.Tensor`. + +``` +i = 0 +if tf.greater(i, 0): + i = 1 +# i is not a Tensor +``` + +### `if` statements + +`if` statements whose condition is a `tf.Tensor` are executed as TensorFlow +conditionals by converting them to `tf.cond`: + +``` +if tf.random.uniform(()) > 0.5: + x = 1 +else: + x = 2 +``` + +`if` statements whose condition is not a `tf.Tensor` are executed as normal +Python: + +``` +if np.random.uniform() > 0.5: + x = 1 +else: + x = 2 +``` + +`if` statements executed as TensorFlow conditionals are subject to restrictions +(see [limitations](limitations.md)). All symbols affected by the statement and +used thereafter must be: + + * of a data type understood by TensorFlow + * defined in both branches + * of consistent dtypes in both branches, for TensorFlow entities + * of consistent structure in both branches, for static collections (such as + lists or tuples) + +### `while` statements + +`while` statements whose condition is a `tf.Tensor` are executed as TensorFlow +loops by converting them to `tf.while_loop`: + +``` +x = 0 +while tf.random.uniform(()) > 0.5: + x = x + 1 +``` + +`while` statements whose condition is not a `tf.Tensor` are executed as normal +Python: + +``` +x = 0 +while np.random.uniform() > 0.5: + x = x + 1 +``` + +`while` statements executed as TensorFlow loops are subject to restrictions +(see [limitations](limitations.md)). All symbols affected by the statement and +used thereafter must be: + + * of a data type understood by TensorFlow + * defined before the loop + * of consistent dtype at the beginning and the end of the loop, + for TensorFlow entities + * either of consistent shape at the beginning and the end of the loop, + for TensorFlow entities, or declared in `shape_invariants` + * of consistent structure at the beginning and the end of the loop, for + static collections (such as lists or tuples) + +Caution: A `while` loop whose condition is a Python scalar will execute as +normal Python. If you intended to run the loop as a TensorFlow loop, the loop +will replicate its body in the graph (it is unrolled). To avoid that, make sure +its condition is converted to a `tf.Tensor`, using for instance `tf.constant`. + +For example, the following loop is unrolled, even though the list contains +`tf.Tensor` values, because the type of `l` is a Python `list`: + +``` +l = [tf.constant(1), tf.constant(2), tf.constant(3)] +for i in l: + tf.print(i) # This is unrolled - three `tf.print`s are built in the graph. +``` + +If you wish for the loop to run as a TensorFlow loop, stack the loop: + +``` +l = [tf.constant(1), tf.constant(2), tf.constant(3)] +for i in tf.stack(l): + tf.print(i) # This runs as a TensorFlow loop. +``` + + +Caution: A loop in which the type of the condition condition changes across +iterations, in a way that would influence the way the loop is executed, is not +allowed in AutoGraph. + +For example, the loop below will generate an error. After the first iteration, +`i` becomes a tf.Tensor, because + +``` +i = 0 +while i < 10: # `i < 10` is a Python bool - run as normal while loop + i = tf.constant(1) # Error -- `i < 10` would now be a `tf.Tensor` +``` + +### `for` statements + +`for` statements that iterate over a `tf.Tensor` are executed as TensorFlow +loops by converting them to a `tf.while_loop` which iterates over the first +dimension (equivalent to NumPy): + +``` +for i in tf.constant(((1, 2), (3, 4))): + tf.print('iteration:', i) +``` + +``` +iteration: [1, 2] +iteration: [3, 4] +``` + +Note: If possible, AutoGraph will also set the `maximum_iteration` parameter +of the `tf.while_loop`. + +`for` statements that iterate over a the output of a `tf.range` are executed as +TensorFlow loops by converting them to a `tf.while_loop` which uses the +arguments passed to the `tf.range`: + +``` +for i in tf.range(3): + tf.print('iteration:', i) +``` + +`for` statements that iterate over a `tf.data.Dataset` and which do not contain +`break` or `return` statements are executed as TensorFlow loops by converting +them to `tf.data.Dataset.reduce` ops: + +``` +for i in tf.data.Dataset.range(3): + tf.print('iteration:', i) +``` + +`for` statements that iterate over a _distributed_ `tf.data.Dataset` and which +do not contain `break` or `return` statements are executed as TensorFlow loops +by converting them to the datasets' `reduce` ops: + +``` +for i in tf.distribute.OneDeviceStrategy('cpu').experimental_distribute_dataset( + tf.data.Dataset.range(3)): + tf.print('iteration:', i) +``` + +`for` statements that iterate over a `tf.data.Dataset` and which contain +`break` or `return` statements are executed as TensorFlow loops by converting +them to a combination of `tf.data.Dataset.scan`, `tf.data.Dataset.take_while` +and `tf.data.Dataset.reduce` ops: + +``` +for i in tf.data.Dataset.range(3): + tf.print('iteration:', i) + break +``` + +``` +iteration: 1 +``` + +`for` statements that iterate over a `tf.data.Dataset` _iterator_ are executed +as TensorFlow loops by converting them to a combination of `tf.while_loop`, +and `tf.cond` ops: + +``` +for i in iter(tf.data.Dataset.range(3)): + tf.print('iteration:', i) +``` + +`for` statements that iterate over a type different from any of the above are +executed as normal Python: + +``` +for i in [1, 2, 3]: + print('iteration:', i) +``` + +Caution: A `for` loop over a `list` or `tuple` of `tf.Tensor` is considered to +iterate over a Python `list` (or respectively `tuple`), therefore will be +executed as normal Python. If you intended to run it as a TensorFlow loop, +use `tf.stack` or `tf.concat`. + +Caution: A `for` loop over a Python `range` will be executed as normal Python. +If you intended to run it as a TensorFlow loop, `tf.range`. + +Note: AutoGraph may output a warning when it believes that you are unrolling +a loop inefficiently. However, the warning thresholds are very conservative. + +### `break` statements + +Code blocks in which `break` statements are used are rewritten with equivalent +code that uses extra control booleans and conditionals. The control booleans are +used directly in `while` loops. In the case of `for` loops, the AutoGraph +corresponding operator accepts an `extra_test` argument which is similar to +the conditional of a while loop, and which contains the control boolean. + +For example, the `while` loop below is rewritten as (showing the output of the +`break` transformation only): + +``` +while i < 10: + if i > 3: + break + i += 1 +``` + +``` +break_ = False +while i < 10 and not break_: + if i > 3: + break_ = True + continue # The continue statement is also rewritten in a subsequent pass + i += 1 +``` + +Another example shows how the control boolean is used in the overload of a `for` +loop (showing portions of the final output): + +``` +for i in range(10): + if i > 3: + break +``` + +``` +break_ = False +... +def extra_test(break_): + return ag__.not_(break_) +# break_ becomes a loop variable. +break_, = ag__.for_stmt(range(10), extra_test, ..., (break_,)) +``` + +### `continue` statements + +Code blocks in which `continue` statements are used are rewritten with +equivalent code that uses extra control booleans and conditionals, similar to +how `break` is handled. + +For example, the `for` loop below is rewritten as (showing the output of the +`continue` transformation only): + +``` +for i in range(10): + if i > 3: + continue +``` + +``` +for i in range(10): + continue_ = False + if i > 3: + continue_ = True + if not continue_: + i += 1 +``` + +Notice that unlike `break`, `continue` statements are local to the loop and do +not influence the number of iterations. + +### `return` statements + +`return` statements are also rewritten using control symbols, in a manner +similar to how `break` is converted. In the case of `return` statements, an +additional symbol keeps track of the return value. + +Depending on the structure of the code, the return value might be undefined +in parts of the code (for example on code paths in which no return statement +has executed). AutoGraph keeps track of this by using a special value. +This special value is converted to `None` (the default return value) upon +exiting the function. + +Caution: TensorFlow control flow doe not support undefined values, and an +undefined return value is no exception. Therefore, AutoGraph will raise an +error for TensorFlow control flow in which the return value is not known for +all code paths. + +For example, the following code raises an error because the return value would +be undefined when the random number would be less than 0.5: + +``` +if tf.random.uniform(()) > 0.5: + return 1 +``` + +``` +ValueError: A value must also be returned from the else branch. +``` + +An example of rewriting a `while` (showing the output of the `return` +transformation only): + +``` +def f(): + while i < 10: + if i > 3: + return 1 + i += 1 +``` + +``` +def f(): + do_return = False + retval_ = ag__.UndefinedReturnValue() + while i < 10 and not do_return: + if i > 3: + do_return = True + retval_ = 1 + if not do_return: + i += 1 + return ag__.retval(retval_) # Transforms any UndefinedReturnValue to None +``` + +Note: AutoGraph performs an additional code normalization in which an `if` +statement with no `else` branch contains a `return` statement it is rewritten as +an `if-else` statement in which the code that follows the statement is moved +under the `else` branch. + +Example (showing the normalization only): + +``` +def f(): + if i > 3: + return 1 + i += 1 +``` + +``` +def f(): + if i > 3: + return 1 + else: + i += 1 +``` + + diff --git a/tensorflow/python/autograph/g3doc/reference/index.md b/tensorflow/python/autograph/g3doc/reference/index.md index 28a9b37439f..700bf34c6f1 100644 --- a/tensorflow/python/autograph/g3doc/reference/index.md +++ b/tensorflow/python/autograph/g3doc/reference/index.md @@ -8,16 +8,17 @@ graph. * [Introduction](intro.md) * [Interacting with the generated code](generated_code.md) * [Debugging AutoGraph code](debugging.md) -* Control Flow (coming soon) +* [Control flow](control_flow.md) +* Functions calls (coming soon) +* Exception handling (coming soon) +* Conversion mechanics (coming soon) * Collections (coming soon) -* Exceptions (coming soon) -* Builtin Functions (coming soon) -* Datasets (coming soon) * [Limitations](limitations.md) +* Common errors (coming soon) For more information on AutoGraph, see the following articles: -* [AutoGraph tutorial](https://www.tensorflow.org/alpha/guide/autograph) +* [AutoGraph tutorial](https://www.tensorflow.org/alpha/beta/autograph) * [Eager tutorial](https://www.tensorflow.org/alpha/guide/eager) * [TensorFlow 2.0 Alpha](https://www.tensorflow.org/alpha) * [AutoGraph blog post](https://medium.com/tensorflow/autograph-converts-python-into-tensorflow-graphs-b2a871f87ec7) diff --git a/tensorflow/python/autograph/g3doc/reference/limitations.md b/tensorflow/python/autograph/g3doc/reference/limitations.md index dd15d50afcb..a483cd46a55 100644 --- a/tensorflow/python/autograph/g3doc/reference/limitations.md +++ b/tensorflow/python/autograph/g3doc/reference/limitations.md @@ -10,6 +10,12 @@ However, when applied to TensorFlow control flow (for example, an if statement with a `tf.Tensor` condition), there are certain limitations. This section describes these limitations and practices that will allow you to avoid them. +Key Term: Python variables refer to Python symbols (or symbols for short) and +should not be confused with TensorFlow variables. + +Key Term: A TensorFlow loop variable (or loop variable for short) refers to a +value (typically a `tf.Tensor`) modified by a loop. See `tf.while_loop`. + ### Indirect modifications and hidden side effects in TensorFlow control flow @@ -22,12 +28,10 @@ flow statements into equivalent TensorFlow ops. This process requires "wiring" variables in the Python code whose values are affected these statements control flow into the respective ops. -Note: Python variables should not be confused with TensorFlow variables. - The examples below use a `while` loop, but the same notions extend to all control flow: `if` and `for` statements. -In the example below, `x` needs to become a _loop variable_ of the +In the example below, `x` needs to become a loop variable of the corresponding `tf.while_loop': ``` @@ -255,6 +259,156 @@ for i in tf.range(10): d = {key: value + i for key, value in d.items()} # Okay ``` +### Shape and dtype consistency in TensorFlow control flow + +Unlike Python, TensorFlow has limited support for dynamic typing. This means +that tensors must maintain consistent shapes and dtypes across control flow +paths. + +Note: In general, these restrictions do not apply in control flow in Eager +execution, because Eager execution uses Python control flow, rather than +TensorFlow control flow ops. + +#### Consistency of dtype + +The dtypes across all code paths must be consistent in conditionals and loops. + +For example, if a `tf.cond` (and correspondingly, an AutoGraph `if`) sets a +tensor value conditionally, then that tensor must have the same shape and dtype +in both branches of the conditional. + +Example of illegal dtype change in a conditional: + +``` +x = tf.cond( + tf.random.uniform(()) > 0.5, + lambda: tf.constant(1, dtype=tf.int32), + lambda: tf.constant(1, dtype=tf.float32)) # Error -- inconsistent dtypes: int32, float32 +``` + +The same restriction in AutoGraph code: + +``` +if tf.random.uniform(()) > 0.5: + x = tf.constant(1, dtype=tf.int32) +else: + x = tf.constant(1, dtype=tf.float32) # Error -- inconsistent dtypes: int32, float32 +``` + +Example of illegal dtype change in a loop: + +``` +# This won't work - "x" changes dtype inside the loop. +x = tf.while_loop( + lambda _: tf.random.uniform(()) > 0.5, + lambda x: tf.constant(1, dtype=tf.float32), + loop_vars=(tf.constant(1, dtype=tf.int32),)) # Error -- inconsistent dtypes: int32, float32 +``` + +The same restriction in AutoGraph code: + +``` +x = tf.constant(0, dtype=tf.int32) +while tf.random.uniform(()) > 0.5: + x = tf.constant(0, dtype=tf.float32) # Error -- inconsistent dtypes: int32, float32 +``` + +#### Consistency of shape + +The shapes across all code paths must be consistent in loops only. When tensors +do need to change shape across iterations, use `shape_invariants`. + +Note: Shapes are allowed to be inconsistent in conditionals. The result will be +a partially dynamic shape. + +In a `tf.while_loop` (and correspondingly, an AutoGraph `while` or `for` loop) +all loop variables must maintain consistent shape and dtype across iterations. +That is, every loop variable must have the same shape at the end of the loop +body as the shape that it had at the beginning of the loop body. + +Example of illegal shape change in a loop: + +``` +def loop_body(x): # x.shape is () + return tf.constant((1, 2, 3)) # Error -- inconsistent shapes: (), (3,) + +x = tf.while_loop( + lambda _: tf.random.uniform(()) > 0.5, + loop_body, + loop_vars=(tf.constant(1,)) +``` + +The same restriction in AutoGraph code: + +``` +x = tf.constant(0, dtype=tf.int32) +while tf.random.uniform(()) > 0.5: + x = tf.constant(0, dtype=tf.float32) # Error -- inconsistent shapes: (), (3,) +``` + +### Undefined and None values in TensorFlow + +TensorFlow does not support undefined and `None` values. All tensors must have +a value. + +Example: + +``` +x = tf.cond( + tf.random.uniform(()) > 0.5, + lambda: tf.constant(1), + lambda: None) # Error -- a Tensor cannot be None +``` + +The same restriction carries over in AutoGraph, but only if the symbol is used +after the conditional (otherwise AutoGraph avoids making it a return value +of the `tf.cond`): + +``` +if tf.random.uniform(()) > 0.5: + x = tf.constant(1) +else: + x = None +tf.print(x) # Error -- x may be None here +``` + +A related but less obvious restriction in AutoGraph forbids symbols to be +defined in only one branch of TensorFlow control flow, if the symbol is +used afterwards: + +``` +del x +if tf.random.uniform(()) > 0.5: + x = tf.constant(1) +else: + pass +tf.print(x) # Error -- x may be undefined here +``` + +Similarly, variables defined in a loop may not be used outside the loop, again +if the symbol is used afterwards: + +``` +del x +if tf.random.uniform(()) > 0.5: + x = tf.constant(1) +tf.print(x) # Error -- x may be undefined here +``` + +Avoid these limitations by defining a default value before the control flow +statement: + +``` +x = tf.constant() +if tf.random.uniform(()) > 0.5: + x = tf.constant(1) +tf.print(x) # Okay -- x is either 0 or 1 +``` + +Note: `None` values and undefined symbols are allowed in Eager control flow, +because Eager execution uses Python control flow, rather than TensorFlow +control flow ops. + ### Access to source code Key point: AutoGraph can only handle functions whose source code can be