frameworks/tensorflow2/TF2_Eager_tfFunc.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"accelerator":"GPU","colab":{"name":"TF2_Eager_tfFunc.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"}},"cells":[{"cell_type":"code","metadata":{"colab_type":"code","id":"IT-Ja79c8ioA","colab":{}},"source":["!pip install tensorflow-gpu==2.0.0"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"YozNmFKbsM7a"},"source":["## New Ways Building Network"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"OXsUvs_FLh25","colab":{}},"source":["import tensorflow as tf"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576570770797,"user_tz":-480,"elapsed":1696,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"H4RlLSdOLv9B","outputId":"e3473bfb-dbae-4d24-f393-13a265e3c434","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["print(\"Tensorflow Version: {}\".format(tf.__version__))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Tensorflow Version: 2.0.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"_s-PgoPQSQQi"},"source":["## Eager Execution"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"hn3XA1YPSgK5"},"source":["Eager execution is one of the biggest changes between Tensorflow 1.x and 2.x. "]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"Lbq-3t_G2e3V"},"source":["### Basic Concept"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"-IcXsRyITp2p"},"source":["In Tensorflow 1, all operations on the tensor are under an active session. A simple example is below.\n","\n","```python\n","# Tensorflow 1.x\n","with tf.Session() as sess:\n","  add_res = tf.constant(2.0, dtype=float) + tf.constant(1.5, dtype=float)\n","  print(sess.run(add_res))   # 3.5\n","```"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"VLEgmLjaTPRM"},"source":["In Tensorflow 2 with **eager** mode, you no need start a session to run the operation, rather you can do calculate more like in native python."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576563149278,"user_tz":-480,"elapsed":2653,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"UyVbcLN1SPud","outputId":"17133a96-4944-451e-b69f-d1e321d5426d","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# tensor\n","tf.constant(2.0, dtype=float) + tf.constant(1.5, dtype=float)"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: id=2, shape=(), dtype=float32, numpy=3.5>"]},"metadata":{"tags":[]},"execution_count":3}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"079SNvjTUuDs"},"source":["You can even get the calculation result by simply calling a method `numpy()`."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576563343024,"user_tz":-480,"elapsed":761,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"vD_lcWNeUjdZ","outputId":"b541bf65-f3ee-4bfc-9402-93cfd96b989d","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# numpy result (default float type is tf.float32)\n","(tf.constant(2.0, dtype=float) + tf.constant(1.5, dtype=float)).numpy()"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["3.5"]},"metadata":{"tags":[]},"execution_count":7}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"UipyzRuqabJO"},"source":["In Tensorflow 2.0, eager execution is on by default. Unlike Tensorflow 1.x, operations under the eager model are executing immediately without building a graph."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"e92Re-dNyBbt"},"source":["### No Initializer Required\n","\n","In Tensorflow 1.x, a simple variable must be initialized after it was declared.\n","\n","In Tensorflow 2.x, no matter in `eager` execution or in `tf.function()` no initialization operation is required."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"vWB5kBLW1T-v"},"source":["An example in the **eager** mode. "]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576567416943,"user_tz":-480,"elapsed":857,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"ivYpd_Ee1YOw","outputId":"1fa3d1c0-51af-4873-9c82-de6c38dd43b2","colab":{"base_uri":"https://localhost:8080/","height":187}},"source":["v_add_seq = tf.Variable(1.0)\n","\n","for i in range(10):\n","  # this step is not adding an operation to the graph\n","  # you can simply think this is equivalent to\n","  # `v_add_seq += i`\n","  v_add_seq.assign_add(i)\n","  print(v_add_seq)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=1.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=2.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=4.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=7.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=11.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=16.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=22.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=29.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=37.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=46.0>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Ncf9wulWv2nQ","colab_type":"text"},"source":["The result of the below script is equal to the above one but different from the types of them. One is a variable and the other is a constant."]},{"cell_type":"code","metadata":{"id":"yS_WbGJzvY7K","colab_type":"code","outputId":"f745b297-be80-4753-baa7-fb251b79aaeb","executionInfo":{"status":"ok","timestamp":1576567422299,"user_tz":-480,"elapsed":911,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"colab":{"base_uri":"https://localhost:8080/","height":187}},"source":["v_add_seq = tf.Variable(1.0)\n","\n","for i in range(10):\n","  v_add_seq = v_add_seq + i\n","  print(v_add_seq)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tf.Tensor(1.0, shape=(), dtype=float32)\n","tf.Tensor(2.0, shape=(), dtype=float32)\n","tf.Tensor(4.0, shape=(), dtype=float32)\n","tf.Tensor(7.0, shape=(), dtype=float32)\n","tf.Tensor(11.0, shape=(), dtype=float32)\n","tf.Tensor(16.0, shape=(), dtype=float32)\n","tf.Tensor(22.0, shape=(), dtype=float32)\n","tf.Tensor(29.0, shape=(), dtype=float32)\n","tf.Tensor(37.0, shape=(), dtype=float32)\n","tf.Tensor(46.0, shape=(), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"qaqzjy6RsV0d","colab_type":"text"},"source":["In Tensorflow 1.x, the above script would create a sequential of operations for adding values.\n","\n","```python\n","# Tensorflow 1.x\n","tf.reset_default_graph()\n","with tf.Graph().as_default():\n","  v_add_seq = tf.Variable(1.0)\n","  with tf.Session() as sess:\n","    sess.run(tf.global_variables_initializer())\n","\n","    for i in range(5):\n","      # a `wrong` way to add an integer\n","      # the following script would add operations on the graph\n","      v_add_seq = tf.assign_add(v_add_seq, i)\n","      \"\"\"\n","      +0 = 1.0\n","      +1 = 2.0\n","      +2 = 5.0\n","      +3 = 11.0\n","      +4 = 21.0\n","      \"\"\"\n","      print(\"+{} = {}\".format(i, sess.run(v_add_seq)))\n","\n","      # a correct way to add the integer sequentially\n","      new_value = tf.assign_add(v_add_seq, i)\n","      print(\"+{} = {}\".format(i, sess.run(v_add_seq)))\n","```"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"v37UdKUt1MNZ"},"source":["Example In the `tf.function()`."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"elapsed":1875,"status":"ok","timestamp":1558432848995,"user":{"displayName":"王DevOps","photoUrl":"https://lh6.googleusercontent.com/-DzgWpvOQ_c0/AAAAAAAAAAI/AAAAAAAAA8I/fX9gQd_TAjU/s64/photo.jpg","userId":"04300517850278510646"},"user_tz":-480},"id":"yPai1M-N0cde","outputId":"6358ec62-3e2f-4b50-f08c-9ab5a7738232","colab":{"base_uri":"https://localhost:8080/","height":199}},"source":["v_add_seq_2 = tf.Variable(1.0)\n","\n","@tf.function()\n","def add_seq(x):\n","  return v_add_seq_2.assign_add(x)\n","\n","for i in range(10):\n","  print(add_seq(i))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tf.Tensor(1.0, shape=(), dtype=float32)\n","tf.Tensor(2.0, shape=(), dtype=float32)\n","tf.Tensor(4.0, shape=(), dtype=float32)\n","tf.Tensor(7.0, shape=(), dtype=float32)\n","tf.Tensor(11.0, shape=(), dtype=float32)\n","tf.Tensor(16.0, shape=(), dtype=float32)\n","tf.Tensor(22.0, shape=(), dtype=float32)\n","tf.Tensor(29.0, shape=(), dtype=float32)\n","tf.Tensor(37.0, shape=(), dtype=float32)\n","tf.Tensor(46.0, shape=(), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"qBoP084WL_6O"},"source":["## TF Function"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"ZLq39rIuQcMs"},"source":["### Basic Concept"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"KPKnoAUAQbMn"},"source":["`tf.function()` is a new decorator declaring operations in Tensorflow 2.0.\n","\n","The basic declearation is like below:\n","```python\n","# the necessary decorator\n","@tf.function\n","def func(*params):\n","  '''define the operations'''\n","  results = ...\n","  return results\n","```\n","\n","With `tf.function()`, you **don't** need the following:\n","* running via `tf.Session()`\n","* using `tf.control_dependencies()`\n","* using `tf.global_variables_initializers()`\n","* using `tf.cond()` and `tf.while_loop()`.\n","\n","With `tf.function()`, you are supported to **do** the following:\n","* building concrete functions"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"koreelotL_cC","colab":{}},"source":["@tf.function\n","def add(a, b):\n","  '''An add operation.'''\n","  return a + b"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"cLhYvAO7PIqD"},"source":["You can use `tf.function()` to process tensor. The `tf.function()` can return the result as the tensor."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576567795423,"user_tz":-480,"elapsed":751,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"GrEc1m4_O2Bi","outputId":"ece847af-5305-4629-b2d9-4f0552f2e295","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["add(tf.constant(2.0, dtype=float), tf.constant(3.5, dtype=float))"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: id=593, shape=(), dtype=float32, numpy=5.5>"]},"metadata":{"tags":[]},"execution_count":27}]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576567893819,"user_tz":-480,"elapsed":917,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"kIYlhbf4PfTK","outputId":"a993df08-ac95-4aa7-9a70-6389dce753a6","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["add(tf.constant(3.0, dtype=float), tf.constant(7.5, dtype=float)).numpy()"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["10.5"]},"metadata":{"tags":[]},"execution_count":28}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"QdPa_YoNPhkD"},"source":["* You can also use `tf.function()` as the normal python function."]},{"cell_type":"code","metadata":{"id":"08YoAiOUxNSv","colab_type":"code","outputId":"29662040-b7b3-46a3-fe4a-a80771599ec6","executionInfo":{"status":"ok","timestamp":1576567902319,"user_tz":-480,"elapsed":1222,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["add(1.1, 1.3)"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: id=601, shape=(), dtype=float32, numpy=2.4>"]},"metadata":{"tags":[]},"execution_count":29}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"jkz0f7CNPwBz"},"source":["Furthermore, you might not need to care about the data type of tensors while doing the same operation. \n","\n","Compare to Tensorflow 1.x, you don't need to declare the same operation but for different data types, for example, for integer or for string variables."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"elapsed":9116,"status":"ok","timestamp":1558432712261,"user":{"displayName":"王DevOps","photoUrl":"https://lh6.googleusercontent.com/-DzgWpvOQ_c0/AAAAAAAAAAI/AAAAAAAAA8I/fX9gQd_TAjU/s64/photo.jpg","userId":"04300517850278510646"},"user_tz":-480},"id":"Ifdtt7RnQQ4g","outputId":"e16b3cab-5923-4922-823c-0c3e03f96854","colab":{"base_uri":"https://localhost:8080/","height":35}},"source":["add(\"a\", \"a\")"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: id=128, shape=(), dtype=string, numpy=b'aa'>"]},"metadata":{"tags":[]},"execution_count":11}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"Hi5X7991QiDi"},"source":["### More Faster"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"4nIP63hdQngj"},"source":["One of the biggest changes between Tensorflow 1.x and 2.x is the eager mode. It is no required to execute the operations on a graph under a session via `tf.Session()`.\n","\n","But with `tf.function()`, the calculation would be faster then eager execution. (Due to the optimization in `tf.function()`.)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"1m7RqhZZVb-Z"},"source":["Here we demo the time consumption between **eager model** and **tf.function()**. \n","\n","LSTM cell is one type of gated RNN cell improving memorizing mechanism along with time-series data. But the state in each time moment depended on the previous time moment. This dependent calculation causes itself to consume more time and hard to accelerate during the training."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"HbC7eQbtQlAs","colab":{}},"source":["lstm_cell = tf.keras.layers.LSTMCell(10)\n","\n","@tf.function\n","def fn(input, state):\n","  return lstm_cell(input, state)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576568756141,"user_tz":-480,"elapsed":1709,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"rZa44Cd9XDTh","outputId":"57254119-63e2-4160-c852-1271f1306ca0","colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["input = tf.zeros([10, 10])\n","state = [tf.zeros([10, 10])] * 2\n","\n","import timeit\n","print(\"Eager mode: {}\".format(timeit.timeit(lambda: lstm_cell(input, state), number=10)))\n","print(\"tf.function mode: {}\".format(timeit.timeit(lambda: fn(input, state), number=10)))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Eager mode: 0.641535319000468\n","tf.function mode: 0.06657306199940649\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"FWeqUAQrjE77"},"source":["### Automatic Control Dependencies\n","\n","In Tensorflow 1.x, we use `tf.control_dependencies()` to control the partial calculation order in the graph. But it is easy to ignore the updated relation among tensors.\n","\n","A simple example is below.\n","\n","```python\n","# Tensorflow 1.x\n","\n","with tf.Graph().as_default():\n","  v1 = tf.Variable(tf.constant(1.0))\n","  v2 = tf.Variable(2.0)\n","  x = tf.placeholder(tf.float32)\n","  y = tf.placeholder(tf.float32)\n","  \n","  # =============================================\n","  # wrong way (easier way to build the operation)\n","  # =============================================\n","  #v1_assign = v1.assign(y * v2)\n","  #v2_assign = v2.assign_add(x * v1)\n","  #v_sum = v1 + v2  # 3.0\n","  \n","  # ===============================================================\n","  # correct way (have to notice the dependencies among several operations)\n","  # ===============================================================\n","  v1_assign = v1.assign(y * v2)\n","  with tf.control_dependencies([v1_assign]):\n","    # control is dependent on operator v1_assign\n","    v2_assign = v2.assign_add(x * v1)\n","    with tf.control_dependencies([v2_assign]):\n","      # control is dependent on both operators v1_assign, v2_assign\n","      v_sum = v1 + v2\n","  \n","  with tf.Session() as sess:\n","    sess.run(tf.global_variables_initializer())\n","    print(sess.run([v1, v2, v_sum], feed_dict={x: 3.0, y: 4.0}))  # [8.0, 26.0, 34.0]\n","```"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"QaQhO-Tlvts9"},"source":["In Tensorflow 1.x, `tf.control_dependency()` controls the calculation order in the flow, especially for the operations dependent on the previous and next results. Lots of loop-like operations would encounter such problems. (It is really not convenient to building partial operation.)\n","\n","In Tensorflow 2.x, the control dependencies are handed over to Tensorflow, you can simply build the operation like native python."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"elapsed":10859,"status":"ok","timestamp":1558432714009,"user":{"displayName":"王DevOps","photoUrl":"https://lh6.googleusercontent.com/-DzgWpvOQ_c0/AAAAAAAAAAI/AAAAAAAAA8I/fX9gQd_TAjU/s64/photo.jpg","userId":"04300517850278510646"},"user_tz":-480},"id":"QeGfI1GbjNm2","outputId":"53a0431e-13d4-43e6-de17-3727d7193c76","colab":{"base_uri":"https://localhost:8080/","height":72}},"source":["v1 = tf.Variable(1.0)\n","v2 = tf.Variable(2.0)\n","\n","@tf.function\n","def fn_assign(x, y):\n","  v1.assign(y * v2)       # 8\n","  v2.assign_add(x * v1)   # 26 = 2 + 24\n","  return v1 + v2          # 34 = 8 + 26\n","\n","print(\"{}\\n{}\\n{}\".format(v1, v2, fn_assign(3.0, 4.0)))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=8.0>\n","<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=26.0>\n","34.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"A3JduocQY_tw"},"source":["### Polymorphic / Concrete Function\n","\n","Tensorflow 2.0 with eager mode makes development and debugging more interactive, but it is not allowed for deploying on devices. More operations are required while exporting a saved model or deploying on devices. It requires a **concrete function** defining the graph that can be exported into a saved model. A concrete function can be exported from a polymorphic function. A polymorphic function means python callable that encapsulates several concrete function graphs behind one API. (You can simply think it is required to define tensor spec as passed parameters to the function.)\n","\n","There are three basic steps to define or export a concrete function.\n","* Define `input_signature` parameter in `tf.function`.\n","* Pass in tf.TenorSpec into `get_concrete_function`, e.g. tf.TensorSpec(shape=[1], dtype=tf.float32).\n","* Pass a sample input tensor into `get_concrete_function`, e.g. tf.constant(2.0, shape=[1])."]},{"cell_type":"code","metadata":{"id":"LQAhwfb83sVr","colab_type":"code","colab":{}},"source":["@tf.function\n","def simple_add(v1, v2):\n","  \"\"\"Add two values, v1 and v2, with their relative data type.\"\"\"\n","  return v1 + v2"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576569721210,"user_tz":-480,"elapsed":741,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"xVOZHIJ0Zdae","outputId":"a001fb79-41b5-4878-ea38-34b9f6eaa571","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["c = simple_add.get_concrete_function(tf.TensorSpec(shape=None, dtype=float),\n","                                     tf.TensorSpec(shape=None, dtype=float))\n","\n","print(c(tf.constant(2.0), tf.constant(3.5)))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tf.Tensor(5.5, shape=(), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"wTbkc_Vui4dX"},"source":["You can use attributes `graph` accessing the graph and its relative operations."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576570068908,"user_tz":-480,"elapsed":779,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"TexQn37eicH3","outputId":"6afb26e3-2233-45f3-f2ba-5ebd5d2d2395","colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["print(c.graph)\n","print(c.graph.inputs)\n","print(c.graph.outputs)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["FuncGraph(name=simple_add, id=140623849765576)\n","[<tf.Tensor 'v1:0' shape=<unknown> dtype=float32>, <tf.Tensor 'v2:0' shape=<unknown> dtype=float32>]\n","[<tf.Tensor 'Identity:0' shape=<unknown> dtype=float32>]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"kBsFs3Y22myI"},"source":["### Tensors like Objects"]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"elapsed":1083,"status":"ok","timestamp":1558433527851,"user":{"displayName":"王DevOps","photoUrl":"https://lh6.googleusercontent.com/-DzgWpvOQ_c0/AAAAAAAAAAI/AAAAAAAAA8I/fX9gQd_TAjU/s64/photo.jpg","userId":"04300517850278510646"},"user_tz":-480},"id":"0igbTsag3Xc0","outputId":"4d83d9a9-55e6-4334-9aa5-f16bf679c4ec","colab":{"base_uri":"https://localhost:8080/","height":72}},"source":["state = []\n","\n","@tf.function\n","def append_list(x):\n","  if len(state) < 1:\n","    state.append(tf.Variable(2.0))\n","  return state[0] * x\n","\n","print(append_list(tf.constant(1.0)))\n","print(append_list(tf.constant(2.0)))\n","print(state)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tf.Tensor(2.0, shape=(), dtype=float32)\n","tf.Tensor(4.0, shape=(), dtype=float32)\n","[<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=2.0>]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"WdEyJAL14QXQ"},"source":["### Easier Control Flow\n","\n","No more need `tf.cond()` and `tf.while_loop()`. \n","\n","In Tensorflow 2.x, you can operate the tensor in the same way as usual in python while for branch control."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576570781818,"user_tz":-480,"elapsed":1912,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"MyoAkCBY4orm","outputId":"31fc2e38-aa86-4965-9754-54cfc6497a2a","colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["@tf.function\n","def while_loop(x):\n","  while tf.reduce_sum(x) > 1:\n","    x = tf.tanh(x)\n","  return x\n","\n","while_loop(tf.random.uniform([10]))"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: id=48, shape=(10,), dtype=float32, numpy=\n","array([0.10283414, 0.10629437, 0.10663301, 0.1084716 , 0.10902831,\n","       0.10768783, 0.09392301, 0.10888813, 0.10447586, 0.05031879],\n","      dtype=float32)>"]},"metadata":{"tags":[]},"execution_count":3}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"_07Pihvo5wFb"},"source":["It is easy to use `for loop` to operate the tensor. "]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576570974345,"user_tz":-480,"elapsed":669,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"-Mcq44qz5vys","outputId":"dc35088c-009b-401a-a1a2-477782eaa6e8","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["@tf.function\n","def for_loop(x):\n","  # notice index (i) here is a Tensor, its type is placeholder\n","  # so that you can't replace `tf.range` with `range`\n","  for i in tf.range(10):\n","    x += i\n","  return x\n","\n","print(for_loop(tf.Variable(0)))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tf.Tensor(45, shape=(), dtype=int32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"KstRuMUW6zQV"},"source":["Easier way operates a TensorArray data structure."]},{"cell_type":"code","metadata":{"colab_type":"code","executionInfo":{"status":"ok","timestamp":1576571156574,"user_tz":-480,"elapsed":791,"user":{"displayName":"王DevOps","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAzLB0C3whTHAdHpq24UrEWqGtbhJElQxTU5_b_4g=s64","userId":"04300517850278510646"}},"id":"xKUi3Dod6y1W","outputId":"7c07c505-e5e3-4c78-e101-0f9ca23cc381","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["@tf.function\n","def write_array(x):\n","  ta = tf.TensorArray(tf.float32, size=10)\n","  for i in tf.range(10):\n","    x += tf.cast(i, float)\n","    ta = ta.write(i, x)\n","  return ta.stack()\n","\n","write_array(1.0).numpy()"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([ 1.,  2.,  4.,  7., 11., 16., 22., 29., 37., 46.], dtype=float32)"]},"metadata":{"tags":[]},"execution_count":12}]},{"cell_type":"code","metadata":{"id":"2yJg35tn9khR","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}