forked from fchollet/keras-blog
-
Notifications
You must be signed in to change notification settings - Fork 0
/
keras-as-a-simplified-interface-to-tensorflow-tutorial.html
413 lines (318 loc) · 49.2 KB
/
keras-as-a-simplified-interface-to-tensorflow-tutorial.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
<!DOCTYPE html>
<html lang="en">
<head>
<title>Keras as a simplified interface to TensorFlow: tutorial</title>
<meta charset="utf-8" />
<link rel="stylesheet" href="http://blog.keras.io/theme/css/main.css" type="text/css" />
<link rel="stylesheet" href="http://blog.keras.io/theme/css/pygment.css" type="text/css" />
<link href="https://fonts.googleapis.com/css?family=Lato:400,700|Source+Sans+Pro:400,700|Inconsolata:400,700" rel="stylesheet" type="text/css">
<link href="http://blog.keras.io/" type="application/atom+xml" rel="alternate" title="The Keras Blog ATOM Feed" />
<!--[if IE]>
<script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script><![endif]-->
<!--[if lte IE 7]>
<link rel="stylesheet" type="text/css" media="all" href="http://blog.keras.io/css/ie.css"/>
<script src="http://blog.keras.io/js/IE8.js" type="text/javascript"></script><![endif]-->
<!--[if lt IE 7]>
<link rel="stylesheet" type="text/css" media="all" href="http://blog.keras.io/css/ie6.css"/><![endif]-->
</head>
<body id="index" class="home">
<header id="banner" class="body">
<h1>
<a href="http://blog.keras.io/index.html">The Keras Blog </a>
</h1>
<p id="side">
<a href="https://github.com/fchollet/keras">Keras</a> is a Deep Learning library for Python, that is simple, modular, and extensible.
</p>
<nav><ul>
<li><a href="http://blog.keras.io/">Archives</a></li>
<li >
<a href="https://github.com/fchollet/keras">Github</a>
</li>
<li >
<a href="http://keras.io/">Documentation</a>
</li>
<li >
<a href="https://groups.google.com/forum/#!forum/keras-users">Google Group</a>
</li>
</ul></nav>
</header><!-- /#banner -->
<section id="content" class="body">
<article>
<header> <h1 class="entry-title"><a href="keras-as-a-simplified-interface-to-tensorflow-tutorial.html"
rel="bookmark" title="Permalink to Keras as a simplified interface to TensorFlow: tutorial">Keras as a simplified interface to TensorFlow: tutorial</a></h1> </header>
<div class="entry-content">
<footer class="post-info">
<abbr class="published" title="2016-04-24T00:00:00+02:00">
Sun 24 April 2016
</abbr>
<address class="vcard author">
By <a class="url fn" href="https://twitter.com/fchollet">Francois Chollet</a>
</address>
<p>In <a href="http://blog.keras.io/category/tutorials.html">Tutorials</a>. </p>
</p></footer><!-- /.post-info --><!-- /.post-info -->
<h1>A complete guide to using Keras as part of a TensorFlow workflow</h1>
<p>If TensorFlow is your primary framework, and you are looking for a simple & high-level model definition interface to make your life easier, this tutorial is for you.</p>
<p>Keras layers and models are fully compatible with pure-TensorFlow tensors, and as a result, Keras makes a great model definition add-on for TensorFlow, and can even be used alongside other TensorFlow libraries. Let's see how.</p>
<p><strong>Note that this tutorial assumes that you have configured Keras to use the TensorFlow backend (instead of Theano)</strong>. <a href="http://keras.io/backend/#switching-from-one-backend-to-another">Here are instructions on how to do this</a>.</p>
<p>We will cover the following points:</p>
<p><a href="/#calling-keras-layers-on-tensorflow-tensors">I: Calling Keras layers on TensorFlow tensors</a></p>
<p><a href="/#using-keras-models-with-tensorflow">II: Using Keras models with TensorFlow</a></p>
<p><a href="/#multi-gpu-and-distributed-training">III: Multi-GPU and distributed training</a></p>
<p><a href="/#exporting-a-model-with-tensorflow-serving">IV: Exporting a model with TensorFlow-serving</a></p>
<p><img alt="keras+tensorflow" src="/img/keras-tensorflow-logo.jpg" /></p>
<hr />
<h2 id="calling-keras-layers-on-tensorflow-tensors">I: Calling Keras layers on TensorFlow tensors</h2>
<p>Let's start with a simple example: MNIST digits classification. We will build a TensorFlow digits classifier using a stack of Keras <code>Dense</code> layers (fully-connected layers).</p>
<p>We should start by creating a TensorFlow session and registering it with Keras. This means that Keras will use the session we registered to initialize all variables that it creates internally.</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">tensorflow</span> <span class="kn">as</span> <span class="nn">tf</span>
<span class="n">sess</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
<span class="kn">from</span> <span class="nn">keras</span> <span class="kn">import</span> <span class="n">backend</span> <span class="k">as</span> <span class="n">K</span>
<span class="n">K</span><span class="o">.</span><span class="n">set_session</span><span class="p">(</span><span class="n">sess</span><span class="p">)</span>
</pre></div>
<p>Now let's get started with our MNIST model. We can start building a classifier exactly as you would do in TensorFlow:</p>
<div class="highlight"><pre><span class="c"># this placeholder will contain our input digits, as flat vectors</span>
<span class="n">img</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">784</span><span class="p">))</span>
</pre></div>
<p>We can then use Keras layers to speed up the model definition process:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras.layers</span> <span class="kn">import</span> <span class="n">Dense</span>
<span class="c"># Keras layers can be called on TensorFlow tensors:</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">)(</span><span class="n">img</span><span class="p">)</span> <span class="c"># fully-connected layer with 128 units and ReLU activation</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span>
<span class="n">preds</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'softmax'</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="c"># output layer with 10 units and a softmax activation</span>
</pre></div>
<p>We define the placeholder for the labels, and the loss function we will use:</p>
<div class="highlight"><pre><span class="n">labels</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
<span class="kn">from</span> <span class="nn">keras.objectives</span> <span class="kn">import</span> <span class="n">categorical_crossentropy</span>
<span class="n">loss</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">categorical_crossentropy</span><span class="p">(</span><span class="n">labels</span><span class="p">,</span> <span class="n">preds</span><span class="p">))</span>
</pre></div>
<p>Let's train the model with a TensorFlow optimizer:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">tensorflow.examples.tutorials.mnist</span> <span class="kn">import</span> <span class="n">input_data</span>
<span class="n">mnist_data</span> <span class="o">=</span> <span class="n">input_data</span><span class="o">.</span><span class="n">read_data_sets</span><span class="p">(</span><span class="s">'MNIST_data'</span><span class="p">,</span> <span class="n">one_hot</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">train_step</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">GradientDescentOptimizer</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)</span><span class="o">.</span><span class="n">minimize</span><span class="p">(</span><span class="n">loss</span><span class="p">)</span>
<span class="k">with</span> <span class="n">sess</span><span class="o">.</span><span class="n">as_default</span><span class="p">():</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
<span class="n">batch</span> <span class="o">=</span> <span class="n">mnist_data</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">next_batch</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span>
<span class="n">train_step</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">feed_dict</span><span class="o">=</span><span class="p">{</span><span class="n">img</span><span class="p">:</span> <span class="n">batch</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
<span class="n">labels</span><span class="p">:</span> <span class="n">batch</span><span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
</pre></div>
<p>We can now evaluate the model:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras.metrics</span> <span class="kn">import</span> <span class="n">categorical_accuracy</span> <span class="k">as</span> <span class="n">accuracy</span>
<span class="n">acc_value</span> <span class="o">=</span> <span class="n">accuracy</span><span class="p">(</span><span class="n">labels</span><span class="p">,</span> <span class="n">preds</span><span class="p">)</span>
<span class="k">with</span> <span class="n">sess</span><span class="o">.</span><span class="n">as_default</span><span class="p">():</span>
<span class="k">print</span> <span class="n">acc_value</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">feed_dict</span><span class="o">=</span><span class="p">{</span><span class="n">img</span><span class="p">:</span> <span class="n">mnist_data</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">images</span><span class="p">,</span>
<span class="n">labels</span><span class="p">:</span> <span class="n">mnist_data</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">labels</span><span class="p">})</span>
</pre></div>
<p>In this case, we use Keras only as a syntactical shortcut to generate an op that maps some tensor(s) input to some tensor(s) output, and that's it. The optimization is done via a native TensorFlow optimizer rather than a Keras optimizer. We don't even use any Keras <code>Model</code> at all!</p>
<p>A note on the relative performance of native TensorFlow optimizers and Keras optimizers: there are slight speed differences when optimizing a model "the Keras way" vs. with a TensorFlow optimizer. Somewhat counter-intuitively, Keras seems faster most of the time, by 5-10%. However these differences are small enough that it doesn't really matter, at the end of the day, whether you optimize your models via Keras optimizers or native TF optimizers.</p>
<h4>Different behaviors during training and testing</h4>
<p>Some Keras layers (e.g. <code>Dropout</code>, <code>BatchNormalization</code>) behave differently at training time and testing time. You can tell whether a layer uses the "learning phase" (train/test) by printing <code>layer.uses_learning_phase</code>, a boolean: <code>True</code> if the layer has a different behavior in training mode and test mode, <code>False</code> otherwise.</p>
<p>If your model includes such layers, then you need to specify the value of the learning phase as part of <code>feed_dict</code>, so that your model knows whether to apply dropout/etc or not.</p>
<p>The Keras learning phase (a scalar TensorFlow tensor) is accessible via the Keras backend:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras</span> <span class="kn">import</span> <span class="n">backend</span> <span class="k">as</span> <span class="n">K</span>
<span class="k">print</span> <span class="n">K</span><span class="o">.</span><span class="n">learning_phase</span><span class="p">()</span>
</pre></div>
<p>To make use of the learning phase, simply pass the value "1" (training mode) or "0" (test mode) to <code>feed_dict</code>:</p>
<div class="highlight"><pre><span class="c"># train mode</span>
<span class="n">train_step</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">feed_dict</span><span class="o">=</span><span class="p">{</span><span class="n">x</span><span class="p">:</span> <span class="n">batch</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">labels</span><span class="p">:</span> <span class="n">batch</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">K</span><span class="o">.</span><span class="n">learning_phase</span><span class="p">():</span> <span class="mi">1</span><span class="p">})</span>
</pre></div>
<p>For instance, here's how to add <code>Dropout</code> layers to our previous MNIST example:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras.layers</span> <span class="kn">import</span> <span class="n">Dropout</span>
<span class="kn">from</span> <span class="nn">keras</span> <span class="kn">import</span> <span class="n">backend</span> <span class="k">as</span> <span class="n">K</span>
<span class="n">img</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">784</span><span class="p">))</span>
<span class="n">labels</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">)(</span><span class="n">img</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">Dropout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">Dropout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span>
<span class="n">preds</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'softmax'</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span>
<span class="n">loss</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">categorical_crossentropy</span><span class="p">(</span><span class="n">labels</span><span class="p">,</span> <span class="n">preds</span><span class="p">))</span>
<span class="n">train_step</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">GradientDescentOptimizer</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)</span><span class="o">.</span><span class="n">minimize</span><span class="p">(</span><span class="n">loss</span><span class="p">)</span>
<span class="k">with</span> <span class="n">sess</span><span class="o">.</span><span class="n">as_default</span><span class="p">():</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
<span class="n">batch</span> <span class="o">=</span> <span class="n">mnist_data</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">next_batch</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span>
<span class="n">train_step</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">feed_dict</span><span class="o">=</span><span class="p">{</span><span class="n">img</span><span class="p">:</span> <span class="n">batch</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
<span class="n">labels</span><span class="p">:</span> <span class="n">batch</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span>
<span class="n">K</span><span class="o">.</span><span class="n">learning_phase</span><span class="p">():</span> <span class="mi">1</span><span class="p">})</span>
<span class="n">acc_value</span> <span class="o">=</span> <span class="n">accuracy</span><span class="p">(</span><span class="n">labels</span><span class="p">,</span> <span class="n">preds</span><span class="p">)</span>
<span class="k">with</span> <span class="n">sess</span><span class="o">.</span><span class="n">as_default</span><span class="p">():</span>
<span class="k">print</span> <span class="n">acc_value</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">feed_dict</span><span class="o">=</span><span class="p">{</span><span class="n">img</span><span class="p">:</span> <span class="n">mnist_data</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">images</span><span class="p">,</span>
<span class="n">labels</span><span class="p">:</span> <span class="n">mnist_data</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">labels</span><span class="p">,</span>
<span class="n">K</span><span class="o">.</span><span class="n">learning_phase</span><span class="p">():</span> <span class="mi">0</span><span class="p">})</span>
</pre></div>
<h4>Compatibility with name scopes, device scopes</h4>
<p>Keras layers and models are fully compatible with TensorFlow name scopes. For instance, consider the following code snippet:</p>
<div class="highlight"><pre><span class="n">x</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">64</span><span class="p">))</span>
<span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">name_scope</span><span class="p">(</span><span class="s">'block1'</span><span class="p">):</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">LSTM</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'mylstm'</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span>
</pre></div>
<p>The weights of our LSTM layer will then be named <code>block1/mylstm_W_i</code>, <code>block1/mylstm_U_i</code>, etc... </p>
<p>Similarly, device scopes work as you would expect:</p>
<div class="highlight"><pre><span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s">'/gpu:0'</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">64</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">LSTM</span><span class="p">(</span><span class="mi">32</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="c"># all ops / variables in the LSTM layer will live on GPU:0</span>
</pre></div>
<h4>Compatibility with graph scopes</h4>
<p>Any Keras layer or model that you define inside a TensorFlow graph scope will have all of its variables and operations created as part of the specified graph. For instance, the following works as you would expect:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras.layers</span> <span class="kn">import</span> <span class="n">LSTM</span>
<span class="kn">import</span> <span class="nn">tensorflow</span> <span class="kn">as</span> <span class="nn">tf</span>
<span class="n">my_graph</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">Graph</span><span class="p">()</span>
<span class="k">with</span> <span class="n">my_graph</span><span class="o">.</span><span class="n">as_default</span><span class="p">():</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">64</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">LSTM</span><span class="p">(</span><span class="mi">32</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="c"># all ops / variables in the LSTM layer are created as part of our graph</span>
</pre></div>
<h4>Compatibility with variable scopes</h4>
<p>Variable sharing should be done via calling a same Keras layer (or model) instance multiple times, NOT via TensorFlow variable scopes. A TensorFlow variable scope will have no effect on a Keras layer or model. For more information about weight sharing with Keras, please see <a href="http://keras.io/getting-started/functional-api-guide/#shared-layers">the "weight sharing" section in the functional API guide</a>.</p>
<p>To summarize quickly how weight sharing works in Keras: by reusing the same layer instance or model instance, you are sharing its weights. Here's a simple example:</p>
<div class="highlight"><pre><span class="c"># instantiate a Keras layer</span>
<span class="n">lstm</span> <span class="o">=</span> <span class="n">LSTM</span><span class="p">(</span><span class="mi">32</span><span class="p">)</span>
<span class="c"># instantiate two TF placeholders</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">64</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">64</span><span class="p">))</span>
<span class="c"># encode the two tensors with the *same* LSTM weights</span>
<span class="n">x_encoded</span> <span class="o">=</span> <span class="n">lstm</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">y_encoded</span> <span class="o">=</span> <span class="n">lstm</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
</pre></div>
<h4>Collecting trainable weights and state updates</h4>
<p>Some Keras layers (stateful RNNs and <code>BatchNormalization</code> layers) have internal updates that need to be run as part of each training step. There are stored as a list of tensor tuples, <code>layer.updates</code>. You should generate <code>assign</code> ops for those, to be run at each training step. Here's an example:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras.layers</span> <span class="kn">import</span> <span class="n">BatchNormalization</span>
<span class="n">layer</span> <span class="o">=</span> <span class="n">BatchNormalization</span><span class="p">()(</span><span class="n">x</span><span class="p">)</span>
<span class="n">update_ops</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">old_value</span><span class="p">,</span> <span class="n">new_value</span> <span class="ow">in</span> <span class="n">layer</span><span class="o">.</span><span class="n">updates</span><span class="p">:</span>
<span class="n">update_ops</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">assign</span><span class="p">(</span><span class="n">old_value</span><span class="p">,</span> <span class="n">new_value</span><span class="p">))</span>
</pre></div>
<p>Note that if you are using a Keras model (<code>Model</code> instance or <code>Sequential</code> instance), <code>model.udpates</code> behaves in the same way (and collects the updates of all underlying layers in the model).</p>
<p>In addition, in case you need to explicitly collect a layer's trainable weights, you can do so via <code>layer.trainable_weights</code> (or <code>model.trainable_weights</code>), a list of TensorFlow <code>Variable</code> instances:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras.layers</span> <span class="kn">import</span> <span class="n">Dense</span>
<span class="n">layer</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">32</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="c"># instantiate and call a layer</span>
<span class="k">print</span> <span class="n">layer</span><span class="o">.</span><span class="n">trainable_weights</span> <span class="c"># list of TensorFlow Variables</span>
</pre></div>
<p>Knowing this allows you to implement your own training routine based on a TensorFlow optimizer.</p>
<hr />
<h2 id="using-keras-models-with-tensorflow">II: Using Keras models with TensorFlow</h2>
<h4>Converting a Keras <code>Sequential</code> model for use in a TensorFlow workflow</h4>
<p>You have found a Keras <code>Sequential</code> model that you want to reuse in your TensorFlow project (consider, for instance, <a href="https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3">this VGG16 image classifier with pre-trained weights</a>). How to proceed?</p>
<p>First of all, note that if your pre-trained weights include convolutions (layers <code>Convolution2D</code> or <code>Convolution1D</code>) that were trained with <strong>Theano</strong>, you need to flip the convolution kernels when loading the weights. This is due Theano and TensorFlow implementing convolution in different ways (TensorFlow actually implements correlation, much like Caffe). <a href="https://github.com/fchollet/keras/wiki/Converting-convolution-kernels-from-Theano-to-TensorFlow-and-vice-versa">Here's a short guide on what you need to do in this case</a>. </p>
<p>Let's say that you are starting from the following Keras model, and that you want to modify so that it takes as input a specific TensorFlow tensor, <code>my_input_tensor</code>. This input tensor could be a data feeder op, for instance, or the output of a previous TensorFlow model.</p>
<div class="highlight"><pre><span class="c"># this is our initial Keras model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">first_layer</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">,</span> <span class="n">input_dim</span><span class="o">=</span><span class="mi">784</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'softmax'</span><span class="p">))</span>
</pre></div>
<p>You just need to use <code>set_input</code> to modify the input of the first layer right after you instantiate it, then build the rest of the model on top:</p>
<div class="highlight"><pre><span class="c"># this is our modified Keras model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">first_layer</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">,</span> <span class="n">input_dim</span><span class="o">=</span><span class="mi">784</span><span class="p">)</span>
<span class="n">first_layer</span><span class="o">.</span><span class="n">set_input</span><span class="p">(</span><span class="n">my_input_tensor</span><span class="p">)</span>
<span class="c"># build the rest of the model as before</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">first_layer</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'softmax'</span><span class="p">))</span>
</pre></div>
<p>At this stage, you can call <code>model.load_weights(weights_file)</code> to load your pre-trained weights.</p>
<p>Then you will probably want to collect the <code>Sequential</code> model's output tensor:</p>
<div class="highlight"><pre><span class="n">output_tensor</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">output</span>
</pre></div>
<p>You can now add new TensorFlow ops on top of <code>output_tensor</code>, etc.</p>
<h4>Calling a Keras model on a TensorFlow tensor</h4>
<p>A Keras model acts the same as a layer, and thus can be called on TensorFlow tensors:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras.models</span> <span class="kn">import</span> <span class="n">Sequential</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">,</span> <span class="n">input_dim</span><span class="o">=</span><span class="mi">784</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'softmax'</span><span class="p">))</span>
<span class="c"># this works! </span>
<span class="n">x</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">784</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</pre></div>
<p>Note: by calling a Keras model, your are reusing both its architecture and its weights. When you are calling a model on a tensor, you are creating new TF ops on top of the input tensor, and these ops are reusing the TF <code>Variable</code> instances already present in the model.</p>
<hr />
<h2 id="multi-gpu-and-distributed-training">III: Multi-GPU and distributed training</h2>
<h4>Assigning part of a Keras model to different GPUs</h4>
<p>TensorFlow device scopes are fully compatible with Keras layers and models, hence you can use them to assign specific parts of a graph to different GPUs. Here's a simple example:</p>
<div class="highlight"><pre><span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s">'/gpu:0'</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">64</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">LSTM</span><span class="p">(</span><span class="mi">32</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="c"># all ops in the LSTM layer will live on GPU:0</span>
<span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s">'/gpu:1'</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">64</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">LSTM</span><span class="p">(</span><span class="mi">32</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="c"># all ops in the LSTM layer will live on GPU:1</span>
</pre></div>
<p>Note that the variables created by the LSTM layers will not live on GPU: all TensorFlow variables always live on CPU independently from the device scope where they were created. TensorFlow handles device-to-device variable transfer behind the scenes.</p>
<p>If you want to train multiple replicas of a same model on different GPUs, while sharing the same weights across the different replicas, you should first instantiate your model (or layers) under one device scope, then call the same model instance multiple times in different GPU device scopes, such as:</p>
<div class="highlight"><pre><span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s">'/cpu:0'</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">placeholder</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">784</span><span class="p">))</span>
<span class="c"># shared model living on CPU:0</span>
<span class="c"># it won't actually be run during training; it acts as an op template</span>
<span class="c"># and as a repository for shared variables</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">,</span> <span class="n">input_dim</span><span class="o">=</span><span class="mi">784</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'softmax'</span><span class="p">))</span>
<span class="c"># replica 0</span>
<span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s">'/gpu:0'</span><span class="p">):</span>
<span class="n">output_0</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="c"># all ops in the replica will live on GPU:0</span>
<span class="c"># replica 1</span>
<span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s">'/gpu:1'</span><span class="p">):</span>
<span class="n">output_1</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="c"># all ops in the replica will live on GPU:1</span>
<span class="c"># merge outputs on CPU</span>
<span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s">'/cpu:0'</span><span class="p">):</span>
<span class="n">preds</span> <span class="o">=</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="p">(</span><span class="n">output_0</span> <span class="o">+</span> <span class="n">output_1</span><span class="p">)</span>
<span class="c"># we only run the `preds` tensor, so that only the two</span>
<span class="c"># replicas on GPU get run (plus the merge op on CPU)</span>
<span class="n">output_value</span> <span class="o">=</span> <span class="n">sess</span><span class="o">.</span><span class="n">run</span><span class="p">([</span><span class="n">preds</span><span class="p">],</span> <span class="n">feed_dict</span><span class="o">=</span><span class="p">{</span><span class="n">x</span><span class="p">:</span> <span class="n">data</span><span class="p">})</span>
</pre></div>
<h4>Distributed training</h4>
<p>You can trivially make use of TensorFlow distributed training by registering with Keras a TF session linked to a cluster:</p>
<div class="highlight"><pre><span class="n">server</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Server</span><span class="o">.</span><span class="n">create_local_server</span><span class="p">()</span>
<span class="n">sess</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">Session</span><span class="p">(</span><span class="n">server</span><span class="o">.</span><span class="n">target</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">keras</span> <span class="kn">import</span> <span class="n">backend</span> <span class="k">as</span> <span class="n">K</span>
<span class="n">K</span><span class="o">.</span><span class="n">set_session</span><span class="p">(</span><span class="n">sess</span><span class="p">)</span>
</pre></div>
<p>For more information about using TensorFlow in a distributed setting, see <a href="https://www.tensorflow.org/versions/r0.8/how_tos/distributed/index.html">this tutorial</a>.</p>
<hr />
<h2 id="exporting-a-model-with-tensorflow-serving">IV: Exporting a model with TensorFlow-serving</h2>
<p><a href="https://github.com/tensorflow/serving">TensorFlow Serving</a> is a library for serving TensorFlow models in a production setting, developed by Google.</p>
<p>Any Keras model can be exported with TensorFlow-serving (as long as it only has one input and one output, which is a limitation of TF-serving), whether or not it was training as part of a TensorFlow workflow. In fact you could even train your Keras model with Theano then switch to the TensorFlow Keras backend and export your model.</p>
<p>Here's how it works.</p>
<p>If your graph makes use of the Keras learning phase (different behavior at training time and test time), the very first thing to do before exporting your model is to hard-code the value of the learning phase (as 0, presumably, i.e. test mode) into your graph. This is done by 1) registering a constant learning phase with the Keras backend, and 2) re-building your model afterwards.</p>
<p>Here are these two simple steps in action:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">keras</span> <span class="kn">import</span> <span class="n">backend</span> <span class="k">as</span> <span class="n">K</span>
<span class="n">K</span><span class="o">.</span><span class="n">set_learning_phase</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c"># all new operations will be in test mode from now on</span>
<span class="c"># serialize the model and get its weights, for quick re-building</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">previous_model</span><span class="o">.</span><span class="n">get_config</span><span class="p">()</span>
<span class="n">weights</span> <span class="o">=</span> <span class="n">previous_model</span><span class="o">.</span><span class="n">get_weights</span><span class="p">()</span>
<span class="c"># re-build a model where the learning phase is now hard-coded to 0</span>
<span class="kn">from</span> <span class="nn">keras.models</span> <span class="kn">import</span> <span class="n">model_from_config</span>
<span class="n">new_model</span> <span class="o">=</span> <span class="n">model_from_config</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">new_model</span><span class="o">.</span><span class="n">set_weights</span><span class="p">(</span><span class="n">weights</span><span class="p">)</span>
</pre></div>
<p>We can now use TensorFlow-serving to export the model, following the instructions found in <a href="https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/serving_basic.md">the official tutorial</a>:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">tensorflow_serving.session_bundle</span> <span class="kn">import</span> <span class="n">exporter</span>
<span class="n">export_path</span> <span class="o">=</span> <span class="o">...</span> <span class="c"># where to save the exported graph</span>
<span class="n">export_version</span> <span class="o">=</span> <span class="o">...</span> <span class="c"># version number (integer)</span>
<span class="n">saver</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Saver</span><span class="p">(</span><span class="n">sharded</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">model_exporter</span> <span class="o">=</span> <span class="n">exporter</span><span class="o">.</span><span class="n">Exporter</span><span class="p">(</span><span class="n">saver</span><span class="p">)</span>
<span class="n">signature</span> <span class="o">=</span> <span class="n">exporter</span><span class="o">.</span><span class="n">classification_signature</span><span class="p">(</span><span class="n">input_tensor</span><span class="o">=</span><span class="n">model</span><span class="o">.</span><span class="n">input</span><span class="p">,</span>
<span class="n">scores_tensor</span><span class="o">=</span><span class="n">model</span><span class="o">.</span><span class="n">output</span><span class="p">)</span>
<span class="n">model_exporter</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">sess</span><span class="o">.</span><span class="n">graph</span><span class="o">.</span><span class="n">as_graph_def</span><span class="p">(),</span>
<span class="n">default_graph_signature</span><span class="o">=</span><span class="n">signature</span><span class="p">)</span>
<span class="n">model_exporter</span><span class="o">.</span><span class="n">export</span><span class="p">(</span><span class="n">export_path</span><span class="p">,</span> <span class="n">tf</span><span class="o">.</span><span class="n">constant</span><span class="p">(</span><span class="n">export_version</span><span class="p">),</span> <span class="n">sess</span><span class="p">)</span>
</pre></div>
<hr />
<p>Want to see a new topic covered in this guide? <a href="https://twitter.com/fchollet">Reach out on Twitter</a>.</p>
<hr />
</div><!-- /.entry-content -->
</article>
</section>
<footer id="footer" class="body">
<address id="about" class="vcard body">
Powered by <a href="http://alexis.notmyidea.org/pelican/">pelican</a>, which takes great advantages of <a href="http://python.org">python</a>.
</address><!-- /#about -->
</footer><!-- /#footer -->
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-61785484-1");
pageTracker._trackPageview();
} catch(err) {}</script>
</body>
</html>