<p>TensorFlow 2.0 preview version has just been <a href="https://groups.google.com/a/tensorflow.org/forum/#!topic/developers/aKdmUOiyzGM">announced</a>.
In this blog post, I’ll summarize what to expect for migrating your code to this new version.</p>
<h2>Official resources</h2>
<ul>
<li><a href="https://groups.google.com/a/tensorflow.org/forum/?#!topic/announce/qXfsxr2sF-0">2.0 first official announcement</a></li>
<li><a href="https://groups.google.com/a/tensorflow.org/forum/#!topic/developers/aKdmUOiyzGM">Preview version announcement</a></li>
<li><a href="https://www.tensorflow.org/versions/r2.0/api_docs/python/tf">2.0 docs</a></li>
<li><a href="https://github.com/tensorflow/community/tree/master/rfcs">2.0 design documents explaining the new APIs</a></li>
<li><a href="https://github.com/tensorflow/docs/tree/master/site/en/r2">2.0 guides and tutorials</a> (might be moved, <a href="https://github.com/tensorflow/docs/tree/ab40de5b068f135a6f1840ba2dd7465386f4edc8/site/en/r2">permalink)</a></li>
</ul>
<h2>Migrating your code automatically</h2>
<p>Here's the good news: TensorFlow 2.0 provides an <strong>automatic migration tool</strong>. After <a href="https://www.tensorflow.org/versions/r2.0/api_docs/python/tf">installing the new version</a>, you can run:</p>
<pre><code>tf_upgrade_v2 --intree ./project_directory --outtree ./project_directory_updated
</code></pre>
<p>So what is this tool doing ? It's mostly moving your code to <code>tf.compat.v1</code> and changing the name of the arguments if necessary.
The script should update your code to make it work, but will not make it follow TensorFlow 2.0 best practices.</p>
<p>Moreover, it does not work with <code>tf.contrib</code>. Indeed, this module has been removed from TensorFlow 2.0,
with most of the APIs moved to TensorFlow core or external repositories. <a href="https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md">See the details here</a></p>
<h2>Migrating your code manually</h2>
<p>While your code will work using <code>tf.compat.v1</code>, to follow TensorFlow 2.0 new philosophy you will have to update your code manually.
Here are the biggest changes:</p>
<ul>
<li>as stated above, <code>tf.contrib</code> will disappear</li>
<li>using eager execution is recommended (<a href="https://github.com/tensorflow/docs/blob/master/site/en/r2/guide/eager.ipynb">see example notebook</a>)</li>
<li>the <code>tf.Session</code> API will disappear in favor of <code>tf.function</code> (<a href="https://github.com/tensorflow/docs/blob/master/site/en/r2/guide/autograph.ipynb">see example notebook</a>)</li>
<li><code>tf.layers</code> will disappear, model definition should be done using <code>tf.keras</code></li>
</ul>
<p>A <strong>very good explanation</strong> is available in the official TensorFlow 2.0 guides <a href="https://github.com/tensorflow/community/blob/b1d83bf2ee3fc72650140b89656e29932db36226/rfcs/20180918-functions-not-sessions-20.md">here</a>.</p>
<h2>Conclusion</h2>
<p>Note that this is only a preview version: expect some bugs and inconsistencies. Nevertheless,
it should give you a good vision on this substantial TensorFlow update. In upcoming blogposts, I'll detail each of the new
changes in TensorFlow.</p>
<p><em>I'm still unsure about the future of TensorFlow Estimators and TPUs.
If you have any info about this, reach out to me <a href="https://twitter.com/EliotAndres">on Twitter</a>.</em></p>


How to migrate your code to TensorFlow 2.0

Migrating to TensorFlow 2.0

<p>In this article, I’ll detail the technique I use to solve almost any AI / machine learning project.
I can already hear you screaming behind your screen <em>« there is no magic approach to ML »</em>  and you’d be right!</p>
<p>I would say that this technique applied to 90% of my projects. Without further ado, <strong>here’s the approach</strong>:</p>
<ol>
<li>Find a machine learning competition with a problem close to the one you want to solve</li>
<li>Find the winning team’s solution</li>
<li>Adapt this solution to your problem</li>
</ol>
<p>Before detailing each step, I’d like to point out that the winning team solution is not used as is.
Mostly because your problem will not be perfectly identical to the competition’s problem.</p>
<h2>1 - Finding a matching machine learning competition</h2>
<p>To do so, I recommend using Google or <a href="https://ndres.me/kaggle-past-solutions/">Kaggle Past Solutions</a>. (<strong>disclaimer</strong>: I created Kaggle Past Solutions, precisely for this reason. It’s open-source and free)</p>
<p>How to find a matching competition ? Let’s use an example. I was recently working
on a project to predict the inventory in a warehouse. Since we want to predict future orders,
the keywords here are « time series » and « forecast ».
Looking up those keywords in Kaggle Past Solutions yields, among other results,
the <a href="https://www.kaggle.com/c/favorita-grocery-sales-forecasting">Corporación Favorita Grocery Sales Forecasting competition</a>
("Can you accurately predict sales for a large grocery chain? »).</p>
<p>This competition seems similar enough. Indeed, an order from a warehouse must be
correlated to the sales, so with the same input we might be able to get good results.
Now let’s move on to choosing a solution.</p>
<h2>2 - Finding the winning team’s solution</h2>
<p>This step should be rephrased into "finding a good winning solution". It mostly depends on:</p>
<ul>
<li>if the source code is available</li>
<li>if a detailed explanation is available</li>
<li>the complexity of the solution. Many Kaggle solutions use
<a href="http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/">stacking</a>
and <a href="https://en.wikipedia.org/wiki/Ensemble_learning">ensembling</a>, sometimes making
them very complex. Thankfully, solutions sometimes offer a « best single model ».</li>
</ul>
<p>In our example, the <a href="https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47582">first place solution</a>
is of very good quality. It quotes sources, contains a good explanation and the linked code is concise and of good quality.</p>
<h2>3 - Adapt this solution to your problem</h2>
<p>The solution will be used as a baseline. You will know what technology to use
(convolutions, gradient boosting, LSTMs) and what part of the data to exploit.
While you will not get a working solution right away, you’ll at least know that
you’re using the right tools and the right data.</p>
<p>Moreover, you can draw inspiration and learn from:</p>
<ul>
<li>the way the code is architectured</li>
<li>the <a href="https://en.wikipedia.org/wiki/Feature_engineering">feature engineering</a></li>
<li>the libraries used</li>
</ul>
<p>In our example, the solution is using an ensemble of LSTMs and gradient boosting.
Reading the code, I also learned about <a href="https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html">pandas.read_csv</a>'s
<code>converters</code> argument.</p>
<h2>What's next</h2>
<p>At this point you should have a very good baseline, built upon <a href="https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_giants">the shoulders of kaggle masters</a>.
From this point, it's all about improving your model. And if you get good results, don't forget to thank the authors of the solution.</p>


A technique to solve machine learning problems

My approach to solving (almost) any machine learning problem

<h2>About style transfer</h2>
<p>Pioneered in 2015, style transfer is a concept that uses transfers the style of a painting to an existing photography, using neural networks.
The original paper is <a href="https://arxiv.org/abs/1508.06576">A Neural Algorithm of Artistic Style</a> by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.</p>
<p>Here are a few examples taken from it:</p>
<figure class="my-8 text-center"><img src="/images/archive/style-transfer-examples.jpg" alt="Style transfer example from the original paper" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Style transfer example from the original paper</figcaption></figure>
<h2>How it works</h2>
<p>This gif is meant to give you a rough idea on how style transfer works in the orignal paper:</p>
<figure class="my-8 text-center"><img src="/images/archive/style-transfer.gif" alt="Style transfer explained with a gif (click to enlarge)" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Style transfer explained with a gif (click to enlarge)</figcaption></figure>
<p>Although I tried to make the gif self-explanatory, here are a few more details:</p>
<ul>
<li>"Filtered image" means the output of a truncated VGG16 network. This means that we run the image through VGG16
but we stop at some layer. Depending on the goal, we stop at different layers.</li>
<li>Gram matrix: you'll find an explanation below The general idea is that if coefficient (n1, n2) is high, it means that filters n1 and n2 activate for the same pixels.</li>
<li>In practice there is a bit more than 5 steps to optimize the random pixels</li>
</ul>
<p>The paper is really easy to read, I really recommend <a href="https://arxiv.org/abs/1508.06576">having a look</a></p>
<h2>Bonus: how to calculate a Gram Matrix</h2>
<p>(This explanation comes from <a href="https://github.com/aleju/papers/blob/master/neural-nets/A_Neural_Algorithm_for_Artistic_Style.md">Alexander Jung's summary of the paper</a>)</p>
<ul>
<li>Take the activations of a layer. That layer will contain some convolution filters (e.g. 128), each one having its own activations.</li>
<li>Convert each filter's activations to a (1-dimensional) vector.</li>
<li>Pick all pairs of filters. Calculate the scalar product of both filter's vectors.</li>
<li>Add the scalar product result as an entry to a matrix of size #filters x #filters (e.g. 128x128).</li>
<li>Repeat that for every pair to get the Gram Matrix.</li>
<li>The Gram Matrix roughly represents the texture of the image.</li>
</ul>
<h2>More ressources</h2>
<ul>
<li><a href="https://arxiv.org/abs/1508.06576">Original paper</a></li>
<li><a href="https://github.com/keras-team/keras/blob/master/examples/neural_style_transfer.py">Code in Keras</a></li>
</ul>
<h2>Style transfer today</h2>
<p>Since the original paper, style transfer improved a lot, both in speed and quality. Here's an example of what you can do with the latest paper:</p>
<blockquote class="twitter-tweet" data-lang="fr"><p lang="en" dir="ltr">Style transfer just reached a new wow moment ! Over the week-end I played with &quot;Deep Painterly Harmonization&quot;, Fujun Luan et al. Can you tell the difference with a real painting ? Code: <a href="https://t.co/luFdB5F3M3">https://t.co/luFdB5F3M3</a> Paper (very well written): <a href="https://t.co/gSGoqZ2WEM">https://t.co/gSGoqZ2WEM</a> <a href="https://t.co/maRJRXpffK">pic.twitter.com/maRJRXpffK</a></p>&mdash; Eliot Andres (@EliotAndres) <a href="https://twitter.com/EliotAndres/status/985817482358415361?ref_src=twsrc%5Etfw">16 avril 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>


Understanding style transfer with animated gifs

Machine learning explained with gifs: style transfer

<p><strong>Edit</strong>: There is now a much better and simpler way of creating gifs, a <a href="https://github.com/maxhumber/gif">libray called "gif"</a></p>
<h2>The problem</h2>
<p>If you Google how to make an animated Matplotlib graph, you end up with code like that:</p>
<pre><code>fig, ax = plt.subplots()

x = np.arange(0, 2*np.pi, 0.01)
line, = ax.plot(x, np.sin(x))

def animate(i):
    line.set_ydata(np.sin(x + i/10.0))  # update the data
    return line,

# Init only required for blitting to give a clean slate.
def init():
    line.set_ydata(np.ma.array(x, mask=True))
    return line,

ani = animation.FuncAnimation(fig, animate, np.arange(1, 200), init_func=init, interval=25, blit=True)
</code></pre>
<p>The logic of using an init function and methods such as <code>set_ydata</code> is a bit <strong>too complex</strong> for my taste, so I ended up using a different approach</p>
<h2>The simple solution</h2>
<p>The idea is to build an array of images and to convert it to a gif. To do so you need <a href="https://imageio.github.io/">Imageio</a>.</p>
<pre><code>import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import imageio

def plot_for_offset(power, y_max):
    # Data for plotting
    t = np.arange(0.0, 100, 1)
    s = t**power

    fig, ax = plt.subplots(figsize=(10,5))
    ax.plot(t, s)
    ax.grid()
    ax.set(xlabel='X', ylabel='x^{}'.format(power),
           title='Powers of x')

    # IMPORTANT ANIMATION CODE HERE
    # Used to keep the limits constant
    ax.set_ylim(0, y_max)

    # Used to return the plot as an image rray
    fig.canvas.draw()       # draw the canvas, cache the renderer
    image = np.frombuffer(fig.canvas.tostring_rgb(), dtype='uint8')
    image  = image.reshape(fig.canvas.get_width_height()[::-1] + (3,))

    return image

kwargs_write = {'fps':1.0, 'quantizer':'nq'}
imageio.mimsave('./powers.gif', [plot_for_offset(i/4, 100) for i in range(10)], fps=1)
</code></pre>
<h2>Result</h2>
<figure class="my-8 text-center"><img src="/images/archive/powers.gif" alt="Result of the code above" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Result of the code above</figcaption></figure>


In this tutorial, I'll show you how to export your matplotlib plots as animated gifs

How to make animated gifs from Matplotlib plots easily

<p>Inspired by a <a href="https://medium.com/@hondanhon/i-trained-a-neural-net-to-generate-british-placenames-9460e907e4e9">Dan Hon article</a> on how to generate British placenames, I decided to train my own network to generate startup names.</p>
<p>The <a href="https://github.com/karpathy/char-rnn">original code</a> was made by Andej Karpathy, but there is a more modern and concise version in Keras, <a href="https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py">available here</a>.</p>
<h1>The results</h1>
<p>Here are a few of my favorites:</p>
<ul>
<li>Cuyation</li>
<li>SaleBott</li>
<li>Airr Energy</li>
<li>Costapi</li>
<li>Excel Worldeh</li>
<li>Hight Biotherapeutics</li>
<li>Joyai</li>
<li>YouLab</li>
<li>Rygo Inc.</li>
<li>Lifester</li>
<li>Mobilivy</li>
<li>bitran</li>
<li>OpenFactor</li>
</ul>
<p>You can find a <a href="https://github.com/EliotAndres/char-rnn-tensorflow-js/blob/master/generated.txt">longer list here</a>.</p>
<p><strong>Note</strong>: you can't prevent the neural net from generating startup names that already exist. So if you were planing on using any of those, do your own research first.</p>
<h1>How it works</h1>
<p>I trained the neural network on 170k startup names. To generate a startup name, you start with 10 random characters as a seed, feed it to the network (a <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">LSTM</a>) and ask it to predict the next character.</p>
<p>You then append the predicted character to the seed, remove the first character (so its length is still 10) and repeat.</p>
<p>If we start with this seed: <code>Kdsdgasdgd</code>, for our last example it looks like this:</p>
<ul>
<li>model.predict('Kdsdgasdgd') -> returns O</li>
<li>model.predict('dsdgasdgdO') -> returns p</li>
<li>model.predict('sdgasdgdOp') -> returns e</li>
<li>model.predict('dgasdgdOpe') -> returns n</li>
<li>[...]</li>
<li>model.predict('OpenFactor') -> returns '\n' (new line, meaning it's the end of the word)</li>
</ul>
<p>In summary, we repeatedly call a recurrent neural network. Deepception <a href="https://www.youtube.com/watch?v=UXd0EDy7aTY">(see video)</a></p>
<h1>The code</h1>
<p>If you want to dig deeper, the full code and dataset is available in a <a href="https://github.com/EliotAndres/char-rnn-tensorflow-js/blob/master/python/char-rnn-tensorflow-js.ipynb">notebook here</a>.</p>
<h1>Conclusion</h1>
<p>A few months ago, I made a startup pitch generator using Markov chains. You can find the <a href="http://ndres.me/markov-incubator/">demo here</a>. The 2 projects pair well together :-)</p>


I'll show you how to use a LSTM to generate startup names

Using a neural network to generate your next startup name

<p>Last week I published <a href="/post/facebook-graph-network/">an article</a> showing you how I built a friend graph using you Facebook data.</p>
<p>This article is a detailed version showing you how to do it yourself. Here's what we'll end up with:</p>
<figure class="my-8 text-center"><img src="/images/archive/facebook-graph.jpg" alt="Facebook friend network (click to enlarge)" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Facebook friend network (click to enlarge)</figcaption></figure>
<p><strong>Warning</strong>: To make such a graph, you need to scape all your mutual friends. Scrapping is automated but can take a few hours.</p>
<p>Let's get started !</p>
<h2>Scrapping the data</h2>
<p>Facebook does not let you access friends you have in common via any API, so we'll have to scrappe the data.</p>
<p>Clone the repo, and install requirements:</p>
<pre><code>git clone https://github.com/EliotAndres/facebook-friend-graph
cd facebook-friend-graph
pip install -r requirements.txt
</code></pre>
<p>The scrapping script will open a Chrome window and go to each of your friends profile. It'll then read the DOM to save your mutual friends.</p>
<p>It's using the Chrome drive, so you must also <a href="https://sites.google.com/a/chromium.org/chromedriver/">install it</a>. Once done, you can run:</p>
<pre><code>python facebookFOF.py
</code></pre>
<p>The script will prompt you for your username/password. You'll see a progress bar. It will periodically save the graph. So if it fails, you can just restart it and it'll start where it stopped.
It can take up to a few hours depending on your connection.</p>
<p>You'll end up with a .pickle file containing you friend graph information. Now let's plot this data !</p>
<h2>Building the graph</h2>
<p>You'll find all the details in the notebook below. <a href="https://github.com/EliotAndres/facebook-friend-graph/blob/master/Facebook%20friend%20graph.ipynb">It's available here.</a></p>
<script src="https://gist.github.com/EliotAndres/10d87567a2f81209f4c9be62c421a5df.js"></script>
<h2>Conclusion</h2>
<p>The graph should be made of differentiated clusters. You can hover over the clusters to see your friends names.
You can also play with the parameters and the colors to make it suit your needs. <strong>If you encounter any problem, open an issue on Github.</strong></p>
<h2>Acknowledgements</h2>
<p>Original notebook author: <a href="https://twitter.com/lucasallenio">Lucas Allen</a>, <a href="https://github.com/lgallen/twitter-graph">original code</a></p>
<p>Original scrapping code (Python 2) <a href="https://github.com/giladravid/FacebookFOF">here</a></p>


Tutorial (with code) on how to plot your Facebook friend network

Tutorial: Plotting your Facebook friend network using NetworkX and python-louvain

<p>Inspired by a friend's  post, I decided to plot my Facebook network. To do so, I scrapped "mutual friends" and made the following graph:</p>
<figure class="my-8 text-center"><img src="/images/archive/facebook-graph.jpg" alt="Facebook friend network (click to enlarge)" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Facebook friend network (click to enlarge)</figcaption></figure>
<p>In this blog post, I'll explain how the graph is made and how clusters are created. In a following blog post, I'll explain how to make it yourself.</p>
<h1>A few comments on the graph</h1>
<p>In this graph, each friend is represented by a node. Each edge represents a friendship.
You'll see that clusters are roughly made of where I went to school or in holidays.
One cluster is the group of people I'm traveling with as part of my <a href="http://remoteyear.com/">Remote Year</a>.
The node in the middle represents me.</p>
<h1>How it was made</h1>
<h3>1 - Scrapping mutual friends</h3>
<p>Facebook does not let you access friends you have in common via any API, so I had to make a script to gather the relationships. I ended up with a list of edges (friend1 &#x3C;-> friend2)</p>
<h3>2 - Plotting the graph</h3>
<p>The position of the nodes is computed using the <a href="https://en.wikipedia.org/wiki/Force-directed_graph_drawing">Fruchterman-Reingold Algorithm</a>.
It is a is a force-directed algorithm. The general idea is to represent each friendship by a spring. The consequences are:</p>
<ul>
<li>When you are part of a large group of people, the numerous springs will bring you closer to the group.</li>
<li>If you are friend with 2 large groups, you'll appear in the middle of the 2 groups because of the opposing forces.</li>
<li>If you are friend with only a few people, you'll appear on the outter border of the graph</li>
</ul>
<h3>3 - Clustering</h3>
<p>You'll also notice that each group has a different color. Each color represents a community. To create those communities, I used the <a href="https://en.wikipedia.org/wiki/Louvain_Modularity">Louvain Method for community detection</a>. The method attempts to maximize <a href="https://en.wikipedia.org/wiki/Modularity_(networks)">modularity</a>. The general idea of modularity is to measure the number of connections between people of different communities.</p>
<p>Therefore if the graph has a high modularity, it means that there are few edges connecting 2 nodes of different communities but numerous edges connecting 2 nodes of the same community.</p>
<p>In the Louvain Method of community detection, small communities are found by optimizing modularity locally on all nodes. Then each small community is grouped into one larger community.</p>
<p>You'll find an animation of the community merging below:</p>
<div class="my-8 aspect-video w-full"><iframe class="w-full h-full" src="https://www.youtube.com/embed/dGa-TXpoPz8" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
<p>Video credits to <a href="https://github.com/keithpjolley">Keith Jolley</a></p>
<h1>Make it yourself</h1>
<p>I'm currently writing a tutorial on how to make it yourself. When it'll be ready, I'll update this blog post.
<strong>Update</strong>: <a href="/post/friend-graph-tutorial/">here's the blog post</a></p>


In this blog post, I'll explain how to plot a Facebook friend network

Using Facebook data to plot my friend network

<p>Notebook extensions are plug-ins that you can easily add to your Jupyter notebooks.
The best way to install them is to use <a href="https://github.com/Jupyter-contrib/jupyter_nbextensions_configurator">Jupyter NbExtensions Configurator</a>.
It will add a tab to let you enable/disable extensions:</p>
<figure class="my-8 text-center"><img src="/images/archive/jupyter-nbextensions.png" alt="Screenshot of NbExtensions Configurator" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Screenshot of NbExtensions Configurator</figcaption></figure>
<h2>Installation</h2>
<p>Installation with conda:</p>
<pre><code>conda install -c conda-forge jupyter_contrib_nbextensions
conda install -c conda-forge jupyter_nbextensions_configurator
</code></pre>
<p>Or with pip:</p>
<pre><code>pip install jupyter_nbextensions_configurator jupyter_contrib_nbextensions
jupyter contrib nbextension install --user
jupyter nbextensions_configurator enable --user
</code></pre>
<p>Find more info about installation <a href="https://github.com/Jupyter-contrib/jupyter_nbextensions_configurator#installation">here</a></p>
<h2>1 - Collapsible headings</h2>
<p>Very useful when dealing with large notebooks, collapsible headings allow you to collapse some parts of the notebooks.</p>
<figure class="my-8 text-center"><img src="/images/archive/collapsible-headings.gif" alt="Using collapsible headings" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Using collapsible headings</figcaption></figure>
<h2>2 - Notify</h2>
<p>For long running task, the notify extension sends a notification when the notebook becomes idle.</p>
<figure class="my-8 text-center"><img src="/images/archive/jupyter-notification.gif" alt="Using notify" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Using notify</figcaption></figure>
<p><em>To use it, enable the extension and then enable it in the button bar.
The number you select is the minimum time the notebook has to run for you to get a notification (Note that you have to keep the notebook open in the browser for the notification to work)</em></p>
<h2>3 - Code folding</h2>
<figure class="my-8 text-center"><img src="/images/archive/jupyter-code-folding.gif" alt="Using code folding" class="mx-auto" /><figcaption class="text-center text-sm text-gray-500 mt-2">Using code folding</figcaption></figure>
<h2>4 - tqdm_notebook</h2>
<p>This one is not really an notebook extension. TQDM is a progress bar library. But it sometimes fails to work properly on Jupyter Notebooks. Thanks to <a href="https://twitter.com/randal_olson">Randy Olson</a> for the tip:</p>
<blockquote class="twitter-tweet" data-lang="fr"><p lang="en" dir="ltr">TIL: tqdm (the <a href="https://twitter.com/hashtag/Python?src=hash&amp;ref_src=twsrc%5Etfw">#Python</a> progress bar library) has a specialized &quot;tqdm_notebook&quot; function for use in Jupyter Notebooks.<br><br>No more messed up progress bars in my notebooks - hooray!<a href="https://t.co/r0jAQXQ6TM">https://t.co/r0jAQXQ6TM</a> <a href="https://t.co/FyYBRm2qE1">pic.twitter.com/FyYBRm2qE1</a></p>&mdash; Randy Olson (@randal_olson) <a href="https://twitter.com/randal_olson/status/969657169342734336?ref_src=twsrc%5Etfw">2 mars 2018</a></blockquote>
<h2>5 - %debug</h2>
<p>Not a notebook extension but an <a href="http://ipython.readthedocs.io/en/stable/interactive/magics.html">IPython magic</a> command. For a great explanation, I suggest reading the <a href="https://twitter.com/radekosmulski/status/945739571735748609">full twitter thread</a> from <a href="https://twitter.com/radekosmulski">Radek Osmulski</a></p>
<blockquote class="twitter-tweet" data-lang="fr"><p lang="en" dir="ltr">Favorite recent jupyter notebook discovery - the %debug magic:<br><br>1. Get an exception.<br>2. Insert a new cell, type %debug and run it.<br><br>An interactive debugger will open bringing you to where the exception occurred and allowing you to look around! <a href="https://t.co/9DSnSbpu15">pic.twitter.com/9DSnSbpu15</a></p>&mdash; Radek (@radekosmulski) <a href="https://twitter.com/radekosmulski/status/945739571735748609?ref_src=twsrc%5Etfw">26 décembre 2017</a></blockquote>
<h2>6 - Smaller extensions and other tips</h2>
<ul>
<li>
<p><strong>%lsmagic</strong>: Run this in a cell to list all available IPython magics</p>
</li>
<li>
<p><strong>Zen mode extension</strong>: removes the menus for less distractions</p>
</li>
<li>
<p><strong>Execute time extension</strong>: shows how long a cell took to run</p>
</li>
<li>
<p><strong>autoreload</strong>: Autoreloads external files without having to restart the notebook. To enable it:</p>
<pre><code>  %load_ext autoreload
  %autoreload 2
</code></pre>
</li>
</ul>
<h2>You know a must-have notebook extension ? Hit me on twitter or make a pull request on this blog post !</h2>
<h3>Edit on Wed, Mar 7, 2018:</h3>
<p>Some people <a href="https://www.reddit.com/r/MachineLearning/comments/82fcw0/d_top_5_best_jupyter_notebook_extensions/">on Reddit</a> suggested a few more:</p>
<ul>
<li><strong><a href="http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/varInspector/README.html">Variable inspector</a></strong>: displays all variables in a floating window</li>
<li><strong>CodeMirror Keymap</strong>: lets you choose between key bindings, such as vim</li>
<li><strong>Scratchpad</strong>: executes code against the current kernel without modifying the notebook document</li>
<li><strong>Splitcells</strong>: splits cells vertically</li>
</ul>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>


The best extensions to be more productive with notebooks

Top 5 Best Jupyter Notebook Extensions

<h2>Converting a Caffe model to TensorFlow</h2>
<p>The <a href="https://github.com/BVLC/caffe/wiki/Model-Zoo">Caffe Model Zoo</a> is an extraordinary place where reasearcher share their models. Caffe is an awesome framework, but you might want to use TensorFlow instead. In this blog post, I'll show you how to convert the <a href="https://github.com/metalbubble/places365">Places 365 model</a> to TensorFlow.</p>
<h2>Using Caffe-Tensorflow to convert your model</h2>
<p>Your best bet is to use the awesome <a href="https://github.com/ethereon/caffe-tensorflow">caffe-tensorflow</a>. This project takes a <code>prototxt</code> file as an input and converts it to a python file so you can use the model with TensorFlow. I had to use <a href="https://github.com/ethereon/caffe-tensorflow/pull/76">this pull request</a> to get a standalone model. I forked the repo with a <a href="https://github.com/ethereon/caffe-tensorflow/pull/105/files#diff-e9719ee0bc9dcdda8d08a09378a691b1R17">few</a> <a href="https://github.com/linkfluence/caffe-tensorflow/commit/4068ea31a1ff7dc91cd429e091e61b039448ce1b">other</a> tweaks as well.</p>
<h3>1 - Install caffe-tensorflow</h3>
<pre><code>git clone https://github.com/linkfluence/caffe-tensorflow
source activate Python27 # You need Python 2.7
</code></pre>
<h3>2 - (Optional) Switch to TensorFlow CPU</h3>
<p>You might bump into memory issues if you don't have enough memory. In this case just uninstall <code>tensorflow-gpu</code> and install <code>tensorflow</code></p>
<h3>3 - Convert your model</h3>
<pre><code>python convert.py --caffemodel ./places/vgg16_hybrid1365.caffemodel ./places/deploy_vgg16_hybrid1365.prototxt --data-output-path ./output.mat --code-output-path ./output2.py --standalone-output-path ./standalonehybrid.pb`
</code></pre>
<h3>4 - (Optional) Re-install TensorFlow GPU</h3>
<h2>Using the pb file</h2>
<p>If the the previous command succeeded, you'll end up with a <code>./standalonehybrid.pb</code> file. This file contains the weights and the architecture of the network. Here's how to use it:</p>
<pre><code>import tensorflow as tf
import cv2
import numpy as np

def load_graph(frozen_graph_filename):
    # We load the protobuf file from the disk and parse it to retrieve the
    # unserialized graph_def
    with tf.gfile.GFile(frozen_graph_filename, "rb") as f:
        graph_def = tf.GraphDef()
        graph_def.ParseFromString(f.read())

    # Then, we can use again a convenient built-in function to import a graph_def into the
    # current default Graph
    with tf.Graph().as_default() as graph:
        tf.import_graph_def(
            graph_def,
            input_map=None,
            return_elements=None,
            name="prefix",
            op_dict=None,
            producer_op_list=None
        )
    return graph

graph = load_graph('./standalonehybrid.pb')

x = graph.get_tensor_by_name('prefix/data:0')
y = graph.get_tensor_by_name('prefix/prob:0')


im = cv2.imread('./test_image.jpg')

WIDTH, HEIGHT = 224, 224
im = cv2.resize(im,  (WIDTH, HEIGHT))

# Places was using batches of 10 images
batch = np.array([im for i in range(10)])

with tf.Session(graph=graph) as sess:
    y_out = sess.run(y, feed_dict={ x: batch })
</code></pre>
<p><em>(the handy function comes from <a href="https://blog.metaflow.fr/tensorflow-how-to-freeze-a-model-and-serve-it-with-a-python-api-d4f3596b3adc">this blog post</a>)</em></p>
<p>Note: You'll see that we're building a batch of 10 images... with only one image. This is because the network is set like this and I want to keep this article simple. It should be possible to change the batch size. We'll see this in an upcoming blog post</p>
<h2>Conclusion</h2>
<p>We've seen how easy it is to use the models from the Caffe model zoo with TensorFlow. You are now able to classify cars, predict places, detect facial landmarks and so many magical things !</p>
<h2>Bonus: What if the model is based on a custom version of Caffe</h2>
<p>It can happen that researchers need custom layers: they usually fork caffe. In this case, <a href="https://gist.github.com/FranciscoCanas/844aab52fcff6b1c652a">this gist</a> describes the steps to extract the weights and <a href="https://github.com/machrisaa/tensorflow-vgg/blob/master/vgg16.py#L20">this piece of code</a> should give you some hints on how to load them into your TensorFlow graph.</p>
<p>For complex layers, there are some small differences between Caffe and TensorFlow: you will have to look at the source code. For instance, LSTM gates are not concatenated in the same order in TensorFlow and in Caffe.</p>
<p>What worked best for me was to:</p>
<ul>
<li>Export Caffe's weights into an Numpy matrix</li>
<li>Run a dummy example into the first N layers in Caffe, store the output</li>
<li>Load the weights into your TensorFlow Graph, run the same example into the same first N layers but using TensorFlow this time</li>
<li>Compare the output. If it's not matching, check what's wrong</li>
<li>Increment N and repeat</li>
</ul>
<p>I was able to get a 10<sup>-3</sup> mean difference in the final output when transfering a convnet and a 10<sup>-2</sup> mean difference after a bi-LSTM. Not too bad !</p>


In this tutorial, I'll show you how to convert the Places 365 model to TensorFlow

Converting a Caffe model to TensorFlow

<h1>About inference</h1>
<p>Using neural networks is primarily made of 2 phases: training your model and using it. The later part can also be called inference, forward pass or evaluation.</p>
<p>For most researchers, most of the time is used by training : they have to retrain using different architectures or different parameters. However, if you are using deep learning in production, inference is the most critical step.</p>
<h1>What is XLA</h1>
<p>During the TensorFlow summit 2017, the TensorFlow team introduced <a href="https://www.tensorflow.org/performance/xla/">XLA</a> (Accelerated Linear Algebra). It is an experimental framework aiming at optimizing TensorFlow graphs.
It should be able to improve memory usage, model footprint, portability and more importantly, execution speed. Using XLA can lead to improvement of up to 80% !</p>
<p>If you want to learn more about XLA, you should watch the summit's presentation
<a href="https://www.youtube.com/watch?v=kAOanJczHA0">here</a> or read about it <a href="https://www.tensorflow.org/performance/xla/">here</a>.</p>
<p>Basically, XLA is about optimizing your graph. One example might be that instead
of doing:</p>
<pre><code>a = b * c + g
d = b * c * e
</code></pre>
<p>It would be better to do:</p>
<pre><code>tmp = b * c
a = tmp + g
d = tmp * e
</code></pre>
<p>This is one optimisation example but XLA comes with many others</p>
<h1>Using XLA</h1>
<p>In this tutorial, I’ll show you how to use XLA.</p>
<p><strong>Important</strong>: At the time of writing, you have to compile TensorFlow with a special flag in order benefit from XLA. <a href="https://www.tensorflow.org/install/install_sources">Building TensorFlow</a> is fairly straightforward and takes about one hour on a regular machine.</p>
<p>Once built, enabling XLA for your graph is very simple:</p>
<pre><code>config = tf.ConfigProto()

config.graph_options.optimizer_options.global_jit_level = tf.OptimizerOptions.ON_1

sess = tf.Session(config=config)
</code></pre>
<h1>What are the performance gains ?</h1>
<p>I ran a small benchmark to test-out XLA's performance on a small RNN. I got a
~30% gain in speed on GPU! The code is available <a href="https://gist.github.com/EliotAndres/5497b763932f03dc46d3089e3b64c341">here</a>.</p>
<h1>Conclusion</h1>
<p>I should underline that your mileage may vary, XLA is not a magic perfomance flag.
But the TensorFlow team is constantly working on it so it is worth trying.</p>


In this blog post, we'll investigate how to make TensorFlow faster with XLA

Faster inference in TensorFlow using XLA.

<p>Jupyter Notebooks are pretty awesome. They allow you to prototype and experiment with ease. But did you know that you can also use them as a backend ?</p>
<p>In this tutorial, I'll show you how to make a simple api that converts degrees to radians. We should end up with something like this:</p>
<pre><code>$ curl "http://serverIp:8888/convert?angle=180"
#{"convertedAngle": 3.141592653589793}
</code></pre>
<h1>Installation</h1>
<p>Assuming that jupyter notebook is already installed, first install the kernel_gateway:</p>
<pre><code>pip install jupyter_kernel_gateway
jupyter kernelgateway --generate-config
</code></pre>
<p>If you want your api to be accessible from other computers, edit the <code>~/.jupyter/jupyter_kernel_gateway_config.py</code> file. Replace</p>
<pre><code># c.KernelGatewayApp.ip = '127.0.0.1'
</code></pre>
<p>with</p>
<pre><code>c.KernelGatewayApp.ip = '*'
</code></pre>
<h1>Creating the API</h1>
<p>Create a notebook with 2 cells. The first one:</p>
<pre><code>import math
import json
</code></pre>
<p>This cell will only be run once, when the notebook is loaded. You can perform any initialization you want here.</p>
<p>And the second one:</p>
<pre><code># GET /convert
req = json.loads(REQUEST)
args = req['args']

if 'angle' not in args:
  print(json.dumps({'convertedAngle': None}))
else:
  # Note the [0] when retrieving the argument.
  # This is because you could potentially pass multiple angles.
  angle = int(args['angle'][0])
  converted = math.radians(angle)
  print(json.dumps({'convertedAngle': converted}))
</code></pre>
<p>The first line is to indicate to the kernel gateway that we want to listen to the <code>/convert</code> route. This cell will be run on each call.</p>
<p>Finally, run your notebook:</p>
<pre><code>$ jupyter kernelgateway --KernelGatewayApp.api='kernel_gateway.notebook_http' --KernelGatewayApp.seed_uri='/home/username/Notebook.ipynb'
#[KernelGatewayApp] Kernel started: 72515fd8-6314-4d65-9f48-8214756850d7
#[KernelGatewayApp] Registering resource: /convert, methods: (['GET'])
#[KernelGatewayApp] Registering resource: /_api/spec/swagger.json, methods: (GET)
#[KernelGatewayApp] Jupyter Kernel Gateway at http://*:8888
</code></pre>
<p>And there you go, you have your api ! This can of course be used to serve predictions from a deep learning model.</p>


In this tutorial, I'll show you how to create a small api for your Jupyter Notebook

Eliot's blog archive