search.xml

<?xml version="1.0" encoding="utf-8"?>
<search> 
  
  
    
    <entry>
      <title>The 4 Hard Truths Data Science Blogs Don&#39;t Teach You About</title>
      <link href="2020/11/02/data-science-reality/"/>
      <url>2020/11/02/data-science-reality/</url>
      
        <content type="html"><![CDATA[<p><strong>I’m angry.</strong> Yes, I’m angry at <em>data science blogs</em>. Which is weird, because data science is blooming right now and we see a wide variety of knowledge being spread at every corner of the Internet. So, explaining why I am angry at them will be a tough one, but bear with me while I describe my journey through this industry.</p><h2 id="The-wrong-path-to-data-science"><a href="#The-wrong-path-to-data-science" class="headerlink" title="The wrong path to data science"></a>The wrong path to data science</h2><p>Let me give you some context first. A few years ago I was on the road to data science, I wanted to learn everything about this field, the sole idea of building something <em>intelligent</em> that can help someone predict something amazed me.</p><p>Inspired by this idea I decided I wanted to become a data scientist, and like many others, I jumped from engineering to this new landscape. Not knowing where to start I began searching through the Internet to see for myself how data science looks like.</p><p>Soon enough, I ended up in the vast sea of blogs, poured with hype and expectations, I was reading titles such as:</p><ul><li><strong>Data Science for Beginners: FULL COURSE!</strong></li><li><strong>Must-read books for Machine Learning and Data Science</strong></li><li><strong>How to speed up pandas with one line of code!</strong></li><li><strong>10 BEST machine learning courses</strong></li></ul><p>I was ready to read them all. I thought to myself:</p><blockquote><p>If I’m able to learn the best algorithm, if I build the best model to predict X thing, if I apply bleeding-edge techniques, I will surely stay ahead of the competition, I will be a <em>good</em> data scientist.</p></blockquote><p><strong>I was about to learn that <em>skill</em> only gets you so far.</strong></p><h2 id="Hard-truths-and-disappointments"><a href="#Hard-truths-and-disappointments" class="headerlink" title="Hard truths and disappointments"></a>Hard truths and disappointments</h2><p>After months of hard study, I was quickly looking for a job in this new and exciting field. I was pretty good at Pandas, and I was able to get my head around scikit-learn. Tensorflow? No problem.</p><p>I landed a job as a Data Analyst. I was lucky! And this company was great. I was hyped, I felt awesome, and I wanted to show what could I do. I wanted to help the company grow, and show them how can we apply data analysis and machine learning into their operations.</p><p>But in my small mind, <strong>I had no idea how utterly wrong I was,</strong> and let me tell you why.</p><h3 id="1-Data-doesn’t-appear-magically"><a href="#1-Data-doesn’t-appear-magically" class="headerlink" title="1. Data doesn’t appear magically"></a>1. Data doesn’t appear magically</h3><!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1551269901-5c5e14c25df7?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="person holding wand on top of bowl " title="person holding wand on top of bowl ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/3n7DdlkMfEg" target="_blank">Photo</a> by <a href="https://unsplash.com/@art_maltsev" target="_blank">@art_maltsev</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 02/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p>Guess what? <strong>Every blog out there will lay its analysis over the foundation that data is already there, waiting to be analyzed.</strong> They run over this assumption, and I, as a rookie, was lied by omission.</p><p>As a data analyst, I was tasked to analyze our sales, monthly revenue, cancellations, and everything that is without a doubt important for a SaaS company, and to get a dataset I had to connect to production servers, API’s, buckets, etc.</p><p>And you could say: <strong><em>Well, of course, that’s expected!</em></strong> The only problem is that <strong>programs do not generate datasets for human consumption.</strong></p><p>Most of the time you will have a SQL table in a production database ridden with a huge number of columns that you don’t even understand what they mean. Or JSON files that don’t even have a proper structure. Or incomplete datasets that I needed to join from multiple sources to have a working dataset.</p><p>Now, imagine doing that over, and over, and over.</p><blockquote><p>The first lesson I learned was that data doesn’t magically appear: Something has to generate it, and someone has to put it together.</p></blockquote><h3 id="2-Scalability-will-be-an-issue"><a href="#2-Scalability-will-be-an-issue" class="headerlink" title="2. Scalability will be an issue"></a>2. Scalability will be an issue</h3><!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1487837647815-bbc1f30cd0d2?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="white staircase with pink background " title="white staircase with pink background ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/w6OniVDCfn0" target="_blank">Photo</a> by <a href="https://unsplash.com/@maxon" target="_blank">@maxon</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 02/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p>I struggled to get data, but I finally knew my way around. I was already building and putting together datasets, and it was about time to fire up Jupyter and start tinkering with it.</p><p>My objective was straight, I wanted to know what are the reasons why people cancel their subscription with us. I started my EDA right away. Found some hard truths, cleared up assumptions and built a small model that can predict the probability for someone to churn.</p><p>It wasn’t the best model, but it was good enough, and I was proud of it. I presented my findings to the stakeholders and they were delighted. They now expect a report of the accounts that most likely are going to cancel every month in their email. The experiment was a success!</p><p><strong>Dear reader, did you just realized what I just said?</strong> If you haven’t managed a data science team before you will think there’s no issue here. However, for someone who has to deal with the coordination, capacity, and planning of such team you will soon realize that <em>this strategy is not scalable.</em></p><p>You’re having a data analyst (or a data scientist), extracting data by himself, running a report locally, on a Jupyter notebook who only works in his computer, manually delivering a report to stakeholders. <strong>If you don’t think this is a recipe for a disaster then I invite you to reconsider your strategy to build scalable teams.</strong></p><p>Moreover, with this approach you’re going to burn out, stakeholders will depend on your ability to send them the report on time, and <em>you’re teaching the company that they don’t need to learn about data, <strong>they have you.</strong></em></p><blockquote><p>Soon enough, I learned my second lesson the hard way: Data Science is nothing without the architecture to support it.</p></blockquote><h3 id="3-Models-with-no-business-case-are-useless"><a href="#3-Models-with-no-business-case-are-useless" class="headerlink" title="3. Models with no business case are useless"></a>3. Models with no business case are useless</h3><!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1581291518857-4e27b48ff24e?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="person writing on white paper " title="person writing on white paper ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/v9FQR4tbIq8" target="_blank">Photo</a> by <a href="https://unsplash.com/@kellysikkema" target="_blank">@kellysikkema</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 02/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p>Even though our processes were not scalable, we kept going, and we were developing model after model, even the same models with different algorithms. We just discovered AI, and we wanted to make it ours!</p><p>After all those models I built, I realized that people were asking me for things that seemed to tackle no business problem. Predict revenue? Ok. Cancellations? Here you go. Forecast new accounts? No problem.</p><p>Now, let me ask some harsh questions to my past self:</p><ol><li><em>Why someone wanted me to predict the revenue? Was there a plan to execute if our prediction was that we were about to hit a bad month?</em></li><li><em>We have a model to predict churn. Do we have an action plan for users at risk?</em></li><li><em>Why do you want to forecast new accounts? Is there any reason to do it at all?</em></li></ol><p>If you layout a myriad of models out there without any business objective, with no execution plan, to only please stakeholders’ wonder and amazement, then let me politely tell you that <strong>you’re providing nothing of value.</strong></p><p>Having a business objective, and a execution plan, is <em>paramount</em> to building successful AI products that are going to change the way you do business. Let’s go even further, <strong>your responsibility as data scientist is to educate your stakeholders!</strong> They trust your expertise and field knowledge to guide them through this AI revolution.</p><p>Have them prepare a business case, ask them difficult questions and execution plans, prepare to mitigate failure, clear assumptions around the business risks involved. You’ll provide a more clear agenda, better models, and you’ll bring more value to the company.</p><blockquote><p>Yet again, I learned another lesson: To build without a plan is to build nothing at all.</p></blockquote><h3 id="4-Data-Science-is-not-exclusive-for-a-team"><a href="#4-Data-Science-is-not-exclusive-for-a-team" class="headerlink" title="4. Data Science is not exclusive for a team"></a>4. Data Science is not exclusive for a team</h3><!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1551288049-bebda4e38f71?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="graphs of performance analytics on a laptop screen " title="graphs of performance analytics on a laptop screen ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/JKUTrJ4vK00" target="_blank">Photo</a> by <a href="https://unsplash.com/@lukechesser" target="_blank">@lukechesser</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 02/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p><strong>We are at the Fourth Industrial Revolution, and data is at the front-line.</strong></p><p>That is something that I bet most people don’t understand. Data has such a breakthrough, that it transformed businesses in its <em>entirety!</em></p><p>Remember how important it was to know how to use a computer? I still remember that having a Microsoft Office learning certificate guarantee you a job somewhere. People were replaced one-by-one by the newer generations who were more adept at computers.</p><p>Years later that was not a qualification requirement, it is an <em>expectation.</em> Companies now <em>expect</em> you to know how to use a computer, they <em>expect</em> you to know how to use Excel, they <em>expect</em> you to be able to browse the web without an issue.</p><p>If you think that data is going to be different, then I beg you to reconsider your priorities. We see companies investing in <em>data democratization</em> like there’s no tomorrow. Teaching their employees how to handle data, interpret it, and use it to enhance their operations.</p><p><strong>They have seen the value that data brings to the business.</strong> They <em>know</em> how important it is to make informed decisions, and develop strategies around data is now becoming <em>the norm.</em></p><p>If you think Data Science is going to be reserved for teams who know how to handle data, then I’m afraid you’re wrong. If you want to succeed in your business, whether you are a data analyst, or a Chief Data Officer of an organization, then <strong>you have to push for data democratization.</strong></p><blockquote><p>And with this I learned a valuable lesson: If you don’t invest data democratization, your strategies will be superseded for a company that does it.</p></blockquote><h2 id="Conclusion-and-recommendations"><a href="#Conclusion-and-recommendations" class="headerlink" title="Conclusion and recommendations"></a>Conclusion and recommendations</h2><p>If you see yourself in one of these points, then let me give you some recommendations:</p><ol><li><strong>Hire a data engineer first:</strong> This person will lay foundations for analysts and data scientists to scale their operations with ease. It’s one of the best single decisions you can make.</li><li><strong>Layout your data strategy:</strong> Think about your data strategy, try to find loopholes in it. Think about how you are going to move data (Apache Airflow?), how are you going to analyze data (Deepnote?), how are you going to deploy products (MLFlow?)</li><li><strong>Build with a business case:</strong> Make a quick checklist with objectives, risks involved, mitigation plans, what-if’s, and so on.</li><li><strong>Prepare to scale data:</strong> Invest in people education. Teach them Python or R, SQL is a must these days. These are people who are going to be at the front of your business. Maybe they are in marketing, sales, or other operations, but they <em>need</em> to read and manipulate data.</li></ol><p>I hope you liked the entry. <strong>Follow me on Twitter</strong> if you want to read more entries like this 👉🏻 <a href="https://twitter.com/__franccesco?ref_src=twsrc%5Etfw" target="_blank" rel="noopener" class="twitter-follow-button" data-show-count="false">Follow @__franccesco</a><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></p><p>Also, share this article if you found it interesting. See you soon.</p>]]></content>
      
      
      
        <tags>
            
            <tag> data science </tag>
            
            <tag> opinion </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Monitoring Nginx With @Sherlog/cli</title>
      <link href="2020/10/07/monitoring-nginx-with-sherlog/"/>
      <url>2020/10/07/monitoring-nginx-with-sherlog/</url>
      
        <content type="html"><![CDATA[<p>Logs play a very important role throughout the entire life cycle of an application development as well as troubleshooting and replicating bugs on production that could lead to service interruption and harm our user’s experience.</p><p>A few months ago, I went on a journey for finding a tool that will allow me to improve logs visibility and to take action as quickly as possible, and of course with a minimum amount of effort and server requirements. I found many of them, the vast majority very appealing with endless features to the point where it started feeling somewhat overwhelmed. None of these tools however, were easy to set up and they all required a learning curve to take advantage of it’s full potential. Not to mention, the majority weren’t free and pricing will range depending on the retention period, number of instances, license, etc.</p><p>My goal was to simply run a command, get what I needed and continue with my life. All in sudden, I was hit with the <strong>aha moment!</strong> Why don’t I write a simple tool that attempts to solve the problem? So here I am, a few months after, sharing my approach at tackling this problem.</p><p><a href="https://www.npmjs.com/package/@sherlog/cli" target="_blank" rel="noopener">@sherlog/cli</a> requires <code>node &gt;= 12</code>.</p><p>For this example, I’m using <a href="https://github.com/nvm-sh/nvm" target="_blank" rel="noopener">nvm</a> to install the minimum required version</p><p>Run the following commands in your terminal:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ nvm install v12.16.1</span><br><span class="line">$ npm install -g @sherlog/cli</span><br></pre></td></tr></table></figure><p>Initialize the project</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sherlog init</span><br></pre></td></tr></table></figure><p>The previous command generates a <code>.sherlog</code> config file in your current working directory (no biggie, just another JSON). Fill in the blanks. Once configured, it should look similar to this.</p><figure class="highlight json"><table><tr><td class="code"><pre><span class="line">&#123;</span><br><span class="line">  <span class="attr">"hostname"</span>: <span class="string">"192.168.10.108"</span>,</span><br><span class="line">  <span class="attr">"backpressure"</span>: <span class="number">1000</span>,</span><br><span class="line">  <span class="attr">"chunks"</span>: <span class="number">500</span>,</span><br><span class="line">  <span class="attr">"compression"</span>: <span class="literal">true</span>,</span><br><span class="line">  <span class="attr">"files"</span>: [&#123;</span><br><span class="line">    <span class="attr">"metric"</span>: <span class="string">"nginx"</span>,</span><br><span class="line">    <span class="attr">"file"</span>: <span class="string">"/var/log/nginx/access.log"</span>,</span><br><span class="line">    <span class="attr">"eventType"</span>: <span class="string">"http"</span>,</span><br><span class="line">    <span class="attr">"timezone"</span>: <span class="string">"UTC"</span>,</span><br><span class="line">    <span class="attr">"fromBeginning"</span>: <span class="literal">true</span></span><br><span class="line">  &#125;]</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>This file can be committed to your repository to speed up the process next time you need to check your logs on different environments. Lets go ahead and start the service.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">$ sherlog start</span><br></pre></td></tr></table></figure><p>This will output the following in your terminal</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Sherlog listening on:</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">   - Dashboard:   http:&#x2F;&#x2F;localhost:8000</span><br><span class="line">   - Local:       ws:&#x2F;&#x2F;localhost:8000</span><br><span class="line">   - Network:     ws:&#x2F;&#x2F;192.168.10.108:8000</span><br></pre></td></tr></table></figure><p>Navigate to the dashboard url <a href="http://localhost:8000/" target="_blank" rel="noopener">http://localhost:8000</a></p><p><img src="https://dev-to-uploads.s3.amazonaws.com/i/mcz73m9ors0wmzqkw3wk.png" alt="Alt Text"></p><p>Now you can navigate through your logs as if you were watching a YouTube video going back and forth in case you missed something.</p><p><a href="https://www.npmjs.com/package/@sherlog/cli" target="_blank" rel="noopener">@sherlog/cli</a> supports the following default log formats out of the box:</p><ul><li><strong>Apache2</strong><ul><li>HTTP</li><li>Error</li></ul></li><li><strong>Monolog</strong> (e.g. Laravel)</li><li><strong>Mysql</strong><ul><li>General</li></ul></li><li><strong>Nginx</strong><ul><li>HTTP</li><li>Error</li></ul></li><li><strong>PHP-fpm</strong></li><li><strong>Redis</strong></li></ul><p>That’s it for now folks, if you wish to get updates about <strong>@sherlog/cli</strong> and possible use cases you can follow me on Twitter <a href="https://twitter.com/Brucelampson" target="_blank" rel="noopener">@Brucelampson</a> or feel free to submit a pull request to the project on GitHub <a href="https://github.com/sherl0g/cli" target="_blank" rel="noopener">sherl0g</a></p>]]></content>
      
      
      
        <tags>
            
            <tag> guest </tag>
            
            <tag> project </tag>
            
            <tag> javascript </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Python Logging Basics: Why Is It Important and How to Use It?</title>
      <link href="2020/06/27/Logging-Events-in-Python/"/>
      <url>2020/06/27/Logging-Events-in-Python/</url>
      
        <content type="html"><![CDATA[<!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1526708286628-e9e0b57f84fa?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="gray and black film projector " title="gray and black film projector ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/mwWZTLr9Tcg" target="_blank">Photo</a> by <a href="https://unsplash.com/@ingoschulz" target="_blank">@ingoschulz</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 01/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p>If you find yourself having troubles debugging your code, or wondering what went wrong, then you should start logging events in your python code.</p><p>Using the <code>logging</code> library you can basically record what actions is your code doing, i.e making a web request, reading a file, monitoring something, etc. It can help you to narrow down your faulty code for debugging.</p><p>Moreover, logging is not only helpful for debugging, but it is also helpful for collaboration, and many platforms use the logging module in your code so you navigate between events easily. It will not only help you on your projects, but also in professional environments.</p><h2 id="The-logging-module"><a href="#The-logging-module" class="headerlink" title="The logging module"></a>The logging module</h2><p>You can start by importing the <code>logging</code> library in your python shell and after importing it you can log different event levels such as <code>INFO</code>, <code>WARNING</code> and <code>ERROR</code>.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">In [<span class="number">1</span>]: <span class="keyword">import</span> logging</span><br><span class="line"></span><br><span class="line">In [<span class="number">2</span>]: logging.info(<span class="string">'Hello'</span>)</span><br><span class="line"></span><br><span class="line">In [<span class="number">3</span>]: logging.warning(<span class="string">'Something might happen...'</span>)</span><br><span class="line">WARNING:root:Something might happen...</span><br><span class="line"></span><br><span class="line">In [<span class="number">4</span>]: logging.error(<span class="string">'Something BAD happened!'</span>)</span><br><span class="line">ERROR:root:Something BAD happened!</span><br></pre></td></tr></table></figure><p>I assume you noticed that our <code>info</code> event wasn’t logged. This is because the default logging level is set to record events equals to warnings and above. There are 4 types of levels you should know about.</p><table><thead><tr><th>Level</th><th>Attribute</th><th>Code</th></tr></thead><tbody><tr><td>Debug</td><td>logging.DEBUG</td><td>10</td></tr><tr><td>Info</td><td>logging.INFO</td><td>20</td></tr><tr><td>Warning</td><td>logging.WARNING</td><td>30</td></tr><tr><td>Error</td><td>logging.ERROR</td><td>40</td></tr><tr><td>Critical</td><td>logging.CRITICAL</td><td>50</td></tr><tr><td>Fatal</td><td>logging.FATAL</td><td>50</td></tr></tbody></table><p>We can allow the logging module to log <code>info</code> records using the <code>setLevel()</code> function.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">In [<span class="number">9</span>]: logging.getLogger().setLevel(logging.INFO)</span><br><span class="line">In [<span class="number">10</span>]: logging.info(<span class="string">'info!'</span>)</span><br><span class="line">INFO:root:info!</span><br></pre></td></tr></table></figure><p>However, what we should be doing, is to instantiate a <em>logger</em> class. This will allow us to set up the format, a handler and the logging level without having to call it from the main module.</p><blockquote><p><em>Loggers have the following attributes and methods.  Note that Loggers should NEVER be instantiated directly, but always through the module-level function <code>logging.getLogger(name)</code>.  Multiple calls to <code>getLogger()</code> with the same name will always return a reference to the same Logger object.</em></p><p><em>— Python 3 Docs</em></p></blockquote><h2 id="Loading-a-logger"><a href="#Loading-a-logger" class="headerlink" title="Loading a logger"></a>Loading a logger</h2><p>We can instantiate logger using the method <code>getLogger()</code>, we just have to provide a name for that logger.</p><p>Usually what you will want to do is to provide a name to the <code>getLogger()</code> function, this name can be the special variable <code>__name__</code>. This way you can identify from which module is the log coming from.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">In [<span class="number">1</span>]: <span class="keyword">import</span> logging</span><br><span class="line">In [<span class="number">2</span>]: logger = logging.getLogger(__name__)</span><br></pre></td></tr></table></figure><p>After you have instantiated your logger class, you have to define a handler. Now, a handler is what tells the logger where it should store the logs. The two most common options are the <code>FileHandler</code> and the <code>StreamHandler</code>, but there are <a href="https://docs.python.org/3.8/library/logging.handlers.html#module-logging.handlers" target="_blank" rel="noopener">other options</a> as well.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">In [<span class="number">1</span>]: <span class="keyword">import</span> logging</span><br><span class="line"></span><br><span class="line">In [<span class="number">2</span>]: logger = logging.getLogger(__name__)</span><br><span class="line">In [<span class="number">3</span>]: logger.setLevel(logging.INFO)</span><br><span class="line"></span><br><span class="line">In [<span class="number">4</span>]: stream_handler = logging.StreamHandler()</span><br><span class="line">In [<span class="number">5</span>]: logger.addHandler(stream_handler)</span><br><span class="line"></span><br><span class="line">In [<span class="number">6</span>]: logger.info(<span class="string">'I\'m displaying info!'</span>)</span><br><span class="line">I<span class="string">'m displaying info!</span></span><br></pre></td></tr></table></figure><h2 id="Changing-logger-format"><a href="#Changing-logger-format" class="headerlink" title="Changing logger format"></a>Changing logger format</h2><p>However, I would like to add more formatting to the logs,  in this case, a time, a level name, and a message will suffice. Luckily, this is fairly easy using the <code>Formatter</code> class.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">In [<span class="number">7</span>]: log_format = logging.Formatter(<span class="string">"%(asctime)s - %(levelname)s: %(message)s"</span>)</span><br><span class="line"></span><br><span class="line">In [<span class="number">8</span>]: stream_handler.setFormatter(log_format)</span><br><span class="line"></span><br><span class="line">In [<span class="number">9</span>]: logger.info(<span class="string">'I\'m displaying info!'</span>)</span><br><span class="line"><span class="number">2020</span><span class="number">-05</span><span class="number">-25</span> <span class="number">20</span>:<span class="number">18</span>:<span class="number">25</span>,<span class="number">322</span> - INFO: I<span class="string">'m displaying info!</span></span><br></pre></td></tr></table></figure><p>You can see a list of attribute names in the python documentation <a href="https://docs.python.org/3.8/library/logging.html#logrecord-attributes" target="_blank" rel="noopener">here</a>.</p><h2 id="Putting-it-all-together"><a href="#Putting-it-all-together" class="headerlink" title="Putting it all together"></a>Putting it all together</h2><p>Let’s gather our thoughts and put this scenario. Let’s say we have a <code>TXT</code> file in which we have some Urls and we have build a script that goes through each one of them to test if they are up and running, throwing error codes (HTTP 5xx) or maybe completely down.</p><p>We also want to log everything into a file, more specifically <code>INFO</code> levels and up, but we also want to log only <code>WARNING</code> events into our terminal so we don’t fill it up with information we don’t care about.</p><p>To do this we are going to split our script into 2 modules. Is not necessary but I don’t want to cramp it up everything into a single file. Let’s first create a <code>TXT</code> file and fill it out with Urls:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">https:&#x2F;&#x2F;httpstat.us&#x2F;</span><br><span class="line">https:&#x2F;&#x2F;httpstat.us&#x2F;500</span><br><span class="line">https:&#x2F;&#x2F;httpstat.us&#x2F;400</span><br><span class="line">https:&#x2F;&#x2F;localhost&#x2F;</span><br></pre></td></tr></table></figure><p>Now, let’s create a python script and call it <code>load_logger.py</code>. Here we’ll define our logger instance, the formatter, and handlers appended to it with their respective logging level.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> logging</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">load_logger</span><span class="params">()</span> -&gt; logging.Logger:</span></span><br><span class="line">    <span class="string">"""Return a logger instance."""</span></span><br><span class="line"></span><br><span class="line">    logger = logging.getLogger(__name__)</span><br><span class="line">    logger.setLevel(logging.INFO)</span><br><span class="line"></span><br><span class="line">    file_handler = logging.FileHandler(<span class="string">"status.log"</span>)</span><br><span class="line">    stream_handler = logging.StreamHandler()</span><br><span class="line"></span><br><span class="line">    file_handler.setLevel(logging.INFO)</span><br><span class="line">    stream_handler.setLevel(logging.WARNING)</span><br><span class="line"></span><br><span class="line">    log_format = logging.Formatter(<span class="string">"%(asctime)s - %(levelname)s: %(message)s"</span>)</span><br><span class="line"></span><br><span class="line">    file_handler.setFormatter(log_format)</span><br><span class="line">    stream_handler.setFormatter(log_format)</span><br><span class="line"></span><br><span class="line">    logger.addHandler(file_handler)</span><br><span class="line">    logger.addHandler(stream_handler)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> logger</span><br></pre></td></tr></table></figure><p>Now that we have defined our logger, let’s create our main script <code>site_monitor.py</code>. What this file will do is:</p><ol><li>Load the event logger</li><li>Read our file domains.txt</li><li>Test the status code of each destination</li><li>Log the event of the status code</li></ol><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> logging</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> sys <span class="keyword">import</span> argv</span><br><span class="line"><span class="keyword">from</span> sys <span class="keyword">import</span> exit</span><br><span class="line"><span class="keyword">from</span> load_logger <span class="keyword">import</span> load_logger</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(domains_file: str)</span> -&gt; <span class="keyword">None</span>:</span></span><br><span class="line">    <span class="string">"""Check for a number of HTTP codes and log them."""</span></span><br><span class="line"></span><br><span class="line">    logger = load_logger()</span><br><span class="line"></span><br><span class="line">    <span class="keyword">with</span> open(domains_file, <span class="string">"r"</span>) <span class="keyword">as</span> file:</span><br><span class="line">        logger.info(<span class="string">f"Reading file: <span class="subst">&#123;domains_file&#125;</span>"</span>)</span><br><span class="line">        domains = file.read().splitlines()</span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> url <span class="keyword">in</span> domains:</span><br><span class="line">        logger.info(<span class="string">f"Sending get request to: <span class="subst">&#123;url&#125;</span>"</span>)</span><br><span class="line"></span><br><span class="line">        <span class="keyword">try</span>:</span><br><span class="line">            req = requests.get(url)</span><br><span class="line">        <span class="keyword">except</span> Exception <span class="keyword">as</span> e:</span><br><span class="line">            logger.error(<span class="string">f"Destination unreachable for url: <span class="subst">&#123;url&#125;</span>"</span>)</span><br><span class="line">            exit(<span class="number">1</span>)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            <span class="keyword">if</span> req.ok:</span><br><span class="line">                logger.info(<span class="string">f"Status OK for url: <span class="subst">&#123;url&#125;</span>"</span>)</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                logger.warning(<span class="string">f"Received status code: <span class="subst">&#123;req.status_code&#125;</span>"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</span><br><span class="line">    main(argv[<span class="number">1</span>])</span><br></pre></td></tr></table></figure><p>To run our script you can call it using the command line:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ python site_monitor.py domains.txt</span><br></pre></td></tr></table></figure><p>After that you will see that only <code>WARNING</code> events are being recorded in the console, but in our <code>status.log</code> file you will see <code>INFO</code> logs and up.</p><p>In this example, I’m executing code and I’m using the command <code>tail</code> with the flag <code>-f</code> to see what’s being recorded in that file.</p><p><video src="/assets/images/logging/logging_tail.webm" preload autoplay loop controls></video></p><p>As you can see, in our example we only log <code>WARNING</code> events in our terminal, but we are able to dump everything in our <code>status.log</code> file.</p><p>If you want to take a closer look at the code, I’ve set up a sample repository at: <a href="https://github.com/franccesco/status_monitor_example" target="_blank" rel="noopener">https://github.com/franccesco/status_monitor_example</a></p><p>Feel free to reach out if you have any doubts.</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> logging </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>How to Save Dictionaries, Lists, Tuples and Other Objects With Pickle</title>
      <link href="2019/06/26/Save-your-dictionaries-lists-tuples-and-other-objects-with-Pickle/"/>
      <url>2019/06/26/Save-your-dictionaries-lists-tuples-and-other-objects-with-Pickle/</url>
      
        <content type="html"><![CDATA[<p><img src="/assets/images/pickle/carbon.png"></p><p>Sometimes you just want to save a dictionary, a list, a string, or anything into a file so you can use it later. This can be easily achieved with the module <strong>Pickle.</strong></p><blockquote><p><em><strong>Warning:</strong></em><br>The pickle module is not secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source. <strong><em>— <a href="https://docs.python.org/3.7/library/pickle.html" target="_blank" rel="noopener">Pickle Documentation</a></em></strong></p></blockquote><h2 id="What-is-Pickle"><a href="#What-is-Pickle" class="headerlink" title="What is Pickle"></a>What is Pickle</h2><p>Pickle is a module in Python that can be implemented to serialize or de-serialize a Python object, meaning that it can be used to save an <strong>object</strong> into a <strong>file</strong>; Just have in mind that this is not the same as saving a <strong>configuration</strong> file, there are other data structures that we can use to achieve that task such as JSON, CSV, YAML/TOML, etc.</p><h2 id="How-to-save-an-dictionary-with-Pickle"><a href="#How-to-save-an-dictionary-with-Pickle" class="headerlink" title="How to save an dictionary with Pickle"></a>How to save an dictionary with Pickle</h2><p>Saving an object with Pickle is really easy, all you need to do is to save the object into a file providing the <strong>object</strong> and the <strong>filename</strong>, here’s a quick snippet:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> pickle</span><br><span class="line"></span><br><span class="line">dictionary = &#123;<span class="string">'string_1'</span>: <span class="number">1</span>, <span class="string">'string_2'</span>: <span class="number">2.2</span>, <span class="string">'string_3'</span>: <span class="literal">True</span>&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment"># Pickling (serializing) a dictionary into a file</span></span><br><span class="line"><span class="keyword">with</span> open(<span class="string">'saved_object.pickle'</span>, <span class="string">'wb'</span>) <span class="keyword">as</span> filename:</span><br><span class="line">    pickle.dump(dictionary, filename)</span><br></pre></td></tr></table></figure><p>And there you go! You have saved the contents of an object into a file. Just remember to always save the file as <em>binary</em> providing the arguments <code>w</code> (<em>writing</em>) and <code>b</code> (<em>binary</em>)</p><h2 id="Loading-a-dictionary"><a href="#Loading-a-dictionary" class="headerlink" title="Loading a dictionary"></a>Loading a dictionary</h2><p>If you have used JSON before then you can find the syntax to be very familiar. We can load a previously pickled object using the <code>.load()</code> method and providing a filename:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> pickle</span><br><span class="line"></span><br><span class="line"><span class="comment"># Unpickling (de-serializing) a dictionary</span></span><br><span class="line"><span class="keyword">with</span> open(<span class="string">'saved_object.pickle'</span>, <span class="string">'rb'</span>) <span class="keyword">as</span> filename:</span><br><span class="line">    dictionary = pickle.load(filename)</span><br><span class="line"></span><br><span class="line">print(dictionary)</span><br><span class="line"></span><br><span class="line"><span class="comment"># &gt;&gt;&gt; &#123;'string_1': 1, 'string_2': 2.2, 'string_3': True&#125;</span></span><br></pre></td></tr></table></figure><p>As you can see, our dictionary object was loaded correctly. To double-check this, here’s a script that compares if the contents of a dictionary and a saved pickle are the same:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> pickle</span><br><span class="line"></span><br><span class="line">dictionary_a = &#123;<span class="string">'string_1'</span>: <span class="number">1</span>, <span class="string">'string_2'</span>: <span class="number">2.2</span>, <span class="string">'string_3'</span>: <span class="literal">True</span>&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment"># Pickling (serializing) dictionary A into a file</span></span><br><span class="line"><span class="keyword">with</span> open(<span class="string">'saved_object.pickle'</span>, <span class="string">'wb'</span>) <span class="keyword">as</span> filename:</span><br><span class="line">    pickle.dump(dictionary_a, filename)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Unpickling (de-serializing) dictionary A into B</span></span><br><span class="line"><span class="keyword">with</span> open(<span class="string">'saved_object.pickle'</span>, <span class="string">'rb'</span>) <span class="keyword">as</span> filename:</span><br><span class="line">    dictionary_b = pickle.load(filename)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Dictionaries A and B remaing the same</span></span><br><span class="line">print(<span class="string">f'Is dictionary_a == dictionary_b?: <span class="subst">&#123;dictionary_a == dictionary_b&#125;</span>'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># &gt;&gt;&gt; Is dictionary_a == dictionary_b?: True</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Dictionaries have the same content</span></span><br><span class="line">print(<span class="string">f'Dictionary A: <span class="subst">&#123;dictionary_a&#125;</span>'</span>)</span><br><span class="line">print(<span class="string">f'Dictionary B: <span class="subst">&#123;dictionary_b&#125;</span>'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># &gt;&gt;&gt; Dictionary A: &#123;'string_1': 1, 'string_2': 2.2, 'string_3': True&#125;</span></span><br><span class="line"><span class="comment"># &gt;&gt;&gt; Dictionary B: &#123;'string_1': 1, 'string_2': 2.2, 'string_3': True&#125;</span></span><br></pre></td></tr></table></figure><h2 id="What-kind-of-data-can-be-Pickled"><a href="#What-kind-of-data-can-be-Pickled" class="headerlink" title="What kind of data can be Pickled?"></a>What kind of data can be Pickled?</h2><p>According to the official documentation, this is the list of object types that can be pickle/unpickled:</p><ul><li><code>None</code>, <code>True</code>, and <code>False</code></li><li>integers, floating point numbers, complex numbers</li><li>strings, bytes, bytearrays</li><li>tuples, lists, sets, and dictionaries containing only picklable objects</li><li>functions defined at the top level of a module (using def, not lambda)</li><li>built-in functions defined at the top level of a module</li><li>classes that are defined at the top level of a module<br>instances of such classes whose <strong>dict</strong> or the result of calling <strong>getstate</strong>() is picklable (see section Pickling Class Instances for * details).</li></ul><p>This is a very interesting approach if you need to save an object into a file. Just beware that there’s a lot of data structures out there than can help you to save a <em>configuration</em> file. But if you need to save an <strong>object</strong>, then this seems the right choice!</p><h2 id="Further-Reading"><a href="#Further-Reading" class="headerlink" title="Further Reading"></a>Further Reading</h2><ul><li><a href="https://docs.python.org/3.7/library/pickle.html" target="_blank" rel="noopener">Pickle — Python object serialization</a></li></ul>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Develop and Publish Your Python Packages With Poetry</title>
      <link href="2019/06/16/develop-and-publish-with-poetry/"/>
      <url>2019/06/16/develop-and-publish-with-poetry/</url>
      
        <content type="html"><![CDATA[<!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1473186505569-9c61870c11f9?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="black and silver fountain pen " title="black and silver fountain pen ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/hjwKMkehBco" target="_blank">Photo</a> by <a href="https://unsplash.com/@alvaroserrano" target="_blank">@alvaroserrano</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 01/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p>I’ve been trying to publish my packages to <a href="http://pypi.org/" target="_blank" rel="noopener">PyPi</a> so people can access my software more easily. But I have to be honest, Python’s publishing system is not the best out there and it has to improve quite a lot.</p><p>Wandering around I stumbled upon <a href="https://poetry.eustace.io/" target="_blank" rel="noopener"><strong>Poetry</strong></a>, which is a python packager and dependency manager created by <em>Sébastien Eustace</em> for people that doesn’t want to lose their head managing a Python project.</p><p>Let’s say that we want to make a small command line application that checks the status of a web page, let’s call it <strong>checkstat</strong>. So how can we <strong>create</strong>, <strong>develop</strong>, and <strong>publish</strong> our software with Poetry?</p><hr><blockquote><p>Courtesy Notice:<br>This post has been revisited and updated on <em>2019-06-16 11:20:44</em>.</p></blockquote><hr><h2 id="Installing-Poetry"><a href="#Installing-Poetry" class="headerlink" title="Installing Poetry"></a>Installing Poetry</h2><p>You can install poetry effortlessly with a python script provided by the creator.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python</span><br></pre></td></tr></table></figure><p>Now we are able to see Poetry’s options.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ poetry</span><br><span class="line">Poetry 0.12.14</span><br><span class="line"></span><br><span class="line">Usage:</span><br><span class="line">  <span class="built_in">command</span> [options] [arguments]</span><br><span class="line"></span><br><span class="line">Options:</span><br><span class="line">  -h, --<span class="built_in">help</span>                      Display this <span class="built_in">help</span> message</span><br><span class="line">  -q, --quiet                     Do not output any message</span><br><span class="line">  -V, --version                   Display this application version</span><br><span class="line">      --ansi                      Force ANSI output</span><br><span class="line">      --no-ansi                   Disable ANSI output</span><br><span class="line">  -n, --no-interaction            Do not ask any interactive question</span><br><span class="line">  -v|vv|vvv, --verbose[=VERBOSE]  Increase the verbosity of messages: 1 <span class="keyword">for</span> normal output, 2 <span class="keyword">for</span> more verbose output and 3 <span class="keyword">for</span> debug</span><br><span class="line"></span><br><span class="line">Available commands:</span><br><span class="line">  about          Short information about Poetry.</span><br><span class="line">  add            Add a new dependency to pyproject.toml.</span><br><span class="line">  build          Builds a package, as a tarball and a wheel by default.</span><br><span class="line">  check          Checks the validity of the pyproject.toml file.</span><br><span class="line">  config         Sets/Gets config options.</span><br><span class="line"></span><br><span class="line">--- SNIP ---</span><br><span class="line"></span><br><span class="line"> debug</span><br><span class="line">  debug:info     Shows debug information.</span><br><span class="line">  debug:resolve  Debugs dependency resolution.</span><br><span class="line"> self</span><br><span class="line">  self:update    Updates poetry to the latest version.</span><br></pre></td></tr></table></figure><h2 id="Creating-our-package-and-adding-dependencies"><a href="#Creating-our-package-and-adding-dependencies" class="headerlink" title="Creating our package and adding dependencies"></a>Creating our package and adding dependencies</h2><p>First we have to create our package with the <code>new</code> sub-command, this will create a file structure in which we can work on.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Create a new package</span></span><br><span class="line">$ poetry new checkstat</span><br><span class="line">Created package checkstat <span class="keyword">in</span> checkstat</span><br><span class="line"></span><br><span class="line"><span class="comment"># Enter the new package directory</span></span><br><span class="line">$ <span class="built_in">cd</span> checkstat</span><br><span class="line"></span><br><span class="line"><span class="comment"># Prints the directory tree</span></span><br><span class="line">$ tree</span><br><span class="line">.</span><br><span class="line">├── checkstat</span><br><span class="line">│   └── __init__.py</span><br><span class="line">├── pyproject.toml</span><br><span class="line">├── README.rst</span><br><span class="line">└── tests</span><br><span class="line">    ├── __init__.py</span><br><span class="line">    └── test_checkstat.py</span><br><span class="line"></span><br><span class="line">2 directories, 5 files</span><br></pre></td></tr></table></figure><p>As we can see, we only need the sub-command <code>new</code> followed by a name argument to create a directory structure for our little project. This will also generate a configuration file for our project called <a href="https://poetry.eustace.io/docs/pyproject/" target="_blank" rel="noopener"><strong><code>pyproject.toml</code></strong></a> which is a much better way to define your project configuration than <strong><code>setup.py</code></strong>, specially for people who are new to python, you can <a href="https://www.python.org/dev/peps/pep-0518/" target="_blank" rel="noopener">read more about this implementation here</a>.</p><figure class="highlight toml"><table><tr><td class="code"><pre><span class="line"><span class="section">[tool.poetry]</span></span><br><span class="line"><span class="attr">name</span> = <span class="string">"checkstat"</span></span><br><span class="line"><span class="attr">version</span> = <span class="string">"0.1.0"</span></span><br><span class="line"><span class="attr">description</span> = <span class="string">""</span></span><br><span class="line"><span class="attr">authors</span> = [<span class="string">"Franccesco Orozco &lt;franccesco@codingdose.info&gt;"</span>]</span><br><span class="line"></span><br><span class="line"><span class="section">[tool.poetry.dependencies]</span></span><br><span class="line"><span class="attr">python</span> = <span class="string">"^3.7"</span></span><br><span class="line"></span><br><span class="line"><span class="section">[tool.poetry.dev-dependencies]</span></span><br><span class="line"><span class="attr">pytest</span> = <span class="string">"^3.0"</span></span><br></pre></td></tr></table></figure><p>As you can see, it describes basic information about the project, dependencies and development dependencies. We will get back to the TOML file shortly.</p><p>What we want to do next is to create the virtual environment of our project and install the development dependencies with the command <code>install</code>.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Create package isolated virtualenv and install dependencies</span></span><br><span class="line">$ poetry install</span><br></pre></td></tr></table></figure><p><video src="/assets/images/poetry/poetry-install.webm" preload autoplay loop controls></video></p><p>Now that we have installed the requirements and initiated the virtual environment let’s add more libraries to our project, shall we? Let’s add <a href="http://click.pocoo.org/" target="_blank" rel="noopener">Click</a>, <a href="https://github.com/tartley/colorama" target="_blank" rel="noopener">Colorama</a> and <a href="http://docs.python-requests.org/en/master/" target="_blank" rel="noopener">Requests</a>.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Add dependencies</span></span><br><span class="line">$ poetry add click colorama requests</span><br></pre></td></tr></table></figure><p><video src="/assets/images/poetry/poetry-add.webm" preload autoplay loop controls></video></p><p>There you go, now if we check the our <strong><code>pyproject.toml</code></strong> again, you can see that it has the new package dependencies.</p><figure class="highlight toml"><table><tr><td class="code"><pre><span class="line"><span class="section">[tool.poetry.dependencies]</span></span><br><span class="line"><span class="attr">python</span> = <span class="string">"^3.7"</span></span><br><span class="line"><span class="attr">click</span> = <span class="string">"^7.0"</span></span><br><span class="line"><span class="attr">colorama</span> = <span class="string">"^0.4.1"</span></span><br><span class="line"><span class="attr">requests</span> = <span class="string">"^2.22"</span></span><br></pre></td></tr></table></figure><h2 id="Developing-our-module-and-our-CLI"><a href="#Developing-our-module-and-our-CLI" class="headerlink" title="Developing our module and our CLI"></a>Developing our module and our CLI</h2><p>We’re ready to develop our <em>checkstat</em> module and command line interface. Let’s start by making a test with <a href="https://docs.pytest.org/en/latest/" target="_blank" rel="noopener">Pytest</a>. For this, let’s open <code>test_checkstat.py</code> inside our <em>tests</em> folder.</p><p>Here we’re going to set an <em>expectation</em>. Our module <strong><code>checkstat</code></strong> should have a method called <strong><code>is_up</code></strong> which should return <strong>True</strong> if the webpage returns the code <a href="https://httpstatuses.com/200" target="_blank" rel="noopener"><em>200</em></a> for any other HTTP code it should return <strong>False</strong>. Here’s the test:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> checkstat</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test_checkstat</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Test if checkstat module returns True on 200 code."""</span></span><br><span class="line">    <span class="keyword">assert</span> checkstat.is_up(<span class="string">'https://codingdose.info'</span>)</span><br></pre></td></tr></table></figure><p>Now if we run this test with pytest it will show red because we haven’t build any modules yet, but for the sake of demonstration, let’s show the red test first.</p><hr><blockquote><p>Remember:<br>In order to use the packages <em>inside</em> your virtual environment, you have to activate it first. With Poetry, you can either use <code>poetry run _command_</code> or simply activate it once using <code>poetry shell</code>.</p></blockquote><hr><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Running our test</span></span><br><span class="line">$ poetry run pytest</span><br></pre></td></tr></table></figure><p><img src="/assets/images/poetry/pytest_red_1.png" alt="Pytest red test"></p><p>Now, let’s build the module shall we? Let’s make a file called <strong><code>checkstat.py</code></strong> under the <em>checkstat</em> directory and define a <strong><code>is_up</code></strong> method that returns <strong><code>True</code></strong> if the webpage is reachable and returns a <em>200</em> HTTP code.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># checkstat/checkstat.py</span></span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">is_up</span><span class="params">(webpage)</span>:</span></span><br><span class="line">    <span class="string">"""Return True if 200 code was received, else return False."""</span></span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        req = requests.get(webpage)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># On connection error, return False.</span></span><br><span class="line">    <span class="keyword">except</span> requests.exceptions.ConnectionError:</span><br><span class="line">        <span class="keyword">return</span> <span class="literal">False</span></span><br><span class="line">    <span class="comment"># Connection was successful, return True on 200 code, else return False.</span></span><br><span class="line">    <span class="keyword">else</span>:</span><br><span class="line">        <span class="keyword">if</span> req.status_code == <span class="number">200</span>:</span><br><span class="line">            <span class="keyword">return</span> <span class="literal">True</span></span><br><span class="line">        <span class="keyword">return</span> <span class="literal">False</span></span><br></pre></td></tr></table></figure><p>Now let’s add our module to our initialization package so it gets correctly loaded.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># checkstat/__init__.py</span></span><br><span class="line"><span class="keyword">from</span> .checkstat <span class="keyword">import</span> is_up</span><br></pre></td></tr></table></figure><p>Let’s re-run our test to see if its passing now.</p><p><img src="/assets/images/poetry/pytest_green_1.png" alt="Pytest green test"></p><p>Perfect, now that our test is passing we can go ahead and make our command line interface. Let’s create a file called <strong><code>cli.py</code></strong> inside our <em>checkstat</em> folder.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># checkstat/cli.py</span></span><br><span class="line"><span class="keyword">import</span> click</span><br><span class="line"><span class="keyword">import</span> checkstat</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@click.command()</span></span><br><span class="line"><span class="meta">@click.argument('host')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(host)</span>:</span></span><br><span class="line">    <span class="string">"""CLI for checkstat package."""</span></span><br><span class="line">    print(<span class="string">'Status: '</span>, end=<span class="string">''</span>)</span><br><span class="line">    <span class="keyword">if</span> checkstat.is_up(host):</span><br><span class="line">        click.secho(<span class="string">'Up and running.'</span>, bold=<span class="literal">True</span>, fg=<span class="string">'green'</span>)</span><br><span class="line">    <span class="keyword">else</span>:</span><br><span class="line">        click.secho(<span class="string">'Server is down.'</span>, bold=<span class="literal">True</span>, fg=<span class="string">'red'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    main()</span><br></pre></td></tr></table></figure><p>Perfect, now we have completed our CLI, but we haven’t tried it yet, how do we make it run on the terminal without calling <code>python x_file.py</code>? Easy-peasy, let’s edit the <strong><code>pyproject.toml</code></strong> and add a <strong><em>script</em></strong> section.</p><figure class="highlight toml"><table><tr><td class="code"><pre><span class="line"><span class="section">[tool.poetry]</span></span><br><span class="line"><span class="attr">name</span> = <span class="string">"checkstat"</span></span><br><span class="line"><span class="attr">version</span> = <span class="string">"0.1.0"</span></span><br><span class="line"><span class="attr">description</span> = <span class="string">""</span></span><br><span class="line"><span class="attr">authors</span> = [<span class="string">"Franccesco Orozco &lt;franccesco.orozco@codingdose.info&gt;"</span>]</span><br><span class="line"></span><br><span class="line">--- SNIP ---</span><br><span class="line"></span><br><span class="line"><span class="comment"># New section</span></span><br><span class="line"><span class="section">[tool.poetry.scripts]</span></span><br><span class="line"><span class="attr">checkstat</span> = <span class="string">"checkstat.cli:main"</span></span><br></pre></td></tr></table></figure><p>What does <code>checkstat.cli:main</code> means? It means: <em>Hey python, whenever I type <strong>checkstat</strong> I want you to execute the function <strong>main()</strong> inside of the module <strong>cli.py</strong> in the package <strong>checkstat</strong></em>.</p><p>Now that we have updated our configuration file, our CLI gets automatically loaded into our virtual environment and we can execute it directly with Poetry. Let’s test it ourselves with a working webpage and another one that returns a 500 HTTP status.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ poetry run checkstat https://codingdose.info</span><br><span class="line">Status: Up and running.</span><br><span class="line"></span><br><span class="line">$ poetry run checkstat https://httpstat.us/500</span><br><span class="line">Status: Server is down.</span><br></pre></td></tr></table></figure><p><video src="/assets/images/poetry/checkstat-updown.webm" preload autoplay loop controls></video></p><p>Awesome! Our little command line application is ready to be published.</p><h2 id="Publishing-our-tool"><a href="#Publishing-our-tool" class="headerlink" title="Publishing our tool"></a>Publishing our tool</h2><p>Publishing our tool to the PyPi is really effortless, let’s try to do it right now. First of all we have to build our package.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ poetry build</span><br><span class="line">Building checkstat (0.1.0)</span><br><span class="line"> - Building sdist</span><br><span class="line"> - Built checkstat-0.1.0.tar.gz</span><br><span class="line"></span><br><span class="line"> - Building wheel</span><br><span class="line"> - Built checkstat-0.1.0-py2.py3-none-any.whl</span><br></pre></td></tr></table></figure><p>Now we have to upload it to the pypi.org repository.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ poetry publish</span><br><span class="line">Username: franccesco</span><br><span class="line">Password: **********</span><br><span class="line"></span><br><span class="line"> - Uploading checkstat-0.1.0-py2.py3-none-any.whl 100%</span><br><span class="line"> - Uploading checkstat-0.1.0.tar.gz 100%</span><br></pre></td></tr></table></figure><p>And that’s it! Our tools is now available to <em>anyone</em> using <code>pip</code> to install packages. Let’s check it out: <a href="https://pypi.org/project/checkstat/" target="_blank" rel="noopener">https://pypi.org/project/checkstat/</a></p><h2 id="Installing-our-tool"><a href="#Installing-our-tool" class="headerlink" title="Installing our tool"></a>Installing our tool</h2><p>Let’s go and install our tool to see if it’s available now.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ pip install checkstat</span><br></pre></td></tr></table></figure><p>And let’s execute it.</p><p><video src="/assets/images/poetry/install-checkstat.webm" preload autoplay loop controls></video></p><p>It works!</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>I hope you like the entry, today we have learn how to <strong>create, develop and ship</strong> a very basic CLI. Poetry is an incredibly easy to use and flexible tool to create your packages and distribute them. Of course, there’s a lot more than I have shown here, remember to check out Poetry’s documentation.</p><h2 id="Further-reading"><a href="#Further-reading" class="headerlink" title="Further reading"></a>Further reading</h2><ul><li><a href="https://poetry.eustace.io/" target="_blank" rel="noopener">Poetry</a></li><li><a href="https://poetry.eustace.io/docs/" target="_blank" rel="noopener">Poetry’s Documentation</a></li><li><a href="https://docs.python-guide.org/scenarios/cli/" target="_blank" rel="noopener">Command Line Applications</a></li><li><a href="https://python-guide-cn.readthedocs.io/en/latest/writing/structure.html" target="_blank" rel="noopener">Structuring Your Project</a></li><li><a href="http://docs.python-requests.org/en/master/" target="_blank" rel="noopener">Requests</a></li><li><a href="http://click.pocoo.org/" target="_blank" rel="noopener">Click</a></li><li><a href="https://docs.pytest.org/en/latest/" target="_blank" rel="noopener">Pytest</a></li></ul>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> packaging </tag>
            
            <tag> poetry </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>How to Easily Use a Progress Bar in Python</title>
      <link href="2019/06/15/how-to-use-a-progress-bar-in-python/"/>
      <url>2019/06/15/how-to-use-a-progress-bar-in-python/</url>
      
        <content type="html"><![CDATA[<img src="/assets/images/progressbars/carbon.png"><p>We’ve all been there, your code is performing a job and it’s going to take a while. I’m an impatient guy, so it would be nice to have an ETA or a progress bar to show us. Fortunately, there are libraries out there than can help us to achieve this!</p><p>There’s two ways in which we can integrate a progress bar into our loops, via a <a href="https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/" target="_blank" rel="noopener"><em>Context Manager</em></a> or just wrapping up an iterable object into a method.</p><p>We’re going to be testing Progress, ProgressBar2, TQDM, Click and Clint, so make sure to create your testing environment with <a href="https://codingdose.info/2018/02/20/pipenv-development-workflow/">Pipenv</a>:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ mkdir progressbar-testing</span><br><span class="line">$ <span class="built_in">cd</span> progressbar-testing</span><br><span class="line">$ pipenv install tqdm progressbar2 click clint</span><br></pre></td></tr></table></figure><h2 id="Progress"><a href="#Progress" class="headerlink" title="Progress"></a><a href="https://github.com/verigak/progress/" target="_blank" rel="noopener">Progress</a></h2><p>While I was testing each library, this is one I really liked, and this is because it has a lot of progress bar styles that you can play with. We’re not going to look at <em>all</em> of them, but you can take a look at the source code and documentation if you have further questions.</p><h3 id="Progress-BAR"><a href="#Progress-BAR" class="headerlink" title="Progress - BAR"></a>Progress - BAR</h3><p>This is the most basic one, is basically a progress bar that is being filled by a hash. It works pretty easily with a context manager, you can use this snippet as an example:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"><span class="keyword">from</span> progress.bar <span class="keyword">import</span> Bar</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> Bar(<span class="string">'Processing...'</span>) <span class="keyword">as</span> bar:</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">100</span>):</span><br><span class="line">        sleep(<span class="number">0.02</span>)</span><br><span class="line">        bar.next()</span><br></pre></td></tr></table></figure><p>As you can see, in this case we only import the <code>Bar</code> class and we add a <em>label</em> to our progress bar and it automatically handles our loop. At the end of each iteration, we have to append the <code>.next()</code> method to our <code>bar</code> object so we can update the progress bar.</p><p><video src="/assets/images/progressbars/progress-bar.webm" preload autoplay loop controls></video></p><h3 id="Progress-PixelBar"><a href="#Progress-PixelBar" class="headerlink" title="Progress - PixelBar"></a>Progress - PixelBar</h3><p>We can achieve the same with other progress bar styles, let’s try the Pixel Bar:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"><span class="keyword">from</span> progress.bar <span class="keyword">import</span> PixelBar</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> PixelBar(<span class="string">'Processing...'</span>) <span class="keyword">as</span> bar:</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">100</span>):</span><br><span class="line">        sleep(<span class="number">0.02</span>)</span><br><span class="line">        bar.next()</span><br></pre></td></tr></table></figure><p><video src="/assets/images/progressbars/progress-pixelbar.webm" preload autoplay loop controls></video></p><h3 id="Progress-PixelSpinner"><a href="#Progress-PixelSpinner" class="headerlink" title="Progress - PixelSpinner"></a>Progress - PixelSpinner</h3><p>Sometimes you don’t know how long it might take to perform an operation. If this is the case, then you can use the Pixel Spinner to display a pixel spinner (duh!) without an actual <em>progress bar</em>.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"><span class="keyword">from</span> progress.spinner <span class="keyword">import</span> PixelSpinner</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> PixelSpinner(<span class="string">'Processing...'</span>) <span class="keyword">as</span> bar:</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">100</span>):</span><br><span class="line">        sleep(<span class="number">0.06</span>)</span><br><span class="line">        bar.next()</span><br></pre></td></tr></table></figure><p><video src="/assets/images/progressbars/progress-pixelspinner.webm" preload autoplay loop controls></video></p><p>Pretty neat right? You can read more about Progress here:</p><ul><li><a href="https://github.com/verigak/progress/" target="_blank" rel="noopener">Source Code</a></li></ul><hr><h2 id="TQDM"><a href="#TQDM" class="headerlink" title="TQDM"></a><a href="https://github.com/tqdm/tqdm" target="_blank" rel="noopener">TQDM</a></h2><p><a href="https://github.com/tqdm/tqdm" target="_blank" rel="noopener">TQDM</a>, a short for <em>taqadum</em> which means <em>progress</em> in Arabic, is a very popular choice, even more for data scientist or analysts, as it provides a really fast framework with a lot of customizations and information for you to work with.</p><p>It’s also smart enough to use it with barely two lines of code. Just provide an iterable to the function <code>tqdm()</code> and your good to go.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> tqdm <span class="keyword">import</span> tqdm</span><br><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> tqdm(range(<span class="number">100</span>)):</span><br><span class="line">    sleep(<span class="number">0.02</span>)</span><br></pre></td></tr></table></figure><p>And there you go! You have a lot of information on your progress bar such as a percentage, the length of your iterable, an ETA and even <em>iterables per seconds</em>!</p><p><video src="/assets/images/progressbars/tqdm.webm" preload autoplay loop controls></video></p><p>You can also add a label to your progress bar, displaying each object along the way:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> string</span><br><span class="line"><span class="keyword">from</span> tqdm <span class="keyword">import</span> tqdm</span><br><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"></span><br><span class="line"><span class="comment"># A list from A to Z wrapped around TQDM function</span></span><br><span class="line">progress_bar = tqdm(list(string.ascii_lowercase))</span><br><span class="line"><span class="keyword">for</span> letter <span class="keyword">in</span> progress_bar:</span><br><span class="line">    progress_bar.set_description(<span class="string">f'Processing <span class="subst">&#123;letter&#125;</span>...'</span>)</span><br><span class="line">    sleep(<span class="number">0.09</span>)</span><br></pre></td></tr></table></figure><p><video src="/assets/images/progressbars/tqdm-label.webm" preload autoplay loop controls></video></p><p>TQDM, from my perspective, is a really important project if you’re into Data Science or need a incredibly fast way to show progress to your operations, you can read more about it here:</p><ul><li><a href="https://github.com/tqdm/tqdm" target="_blank" rel="noopener">Source Code &amp; Documentation</a></li><li><a href="https://github.com/tqdm/tqdm/wiki" target="_blank" rel="noopener">Wiki</a></li></ul><hr><h2 id="Click"><a href="#Click" class="headerlink" title="Click"></a><a href="https://click.palletsprojects.com/" target="_blank" rel="noopener">Click</a></h2><p>Click is one of the best libraries out there to create a <a href="https://en.wikipedia.org/wiki/Command-line_interface" target="_blank" rel="noopener">Command Line Interface</a> to your apps or libraries. I cannot recommend it enough!</p><p>It also includes a very simple progress bar as an utility, you can use it inside a context manager, just like this:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> click</span><br><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"></span><br><span class="line"><span class="comment"># Fill character is # by default, you can change it</span></span><br><span class="line"><span class="comment"># for any other char you want, or even change the color.</span></span><br><span class="line">fill_char = click.style(<span class="string">'='</span>, fg=<span class="string">'yellow'</span>)</span><br><span class="line"><span class="keyword">with</span> click.progressbar(range(<span class="number">100</span>), label=<span class="string">'Loading...'</span>, fill_char=fill_char) <span class="keyword">as</span> bar:</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> bar:</span><br><span class="line">        sleep(<span class="number">0.02</span>)</span><br></pre></td></tr></table></figure><p>You can also see that we can change the color of the progress bar meter, and we also have an ETA.</p><p><video src="/assets/images/progressbars/click.webm" preload autoplay loop controls></video></p><p>If you want to know more about the progress bar utility in Click, you can check it out here:</p><ul><li><a href="https://click.palletsprojects.com/en/7.x/utils/#showing-progress-bars" target="_blank" rel="noopener">Click - Showing Progress Bars</a></li></ul><hr><h2 id="ProgressBar2"><a href="#ProgressBar2" class="headerlink" title="ProgressBar2"></a><a href="https://github.com/WoLpH/python-progressbar" target="_blank" rel="noopener">ProgressBar2</a></h2><p>This is also a very popular choice and one that is easy to use. It also work with widgets to calculate the current progress, such as AbsoluteETA, AdaptiveETA, AdaptiveTransferSpeed and others which are very interesting.</p><p>It’s implementation is also very simple. As with TQDM, only need 2 lines of code are enough to get us started:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"><span class="keyword">from</span> progressbar <span class="keyword">import</span> progressbar</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> progressbar(range(<span class="number">100</span>)):</span><br><span class="line">    sleep(<span class="number">0.02</span>)</span><br></pre></td></tr></table></figure><p>And yes, although you install it as <em>progressbar2</em> make sure you import it as <em>progressbar</em>. Here’s how it displays the progress bar by default.</p><p><video src="/assets/images/progressbars/progressbar2.webm" preload autoplay loop controls></video></p><p>You can checkout the ProgressBar2 documentation and homepage here:</p><ul><li><a href="https://github.com/WoLpH/python-progressbar" target="_blank" rel="noopener">Homepage</a></li><li><a href="https://progressbar-2.readthedocs.io/en/latest/index.html" target="_blank" rel="noopener">Documentation</a></li><li><a href="https://progressbar-2.readthedocs.io/en/latest/_modules/progressbar/widgets.html" target="_blank" rel="noopener">Widgets</a></li></ul><hr><h2 id="Clint"><a href="#Clint" class="headerlink" title="Clint"></a><a href="https://github.com/kennethreitz/clint" target="_blank" rel="noopener">Clint</a></h2><p>And lastly we have Clint, which stands for <em><strong>C</strong> ommand <strong>L</strong> ine <strong>IN</strong> terface <strong>T</strong> ools</em>, which is not maintained anymore, but I will show it here just to pay my respects.</p><p>Creating a progress bar is just as easy as we have seen with the other tools, this one doesn’t require a context manager. Here we can see a regular progress bar and a <em>Mill</em> style progress bar:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"><span class="keyword">from</span> clint.textui <span class="keyword">import</span> progress</span><br><span class="line"></span><br><span class="line">print(<span class="string">'Clint - Regular Progress Bar'</span>)</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> progress.bar(range(<span class="number">100</span>)):</span><br><span class="line">    sleep(<span class="number">0.02</span>)</span><br><span class="line"></span><br><span class="line">print(<span class="string">'Clint - Mill Progress Bar'</span>)</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> progress.mill(range(<span class="number">100</span>)):</span><br><span class="line">    sleep(<span class="number">0.02</span>)</span><br></pre></td></tr></table></figure><p>And here you can see it in action:</p><p><video src="/assets/images/progressbars/clint.webm" preload autoplay loop controls></video></p><hr><p>There are other libraries out there but these are the ones that I definitely recommend you to check out. Also, here’s a snippet of code that tests all of the progress bar styles and libraries that we have tested so far. Just make sure to install the appropriate libraries.</p><p>You can clone it with Git:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git <span class="built_in">clone</span> https://gist.github.com/33e56c93c3c43cf70f19ecbfc921e358.git progressbar-testing</span><br></pre></td></tr></table></figure><script src="https://gist.github.com/franccesco/33e56c93c3c43cf70f19ecbfc921e358.js"></script><p>Also, I should recall that all these libraries have more features than the ones that we have seen here, so make sure to check them out. Have fun!</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Create a Project Page for Your Repos Easily With Jekyll and GitHub Pages</title>
      <link href="2019/06/08/create-a-project-page-for-your-repositories-easily-with-Jekyll/"/>
      <url>2019/06/08/create-a-project-page-for-your-repositories-easily-with-Jekyll/</url>
      
        <content type="html"><![CDATA[<!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1505373877841-8d25f7d46678?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="person discussing while standing in front of a large screen in front of people inside dim-lighted room " title="person discussing while standing in front of a large screen in front of people inside dim-lighted room ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/bzdhc5b3Bxs" target="_blank">Photo</a> by <a href="https://unsplash.com/@xteemu" target="_blank">@xteemu</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 01/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p>I’m going to show you how to create a <em>project page</em> using the ever popular Jekyll and GitHub Pages so your projects can have a face, and of course, to show it of to your friends out there.</p><p>So, without further ado, we should get to the point already, shall we?</p><h2 id="Building-our-site-with-Jekyll"><a href="#Building-our-site-with-Jekyll" class="headerlink" title="Building our site with Jekyll"></a>Building our site with Jekyll</h2><p>Let’s say that you have a project called <a href="https://github.com/franccesco/chuck-says/" target="_blank" rel="noopener">chuck-says</a> that acts like a fortune cookie + cowsay whenever you call it in the command line.</p><p>To provide our repository with a project page, we can use <strong>Jekyll</strong>, which is a tool made in Ruby that can use a combination of Markdown and Font Matter to create static websites and blogs. It’s a very easy to use site generator that can aid us with what we want to accomplish in this case. We can install it using the <code>gem</code> command:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gem install jekyll</span><br></pre></td></tr></table></figure><p>Now, let’s go to the folder in which our project is located. There, we can create our project’s page on the folder <code>docs</code>. There’s no need to create the folder as Jekyll can do this for us:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ jekyll new docs</span><br><span class="line">Running bundle install <span class="keyword">in</span> /home/your_username/chuck-says/docs...</span><br><span class="line">  Bundler: Fetching gem metadata from https://rubygems.org/...........</span><br><span class="line">  Bundler: Fetching gem metadata from https://rubygems.org/.</span><br><span class="line">  Bundler: Resolving dependencies...</span><br><span class="line">  Bundler: Using public_suffix 3.1.0</span><br><span class="line">--- SNIP ---</span><br></pre></td></tr></table></figure><p>After that we can see our page structure, which can be something like this:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">$ tree docs&#x2F;</span><br><span class="line">.</span><br><span class="line">├── 404.html</span><br><span class="line">├── about.md</span><br><span class="line">├── _config.yml</span><br><span class="line">├── Gemfile</span><br><span class="line">├── Gemfile.lock</span><br><span class="line">├── index.md</span><br><span class="line">└── _posts</span><br><span class="line">    └── 2019-06-09-welcome-to-jekyll.markdown</span><br><span class="line"></span><br><span class="line">1 directory, 7 files</span><br></pre></td></tr></table></figure><p>Here’s a brief description of each one of them, the bold one’s, such as <strong>config.yml</strong> and <strong>index.md</strong> are the ones that we’re actually going to use!</p><table><thead><tr><th>File</th><th>Description</th></tr></thead><tbody><tr><td>404.html</td><td>Not Found template</td></tr><tr><td>about.md</td><td>A short about page</td></tr><tr><td><strong>config.yml</strong></td><td>Configuration file</td></tr><tr><td>Gemfile</td><td>Dependency gem requirements and configuration</td></tr><tr><td>Gemfile.lock</td><td>Pinned gem versions</td></tr><tr><td><strong>index.md</strong></td><td>Front Page (And the one we’ll use!)</td></tr><tr><td>posts/</td><td>Folder containing blog posts (we won’t use them!)</td></tr></tbody></table><p>As we’re not trying to create a blog (we can see how to do it on a another post), we can remove <code>about.md</code> and the <code>posts/</code> folder.</p><p>Before we move on, there’s something important we need to do first! We have to add the <strong><a href="https://github.com/github/pages-gem" target="_blank" rel="noopener"><code>github-pages</code></a></strong> gem, provided by GitHub, in order to have access to themes and to ensure maximum compatibility to make our site deploy on GitHub Pages. Let’s open up <code>Gemfile</code> and add the gem as follows:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"></span><br><span class="line"><span class="comment"># Main dependencies and group definitions</span></span><br><span class="line">source <span class="string">"https://rubygems.org"</span></span><br><span class="line">gem <span class="string">"jekyll"</span>, <span class="string">"~&gt; 3.8.5"</span></span><br><span class="line">gem <span class="string">"minima"</span>, <span class="string">"~&gt; 2.0"</span></span><br><span class="line">group <span class="symbol">:jekyll_plugins</span> <span class="keyword">do</span></span><br><span class="line">  gem <span class="string">"jekyll-feed"</span>, <span class="string">"~&gt; 0.6"</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line">gem <span class="string">"tzinfo-data"</span>, <span class="symbol">platforms:</span> [<span class="symbol">:mingw</span>, <span class="symbol">:mswin</span>, <span class="symbol">:x64_mingw</span>, <span class="symbol">:jruby</span>]</span><br><span class="line">gem <span class="string">"wdm"</span>, <span class="string">"~&gt; 0.1.0"</span> <span class="keyword">if</span> Gem.win_platform?</span><br><span class="line"></span><br><span class="line"><span class="comment"># Add the GitHub Pages gem</span></span><br><span class="line">gem <span class="string">'github-pages'</span>, <span class="symbol">group:</span> <span class="symbol">:jekyll_plugins</span></span><br></pre></td></tr></table></figure><p>Now, let’s update our packages:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Remember to to this inside the /docs folder</span></span><br><span class="line">$ bundle update</span><br><span class="line">Fetching gem metadata from https://rubygems.org/..........</span><br><span class="line">Fetching gem metadata from https://rubygems.org/.</span><br><span class="line">Resolving dependencies...</span><br><span class="line">Using concurrent-ruby 1.1.5</span><br><span class="line">...</span><br><span class="line">Bundle updated!</span><br><span class="line"></span><br><span class="line">$ bundle install</span><br><span class="line">Using concurrent-ruby 1.1.5</span><br><span class="line">Using i18n 0.9.5</span><br><span class="line">...</span><br><span class="line">Bundle complete! 5 Gemfile dependencies, 85 gems now installed.</span><br><span class="line">Use `bundle info [gemname]` to see <span class="built_in">where</span> a bundled gem is installed.</span><br></pre></td></tr></table></figure><p>To ensure that we can run our page, let’s serve it locally with the <code>serve</code> or <code>s</code> subcommand:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> jekyll serve</span><br><span class="line">Configuration file: /home/your_home/workspace/chuck-says/docs/_config.yml</span><br><span class="line">            Source: /home/your_home/workspace/chuck-says/docs</span><br><span class="line">       Destination: /home/your_home/workspace/chuck-says/docs/_site</span><br><span class="line"> Incremental build: disabled. Enable with --incremental</span><br><span class="line">      Generating...</span><br><span class="line">       Jekyll Feed: Generating feed <span class="keyword">for</span> posts</span><br><span class="line">                    <span class="keyword">done</span> <span class="keyword">in</span> 0.111 seconds.</span><br><span class="line"> Auto-regeneration: enabled <span class="keyword">for</span> <span class="string">'/home/your_home/workspace/chuck-says/docs'</span></span><br><span class="line">    Server address: http://127.0.0.1:4000/</span><br><span class="line">  Server running... press ctrl-c to stop.</span><br></pre></td></tr></table></figure><p>There you go! You can now visit <a href="http://127.0.0.1:4000/" target="_blank" rel="noopener">http://127.0.0.1:4000/</a> to see your Jekyll web page up and running!</p><p><img src="/assets/images/jekyll/jekyll_default.png" alt="Default Jekyll Landing Page"></p><p>Now this is great, but this looks like a blog, and we’re not trying to make a blog here, we have to perform a few tweaks to get our one-page project website come to life.</p><h1 id="Configuring-Jekyll"><a href="#Configuring-Jekyll" class="headerlink" title="Configuring Jekyll"></a>Configuring Jekyll</h1><p>We got our site up and running, which is awesome, you can make blog posts! But we’re not looking for that. Instead, we want to create a one-page site for our project.</p><p>To achieve this, we’re going to open up <strong><code>_config.yml</code></strong> and make a few changes, as well as changing the main theme. We will want to apply the <a href="https://github.com/pages-themes/cayman" target="_blank" rel="noopener"><strong>Cayman</strong> theme</a> for our site. Here’s the configuration file:</p><figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">title:</span> <span class="string">Your</span> <span class="string">awesome</span> <span class="string">title</span></span><br><span class="line"><span class="attr">email:</span> <span class="string">your-email@example.com</span></span><br><span class="line"><span class="attr">description:</span> <span class="string">&gt;-</span> <span class="comment"># this means to ignore newlines until "baseurl:"</span></span><br><span class="line">  <span class="string">Write</span> <span class="string">an</span> <span class="string">awesome</span> <span class="string">description</span> <span class="string">for</span> <span class="string">your</span> <span class="string">new</span> <span class="string">site</span> <span class="string">here.</span> <span class="string">You</span> <span class="string">can</span> <span class="string">edit</span> <span class="string">this</span></span><br><span class="line">  <span class="string">line</span> <span class="string">in</span> <span class="string">_config.yml.</span> <span class="string">It</span> <span class="string">will</span> <span class="string">appear</span> <span class="string">in</span> <span class="string">your</span> <span class="string">document</span> <span class="string">head</span> <span class="string">meta</span> <span class="string">(for</span></span><br><span class="line">  <span class="string">Google</span> <span class="string">search</span> <span class="string">results)</span> <span class="string">and</span> <span class="string">in</span> <span class="string">your</span> <span class="string">feed.xml</span> <span class="string">site</span> <span class="string">description.</span></span><br><span class="line"><span class="attr">baseurl:</span> <span class="string">""</span> <span class="comment"># the subpath of your site, e.g. /blog</span></span><br><span class="line"><span class="attr">url:</span> <span class="string">""</span> <span class="comment"># the base hostname &amp; protocol for your site, e.g. http://example.com</span></span><br><span class="line"><span class="attr">twitter_username:</span> <span class="string">jekyllrb</span></span><br><span class="line"><span class="attr">github_username:</span>  <span class="string">jekyll</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Build settings</span></span><br><span class="line"><span class="attr">markdown:</span> <span class="string">kramdown</span></span><br><span class="line"><span class="attr">theme:</span> <span class="string">minima</span></span><br><span class="line"><span class="attr">plugins:</span></span><br><span class="line">  <span class="bullet">-</span> <span class="string">jekyll-feed</span></span><br></pre></td></tr></table></figure><p>We will want to change these variables, more importantly, the <strong><code>baseurl</code></strong>. For GitHub projects, your <code>baseurl</code> will be your repository name. If you change your repository name in the future, then remember to edit this line! Let’s make our changes:</p><figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">title:</span> <span class="string">Chuck</span> <span class="string">Says.</span></span><br><span class="line"><span class="attr">email:</span> <span class="string">franccesco@codingdose.info</span></span><br><span class="line"><span class="attr">description:</span> <span class="string">Replace</span> <span class="string">weak</span> <span class="string">fortune</span> <span class="string">cookies</span> <span class="string">with</span> <span class="string">all-mighty</span> <span class="string">Chuck</span> <span class="string">Norris</span> <span class="string">facts.</span></span><br><span class="line"><span class="attr">baseurl:</span> <span class="string">"/chuck-says"</span> <span class="comment"># the subpath of your site, e.g. /blog</span></span><br><span class="line"><span class="attr">url:</span> <span class="string">""</span> <span class="comment"># If you have a domain name, then you should fill this out!</span></span><br><span class="line"><span class="attr">twitter_username:</span> <span class="string">__franccesco</span></span><br><span class="line"><span class="attr">github_username:</span>  <span class="string">franccesco</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Build settings</span></span><br><span class="line"><span class="attr">markdown:</span> <span class="string">kramdown</span></span><br><span class="line"><span class="attr">theme:</span> <span class="string">jekyll-theme-cayman</span> <span class="comment"># &lt;&lt; Set the Cayman theme!</span></span><br><span class="line"><span class="attr">plugins:</span></span><br><span class="line">  <span class="bullet">-</span> <span class="string">jekyll-feed</span></span><br></pre></td></tr></table></figure><p>That was fast, right? But before we move on, it’s time for us to update the layout in our <strong><code>index.md</code></strong> file. You see, Jekyll works using a pre-defined <a href="https://jekyllrb.com/docs/layouts/" target="_blank" rel="noopener">layout</a> that wraps the page in it, these are defined in the directory <strong><code>_layouts</code></strong> in a HTML file, luckily they are added automatically in the <code>github-pages</code> gem, so you won’t have to create one.</p><p>Let’s add the default layout to our <code>index.md</code> page using <a href="https://jekyllrb.com/docs/front-matter/" target="_blank" rel="noopener">Font Matter</a>. Let’s open it up and change the format; After that, we can finally fill out our page!</p><figure class="highlight markdown"><table><tr><td class="code"><pre><span class="line">---</span><br><span class="line"><span class="section"># You can copy and paste your `README.md` contents, or write something else!</span></span><br><span class="line"><span class="section"># Feel free to use your imagination here, just remember to respect the font matter division (`---`).</span></span><br><span class="line"></span><br><span class="line">layout: default # <span class="xml"><span class="tag">&lt;&lt; <span class="attr">Change</span> <span class="attr">this</span> <span class="attr">line</span> <span class="attr">from</span> '<span class="attr">home</span>' <span class="attr">to</span> '<span class="attr">default</span>'</span></span></span><br><span class="line"><span class="xml">---</span></span><br><span class="line"></span><br><span class="line"><span class="xml">Chuck Norris Facts, right in your terminal!</span></span><br><span class="line"></span><br><span class="line"><span class="xml">Because, who wouldn't want that? You better...</span></span><br><span class="line"></span><br><span class="line"><span class="xml">Get your day started as soon you see that sweet Chuck Norris fact of the day in your terminal; there's more than 600+ facts here, baby!</span></span><br><span class="line"><span class="xml">They're all real... allegedly.</span></span><br><span class="line"></span><br><span class="line"><span class="xml">--- SNIP ---</span></span><br></pre></td></tr></table></figure><h2 id="Deploying-our-site-with-GitHub-Pages"><a href="#Deploying-our-site-with-GitHub-Pages" class="headerlink" title="Deploying our site with GitHub Pages"></a>Deploying our site with GitHub Pages</h2><p>After you have pushed your <code>/docs</code> folder to your repository with all of the changes we previously made, let’s configure GitHub so it can deploy our site. Fortunately, this is very easy.</p><ul><li>In your GitHub repository, click the <strong>Settings</strong> tab.</li><li>Scroll down until you see <strong>GitHub Pages</strong>.</li><li>Click the button under <strong>Source</strong> and select <strong>master branch /docs folder</strong></li></ul><p><img src="/assets/images/jekyll/github_settings.png"></p><p>After that, you should see <strong>Your site is published at <a href="https://yourusername.github.io/your-project/" target="_blank" rel="noopener">https://yourusername.github.io/your-project/</a></strong>, it may take a few seconds to appear.</p><p>After you have applied the settings, you should be able to check out your new project site at the URL that GitHub provides you.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>Jekyll is a very powerful site generator. You can use it to make a blog, a CV online, your project’s sites, and it doesn’t cost a dime to host it at GitHub pages! Here’s an example of how it looks: <a href="https://franccesco.github.io/chuck-says/" target="_blank" rel="noopener">https://franccesco.github.io/chuck-says/</a></p><p><a href="https://franccesco.github.io/chuck-says/" target="_blank" rel="noopener"><img src="/assets/images/jekyll/chuck_page.png"></a></p><p>Pretty neat, right? You can check the example repository by clicking on the image. Also, if you’re new and you would like to make your first contribution in the GitHub community, <a href="https://github.com/franccesco/chuck-says/issues" target="_blank" rel="noopener">check out the issues pages</a>.</p><hr><p>I think I’ll make a post on how to create a blog using Jekyll, but we will also check out other static sites generators too. I hope you really liked the post, if you have any question then don’t hesitate to let me know.</p>]]></content>
      
      
      
        <tags>
            
            <tag> ruby </tag>
            
            <tag> github </tag>
            
            <tag> jekyll </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>How to Create a Ruby Gem With Bundler</title>
      <link href="2019/03/06/How-to-create-a-Ruby-gem-with-Bundler/"/>
      <url>2019/03/06/How-to-create-a-Ruby-gem-with-Bundler/</url>
      
        <content type="html"><![CDATA[<!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1575833947349-69324d765146?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="small brown cardboard box with handle " title="small brown cardboard box with handle ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/5R5Trsu1aIM" target="_blank">Photo</a> by <a href="https://unsplash.com/@kellysikkema" target="_blank">@kellysikkema</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 01/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p>I’ve been writing and focusing on Python lately and I’ve been wanting to make more content about Ruby. Ruby was my very first language and the one that got me into this programming world.</p><p>For this entry I’m going to write how to create, test and publish our gem to <a href="https://rubygems.org/" target="_blank" rel="noopener">RubyGems.org</a> to make it available for everyone, and in future entries we’re going to see <strong>how to setup a CI/CD</strong> for automatic testing and deployment, <strong>Behavior Driven Testing</strong> with Cucumber/Aruba and <strong>Code Coverage</strong> with SimpleCov.</p><p>Let’s start with the basics. You can skip the basics <a href="#Creating-a-gem-with-bundler">clicking here</a></p><h2 id="What’s-a-ruby-gem"><a href="#What’s-a-ruby-gem" class="headerlink" title="What’s a ruby gem?"></a>What’s a ruby gem?</h2><p>A ruby gem is a piece of code that you can integrate to your software (made in ruby) to help you achieve some tasks more easily. Think of it as a <em>library</em>, because that’s exactly what it is!</p><p>An example of this would be requiring a gem that can make <em>http</em> requests for us. One gem (read <em>library</em>) that can perform this would be <a href="https://github.com/jnunemaker/httparty" target="_blank" rel="noopener">httparty</a>.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="meta">irb(main):001:0&gt;</span> require <span class="string">'httparty'</span></span><br><span class="line">=&gt; true</span><br><span class="line"><span class="meta">irb(main):002:0&gt;</span> response = HTTParty.get(<span class="string">'https://google.com'</span>)</span><br><span class="line"><span class="meta">irb(main):003:0&gt;</span> response.header</span><br><span class="line">=&gt; #&lt;Net::HTTPOK 200 OK readbody=true&gt;</span><br></pre></td></tr></table></figure><p>Another good example is the gem <a href="https://github.com/janlelis/clipboard" target="_blank" rel="noopener">Clipboard</a> that allow us to copy, paste and clear the clipboard on Linux, MacOS and Windows.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="meta">irb(main):001:0&gt;</span> require <span class="string">'clipboard'</span></span><br><span class="line">=&gt; true</span><br><span class="line"><span class="meta">irb(main):002:0&gt;</span> Clipboard.copy(<span class="string">'Hello world!'</span>)</span><br><span class="line">=&gt; <span class="string">"Hello world!"</span></span><br><span class="line"><span class="meta">irb(main):003:0&gt;</span> Clipboard.paste</span><br><span class="line">=&gt; <span class="string">"Hello world!"</span></span><br><span class="line"><span class="meta">irb(main):004:0&gt;</span> Clipboard.clear</span><br><span class="line">=&gt; <span class="string">""</span></span><br></pre></td></tr></table></figure><h2 id="How-to-install-a-gem"><a href="#How-to-install-a-gem" class="headerlink" title="How to install a gem"></a>How to install a gem</h2><p>Installing a gem is pretty straight forward, we can do this with the <code>gem</code> command line application provided by RubyGems, you shouldn’t worry about installing it as it comes bundled with Ruby since version 1.9:</p><pre><code>$ gem install _gem_name_here_</code></pre><p>To install the Clipboard gem, then we can install it like this.</p><pre><code>$ gem install clipboard</code></pre><p>The end users that are going to use our gem will also install our gem (or library) like this.</p><h2 id="Dependency-issues"><a href="#Dependency-issues" class="headerlink" title="Dependency issues"></a>Dependency issues</h2><p>There’s only one issue with this implementation, and that is when you have two pieces of code that requires different versions of (let’s say) Clipboard. For example, <strong>SoftwareA</strong> requires the Clipboard gem version <em>0.5.8</em>, and <strong>SoftwareB</strong> requires the version <em>1.1.2</em> of the same gem and it brings breaking changes as it is not backwards compatible with the previous versions of the gem.</p><p><strong>SoftwareA</strong> was installed first, so you have the Clipboard version <em>0.5.8</em> but as soon as you install the <strong>SoftwareB</strong> using the gem command line, it proceeds to install Clipboard’s newest version, which would be <em>1.1.2</em>.</p><p>As this new version brings breaking changes and it’s not backwards compatible with previous versions, due to refactoring, renamed functions/method/classes, etc. It becomes pretty obvious that <strong>SoftwareA</strong> won’t work.</p><p>You reinstall the previous version of Clipboard <em>0.5.8</em> and it works now! But guess what, <strong>SoftwareB</strong> just broke. Welcome to the <a href="https://en.wikipedia.org/wiki/Dependency_hell" target="_blank" rel="noopener">Dependency Hell</a>.</p><h2 id="Bundler-comes-to-the-rescue"><a href="#Bundler-comes-to-the-rescue" class="headerlink" title="Bundler comes to the rescue"></a>Bundler comes to the rescue</h2><p>To resolve this issue, we need a sort of isolated environment where we can develop or deploy our software without meddling with the version numbers of our dependencies in our other projects ourselves.</p><p>Bundler was designed with this idea in mind, where you can build your own library or app without affecting the version numbers in your multiple projects. If you’re familiar with virtualenv, venv, pipenv or poetry in Python, then you’ll get the hang of it in no time.</p><p>To install bundler we follow the same procedure when installing any other gem.</p><pre><code>gem install bundler</code></pre><p>After that, we’re able to use bundler with any application to install its requirements, for this we can go to the project folder and create a gemfile (or gemspec sometimes) and create a <code>Gemfile</code>.</p><h2 id="Require-gems-in-your-Gemfile"><a href="#Require-gems-in-your-Gemfile" class="headerlink" title="Require gems in your Gemfile"></a>Require gems in your Gemfile</h2><p>This <code>Gemfile</code> will contain the other libraries that we’re going to use to make our gem work. For this example we’re going to create a folder to hold a new project and then we’re creating a <code>Gemfile</code> to hold our dependencies.</p><pre><code>$ mkdir mygem$ cd mygem$ touch Gemfile</code></pre><p>Now we have a Gemfile in our directory <code>mygem</code>, let’s fill it with a gem that we’re going to require to build our command line interface, this gem is <a href="http://whatisthor.com/" target="_blank" rel="noopener">Thor</a>.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Gemfile</span></span><br><span class="line">source <span class="string">"https://rubygems.org"</span></span><br><span class="line">gem <span class="string">'thor'</span>, <span class="string">'~&gt; 0.20'</span></span><br></pre></td></tr></table></figure><p>What’s happening here? Well, the first line is going to tell Bundler that we’re going to require our gems from the server <a href="https://rubygems.org/" target="_blank" rel="noopener">rubygems.org</a>. The second line is telling Bundler to install the gem Thor.</p><p>But what’s that <code>~&gt;</code> doing there? It’s basically a way of saying <em>“I want the highest version of thor between the range of &gt;= 0.20 and &lt; 1.0</em>. This translates to the highest version of thor available since <em>0.20</em> but <strong>less</strong> than <em>1.0</em>.</p><p>This is called the <a href="https://robots.thoughtbot.com/rubys-pessimistic-operator" target="_blank" rel="noopener">Ruby’s Pessimistic Operator</a>, the <a href="https://guides.rubygems.org/patterns/#pessimistic-version-constraint" target="_blank" rel="noopener">twiddle-wakka</a> or the <a href="https://github.com/rubygems/rubygems/pull/123" target="_blank" rel="noopener"><em>spermy operator</em></a> if you prefer it that way.</p><p>Now that we have defined our requirements we can fetch the lastest gem versions available for us using the <code>update</code> command in Bundler.</p><pre><code>$ bundle updateFetching gem metadata from https://rubygems.org/.Resolving dependencies...Using bundler 1.16.4Fetching thor 0.20.0Installing thor 0.20.0Bundle updated!</code></pre><p>Now, that we have fetch the latest versions (within our version constrain define in the Gemfile, of course) we can proceed to install them using the <code>install</code> command.</p><pre><code>$ bundle installUsing bundler 1.16.4Using thor 0.20.0Bundle complete! 1 Gemfile dependency, 2 gems now installed.Use `bundle info [gemname]` to see where a bundled gem is installed</code></pre><p>This will generate a <code>Gemfile.lock</code> file that will pin our gem versions.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">GEM</span><br><span class="line">  <span class="symbol">remote:</span> <span class="symbol">https:</span>/<span class="regexp">/rubygems.org/</span></span><br><span class="line">  <span class="symbol">specs:</span></span><br><span class="line">    thor (<span class="number">0</span>.<span class="number">20.0</span>)</span><br><span class="line"></span><br><span class="line">PLATFORMS</span><br><span class="line">  ruby</span><br><span class="line"></span><br><span class="line">DEPENDENCIES</span><br><span class="line">  thor (~&gt; <span class="number">0</span>.<span class="number">20</span>)</span><br><span class="line"></span><br><span class="line">BUNDLED WITH</span><br><span class="line">   <span class="number">1.16</span>.<span class="number">4</span></span><br></pre></td></tr></table></figure><p>You can see here that there’s specifications about the version numbers of our dependencies, platforms, the remote server where we’re going to retrieve our gems and the bundler version.</p><p>Now we’re able to use the <code>thor</code> gem in our library. To our surprise, this gem comes also with a <a href="https://en.wikipedia.org/wiki/Command-line_interface" target="_blank" rel="noopener">command line interface</a> (CLI) that we can use.</p><p>To execute it, we have to do it under the environment that bundler has prepared for us. We can execute it using the <code>exec</code> command in bundler.</p><pre><code>$ bundle exec thorCommands:  thor help [COMMAND]  # Describe available commands or one specific command  thor install NAME    # Install an optionally named Thor file into your system commands  thor installed       # List the installed Thor modules and commands  thor list [SEARCH]   # List the available thor commands (--substring means .*SEARCH)  thor uninstall NAME  # Uninstall a named Thor module  thor update NAME     # Update a Thor file from its original location  thor version         # Show Thor version</code></pre><p>Awesome! Now we know how to define requirements and versions constrains within our application’s project, we can go and create a directory structure for our gem. But beware, developing gems is a <em>bit different</em> than developing applications with Ruby.</p><h1 id="Creating-our-project"><a href="#Creating-our-project" class="headerlink" title="Creating our project"></a>Creating our project</h1><h2 id="Project-description"><a href="#Project-description" class="headerlink" title="Project description"></a>Project description</h2><p>What we’re going to create is a gem with a command line interface called DiceMyPass (or <strong>DMP</strong>, from now on). This gem will provide you with a secure passphrase extracted from EFF’s long wordlist with an optional length. For example:</p><pre><code>$ dmp gen- Passphrase: slashed uncharted evoke placard outweigh revision</code></pre><p>Additionally, we’re going to add an option to our <code>gen</code> (read generate) command that will check if our newly generated passphrase was found on a dataset on <a href="https://haveibeenpwned.com/" target="_blank" rel="noopener">HIBP</a> to check if its vulnerable. For example:</p><pre><code>$ dmp gen --hibp- Passphrase: rockstar brunt stunt remindful astronaut bats- Password was not found in a dataset.</code></pre><p>And lastly, we’re going to add a option to copy the new passphrase to the clipboard with the flag <code>--clipboard</code>.</p><h2 id="Creating-a-gem-with-bundler"><a href="#Creating-a-gem-with-bundler" class="headerlink" title="Creating a gem with bundler"></a>Creating a gem with bundler</h2><p>Now we’re going to get our hands dirty, you might think that we will have to create a directory structure and a Gemfile for our Ruby gem DMP, fortunately bundler got things covered for us and is able to scaffold one for us.</p><p>Allow bundler to create a scaffold of your project using the <code>gem</code> command in bundler. It’s going to ask you a couple of questions, they’re all important but when it asks you about testing then you should write <strong>minitest</strong>, which is a gem that will help us to test the functionality of our gem.</p><pre><code>$ bundler gem dmpCreating gem &#39;dmp&#39;...MIT License enabled in configCode of conduct enabled in config      create  dmp/Gemfile      create  dmp/lib/dmp.rb      create  dmp/lib/dmp/version.rb      create  dmp/dmp.gemspec      create  dmp/Rakefile      create  dmp/README.md      create  dmp/bin/console      create  dmp/bin/setup      create  dmp/.gitignore      create  dmp/.travis.yml      create  dmp/test/test_helper.rb      create  dmp/test/dmp_test.rb      create  dmp/LICENSE.txt      create  dmp/CODE_OF_CONDUCT.mdInitializing git repo in /home/franccesco/workspace/dmpGem &#39;dmp&#39; was successfully created.</code></pre><p>This will generate a directory structure and there’s a couple of files that requires your attention.</p><table><thead><tr><th align="left">File</th><th align="right">Description</th></tr></thead><tbody><tr><td align="left">Gemfile</td><td align="right">Gemfile holding our <strong>application dependencies</strong></td></tr><tr><td align="left"><strong>dmp.gemspec</strong></td><td align="right">Gemspec holding our <strong>gem dependencies</strong></td></tr><tr><td align="left">Rakefile</td><td align="right"><strong>Rake commands</strong> to handle our build cycle</td></tr><tr><td align="left">CODE_OF_CONDUCT.md</td><td align="right">Code of Conduct to let people know <strong>how to contribute</strong></td></tr><tr><td align="left">LICENSE.txt</td><td align="right"><strong>License your project</strong> under the MIT license</td></tr><tr><td align="left"><strong>.gitignore</strong></td><td align="right">List of <strong>files excluded</strong> from version control (git)</td></tr><tr><td align="left"><strong>lib/dmp.rb</strong></td><td align="right">The file where we’re going to develop our gem</td></tr><tr><td align="left">lib/dmp/version.rb</td><td align="right">Here we’re going to <strong>bump the version number of our gem</strong></td></tr></tbody></table><p>Obviously, there are others that are also important, but we’re going to see them in other posts.</p><p>Now, we have to define the dependencies of our project, but hold on, we’re not using the <strong>Gemfile</strong> to define our dependencies in our gem, we’re using the <code>.gemspec</code> here.</p><p>This is because there’s a difference between developing a gem and developing an application. I’m not going through the details about the differences, you can find that in <a href="https://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/" target="_blank" rel="noopener">this excellent article made by Yehuda Katz</a>.</p><p>To make it easier for you, just remember:</p><ul><li>When developing an app: Use the <strong>Gemfile.</strong></li><li>When developing a gem: Use the <strong>gemspec</strong>.</li></ul><p>Let’s open up our gemspec and fill it with the necessary information, remember to replace the <code>TODO</code>‘s with relevant information.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">lib = File.expand_path(<span class="string">"../lib"</span>, __FILE_<span class="number">_</span>)</span><br><span class="line">$LOAD_PATH.unshift(lib) <span class="keyword">unless</span> $LOAD_PATH.<span class="keyword">include</span>?(lib)</span><br><span class="line"><span class="keyword">require</span> <span class="string">"dmp/version"</span></span><br><span class="line"></span><br><span class="line">Gem::Specification.new <span class="keyword">do</span> <span class="params">|spec|</span></span><br><span class="line">  spec.name          = <span class="string">"dmp"</span></span><br><span class="line">  spec.version       = Dmp::VERSION</span><br><span class="line">  spec.authors       = [<span class="string">"Franccesco Orozco"</span>]</span><br><span class="line">  spec.email         = [<span class="string">"franccesco@codingdose.info"</span>]</span><br><span class="line"></span><br><span class="line">  spec.summary       = <span class="string">%q&#123;Generate a secure passphrase.&#125;</span></span><br><span class="line">  spec.description   = <span class="string">%q&#123;Generates a passphrase using EFF's long wordlist.&#125;</span></span><br><span class="line">  spec.homepage      = <span class="string">"https://github.com/franccesco/dmp"</span></span><br><span class="line">  spec.license       = <span class="string">"MIT"</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># Specify which files should be added to the gem when it is released.</span></span><br><span class="line">  <span class="comment"># The `git ls-files -z` loads the files in the RubyGem that have been added into git.</span></span><br><span class="line">  spec.files         = Dir.chdir(File.expand_path(<span class="string">'..'</span>, __FILE_<span class="number">_</span>)) <span class="keyword">do</span></span><br><span class="line">    <span class="string">`git ls-files -z`</span>.split(<span class="string">"\x0"</span>).reject &#123; <span class="params">|f|</span> f.match(<span class="regexp">%r&#123;^(test|spec|features)/&#125;</span>) &#125;</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line">  spec.bindir        = <span class="string">"exe"</span></span><br><span class="line">  spec.executables   = spec.files.grep(<span class="regexp">%r&#123;^exe/&#125;</span>) &#123; <span class="params">|f|</span> File.basename(f) &#125;</span><br><span class="line">  spec.require_paths = [<span class="string">"lib"</span>]</span><br><span class="line"></span><br><span class="line">  <span class="comment"># add dependencies</span></span><br><span class="line">  spec.add_dependency <span class="string">'thor'</span>, <span class="string">'~&gt; 0'</span></span><br><span class="line">  spec.add_dependency <span class="string">'colorize'</span>, <span class="string">'~&gt; 0.8'</span></span><br><span class="line">  spec.add_dependency <span class="string">'clipboard'</span>, <span class="string">'~&gt; 1.1'</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># add dependencies specially for development needs</span></span><br><span class="line">  spec.add_development_dependency <span class="string">"bundler"</span>, <span class="string">"~&gt; 1.16"</span></span><br><span class="line">  spec.add_development_dependency <span class="string">"rake"</span>, <span class="string">"~&gt; 10.0"</span></span><br><span class="line">  spec.add_development_dependency <span class="string">"irb"</span>, <span class="string">"~&gt; 1.0.0"</span></span><br><span class="line">  spec.add_development_dependency <span class="string">"minitest"</span>, <span class="string">"~&gt; 5.0"</span></span><br><span class="line">  spec.add_development_dependency <span class="string">"minitest-reporters"</span>, <span class="string">"~&gt; 1.3"</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>As you can see, we have define our dependencies. <strong>Thor</strong> to handle our command line interface, <strong>Colorize</strong> to print colored strings to the command line, and <strong>Clipboard</strong> to copy the output to our clipboard automatically.</p><p>For our development dependencies we have added <strong>minitest-reporters</strong> which displays a nice report of our tests right in the command line.</p><p>As you can see, they all have version constrains to avoid that our gem breaks when the dependencies are updated with major versions and incompatible changes. Let’s install our dependencies.</p><pre><code>$ bundle updateFetching gem metadata from https://rubygems.org/........Resolving dependencies...Using rake 10.5.0Using ansi 1.5.0Using builder 3.2.3Using bundler 1.16.4Using clipboard 1.1.2Using colorize 0.8.1Using thor 0.20.0Using dmp 0.1.0 from source at `.`Using minitest 5.11.3Using ruby-progressbar 1.10.0Using minitest-reporters 1.3.4Bundle updated!$ bundle installUsing rake 10.5.0Using ansi 1.5.0Using builder 3.2.3Using bundler 1.16.4Using clipboard 1.1.2Using colorize 0.8.1Using thor 0.20.0Using dmp 0.1.0 from source at `.`Using minitest 5.11.3Using ruby-progressbar 1.10.0Using minitest-reporters 1.3.4Bundle complete! 5 Gemfile dependencies, 11 gems now installed.Use `bundle info [gemname]` to see where a bundled gem is installed.</code></pre><p>There you go, we have now installed our dependencies, and if you look closely, we have also installed our gem in development mode.</p><pre><code>Using dmp 0.1.0 from source at `.`</code></pre><p>After we have updated and installed our dependencies, it is important that we exclude the newly generated <code>Gemfile.lock</code> from version control, as we’re developing a gem, not an application, and we’re not trying to replicate our environment in another one.</p><pre><code>$ echo Gemfile.lock &gt;&gt; .gitignore</code></pre><p>Now that we have excluded the lock file, let’s make a simple test before making our first commit.</p><p>Open up <strong>test/dmp_test.rb</strong> and let’s fill it with the following content.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">"test_helper"</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'dmp'</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DmpTest</span> &lt; Minitest::Test</span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">test_that_it_has_a_version_number</span></span></span><br><span class="line">    refute_nil <span class="symbol">:</span><span class="symbol">:Dmp</span><span class="symbol">:</span><span class="symbol">:VERSION</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">test_say_hi</span></span></span><br><span class="line">    assert_equal Dmp.say_hi(<span class="string">'Franccesco'</span>), <span class="string">'Hello, Franccesco!'</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Now, we can see that we have 2 tests here. The first one test if our module has a <code>VERSION</code> number and if it <em>doesn’t</em> have this constant in our module then it should complain. We can check this out in the dmp module.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  VERSION = <span class="string">"0.1.0"</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Now for the other one, we have written a test that checks that the result of our module function <code>say_hi(&#39;Franccesco&#39;)</code> returns <code>Hello, Franccesco!</code>. As we haven’t written any modules yet, it will fail. Let’s respect the <em>red, green, refactor</em> cycle and let’s make it fail.</p><p>For this, let’s run <code>rake test</code> to begin our minitests.</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> rake <span class="built_in">test</span></span><br><span class="line">  1) Error:</span><br><span class="line">DmpTest<span class="comment">#test_say_hi:</span></span><br><span class="line">NoMethodError: undefined method `say_hi<span class="string">' for Dmp:Module</span></span><br><span class="line"><span class="string">    /home/franccesco/dmp/test/dmp_test.rb:10:in `test_say_hi'</span></span><br><span class="line"></span><br><span class="line">2 runs, 1 assertions, 0 failures, 1 errors, 0 skips</span><br></pre></td></tr></table></figure><p>Minitest complains that it cannot find the method <code>say_hi</code>, this is because we haven’t created our module yet. Let’s create it right now on <strong>lib/dmp.rb</strong> to make it pass.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># lib/dmp.rb</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">"dmp/version"</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">say_hi</span><span class="params">(name)</span></span></span><br><span class="line">    <span class="string">"Hello, <span class="subst">#&#123;name&#125;</span>!"</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>There you go! We have written a simple module that takes a name as a parameter and returns a salute with your name of choice. Let’s run the test again.</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> rake <span class="built_in">test</span></span><br><span class="line"><span class="comment"># Running:</span></span><br><span class="line"></span><br><span class="line">..</span><br><span class="line"></span><br><span class="line">Finished <span class="keyword">in</span> 0.000661s, 3027.9266 runs/s, 3027.9266 assertions/s.</span><br><span class="line"></span><br><span class="line">2 runs, 2 assertions, 0 failures, 0 errors, 0 skips</span><br></pre></td></tr></table></figure><p>All good now! Our module returns no failures now, meaning that our module successfully returns a salute with our name. Now, our tests are fine, but we can make our minitest report more friendly!</p><h2 id="Friendlier-reports-with-minitest-reports"><a href="#Friendlier-reports-with-minitest-reports" class="headerlink" title="Friendlier reports with minitest-reports"></a>Friendlier reports with minitest-reports</h2><p>Our <em>red-green-refactor</em> cycle is not showing neither red or green yet, so let’s add that to our environment. We can modify the behavior and presentation of our test opening the test helper found in <strong>test/test_helper.rb</strong>.</p><p>Just before, we added a development dependency in our <code>gemspec</code>, a gem called <a href="https://github.com/kern/minitest-reporters" target="_blank" rel="noopener">minitest-reporters</a> which modifies the presentation in our reports, open <strong>test/test_helper.rb</strong> and spin up the minitest-reporter gem.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">$LOAD_PATH.unshift File.expand_path(<span class="string">"../../lib"</span>, __FILE_<span class="number">_</span>)</span><br><span class="line"><span class="keyword">require</span> <span class="string">"dmp"</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">"minitest/autorun"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># add default progress bar to reports</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'minitest/reporters'</span></span><br><span class="line"></span><br><span class="line">Minitest::Reporters.use!</span><br></pre></td></tr></table></figure><p>This will use the default progress bar reporter when we run our test. To try it out, let’s open up <strong>lib/dmp.rb</strong> and modify our code to make it fail.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">"dmp/version"</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">say_hi</span><span class="params">(name)</span></span></span><br><span class="line">    <span class="string">"Hello, Palmer!"</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>We now that our test is expecting to return another name, the one that we provide, but it will return <strong>Palmer</strong> instead, this way our test should represent a failed test with a red progress bar.</p><p><img src="/assets/images/bundler/test_fail.png" alt="red test"></p><p>There we go! this step is not entirely necessary for the development of our projects but it’s surely a nice addition as it adds visual aid and also makes testing a lot more enjoyable for sure. Let’s make the test pass, shall we?</p><p>Let’s fix our method <code>say_hi</code> so it returns our name instead of <strong>Palmer</strong>, you know how to do that ;). After that, our test should be green now.</p><p><img src="/assets/images/bundler/test_pass.png" alt="green test"></p><p>Great! Now that we have setup our minitests correctly, we can move on and make a more robust test, one that actually will test the functionality of our project, but before we write those tests, it would be useful to explain how the HaveIBeenPwned API works.</p><h2 id="HaveIBeenPwned-API"><a href="#HaveIBeenPwned-API" class="headerlink" title="HaveIBeenPwned API"></a>HaveIBeenPwned API</h2><p>In order to write the test, we have to learn how the HaveIBeenPwned API works first, and it is actually not difficult at all. As we only need to access a certain function of the API, we don’t need to learn every aspect of the API itself, only the one to check if our password is vulnerable.</p><p>You can find the documentation of this aspect of the API <a href="https://haveibeenpwned.com/API/v2#PwnedPasswords" target="_blank" rel="noopener">clicking here</a>, but let me give you an overview of how it works and what kind of requests we can submit.</p><p>Let’s say that our password is <em>‘passw0rd’</em>, but if we want to check this password through the API, we cannot submit the password in clear text as this would be an insecure practice. Instead we have to submit a <em>partial</em> hash encoded in SHA-1 which are the first 5 characters of the hash and then submit it to:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">https:&#x2F;&#x2F;api.pwnedpasswords.com&#x2F;range&#x2F;&#123;first 5 hash chars&#125;</span><br></pre></td></tr></table></figure><p>This will return a list of suffixes of all the hashes that matches the first 5 characters of our hash, followed by a count of how many that hash was found in vulnerable datasets. Here are the simplified steps:</p><ol><li>We encode our ‘passw0rd’ string to SHA-1 which would be <code>7C6A61C68EF8B9B6B061B28C348BC1ED7921CB53</code></li><li>We submit the first 5 characters of our hash <code>7C6A6</code> to the API, which would be <a href="https://api.pwnedpasswords.com/range/7C6A6" target="_blank" rel="noopener">https://api.pwnedpasswords.com/range/7C6A6</a></li><li>It will return a long list of hashes, we only need to find the suffix of our hash, which would be <code>1C68EF8B9B6B061B28C348BC1ED7921CB53</code></li></ol><p>Example:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line"># If the password is secure, we wouldn&#39;t be able</span><br><span class="line"># to find the suffix of the hash here.</span><br><span class="line"># This is clearly not the case.</span><br><span class="line"></span><br><span class="line">-- SNIP --</span><br><span class="line">1BC4E6F00BECB5998201277DC62F89E08B0:7</span><br><span class="line">1BC5AF255E721AF1C4AA83FD0F8EE8A79B8:3</span><br><span class="line">1C68EF8B9B6B061B28C348BC1ED7921CB53:216221 &lt;&lt;- Here</span><br><span class="line">1CEA692E5FA3ED23B956839B4B8BFCCC5F5:4</span><br><span class="line">1DC4A0F7305069370733B17882579EBDF4E:3</span><br><span class="line">-- SNIP --</span><br></pre></td></tr></table></figure><p>This method is called the <a href="https://en.wikipedia.org/wiki/K-anonymity" target="_blank" rel="noopener">K-Anonymity model</a>, and as we can see, the password <em>‘passw0rd’</em> was found in 216221 datasets. This makes it obvious that this is not a secure password at all, so let’s implement this functionality into our code, shall we?</p><h2 id="Generating-a-secure-passphrase"><a href="#Generating-a-secure-passphrase" class="headerlink" title="Generating a secure passphrase"></a>Generating a secure passphrase</h2><p>As already stated in our project description, we’re going to create a single gem that generates a passphrase and also it checks if the generated password is a <em>known vulnerable password</em>.</p><p>Right now, we’re going to delete our previous <code>say_hi</code> test and we’re going to create three more empty tests that we will fill-out eventually. Here’s how the tests should look:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">"test_helper"</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'dmp'</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DmpTest</span> &lt; Minitest::Test</span></span><br><span class="line">  <span class="comment"># def setup; end</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">test_that_it_has_a_version_number</span></span></span><br><span class="line">    refute_nil <span class="symbol">:</span><span class="symbol">:Dmp</span><span class="symbol">:</span><span class="symbol">:VERSION</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># def test_gen_passphrase; end</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># def test_vulnerable_pass; end</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># def test_secure_pass; end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Now, we can see that we have three more tests and a setup method:</p><ol><li>First, we’re going to test if our module (dmp) generates a secure passphrase.</li><li>We’re going to check if our generated passphrase is secure enough.</li><li>Lastly, We’re going to test if our program alerts us if a password is vulnerable.</li></ol><p>Let’s fill-out <code>test_gen_passphrase</code> first. As we know, when we create tests we create <em>expectations</em>. Following this idea, we’re going to define how our program should create our secure passphrase. Our “secure” passphrase for these tests will be <em>“coding dose dot com”</em> in the meantime.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test_gen_passphrase</span></span></span><br><span class="line">  <span class="comment"># gen_passphrase should generate and respect prassphrase length</span></span><br><span class="line">  passphrase3 = Dmp.gen_passphrase(<span class="number">3</span>)</span><br><span class="line">  passphrase_default = Dmp.gen_passphrase</span><br><span class="line">  passphrase12 = Dmp.gen_passphrase(<span class="number">12</span>)</span><br><span class="line">  assert_equal passphrase3.length, <span class="number">3</span>, <span class="string">'Passphrase length != 3'</span></span><br><span class="line">  assert_equal passphrase_default.length, <span class="number">7</span>, <span class="string">'Passphrase length != 7'</span></span><br><span class="line">  assert_equal passphrase12.length, <span class="number">12</span>, <span class="string">'Passphrase length != 12'</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>There’s a lot going on here, huh? No worries though, this is more of the same. Let’s analyze the first variable declaration and the first assertion:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Here we generate a passphrase which will consist on three words</span></span><br><span class="line">passphrase3 = Dmp.gen_passphrase(<span class="number">3</span>)</span><br><span class="line"></span><br><span class="line">-- SNIP --</span><br><span class="line"></span><br><span class="line"><span class="comment"># Now we check if the passphrase previously created has a length of three words.</span></span><br><span class="line">assert_equal passphrase3.length, <span class="number">3</span>, <span class="string">'Passphrase length != 3'</span></span><br></pre></td></tr></table></figure><p>If the passphrase is not equal to <em>three</em> words then it complains with the message <em>‘Passphrase length != 3’</em>. Let’s run this test.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="symbol">NoMethodError:</span> undefined method <span class="string">`gen_passphrase' for Dmp:Module</span></span><br></pre></td></tr></table></figure><p>Our first expectation did not run correctly, but of course, this is what we’re looking for. So as there’s no method named <em>gen_passphrase</em> in our module yet, we’ll have to create it first. But before we do that, we’ll have to look for a dictionary that gives us a list of words which we’ll use to generate a passphrase consisting in 3, 4, 8, 100 words if we need to. You can find the list here: <a href="https://raw.githubusercontent.com/franccesco/dmp/master/lib/dmp/assets/eff_long_wordlist.txt" target="_blank" rel="noopener">eff_long_wordlist.txt</a></p><p>Now, let’s save this file in <code>lib/dmp/assets/eff_long_wordlist.txt</code>.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ mkdir lib/dmp/assets</span><br><span class="line">$ wget -O lib/dmp/assets/eff_long_wordlist.txt https://raw.githubusercontent.com/franccesco/dmp/master/lib/dmp/assets/eff_long_wordlist.txt</span><br></pre></td></tr></table></figure><p>Now that we have saved the dictionary in the assets folder, we can open up our module and code the core functionality of our gem, which is to generate a secure passphrase using this dictionary.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="comment"># First we load the absolute path of our eff_long_wordlist.txt.</span></span><br><span class="line">  @eff_wordlist = File.dirname(__FILE_<span class="number">_</span>) + <span class="string">'/dmp/assets/eff_long_wordlist.txt'</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># The default passphrase length should be 7</span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">gen_passphrase</span><span class="params">(pass_length = <span class="number">7</span>)</span></span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># Read filename eff_long_wordlist and save it as a list.</span></span><br><span class="line">    wordlist = File.readlines(@eff_wordlist)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># Strip the '\n' out of every line.</span></span><br><span class="line">    wordlist.map(&amp;<span class="symbol">:strip!</span>)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># Shuffle the list and return a list up to pass_length words</span></span><br><span class="line">    <span class="comment"># which in the case would be equal to 7 words.</span></span><br><span class="line">    wordlist.shuffle[<span class="number">0</span>...pass_length]</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>This should be pretty easy:</p><ol><li>We create an instance variable holding the absolute path of our wordlist</li><li>Define our method <em>gen_passphrase</em> with a default length of 7</li><li>Load our wordlist as a list and hold it in the <em>wordlist</em> variable.</li><li>Strip every ‘\n’ in each word</li><li>Lastly, we scramble the words and return a list of words with our desired length.</li></ol><p>Let’s test it out:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">Started with run options --seed 2535</span><br><span class="line"></span><br><span class="line">  2/2: [==========================================================================] 100% Time: 00:00:00, Time: 00:00:00</span><br><span class="line"></span><br><span class="line">Finished <span class="keyword">in</span> 0.01531s</span><br><span class="line">2 tests, 4 assertions, 0 failures, 0 errors, 0 skips</span><br></pre></td></tr></table></figure><p>Awesome! Our code works! But how can we actually check how it works? We’ll we can definitely import it into IRB and try it out with <code>bundle exec irb</code>:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="meta">irb(main):001:0&gt;</span> require <span class="string">'dmp'</span></span><br><span class="line">=&gt; true</span><br><span class="line"><span class="meta">irb(main):002:0&gt;</span> Dmp.gen_passphrase</span><br><span class="line">=&gt; [<span class="string">"autopilot"</span>, <span class="string">"ivy"</span>, <span class="string">"overlay"</span>, <span class="string">"down"</span>, <span class="string">"visitor"</span>, <span class="string">"prenatal"</span>, <span class="string">"flirt"</span>]</span><br><span class="line"><span class="meta">irb(main):003:0&gt;</span> Dmp.gen_passphrase(<span class="number">3</span>)</span><br><span class="line">=&gt; [<span class="string">"roulette"</span>, <span class="string">"earthen"</span>, <span class="string">"garbage"</span>]</span><br><span class="line"><span class="meta">irb(main):004:0&gt;</span> Dmp.gen_passphrase(<span class="number">6</span>)</span><br><span class="line">=&gt; [<span class="string">"tartly"</span>, <span class="string">"happier"</span>, <span class="string">"juice"</span>, <span class="string">"itunes"</span>, <span class="string">"job"</span>, <span class="string">"eastward"</span>]</span><br></pre></td></tr></table></figure><p>You see? Now we can create a secure passphrase with our method. Let’s move onto the next test.</p><h2 id="Checking-passphrases-with-HIBP-API"><a href="#Checking-passphrases-with-HIBP-API" class="headerlink" title="Checking passphrases with HIBP API"></a>Checking passphrases with HIBP API</h2><p>Let’s write the test of our method to check if a passphrase or password is vulnerable (read <em>found in a dataset</em>).</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test_vulnerable_pass</span></span></span><br><span class="line">  <span class="comment"># check_pwned should flag this password</span></span><br><span class="line">  vuln_count = Dmp.check_pwned(<span class="string">'passw0rd'</span>)</span><br><span class="line">  refute_nil vuln_count</span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Let’s analyze the first piece of code.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">vuln_count = Dmp.check_pwned(<span class="string">'passw0rd'</span>)</span><br></pre></td></tr></table></figure><p>It gets clearer if we read this code in <em>reverse</em>. We have the password <code>&#39;passw0rd&#39;</code> that we want to check if it’s unsafe using the method <code>check_pwned</code> that belongs to the module <code>Dmp</code> and we want to hold the value or output of this action to the variable <code>vuln_count</code>. This output should tell us in how many datasets was the password found.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">assert_nil vuln_count</span><br></pre></td></tr></table></figure><p>As our password is not secure at all, then the value of <em>vuln_count</em> should <strong>not</strong> be <code>nil</code>, this is because we out method <em>check_pwned</em> should find the suffix of our hash in the HaveIBeenPwned (HIBP) datasets. With this in mind, we test the the value of <em>vuln_count</em> should not be <code>nil</code> with <code>refute_nil</code>. Let’s run the test.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="symbol">NoMethodError:</span> undefined method <span class="string">`check_pwned' for Dmp:Module</span></span><br></pre></td></tr></table></figure><p>As there’s no method named <strong>check_pwned</strong> then our test complains, let’s open up the module <em>dmp.rb</em> and fill-out the code.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="symbol">NoMethodError:</span> undefined method <span class="string">`check_pwned' for Dmp:Module</span></span><br></pre></td></tr></table></figure><p>No method? no problem, let’s fill it out in our module.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">"dmp/version"</span></span><br><span class="line"><span class="comment"># require SHA-1 digest and http utilities</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'digest/sha1'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'net/http'</span></span><br><span class="line">mod</span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">self</span>.<span class="title">check_pwned</span><span class="params">(passphrase)</span></span></span><br><span class="line">    <span class="comment"># If the passphrase is an array generated by gen_passphrase we convert</span></span><br><span class="line">    <span class="comment"># the passphrase to an unified string, if it's a string already then</span></span><br><span class="line">    <span class="comment"># no changes are applied to the passphrase variable.</span></span><br><span class="line">    passphrase = passphrase.join(<span class="string">' '</span>) <span class="keyword">if</span> passphrase.is_a?(Array)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># We encode our passphrase to SHA-1, and save or prefix consisting</span></span><br><span class="line">    <span class="comment"># in 5 characters to the variable sha1_excerpt and the suffix to</span></span><br><span class="line">    <span class="comment"># the variable sha1_to_look_for.</span></span><br><span class="line">    sha1_pass = Digest::SHA1.hexdigest(passphrase)</span><br><span class="line">    sha1_excerpt = sha1_pass[<span class="number">0</span>...<span class="number">5</span>]</span><br><span class="line">    sha1_to_look_for = sha1_pass[<span class="number">5</span>..-<span class="number">1</span>]</span><br><span class="line"></span><br><span class="line">    <span class="comment"># We make the API call with our SHA-1 prefix and store the response to</span></span><br><span class="line">    <span class="comment"># the variable api_request</span></span><br><span class="line">    api_url = URI(<span class="string">"https://api.pwnedpasswords.com/range/<span class="subst">#&#123;sha1_excerpt&#125;</span>"</span>)</span><br><span class="line">    api_request = Net::HTTP.get(api_url)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="comment"># The response is text instead of JSON, needs to format the response</span></span><br><span class="line">    <span class="comment"># to a dictionary so the rest of the hash can be located easier.</span></span><br><span class="line">    <span class="comment"># =&gt; String '0018A45C4D1DEF81644B54AB7F969B88D65:21'</span></span><br><span class="line">    <span class="comment"># =&gt; Array ['0018A45C4D1DEF81644B54AB7F969B88D65:21', ...]</span></span><br><span class="line">    <span class="comment"># =&gt; 2D Array [['0018A45C4D1DEF81644B54AB7F969B88D65', '21'], ...]</span></span><br><span class="line">    <span class="comment"># =&gt; Hash &#123;'0018A45C4D1DEF81644B54AB7F969B88D65': 21, ...&#125;</span></span><br><span class="line">    striped_list = api_request.split(<span class="string">"\r\n"</span>)</span><br><span class="line">    pass_list = striped_list.map &#123; <span class="params">|hash|</span> hash.split(<span class="string">':'</span>) &#125;</span><br><span class="line">    hash_list = Hash[*pass_list.flatten!]</span><br><span class="line">    hash_list[sha1_to_look_for.upcase]</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Now there’s a lot going on here, let’s simplify the steps:</p><ol><li>The passphrase should be a <strong>string</strong>, as in <em>roulette earthen garbage”</em>, but if the password was generated by <code>gen_passphrase</code> then it will return an array, only in that case we convert it to a string.</li><li>We use the digest module to convert our string to SHA-1, we save the 5 characters prefix to <code>sha_excerpt</code> and the suffix to <code>sha1_to_look_for</code>.</li><li>We perform a <strong>GET</strong> request to the API using the prefix of our hash. The response should be a text list as we saw previously.</li><li>We have to format that list in a way that we can <strong>search</strong> for our hash suffix and have the value of how many times the password is found in datasets.</li></ol><p>And this is the slightly tricky part, when we receive the response from the API, it doesn’t provide us with a pretty JSON response which we can work on, it provides us with a bare list like this:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">003E7C1C94342454421573ADECD156C6AE8:2\r\n00A4DB094C56008C81D9DA2C55166F1A5BA:4\r\n00F042A842B821E2F727B0A4A3C0555E4A0:2\r\n01F14311110773C8064336D0D52736141D2:3\r\n01F6581B8152E00CBA4F8261335A78DA26F:1\r\n020290C96F182C924647A747F21681697B9:2\r\n02146D9588F55A6751CE580AA1AC6E16106:2\r\n02C2409C5E2AAC99D2937CAB31EB4677EAD:2\r\n02EFB814079D668ACF7308FAA18583D8CED:2\r\n033211C0B3B8B0EBC0BFDF2000CE0FFA166:1\r\n0378E7D9BC61CE282E9664D404505F66457:1\r\n03D801A3E713009943D0A76217278ABE2DD:3\r\n0412EEBFCB315371F4CDEAEB3AFDBEA43CD:1...</span><br></pre></td></tr></table></figure><p>Pretty, right? (Sarcasm intended) now, I’m sure that there are better ways to convert this mess into a dictionary, but for clarity and brevity we will not get into a regex mind-boggling tricks right now. Let’s try this piece of code in IRB. First, let’s remove all the \r\n’s that we can find with <code>striped_list = api_request.split(&quot;\r\n&quot;)</code>.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="meta">irb(main):009:0&gt;</span> striped_list = api_request.split(<span class="string">"\r\n"</span>)</span><br><span class="line">=&gt; [<span class="string">"003E7C1C94342454421573ADECD156C6AE8:2"</span>, <span class="string">"00A4DB094C56008C81D9DA2C55166F1A5BA:4"</span>, <span class="string">"00F042A842B821E2F727B0A4A3C0555E4A0:2"</span>, <span class="string">"01F14311110773C8064336D0D52736141D2:3"</span>, <span class="string">"01F6581B8152E00CBA4F8261335A78DA26F:1"</span>, <span class="string">"020290C96F182C924647A747F21681697B9:2"</span>, <span class="string">"02146D9588F55A6751CE580AA1AC6E16106:2"</span>, <span class="string">"02C2409C5E2AAC99D2937CAB31EB4677EAD:2"</span>, <span class="string">"02EFB814079D668ACF7308FAA18583D8CED:2"</span>, <span class="string">"033211C0B3B8B0EBC0BFDF2000CE0FFA166:1"</span>, <span class="string">"..."</span>]</span><br></pre></td></tr></table></figure><p>Good, now let’s map that list and create a 2D Array:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="meta">irb(main):010:0&gt;</span> pass_list = striped_list.map &#123; <span class="params">|hash|</span> hash.split(<span class="string">':'</span>) &#125;</span><br><span class="line">=&gt; [[<span class="string">"003E7C1C94342454421573ADECD156C6AE8"</span>, <span class="string">"2"</span>], [<span class="string">"00A4DB094C56008C81D9DA2C55166F1A5BA"</span>, <span class="string">"4"</span>], [<span class="string">"00F042A842B821E2F727B0A4A3C0555E4A0"</span>, <span class="string">"2"</span>], [<span class="string">"01F14311110773C8064336D0D52736141D2"</span>, <span class="string">"3"</span>], [<span class="string">"..."</span>],</span><br><span class="line"></span><br><span class="line"><span class="comment"># This individualizes our suffixes:</span></span><br><span class="line"><span class="meta">irb(main):011:0&gt;</span> pass_list[<span class="number">0</span>]</span><br><span class="line">=&gt; [<span class="string">"003E7C1C94342454421573ADECD156C6AE8"</span>, <span class="string">"2"</span>]</span><br><span class="line"><span class="meta">irb(main):013:0&gt;</span> pass_list[<span class="number">15</span>]</span><br><span class="line">=&gt; [<span class="string">"04BC55FD524B3E42D6A732E2EA8076A9178"</span>, <span class="string">"5"</span>]</span><br></pre></td></tr></table></figure><p>Perfect… well not quite so, let’s create a dictionary out of the 2D array.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="meta">irb(main):014:0&gt;</span> hash_list = Hash[*pass_list.flatten!]</span><br><span class="line">=&gt; &#123;<span class="string">"003E7C1C94342454421573ADECD156C6AE8"</span>=&gt;<span class="string">"2"</span>, <span class="string">"00A4DB094C56008C81D9DA2C55166F1A5BA"</span>=&gt;<span class="string">"4"</span>, <span class="string">"00F042A842B821E2F727B0A4A3C0555E4A0"</span>=&gt;<span class="string">"2"</span>, <span class="string">"01F14311110773C8064336D0D52736141D2"</span>=&gt;<span class="string">"3"</span>, <span class="string">"01F6581B8152E00CBA4F8261335A78DA26F"</span>=&gt;<span class="string">"1"</span>, <span class="string">"020290C96F182C924647A747F21681697B9"</span>=&gt;<span class="string">"2"</span>, <span class="string">"02146D9588F55A6751CE580AA1AC6E16106"</span>=&gt;<span class="string">"2"</span>, <span class="string">"02C2409C5E2AAC99D2937CAB31EB4677EAD"</span>=&gt;<span class="string">"2"</span>, <span class="string">"02EFB814079D668ACF7308FAA18583D8CED"</span>=&gt;<span class="string">"2"</span>, <span class="string">"033211C0B3B8B0EBC0BFDF2000CE0FFA166"</span>=&gt;<span class="string">"1"</span>, <span class="string">"0378E7D9BC61CE282E9664D404505F66457"</span>=&gt;<span class="string">"1"</span>, <span class="string">"03D801A3E713009943D0A76217278ABE2DD"</span>=&gt;<span class="string">"3"</span>, <span class="string">"0412EEBFCB315371F4CDEAEB3AFDBEA43CD"</span>=&gt;<span class="string">"1"</span>, <span class="string">"0422590C0BC43132207FF55FD78717074A4"</span>=&gt;<span class="string">"2"</span>, <span class="string">"04487E63244F1E2E868870AF5AE42ED8F1D"</span>=&gt;<span class="string">"2"</span>, <span class="string">"04BC55FD524B3E42D6A732E2EA8076A9178"</span>=&gt;<span class="string">"5"</span>, <span class="string">"051394B2B64EF899A10064E2A068924A46C"</span>=&gt;<span class="string">"2"</span>, <span class="string">"05AB0063CC2A0C1B857329D914932DF7C5B"</span>=&gt;<span class="string">"1"</span>, <span class="string">"06AD55DDE7997263212B916CDA2D9439924"</span>=&gt;<span class="string">"4"</span>, <span class="string">"083C47463AAF42031B31DDA54E2F68DC807"</span>=&gt;<span class="string">"1"</span>, <span class="string">"08765B6BDFAF683851AF48258A042D591C1"</span>=&gt;<span class="string">"2"</span>, <span class="string">"099FC9301DB35018687F5BEB5254530020A"</span>=&gt;<span class="string">"2"</span>, <span class="string">"09B30BF127F929D1D9CD946E84C7F7E8FBF"</span>=&gt;<span class="string">"4"</span>, <span class="string">"09D44DA6F15D940BFB19315A0C54CEAECBF"</span>=&gt;<span class="string">"5"</span>, <span class="string">"0A649886EE897919604D2D8F35384ECC90F"</span>=&gt;<span class="string">"3"</span>, <span class="string">"0C720EC0E1BED69EE7DE19C3EA4326E3DFF"</span>=&gt;<span class="string">"7"</span>, <span class="string">"0C9279D46756FDA6911146D2245A013C4F4"</span>=&gt;<span class="string">"3"</span>, <span class="string">"..."</span>,</span><br></pre></td></tr></table></figure><p>Isn’t that better? Now we can search the suffix of our hash within the variable <code>hash_list</code> effortlessly:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="meta">irb(main):015:0&gt;</span> hash_list[sha1_to_look_for.upcase]</span><br><span class="line">=&gt; <span class="string">"216221"</span></span><br></pre></td></tr></table></figure><p>It’s working! And right now that’s what we need to know, either way we can refactor it anytime later. Let’s load our module into IRB and let’s check some passwords.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="meta">irb(main):001:0&gt;</span> require <span class="string">'dmp'</span></span><br><span class="line">=&gt; true</span><br><span class="line"><span class="meta">irb(main):002:0&gt;</span> Dmp.check_pwned(<span class="string">'passw0rd'</span>)</span><br><span class="line">=&gt; <span class="string">"216221"</span></span><br><span class="line"><span class="meta">irb(main):003:0&gt;</span> Dmp.check_pwned(<span class="string">'iloveyou'</span>)</span><br><span class="line">=&gt; <span class="string">"1593388"</span></span><br><span class="line"><span class="meta">irb(main):004:0&gt;</span> Dmp.check_pwned(<span class="string">'coding dose dot com'</span>)</span><br><span class="line">=&gt; nil</span><br></pre></td></tr></table></figure><p>As you can see, the password <em>‘passw0rd’</em> was found in <strong>216221</strong> dictionaries (or leaks), <em>iloveyou</em> was found in <strong>1593388</strong> and, fortunately, <em>coding dose dot com</em> was not found in any dataset, which makes it as secure against dictionary attacks at the moment.</p><h2 id="Asserting-secure-passwords"><a href="#Asserting-secure-passwords" class="headerlink" title="Asserting secure passwords"></a>Asserting secure passwords</h2><p>This is the last test unit we’re going to perform here. More precisely we are going to check if a password is secure, we know that <em>coding dose dot com</em> is secure enough so we can test it and our gem should not flag this password as insecure. Let’s fill out the test right now.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test_secure_pass</span></span></span><br><span class="line">  <span class="comment"># check_pwned should not flag this passphrase</span></span><br><span class="line">  vuln_count = Dmp.check_pwned(<span class="string">'iloveyou'</span>)</span><br><span class="line">  assert_nil vuln_count</span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>What we’re doing here is pretty much the same as with our previous test and the keyword <code>refute_nil</code>, but this time <code>vuln_count</code> should be <code>nil</code> because our secure password should not be found in any dataset on the HIBP API. Right now we want to see the test fail, so we have set the password as <em>‘iloveyou’</em>.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> rake <span class="built_in">test</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># --- SNIP ---</span></span><br><span class="line"></span><br><span class="line">        Expected <span class="comment"># encoding: ASCII-8BIT</span></span><br><span class="line">        <span class="string">"1593388"</span> to be nil.</span><br><span class="line"></span><br><span class="line"><span class="comment"># --- SNIP ---</span></span><br></pre></td></tr></table></figure><p>There we see our test failing, now let’s make it pass changing the password to <em>‘coding dose dot com’</em> and we should see that our test pass without any issue.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> rake <span class="built_in">test</span></span><br><span class="line"></span><br><span class="line">Started with run options --seed 24537</span><br><span class="line"></span><br><span class="line">  4/4: [==========================================================================] 100% Time: 00:00:00, Time: 00:00:00</span><br><span class="line"></span><br><span class="line">Finished <span class="keyword">in</span> 0.53177s</span><br><span class="line">4 tests, 6 assertions, 0 failures, 0 errors, 0 skips</span><br></pre></td></tr></table></figure><h2 id="Refactoring-our-tests"><a href="#Refactoring-our-tests" class="headerlink" title="Refactoring our tests"></a>Refactoring our tests</h2><p>There’s one issue with our test that can be very easily fixed. Let’s take a closer look at the last test.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DmpTest</span> &lt; Minitest::Test</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">test_secure_pass</span></span></span><br><span class="line">    <span class="comment"># check_pwned should not flag this passphrase</span></span><br><span class="line">    vuln_count = Dmp.check_pwned(<span class="string">'coding dose dot com'</span>)</span><br><span class="line">    assert_nil vuln_count</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>If there’s one issue that bothers me, is that we’re not <em>actually</em> testing the core functionality of our gem. Why are we testing for a hard coded passphrase (<em>‘coding dose dot com’</em>) if our gem CAN generate one for us? We could even change our test like this and it could work perfectly fine!</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DmpTest</span> &lt; Minitest::Test</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">test_secure_pass</span></span></span><br><span class="line">    <span class="comment"># check_pwned should not flag this passphrase</span></span><br><span class="line">    safe_pass = Dmp.gen_passphrase(<span class="number">7</span>)</span><br><span class="line">    vuln_count = Dmp.check_pwned(safe_pass)</span><br><span class="line">    assert_nil vuln_count</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>This is perfectly fine, however, what about if we make more tests and we need to generate more passwords? We would have to generate a new passphrase in every test and this would affect performance and it wouldn’t be <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself" target="_blank" rel="noopener">DRY</a> code.</p><p>In order to avoid repetition and make our lives way easier, we can write a <code>setup</code> method, in which we can create an instance variable that holds our generated secure password, along with an insecure one, in order to reuse it in many tests.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DmpTest</span> &lt; Minitest::Test</span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">setup</span></span></span><br><span class="line">    @unsafe_pass = <span class="string">'passw0rd'</span></span><br><span class="line">    @safe_pass = Dmp.gen_passphrase(pass_length = <span class="number">12</span>)</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">test_vulnerable_pass</span></span></span><br><span class="line">    <span class="comment"># check_pwned should flag this password</span></span><br><span class="line">    vuln_count = Dmp.check_pwned(@unsafe_pass)</span><br><span class="line">    refute_nil vuln_count</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">test_secure_pass</span></span></span><br><span class="line">    <span class="comment"># check_pwned should not flag this passphrase</span></span><br><span class="line">    vuln_count = Dmp.check_pwned(@safe_pass)</span><br><span class="line">    assert_nil vuln_count</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Now if we run our tests everything should work correctly and we can reuse this passwords without having to declare them in each test.</p><h1 id="Command-Line-Interface-CLI-and-publishing-our-gem-in-RubyGems"><a href="#Command-Line-Interface-CLI-and-publishing-our-gem-in-RubyGems" class="headerlink" title="Command Line Interface (CLI) and publishing our gem in RubyGems"></a>Command Line Interface (CLI) and publishing our gem in RubyGems</h1><h2 id="Creating-our-CLI-with-Thor"><a href="#Creating-our-CLI-with-Thor" class="headerlink" title="Creating our CLI with Thor"></a>Creating our CLI with Thor</h2><p>Now that we have created the core functions of our gem, we can actually create a nice CLI interface. Luckily, this is made easier with the framework called <a href="http://whatisthor.com/" target="_blank" rel="noopener">Thor</a>.</p><blockquote><p>Thor is a toolkit for building powerful command-line interfaces. It is used in Bundler, Vagrant, Rails and others.</p></blockquote><p>I invite you to go into the website and read a bit about Thor so you understand how the framework works, but don’t worry, it’s actually pretty simple and straight forward. To begin with, we’re going to create the file in which we’re going to construct our CLI.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Create the filename cli.rb under lib/dmp/</span></span><br><span class="line">$ touch lib/dmp/cli.rb</span><br></pre></td></tr></table></figure><p>Now that we have created our file, let’s recall our project description:</p><ol><li>We want our gem to generate a passphrase.</li><li>We want our gem to check this passphrase with the HIBP to test if it’s secure.</li><li>And we also want an option to automatically copy this passphrase to the clipboard.</li></ol><p>Ideally, we want to recreate this behavior:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ dmp gen 4 --hibp --clipboard</span><br><span class="line">- Passphrase: cobweb desolate pushy mulled</span><br><span class="line">- Copied to clipboard.</span><br><span class="line">- Password was not found <span class="keyword">in</span> a dataset.</span><br></pre></td></tr></table></figure><p>Now, I love colors in my terminal, so we’ll add colors to our output, in fact, I want to have every word in the passphrase with a random color. But let’s not get ahead of ourselves and add a little bit of code to our CLI to get started, open the file.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># lib/dmp/cli.rb</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">require</span> <span class="string">'thor'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'dmp'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'colorize'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'clipboard'</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="class"><span class="keyword">class</span> <span class="title">CLI</span> &lt; Thor</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>This is the bare minimum that we need to get our CLI started, however, when we try to execute our gem in a terminal using bundler, it doesn’t detect it. This is because we need to define our executable file in our gem before we proceed.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Create the file exe/dmp</span></span><br><span class="line">$ mkdir exe</span><br><span class="line">$ touch exe/dmp</span><br><span class="line"></span><br><span class="line"><span class="comment"># Make dmp executable</span></span><br><span class="line">$ chmod +x exe/dmp</span><br></pre></td></tr></table></figure><p>Please, notice that our <code>dmp</code> file does not have any extension, this is intended. Let’s open this file and fill it with some code.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment">#!/usr/bin/env ruby</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'dmp/cli'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">Dmp::CLI.start</span><br></pre></td></tr></table></figure><p>In order for bundler to detect our new executable, we must integrate it into our version control with git.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git add .</span><br><span class="line">$ git commit -am <span class="string">"Add CLI with Thor"</span></span><br></pre></td></tr></table></figure><p>After that, let’s update our gems</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># the command 'bundle' automatically updates and install your gems</span></span><br><span class="line">$ bundle</span><br><span class="line">Using rake 10.5.0</span><br><span class="line">Using ansi 1.5.0</span><br><span class="line">Using builder 3.2.3</span><br><span class="line">Using bundler 1.17.2</span><br><span class="line">Using clipboard 1.3.3</span><br><span class="line">Using colorize 0.8.1</span><br><span class="line">Using thor 0.20.3</span><br><span class="line">Using dmp 0.1.0 from <span class="built_in">source</span> at `.`</span><br><span class="line">Using irb 1.0.0</span><br><span class="line">Using minitest 5.11.3</span><br><span class="line">Using ruby-progressbar 1.10.0</span><br><span class="line">Using minitest-reporters 1.3.6</span><br><span class="line">Bundle complete! 6 Gemfile dependencies, 12 gems now installed.</span><br><span class="line">Use `bundle info [gemname]` to see <span class="built_in">where</span> a bundled gem is installed.</span><br></pre></td></tr></table></figure><p>And finally we can see that our CLI comes to life.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> dmp</span><br><span class="line">Commands:</span><br><span class="line">  dmp <span class="built_in">help</span> [COMMAND]  <span class="comment"># Describe available commands or one specific command</span></span><br></pre></td></tr></table></figure><p>But we’re missing all the functionality that we want to create though, also there’s no description on what our program do, let’s try to fix this by adding a description and also three tasks to our CLI.</p><p>These tasks will be <code>gen_pass</code> which we’ll use to generate passphrases, <code>check_pass</code> that will check if a passphrase or password is found in a HIBP dataset and lastly an <code>about</code> task which will be used to describe information about the program and the author of it, alright? Let’s do it.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">'thor'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'dmp'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'colorize'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'clipboard'</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="class"><span class="keyword">class</span> <span class="title">CLI</span> &lt; Thor</span></span><br><span class="line">    desc <span class="string">'gen [length]'</span>, <span class="string">'Generate a passphrase of the desired length.'</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">gen_pass</span>;</span> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">    desc <span class="string">'check'</span>, <span class="string">'Check if a password/passphrase is vulnerable.'</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">check_pass</span>;</span> <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">    desc <span class="string">'about'</span>, <span class="string">'Displays version number and information'</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">about</span>;</span> <span class="keyword">end</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Now, we have added empty tasks, and that’s O.K. for now, we will fill them one by one, but if we execute our gem we will see that new tasks were added.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> dmp</span><br><span class="line">Commands:</span><br><span class="line">  dmp about           <span class="comment"># Displays version number and information</span></span><br><span class="line">  dmp check           <span class="comment"># Check if a password/passphrase is vulnerable.</span></span><br><span class="line">  dmp gen [length]    <span class="comment"># Generate a passphrase of the desired length.</span></span><br><span class="line">  dmp <span class="built_in">help</span> [COMMAND]  <span class="comment"># Describe available commands or one specific command</span></span><br></pre></td></tr></table></figure><p>Let’s code the task <code>gen</code> first. I want to add two options to the <code>gen</code> command, when I generate a passphrase of my desired length, I want to automatically copy the new passphrase to the clipboard, and also I want to check if the passphrase shows up in a HIBP database, which would make it insecure. We can add these optional functionality to our <code>gen</code> command using the keyword <code>method_option</code>.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">'thor'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'dmp'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'colorize'</span></span><br><span class="line"><span class="keyword">require</span> <span class="string">'clipboard'</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="class"><span class="keyword">class</span> <span class="title">CLI</span> &lt; Thor</span></span><br><span class="line">    desc <span class="string">'gen [length]'</span>, <span class="string">'Generate a passphrase of the desired length.'</span></span><br><span class="line">    method_option <span class="symbol">:clipboard</span>,</span><br><span class="line">                  <span class="symbol">aliases:</span> <span class="string">'-c'</span>,</span><br><span class="line">                  <span class="symbol">type:</span> <span class="symbol">:boolean</span>,</span><br><span class="line">                  <span class="symbol">desc:</span> <span class="string">'Copy passphrase to clipboard.'</span></span><br><span class="line">    method_option <span class="symbol">:hibp</span>,</span><br><span class="line">                  <span class="symbol">aliases:</span> <span class="string">'-H'</span>,</span><br><span class="line">                  <span class="symbol">type:</span> <span class="symbol">:boolean</span>,</span><br><span class="line">                  <span class="symbol">desc:</span> <span class="string">'Check if passphrase is vulnerable in HIBP database.'</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">gen_pass</span><span class="params">(pass_length = <span class="number">7</span>)</span></span></span><br><span class="line">    <span class="keyword">end</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>We can see that we have added the option <code>clipboard</code> which will be activated using the alias <code>-c</code>, this option is <em>boolean</em> which will be treated as a flag and has it’s own description, the same goes to the <code>hibp</code> option. We can see this reflected when we check the help command in our gem.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> dmp <span class="built_in">help</span> gen</span><br><span class="line">Usage:</span><br><span class="line">  dmp gen [length]</span><br><span class="line"></span><br><span class="line">Options:</span><br><span class="line">  -c, [--clipboard], [--no-clipboard]  <span class="comment"># Copy passphrase to clipboard.</span></span><br><span class="line">  -H, [--hibp], [--no-hibp]            <span class="comment"># Check if passphrase is vulnerable in HIBP database.</span></span><br><span class="line"></span><br><span class="line">Generate a passphrase of the desired length.</span><br></pre></td></tr></table></figure><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="class"><span class="keyword">class</span> <span class="title">CLI</span> &lt; Thor</span></span><br><span class="line">    <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">gen_pass</span><span class="params">(pass_length = <span class="number">7</span>)</span></span></span><br><span class="line">      <span class="comment"># Force our length value to be an integer</span></span><br><span class="line">      new_passphrase = Dmp.gen_passphrase(pass_length.to_i)</span><br><span class="line"></span><br><span class="line">      <span class="comment"># If our options :clipboard and :hibp are true, then proceeds</span></span><br><span class="line">      <span class="comment"># to copy the contents of the passphrase to the clipboard</span></span><br><span class="line">      <span class="comment"># and check if the passphrase is vulnerable.</span></span><br><span class="line">      Clipboard.copy(new_passphrase.join(<span class="string">' '</span>)) <span class="keyword">if</span> options[<span class="symbol">:clipboard</span>]</span><br><span class="line">      dataset_count = Dmp.check_pwned(new_passphrase) <span class="keyword">if</span> options[<span class="symbol">:hibp</span>]</span><br><span class="line"></span><br><span class="line">      <span class="comment"># To add colors, first we store the available colors in a variable</span></span><br><span class="line">      <span class="comment"># but we eliminate the color black which makes some words to be</span></span><br><span class="line">      <span class="comment"># unreadable in the terminal. After that, we map the passphrase list</span></span><br><span class="line">      <span class="comment"># and assign a random color to each one of them.</span></span><br><span class="line">      colors = String.colors</span><br><span class="line">      colors.delete(<span class="symbol">:black</span>) <span class="comment"># black color looks ugly in the terminal</span></span><br><span class="line">      new_passphrase.map! <span class="keyword">do</span> <span class="params">|phrase|</span></span><br><span class="line">        random_color = colors.sample</span><br><span class="line">        phrase.colorize(random_color)</span><br><span class="line">      <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">      <span class="comment"># We add default messages in case our options are activated. A green bold</span></span><br><span class="line">      <span class="comment"># message when the user wants to copy the passphrase, another one if the</span></span><br><span class="line">      <span class="comment"># passphrase is safe, and a red bold one if the passphrase is vulnerable.</span></span><br><span class="line">      copy_msg = <span class="string">'- Copied to clipboard.'</span>.bold.green</span><br><span class="line">      vuln_pass_msg = <span class="string">"- WARNING: Passphrase appears in <span class="subst">#&#123;dataset_count&#125;</span> datasets!"</span>.red.bold</span><br><span class="line">      safe_pass_msg = <span class="string">'- Password was not found in a dataset.'</span>.green.bold</span><br><span class="line"></span><br><span class="line">      <span class="comment"># Bold the title passphrase and then join the passphrase to make it a string.</span></span><br><span class="line">      puts <span class="string">'- Passphrase: '</span>.bold + new_passphrase.join(<span class="string">' '</span>)</span><br><span class="line"></span><br><span class="line">      <span class="comment"># If the clipboard option is active then display the clipboard message</span></span><br><span class="line">      puts copy_msg <span class="keyword">if</span> options[<span class="symbol">:clipboard</span>]</span><br><span class="line">      <span class="comment"># If the option :hibp is True then check if the pass is found in a dataset.</span></span><br><span class="line">      <span class="comment"># If dataset_cout is not nil then display vuln_pass_msg, else display</span></span><br><span class="line">      <span class="comment"># the safe_pass_msg.</span></span><br><span class="line">      puts dataset_count ? vuln_pass_msg : safe_pass_msg <span class="keyword">if</span> options[<span class="symbol">:hibp</span>]</span><br><span class="line">    <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Looks a little bit long because all of the comments there, I only wrote them from clarity but you can leave them behind so you don’t have your code so cluttered. The good news is that we have completed our first task! It should be executed flawlessly. Let’s check it out.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> dmp gen</span><br><span class="line">- Passphrase: molecular lubricate press net plank crook subpanel</span><br><span class="line"></span><br><span class="line">$ bundle <span class="built_in">exec</span> dmp gen 3</span><br><span class="line">- Passphrase: preppy dividing epidural</span><br><span class="line"></span><br><span class="line">$ bundle <span class="built_in">exec</span> dmp gen 3 -c -H</span><br><span class="line">- Passphrase: jaunt nurture reason</span><br><span class="line">- Copied to clipboard.</span><br><span class="line">- Password was not found <span class="keyword">in</span> a dataset.</span><br><span class="line"></span><br><span class="line">$ bundle <span class="built_in">exec</span> dmp gen 1 -H</span><br><span class="line">- Passphrase: capacity</span><br><span class="line">- WARNING: Passphrase appears <span class="keyword">in</span> 1879 datasets!</span><br></pre></td></tr></table></figure><p>Awesome! Our gem works as expected, with all of the functionality we intended for it. Now, let’s fill out the next task, which would be <code>check_pass</code>. This task will not generate a passphrase, instead it will check for a password or passphrase that we currently have.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="class"><span class="keyword">class</span> <span class="title">CLI</span> &lt; Thor</span></span><br><span class="line">    <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line"></span><br><span class="line">    desc <span class="string">'check'</span>, <span class="string">'Check if a password/passphrase is vulnerable.'</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">check_pass</span></span></span><br><span class="line">      puts <span class="string">"Enter your password, press ENTER when you're done."</span></span><br><span class="line">      password = ask(<span class="string">'Password (hidden):'</span>.yellow, <span class="symbol">echo:</span> <span class="literal">false</span>)</span><br><span class="line">      (puts <span class="string">"Aborted."</span>.red.bold; exit) <span class="keyword">if</span> password.empty?</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">      dataset_count = Dmp.check_pwned(password)</span><br><span class="line">      vuln_msg = <span class="string">"Your password appears in <span class="subst">#&#123;dataset_count&#125;</span> datasets!"</span>.red.bold</span><br><span class="line">      safe_msg = <span class="string">"Your password was not found in a dataset."</span>.green.bold</span><br><span class="line">      puts dataset_count ? vuln_msg : safe_msg</span><br><span class="line">    <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>As we can see, this task is very simple and short, one thing that we can highlight is the fact that we can <code>ask</code> for a password without disclose it in the terminal, for security reasons of course. We can do this with the <code>ask</code> method followed by <code>echo: false</code> if we want to turn off the echo when people type in.</p><p>Now that our <code>check_pass</code> method is finished, let’s try it out on the console.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> dmp check</span><br><span class="line">Enter your password, press ENTER when you<span class="string">'re done.</span></span><br><span class="line"><span class="string">Password (hidden): Your password was not found in a dataset.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">$ bundle exec dmp check</span></span><br><span class="line"><span class="string">Enter your password, press ENTER when you'</span>re <span class="keyword">done</span>.</span><br><span class="line">Password (hidden): Your password appears <span class="keyword">in</span> 15996 datasets!</span><br></pre></td></tr></table></figure><p>There we go! For the final task, let’s fill in the <code>about</code> task. For old times sake (and a little bit of cockiness) we’ll add a little banner when we call the <code>about</code> task.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="class"><span class="keyword">class</span> <span class="title">CLI</span> &lt; Thor</span></span><br><span class="line">    <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line"></span><br><span class="line">    desc <span class="string">'about'</span>, <span class="string">'Displays version number and information'</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">about</span></span></span><br><span class="line">      puts Dmp::BANNER.bold.red</span><br><span class="line">      puts <span class="string">'version: '</span>.bold + Dmp::VERSION.green</span><br><span class="line">      puts <span class="string">'author: '</span>.bold + <span class="string">'@__franccesco'</span>.green</span><br><span class="line">      puts <span class="string">'homepage: '</span>.bold + <span class="string">'https://github.com/franccesco/dmp'</span>.green</span><br><span class="line">      puts <span class="string">'learn more: '</span>.bold + <span class="string">'https://codingdose.info'</span>.green</span><br><span class="line">      puts <span class="comment"># extra line, somehow I like them.</span></span><br><span class="line">    <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>We can see that we display the banner first, and a few details about ourselves and the program there like version, homepage, author and you can add as much as you want. But we haven’t declared a <code>BANNER</code> yet. So let’s do that, shall we? Let’s open up our version file in <code>lib/dmp/version.rb</code> and add our banner there.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  VERSION = <span class="string">"0.1.0"</span></span><br><span class="line">  BANNER = <span class="string">''</span><span class="string">'</span></span><br><span class="line"><span class="string"> ____    __  __   ____</span></span><br><span class="line"><span class="string">|  _ \  |  \/  | |  _ \</span></span><br><span class="line"><span class="string">| | | | | |\/| | | |_) |</span></span><br><span class="line"><span class="string">| |_| | | |  | | |  __/</span></span><br><span class="line"><span class="string">|____/  |_|  |_| |_|</span></span><br><span class="line"><span class="string">               '</span><span class="string">''</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>There we go, now let’s test our <code>about</code> task.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> dmp about</span><br><span class="line"> ____    __  __   ____</span><br><span class="line">|  _ \  |  \/  | |  _ \</span><br><span class="line">| | | | | |\/| | | |_) |</span><br><span class="line">| |_| | | |  | | |  __/</span><br><span class="line">|____/  |_|  |_| |_|</span><br><span class="line"></span><br><span class="line">version: 0.1.0</span><br><span class="line">author: @__franccesco</span><br><span class="line">homepage: https://github.com/franccesco/dmp</span><br><span class="line">learn more: https://codingdose.info</span><br></pre></td></tr></table></figure><p>Cool, right?! Now we have a fully functional gem! Let’s save our final changes into git.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git commit -am <span class="string">"Complete CLI tasks"</span></span><br><span class="line">[develop ba6f18e] Complete CLI tasks</span><br><span class="line"> 2 files changed, 57 insertions(+), 4 deletions(-)</span><br></pre></td></tr></table></figure><h2 id="Managing-versions-with-gem-release"><a href="#Managing-versions-with-gem-release" class="headerlink" title="Managing versions with gem-release"></a>Managing versions with gem-release</h2><p>I assume you’re going to refactor this gem, which you should do! There’s a lot of dirty code in it, right? I’m not doing that in this article though. But once you do, you will probably add more features or maybe remove some of them (like the colors for example), and when that happens, you would like to reflect this changes by <em>bumping</em> your version number.</p><p>I use the <a href="https://semver.org/" target="_blank" rel="noopener"><strong>Semantic Versioning</strong></a>, you should check it out if you don’t know about it. If you made a change and you want to bump your version number, you would have to do it manually opening the <code>version.rb</code> file and then committing your changes. However, we can do this way more easily with the <code>gem-release</code> extension. Let’s install it right now.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gem install gem-release</span><br><span class="line">Fetching gem-release-2.0.1.gem</span><br><span class="line">Successfully installed gem-release-2.0.1</span><br><span class="line">Parsing documentation <span class="keyword">for</span> gem-release-2.0.1</span><br><span class="line">Installing ri documentation <span class="keyword">for</span> gem-release-2.0.1</span><br><span class="line">Done installing documentation <span class="keyword">for</span> gem-release after 0 seconds</span><br><span class="line">1 gem installed</span><br></pre></td></tr></table></figure><p>Now, I want to make a little change to our gem, I don’t want to write <code>dmp gen</code> each time I want to generate a new passphrase, I want it to do it by default! Luckily, we can do this with the keyword <code>default_task</code>. Let’s open up our CLI.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">module</span> <span class="title">Dmp</span></span></span><br><span class="line">  <span class="class"><span class="keyword">class</span> <span class="title">CLI</span> &lt; Thor</span></span><br><span class="line">    default_task <span class="symbol">:gen_pass</span></span><br><span class="line">    <span class="comment"># -- CODE SNIPPED --</span></span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>Now let’s check and commit our change.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ bundle <span class="built_in">exec</span> dmp</span><br><span class="line">- Passphrase: whinny capitol balsamic colt washout lend cradling</span><br><span class="line"></span><br><span class="line">$ git commit -am <span class="string">"Add default task to CLI"</span></span><br></pre></td></tr></table></figure><p>After we have committed our changes, we can go ahead and bump our version number. As we might know, we should bump a minor version as we added functionality to our gem.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gem bump --version minor</span><br><span class="line"></span><br><span class="line">Bumping dmp from version 0.1.0 to 0.2.0</span><br><span class="line">Changing version <span class="keyword">in</span> lib/dmp/version.rb from 0.1.0 to 0.2.0</span><br><span class="line"></span><br><span class="line">Staging lib/dmp/version.rb</span><br><span class="line">$ git add lib/dmp/version.rb</span><br><span class="line"></span><br><span class="line">Creating commit</span><br><span class="line">$ git commit -m <span class="string">"Bump dmp to 0.2.0"</span></span><br><span class="line">[develop b54f1f4] Bump dmp to 0.2.0</span><br><span class="line"> 1 file changed, 1 insertion(+), 1 deletion(-)</span><br><span class="line"></span><br><span class="line">All is good, thanks my friend.</span><br></pre></td></tr></table></figure><p>As you can see, the <code>gem-release</code> bumps our minor version and also <em>commits</em> our bump change in a single pass. After we have made all of our changes, it is a good idea to publish or gem so other people can download it and use it!</p><h2 id="Publishing-our-gem"><a href="#Publishing-our-gem" class="headerlink" title="Publishing our gem"></a>Publishing our gem</h2><p>Publishing our gem its <em>really</em> easy. For this you should go to <a href="https://rubygems.org/" target="_blank" rel="noopener">Rubygems.org</a> and sign up with an email and password, as you will need to authenticate in order to manage your uploaded gems. Here’s how you do it.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># First we build our gem with our gem specifications</span></span><br><span class="line">$ gem build dmp.gemspec</span><br><span class="line">  Successfully built RubyGem</span><br><span class="line">  Name: dmp</span><br><span class="line">  Version: 0.1.0</span><br><span class="line">  File: dmp-0.1.0.gem</span><br></pre></td></tr></table></figure><p>After we have built our gem, it’s time to publish it.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gem push dmp-0.1.0.gem</span><br><span class="line">Enter your RubyGems.org credentials.</span><br><span class="line">Don<span class="string">'t have an account yet? Create one at https://rubygems.org/sign_up</span></span><br><span class="line"><span class="string">   Email:   gem_author@example</span></span><br><span class="line"><span class="string">Password:</span></span><br><span class="line"><span class="string">Signed in.</span></span><br><span class="line"><span class="string">Pushing gem to RubyGems.org...</span></span><br><span class="line"><span class="string">Successfully registered gem: dmp-0.1.0.gem</span></span><br></pre></td></tr></table></figure><p>And that’s it! Your gem is available to the public, you can try this yourself by installing your gem hosted on the rubygems servers.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gem install dmp</span><br><span class="line">Fetching dmp-0.2.4.gem</span><br><span class="line">Successfully installed dmp-0.2.4</span><br><span class="line">Parsing documentation <span class="keyword">for</span> dmp-0.2.4</span><br><span class="line">Installing ri documentation <span class="keyword">for</span> dmp-0.2.4</span><br><span class="line">Done installing documentation <span class="keyword">for</span> dmp after 0 seconds</span><br><span class="line">1 gem installed</span><br><span class="line"></span><br><span class="line">$ dmp about</span><br><span class="line"> ____    __  __   ____</span><br><span class="line">|  _ \  |  \/  | |  _ \</span><br><span class="line">| | | | | |\/| | | |_) |</span><br><span class="line">| |_| | | |  | | |  __/</span><br><span class="line">|____/  |_|  |_| |_|</span><br><span class="line"></span><br><span class="line">version: 0.2.4</span><br><span class="line">author: @__franccesco</span><br><span class="line">homepage: https://github.com/franccesco/dmp</span><br><span class="line">learn more: https://codingdose.info</span><br></pre></td></tr></table></figure><h1 id=""><a href="#" class="headerlink" title=""></a><a href=""></a></h1><h2 id="EOF"><a href="#EOF" class="headerlink" title="EOF"></a>EOF</h2><p>This was quite a ride right? If you have any questions then let me know in the comments bellow. Thank you for reading this article and I appreciate your time and efforts if you have followed this tutorial.</p><p>You can checkout the code for DMP in the github repository here: <a href="https://github.com/franccesco/dmp" target="_blank" rel="noopener">https://github.com/franccesco/dmp</a><br>And also you can check out the gem at RubyGems.org: <a href="https://rubygems.org/gems/dmp" target="_blank" rel="noopener">https://rubygems.org/gems/dmp</a></p><p>If you feel that you need to rewrite the dirty code or make things your way, then who am I to stop you? Go do it! I’m sure you’ll do a hell of a job! :)</p>]]></content>
      
      
      
        <tags>
            
            <tag> ruby </tag>
            
            <tag> bundler </tag>
            
            <tag> packaging </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Generate and Verify Your Commits With GPG in Github</title>
      <link href="2018/08/06/Generate-and-sign-your-commits-with-GPG-in-Github/"/>
      <url>2018/08/06/Generate-and-sign-your-commits-with-GPG-in-Github/</url>
      
        <content type="html"><![CDATA[<!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1556075798-4825dfaaf498?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="Free Text Image on Unsplash" title="Free Text Image on Unsplash">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/842ofHC6MaI" target="_blank">Photo</a> by <a href="https://unsplash.com/@yancymin" target="_blank">@yancymin</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 01/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><h2 id="Why-should-you-do-it"><a href="#Why-should-you-do-it" class="headerlink" title="Why should you do it?"></a>Why should you do it?</h2><blockquote><p>Using GPG, you can sign and verify tags and commits. With GPG keys, tags or commits that you’ve authored on GitHub are verified and other people can trust that the changes you’ve made really were made by you.</p></blockquote><ul><li><a href="https://help.github.com/articles/about-gpg/" target="_blank" rel="noopener">About GPG | Github</a></li></ul><p>Signing our commits is a great way to verify your commits and let your collaborators know that they can trust that you committed those changes in your project. We’re going to see how can we use a GPG key to sign our commits and also how to change git settings so it signs our commits automatically.</p><p><em>This guide will assume that you haven’t setup your GPG key yet.</em></p><h2 id="Installing-GPG"><a href="#Installing-GPG" class="headerlink" title="Installing GPG"></a>Installing GPG</h2><p>GnuPG should be available on your system, if it’s not, then you can download it and follow the instructions to install it depending on your distribution here: <a href="https://www.gnupg.org/download/" target="_blank" rel="noopener">https://www.gnupg.org/download/</a></p><h2 id="Setting-up-GPG"><a href="#Setting-up-GPG" class="headerlink" title="Setting up GPG"></a>Setting up GPG</h2><p>This is a pretty easy step as the GnuPG has a wizard that can help us fill the requirements to create our GPG key with the <code>gpg --full-generate-key</code> command.</p><p>We’re going to generate a RSA 4096 bit long key with no expiration date on our email.</p><p><em><strong>NOTE:</strong> In order to verify your commits in Github, you <strong>must</strong> enter the email address that you have registered in Github and it should also match your email that you previously configured in git.</em></p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gpg --full-generate-key</span><br><span class="line">Please select what kind of key you want:</span><br><span class="line">   (1) RSA and RSA (default)</span><br><span class="line">   (2) DSA and Elgamal</span><br><span class="line">   (3) DSA (sign only)</span><br><span class="line">   (4) RSA (sign only)</span><br><span class="line">Your selection? 1</span><br><span class="line">RSA keys may be between 1024 and 4096 bits long.</span><br><span class="line">What keysize <span class="keyword">do</span> you want? (3072) 4096</span><br><span class="line">Requested keysize is 4096 bits</span><br><span class="line">Please specify how long the key should be valid.</span><br><span class="line">         0 = key does not expire</span><br><span class="line">      &lt;n&gt;  = key expires <span class="keyword">in</span> n days</span><br><span class="line">      &lt;n&gt;w = key expires <span class="keyword">in</span> n weeks</span><br><span class="line">      &lt;n&gt;m = key expires <span class="keyword">in</span> n months</span><br><span class="line">      &lt;n&gt;y = key expires <span class="keyword">in</span> n years</span><br><span class="line">Key is valid <span class="keyword">for</span>? (0) 0</span><br><span class="line">Key does not expire at all</span><br><span class="line">Is this correct? (y/N) y</span><br><span class="line"></span><br><span class="line">GnuPG needs to construct a user ID to identify your key.</span><br><span class="line"></span><br><span class="line">Real name: Franccesco Orozco</span><br><span class="line">Email address: franccesco.orozco@codingdose.info</span><br><span class="line">Comment: My First Key!</span><br><span class="line">You selected this USER-ID:</span><br><span class="line">    <span class="string">"Franccesco Orozco (My First Key!) &lt;franccesco.orozco@codingdose.info&gt;"</span></span><br></pre></td></tr></table></figure><p>After that it will ask us to input a passphrase, remember to choose a <em>good passphrase</em>.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">We need to generate a lot of random bytes. It is a good idea to perform</span><br><span class="line">some other action (type on the keyboard, move the mouse, utilize the</span><br><span class="line">disks) during the prime generation; this gives the random number</span><br><span class="line">generator a better chance to gain enough entropy.</span><br><span class="line">gpg: key 438D3A434DA6E6FA marked as ultimately trusted</span><br><span class="line">gpg: revocation certificate stored as &#39;&#x2F;home&#x2F;franccesco&#x2F;.gnupg&#x2F;openpgp-revocs.d&#x2F;D77C79FFD77BFD0B9BCE58FE438D3A434DA6E6FA.rev&#39;</span><br><span class="line">public and secret key created and signed.</span><br><span class="line"></span><br><span class="line">pub   rsa4096 2018-09-05 [SC]</span><br><span class="line">      D77C79FFD77BFD0B9BCE58FE438D3A434DA6E6FA</span><br><span class="line">uid                      Franccesco Orozco (My First Key!) &lt;franccesco.orozco@codingdose.info&gt;</span><br><span class="line">sub   rsa4096 2018-09-05 [E]</span><br></pre></td></tr></table></figure><p>That’s it! We were able to create our GPG key pretty easily, right? Let’s see what can we do to add it to Github.</p><h2 id="Listing-our-key-s"><a href="#Listing-our-key-s" class="headerlink" title="Listing our key(s)"></a>Listing our key(s)</h2><p>Now that we have created our key, we can list it like this.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gpg --list-secret-keys --keyid-format LONG franccesco.orozco</span><br><span class="line">sec   rsa4096/438D3A434DA6E6FA 2018-09-05 [SC]</span><br><span class="line">      D77C79FFD77BFD0B9BCE58FE438D3A434DA6E6FA</span><br><span class="line">uid                 [ultimate] Franccesco Orozco (My First Key!) &lt;franccesco.orozco@codingdose.info&gt;</span><br><span class="line">ssb   rsa4096/5D21EBE255188195 2018-09-05 [E]</span><br></pre></td></tr></table></figure><p>What we’re looking in our key is the <strong>GPG Key ID</strong>, which is: <em>438D3A434DA6E6FA</em>. With this identifier we can export the GPG <strong>Public</strong> key and paste it in Github.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gpg --armor --<span class="built_in">export</span> 438D3A434DA6E6FA</span><br><span class="line">-----BEGIN PGP PUBLIC KEY BLOCK-----</span><br><span class="line">mQINBFuP6u4BEADA6ZWSjy86eMpS6OczKgkPbytA7b5lzMcdwnSccwuX0w9/fVA7</span><br><span class="line">yx+fuZZuKO1rHNR96wgq4m5Z9iUM7UQ5FG9g93CXUp6kmPcast3fpQ7D13Oq6lEy</span><br><span class="line">iNmxziJ3K/DQnEj8vgEl6vxDusBswRdYXHKytKt2pFngZqF/rtD0Mbf9shrGaI9B</span><br><span class="line">--- SNIP ---</span><br><span class="line">Hlupx07dHpBEsjaiKWL80GhKFSQNKO+oOlSZ537nRqcLUzU7zvc1qLp6Z6ZSAFjl</span><br><span class="line">E3aw43pKynoLYXvxUO1vi0En//jMSG4riLZDiZBkfM21</span><br><span class="line">=QqhC</span><br><span class="line">-----END PGP PUBLIC KEY BLOCK-----</span><br></pre></td></tr></table></figure><p>If you have <code>xclip</code> package in your system and don’t want to select the whole key then here’s a great tip to automatically add it to your clipboard.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ gpg --armor --<span class="built_in">export</span> 438D3A434DA6E6FA | xclip -sel c</span><br></pre></td></tr></table></figure><h2 id="Add-GPG-key-to-Github"><a href="#Add-GPG-key-to-Github" class="headerlink" title="Add GPG key to Github"></a>Add GPG key to Github</h2><p>Now it’s time to paste the key in Github, this is pretty easy and self explanatory, go to <strong><a href="https://github.com/settings/keys" target="_blank" rel="noopener">SSH and GPG Keys</a></strong> in your Github settings and click in <strong>New GPG Key</strong>.</p><p>Here you can past your Public key and submit it.<br><img src="/assets/images/gpg_github/add_key.png" alt="image"></p><p>After that, you can see that you have already added your Github key, if its says <em>unverified</em> it’s because it doesn’t matches your email address in your Github account, so make sure everything matches with the email in your GPG key.<br><img src="/assets/images/gpg_github/added_key.png" alt="image"></p><h2 id="Configure-git-with-your-key"><a href="#Configure-git-with-your-key" class="headerlink" title="Configure git with your key"></a>Configure git with your key</h2><p>Now let’s make git aware of our new key with a global configuration.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git config --global user.signingkey 438D3A434DA6E6FA</span><br></pre></td></tr></table></figure><p>After that we can start committing signed changes using the <code>-S</code> flag.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git commit -S -m <span class="string">"Signed commit!"</span></span><br></pre></td></tr></table></figure><p>But of course, if you don’t want to set the <code>-S</code> flag every time then you can set it as default in your commits.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git config --global commit.gpgsign <span class="literal">true</span></span><br></pre></td></tr></table></figure><h2 id="Push-a-signed-commit"><a href="#Push-a-signed-commit" class="headerlink" title="Push a signed commit"></a>Push a signed commit</h2><p>Now, go ahead and feel free to push a commit in Github. From now on, all pushed commits will show as verified in your commit history.</p><p><img src="/assets/images/gpg_github/verify_badge.png" alt="image"></p><p>This will give you more credibility and security to your projects, specially if you work with sensitive date or with a large number of people.</p><h2 id="Further-reading"><a href="#Further-reading" class="headerlink" title="Further reading"></a>Further reading</h2><ul><li><a href="https://help.github.com/articles/generating-a-new-gpg-key/" target="_blank" rel="noopener">Generating a new GPG Key</a></li><li><a href="https://help.github.com/articles/telling-git-about-your-gpg-key/" target="_blank" rel="noopener">Telling Git about your GPG key</a></li><li><a href="https://help.github.com/articles/signing-commits-using-gpg/" target="_blank" rel="noopener">Signing commits using GPG</a></li><li><a href="https://gitlab.com/help/user/project/repository/gpg_signed_commits/index.md" target="_blank" rel="noopener">Gitlab guide on GPG</a></li></ul>]]></content>
      
      
      
        <tags>
            
            <tag> git </tag>
            
            <tag> github </tag>
            
            <tag> gpg </tag>
            
            <tag> linux </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Change Flask Root Folder for Templates and Static Files</title>
      <link href="2018/05/12/change-flask-root-folder/"/>
      <url>2018/05/12/change-flask-root-folder/</url>
      
        <content type="html"><![CDATA[<p>Today I was facing a problem, I didn’t know how to change Flask root directory. Flask by default look for templates and static files under the <strong>root</strong> directory (<code>/</code>), how can we change that?</p><p><img src="/assets/thumbnails/flask_root.png"></p><h2 id="Changing-the-root-path"><a href="#Changing-the-root-path" class="headerlink" title="Changing the root path"></a>Changing the root path</h2><p>Here’s my directory structure:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">.</span><br><span class="line">├── api_files</span><br><span class="line">│   ├── static</span><br><span class="line">│   │   └── style.css</span><br><span class="line">│   └── templates</span><br><span class="line">│       └── index.html</span><br><span class="line">├── api.py</span><br><span class="line">├── Pipfile</span><br><span class="line">└── Pipfile.lock</span><br><span class="line"></span><br><span class="line">3 directories, 5 files</span><br></pre></td></tr></table></figure><p>I want Flask to be able to process the folders <code>static</code> and <code>templates</code> <em>inside</em> the <code>api_files</code> folder, my main Flask app is <code>api.py</code> which is <em>outside</em> the <code>api_files</code> folder. Let’s open it up and change that behavior so it can process the templates and static files inside that folder.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask, render_template</span><br><span class="line"></span><br><span class="line"><span class="comment"># here we can set a different root path</span></span><br><span class="line">app = Flask(__name__, root_path=<span class="string">'api_files/'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route('/')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">index</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Render home page."""</span></span><br><span class="line">    <span class="keyword">return</span> render_template(<span class="string">'index.html'</span>)  <span class="comment"># we can render templates as usual</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    app.run()</span><br></pre></td></tr></table></figure><p>This way, Flask will no longer look for templates and static files under the project root (<code>/</code>) path, but inside the folder <code>api_files</code> instead.</p><p>There’s a lot of solutions out there about changing jinja2 configuration, I think this is a way more easier and cleaner approach.</p><p>You can read more about variables and the <a href="http://flask.pocoo.org/docs/1.0/api/" target="_blank" rel="noopener"><strong>Flask API here</strong></a>.</p>]]></content>
      
      
      
        <tags>
            
            <tag> flask </tag>
            
            <tag> web-development </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Deploy a Flask Project to Heroku</title>
      <link href="2018/05/10/deploy-a-flask-project-to-heroku/"/>
      <url>2018/05/10/deploy-a-flask-project-to-heroku/</url>
      
        <content type="html"><![CDATA[<p><img src="/assets/images/thumbnails/flaskapp.png" alt="Flask App"></p><p>I was suddenly in the need of deploying a very basic Flask API to the cloud, so it can be available to the public.</p><p>The thing is, Flask is not made for a production and scalable environment, but if you <em>only</em> need to deploy a very basic web server to Heroku then this guide is for you.</p><h2 id="Initialize-a-repository"><a href="#Initialize-a-repository" class="headerlink" title="Initialize a repository"></a>Initialize a repository</h2><p>First of all we will need to set up a virtual enviroment with <strong>Pipenv</strong>, a <strong>Flask</strong> app, and <strong>initialize</strong> a repository.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># create a new folder where we'll initialize our repo.</span></span><br><span class="line">$ mkdir flask_app</span><br><span class="line"></span><br><span class="line"><span class="comment"># Install Flask and Gunicorn</span></span><br><span class="line">$ pipenv install flask gunicorn</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create the necessary files</span></span><br><span class="line">$ touch runtime.txt Procfile app.py</span><br><span class="line"></span><br><span class="line"><span class="comment"># Initialize a repository</span></span><br><span class="line">$ git init</span><br></pre></td></tr></table></figure><h2 id="Create-a-simple-flask-app"><a href="#Create-a-simple-flask-app" class="headerlink" title="Create a simple flask app"></a>Create a simple flask app</h2><p>Open <strong>app.py</strong> and create a very simplistic app that returns a json string to see if it can be run with <strong>gunicorn</strong>:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># app.py</span></span><br><span class="line"><span class="string">"""Flask App Project."""</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask, jsonify</span><br><span class="line">app = Flask(__name__)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route('/')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">index</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Return homepage."""</span></span><br><span class="line">    json_data = &#123;<span class="string">'Hello'</span>: <span class="string">'World!'</span>&#125;</span><br><span class="line">    <span class="keyword">return</span> jsonify(json_data)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    app.run()</span><br></pre></td></tr></table></figure><h2 id="Run-your-Flask-app-with-gunicorn"><a href="#Run-your-Flask-app-with-gunicorn" class="headerlink" title="Run your Flask app with gunicorn"></a>Run your Flask app with gunicorn</h2><p>Heroku will use <strong>Gunicorn</strong> to act as a web server for our Flask app, we have already installed gunicorn with pipenv, we have to run it with our flask app to make sure it works.</p><figure class="highlight shell"><table><tr><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> pipenv run gunicorn app:app</span></span><br><span class="line">[2018-05-11 18:18:13 -0600] [6508] [INFO] Starting gunicorn 19.8.1</span><br><span class="line">[2018-05-11 18:18:13 -0600] [6508] [INFO] Listening at: http://127.0.0.1:8000 (6508)</span><br><span class="line">[2018-05-11 18:18:13 -0600] [6508] [INFO] Using worker: sync</span><br><span class="line">[2018-05-11 18:18:13 -0600] [6561] [INFO] Booting worker with pid: 6561</span><br></pre></td></tr></table></figure><p>It seems that it works flawlessly, let’s visit our project’s home page.</p><p><img src="/assets/images/heroku-flask/working_homepage.png" alt="JSON Flask App"></p><p>And it is working as it should. Now we have to deploy our project with Heroku.</p><h2 id="Deployment-to-Heroku"><a href="#Deployment-to-Heroku" class="headerlink" title="Deployment to Heroku"></a>Deployment to Heroku</h2><p>To deploy our project we will need to edit two files:</p><ul><li><strong>Procfile</strong></li><li><strong>runtime.txt</strong></li></ul><p>A <a href="https://devcenter.heroku.com/articles/procfile" target="_blank" rel="noopener"><strong>Procfile</strong></a> will be used to let Heroku know how to handle our web server.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">web: gunicorn app:app</span><br></pre></td></tr></table></figure><p>By default gunicorn bind the service on port 8000, if we wanted to change this port we could do so by appending the argument <code>--bind ip_address:port</code>. We can also pass environment variables in case that Heroku have a specific port.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">web: gunicorn app:app --bind 0.0.0.0:$PORT</span><br></pre></td></tr></table></figure><p>Now let’s open <strong>runtime.txt</strong>, this file let Heroku know what version of python are we using.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">python-3.6.5</span><br></pre></td></tr></table></figure><p>We’re letting Heroku know that we’re using Python 3.6.5 (which at the time is the latest and stable one).</p><p>Once we have defined our runtime, we should commit our changes.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># add files</span></span><br><span class="line">$ git add -A</span><br><span class="line"></span><br><span class="line"><span class="comment"># commit them</span></span><br><span class="line">$ git commit -am <span class="string">"first commit"</span></span><br></pre></td></tr></table></figure><p>Once we have a working directory tree, we should be able to create a Heroku app and push our project.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># create a heroku app, you can leave name_here empty</span></span><br><span class="line"><span class="comment"># if you wish heroku to pick a name for you.</span></span><br><span class="line">$ heroku create name_here</span><br><span class="line">Creating app... <span class="keyword">done</span>, ⬢ name_here</span><br><span class="line">https://name_here.herokuapp.com/ | https://git.heroku.com/name_here.git</span><br><span class="line"></span><br><span class="line"><span class="comment"># now we can push our project to our heroku app</span></span><br><span class="line">$ git push heroku master</span><br><span class="line">-- SNIP --</span><br><span class="line">remote: -----&gt; Python app detected</span><br><span class="line">remote: -----&gt; Installing python-3.6.5</span><br><span class="line">remote: -----&gt; Installing pip</span><br><span class="line">remote: -----&gt; Installing dependencies with Pipenv 11.8.2…</span><br><span class="line">remote:        Installing dependencies from Pipfile.lock (59a99c)…</span><br><span class="line">remote: -----&gt; Discovering process types</span><br><span class="line">remote:        Procfile declares types -&gt; web</span><br><span class="line">remote:</span><br><span class="line">remote: -----&gt; Compressing...</span><br><span class="line">remote:        Done: 53.9M</span><br><span class="line">remote: -----&gt; Launching...</span><br><span class="line">remote:        Released v3</span><br><span class="line">remote:        https://floating-badlands-13121.herokuapp.com/ deployed to Heroku</span><br><span class="line">remote:</span><br><span class="line">remote: Verifying deploy... <span class="keyword">done</span>.</span><br><span class="line">To https://git.heroku.com/floating-badlands-13121.git</span><br><span class="line"> * [new branch]      master -&gt; master</span><br></pre></td></tr></table></figure><p>And we’re done, we can visit our app by executing <code>heroku open</code>, it will open a browser tab with your app’s domain name.</p><h2 id="flask-heroku-example"><a href="#flask-heroku-example" class="headerlink" title="flask-heroku-example"></a>flask-heroku-example</h2><p>I wrote an example repository that already have a runtime and Procfile so you can deploy your flask project to Heroku, <a href="https://github.com/franccesco/flask-heroku-example" target="_blank" rel="noopener">here’s the repo</a>.</p><p>If you have any questions, feel free to comment bellow. Have fun.</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> web-development </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Importing Variables From a Web Site</title>
      <link href="2018/04/17/case-import-variables-from-a-web-site-Python/"/>
      <url>2018/04/17/case-import-variables-from-a-web-site-Python/</url>
      
        <content type="html"><![CDATA[<!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1548943487-a2e4e43b4853?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="seafood ramen platter " title="seafood ramen platter ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/YmyFBvW7oG8" target="_blank">Photo</a> by <a href="https://unsplash.com/@nickkarvounis" target="_blank">@nickkarvounis</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 01/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><h2 id="Before-we-start-A-short-disclaimer"><a href="#Before-we-start-A-short-disclaimer" class="headerlink" title="Before we start: A short disclaimer"></a>Before we start: A short disclaimer</h2><p><strong>Don’t do this.</strong> Don’t go there importing random code you found on the internet into <em>your</em> code because that can be dangerous and code would be injected into your machine or a client’s machine and <strong>you don’t want that.</strong> Remember to <em>always</em> have your code and modules in a <strong>version control</strong> system and that you have complete knowledge of what you’re loading.</p><hr><p>Today I was talking with some of the guys in a Discord server about python, called <a href="https://pythondiscord.com/" target="_blank" rel="noopener"><strong>Python Discord</strong></a> (which I definitely recommend you to check out!), and stumbled upon a guy who was requesting assistance.</p><p><img src="/assets/images/import_val_case/conv1.png" alt="Help"><br>(Yeah… I’m <strong>theinquisitor.</strong>)</p><p>What he wanted to do was to <strong>import a variable that contains a list object,</strong> which is perfectly fine, the only issue was that the variable was <strong>in a web page called <a href="https://dumptext.com/Ai9Ww8j4/raw/" target="_blank" rel="noopener">dumptext.com</a>.</strong></p><p><img src="/assets/images/import_val_case/impval1.png" alt="List Snippet"></p><p>And to make things worst, when you inspect the web page’s source code you find that the text isn’t even <em>raw</em> text, I don’t know <strong><em>why</em></strong> would anyone add a <em>raw</em> button that doesn’t even return <em>raw</em> text.</p><p><img src="/assets/images/import_val_case/impval2.png" alt="List Snippet"></p><p>This is a <strong>bad idea</strong>, but hey: <em>I like to solve problems</em>, and we will work with this scenario just for the fun of it, so let’s start with <strong>Solution #1.</strong></p><h2 id="Solution-1-Create-a-module-containing-the-variable-name"><a href="#Solution-1-Create-a-module-containing-the-variable-name" class="headerlink" title="Solution #1: Create a module containing the variable name"></a>Solution #1: Create a module containing the variable name</h2><p>The first solution will consist in:</p><ol><li><strong>Request</strong> the web page HTML’s source code.</li><li><strong>Parse</strong> the HTML code with BeautifulSoup</li><li><strong>Extract</strong> the contents inside the HTML’s <code>&lt;pre&gt;&lt;/pre&gt;</code> tags which contains the list we want.</li><li><strong>Save</strong> the list into a module called <code>wordlist.py</code> that we can import.</li><li><strong>Import</strong> the variable <code>words</code> from our newly created <code>wordlist.py</code> module.</li></ol><p>Before we start, we have to create an empty module first that will hold our data:<br><code>wordlist.py</code></p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">$ touch wordlist.py</span><br></pre></td></tr></table></figure><p>Now we proceed to create our main python code called:<br><code>solution1.py</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="string">"""Solution 1."""</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"><span class="keyword">from</span> importlib <span class="keyword">import</span> reload</span><br><span class="line"></span><br><span class="line"><span class="comment"># our module containing the variable with the list object</span></span><br><span class="line"><span class="keyword">import</span> wordlist</span><br><span class="line"></span><br><span class="line"><span class="comment"># request the HTML source and extract the text inside the '&lt;pre&gt;&lt;/pre&gt;' tags.</span></span><br><span class="line">dumptext_data = requests.get(<span class="string">'https://dumptext.com/Ai9Ww8j4/raw/'</span>).text</span><br><span class="line">parsed_dumptext_html = BeautifulSoup(dumptext_data, <span class="string">'html.parser'</span>)</span><br><span class="line">scraped_list = parsed_dumptext_html.pre.text</span><br><span class="line"></span><br><span class="line"><span class="comment"># save the contents of the scraped_list into a module</span></span><br><span class="line"><span class="keyword">with</span> open(<span class="string">'wordlist.py'</span>, <span class="string">'w'</span>) <span class="keyword">as</span> wordlist_object:</span><br><span class="line">    wordlist_object.write(scraped_list)</span><br><span class="line"></span><br><span class="line"><span class="comment"># reload the module to import the variable 'words'</span></span><br><span class="line">reload(wordlist)</span><br><span class="line"></span><br><span class="line"><span class="comment"># now we can use the variable words everywhere we want.</span></span><br><span class="line">words = wordlist.words</span><br><span class="line">print(type(words))</span><br><span class="line">print(words)</span><br></pre></td></tr></table></figure><p>Now, we are familiar with the packages <code>requests</code> and <code>BeautifulSoup</code>, but what about <code>importlib</code>? Well, <code>importlib</code> have a lot of utilities for our <em>imports</em>. In this case we imported the module <code>reload()</code> from <code>importlib</code> that allows us to <em>reload</em> a previous imported module, fresh as new, <strong>including the newly created variable holding our list</strong>. Remember that PEP8 recommends adding your imports at the top of the file, following your docstring, not in the middle of the file.</p><p>Let’s see if our solution works well:</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">$ python solution1.py</span><br><span class="line">&lt;class <span class="string">'list'</span>&gt;</span><br><span class="line">[<span class="string">'aah'</span>, <span class="string">'aal'</span>, <span class="string">'aas'</span>, <span class="string">'aba'</span>, <span class="string">'abo'</span>, <span class="string">'abs'</span>, <span class="string">'aby'</span>, <span class="string">'ace'</span>, <span class="string">'act'</span>, <span class="string">'add'</span>, <span class="string">'ado'</span>, <span class="string">'ads'</span>, <span class="string">'adz'</span>, <span class="string">'aff'</span>, <span class="string">'aft'</span>, <span class="string">'aga'</span>, <span class="string">'age'</span>, <span class="string">'ago'</span>, <span class="string">'ags'</span>, <span class="string">'aha'</span>, <span class="string">'ahi'</span>, <span class="string">'ahs'</span>, <span class="string">'aid'</span>, <span class="string">'ail'</span>, <span class="string">'aim'</span>, <span class="string">'ain'</span>, <span class="string">'air'</span>, <span class="string">'ais'</span>, <span class="string">'ait'</span>, <span class="string">'ala'</span>, <span class="string">'alb'</span>, <span class="string">'ale'</span>, <span class="string">'all'</span>, <span class="string">'alp'</span>, <span class="string">'als'</span>, <span class="string">'alt'</span>, <span class="string">'ama'</span>, <span class="string">'ami'</span>, <span class="string">'amp'</span>, <span class="string">'amu'</span>, <span class="string">'ana'</span>, <span class="string">'and'</span>, <span class="string">'ane'</span>, <span class="string">'ani'</span>, <span class="string">'ant'</span>, <span class="string">'any'</span>, ...]</span><br></pre></td></tr></table></figure><p><strong>It works!</strong>, we can see that our it is an object from the class <em>list</em> and it prints successfully.</p><p>There’s only one thing… <strong><em>we shouldn’t do this</em></strong>. Importing random code from the Internet is not a great option, believe me. You could do this in <em>very</em> specific scenario where you absolutely don’t have any choice, the file is read only and also <em>you</em> are the owner. If it’s a controlled file then good, but realistically I don’t see that happening, so please, avoid this solution.</p><p><strong>It was fun to write though.</strong></p><h2 id="Solution-2-A-Regular-Expression-to-Match-the-List-Items"><a href="#Solution-2-A-Regular-Expression-to-Match-the-List-Items" class="headerlink" title="Solution #2: A Regular Expression to Match the List Items"></a>Solution #2: A Regular Expression to Match the List Items</h2><p>This is a much better (and cleaner) solution, what we’ll do is the same scraping as before, but instead of creating and loading a module, we will pick everything inside a pair of single quotes (<code>&#39;&#39;</code>) from the parsed HTML code with a <em>regular expression</em> and add those results to a list.</p><p>Let’s jump straight into the code:<br><code>solution2.py</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="string">"""Solution 2."""</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"></span><br><span class="line"><span class="comment"># request the HTML source and extract the text inside the '&lt;pre&gt;&lt;/pre&gt;' tags.</span></span><br><span class="line">dumptext_data = requests.get(<span class="string">'https://dumptext.com/Ai9Ww8j4/raw/'</span>).text</span><br><span class="line">parsed_dumptext_html = BeautifulSoup(dumptext_data, <span class="string">'html.parser'</span>)</span><br><span class="line">scraped_list = parsed_dumptext_html.pre.text</span><br><span class="line"></span><br><span class="line"><span class="comment"># define a pattern that captures everything inside a pair of quotes ('')</span></span><br><span class="line">pattern = re.compile(<span class="string">r'\'(.*?)\''</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># create a list of items using list comprehension with all the matching items</span></span><br><span class="line">words = [word <span class="keyword">for</span> word <span class="keyword">in</span> re.findall(pattern, scraped_list)]</span><br><span class="line"></span><br><span class="line"><span class="comment"># now we can use the variable words everywhere we want.</span></span><br><span class="line">print(type(words))</span><br><span class="line">print(words)</span><br></pre></td></tr></table></figure><p>Let’s see if it works correctly:</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">$ python solution2.py</span><br><span class="line">&lt;class <span class="string">'list'</span>&gt;</span><br><span class="line">[<span class="string">'aah'</span>, <span class="string">'aal'</span>, <span class="string">'aas'</span>, <span class="string">'aba'</span>, <span class="string">'abo'</span>, <span class="string">'abs'</span>, <span class="string">'aby'</span>, <span class="string">'ace'</span>, <span class="string">'act'</span>, <span class="string">'add'</span>, <span class="string">'ado'</span>, <span class="string">'ads'</span>, <span class="string">'adz'</span>, <span class="string">'aff'</span>, <span class="string">'aft'</span>, <span class="string">'aga'</span>, <span class="string">'age'</span>, <span class="string">'ago'</span>, <span class="string">'ags'</span>, <span class="string">'aha'</span>, <span class="string">'ahi'</span>, <span class="string">'ahs'</span>, <span class="string">'aid'</span>, <span class="string">'ail'</span>, <span class="string">'aim'</span>, <span class="string">'ain'</span>, <span class="string">'air'</span>, <span class="string">'ais'</span>, <span class="string">'ait'</span>, <span class="string">'ala'</span>, <span class="string">'alb'</span>, <span class="string">'ale'</span>, <span class="string">'all'</span>, <span class="string">'alp'</span>, <span class="string">'als'</span>, <span class="string">'alt'</span>, <span class="string">'ama'</span>, <span class="string">'ami'</span>, <span class="string">'amp'</span>, <span class="string">'amu'</span>, <span class="string">'ana'</span>, <span class="string">'and'</span>, <span class="string">'ane'</span>, <span class="string">'ani'</span>, <span class="string">'ant'</span>, <span class="string">'any'</span>, ...]</span><br></pre></td></tr></table></figure><p>That’s it!, no need for stinkin’ modules and injecting random code into our code, no importing and no reloading:</p><ol><li>We imported the library <code>re</code> which aid us to search for strings using <em>regular expression</em> (Or <em>RegEx</em>).</li><li>We use the same scraping technique as before.</li><li>Now we define a RegEx pattern with <code>re.compile(r&#39;__pattern__&#39;)</code></li><li>Create a List of words, where <code>word</code> is the result of all the words found in <code>re.findall(pattern, scraped_list)</code></li></ol><p>And that all, you can now use your <code>words</code> in a <strong><em>lot</em></strong> more safer way.</p><h1 id="Python-Discord"><a href="#Python-Discord" class="headerlink" title="Python Discord"></a>Python Discord</h1><p>If you’re interested in being part of a community, then I cannot recommend you <strong>Python Discord</strong> more. It has helped me to learn so <em>much</em> about Python, people is really helpful and we’re are <em>always growing</em>.</p><p>So please, be my guest and hop in this awesome community, here’s your invite: <a href="https://pythondiscord.com/invite" target="_blank" rel="noopener">https://pythondiscord.com/invite</a></p><p><strong>Everyone</strong> is welcome.</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> web-scraping </tag>
            
            <tag> crazy </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Create Multiple Directories With Makedirs - Python</title>
      <link href="2018/04/17/create-multiple-directories-with-makedirs-Python/"/>
      <url>2018/04/17/create-multiple-directories-with-makedirs-Python/</url>
      
        <content type="html"><![CDATA[<p>You don’t need a bunch of <em>if’s</em> and <em>else if’s</em> to create an array of directories in python, just the good ol’ <code>makedirs</code>.</p><p><img src="/assets/images/thumbnails/makedirs.png" alt="makedirs"></p><h2 id="Create-an-array-of-directories"><a href="#Create-an-array-of-directories" class="headerlink" title="Create an array of directories"></a>Create an array of directories</h2><p>To create an array of directories you must import the package <code>os</code> and import the method <code>makedirs</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> os <span class="keyword">import</span> makedirs</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>makedirs(<span class="string">'1/2/3/4/5'</span>)</span><br></pre></td></tr></table></figure><p>You’ll see now that your directories are created correctly, let’s run the command <code>tree</code> outside of python to see if they’re were actually created.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ tree 1</span><br><span class="line">1</span><br><span class="line">└── 2</span><br><span class="line">    └── 3</span><br><span class="line">        └── 4</span><br><span class="line">            └── 5</span><br><span class="line"></span><br><span class="line">4 directories, 0 files</span><br></pre></td></tr></table></figure><p>You can see that all our directories where created (it says 4 because it is excluding the root directory - 1).</p><p>We can create any array of directories as long as the <em>last</em> directory that we want to create is <em>not</em> already created. Let’s create the directory number <code>6</code>.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>makedirs(<span class="string">'1/2/3/4/5/6'</span>)</span><br></pre></td></tr></table></figure><p>Python does not complain, usually means it all went well, let’s inspect our tree.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ tree 1</span><br><span class="line">1</span><br><span class="line">└── 2</span><br><span class="line">    └── 3</span><br><span class="line">        └── 4</span><br><span class="line">            └── 5</span><br><span class="line">                └── 6</span><br><span class="line"></span><br><span class="line">5 directories, 0 files</span><br></pre></td></tr></table></figure><p>Perfect, there are 5 directories created under <code>1</code> now. Would <code>makedirs</code> complain if we wanted to overwrite the same directory structure?</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> os <span class="keyword">import</span> makedirs</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>makedirs(<span class="string">'1/2/3/4/5/6'</span>)</span><br><span class="line">Traceback (most recent call last):</span><br><span class="line">  File <span class="string">"&lt;stdin&gt;"</span>, line <span class="number">1</span>, <span class="keyword">in</span> &lt;module&gt;</span><br><span class="line">  File <span class="string">"/home/franccesco/.pyenv/versions/3.6.5/lib/python3.6/os.py"</span>, line <span class="number">220</span>, <span class="keyword">in</span> makedirs</span><br><span class="line">    mkdir(name, mode)</span><br><span class="line">FileExistsError: [Errno <span class="number">17</span>] File exists: <span class="string">'1/2/3/4/5/6'</span></span><br></pre></td></tr></table></figure><p>Yes, this is because <code>makedirs</code> couldn’t find a single directory to create (because they’re already created), but sometimes we don’t want <code>makedirs</code> to make a rant and exits abruptly if it encounters that the directory structure is already there.</p><p>To suppress this behavior we can pass the argument <code>exist_ok=True</code> to avoid <code>makedirs</code> to raise an exception.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> os <span class="keyword">import</span> makedirs</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>makedirs(<span class="string">'1/2/3/4/5/6'</span>, exist_ok=<span class="literal">True</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>makedirs(<span class="string">'1/2/3/4/5/6'</span>, exist_ok=<span class="literal">True</span>)</span><br></pre></td></tr></table></figure><p>See? Now <code>makedirs</code> doesn’t complain if there’s already a directory structure created, our directories are still created, or just ignored if they’re already there.</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ tree 1</span><br><span class="line">1</span><br><span class="line">└── 2</span><br><span class="line">    └── 3</span><br><span class="line">        └── 4</span><br><span class="line">            └── 5</span><br><span class="line">                └── 6</span><br><span class="line"></span><br><span class="line">5 directories, 0 files</span><br></pre></td></tr></table></figure>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> file-management </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>How to Handle a Deque in Python</title>
      <link href="2018/04/14/Deques-in-Python/"/>
      <url>2018/04/14/Deques-in-Python/</url>
      
        <content type="html"><![CDATA[<p>Deques are a great way to handle memory efficient appends to a list-like object, it is a special module that allows you to handle list items in a more appropriate way.</p><p><img src="/assets/images/thumbnails/deques.png" alt="Deque example"></p><h2 id="Create-a-deque"><a href="#Create-a-deque" class="headerlink" title="Create a deque"></a>Create a deque</h2><p>To create a list simply import the <code>deque</code> module from <code>collections</code> library and call <code>deque(_items_)</code> on a variable.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> collections <span class="keyword">import</span> deque</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq = deque(<span class="string">'123'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>])</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>type(dq)</span><br><span class="line">&lt;<span class="class"><span class="keyword">class</span> '<span class="title">collections</span>.<span class="title">deque</span>'&gt;</span></span><br></pre></td></tr></table></figure><p>Or if you wish to create an empty <code>deque</code>.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq = deque()</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([])</span><br></pre></td></tr></table></figure><p>What happens if you want to create a deque of integers?</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq = deque(<span class="number">123</span>)</span><br><span class="line">Traceback (most recent call last):</span><br><span class="line">  File <span class="string">"&lt;stdin&gt;"</span>, line <span class="number">1</span>, <span class="keyword">in</span> &lt;module&gt;</span><br><span class="line">TypeError: <span class="string">'int'</span> object <span class="keyword">is</span> <span class="keyword">not</span> iterable</span><br></pre></td></tr></table></figure><p>You simply can’t. Why is that? Because Integers are not iterable in python but String <em>are</em>.</p><p>This is because Integers, unlike strings, don’t have a <code>__iter__</code> method and therefore they don’t return iterables.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>str.__iter__</span><br><span class="line">&lt;slot wrapper <span class="string">'__iter__'</span> of <span class="string">'str'</span> objects&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>int.__iter__</span><br><span class="line">Traceback (most recent call last):</span><br><span class="line">  File <span class="string">"&lt;stdin&gt;"</span>, line <span class="number">1</span>, <span class="keyword">in</span> &lt;module&gt;</span><br><span class="line">AttributeError: type object <span class="string">'int'</span> has no attribute <span class="string">'__iter__'</span></span><br></pre></td></tr></table></figure><h2 id="Accessing-items-by-index"><a href="#Accessing-items-by-index" class="headerlink" title="Accessing items by index"></a>Accessing items by index</h2><p>We can access items in a deque with an index number.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq[<span class="number">0</span>]</span><br><span class="line"><span class="string">'1'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq[<span class="number">1</span>]</span><br><span class="line"><span class="string">'2'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq[<span class="number">2</span>]</span><br><span class="line"><span class="string">'3'</span></span><br></pre></td></tr></table></figure><h2 id="Converting-an-item-to-integer"><a href="#Converting-an-item-to-integer" class="headerlink" title="Converting an item to integer"></a>Converting an item to integer</h2><p>You can convert an item to a item simply by wrapping them around an <code>int()</code> method.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>one = int(dq[<span class="number">0</span>])</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>type(one)</span><br><span class="line">&lt;<span class="class"><span class="keyword">class</span> '<span class="title">int</span>'&gt;</span></span><br><span class="line"><span class="class">&gt;&gt;&gt; <span class="title">one</span></span></span><br><span class="line"><span class="class">1</span></span><br></pre></td></tr></table></figure><h2 id="Appending-items-to-a-deque"><a href="#Appending-items-to-a-deque" class="headerlink" title="Appending items to a deque"></a>Appending items to a deque</h2><p>We can append new items to our deque, to either left side or right side.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq.append(<span class="string">'4'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>])</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq.appendleft(<span class="string">'0'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'0'</span>, <span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>])</span><br></pre></td></tr></table></figure><h2 id="Extending-our-deque"><a href="#Extending-our-deque" class="headerlink" title="Extending our deque"></a>Extending our deque</h2><p>We can also add multiple values at once.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq.extend(<span class="string">'567'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'0'</span>, <span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>, <span class="string">'5'</span>, <span class="string">'6'</span>, <span class="string">'7'</span>])</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq.extendleft(<span class="string">'cba'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'a'</span>, <span class="string">'b'</span>, <span class="string">'c'</span>, <span class="string">'0'</span>, <span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>, <span class="string">'5'</span>, <span class="string">'6'</span>, <span class="string">'7'</span>])</span><br></pre></td></tr></table></figure><h2 id="Popping-items"><a href="#Popping-items" class="headerlink" title="Popping items"></a>Popping items</h2><p>We can delete items in both sides.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq.pop()</span><br><span class="line"><span class="string">'7'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'a'</span>, <span class="string">'b'</span>, <span class="string">'c'</span>, <span class="string">'0'</span>, <span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>, <span class="string">'5'</span>, <span class="string">'6'</span>])</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq.popleft()</span><br><span class="line"><span class="string">'a'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'b'</span>, <span class="string">'c'</span>, <span class="string">'0'</span>, <span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>, <span class="string">'5'</span>, <span class="string">'6'</span>])</span><br></pre></td></tr></table></figure><h2 id="Rotating-items"><a href="#Rotating-items" class="headerlink" title="Rotating items"></a>Rotating items</h2><p>Or rotate our items if we want.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq.rotate(<span class="number">-2</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'0'</span>, <span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>, <span class="string">'5'</span>, <span class="string">'6'</span>, <span class="string">'b'</span>, <span class="string">'c'</span>])</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq.rotate(<span class="number">2</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq</span><br><span class="line">deque([<span class="string">'b'</span>, <span class="string">'c'</span>, <span class="string">'0'</span>, <span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>, <span class="string">'5'</span>, <span class="string">'6'</span>])</span><br></pre></td></tr></table></figure><h2 id="Slicing-deques"><a href="#Slicing-deques" class="headerlink" title="Slicing deques"></a>Slicing deques</h2><p>We cannot slice our deques, at least not <em>directly</em>.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dq[<span class="number">2</span>:]</span><br><span class="line">Traceback (most recent call last):</span><br><span class="line">  File <span class="string">"&lt;stdin&gt;"</span>, line <span class="number">1</span>, <span class="keyword">in</span> &lt;module&gt;</span><br><span class="line">TypeError: sequence index must be integer, <span class="keyword">not</span> <span class="string">'slice'</span></span><br></pre></td></tr></table></figure><p>You can import <code>itertools</code> and return a sliced list (not a deque) of items with <code>islice()</code> method.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">import</span> itertools</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>list(itertools.islice(dq, <span class="number">2</span>, <span class="number">9</span>))</span><br><span class="line">[<span class="string">'0'</span>, <span class="string">'1'</span>, <span class="string">'2'</span>, <span class="string">'3'</span>, <span class="string">'4'</span>, <span class="string">'5'</span>, <span class="string">'6'</span>]</span><br></pre></td></tr></table></figure><p>You can find more information about deques in the official Python documentation: <a href="https://docs.python.org/3/library/collections.html#collections.deque" target="_blank" rel="noopener">Deque Objects</a></p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Create a Basic Api With Flask</title>
      <link href="2018/04/07/create-a-basic-api-with-flask/"/>
      <url>2018/04/07/create-a-basic-api-with-flask/</url>
      
        <content type="html"><![CDATA[<h2 id="What-is-Flask"><a href="#What-is-Flask" class="headerlink" title="What is Flask?"></a>What is Flask?</h2><blockquote><p>Flask is a microframework for Python based on Werkzeug, Jinja 2 and good intentions.</p></blockquote><p>Flask is a <em>microframework</em> for Python that you can use to quickly create API’s and websites. It’s a great and easy to use platform, let’s create a simple API, but first we will go through the basics.</p><h2 id="Flask-Installation"><a href="#Flask-Installation" class="headerlink" title="Flask Installation"></a>Flask Installation</h2><p>We’re going to install Flask in our virtual environment to later import it into our code.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pipenv install flask</span><br></pre></td></tr></table></figure><h2 id="A-basic-web-server"><a href="#A-basic-web-server" class="headerlink" title="A basic web server"></a>A basic web server</h2><p>Ok, now that we have installed Flask in our virtual environment, we will create the main page, let’s create a file named <code>api.py</code> and input the following code snippet.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="string">"""Flask API."""</span></span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</span><br><span class="line">app = Flask(__name__)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route('/')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">index</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Return main page."""</span></span><br><span class="line">    <span class="keyword">return</span> <span class="string">'It Works!'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    app.run()</span><br></pre></td></tr></table></figure><p>That’s enough to lift our API and return <strong>It Works!</strong> in the main page.</p><p><img src="/assets/images/flask/Selection_001.png" alt="Main Page"></p><h2 id="Working-with-Variables"><a href="#Working-with-Variables" class="headerlink" title="Working with Variables"></a>Working with Variables</h2><p>Let’s create a function to our, let’s add a <code>salute(name)</code> method that accepts one argument as a name and returns <code>&#39;Hello, {name}!&#39;</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="string">"""Flask API."""</span></span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</span><br><span class="line">app = Flask(__name__)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route('/')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">index</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Return main page."""</span></span><br><span class="line">    <span class="keyword">return</span> <span class="string">'It Works!'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route('/&lt;name&gt;')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">salute</span><span class="params">(name)</span>:</span></span><br><span class="line">    <span class="string">"""Salute someone."""</span></span><br><span class="line">    <span class="keyword">return</span> <span class="string">f'Hello, <span class="subst">&#123;name&#125;</span>!'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    app.run()</span><br></pre></td></tr></table></figure><p><img src="/assets/images/flask/Selection_002.png" alt="salutation"></p><p>You can also define if a variable should be Integer, String, etc, like this:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">@app.route('/&lt;string:name&gt;')</span></span><br></pre></td></tr></table></figure><p>Here’s a <a href="http://flask.pocoo.org/docs/0.12/quickstart/#variable-rules" target="_blank" rel="noopener" title="Variable Rules">reference table</a>:</p><table><thead><tr><th align="left">Type</th><th align="left">Comment</th></tr></thead><tbody><tr><td align="left">string</td><td align="left">Accepts a string without slashes</td></tr><tr><td align="left">int</td><td align="left">Accepts integers</td></tr><tr><td align="left">float</td><td align="left">For floating point value</td></tr><tr><td align="left">path</td><td align="left">Accept strings with slashe</td></tr><tr><td align="left">any</td><td align="left">Accepts one of the provided item</td></tr><tr><td align="left">UUID</td><td align="left">Accepts UUID strings</td></tr></tbody></table><h2 id="Return-a-JSON-Encoded-String"><a href="#Return-a-JSON-Encoded-String" class="headerlink" title="Return a JSON Encoded String"></a>Return a JSON Encoded String</h2><p>We can <strong>Jsonify</strong> a dictionary and return it to the browser as a JSON serialized data, let’s input our dog data into a dictionary and serialize it:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment">#  -- SNIP --</span></span><br><span class="line"><span class="meta">@app.route('/mydog')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_dog</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Return JSON data about our dog."""</span></span><br><span class="line">    doggo = &#123;<span class="string">'name'</span>: <span class="string">'sam'</span>, <span class="string">'food'</span>: <span class="string">'meat'</span>, <span class="string">'hobby'</span>: <span class="string">'sleep'</span>&#125;</span><br><span class="line">    <span class="keyword">return</span> jsonify(doggo)</span><br></pre></td></tr></table></figure><p>And when we go to <code>/mydog</code> URL we can see our serialized dictionary:</p><p><img src="/assets/images/flask/Selection_003.png" alt="serialized data"></p><h2 id="A-Flask-Project"><a href="#A-Flask-Project" class="headerlink" title="A Flask Project"></a>A Flask Project</h2><p>Let’s create a little Flask project to warm up, shall we? We’re going to do an API that scans localhost with Nmap and return its open ports with only one method.</p><p><code>api.py</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="string">"""An Nmap Scanner and Parser."""</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> xmltodict</span><br><span class="line"><span class="keyword">from</span> subprocess <span class="keyword">import</span> run</span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask, jsonify</span><br><span class="line">app = Flask(__name__)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route('/')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">index</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Return the main page."""</span></span><br><span class="line">    <span class="keyword">return</span> <span class="string">"Nmap Scanner, a more sofisticated page is coming soon."</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route('/openports')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">scan_localhost</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Scan a host with Nmap."""</span></span><br><span class="line">    <span class="comment"># run nmap scan from another proccess</span></span><br><span class="line">    run([<span class="string">'nmap'</span>, <span class="string">'-T5'</span>, <span class="string">'--open'</span>, <span class="string">'-oX'</span>, <span class="string">'scan.xml'</span>, <span class="string">'localhost'</span>])</span><br><span class="line"></span><br><span class="line">    <span class="comment"># parse Nmap XML report and covert it to a dictionary</span></span><br><span class="line">    <span class="keyword">with</span> open(<span class="string">'scan.xml'</span>) <span class="keyword">as</span> raw_xml:</span><br><span class="line">        nmap_scan = xmltodict.parse(raw_xml.read())</span><br><span class="line"></span><br><span class="line">    <span class="comment"># Jsonify the dictionary and return it</span></span><br><span class="line">    <span class="keyword">return</span> jsonify(nmap_scan)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    app.run()</span><br></pre></td></tr></table></figure><p>So, what are we doing here?</p><ul><li>Firstly, we import the libraries <strong>Flask</strong> for our API of course, we also import <strong>run</strong> from the package <strong>subprocess</strong> which allows us to run subprocesses from Python, the library <strong>xmltodict</strong> to parse XML data and convert it into a dictionary so we can return it later as a <strong>Jsonified</strong> data.</li><li>Now we defined our route <code>openports</code><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">@app.route('/openports')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">scan_localhost</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""Scan a host with Nmap."""</span></span><br></pre></td></tr></table></figure></li><li>Spawn a Nmap subprocess that scan our localhost looking for open ports only, we designate our XML output to <code>scan.xml</code>.<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># run nmap scan from another proccess</span></span><br><span class="line">run([<span class="string">'nmap'</span>, <span class="string">'-T5'</span>, <span class="string">'--open'</span>, <span class="string">'-oX'</span>, <span class="string">'scan.xml'</span>, <span class="string">'localhost'</span>])</span><br></pre></td></tr></table></figure></li><li>We open our report <code>scan.xml</code>, convert it to a dictionary with <code>xmltodict.parse(raw_xml.read())</code> and store it into <code>nmap_scan</code>.<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># parse Nmap XML report and covert it to a dictionary</span></span><br><span class="line"><span class="keyword">with</span> open(<span class="string">'scan.xml'</span>) <span class="keyword">as</span> raw_xml:</span><br><span class="line">    nmap_scan = xmltodict.parse(raw_xml.read())</span><br></pre></td></tr></table></figure></li><li>Last, but not least, we return the dictionary into a Jsonified data.<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Jsonify the dictionary and return it</span></span><br><span class="line"><span class="keyword">return</span> jsonify(nmap_scan)</span><br></pre></td></tr></table></figure></li></ul><p>And when we visit <a href="https://localhost/openports" target="_blank" rel="noopener">https://localhost/openports</a> we can successfully see our API in action:<br><img src="/assets/images/flask/api_nmap.png" alt="API Nmap"></p><p>Of course that our code could be way, <em>waaaaay</em> better, but a simple example will suffice.</p><p>Remember to checkout the Flask documentation, it is immensely useful and <a href="http://flask.pocoo.org/docs/0.12/" target="_blank" rel="noopener">we only scratched the surface here</a>.</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> web-development </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Create and Object With Namedtuple</title>
      <link href="2018/04/07/Create-and-object-with-namedtuple/"/>
      <url>2018/04/07/Create-and-object-with-namedtuple/</url>
      
        <content type="html"><![CDATA[<p>Namedtuples are a subclass of <em>typename</em> which allow us to create objects (or classes) with tuple-like attributes, it’s easy to create a class with immutable attributes with this library.</p><h2 id="Creating-an-object"><a href="#Creating-an-object" class="headerlink" title="Creating an object"></a>Creating an object</h2><p>To create a namedtuple we only need the name of the class <em>first</em> and then the attribute list.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> collections <span class="keyword">import</span> namedtuple</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>Person = namedtuple(<span class="string">'Person'</span>, <span class="string">'name job_position sex married'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>john = Person(<span class="string">'john'</span>, <span class="string">'developer'</span>, <span class="string">'male'</span>, <span class="literal">False</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>john</span><br><span class="line">Person(name=<span class="string">'john'</span>, job_position=<span class="string">'developer'</span>, sex=<span class="string">'male'</span>, married=<span class="literal">False</span>)</span><br></pre></td></tr></table></figure><p>Be aware that you can pass an argument list as a string (<strong>‘att1 att2’</strong>), as an <em>actual</em> list (<strong>[‘att1’, ‘att2’]</strong>) or a string with comma separated values (<strong>‘att1, att2’</strong>).</p><h2 id="Accessing-attributes"><a href="#Accessing-attributes" class="headerlink" title="Accessing attributes"></a>Accessing attributes</h2><p>We can access the person attributes just like any other class.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>john.name</span><br><span class="line"><span class="string">'john'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>john.job_position</span><br><span class="line"><span class="string">'developer'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>john.sex</span><br><span class="line"><span class="string">'male'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>john.married</span><br><span class="line"><span class="literal">False</span></span><br></pre></td></tr></table></figure><h2 id="Docstring"><a href="#Docstring" class="headerlink" title="Docstring"></a>Docstring</h2><p>We can also assign docstring to our attributes.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>Person.__doc__ = <span class="string">'A person attributes'</span></span><br><span class="line"><span class="string">'A person attributes'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>Person.name.__doc__ = <span class="string">"A person's name"</span></span><br><span class="line"><span class="string">"A person's name"</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>Person.job_position.__doc__ = <span class="string">"A person's job position"</span></span><br><span class="line"><span class="string">"A person's job position"</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>Person.sex.__doc__ = <span class="string">'Yes'</span></span><br><span class="line"><span class="string">'Yes'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>Person.married.__doc__ = <span class="string">'Is the person married?'</span></span><br><span class="line"><span class="string">'Is the person married?'</span></span><br></pre></td></tr></table></figure><h2 id="Changing-values"><a href="#Changing-values" class="headerlink" title="Changing values"></a>Changing values</h2><p>As our attribute are <em>tuples</em> we cannot change these attributes.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>john.name = <span class="string">'Johnny'</span></span><br><span class="line">Traceback (most recent call last):</span><br><span class="line">  File <span class="string">"&lt;stdin&gt;"</span>, line <span class="number">1</span>, <span class="keyword">in</span> &lt;module&gt;</span><br><span class="line">AttributeError: can<span class="string">'t set attribute</span></span><br></pre></td></tr></table></figure><p>But we can copy the class with the new values.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>johnny = john._replace(name=<span class="string">'johnny'</span>, married=<span class="literal">True</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>johnny</span><br><span class="line">Person(name=<span class="string">'johnny'</span>, job_position=<span class="string">'developer'</span>, sex=<span class="string">'male'</span>, married=<span class="literal">True</span>)</span><br></pre></td></tr></table></figure><h2 id="Convert-a-Dictionary-to-a-Class"><a href="#Convert-a-Dictionary-to-a-Class" class="headerlink" title="Convert a Dictionary to a Class"></a>Convert a Dictionary to a Class</h2><p>Using the same class <em>Person</em> that we defined above.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>d = &#123;<span class="string">'name'</span>: <span class="string">'john'</span>, <span class="string">'job_position'</span>: <span class="string">'developer'</span>, <span class="string">'sex'</span>: <span class="string">'male'</span>, <span class="string">'married'</span>: <span class="literal">False</span>&#125;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>john = Person(**d)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>john</span><br></pre></td></tr></table></figure><p><a href="https://docs.python.org/3.6/library/collections.html#collections.namedtuple" target="_blank" rel="noopener">You can read more about namedtuples here</a></p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> classes </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Return a List of Files and Folders With Glob</title>
      <link href="2018/03/29/return-a-list-of-files-and-absolute-path-with-python/"/>
      <url>2018/03/29/return-a-list-of-files-and-absolute-path-with-python/</url>
      
        <content type="html"><![CDATA[<p>There’s a great library added in Python 3.5 that lets you return a list of filenames and folders called <code>glob</code>, here’s how to use it.</p><h2 id="Return-files-and-folders-in-current-folder"><a href="#Return-files-and-folders-in-current-folder" class="headerlink" title="Return files and folders in current folder."></a>Return files and folders in current folder.</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>glob(<span class="string">'**'</span>)</span><br><span class="line">[<span class="string">'scaffolds'</span>, <span class="string">'node_modules'</span>, <span class="string">'yarn.lock'</span>, <span class="string">'_config.yml'</span>, <span class="string">'source'</span>, <span class="string">'db.json'</span>, <span class="string">'themes'</span>, <span class="string">'package.json'</span>, <span class="string">'package-lock.json'</span>]</span><br></pre></td></tr></table></figure><h2 id="Return-files-and-folders-recursively"><a href="#Return-files-and-folders-recursively" class="headerlink" title="Return files and folders recursively"></a>Return files and folders recursively</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>glob(<span class="string">'**'</span>, recursive=<span class="literal">True</span>)</span><br><span class="line">[<span class="string">'scaffolds'</span>, <span class="string">'scaffolds/post.md'</span>, <span class="string">'scaffolds/page.md'</span>, <span class="string">'scaffolds/draft.md'</span>, <span class="string">'node_modules'</span>, <span class="string">'...'</span>]</span><br></pre></td></tr></table></figure><h2 id="Return-only-specific-type-of-files"><a href="#Return-only-specific-type-of-files" class="headerlink" title="Return only specific type of files"></a>Return only specific type of files</h2><ul><li><p><strong>recursively</strong></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>glob(<span class="string">'*.json'</span>, recursive=<span class="literal">True</span>)</span><br><span class="line">[<span class="string">'db.json'</span>, <span class="string">'package.json'</span>, <span class="string">'package-lock.json'</span>]</span><br></pre></td></tr></table></figure></li><li><p><strong>non-recursively</strong></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>glob(<span class="string">'**/*.md'</span>, recursive=<span class="literal">True</span>)</span><br><span class="line">[<span class="string">'scaffolds/post.md'</span>, <span class="string">'scaffolds/page.md'</span>, <span class="string">'scaffolds/draft.md'</span>, <span class="string">'...'</span>]</span><br></pre></td></tr></table></figure></li><li><p><strong>recursively from another folder</strong></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>glob(<span class="string">'source/**/*.md'</span>, recursive=<span class="literal">True</span>)</span><br><span class="line">[<span class="string">'source/README.md'</span>, <span class="string">'source/about/index.md'</span>, <span class="string">'source/_posts/class-inheritance-with-python.md'</span>, <span class="string">'source/_posts/How-to-securely-store-sensitive-configuration-with-dotenv.md'</span>, <span class="string">'...'</span>]</span><br></pre></td></tr></table></figure></li></ul><h1 id="Print-a-list-of-filenames"><a href="#Print-a-list-of-filenames" class="headerlink" title="Print a list of filenames:"></a>Print a list of filenames:</h1><ul><li><p><strong>Relative path</strong></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">for</span> file <span class="keyword">in</span> glob(<span class="string">'source/**/*.md'</span>, recursive=<span class="literal">True</span>):</span><br><span class="line"><span class="meta">... </span>    print(file)</span><br><span class="line">...</span><br><span class="line"><span class="string">'source/README.md'</span></span><br><span class="line"><span class="string">'source/about/index.md'</span></span><br><span class="line"><span class="string">'source/_posts/class-inheritance-with-python.md'</span></span><br><span class="line"><span class="string">'source/_posts/How-to-securely-store-sensitive-configuration-with-dotenv.md'</span></span><br><span class="line"><span class="string">'source/_posts/Hello-all.md'</span></span><br><span class="line"><span class="string">'source/_posts/return-a-list-of-files-and-absolute-path-with-python.md'</span></span><br><span class="line"><span class="string">'source/_posts/scraping-web-data-with-requests-and-beautifulsoup-part-2.md'</span></span><br></pre></td></tr></table></figure></li><li><p><strong>Absolute path</strong></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">import</span> os</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> glob <span class="keyword">import</span> glob</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">for</span> file <span class="keyword">in</span> glob(<span class="string">'source/**/*.md'</span>, recursive=<span class="literal">True</span>):</span><br><span class="line"><span class="meta">... </span>    print(os.path.abspath(file))</span><br><span class="line">...</span><br><span class="line"><span class="string">'/home/franccesco/workspace/codingdose.info/source/README.md'</span></span><br><span class="line"><span class="string">'/home/franccesco/workspace/codingdose.info/source/about/index.md'</span></span><br><span class="line"><span class="string">'/home/franccesco/workspace/codingdose.info/source/_posts/class-inheritance-with-python.md'</span></span><br><span class="line"><span class="string">'/home/franccesco/workspace/codingdose.info/source/_posts/How-to-securely-store-sensitive-configuration-with-dotenv.md'</span></span><br><span class="line"><span class="string">'/home/franccesco/workspace/codingdose.info/source/_posts/Hello-all.md'</span></span><br><span class="line"><span class="string">'/home/franccesco/workspace/codingdose.info/source/_posts/return-a-list-of-files-and-absolute-path-with-python.md'</span></span><br><span class="line"><span class="string">'/home/franccesco/workspace/codingdose.info/source/_posts/scraping-web-data-with-requests-and-beautifulsoup-part-2.md'</span></span><br></pre></td></tr></table></figure></li></ul><p>Have fun.</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> snippet </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Unit Testing With Python: An Introduction</title>
      <link href="2018/03/27/unit-testing-with-python-an-introduction/"/>
      <url>2018/03/27/unit-testing-with-python-an-introduction/</url>
      
        <content type="html"><![CDATA[<blockquote><p>I encourage developers to see the value of unit testing; I urge them to get into the habit of writing structured tests alongside their code.<br>— <a href="https://blog.codinghorror.com/i-pity-the-fool-who-doesnt-write-unit-tests/" target="_blank" rel="noopener">CodingHorror</a></p></blockquote><p>I was reading about Unit Testing and found a blog entry in CodingHorror called <em><a href="https://blog.codinghorror.com/i-pity-the-fool-who-doesnt-write-unit-tests/" target="_blank" rel="noopener">I Pity The Fool Who Doesn’t Write Unit Tests</a></em>, and guess what, <strong>he’s right</strong>.</p><p>Sadly, I’ve encountered a lot of people who doesn’t write tests for their code, and have a <a href="https://www.agilealliance.org/glossary/continuous-integration" target="_blank" rel="noopener">CI System</a> (Travis, Jenkins, Gitlab, etc.) only to test the syntax of the code with flake8 or pycodestyle, or the result of calling the software without returning an exception… this is <strong><em>wrong</em></strong>, you’re not testing anything there.</p><p>You should be able to test your code to ensure that you’re getting the expected output and then have the confidence to refactor or write new functions without the fear of breaking anything.</p><p>There is <em>value</em> in testing, and <strong>I’m not talking about religiously test <em>first</em> code <em>later</em></strong>, for me that’s a personal matter, all that I’m saying is that <strong>your code won’t have the quality it needs without testing.</strong> So stop neglecting Unit Testing and I invite you to read the CodingHorror blog post.</p><p>Let’s jump right into it with a primer for python.</p><h2 id="Unit-Testing-a-Method"><a href="#Unit-Testing-a-Method" class="headerlink" title="Unit Testing a Method"></a>Unit Testing a Method</h2><p>Honestly, you should <strong><em>write a failing test</em></strong> first, then <strong><em>write just the necessary code</em></strong> to make it work, and lastly <strong><em>refactor</em></strong>, from now on just rinse and repeat. But as we’re still learning how to code I’ll write the code first and let’s test it later.</p><p>We’re going to write a module called <code>say_hi.pi</code> that has a function called <code>salute(name)</code> which takes a <code>name</code> as a single argument and returns <code>Hello, &lt;name&gt;!</code>. Here’s our code for <code>say_hi.py</code>:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">salute</span><span class="params">(name)</span>:</span></span><br><span class="line">    <span class="keyword">return</span> <span class="string">'Hello, &#123;&#125;!'</span>.format(name)</span><br></pre></td></tr></table></figure><p>Now let’s write a test called <code>test_say_hi.py</code> for this module to ensure that our code always return the desired string:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> unittest</span><br><span class="line"><span class="keyword">from</span> say_hi <span class="keyword">import</span> salute</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestSayHi</span><span class="params">(unittest.TestCase)</span>:</span></span><br><span class="line">    <span class="string">"""Class for testing say_hi.py"""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">test_salute</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="string">"""Test salute() function."""</span></span><br><span class="line">        self.assertEqual(salute(<span class="string">'Anne'</span>), <span class="string">'Hello, Anne!'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    unittest.main()</span><br></pre></td></tr></table></figure><p>Seems a bit long, huh? We’ll break it later, for now let’s see if our code works:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ python test_say_hi.py</span><br></pre></td></tr></table></figure><p>Aaand guess what?:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">.</span><br><span class="line">----------------------------------------------------------------------</span><br><span class="line">Ran 1 test in 0.000s</span><br><span class="line"></span><br><span class="line">OK</span><br></pre></td></tr></table></figure><p>It works correctly, so what is happening in that test? Let’s break it down.</p><h2 id="Inspecting-the-Test"><a href="#Inspecting-the-Test" class="headerlink" title="Inspecting the Test"></a>Inspecting the Test</h2><ol><li><p>First we import the <strong>unittest</strong> module that will let us write tests for our code:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> unittest</span><br></pre></td></tr></table></figure></li><li><p>Now we import the function <code>salute</code> from the python file <code>say_hi.py</code> that we just wrote.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> say_hi <span class="keyword">import</span> salute</span><br></pre></td></tr></table></figure></li><li><p>We create a new class that inherits from <code>unittest.TestCase</code>, we’re are making a new test case class that will hold our methods to test our <code>say_hi.py</code> functions.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestSayHi</span><span class="params">(unittest.TestCase)</span>:</span></span><br><span class="line">    <span class="string">"""Class for testing say_hi.py"""</span></span><br></pre></td></tr></table></figure></li><li><p>Now we define our first class method, this method will test our function <code>salute</code>, and this is the important part, what are we doing with <code>self.assertEqual</code>? We are making a comparison here, we are asking our test: <em>The function <code>salute(&#39;Anne&#39;)</code> <strong>should</strong> return exactly: <code>Hello, Anne!</code>, if it’s not the same, then complain!</em></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test_salute</span><span class="params">(self)</span>:</span></span><br><span class="line">    <span class="string">"""Test salute() function."""</span></span><br><span class="line">    self.assertEqual(salute(<span class="string">'Anne'</span>), <span class="string">'Hello, Anne!'</span>)</span><br></pre></td></tr></table></figure></li><li><p>Lastly this line here is in charge of executing or Unit Tests if it’s called directly as a standalone program.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    unittest.main()</span><br></pre></td></tr></table></figure></li></ol><h2 id="Testing-a-Class"><a href="#Testing-a-Class" class="headerlink" title="Testing a Class"></a>Testing a Class</h2><p>Now that we have the hang of it, let’s create our own class, it shouldn’t be hard at all.</p><p>Let’s create a <code>Raccoon</code> class with a name (because, <em>why not?</em>) and a <code>rabid</code> status of <code>False</code>… because we don’t want a rabid raccoon.</p><p>Tha <em>only</em> thing that is going to change is that we are making things the <em>“right way”</em> now. We’re making a <em>Test first</em>, then we <em>write code</em>, and finally we <em>refactor</em>, so let’s dive directly into the unit test.</p><p><code>test_raccoon.py</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> unittest</span><br><span class="line"><span class="keyword">from</span> raccoon <span class="keyword">import</span> Raccoon</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestRabidRaccoon</span><span class="params">(unittest.TestCase)</span>:</span></span><br><span class="line">    <span class="string">"""Test if raccoon class."""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">test_raccoon_health</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="string">"""Test if the raccoon is rabid or not."""</span></span><br><span class="line"></span><br><span class="line">        <span class="comment"># let's name our favorite raccoon 'Helga'</span></span><br><span class="line">        helga = Raccoon(<span class="string">'helga'</span>)</span><br><span class="line">        self.assertEqual(helga.rabid, <span class="literal">False</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    unittest.main()</span><br></pre></td></tr></table></figure><p>Now let’s execute our test and let’s see how our tests <em>guide us</em> to write our code:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ python test_raccoon.py</span><br><span class="line">Traceback (most recent call last):</span><br><span class="line">  File <span class="string">"test_raccoon.py"</span>, line 2, <span class="keyword">in</span> &lt;module&gt;</span><br><span class="line">    from raccoon import Raccoon</span><br><span class="line">ImportError: cannot import name <span class="string">'Raccoon'</span></span><br></pre></td></tr></table></figure><p>Our test complains that there’s no such module called <code>Raccoon</code>, of course there isn’t one because we haven’t created one, so let’s do it:</p><p><code>racoon.py</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Raccoon</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name)</span>:</span></span><br><span class="line">        self.name = name</span><br><span class="line">        self.rabid = <span class="literal">False</span></span><br></pre></td></tr></table></figure><p>Now, we execute our test again:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ python test_raccoon.py</span><br><span class="line">.</span><br><span class="line">----------------------------------------------------------------------</span><br><span class="line">Ran 1 <span class="built_in">test</span> <span class="keyword">in</span> 0.000s</span><br><span class="line"></span><br><span class="line">OK</span><br></pre></td></tr></table></figure><p>And we find out that it’s all alright, now it’s time to refactor, as we haven’t written almost anything, we should add at least documentation to our class without the fear of breaking anything, <strong>unit tests got our backs.</strong></p><p><code>racoon.py</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="string">"""Module holding Raccoon class."""</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Raccoon</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="string">"""Class simulating a rabid Raccoon."""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name)</span>:</span></span><br><span class="line">        <span class="string">"""Initialize attributes."""</span></span><br><span class="line">        self.name = name</span><br><span class="line">        self.rabid = <span class="literal">False</span></span><br></pre></td></tr></table></figure><p>We added simple docstrings to our module, let’s see if we didn’t break anything:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ python test_raccoon.py</span><br><span class="line">.</span><br><span class="line">----------------------------------------------------------------------</span><br><span class="line">Ran 1 <span class="built_in">test</span> <span class="keyword">in</span> 0.000s</span><br><span class="line"></span><br><span class="line">OK</span><br></pre></td></tr></table></figure><h2 id="Refactoring-our-Unit-Test"><a href="#Refactoring-our-Unit-Test" class="headerlink" title="Refactoring our Unit Test"></a>Refactoring our Unit Test</h2><p>Our test could’ve been better and we need to refactor it, because there are minor details that we should implement, let’s introduce the <strong>setUp</strong> method.</p><p>First of all, if our test is going to use the name <code>Helga</code> multiple times, then we should define a <strong>setUp</strong> method so we can define our name only one time to avoid repetition, we do this <em>inside</em> our <em>Test Case</em> class <code>TestRabidRaccoon</code>. Also we’re changing <code>assertEqual</code> for a more appropriate a shorter way to check for <code>False</code> which is <code>assertFalse</code>:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> unittest</span><br><span class="line"><span class="keyword">from</span> raccoon <span class="keyword">import</span> Raccoon</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestRabidRaccoon</span><span class="params">(unittest.TestCase)</span>:</span></span><br><span class="line">    <span class="string">"""Test if raccoon class."""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">setUp</span><span class="params">(self)</span>:</span></span><br><span class="line">      <span class="string">"""Setup variables that we're going to use across our class"""</span></span><br><span class="line">        self.name = <span class="string">'Helga'</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">test_raccoon_health</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="string">"""Test if the raccoon is rabid or not."""</span></span><br><span class="line">        <span class="comment"># let's pass the previously defined name in setUp</span></span><br><span class="line">        helga = Raccoon(self.name)</span><br><span class="line"></span><br><span class="line">        <span class="comment"># change assertEqual for a more proper method to check False</span></span><br><span class="line">        self.assertFalse(helga.rabid)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    unittest.main()</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">.</span><br><span class="line">----------------------------------------------------------------------</span><br><span class="line">Ran 1 test in 0.000s</span><br><span class="line"></span><br><span class="line">OK</span><br><span class="line">[Finished in 0.167s]</span><br></pre></td></tr></table></figure><p>It runs as expected. There are other assertion methods that you can use to test your code, here’s a short list:</p><table><thead><tr><th align="left">Method</th><th align="left">Description</th></tr></thead><tbody><tr><td align="left"><strong>assertEqual(A, B)</strong></td><td align="left">Check if A is equal to B</td></tr><tr><td align="left"><strong>assertNotEqual(A, B)</strong></td><td align="left">Check if A is <em>not</em> equal to B</td></tr><tr><td align="left"><strong>assertTrue(A)</strong></td><td align="left">Check if A is returns <strong>True</strong></td></tr><tr><td align="left"><strong>assertFalse(A)</strong></td><td align="left">Check if A returns <strong>False</strong></td></tr><tr><td align="left"><strong>assertIn(<em>item</em>, <em>list</em>)</strong></td><td align="left">Check if <em>item</em> is in <em>list</em></td></tr><tr><td align="left"><strong>assertNotIn(<em>item</em>, <em>list</em>)</strong></td><td align="left">Check if <em>item</em> is <em>not</em> in <em>list</em></td></tr></tbody></table><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>This is enough to get you started, there’s a <em>lot</em> of value in testing, and you should do it, you’ll become a better developer and you’ll also adopt good practices, if you’re barely starting coding then you will be able to <strong>use your test as a guidance</strong> because you will be <em>forced</em> to think <em>what</em> you want your code to do <em>first</em> before actually code it.</p><p>If you are an experienced developer  then <strong>you will catch bugs easier</strong>, you’ll have <strong>confidence when you’re refactoring</strong> your code, <strong>things are less likely to break</strong>, and you will be able to <strong>implement new features exactly as you want them</strong>.</p><p>So, code first and test later, or test first and code later, that’s <strong>OK</strong> for me, <em>as long as you are testing it.</em></p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Suppress Print Output in Python</title>
      <link href="2018/03/22/supress-print-output-in-python/"/>
      <url>2018/03/22/supress-print-output-in-python/</url>
      
        <content type="html"><![CDATA[<h2 id="Why"><a href="#Why" class="headerlink" title="Why?"></a>Why?</h2><p>Sometimes you don’t want output <em>at all</em> to your screen, there are times when I’m writing a function that prints something to the screen, the ideal thing (at least for me) is to use <code>return</code> instead of <code>print()</code> of course, but let’s say that you don’t have another option and when it comes to Unit Testing it becomes annoying. How can we suppress <a href="https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)" target="_blank" rel="noopener">STDOUT</a>?</p><h2 id="Redirecting-STDOUT"><a href="#Redirecting-STDOUT" class="headerlink" title="Redirecting STDOUT"></a>Redirecting STDOUT</h2><p>We’re writing a simple script that greets someone:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">salute</span><span class="params">(name)</span>:</span></span><br><span class="line">    <span class="string">"""Says hi to someone."""</span></span><br><span class="line">    print(<span class="string">'Hi, &#123;&#125;!'</span>.format(name))</span><br></pre></td></tr></table></figure><p>Let’s run this code shall we? We’ll import it and use the function <code>salute(name)</code>:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> say_hi <span class="keyword">import</span> salute</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>salute(<span class="string">'Anne'</span>)</span><br><span class="line">Hi, Anne!</span><br></pre></td></tr></table></figure><p>It works great. Now, to suppress the output of <code>salute(name)</code> we’re going to redirect <code>sys.stdout</code> into a <a href="https://docs.python.org/3/library/io.html" target="_blank" rel="noopener">StringIO</a> to capture any text output:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># remember to import io and sys</span></span><br><span class="line"><span class="keyword">import</span> io</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">salute</span><span class="params">(name)</span>:</span></span><br><span class="line">    <span class="string">"""Says hi to someone."""</span></span><br><span class="line">    print(<span class="string">'Hi, &#123;&#125;!'</span>.format(name))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># create a text trap and redirect stdout</span></span><br><span class="line">text_trap = io.StringIO()</span><br><span class="line">sys.stdout = text_trap</span><br><span class="line"></span><br><span class="line"><span class="comment"># execute our now mute function</span></span><br><span class="line">salute(<span class="string">'Anne'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># now restore stdout function</span></span><br><span class="line">sys.stdout = sys.__stdout__</span><br></pre></td></tr></table></figure><p>If we execute this python code, no output will be displayed. This is useful when we are Unit Testing and we want to suppress <code>print()</code> output. If we want to check the captured test we can do this with <code>.getvalue()</code>:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">-- SNIP --</span><br><span class="line"></span><br><span class="line"><span class="comment"># getting trapped print</span></span><br><span class="line">print(<span class="string">'Captured text:'</span>)</span><br><span class="line">print(text_trap.getvalue())</span><br></pre></td></tr></table></figure><p>Result:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">Captured text:</span><br><span class="line">Hi, Anne!</span><br></pre></td></tr></table></figure><h2 id="Unit-Testing-print"><a href="#Unit-Testing-print" class="headerlink" title="Unit Testing print()"></a>Unit Testing print()</h2><p>If we have a module that prints something to the screen instead of returning a value, we can test that <code>print()</code> string with the method above. First, let’s remove everything from our previous code, leaving only our <code>salute</code> method:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">salute</span><span class="params">(name)</span>:</span></span><br><span class="line">    <span class="string">"""Says hi to someone."""</span></span><br><span class="line">    print(<span class="string">'Hi, &#123;&#125;!'</span>.format(name))</span><br></pre></td></tr></table></figure><p>Using the code above, what we want to do is to test if the method <code>salute(name)</code> always prints a greeting in the following format: <code>Hi, __name__!</code> (keep in mind that the function <code>print()</code> always insert a new line, or <code>\n</code>, at the end)</p><p>Let’s setup our Unit Test:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> unittest</span><br><span class="line"><span class="keyword">from</span> say_hi <span class="keyword">import</span> salute</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestSayHi</span><span class="params">(unittest.TestCase)</span>:</span></span><br><span class="line">    <span class="string">"""Tests for say_hi.py"""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">setUp</span><span class="params">(self)</span>:</span></span><br><span class="line">        self.name = <span class="string">'Anne'</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">test_salute</span><span class="params">(self)</span>:</span></span><br><span class="line">        self.assertEqual(salute(self.name), <span class="string">'Hi, Anne!\n'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    unittest.main()</span><br></pre></td></tr></table></figure><p>When we execute our code we obtain the following error:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ python test_say_hi.py</span><br><span class="line">Hi, Anne!</span><br><span class="line">F</span><br><span class="line">======================================================================</span><br><span class="line">FAIL: test_salute (__main__.TestSayHi)</span><br><span class="line">----------------------------------------------------------------------</span><br><span class="line">Traceback (most recent call last):</span><br><span class="line">  File <span class="string">"test_say_hi.py"</span>, line 12, <span class="keyword">in</span> test_salute</span><br><span class="line">    self.assertEqual(salute(self.name), <span class="string">'Hi, Anne!\n'</span>)</span><br><span class="line">AssertionError: None != <span class="string">'Hi, Anne!\n'</span></span><br><span class="line"></span><br><span class="line">----------------------------------------------------------------------</span><br><span class="line">Ran 1 <span class="built_in">test</span> <span class="keyword">in</span> 0.000s</span><br></pre></td></tr></table></figure><p>Why our assertion didn’t work here? <code>salute(&#39;Anne&#39;)</code> returns <code>&#39;Hi, Anne!\n&#39;</code>, right? It doesn’t. Print doesn’t return anything actually, it just <em>prints</em> something to the screen (<a href="https://docs.python.org/3.5/library/functions.html#print" target="_blank" rel="noopener">to put it mildly</a>), but it doesn’t <em>returns</em> a string.</p><p>But with our previous technique we can capture the string and store it in a value so we can compare our print string:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> unittest</span><br><span class="line"><span class="keyword">import</span> io</span><br><span class="line"><span class="keyword">from</span> say_hi <span class="keyword">import</span> salute</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestSayHi</span><span class="params">(unittest.TestCase)</span>:</span></span><br><span class="line">    <span class="string">"""Tests for say_hi.py"""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">setUp</span><span class="params">(self)</span>:</span></span><br><span class="line">        self.name = <span class="string">'Anne'</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">test_salute</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="string">"""Test print in salute()."""</span></span><br><span class="line">        <span class="comment"># create a trap</span></span><br><span class="line">        text_trap = io.StringIO()</span><br><span class="line">        sys.stdout = text_trap</span><br><span class="line"></span><br><span class="line">        salute(self.name)</span><br><span class="line"></span><br><span class="line">        <span class="comment"># restore stdout</span></span><br><span class="line">        sys.stdout = sys.__stdout__</span><br><span class="line">        self.assertEqual(text_trap.getvalue(), <span class="string">'Hi, Anne!\n'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    unittest.main()</span><br></pre></td></tr></table></figure><p>Inspecting <code>test_salute()</code> we redirect our print to our <code>text_trap</code>, and after restoring <code>stdout</code> to its original functionality we can compare the value of <code>text_trap.getvalue()</code> with our expected output: <code>&#39;Hi, Anne!\n&#39;</code>:</p><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">.</span><br><span class="line">----------------------------------------------------------------------</span><br><span class="line">Ran 1 <span class="built_in">test</span> <span class="keyword">in</span> 0.000s</span><br><span class="line"></span><br><span class="line">OK</span><br></pre></td></tr></table></figure><p>And it works correctly, we can now compare our <code>print</code> value with an unit test, plus it doesn’t print anything in our tests.</p><h1 id="UPDATE"><a href="#UPDATE" class="headerlink" title="UPDATE"></a>UPDATE</h1><p>There’s a much nicer and safer way to do this instead of rewriting <code>stdout</code>, we can use a <em>Context Manager</em> temporarily redirect <code>sys.stdout</code> without touching it.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> io</span><br><span class="line"><span class="keyword">from</span> contextlib <span class="keyword">import</span> redirect_stdout</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">salute</span><span class="params">(name)</span>:</span></span><br><span class="line">    <span class="string">"""Says hi to someone."""</span></span><br><span class="line">    print(<span class="string">'Hi, &#123;&#125;!'</span>.format(name))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># set a trap and redirect stdout</span></span><br><span class="line">trap = io.StringIO()</span><br><span class="line"><span class="keyword">with</span> redirect_stdout(trap):</span><br><span class="line">    salute(<span class="string">'Anne'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># getting redirected output</span></span><br><span class="line">captured_stdout = trap.getvalue()</span><br><span class="line">print(captured_stdout)</span><br></pre></td></tr></table></figure><p>And if we’re going to test <code>salute(name)</code>:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> io</span><br><span class="line"><span class="keyword">import</span> unittest</span><br><span class="line"><span class="keyword">from</span> say_hi <span class="keyword">import</span> salute</span><br><span class="line"><span class="keyword">from</span> contextlib <span class="keyword">import</span> redirect_stdout</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestSayHi</span><span class="params">(unittest.TestCase)</span>:</span></span><br><span class="line">    <span class="string">"""Tests for say_hi.py"""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">setUp</span><span class="params">(self)</span>:</span></span><br><span class="line">        self.name = <span class="string">'Anne'</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">test_salute</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="string">"""Test print in salute()."""</span></span><br><span class="line">        <span class="comment"># create a trap</span></span><br><span class="line">        text_trap = io.StringIO()</span><br><span class="line">        <span class="keyword">with</span> redirect_stdout(text_trap):</span><br><span class="line">            salute(self.name)</span><br><span class="line">        self.assertEqual(text_trap.getvalue(), <span class="string">'Hi, Anne!\n'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    unittest.main()</span><br></pre></td></tr></table></figure><p>This is a much safer and cleaner approach, have fun!</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Classes and Special Methods in Python</title>
      <link href="2018/03/18/class-inheritance-with-python/"/>
      <url>2018/03/18/class-inheritance-with-python/</url>
      
        <content type="html"><![CDATA[<!-- Start of Unsplash Embed Code - Full-width (Embed code by @BirdyOz)--><div style="width:100%; margin: 20px auto;">    <img src="https://images.unsplash.com/photo-1517842645767-c639042777db?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-lge" alt="brown fountain pen on notebook " title="brown fountain pen on notebook ">    <div class="text-muted" style="opacity: 0.5">        <small><a href="https://unsplash.com/photos/5bYxXawHOQg" target="_blank">Photo</a> by <a href="https://unsplash.com/@dtravisphd" target="_blank">@dtravisphd</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 01/11/2020</small>    </div></div><!-- End of Unsplash Embed code --><p>These are my study notes on Classes and special methods. As always, if something is wrong then you can always correct me, it would help me and everybody else.</p><h2 id="Class-‘Employee’-example"><a href="#Class-‘Employee’-example" class="headerlink" title="Class ‘Employee’ example"></a>Class ‘Employee’ example</h2><p>This is the example class that we’re going to use, a class <code>Employee</code> which we will inherit from to create more classes like <code>Manager</code> and <code>Supervisor</code>:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Employee</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="string">"""Class simulating an Employee with basic attributes."""</span></span><br><span class="line"></span><br><span class="line">    total_employees = <span class="number">0</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name, rate, position)</span>:</span></span><br><span class="line">        self.name = name</span><br><span class="line">        self.rate = rate</span><br><span class="line">        self.owed = <span class="number">0</span></span><br><span class="line">        self.position = position</span><br><span class="line">        Employee.total_employees += <span class="number">1</span></span><br></pre></td></tr></table></figure><p>Let’s break this up a little bit, we’re creating the class <code>Employee</code> that holds a variable <code>total_employee</code> and every time we <em>initialize</em> our class we will add a <code>+1</code> to our total employee number. This initialization of this class will require a <code>name</code>, <code>rate</code> and <code>position</code>. Remember that the keyword <em>self</em> refers to the class <em>itself</em>, these variables are shared in the whole class. So far, so good.</p><p>Save this code as <em>class_example.py</em> and import it in python:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> class_example <span class="keyword">import</span> Employee</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>employee1 = Employee(<span class="string">'Anne'</span>, <span class="number">20</span>, <span class="string">'manager'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>employee1.name</span><br><span class="line"><span class="string">'Anne'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>employee1.rate</span><br><span class="line"><span class="number">20</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>employee1.position</span><br><span class="line"><span class="string">'manager'</span></span><br><span class="line">&gt;&gt;&gt;</span><br></pre></td></tr></table></figure><p>Now we can instantiate our class in <code>employee1</code> and inspect the employee’s name, rate and position.</p><h2 id="Initializing-a-class-with-init"><a href="#Initializing-a-class-with-init" class="headerlink" title="Initializing a class with __init__"></a>Initializing a class with <code>__init__</code></h2><p>This method let us initialize an <em>instance</em> of our class <strong>Employee</strong> with <code>x = Class(Args)</code>. This way, every time we want time instantiate an object then we have to pass parameters that will act as attributes to out class object. you can see en the last line that every time we initialize our class it will bump the number of <code>total_employees</code>, let’s check this out:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> class_example <span class="keyword">import</span> Employee</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>emp1 = Employee(<span class="string">'Anne'</span>, <span class="number">20</span>, <span class="string">'manager'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>Employee.total_employees</span><br><span class="line"><span class="number">1</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>emp2 = Employee(<span class="string">'John'</span>, <span class="number">25</span>, <span class="string">'supervisor'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>Employee.total_employees</span><br><span class="line"><span class="number">2</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>emp3 = Employee(<span class="string">'Fran'</span>, <span class="number">30</span>, <span class="string">'developer'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>Employee.total_employees</span><br><span class="line"><span class="number">3</span></span><br></pre></td></tr></table></figure><p>See? Every time a new employee is created, the number of employees goes up.</p><h2 id="Changing-the-String-representation-with-repr"><a href="#Changing-the-String-representation-with-repr" class="headerlink" title="Changing the String representation with __repr__"></a>Changing the String representation with <code>__repr__</code></h2><p>If we check the string representation of our class then we will have something like this:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>emp1</span><br><span class="line">&lt;class_example.Employee object at <span class="number">0x7f3809baec50</span>&gt;</span><br></pre></td></tr></table></figure><p>To give a more accurate description of our object, we can change the default representation to a string that describe the initialized class more appropriately re-defining the <code>__repr__</code> method:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Employee</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="string">"""Class simulating an Employee with basic attributes."""</span></span><br><span class="line"></span><br><span class="line">    total_employees = <span class="number">0</span></span><br><span class="line">    -- SNIP --</span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__repr__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> <span class="string">'Employee object with basic attributes'</span></span><br></pre></td></tr></table></figure><p>Now we can check again to see how our string representation gets displayed:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>emp1</span><br><span class="line">Employee object <span class="keyword">with</span> basic attributes</span><br><span class="line">&gt;&gt;&gt;</span><br></pre></td></tr></table></figure><h2 id="Adding-more-functionality-to-our-class"><a href="#Adding-more-functionality-to-our-class" class="headerlink" title="Adding more functionality to our class"></a>Adding more functionality to our class</h2><p>Let’s add hours worked to our employee and how much do we owe to our employee:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Employee</span><span class="params">(object)</span>:</span></span><br><span class="line">    -- SNIP --</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">add_hours_worked</span><span class="params">(self, total_hours)</span>:</span></span><br><span class="line">        self.owed = total_hours * self.rate</span><br><span class="line">        print(<span class="string">'&#123;&#125; work hours where added to &#123;&#125;'</span>.format(total_hours, self.name))</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">pay</span><span class="params">(self)</span>:</span></span><br><span class="line">        print(<span class="string">'Employee &#123;&#125; was paid with &#123;&#125; USD'</span>.format(self.name, self.owed))</span><br><span class="line">        self.owed = <span class="number">0</span></span><br></pre></td></tr></table></figure><p>with <code>Employee.add_hours_worked(total_hours)</code> method we calculate how much we have to pay to our employee based on the <code>total_hours</code> he has worked. And with the method <code>pay()</code> we let our user know that we have payed the owed amount to our employee and now we owe nothing to him until he has completed more working hours.</p><h2 id="Inheritance"><a href="#Inheritance" class="headerlink" title="Inheritance"></a>Inheritance</h2><p>Now that we have defined a very generic Employee, we’re going to define other classes such as <code>Developer</code> and <code>Manager</code> that inherits the same properties and methods of the class <code>Employee</code> but change a few ones like the string representation and rate, also let’s add a bonus to his payment.</p><p>We’re rewriting <code>add_working_hours()</code> to add a bonus system, if this developer has worked more than 150 hours in a month then a bonus will be added, but we’re not rewriting the whole method,</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Developer</span><span class="params">(Employee)</span>:</span></span><br><span class="line">    <span class="string">"""Employee in charge of development and bug-crushing activities."""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name, rate, position)</span>:</span></span><br><span class="line">        super().__init__(name, rate, position)</span><br><span class="line">        self.bonus = <span class="number">300</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__repr__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> self.__doc__</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">add_working_hours</span><span class="params">(self, total_hours)</span>:</span></span><br><span class="line">        self.owed = total_hours * self.rate</span><br><span class="line">        <span class="keyword">if</span> total_hours &gt; <span class="number">150</span>:</span><br><span class="line">            self.owed += self.bonus</span><br><span class="line">        print(<span class="string">'&#123;&#125; work hours were added to &#123;&#125;'</span>.format(total_hours, self.name))</span><br></pre></td></tr></table></figure><p>This method let us inherits all variables, initializing values and even methods, but what have we changed here exactly? Well, we added a <strong>bonus</strong> of 300 USD as initialization attribute, also we changed the string representation to the special method <code>__doc__</code>, this means that whenever we inspect the our object, it will return the <em>docstring</em> as our representation:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp = Developer(<span class="string">'Anne'</span>, <span class="number">20</span>, <span class="string">'developer'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp</span><br><span class="line">Employee <span class="keyword">in</span> charge of development <span class="keyword">and</span> bug-crushing activities.</span><br></pre></td></tr></table></figure><p>We also modified <code>add_hours_worked()</code> and added a simple control flow</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">...</span><br><span class="line">        <span class="keyword">if</span> total_hours &gt; <span class="number">150</span>:</span><br><span class="line">            self.owed += self.bonus</span><br><span class="line">...</span><br></pre></td></tr></table></figure><p>As you can see here, if our developer has reached more than 150 hours, he can have a bonification of 300USD to his payment, we can see that in action:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp = Developer(<span class="string">'Anne'</span>, <span class="number">20</span>, <span class="string">'developer'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp2 = Developer(<span class="string">'Johnny'</span>, <span class="number">20</span>, <span class="string">'developer'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.add_hours_worked(<span class="number">160</span>)</span><br><span class="line"><span class="number">160</span> work hours where added to Anne</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp2.add_hours_worked(<span class="number">130</span>)</span><br><span class="line"><span class="number">130</span> work hours where added to Johnny</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.pay()</span><br><span class="line">Employee Anne was paid <span class="keyword">with</span> <span class="number">3500</span> USD</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp2.pay()</span><br><span class="line">Employee Johnny was paid <span class="keyword">with</span> <span class="number">2600</span> USD</span><br></pre></td></tr></table></figure><p>The developer ‘Anne’ and ‘Johnny’ earn 20 USD per hour of work, as ‘Anne’ reaches 160 hours of work in her month then she can have a bonification (<em>160 * 20 + 300 = <strong>3500</strong></em>), ‘Johnny’ instead, worked only 130 hours this month, he cannot have his bonification (<em>130 * 20 = <strong>2600</strong></em>).</p><p>But have you realized that we payed our <em>Developers</em> and we didn’t set a <code>pay()</code> method to our <code>Developer</code> class? This is because we inherits the methods found in <code>Employee</code> class so we don’t need to rewrite them.</p><h2 id="Changing-our-values-directly"><a href="#Changing-our-values-directly" class="headerlink" title="Changing our values directly"></a>Changing our values directly</h2><p>We can change our class values directly, for example, if we want to change our employee’s names we can do that easily with:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp = Developer(<span class="string">'Anne'</span>, <span class="number">20</span>, <span class="string">'developer'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.name = <span class="string">'Not Anne'</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.name</span><br><span class="line"><span class="string">'Not Anne'</span></span><br></pre></td></tr></table></figure><h2 id="Other-Special-Methods"><a href="#Other-Special-Methods" class="headerlink" title="Other Special Methods"></a>Other Special Methods</h2><p><code>Class.__getattrib__(attribute)</code> to return the attribute of a class, beware though, this is the same as <code>Class.attribute</code> which is shorter:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.__getattribute__(<span class="string">'rate'</span>)</span><br><span class="line"><span class="number">20</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.rate</span><br><span class="line"><span class="number">20</span></span><br></pre></td></tr></table></figure><p><code>Class.__dict__</code> to return a dictionary of the Class’ attributes:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.__dict__</span><br><span class="line">&#123;<span class="string">'name'</span>: <span class="string">'not anne'</span>, <span class="string">'rate'</span>: <span class="number">20</span>, <span class="string">'owed'</span>: <span class="number">0</span>, <span class="string">'position'</span>: <span class="string">'developer'</span>, <span class="string">'bonus'</span>: <span class="number">300</span>&#125;</span><br></pre></td></tr></table></figure><p><code>Class.__module__</code> to return the module that holds that Class:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.__module__</span><br><span class="line"><span class="string">'class_example'</span></span><br></pre></td></tr></table></figure><h2 id="Set-attributes-to-a-Class"><a href="#Set-attributes-to-a-Class" class="headerlink" title="Set attributes to a Class"></a>Set attributes to a Class</h2><p>Setting an attribute to our class is easy: <code>Class.new_attribute = value</code>.<br>Let’s set the attribute <code>active</code> so we can know if our employee <code>Developer</code> is currently active:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> class_example <span class="keyword">import</span> *</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp = Developer(<span class="string">'Anne'</span>, <span class="number">20</span>, <span class="string">'developer'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.active = <span class="literal">True</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>dev_emp.active</span><br><span class="line"><span class="literal">True</span></span><br></pre></td></tr></table></figure><p>These are my study notes for today, any questions feel free to ask.<br>Here’s the full code used in the examples:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Employee</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="string">"""Class simulating an Employee with basic attributes."""</span></span><br><span class="line">    total_employees = <span class="number">0</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name, rate, position)</span>:</span></span><br><span class="line"></span><br><span class="line">        self.name = name</span><br><span class="line">        self.rate = rate</span><br><span class="line">        self.owed = <span class="number">0</span></span><br><span class="line">        self.position = position</span><br><span class="line">        Employee.total_employees += <span class="number">1</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">add_working_hours</span><span class="params">(self, total_hours)</span>:</span></span><br><span class="line">        self.owed = total_hours * self.rate</span><br><span class="line">        print(<span class="string">'&#123;&#125; work hours where added to &#123;&#125;'</span>.format(total_hours, self.name))</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">pay</span><span class="params">(self)</span>:</span></span><br><span class="line">        print(<span class="string">'Employee &#123;&#125; was paid with &#123;&#125;USD'</span>.format(self.name, self.owed))</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__repr__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> self.__doc__</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Developer</span><span class="params">(Employee)</span>:</span></span><br><span class="line">    <span class="string">"""Employee in charge of development and bug-crushing activities."""</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name, rate, position)</span>:</span></span><br><span class="line">        super().__init__(name, rate, position)</span><br><span class="line">        self.bonus = <span class="number">300</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__repr__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> self.__doc__</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">add_working_hours</span><span class="params">(self, total_hours)</span>:</span></span><br><span class="line">        self.owed = total_hours * self.rate</span><br><span class="line">        <span class="keyword">if</span> total_hours &gt; <span class="number">150</span>:</span><br><span class="line">            self.owed += self.bonus</span><br><span class="line">        print(<span class="string">'&#123;&#125; work hours were added to &#123;&#125;'</span>.format(total_hours, self.name))</span><br></pre></td></tr></table></figure>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> classes </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Figure Out a Download File-Size With Requests</title>
      <link href="2018/03/09/get-a-download-size-with-Requests/"/>
      <url>2018/03/09/get-a-download-size-with-Requests/</url>
      
        <content type="html"><![CDATA[<p>To get the filesize of a download is really easy, servers usually provide a <code>Content-Length</code> in its header response that let us know how heavy is the content we are requesting. We can find out this content length opening our shell and requesting a <code>HEAD</code> response in linux:</p><p><a href="/assets/images/misc/head_request.png"><img src="/assets/images/misc/head_request.png" alt="HEAD response"></a></p><p>As you can see, our content length is display in <em>bytes</em>. Let’s try to get this response with <code>Requests</code></p><h2 id="Display-Content-Length-with-Requests"><a href="#Display-Content-Length-with-Requests" class="headerlink" title="Display Content-Length with Requests"></a>Display Content-Length with Requests</h2><p>Let’s use an image from <a href="https://httpbin.org/image/png" target="_blank" rel="noopener">httpbin</a>. Remember to make a <code>HEAD</code> request instead of a <code>GET</code> request, this way we don’t have to download the entire file</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>req = requests.head(<span class="string">'https://httpbin.org/image/png'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>req.headers[<span class="string">'Content-Length'</span>]</span><br><span class="line"><span class="string">'8090'</span></span><br></pre></td></tr></table></figure><p>When we filter the header with the <code>Content-Length</code> key we can see how heavy our request will be without having to actually download the file.</p><h2 id="Calculating-the-file-size-without-Content-Length"><a href="#Calculating-the-file-size-without-Content-Length" class="headerlink" title="Calculating the file-size without Content-Length"></a>Calculating the file-size without Content-Length</h2><p>This is a tricky one because, well, not always every web server is going to provide you with a <code>Content-Length</code> in its header, this happens sometimes with csv, xml, and other text file-types, luckily they’re not that heavy and we can provide a <code>GET</code> request to know the download size</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>req = requests.get(<span class="string">'http://data.example.com/attachment_id=12'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>len(req.content)</span><br><span class="line"><span class="number">659</span></span><br></pre></td></tr></table></figure><p>As we can see here, we can calculate a file size (mostly text files) using the <em>len()</em> method. If you have a better alternative you can show it off in the comments bellow.</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Scraping Web Data With Requests and Beautifulsoup [Part 2]</title>
      <link href="2018/03/07/scraping-web-data-with-requests-and-beautifulsoup-part-2/"/>
      <url>2018/03/07/scraping-web-data-with-requests-and-beautifulsoup-part-2/</url>
      
        <content type="html"><![CDATA[<p>A quick recap of the last post:</p><ul><li>First we requested content using <code>requests</code> library and parsed the response’s contents using BeautifulSoup library.</li><li>We learned that we can manipulate HTML tags, get its contents and attributes (like <code>href</code> from an <code>a</code> tag).</li><li>We found that the <em>Class</em> <code>post-list</code> is holding an unordered list containing the website’s post titles and links so we proceeded to get the <code>a</code> tag’s content on each list item as well as their <code>href</code> attribute.</li><li>We also store that data into a dictionary in a <code>{post_title: post_link}</code> format to be able to iterate over them and reuse them when needed.</li></ul><p>Now we only have one problem left, and it is <strong><em>pagination</em></strong>. Now as you can see, this is our current source file:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"></span><br><span class="line"><span class="comment"># requesting index page</span></span><br><span class="line">base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br><span class="line">url_contents = BeautifulSoup(base_url.text, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># filter post-list tag</span></span><br><span class="line">post_list = url_contents.find(class_=<span class="string">'post-list'</span>)</span><br><span class="line">post_items = post_list.find_all(<span class="string">'a'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># hold attributes in a dictionary</span></span><br><span class="line">post_dict = &#123;&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> post <span class="keyword">in</span> post_items:</span><br><span class="line">    post_title = post.string</span><br><span class="line">    post_link = base_url.url + post.get(<span class="string">'href'</span>)</span><br><span class="line">    post_dict[post_title] = post_link</span><br><span class="line"></span><br><span class="line"><span class="comment"># iterate our dictionary</span></span><br><span class="line"><span class="keyword">for</span> title, link <span class="keyword">in</span> post_dict.items():</span><br><span class="line">    print(<span class="string">'&#123;&#125;: &#123;&#125;'</span>.format(title, link))</span><br></pre></td></tr></table></figure><p>When we execute our file it gives every post title and link from the main page:</p><figure class="highlight shell"><table><tr><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> pipenv run python extract_links.py</span></span><br><span class="line">Scraping Web Data With Requests and Beautifulsoup [Part 1]: https://codingdose.info//2018/03/05/scraping-web-data-with-requests-and-beautifulsoup/</span><br><span class="line">How to Send a Tweet With Python Using Tweepy: https://codingdose.info//2018/03/01/</span><br><span class="line">-- SNIP --</span><br><span class="line">Migrate From Ghost Blog to Jekyll: https://codingdose.info//2018/02/19/migrate-from-ghost-blog-to-jekyll/</span><br></pre></td></tr></table></figure><p>The thing is, the first post from this web page is <strong><em><a href="/2018/02/18/Hello-all/">Hello All!</a></em></strong> and not <strong><em><a href="/2018/02/19/migrate-from-ghost-blog-to-jekyll/">Migrate From blabla to blabla</a></em></strong>, and chances are that (hopefully) this blog is updated regularly and you’re not even getting the same results! So how do we get to page 2, 3, 4, and so on?</p><h2 id="Looping-over-pages"><a href="#Looping-over-pages" class="headerlink" title="Looping over pages"></a>Looping over pages</h2><p>Most blogs have <code>/page/1</code> or <code>/index.php?page=1</code> or any kind of pagination mechanism to show you more content in an ordered manner. If we wanted to iterate over pages we would first <em>request</em> <strong>/page/1</strong>, and then <strong>/page/2</strong> and so on.</p><p>But there’s a catch: <em>This site doesn’t have page 1.</em> It has <code>/page/2</code> in its URL (<code>https://codingdose.info/page/2/</code>) but page 1 it’s the <em>Index</em> page (<code>https://codingdose.info/</code>). But let’s not get ahead ourselves now, we’re going to get through this bit by bit.</p><h2 id="Finding-our-the-number-of-pages"><a href="#Finding-our-the-number-of-pages" class="headerlink" title="Finding our the number of pages"></a>Finding our the number of pages</h2><p>If we want to iterate over a number of pages, we first have to know how many pages there are in this little blog. If we go to the main page and scroll down we can see the number of pages in this blog: <em>Page 1 of X</em></p><p>If we inspect the string with the developer tools in our browser (Remember: <code>CTRL+SHIFT+C</code>) we find out that the string <em>Page 1 of X</em> is under the class <strong>page-number</strong></p><p><img src="/assets/images/bs4_scraping/pagination.png" alt="Inspection Tool"></p><p>Use the last number of this string to use it as the total pages this blog has and iterate over the total of those pages. At the time of writing, there are only 2 pages on this blog, so we are going to extract that number and convert it to a <em>Integer</em>:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"></span><br><span class="line"><span class="comment"># requesting index page</span></span><br><span class="line">base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br><span class="line">url_contents = BeautifulSoup(base_url.text, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># filter post-list tag</span></span><br><span class="line">post_list = url_contents.find(class_=<span class="string">'post-list'</span>)</span><br><span class="line">post_items = post_list.find_all(<span class="string">'a'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># hold attributes in a dictionary</span></span><br><span class="line">post_dict = &#123;&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment"># get the total number of pages</span></span><br><span class="line">class_pages = url_contents.find(class_=<span class="string">'page-number'</span>)</span><br><span class="line">pages_found = class_pages.get_text()</span><br><span class="line">total_pages = int(pages_found[<span class="number">-1</span>])</span><br><span class="line">print(<span class="string">'Total Pages Found: '</span> + str(total_pages))</span><br><span class="line">-- SNIP --</span><br></pre></td></tr></table></figure><p>What we are doing here is filtering the class <code>page-number</code> to narrow our search and extract the text that it holds with <code>.get_text()</code> to store <em>‘Page 1 of X’</em> in the variable <code>total_pages</code>, now we <em>slice</em> the last character of this string which is ‘2’ with <code>[-1]</code>. Now that we have the total number of pages we convert the string into an <em>integer</em> to be able to iterate over the pages.</p><p>But how we deal with the problem that I have no <code>/page/1</code>? Well, we will first extract the contents from the index page and <em>then</em> we will start iterating over the pages starting from page <strong>number 2</strong>. Remember that this is very unlikely to happen in most blogging systems but programming is all about problem solving, right?</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"></span><br><span class="line">base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br><span class="line">url_contents = BeautifulSoup(base_url.text, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># get the total number of pages</span></span><br><span class="line">class_pages = url_contents.find(class_=<span class="string">'page-number'</span>)</span><br><span class="line">pages_found = class_pages.get_text()</span><br><span class="line">total_pages = int(pages_found[<span class="number">-1</span>])</span><br><span class="line">print(<span class="string">'Total Pages: '</span> + str(total_pages))</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> page <span class="keyword">in</span> range(<span class="number">1</span>, total_pages + <span class="number">1</span>):</span><br><span class="line">    <span class="keyword">if</span> page == <span class="number">1</span>:</span><br><span class="line">        <span class="comment"># requesting index page (page 1)</span></span><br><span class="line">        base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br><span class="line">    <span class="keyword">elif</span> page &gt; <span class="number">1</span>:</span><br><span class="line">        <span class="comment"># after requesting index page, go directly to page 2/3/4...</span></span><br><span class="line">        base_url = requests.get(</span><br><span class="line">            <span class="string">'https://codingdose.info/page/&#123;&#125;/'</span>.format(page))</span><br><span class="line"></span><br><span class="line">    url_contents = BeautifulSoup(base_url.text, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># filter post-list tag</span></span><br><span class="line">    post_list = url_contents.find(class_=<span class="string">'post-list'</span>)</span><br><span class="line">    post_items = post_list.find_all(<span class="string">'a'</span>)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># hold attributes in a dictionary</span></span><br><span class="line">    post_dict = &#123;&#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> post <span class="keyword">in</span> post_items:</span><br><span class="line">        post_title = post.string</span><br><span class="line">        post_link = base_url.url + post.get(<span class="string">'href'</span>)</span><br><span class="line">        post_dict[post_title] = post_link</span><br><span class="line"></span><br><span class="line">    <span class="comment"># iterate our dictionary</span></span><br><span class="line">    <span class="keyword">for</span> title, link <span class="keyword">in</span> post_dict.items():</span><br><span class="line">        print(<span class="string">'&#123;&#125;: &#123;&#125;'</span>.format(title, link))</span><br></pre></td></tr></table></figure><p>Now, there are a number of changes that we have to take into consideration, first we enclosed most of our code in a <code>for</code> control flow using our <code>total_pages</code> Integer as a guide, if you look closely we can see a <code>+ 1</code> at the end of the <code>for</code> line, why is this? Remember that when you are looping with a range of numbers (let’s say from 0 to 9) in a <code>for</code> loop the control flow will stop at <strong>8</strong>, not <strong>9</strong>. This is because the <code>for</code> loop is <em>exclusive</em> not <em>inclusive</em> when looping over numbers, in this case, if we want the <code>for</code> loop to stop at <strong>2</strong> (inclusively, which is our current number of pages) we will have to add a number to our range.</p><p>If we are currently on <em>Page 1</em>, our base URL will be <em>‘<a href="https://codingdose.info&/#39;" target="_blank" rel="noopener">https://codingdose.info&#39;</a></em>, and if we are on <em>Page 2</em> then our base URL is going to be <em>‘<a href="https://codingdose.info/page/2/&#39;">https://codingdose.info/page/2/&#39;</a></em>, and so on.</p><p>Currently our code it’s starting to <a href="https://en.wikipedia.org/wiki/Code_smell" target="_blank" rel="noopener"><em>smell</em></a>, it needs to be refactored to prevent duplication and it could be a lot faster if we integrate concurrency and handle threads, but I will leave that to you, because this is <em>by far</em> not perfect… and it’s getting late here. I’ll end this post here and if you happen to clean the code and come up with a better idea then let me know in the comments bellow, thanks for reading.</p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Scraping Web Data With Requests and Beautifulsoup [Part 1]</title>
      <link href="2018/03/05/scraping-web-data-with-requests-and-beautifulsoup/"/>
      <url>2018/03/05/scraping-web-data-with-requests-and-beautifulsoup/</url>
      
        <content type="html"><![CDATA[<p>Scraping web data is essential if we want to spider web pages for whatever reasons we have, maybe storing posts information (in my case) or monitor a web page, crawl data, etc.</p><p>We are going to see how to handle html data with <a href="https://www.crummy.com/software/BeautifulSoup/" target="_blank" rel="noopener">BeautifulSoup</a> and <a href="http://docs.python-requests.org/en/master/" target="_blank" rel="noopener">Requests</a> using this site as an example.</p><h2 id="What-are-we-going-to-do"><a href="#What-are-we-going-to-do" class="headerlink" title="What are we going to do?"></a>What are we going to do?</h2><ul><li>Make a <code>GET</code> request with the <code>requests</code> library to obtain the web page contents</li><li>Parse those contents into HTML so we can analyze and manipulate data using the powerful <code>BeautifulSoup</code></li><li>Learn how to deal with pagination [in part 2]</li></ul><h2 id="BeautifulSoup-and-Requests-Installation"><a href="#BeautifulSoup-and-Requests-Installation" class="headerlink" title="BeautifulSoup and Requests Installation"></a>BeautifulSoup and Requests Installation</h2><p>As usual using <code>pipenv</code></p><figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">pipenv install bs4 requests</span><br></pre></td></tr></table></figure><h2 id="Requesting-Site-Data-with-GET-method"><a href="#Requesting-Site-Data-with-GET-method" class="headerlink" title="Requesting Site Data with GET method"></a>Requesting Site Data with GET method</h2><p>Using this site as an example, we first have to request the site data before we parse it in HTML</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line"><span class="comment"># requesting index page</span></span><br><span class="line">base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br></pre></td></tr></table></figure><p>This way we have stored our <code>GET</code> response into base_url, we can analyze the response code with <code>base_url.status_code</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>base_url.status_code</span><br><span class="line"><span class="number">200</span></span><br></pre></td></tr></table></figure><p>A <code>200</code> response code means that our request was successful.</p><p>You can find more information on <code>HTTP</code> statuses on <a href="https://httpstatuses.com/" target="_blank" rel="noopener">httpstatuses.com</a>, keep this site as a reference guide when working with web data such as http statuses and requests.</p><h2 id="Parsing-HTML"><a href="#Parsing-HTML" class="headerlink" title="Parsing HTML"></a>Parsing HTML</h2><p>Now with <code>requests</code> we can parse our HTML data with <code>requests</code> using <code>base_url.text</code> but it doesn’t offer the plethora of information and benefits that BeautifulSoup can offer. Let’s import BeautifulSoup and find out what can we do with it.</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"></span><br><span class="line"><span class="comment"># requesting index page</span></span><br><span class="line">base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># parse html data</span></span><br><span class="line">url_contents = BeautifulSoup(base_url.text, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># printing a pretty version of our HTML data</span></span><br><span class="line">print(url_contents.prettify())</span><br></pre></td></tr></table></figure><p>We can see that our output is a prettified version of our HTML requests with proper spacing and new lines</p><figure class="highlight html"><table><tr><td class="code"><pre><span class="line"><span class="meta">&lt;!DOCTYPE <span class="meta-keyword">html</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">html</span>&gt;</span></span><br><span class="line"> <span class="tag">&lt;<span class="name">head</span>&gt;</span></span><br><span class="line">  <span class="comment">&lt;!-- SNIP --&gt;</span></span><br><span class="line">   <span class="tag">&lt;<span class="name">title</span>&gt;</span></span><br><span class="line">    CodingDose()</span><br><span class="line">   <span class="tag">&lt;/<span class="name">title</span>&gt;</span></span><br><span class="line">   <span class="comment">&lt;!-- styles --&gt;</span></span><br><span class="line">   <span class="tag">&lt;<span class="name">link</span> <span class="attr">href</span>=<span class="string">"/css/style.css"</span> <span class="attr">rel</span>=<span class="string">"stylesheet"</span>/&gt;</span></span><br><span class="line">   <span class="comment">&lt;!-- rss --&gt;</span></span><br><span class="line">   <span class="tag">&lt;<span class="name">link</span> <span class="attr">href</span>=<span class="string">"/atom.xml"</span> <span class="attr">rel</span>=<span class="string">"alternate"</span> <span class="attr">title</span>=<span class="string">"CodingDose()"</span> <span class="attr">type</span>=<span class="string">"application/atom+xml"</span>&gt;</span></span><br><span class="line">   <span class="tag">&lt;/<span class="name">link</span>&gt;</span></span><br><span class="line">  <span class="tag">&lt;/<span class="name">meta</span>&gt;</span></span><br><span class="line"> <span class="tag">&lt;/<span class="name">head</span>&gt;</span></span><br><span class="line"><span class="comment">&lt;!-- SNIP --&gt;</span></span><br></pre></td></tr></table></figure><p>Now this is a lot of data, what if we only want to extract the title?</p><figure class="highlight"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>url_contents.title</span><br><span class="line">&lt;title&gt;CodingDose()&lt;/title&gt;</span><br></pre></td></tr></table></figure><p>Ok, what if we want to strip the tags?</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>url_contents.title.string</span><br><span class="line"><span class="string">'CodingDose()'</span></span><br></pre></td></tr></table></figure><p>What about if we want all the the <code>&lt;p&gt;</code> tags?</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">for</span> p <span class="keyword">in</span> url_contents.find_all(<span class="string">'p'</span>):</span><br><span class="line"><span class="meta">... </span>   print(p.string)</span><br></pre></td></tr></table></figure><p>You can find all the documentation and methods in the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" target="_blank" rel="noopener">BS4 docs</a></p><h2 id="Extracting-all-posts-titles-and-links"><a href="#Extracting-all-posts-titles-and-links" class="headerlink" title="Extracting all posts titles and links"></a>Extracting all posts titles and links</h2><p>Now, as an example, we are going to scrape this site and extract all posts titles and links, and store them in a dictionary so we can loop around them, store them or do whatever we want with them.</p><p>There’s a lot of links in this site, so how can we identify only the posts titles and links? We can use the browser inspection tool to identify the <em>Class</em> that holds our links pressing <code>CTRL + SHIFT + C</code> and clicking the first post link to inspect information about it:</p><p><img src="/assets/images/bs4_scraping/inspection_tool.png" alt="Inspection tool"></p><p>Now we see how the link and title is handled in the HTML source code displayed in the Inspection Tool:</p><p><img src="/assets/images/bs4_scraping/source_code.png" alt="Source Code"></p><p>As we can see, our post’s link and title is inside a <code>&lt;li&gt;</code> (<strong>L</strong>ist <strong>I</strong>tem) tag which is inside an <code>&lt;ul&gt;</code> (<strong>U</strong>nordered <strong>L</strong>ist) tag which belongs to the class <code>post-list</code>, now we can narrow our search using BeautifulSoup and the method <code>.find_all(tag)</code> to identify links and strings that only belongs to this class</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"></span><br><span class="line"><span class="comment"># requesting index page</span></span><br><span class="line">base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># parse html data</span></span><br><span class="line">url_contents = BeautifulSoup(base_url.text, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># extract contents from class post-list</span></span><br><span class="line">post_list = url_contents.find(class_=<span class="string">'post-list'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># filter and print link tags ('a') from previous list</span></span><br><span class="line">post_items = post_list.find_all(<span class="string">'a'</span>)</span><br><span class="line">print(post_items)</span><br></pre></td></tr></table></figure><p>Now we have a list including all links and titles from our <code>post-list</code> class but to be honest, it’s quite a mess, so let’s order our posts into a <code>{title}: {link}</code> format using the method <code>.get</code> from each item in our list which extracts the attribute data from our tags, this way if we have:</p><figure class="highlight html"><table><tr><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">a</span> <span class="attr">class</span>=<span class="string">""</span> <span class="attr">href</span>=<span class="string">"/2018/03/01/How-to-send-a-tweet-with-Python-using-Tweepy/"</span>&gt;</span>How to Send a Tweet With Python Using Tweepy<span class="tag">&lt;/<span class="name">a</span>&gt;</span></span><br></pre></td></tr></table></figure><p>We can use the method <code>.get(href)</code> to extract the attribute content (<em>/2018/03/01/How-to-send-a-tweet-with-Python-using-Tweepy/</em>) and of course we would have to attach a our base url to each <code>href</code> attribute so it displays a complete URL:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"></span><br><span class="line"><span class="comment"># requesting index page</span></span><br><span class="line">base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># parse html data</span></span><br><span class="line">url_contents = BeautifulSoup(base_url.text, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># extract contents from class post-list</span></span><br><span class="line">post_list = url_contents.find(class_=<span class="string">'post-list'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># filter and print link tags ('a') from previous list</span></span><br><span class="line">post_items = post_list.find_all(<span class="string">'a'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># extracting post item and post link from 'post_items'</span></span><br><span class="line"><span class="keyword">for</span> post <span class="keyword">in</span> post_items:</span><br><span class="line"></span><br><span class="line">    <span class="comment"># extract title with '.string' method</span></span><br><span class="line">    post_title = post.string</span><br><span class="line"></span><br><span class="line">    <span class="comment"># attach 'base_url' url to each 'href' attribute content</span></span><br><span class="line">    post_link = base_url.url + post.get(<span class="string">'href'</span>)</span><br><span class="line">    print(<span class="string">'&#123;&#125;: &#123;&#125;'</span>.format(post_title, post_link))</span><br></pre></td></tr></table></figure><p>Now we have a formatted list with our titles and links, but we should store that information in a dictionary so we can reuse it later, alright? Let’s add a dictionary holding our values in the following format:<br><code>post_dict[post_title] = post_link</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"></span><br><span class="line"><span class="comment"># requesting index page</span></span><br><span class="line">base_url = requests.get(<span class="string">'https://codingdose.info'</span>)</span><br><span class="line">url_contents = BeautifulSoup(base_url.text, <span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># filter post-list tag</span></span><br><span class="line">post_list = url_contents.find(class_=<span class="string">'post-list'</span>)</span><br><span class="line">post_items = post_list.find_all(<span class="string">'a'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># hold attributes in a dictionary</span></span><br><span class="line">post_dict = &#123;&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> post <span class="keyword">in</span> post_items:</span><br><span class="line">    post_title = post.string</span><br><span class="line">    post_link = base_url.url + post.get(<span class="string">'href'</span>)</span><br><span class="line">    post_dict[post_title] = post_link</span><br><span class="line"></span><br><span class="line"><span class="comment"># iterate our dictionary</span></span><br><span class="line"><span class="keyword">for</span> title, link <span class="keyword">in</span> post_dict.items():</span><br><span class="line">    print(<span class="string">'&#123;&#125;: &#123;&#125;'</span>.format(title, link))</span><br></pre></td></tr></table></figure><p>Now that our values (title and link) are attached to our dictionary, we can reuse them in our code, maybe to create a database with posts and links? Share them to twitter? Crawling? Whatever we want to do with them, we can do it easily if we hold our values in a dictionary.</p><p>We have only one problem left… <em><strong>pagination</strong></em>. You see, we only have the links in the index page, how can we make our script go to the second page, and the third one and so on to extract the titles and links on those pages? We’ll see this in the second part of this post. <em>See ya.</em></p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>How to Send a Tweet With Python Using Tweepy</title>
      <link href="2018/03/01/How-to-send-a-tweet-with-Python-using-Tweepy/"/>
      <url>2018/03/01/How-to-send-a-tweet-with-Python-using-Tweepy/</url>
      
        <content type="html"><![CDATA[<h1 id="Overview"><a href="#Overview" class="headerlink" title="Overview"></a>Overview</h1><p>Here are some simple steps to send a tweet with a Python script to give you an overview on how to authenticate in Twitter using OAuth and simple code snippets to manipulate Twitter data to build a bot.</p><h2 id="Requirements"><a href="#Requirements" class="headerlink" title="Requirements"></a>Requirements</h2><ul><li>python-dotenv</li><li>tweepy</li></ul><p>Installation (pipenv)</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pipenv install python-dotenv tweepy</span><br></pre></td></tr></table></figure><h2 id="Create-an-app"><a href="#Create-an-app" class="headerlink" title="Create an app"></a>Create an app</h2><p>First, go to <a href="https://apps.twitter.com/" target="_blank" rel="noopener">https://apps.twitter.com/</a> to create and <em>app</em> which we will use to access the Twitter API.</p><p><img src="/assets/images/twitter_app/create_app.png" alt="Create app"></p><p>Fill out the bot name, description, and a website (it can be anything if you don’t have a website yet.)</p><p><img src="/assets/images/twitter_app/create_app.png" alt="Fill out forms"></p><p>Tick the <strong><em>Developer Agreement</em></strong> check-box and create your twitter application.</p><p><img src="/assets/images/twitter_app/app_created.png" alt="App created"></p><p>Now, go to the <strong><em>Permissions</em></strong> tab and make sure your app has <strong><em>Read and Write</em></strong> permissions.</p><p><img src="/assets/images/twitter_app/read_write.png" alt="Permissions"></p><p>This is the most important part, go to <strong><em>Keys and Access Tokens</em></strong> and copy your <strong><em>Consumer Key</em></strong> and your <strong><em>Consumer Secret</em></strong></p><p><img src="/assets/images/twitter_app/consumer_keys.png" alt="Consumer Keys"></p><p>Scroll down until you see the section <strong><em>Your Access Tokens</em></strong> and click on <strong><em>Create my access token</em></strong></p><p><img src="/assets/images/twitter_app/copy_tokes.png" alt="Access Tokens"></p><p>Copy those tokens and save them in a safe place, <a href="/2018/02/28/How-to-securely-store-sensitive-configuration-with-dotenv/">remember our <code>.env</code> entry?</a> don’t worry, we’ll use it them here anyway.</p><h2 id="Securing-your-keys"><a href="#Securing-your-keys" class="headerlink" title="Securing your keys"></a>Securing your keys</h2><p>Store your keys in a secure <code>.env</code> text file so we can load them as environment variables later</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">consumer_key &#x3D; JtK10FrJWuAkjtpDSO4lZrhHu</span><br><span class="line">consumer_secret &#x3D; j32pVGFXGUYqcufMH87Npe7bG2vQqL3DYmKHKd8OD9DMfYe6kY</span><br><span class="line">access_token &#x3D; 968177792255971328-Oib8WU1WwjvMX6GqcVkxA3cVcbCVs88</span><br><span class="line">access_secret &#x3D; zcwiPwdOQcJBakxHaN2AVGZrxGrtYIA6DnpXcM0mDo3CG</span><br></pre></td></tr></table></figure><p>Now let’s use our snippet to load environment variables keys into our code without compromising our sensitive data</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> os <span class="keyword">import</span> getenv</span><br><span class="line"><span class="keyword">from</span> dotenv <span class="keyword">import</span> load_dotenv, find_dotenv</span><br><span class="line"></span><br><span class="line"><span class="comment"># load env keys</span></span><br><span class="line">load_dotenv(find_dotenv())</span><br><span class="line"></span><br><span class="line">consumer_key = getenv(<span class="string">'consumer_key'</span>)</span><br><span class="line">consumer_secret = getenv(<span class="string">'consumer_secret'</span>)</span><br><span class="line">access_token = getenv(<span class="string">'access_token'</span>)</span><br><span class="line">access_secret = getenv(<span class="string">'access_secret'</span>)</span><br></pre></td></tr></table></figure><h2 id="Twitter-authentication"><a href="#Twitter-authentication" class="headerlink" title="Twitter authentication"></a>Twitter authentication</h2><p>Access our Twitter app is easy importing <code>tweepy</code>, we also want to return an API object to handle requests</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># import tweepy first</span></span><br><span class="line"><span class="keyword">import</span> tweepy</span><br><span class="line"></span><br><span class="line"><span class="comment"># authentication with OAuth using our keys</span></span><br><span class="line">auth = tweepy.OAuthHandler(consumer_key, consumer_secret)</span><br><span class="line">auth.set_access_token(access_token, access_secret)</span><br><span class="line"></span><br><span class="line"><span class="comment"># return an API using our previous access</span></span><br><span class="line">api = tweepy.API(auth)</span><br></pre></td></tr></table></figure><h2 id="Sending-a-test-tweet"><a href="#Sending-a-test-tweet" class="headerlink" title="Sending a test tweet"></a>Sending a test tweet</h2><p>Once we have access our application we’re ready to send tweets through our twitter app with <code>api.update_status(message)</code></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> tweepy</span><br><span class="line"><span class="keyword">from</span> os <span class="keyword">import</span> getenv</span><br><span class="line"><span class="keyword">from</span> dotenv <span class="keyword">import</span> load_dotenv, find_dotenv</span><br><span class="line"></span><br><span class="line"><span class="comment"># load env keys</span></span><br><span class="line">load_dotenv(find_dotenv())</span><br><span class="line"></span><br><span class="line">consumer_key = getenv(<span class="string">'consumer_key'</span>)</span><br><span class="line">consumer_secret = getenv(<span class="string">'consumer_secret'</span>)</span><br><span class="line">access_token = getenv(<span class="string">'access_token'</span>)</span><br><span class="line">access_secret = getenv(<span class="string">'access_secret'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># OAuth authentication</span></span><br><span class="line">auth = tweepy.OAuthHandler(consumer_key, consumer_secret)</span><br><span class="line">auth.set_access_token(access_token, access_secret)</span><br><span class="line">api = tweepy.API(auth)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Send a tweet</span></span><br><span class="line">api.update_status(<span class="string">'Congrats! You did it!'</span>)</span><br><span class="line">print(<span class="string">'Tweet sent.'</span>)</span><br></pre></td></tr></table></figure><p>Let’s check our twitter account to see if it works</p><p><img src="/assets/images/twitter_app/test_tweet.png" alt="Yup!"></p><p>Yup! It worked! but of course this is only the beginning, there’s a lot more you can do with <strong>tweepy</strong>, check out the API documentation bellow 👇 and some snippets that you can use in your code</p><h2 id="Snippets"><a href="#Snippets" class="headerlink" title="Snippets"></a>Snippets</h2><p>Example snippets extracted from the official documentation</p><p><strong>Follow all</strong></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">for</span> follower <span class="keyword">in</span> tweepy.Cursor(api.followers).items():</span><br><span class="line">    follower.follow()</span><br></pre></td></tr></table></figure><p><strong>Pagination</strong></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Iterate through all of the authenticated user's friends</span></span><br><span class="line"><span class="keyword">for</span> friend <span class="keyword">in</span> tweepy.Cursor(api.friends).items():</span><br><span class="line">    <span class="comment"># Process the friend here</span></span><br><span class="line">    process_friend(friend)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Iterate through the first 200 statuses in the friends timeline</span></span><br><span class="line"><span class="keyword">for</span> status <span class="keyword">in</span> tweepy.Cursor(api.friends_timeline).items(<span class="number">200</span>):</span><br><span class="line">    <span class="comment"># Process the status here</span></span><br><span class="line">    process_status(status)</span><br></pre></td></tr></table></figure><p><strong>Handling Rate Limit</strong></p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># In this example, the handler is time.sleep(15 * 60),</span></span><br><span class="line"><span class="comment"># but you can of course handle it in any way you want.</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">limit_handled</span><span class="params">(cursor)</span>:</span></span><br><span class="line">    <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line">        <span class="keyword">try</span>:</span><br><span class="line">            <span class="keyword">yield</span> cursor.next()</span><br><span class="line">        <span class="keyword">except</span> tweepy.RateLimitError:</span><br><span class="line">            time.sleep(<span class="number">15</span> * <span class="number">60</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> follower <span class="keyword">in</span> limit_handled(tweepy.Cursor(api.followers).items()):</span><br><span class="line">    <span class="keyword">if</span> follower.friends_count &lt; <span class="number">300</span>:</span><br><span class="line">        <span class="keyword">print</span> follower.screen_name</span><br></pre></td></tr></table></figure><p><a href="http://tweepy.readthedocs.io/en/v3.5.0/api.html" target="_blank" rel="noopener">Check out the API Reference</a></p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>How to Securely Store Sensitive Configuration With Dotenv</title>
      <link href="2018/02/28/How-to-securely-store-sensitive-configuration-with-dotenv/"/>
      <url>2018/02/28/How-to-securely-store-sensitive-configuration-with-dotenv/</url>
      
        <content type="html"><![CDATA[<h1 id="TL-DR-Environment-Variables"><a href="#TL-DR-Environment-Variables" class="headerlink" title="TL;DR: Environment Variables"></a>TL;DR: Environment Variables</h1><p>API keys are one example of sensitive information that should remain secret, the problem is that we need to use them in our code to access third-party services like Twitter, Github, DigitalOcean and so on, so how do we manage to use those API keys without hard-coding them into the source code?</p><blockquote><p>The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.<br>— Twelve-Factor App On Configuration</p></blockquote><p>The answer is: <em>Environment Variables</em>. This is based on the <em><a href="https://12factor.net/config" target="_blank" rel="noopener">Twelve-Factor App</a></em> methodology which I recommend you to read, it is an excellent essay that will teach you the process of Software-as-a-Service (SaaS). Let’s dive into it using Python and Ruby as an example.</p><h2 id="Why-this-matters"><a href="#Why-this-matters" class="headerlink" title="Why this matters"></a>Why this matters</h2><p>Let’s put this short and easy, let’s say that we have two type of credentials to access certain web service:</p><ul><li>an API id = ‘846a4da4d06as84d6as84d06’</li><li>and a SECRET id = ‘secret_id_so_secret’</li></ul><p>If we want to use those credentials we could hard code them into our software like this</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Setting credentials</span></span><br><span class="line">api_id = <span class="string">'846a4da4d06as84d6as84d06'</span></span><br><span class="line">secret_id = <span class="string">'secret_id_so_secret'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># login function</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">login</span><span class="params">(api_key, secret)</span>:</span></span><br><span class="line">    print(<span class="string">'Logged with:'</span>)</span><br><span class="line">    print(<span class="string">'API: '</span> + api_id)</span><br><span class="line">    print(<span class="string">'SECRET: '</span> + secret_id)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Logging in to third party app with hard-coded credentials</span></span><br><span class="line">login(api_id, secret_id)</span><br></pre></td></tr></table></figure><p>That would be <strong>extremely wrong</strong> because anyone that is able laid eyes on your code will have the power to steal your credentials and access sensitive information about you, your software and even <a href="https://en.wikipedia.org/wiki/Personally_identifiable_information" target="_blank" rel="noopener">Personally Identifiable Information</a> (PII) about your customers, your team, or anybody who uses your software.</p><p>But, how do we remove these credentials from our code and use them as environment variables?</p><h1 id="Securely-Storing-Credentials-in-Python"><a href="#Securely-Storing-Credentials-in-Python" class="headerlink" title="Securely Storing Credentials in Python"></a>Securely Storing Credentials in Python</h1><p><code>python-dotenv</code> is a very easy to use package that reads a <code>key=value</code> pair from a text file <code>.env</code> (hence <em>dotenv</em>) and load those variables to your environment variables so you can use your API keys securely in your code. You can install <code>python-dotenv</code> with <a href="/2018/02/20/pipenv-development-workflow/">pipenv</a>.</p><p>Now let’s move the API credentials from our code to a safe <code>.env</code> text file:</p><h3 id="Add-a-Git-exclusion-for-env"><a href="#Add-a-Git-exclusion-for-env" class="headerlink" title="Add a Git exclusion for .env"></a>Add a Git exclusion for <code>.env</code></h3><p>First we must add an exclusion to our <code>.gitignore</code> in git so they are not uploaded to Github, commit your exclusions and you’re ready to fill your <code>.env</code> credentials</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">".env"</span> &gt;&gt; .gitignore</span><br><span class="line">git commit .gitignore -m <span class="string">"add exclusion for .env"</span></span><br></pre></td></tr></table></figure><h3 id="Install-python-dotenv-with-Pipenv"><a href="#Install-python-dotenv-with-Pipenv" class="headerlink" title="Install python-dotenv with Pipenv"></a>Install <code>python-dotenv</code> with Pipenv</h3><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">$ pipenv install python-dotenv</span><br><span class="line"><span class="comment"># Output</span></span><br><span class="line">Installing python-dotenv…</span><br><span class="line">-- SNIP --</span><br><span class="line">Adding python-dotenv to Pipfile<span class="string">'s [packages]…</span></span><br><span class="line"><span class="string">Pipfile.lock not found, creating…</span></span><br><span class="line"><span class="string">Locking [dev-packages] dependencies…</span></span><br><span class="line"><span class="string">Locking [packages] dependencies…</span></span><br><span class="line"><span class="string">Updated Pipfile.lock (446908)!</span></span><br><span class="line"><span class="string">Installing dependencies from Pipfile.lock (446908)…</span></span><br><span class="line"><span class="string">  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:00</span></span><br><span class="line"><span class="string">To activate this project'</span>s virtualenv, run the following:</span><br><span class="line"> $ pipenv shell</span><br></pre></td></tr></table></figure><h3 id="Adding-credentials"><a href="#Adding-credentials" class="headerlink" title="Adding credentials"></a>Adding credentials</h3><p>Now let’s strip our credentials from our code and add them to our <em>dotenv</em> file:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line"># .env file</span><br><span class="line">api_id &#x3D; &#39;846a4da4d06as84d6as84d06&#39;</span><br><span class="line">secret_id &#x3D; &#39;secret_id_so_secret&#39;</span><br></pre></td></tr></table></figure><h3 id="Loading-credentials-to-our-environment-variables"><a href="#Loading-credentials-to-our-environment-variables" class="headerlink" title="Loading credentials to our environment variables"></a>Loading credentials to our environment variables</h3><p>Let’s modify our code to add the <code>python-dotenv</code> package</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># import functionality to find and load dotenv credentials</span></span><br><span class="line"><span class="comment"># and getenv to get environment variables from OS</span></span><br><span class="line"><span class="keyword">from</span> os <span class="keyword">import</span> getenv</span><br><span class="line"><span class="keyword">from</span> dotenv <span class="keyword">import</span> load_dotenv, find_dotenv</span><br><span class="line"></span><br><span class="line"><span class="comment"># load environment keys, it will automatically find and load .env</span></span><br><span class="line">load_dotenv(find_dotenv())</span><br><span class="line"></span><br><span class="line"><span class="comment"># login function</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">login</span><span class="params">(api_key, secret)</span>:</span></span><br><span class="line">    print(<span class="string">'Logged with:'</span>)</span><br><span class="line">    print(<span class="string">'API: '</span> + api_id)</span><br><span class="line">    print(<span class="string">'SECRET: '</span> + secret_id)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Logging in to third party app with environment variable credentials</span></span><br><span class="line">api_id = getenv(<span class="string">'api_id'</span>)</span><br><span class="line">secret_id = getenv(<span class="string">'secret_id'</span>)</span><br><span class="line">login(api_id, secret_id)</span><br></pre></td></tr></table></figure><p>Now when we run our code we will successfully access our service with our credentials without compromising our keys</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Output</span></span><br><span class="line">Logged with:</span><br><span class="line">API: 846a4da4d06as84d6as84d06</span><br><span class="line">SECRET: secret_id_so_secret</span><br></pre></td></tr></table></figure><h2 id="Ruby-Example"><a href="#Ruby-Example" class="headerlink" title="Ruby Example"></a>Ruby Example</h2><p>Add <code>dotenv</code> to your <code>Gemfile</code> and remember to add your credentials to your <code>.env</code> file</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">source <span class="string">'https://rubygems.org'</span></span><br><span class="line">gem <span class="string">'dotenv'</span></span><br></pre></td></tr></table></figure><p>Import it and use it in your code as follows</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">'dotenv'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># load dotenv credentials</span></span><br><span class="line">Dotenv.load</span><br><span class="line"></span><br><span class="line">api_id = ENV[<span class="string">'api_id'</span>]</span><br><span class="line">secret_id = ENV[<span class="string">'secret_id'</span>]</span><br><span class="line"></span><br><span class="line">puts <span class="string">"API: <span class="subst">#&#123;api_id&#125;</span>"</span></span><br><span class="line">puts <span class="string">"SECRET: <span class="subst">#&#123;secret_id&#125;</span>"</span></span><br></pre></td></tr></table></figure><p>Now execute your code</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">bundle <span class="built_in">exec</span> ruby test.rb</span><br><span class="line"><span class="comment"># Output</span></span><br><span class="line">API: 846a4da4d06as84d6as84d06</span><br><span class="line">SECRET: secret_id_so_secret</span><br></pre></td></tr></table></figure>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
            <tag> ruby </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Manage Python Versions With Pyenv</title>
      <link href="2018/02/27/manage-python-versions-with-pyenv/"/>
      <url>2018/02/27/manage-python-versions-with-pyenv/</url>
      
        <content type="html"><![CDATA[<p>Pyenv is an excellent tool to have in your tool-set, it manages Python versions much like <code>rbenv</code> for Ruby, in fact it was forked from it.</p><blockquote><p>pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.</p></blockquote><h2 id="Installation"><a href="#Installation" class="headerlink" title="Installation"></a>Installation</h2><p>The automatic installer <a href="https://github.com/pyenv/pyenv-installer" target="_blank" rel="noopener">provided in GitHub</a> will take care of everything so you don’t have to worry about configuring anything.<br>(pro-tip: triple click to select the whole line).</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash</span><br></pre></td></tr></table></figure><p>That’s the only thing you have to do, <strong>if you’re having issues installing a python version</strong> then you will have to install the development libraries:</p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \</span><br><span class="line">libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \</span><br><span class="line">xz-utils tk-dev</span><br></pre></td></tr></table></figure><h2 id="Usage"><a href="#Usage" class="headerlink" title="Usage"></a>Usage</h2><h3 id="Install-another-version"><a href="#Install-another-version" class="headerlink" title="Install another version:"></a>Install another version:</h3><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pyenv install 2.7.14</span><br><span class="line"><span class="comment"># Output</span></span><br><span class="line">Downloading Python-3.6.4.tar.xz...</span><br><span class="line">-&gt; https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tar.xz</span><br><span class="line">Installing Python-3.6.4...</span><br><span class="line">Installed Python-3.6.4 to /home/franccesco/.pyenv/versions/3.6.4</span><br></pre></td></tr></table></figure><h3 id="Check-installed-versions"><a href="#Check-installed-versions" class="headerlink" title="Check installed versions"></a>Check installed versions</h3><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pyenv versions</span><br><span class="line"><span class="comment"># Output</span></span><br><span class="line">  system</span><br><span class="line">* 2.7.14 (<span class="built_in">set</span> by /home/franccesco/.pyenv/version)</span><br><span class="line">  3.6.4</span><br></pre></td></tr></table></figure><h3 id="Check-current-version"><a href="#Check-current-version" class="headerlink" title="Check current version"></a>Check current version</h3><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pyenv version</span><br><span class="line"><span class="comment"># Output</span></span><br><span class="line">2.7.14 (<span class="built_in">set</span> by /home/franccesco/.pyenv/version)</span><br></pre></td></tr></table></figure><h3 id="Set-global-version"><a href="#Set-global-version" class="headerlink" title="Set global version"></a>Set global version</h3><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pyenv global 3.6.4</span><br><span class="line"><span class="comment"># No output; check with version command.</span></span><br></pre></td></tr></table></figure><h3 id="Set-python-version-per-directory"><a href="#Set-python-version-per-directory" class="headerlink" title="Set python version per directory"></a>Set python version per directory</h3><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pyenv <span class="built_in">local</span> 2.7.14</span><br><span class="line"><span class="comment"># No output; check with version command.</span></span><br></pre></td></tr></table></figure><p>A file named <code>.python-version</code> should contain the python version to use uppon entering the folder: <code>cat .python-version</code></p><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Output</span></span><br><span class="line">2.7.14</span><br></pre></td></tr></table></figure><h3 id="Uninstall-a-version"><a href="#Uninstall-a-version" class="headerlink" title="Uninstall a version"></a>Uninstall a version</h3><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pyenv uninstall 2.7.14</span><br><span class="line"><span class="comment"># Output</span></span><br><span class="line">pyenv: remove /home/franccesco/.pyenv/versions/2.7.14? yes</span><br></pre></td></tr></table></figure><h3 id="Update-Pyenv"><a href="#Update-Pyenv" class="headerlink" title="Update Pyenv"></a>Update Pyenv</h3><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pyenv update</span><br><span class="line"><span class="comment"># Output</span></span><br><span class="line">Updating /home/franccesco/.pyenv...</span><br><span class="line">From https://github.com/pyenv/pyenv</span><br><span class="line"> * branch            master     -&gt; FETCH_HEAD</span><br><span class="line">Already up-to-date.</span><br><span class="line">-- SNIP --</span><br><span class="line">Updating /home/franccesco/.pyenv/plugins/pyenv-virtualenv...</span><br><span class="line">From https://github.com/yyuu/pyenv-virtualenv</span><br><span class="line"> * branch            master     -&gt; FETCH_HEAD</span><br><span class="line">Already up-to-date.</span><br><span class="line">Updating /home/franccesco/.pyenv/plugins/pyenv-which-ext...</span><br><span class="line">From https://github.com/yyuu/pyenv-which-ext</span><br><span class="line"> * branch            master     -&gt; FETCH_HEAD</span><br><span class="line">Already up-to-date.</span><br></pre></td></tr></table></figure><h2 id="Final-Words"><a href="#Final-Words" class="headerlink" title="Final Words"></a>Final Words</h2><p>Pyenv has a great integration with <a href="/2018/02/20/pipenv-development-workflow/">Pipenv</a> and you can manage Python versions and package versions with both of them at the same time without any hassle, I greatly recommend you to check them out.</p><p><a href="https://github.com/pyenv/pyenv" target="_blank" rel="noopener">Read more | Pyenv Repository</a></p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Understanding Accessors in Ruby</title>
      <link href="2018/02/25/accessors-in-ruby/"/>
      <url>2018/02/25/accessors-in-ruby/</url>
      
        <content type="html"><![CDATA[<!-- Start of Unsplash Embed Code - Full-width, with text overlay (Embed code by @BirdyOz)--><div class="overlay" style="width:100%; margin: 20px auto; text-align: center; color: white !important; text-shadow: 2px 2px 4px #000000; position: relative;">    <img src="https://images.unsplash.com/photo-1487795924438-a3cb4b7a5556?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=360&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjEyMDd9" class="img-responsive img-fluid img-sml" style="width:100%" alt="child picking strawberries in kitchen " title="child picking strawberries in kitchen ">    <div class="text-overlay" style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); width: 90%">        <p class="" style="font-size: calc( 12px + (40 - 16) * ( 80vw / (1000 - 400) )) !important; line-height: calc( 20px + (32 - 16) * ( 80vw / (1000 - 400) )) !important;">Text overlay goes here</p>        <p class="" style="font-size: calc( 12px + (24 - 16) * ( 60vw / (1000 - 400) )) !important; line-height: calc( 10px + (20 - 10) * ( 80vw / (1000 - 400) )) !important;">Optional byline or citation goes here. Delete if not required.</p>    </div></div><div class="text-muted" style="opacity: 0.5; margin-top: -1.2em">    <small><a href="https://unsplash.com/photos/PMxoh8zJNb0" target="_blank">Photo</a> by <a href="https://unsplash.com/@kellysikkema" target="_blank">@kellysikkema</a> on <a href="https://unsplash.com" target="_blank">Unsplash</a>, accessed 01/11/2020</small></div><!-- End of Unsplash Embed code --><p>One thing that puzzled me as a newbie (disclaimer: I still am) are <strong>accessors</strong> in Ruby, more commonly known as <strong>setters and getters</strong> or explicitly described as <code>attr_reader</code>, <code>attr_writer</code> and <code>attr_accessor</code>. Now let’s dive into the code first and describe the concepts of accessors <em>after</em> we’re done with coding.</p><h2 id="Initializing-a-Class"><a href="#Initializing-a-Class" class="headerlink" title="Initializing a Class"></a>Initializing a Class</h2><p>Let’s say we want to create a class to resemble a <code>Person</code> with a <code>name</code>, and finally let’s try to access that name outside the class:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Class definition</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Person</span></span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">initialize</span><span class="params">(name)</span></span></span><br><span class="line">    @name = name</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Initialize class and inspect it</span></span><br><span class="line">p1 = Person.new(<span class="string">'Anne'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Can't access 'name' variable outside class</span></span><br><span class="line">puts p1.name</span><br></pre></td></tr></table></figure><p>But it seems that we can’t access the variable <code>@name</code> even though we initialized right:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Output:</span></span><br><span class="line">test.<span class="symbol">rb:</span><span class="number">13</span><span class="symbol">:in</span> <span class="string">`&lt;main&gt;': undefined method `</span>name<span class="string">' for #&lt;Person:0x00000002278100 @name="name"&gt; (NoMethodError)</span></span><br></pre></td></tr></table></figure><p>This is because we have to <strong>define a method to access the variable inside the class</strong>.</p><h2 id="Getters-and-Setters"><a href="#Getters-and-Setters" class="headerlink" title="Getters and Setters"></a>Getters and Setters</h2><p>Let’s grab our code and add <strong>two</strong> methods:</p><ul><li>A method to <em>update</em> the person’s name</li><li>And a method to <em>read</em> the person’s name</li></ul><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Class definition</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Person</span></span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">initialize</span><span class="params">(name)</span></span></span><br><span class="line">    @name = name</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># Update 'name'</span></span><br><span class="line">  <span class="comment"># use =() to make a method behave</span></span><br><span class="line">  <span class="comment"># like an attribute assignment</span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">name=</span><span class="params">(name)</span></span></span><br><span class="line">    @name = name</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># Get value of 'name'</span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">name</span></span></span><br><span class="line">    @name</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Initialize class and the value 'name'</span></span><br><span class="line">p1 = Person.new(<span class="string">'Anne'</span>)</span><br><span class="line">puts p1.name</span><br><span class="line"></span><br><span class="line"><span class="comment"># Change 'name'</span></span><br><span class="line">p1.name = <span class="string">'Johnson'</span></span><br><span class="line">puts p1.name</span><br></pre></td></tr></table></figure><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Output</span></span><br><span class="line">Anne</span><br><span class="line">Johnson</span><br></pre></td></tr></table></figure><p>Now this code works perfectly, but a person doesn’t have just one attribute like <code>name</code>, they have age, height, eye color, skin color, hair color. Could you imagine writing endless methods about each attribute? Gladly we have <code>accessors</code></p><h2 id="Write-and-Read-Accessor"><a href="#Write-and-Read-Accessor" class="headerlink" title="Write and Read Accessor"></a>Write and Read Accessor</h2><p>Before we disclose the technical concept of <code>accessors</code> let’s first try them, shall we? We’re changing our code into a more sophisticated and short one with <code>attr_accessor</code>:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Class definition</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Person</span></span></span><br><span class="line">  <span class="keyword">attr_accessor</span> <span class="symbol">:name</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">initialize</span><span class="params">(name)</span></span></span><br><span class="line">    @name = name</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Initialize class and the value 'name'</span></span><br><span class="line">p1 = Person.new(<span class="string">'Anne'</span>)</span><br><span class="line">puts p1.name</span><br><span class="line"></span><br><span class="line"><span class="comment"># Change name</span></span><br><span class="line">p1.name = <span class="string">'Johnson'</span></span><br><span class="line">puts p1.name</span><br></pre></td></tr></table></figure><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Output</span></span><br><span class="line">Anne</span><br><span class="line">Johnson</span><br></pre></td></tr></table></figure><p>What happened here? We wrote an <code>accessor</code> that allow us to read and update the attribute <code>name</code> in the class <code>Person</code>, this way we can add more attributes like <code>height</code> and <code>weight</code> in a single line:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Class definition</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Person</span></span></span><br><span class="line">  <span class="keyword">attr_accessor</span> <span class="symbol">:name</span>, <span class="symbol">:height</span>, <span class="symbol">:weight</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">initialize</span><span class="params">(name)</span></span></span><br><span class="line">    @name = name</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Initialize class with a name and</span></span><br><span class="line"><span class="comment"># add values outside the class</span></span><br><span class="line">p1 = Person.new(<span class="string">'Anne'</span>)</span><br><span class="line">p1.height = <span class="string">'1.80m'</span></span><br><span class="line">p1.weight = <span class="string">'180lbs'</span></span><br><span class="line"></span><br><span class="line">puts <span class="string">"Name: <span class="subst">#&#123;p1.name&#125;</span>"</span></span><br><span class="line">puts <span class="string">"Height: <span class="subst">#&#123;p1.height&#125;</span>"</span></span><br><span class="line">puts <span class="string">"Weight: <span class="subst">#&#123;p1.weight&#125;</span>"</span></span><br></pre></td></tr></table></figure><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Output</span></span><br><span class="line"><span class="symbol">Name:</span> Anne</span><br><span class="line"><span class="symbol">Height:</span> <span class="number">1.80</span>m</span><br><span class="line"><span class="symbol">Weight:</span> <span class="number">180</span>lbs</span><br></pre></td></tr></table></figure><h2 id="Write-Only-Accessor"><a href="#Write-Only-Accessor" class="headerlink" title="Write-Only Accessor"></a>Write-Only Accessor</h2><p>Now that’s a lot easier than to write and read each method, but what about if want to <strong>set</strong> an attribute but not <strong>read</strong> them? For example, a person can have many thoughts but no one else can read them, let’s give this person a <code>thoughts</code> attribute and let’s try to access this person thoughts outside the class:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Class definition</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Person</span></span></span><br><span class="line">  <span class="keyword">attr_accessor</span> <span class="symbol">:name</span>, <span class="symbol">:height</span>, <span class="symbol">:weight</span></span><br><span class="line">  <span class="keyword">attr_writer</span> <span class="symbol">:thoughts</span></span><br><span class="line"></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">initialize</span><span class="params">(name)</span></span></span><br><span class="line">    @name = name</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Initialize class with a name and</span></span><br><span class="line"><span class="comment"># add values outside the class</span></span><br><span class="line">p1 = Person.new(<span class="string">'Anne'</span>)</span><br><span class="line">p1.height = <span class="string">'1.80m'</span></span><br><span class="line">p1.weight = <span class="string">'180lbs'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Set a thought and inspect the class</span></span><br><span class="line">p1.thoughts = <span class="string">'pizza &lt;3'</span></span><br><span class="line">puts p1.inspect</span><br><span class="line"></span><br><span class="line">puts <span class="string">"Name: <span class="subst">#&#123;p1.name&#125;</span>"</span></span><br><span class="line">puts <span class="string">"Height: <span class="subst">#&#123;p1.height&#125;</span>"</span></span><br><span class="line">puts <span class="string">"Weight: <span class="subst">#&#123;p1.weight&#125;</span>"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Try to access that thought</span></span><br><span class="line">puts <span class="string">"<span class="subst">#&#123;p1.name&#125;</span> is thinking about: <span class="subst">#&#123;p1.thoughts&#125;</span>"</span></span><br></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> Output</span></span><br><span class="line"><span class="meta">#</span><span class="bash">&lt;Person:0x00000000e5f538 @name=<span class="string">"Anne"</span>, @height=<span class="string">"1.80 meters"</span>, @weight=<span class="string">"180 lbs"</span>, @thoughts=<span class="string">"pizza &lt;3"</span>&gt;</span></span><br><span class="line">Name: Anne</span><br><span class="line">Height: 1.80 meters</span><br><span class="line">Weight: 180 lbs</span><br><span class="line">test.rb:26:in `&lt;main&gt;': undefined method `thoughts' for #&lt;Person:0x00000000e5f538&gt; (NoMethodError)</span><br></pre></td></tr></table></figure><p>See? We are able to set a value on the <code>thought</code> attribute, but outside the class we are unable to read it (line 26)</p><h2 id="Read-Only-Accessor"><a href="#Read-Only-Accessor" class="headerlink" title="Read-Only Accessor"></a>Read-Only Accessor</h2><p>Now, a person have a lot of things that they can’t change… not by conventional means at least. Let’s add a <em>read-only</em> attribute to this person with <code>attr_reader</code>, like <code>eye_color</code> in line 5 and initialize it:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Class definition</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Person</span></span></span><br><span class="line">  <span class="keyword">attr_accessor</span> <span class="symbol">:name</span>, <span class="symbol">:height</span>, <span class="symbol">:weight</span></span><br><span class="line">  <span class="keyword">attr_writer</span> <span class="symbol">:thoughts</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># read-only accessor</span></span><br><span class="line">  <span class="keyword">attr_reader</span> <span class="symbol">:eye_color</span></span><br><span class="line"></span><br><span class="line">  <span class="comment"># initialize eye_color</span></span><br><span class="line">  <span class="function"><span class="keyword">def</span> <span class="title">initialize</span><span class="params">(name, eye_color)</span></span></span><br><span class="line">    @name = name</span><br><span class="line">    @eye_color = eye_color</span><br><span class="line">  <span class="keyword">end</span></span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Initialize class with a name and</span></span><br><span class="line"><span class="comment"># add values outside the class</span></span><br><span class="line">p1 = Person.new(<span class="string">'Anne'</span>, <span class="string">'Green'</span>)</span><br><span class="line">p1.height = <span class="string">'1.80m'</span></span><br><span class="line">p1.weight = <span class="string">'180lbs'</span></span><br><span class="line">p1.thoughts = <span class="string">'pizza &lt;3'</span></span><br><span class="line"></span><br><span class="line">puts <span class="string">"Name: <span class="subst">#&#123;p1.name&#125;</span>"</span></span><br><span class="line">puts <span class="string">"Height: <span class="subst">#&#123;p1.height&#125;</span>"</span></span><br><span class="line">puts <span class="string">"Weight: <span class="subst">#&#123;p1.weight&#125;</span>"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Read eye color</span></span><br><span class="line">puts <span class="string">"Eye Color: <span class="subst">#&#123;p1.eye_color&#125;</span>"</span></span><br></pre></td></tr></table></figure><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Output</span></span><br><span class="line"><span class="symbol">Name:</span> Anne</span><br><span class="line"><span class="symbol">Height:</span> <span class="number">1.80</span>m</span><br><span class="line"><span class="symbol">Weight:</span> <span class="number">180</span>lbs</span><br><span class="line">Eye <span class="symbol">Color:</span> Green</span><br></pre></td></tr></table></figure><p>As you can see there’s no problem in initializing and accessing the <code>eye_color</code> attribute, but as soon as we try to change it with <code>p1.eye_color = &#39;red&#39;</code>, we can expect the following response:</p><figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">test.rb:30:in `&lt;main&gt;': undefined method `eye_color=' for #&lt;Person:0x000000019defa8&gt; (NoMethodError)</span><br></pre></td></tr></table></figure><p>That, of course, is because we set the accessor as read only.</p><h2 id="Conclusion-and-Technical-Definition"><a href="#Conclusion-and-Technical-Definition" class="headerlink" title="Conclusion and Technical Definition"></a>Conclusion and Technical Definition</h2><p>I cannot describe better what an accessor is more than the definition found in the official <a href="https://ruby-doc.org/docs/ruby-doc-bundle/UsersGuide/rg/accessors.html" target="_blank" rel="noopener">Ruby user’s guide</a>:</p><blockquote><p>An object’s instance variables are its attributes, the things that generally distinguish it from other objects of the same class. It is important to be able to write and read these attributes; doing so requires writing methods called attribute accessors.</p></blockquote><p>Basically, <code>attr_accessor</code>, <code>attr_writer</code> and <code>attr_reader</code> is a short way to define and set class variables without having to define the methods to access and read them outside the class, with:</p><ol><li><code>attr_accessor</code> we are able to <em>read</em> and <em>write</em> values outside the class.</li><li><code>attr_writer</code> allow us to write values without being able to read them outside the class.</li><li>and with <code>attr_reader</code> we can initialize and read the class attributes without being able to <em>reassign</em> it.</li></ol><p>You can read more about accessors and how useful they are in the following documentation:</p><ul><li><a href="https://ruby-doc.org/docs/ruby-doc-bundle/UsersGuide/rg/accessors.html" target="_blank" rel="noopener">Ruby User’s Guide on Accessors</a></li><li><a href="http://ruby-doc.com/docs/ProgrammingRuby/html/tut_classes.html#UC" target="_blank" rel="noopener">The Pragmatic Programmers Guide</a></li><li><a href="https://www.codecademy.com/en/forum_questions/50f0192b102455349200372d" target="_blank" rel="noopener">Codecademy - Accessors</a></li></ul><p>If I got something wrong, you can correct me in the comments bellow.</p>]]></content>
      
      
      
        <tags>
            
            <tag> accessors </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Initialize Rails and Deploy to Heroku</title>
      <link href="2018/02/23/Initialize-Rails-and-Deploy-to-Heroku/"/>
      <url>2018/02/23/Initialize-Rails-and-Deploy-to-Heroku/</url>
      
        <content type="html"><![CDATA[<h2 id="Install-Rails"><a href="#Install-Rails" class="headerlink" title="Install Rails"></a>Install Rails</h2><p>Install the gem:</p><ul><li><code>gem install rails</code></li></ul><h2 id="Create-New-Rails-Project"><a href="#Create-New-Rails-Project" class="headerlink" title="Create New Rails Project"></a>Create New Rails Project</h2><p>Create a new project and <code>cd</code> into it:</p><ul><li><code>rails new ProjectTest</code></li><li><code>cd ProjectTest</code></li></ul><h2 id="Change-Gemfile-to-add-PostgreSQL"><a href="#Change-Gemfile-to-add-PostgreSQL" class="headerlink" title="Change Gemfile to add PostgreSQL"></a>Change Gemfile to add PostgreSQL</h2><p>Heroku works with PostgreSQL as backend database as it doesn’t support <strong>SQLite3</strong>, so you’ll have to add the <code>pg</code> gem in the Gemfile in a <code>production</code> group:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">group <span class="symbol">:production</span> <span class="keyword">do</span></span><br><span class="line">  gem <span class="string">'pg'</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p><strong>IMPORTANT:</strong> After adding PostgreSQL to the production group in the Gemfile <strong>you’ll have to move the SQLite3 gem to a development group or delete it</strong>, if you work with PostgreSQL just delete it entirely but if you would like to keep SQLite3 for local development then move the gem to a dev group like this:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">group <span class="symbol">:development</span> <span class="keyword">do</span></span><br><span class="line">  <span class="comment"># Access an IRB console on exception pages or by using &lt;%= console %&gt; anywhere in the code.</span></span><br><span class="line">  gem <span class="string">'web-console'</span>, <span class="string">'&gt;= 3.3.0'</span></span><br><span class="line">  gem <span class="string">'listen'</span>, <span class="string">'&gt;= 3.0.5'</span>, <span class="string">'&lt; 3.2'</span></span><br><span class="line">  <span class="comment"># Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring</span></span><br><span class="line">  gem <span class="string">'spring'</span></span><br><span class="line">  gem <span class="string">'spring-watcher-listen'</span>, <span class="string">'~&gt; 2.0.0'</span></span><br><span class="line">  <span class="comment"># Use sqlite3 as the database for Active Record</span></span><br><span class="line">  gem <span class="string">'sqlite3'</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><p>This way Heroku won’t touch the SQLite3 gem.</p><h2 id="Update-Bundler"><a href="#Update-Bundler" class="headerlink" title="Update Bundler"></a>Update Bundler</h2><p>Lets update Bundler and install the gems excluding the production group:</p><ul><li><code>bundle update</code></li><li><code>bundle install --without production</code></li></ul><h2 id="Install-Heroku-CLI"><a href="#Install-Heroku-CLI" class="headerlink" title="Install Heroku CLI"></a>Install Heroku CLI</h2><p>Open the terminal and proceed with the installation script:</p><ul><li><code>wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh</code></li></ul><p>After that, enter your <strong>Heroku</strong> credentials:</p><ul><li><code>heroku login</code></li></ul><p>And add you <code>ssh</code> keys:</p><ul><li><code>heroku keys:add</code></li></ul><p>We’ll get back to Heroku later, we have to make some changes to our code first.</p><h2 id="Changing-the-Index-page"><a href="#Changing-the-Index-page" class="headerlink" title="Changing the Index page"></a>Changing the Index page</h2><p>Heroku doesn’t support the default index page of Rails, so let’s write a <em>Hello World!</em> as our index page to confirm it’s deploying correctly to Heroku.</p><p>Generate a controller with our index page:</p><ul><li><code>rails generate controller Welcome index</code></li></ul><p>Now open <strong><em>app/views/welcome/index.html.erb</em></strong> and change your index page as you like, for example:</p><figure class="highlight html"><table><tr><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">h1</span>&gt;</span>Hello world!<span class="tag">&lt;/<span class="name">h1</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">p</span>&gt;</span>Yup! The deployment is working. Checkout <span class="tag">&lt;<span class="name">a</span> <span class="attr">href</span>=<span class="string">"https://codingodse.info"</span>&gt;</span>CodingDose()<span class="tag">&lt;/<span class="name">a</span>&gt;</span><span class="tag">&lt;/<span class="name">p</span>&gt;</span></span><br></pre></td></tr></table></figure><h2 id="Setting-the-Root-Page"><a href="#Setting-the-Root-Page" class="headerlink" title="Setting the Root Page"></a>Setting the Root Page</h2><p>Open the routes file in <strong><em>config/routes.rb</em></strong> and change the line:</p><ul><li><code>get &#39;welcome/index&#39;</code></li></ul><p>For:</p><ul><li><code>root &#39;welcome#index&#39;</code></li></ul><p>This is how the file should look:</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line">Rails.application.routes.draw <span class="keyword">do</span></span><br><span class="line">  root <span class="string">'welcome#index'</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><h2 id="Test-it-Locally"><a href="#Test-it-Locally" class="headerlink" title="Test it Locally"></a>Test it Locally</h2><p>Fire up the server with (<code>s</code> stands for <code>server</code>):</p><ul><li><code>rails s</code></li></ul><p>Go to <a href="http://localhost:3000/" target="_blank" rel="noopener">http://localhost:3000/</a> and your index should be displaying</p><h2 id="Add-and-Commit-files"><a href="#Add-and-Commit-files" class="headerlink" title="Add and Commit files"></a>Add and Commit files</h2><p>Newer versions of <strong>Rails</strong> add a git repository, unless there’s no repository initialized you will have to do it yourself with <code>git init</code>, then add the files and commit them:</p><ul><li><code>git add .</code></li><li><code>git commit -am &quot;initialize repository&quot;</code></li></ul><h2 id="Deploy-to-Heroku"><a href="#Deploy-to-Heroku" class="headerlink" title="Deploy to Heroku"></a>Deploy to Heroku</h2><p>First, create a domain on heroku:</p><ul><li><code>heroku create</code></li></ul><p>And lets push the new repo to heroku’s servers:</p><ul><li><code>git push heroku master</code></li></ul><h2 id="Final-words"><a href="#Final-words" class="headerlink" title="Final words"></a>Final words</h2><p>There you go, you setup a new Rails project, version control and a succesful deployment to Heroku :)</p>]]></content>
      
      
      
        <tags>
            
            <tag> ruby </tag>
            
            <tag> rails </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Calculate Filename SHA1 With Ruby</title>
      <link href="2018/02/22/Calculate-filename-SHA1-with-Ruby/"/>
      <url>2018/02/22/Calculate-filename-SHA1-with-Ruby/</url>
      
        <content type="html"><![CDATA[<p><img src="/assets/thumbnails/sha1.png"></p><h2 id="Basic-usage"><a href="#Basic-usage" class="headerlink" title="Basic usage"></a>Basic usage</h2><p>Where <code>FILENAME</code> is the filename that you want to calculate</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">'digest/sha1'</span></span><br><span class="line">Digest::SHA1.hexdigest(FILENAME)</span><br></pre></td></tr></table></figure><h2 id="More-advanced-usage"><a href="#More-advanced-usage" class="headerlink" title="More advanced usage"></a>More advanced usage</h2><p>Save this code as <code>checkhash.rb</code>, usage: <code>checkhash.rb &lt;filename&gt;</code>.</p><figure class="highlight ruby"><table><tr><td class="code"><pre><span class="line"><span class="keyword">require</span> <span class="string">'digest/sha1'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Usage: checkhash.rb &lt;filename&gt;</span></span><br><span class="line">filename = ARGV.pop</span><br><span class="line"><span class="keyword">if</span> filename.<span class="literal">nil</span>? <span class="comment"># if no filename specified then prints help</span></span><br><span class="line">  puts <span class="string">'Please specify the filename to calculate the hash'</span></span><br><span class="line">  puts <span class="string">"Usage: <span class="subst">#&#123;File.basename($PROGRAM_NAME)&#125;</span> FILENAME"</span></span><br><span class="line">  exit</span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># calculating SHA1 hash</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">calculate_hash</span><span class="params">(file)</span></span></span><br><span class="line">  Digest::SHA1.hexdigest(file)</span><br><span class="line"><span class="keyword">end</span></span><br><span class="line"></span><br><span class="line">file_hash = calculate_hash(filename)</span><br><span class="line"></span><br><span class="line">puts <span class="string">"<span class="subst">#&#123;filename&#125;</span>: <span class="subst">#&#123;file_hash&#125;</span>"</span></span><br></pre></td></tr></table></figure>]]></content>
      
      
      
        <tags>
            
            <tag> cryptography </tag>
            
            <tag> ruby </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>HTTP Requests in Python</title>
      <link href="2018/02/20/http-requests-in-python/"/>
      <url>2018/02/20/http-requests-in-python/</url>
      
        <content type="html"><![CDATA[<h1 id="Before-starting"><a href="#Before-starting" class="headerlink" title="Before starting"></a>Before starting</h1><p>lets try HTTP requests with <a href="https://httpbin.org/" target="_blank" rel="noopener">httpbin.org</a> to test multiple HTTP methods with <code>requests</code> and <code>JSON</code> data. We’ll see how to extract data from a JSON-encoded response (e.g. {‘key’: ‘value’})</p><h2 id="Install-requests-library"><a href="#Install-requests-library" class="headerlink" title="Install requests library"></a>Install requests library</h2><p>With <code>pipenv</code> (<a href="https://codingdose.info/2018/02/20/pipenv-development-workflow/">recommended</a>):</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pipenv install requests</span><br></pre></td></tr></table></figure><p>With <code>pip</code>:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pip install requests</span><br></pre></td></tr></table></figure><h2 id="GET-request"><a href="#GET-request" class="headerlink" title="GET request"></a>GET request</h2><p>Check our IP address:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line"><span class="comment"># get request, response is JSON-encoded</span></span><br><span class="line">myIP = requests.get(<span class="string">'https://httpbin.org/ip'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># extract value from JSON key 'origin'</span></span><br><span class="line"><span class="comment"># &#123;'origin': '38.132.120.4'&#125;</span></span><br><span class="line">print(myIP.json()[<span class="string">'origin'</span>])</span><br></pre></td></tr></table></figure><p>User agent:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">user_agent = requests.get(<span class="string">'https://httpbin.org/user-agent'</span>)</span><br><span class="line">print(user_agent.json()[<span class="string">'user-agent'</span>])</span><br></pre></td></tr></table></figure><p>Time:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">time = requests.get(<span class="string">'https://now.httpbin.org/'</span>)</span><br><span class="line">print(time.json()[<span class="string">'now'</span>][<span class="string">'rfc2822'</span>])</span><br></pre></td></tr></table></figure><p>Passing URL parameters:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">parameters = &#123;<span class="string">'param1'</span>: <span class="string">'value2'</span>, <span class="string">'param2'</span>: <span class="string">'value2'</span>&#125;</span><br><span class="line">get_params = requests.get(<span class="string">'https://httpbin.org/get'</span>, params=parameters)</span><br></pre></td></tr></table></figure><h2 id="POST-request"><a href="#POST-request" class="headerlink" title="POST request"></a>POST request</h2><p>Post Request:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">post_data = requests.post(<span class="string">'https://httpbin.org/post'</span>, data = &#123;<span class="string">'hello'</span>: <span class="string">'world'</span>&#125;)</span><br><span class="line">print(post_data)</span><br></pre></td></tr></table></figure><h2 id="Method-Status-Code"><a href="#Method-Status-Code" class="headerlink" title="Method Status Code"></a>Method Status Code</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">req = requests.get(<span class="string">'https://httpbin.org/'</span>)</span><br><span class="line">req.status_code()</span><br></pre></td></tr></table></figure><h2 id="Authentication"><a href="#Authentication" class="headerlink" title="Authentication"></a>Authentication</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">auth = requests.get(<span class="string">'https://httpbin.org/basic-auth/user/passwd'</span>, auth=(<span class="string">'user'</span>, <span class="string">'passwd'</span>))</span><br></pre></td></tr></table></figure><h2 id="Custom-HTTP-Headers"><a href="#Custom-HTTP-Headers" class="headerlink" title="Custom HTTP Headers"></a>Custom HTTP Headers</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">my_headers = &#123;</span><br><span class="line">    <span class="string">'user-agent'</span>: <span class="string">'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'</span>,</span><br><span class="line">    <span class="string">'Referer'</span>: <span class="string">'google.com'</span>,</span><br><span class="line">    <span class="string">'Cookies'</span>: <span class="string">'AmA=Cookie!'</span>,</span><br><span class="line">&#125;</span><br><span class="line">req = requests.get(<span class="string">'https://httpbin.org/anything'</span>, headers=my_headers)</span><br></pre></td></tr></table></figure><h2 id="Putting-everything-together"><a href="#Putting-everything-together" class="headerlink" title="Putting everything together"></a>Putting everything together</h2><p>A script that reports an IP address geolocation, hostname and country:</p><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> argparse</span><br><span class="line"><span class="keyword">from</span> requests <span class="keyword">import</span> get <span class="keyword">as</span> GET</span><br><span class="line"></span><br><span class="line"><span class="comment"># argparse to enable passing command line arguments</span></span><br><span class="line"><span class="comment"># this is totally optional, it adds functionality</span></span><br><span class="line">parser = argparse.ArgumentParser(description=<span class="string">'Get IP information.'</span>)</span><br><span class="line">parser.add_argument(<span class="string">'host'</span>, help=<span class="string">'host to analize.'</span>)</span><br><span class="line">args = parser.parse_args()</span><br><span class="line"></span><br><span class="line"><span class="comment"># remote IP information</span></span><br><span class="line">ip_remote = GET(<span class="string">'http://ipinfo.io/'</span> + args.host).json()</span><br><span class="line"></span><br><span class="line"><span class="comment"># storing information in a dictionary for iteration</span></span><br><span class="line">ip_info = &#123;</span><br><span class="line">    <span class="string">'Hosname'</span>: ip_remote[<span class="string">'hostname'</span>],</span><br><span class="line">    <span class="string">'Location'</span>: <span class="string">'&#123;&#125;, &#123;&#125;'</span>.format(ip_remote[<span class="string">'region'</span>], ip_remote[<span class="string">'country'</span>]),</span><br><span class="line">    <span class="string">'Coordinates'</span>: ip_remote[<span class="string">'loc'</span>],</span><br><span class="line">    <span class="string">'Organization'</span>: ip_remote[<span class="string">'org'</span>]</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment"># print information about remote IP</span></span><br><span class="line">print(<span class="string">'Information for: &#123;&#125;'</span>.format(ip_remote[<span class="string">'ip'</span>]))</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> key, value <span class="keyword">in</span> ip_info.items():</span><br><span class="line">    print(<span class="string">'&#123;&#125;: &#123;&#125;'</span>.format(key, value))</span><br></pre></td></tr></table></figure>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Pipenv: Installation and Configuration</title>
      <link href="2018/02/20/pipenv-development-workflow/"/>
      <url>2018/02/20/pipenv-development-workflow/</url>
      
        <content type="html"><![CDATA[<h1 id="What-is-pipenv"><a href="#What-is-pipenv" class="headerlink" title="What is pipenv"></a>What is pipenv</h1><p>Essentially <strong>Pipenv</strong> is <code>pip</code> <strong>+</strong> <code>virtualenv</code> and it’s a match made in heaven. It manages dependencies, required python versions (if pyenv is available), generates pipfiles which is more reliable than a <code>requirements.txt</code> file and it generates a virtual environment so you don’t screw other environments and its requirements.</p><blockquote><p>It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages. It also generates the ever–important Pipfile.lock, which is used to produce deterministic builds.<br>— <a href="https://github.com/pypa/pipenv" target="_blank" rel="noopener">From Pipenv Repository</a></p></blockquote><p>and it is officially recommended by Python.org:</p><blockquote><p>While pip alone is often sufficient for personal use, Pipenv is recommended for collaborative projects as it’s a higher-level tool that simplifies dependency management for common use cases.<br>— <a href="https://packaging.python.org/tutorials/managing-dependencies/#installing-pipenv" target="_blank" rel="noopener">Python.org</a></p></blockquote><h2 id="Installation"><a href="#Installation" class="headerlink" title="Installation"></a>Installation</h2><p>Using <code>pip</code></p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pip install --user pipenv</span><br></pre></td></tr></table></figure><h2 id="Create-environment"><a href="#Create-environment" class="headerlink" title="Create environment"></a>Create environment</h2><p>Use argument <code>--two</code> or <code>--three</code> to specify the environment’s python version.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pipenv --three</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Creating a virtualenv for this project…</span><br><span class="line">Using &#x2F;home&#x2F;franccesco&#x2F;.pyenv&#x2F;versions&#x2F;3.6.4&#x2F;bin&#x2F;python3 to create virtualenv…</span><br><span class="line">⠋Running virtualenv with interpreter &#x2F;home&#x2F;franccesco&#x2F;.pyenv&#x2F;versions&#x2F;3.6.4&#x2F;bin&#x2F;python3</span><br><span class="line"></span><br><span class="line">--SNIP --</span><br><span class="line"></span><br><span class="line">Virtualenv location: &#x2F;home&#x2F;franccesco&#x2F;.local&#x2F;share&#x2F;virtualenvs&#x2F;test-PudGTmiz</span><br><span class="line">Creating a Pipfile for this project…</span><br></pre></td></tr></table></figure><h2 id="Install-Dependencies"><a href="#Install-Dependencies" class="headerlink" title="Install Dependencies"></a>Install Dependencies</h2><p>Let’s install <code>requests</code> as an example</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pipenv install requests</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Installing requests…</span><br><span class="line">Collecting requests</span><br><span class="line"></span><br><span class="line">-- SNIP --</span><br><span class="line"></span><br><span class="line">Adding requests to Pipfile&#39;s [packages]…</span><br><span class="line">  PS: You have excellent taste! ✨ 🍰 ✨</span><br><span class="line">Locking [dev-packages] dependencies…</span><br><span class="line">Locking [packages] dependencies…</span><br><span class="line">Updated Pipfile.lock (7b8df8)!</span><br></pre></td></tr></table></figure><h2 id="Managing-environment"><a href="#Managing-environment" class="headerlink" title="Managing environment"></a>Managing environment</h2><p>We can execute our python program without activating the virtual environment shell by running <code>pipenv run python _pythonfile.py_</code> or accessing the virtual environment with <code>pipenv shell</code></p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pipenv shell</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Spawning environment shell (&#x2F;usr&#x2F;bin&#x2F;fish). Use &#39;exit&#39; to leave.</span><br><span class="line">source &#x2F;home&#x2F;franccesco&#x2F;.local&#x2F;share&#x2F;virtualenvs&#x2F;test-PudGTmiz&#x2F;bin&#x2F;activate.fish</span><br><span class="line">-- SNIP --</span><br><span class="line"></span><br><span class="line">~$</span><br></pre></td></tr></table></figure><h2 id="Display-installed-packages"><a href="#Display-installed-packages" class="headerlink" title="Display installed packages"></a>Display installed packages</h2><p>Displays package information and versions.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pipenv graph</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">requests&#x3D;&#x3D;2.18.4</span><br><span class="line">  - certifi [required: &gt;&#x3D;2017.4.17, installed: 2018.1.18]</span><br><span class="line">  - chardet [required: &gt;&#x3D;3.0.2,&lt;3.1.0, installed: 3.0.4]</span><br><span class="line">  - idna [required: &lt;2.7,&gt;&#x3D;2.5, installed: 2.6]</span><br><span class="line">  - urllib3 [required: &lt;1.23,&gt;&#x3D;1.21.1, installed: 1.22]</span><br></pre></td></tr></table></figure><h2 id="Freezing-dependencies"><a href="#Freezing-dependencies" class="headerlink" title="Freezing dependencies"></a>Freezing dependencies</h2><p>It generates a <code>json</code> file with our environment dependencies.</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pipenv lock</span><br></pre></td></tr></table></figure><figure class="highlight"><table><tr><td class="code"><pre><span class="line">&#123;</span><br><span class="line">    <span class="attr">"_meta"</span>: &#123;</span><br><span class="line">        <span class="attr">"hash"</span>: &#123;</span><br><span class="line">            <span class="attr">"sha256"</span>: <span class="string">"33a0ec7c8e3bae6f62dd618f847de92ece20e2bd4efb496927e2524b9c7b8df8"</span></span><br><span class="line">        &#125;,</span><br><span class="line">        <span class="attr">"host-environment-markers"</span>: &#123;</span><br><span class="line">            <span class="attr">"implementation_name"</span>: <span class="string">"cpython"</span>,</span><br><span class="line">            <span class="attr">"implementation_version"</span>: <span class="string">"3.6.4"</span>,</span><br><span class="line">            <span class="attr">"os_name"</span>: <span class="string">"posix"</span>,</span><br><span class="line">            <span class="attr">"platform_machine"</span>: <span class="string">"x86_64"</span>,</span><br><span class="line">            <span class="attr">"platform_python_implementation"</span>: <span class="string">"CPython"</span>,</span><br><span class="line">            <span class="attr">"platform_release"</span>: <span class="string">"4.13.0-32-generic"</span>,</span><br><span class="line">            <span class="attr">"platform_system"</span>: <span class="string">"Linux"</span>,</span><br><span class="line">            <span class="attr">"platform_version"</span>: <span class="string">"#35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018"</span>,</span><br><span class="line">            <span class="attr">"python_full_version"</span>: <span class="string">"3.6.4"</span>,</span><br><span class="line">            <span class="attr">"python_version"</span>: <span class="string">"3.6"</span>,</span><br><span class="line">            <span class="attr">"sys_platform"</span>: <span class="string">"linux"</span></span><br><span class="line">        &#125;,</span><br><span class="line">        <span class="attr">"pipfile-spec"</span>: <span class="number">6</span>,</span><br><span class="line">        <span class="attr">"requires"</span>: &#123;</span><br><span class="line">            <span class="attr">"python_version"</span>: <span class="string">"3.6"</span></span><br><span class="line">        &#125;,</span><br><span class="line">        <span class="attr">"sources"</span>: [</span><br><span class="line">            &#123;</span><br><span class="line">                <span class="attr">"name"</span>: <span class="string">"pypi"</span>,</span><br><span class="line">                <span class="attr">"url"</span>: <span class="string">"https://pypi.python.org/simple"</span>,</span><br><span class="line">                <span class="attr">"verify_ssl"</span>: <span class="literal">true</span></span><br><span class="line">            &#125;</span><br><span class="line">        ]</span><br><span class="line">    &#125;,</span><br><span class="line"></span><br><span class="line">    -- SNIP --</span><br><span class="line"></span><br><span class="line">    "develop": &#123;&#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h2 id="Delete-a-Pipenv-virtual-environment"><a href="#Delete-a-Pipenv-virtual-environment" class="headerlink" title="Delete a Pipenv virtual environment"></a>Delete a Pipenv virtual environment</h2><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pipenv --rm</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Removing virtualenv (&#x2F;home&#x2F;franccesco&#x2F;.local&#x2F;share&#x2F;virtualenvs&#x2F;test-PudGTmiz)…</span><br></pre></td></tr></table></figure><p><em>You can find more documentation at <a href="https://github.com/pypa/pipenv" target="_blank" rel="noopener">Pipenv Guthub repo</a></em></p>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Migrate From Ghost Blog to Jekyll</title>
      <link href="2018/02/19/migrate-from-ghost-blog-to-jekyll/"/>
      <url>2018/02/19/migrate-from-ghost-blog-to-jekyll/</url>
      
        <content type="html"><![CDATA[<h1 id="Jekyll-requirements"><a href="#Jekyll-requirements" class="headerlink" title="Jekyll requirements"></a>Jekyll requirements</h1><p>Jekyll is a <strong>Ruby</strong> gem, so before installing Jekyll you have to install Ruby, I recommend that you install Ruby with <a href="https://rvm.io/" target="_blank" rel="noopener">RVM</a>.</p><h2 id="Installation"><a href="#Installation" class="headerlink" title="Installation"></a>Installation</h2><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">gem install jekyll bundler</span><br></pre></td></tr></table></figure><h2 id="Create-a-site"><a href="#Create-a-site" class="headerlink" title="Create a site"></a>Create a site</h2><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">jekyll new example_blog</span><br></pre></td></tr></table></figure><h2 id="Export-Ghost-posts"><a href="#Export-Ghost-posts" class="headerlink" title="Export Ghost posts"></a>Export Ghost posts</h2><p>Go to <a href="https://yoursite.com/ghost/settings/labs" target="_blank" rel="noopener">https://yoursite.com/ghost/settings/labs</a> and download the <strong><em>Export JSON File</em></strong>, lets save it as <code>mybackup.json</code></p><h2 id="Install-jekyll-ghost-importer"><a href="#Install-jekyll-ghost-importer" class="headerlink" title="Install jekyll_ghost_importer"></a>Install jekyll_ghost_importer</h2><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">gem install jekyll_ghost_importer</span><br></pre></td></tr></table></figure><h2 id="Import-posts-with-jekyll-ghost-importer"><a href="#Import-posts-with-jekyll-ghost-importer" class="headerlink" title="Import posts with jekyll_ghost_importer"></a>Import posts with jekyll_ghost_importer</h2><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">cd example_blog</span><br><span class="line">jekyll_ghost_importer &#x2F;path&#x2F;to&#x2F;mybackup.json</span><br></pre></td></tr></table></figure><h2 id="Generate-and-run-Jekyll"><a href="#Generate-and-run-Jekyll" class="headerlink" title="Generate and run Jekyll"></a>Generate and run Jekyll</h2><p>Once posts have been imported, lets generate and run Jekyll</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">jekyll generate</span><br><span class="line">jekyll serve --watch --livereload</span><br></pre></td></tr></table></figure><p>The flag <code>--watch</code> monitors our files and restarts the server upon changes and <code>--livereload</code> refresh the browser.</p><p>Now we should be able to see our posts:</p><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Configuration file: &#x2F;home&#x2F;franccesco&#x2F;workspace&#x2F;example_blog&#x2F;_config.yml</span><br><span class="line">            Source: &#x2F;home&#x2F;franccesco&#x2F;workspace&#x2F;example_blog</span><br><span class="line">       Destination: &#x2F;home&#x2F;franccesco&#x2F;workspace&#x2F;example_blog&#x2F;_site</span><br><span class="line"> Incremental build: disabled. Enable with --incremental</span><br><span class="line">      Generating... </span><br><span class="line">                    done in 0.799 seconds.</span><br><span class="line"> Auto-regeneration: enabled for &#39;&#x2F;home&#x2F;franccesco&#x2F;workspace&#x2F;example_blog&#39;</span><br><span class="line">    Server address: http:&#x2F;&#x2F;127.0.0.1:4000&#x2F;</span><br><span class="line">LiveReload address: http:&#x2F;&#x2F;127.0.0.1:35729  Server running... press ctrl-c to stop.</span><br></pre></td></tr></table></figure><p>Remember to edit <code>_config.yml</code> to edit your new blog’s name and description.</p>]]></content>
      
      
      
        <tags>
            
            <tag> jekyll </tag>
            
            <tag> ghost </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Sort a Dictionary With Python</title>
      <link href="2018/02/18/sort-a-dictionary-with-python/"/>
      <url>2018/02/18/sort-a-dictionary-with-python/</url>
      
        <content type="html"><![CDATA[<h2 id="Dicionary-Example"><a href="#Dicionary-Example" class="headerlink" title="Dicionary Example:"></a>Dicionary Example:</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">dictionary = &#123;<span class="string">'one'</span>: <span class="number">1</span>, <span class="string">'two'</span>: <span class="number">2</span>, <span class="string">'three'</span>: <span class="number">3</span>, <span class="string">'four'</span>: <span class="number">4</span>, <span class="string">'five'</span>: <span class="number">5</span>&#125;</span><br></pre></td></tr></table></figure><h2 id="Sorting-methods"><a href="#Sorting-methods" class="headerlink" title="Sorting methods"></a>Sorting methods</h2><h3 id="Sort-keys"><a href="#Sort-keys" class="headerlink" title="Sort keys"></a>Sort keys</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">sorted(dictionary)</span><br><span class="line">&gt;&gt; [<span class="string">'five'</span>, <span class="string">'four'</span>, <span class="string">'one'</span>, <span class="string">'three'</span>, <span class="string">'two'</span>]</span><br></pre></td></tr></table></figure><h3 id="Sort-keys-by-value"><a href="#Sort-keys-by-value" class="headerlink" title="Sort keys by value"></a>Sort keys by value</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">sorted(dictionary, key=dictionary.__getitem__)</span><br><span class="line">&gt;&gt; [<span class="string">'one'</span>, <span class="string">'two'</span>, <span class="string">'three'</span>, <span class="string">'four'</span>, <span class="string">'five'</span>]</span><br></pre></td></tr></table></figure><h3 id="Sort-values"><a href="#Sort-values" class="headerlink" title="Sort values"></a>Sort values</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">sorted(dictionary.values())</span><br><span class="line">&gt;&gt; [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br></pre></td></tr></table></figure><h3 id="Reverse-sorting-with-reverse-True"><a href="#Reverse-sorting-with-reverse-True" class="headerlink" title="Reverse sorting with reverse=True"></a>Reverse sorting with <code>reverse=True</code></h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">sorted(dictionary, key=dictionary.__getitem__, reverse=<span class="literal">True</span>)</span><br><span class="line">&gt;&gt; [<span class="string">'one'</span>, <span class="string">'two'</span>, <span class="string">'three'</span>, <span class="string">'four'</span>, <span class="string">'five'</span>]</span><br></pre></td></tr></table></figure>]]></content>
      
      
      
        <tags>
            
            <tag> python </tag>
            
        </tags>
      
    </entry>
    
    
  
  
</search>