Asyncio in Python

Image Credit

In this 'Asyncio with Python' repo, we have covered the following usecases:

• Fetching Pokemon names using Pokemon APIs
• Fetching Weather information for certain cities in the US using Openweather APIs
• Reading Automation Builds & Session information using LambdaTest APIs
• Checking URL Health of links present on LambdaTest Selenium Playground
• Web Scraping of content on LambdaTest E-Commerce Playground

The scenarios mentioned above are executed using Synchronous & Asynchronous (or Async) modes in Python. The Async Programming is realized by leveraging the asyncio library in Python.

Pre-requisites for test execution

Step 1

Create a virtual environment by triggering the virtualenv venv command on the terminal

virtualenv venv

Step 2

Navigate the newly created virtual environment by triggering the source venv/bin/activate command on the terminal

source venv/bin/activate

Follow steps(3) and (4) for using the LambdaTest Cloud Grid (particularly used for Web Scraping & URL Health Checking Scenarios):

Step 3

Procure the LambdaTest User Name and Access Key by navigating to LambdaTest Account Page. You might need to create an an account on LambdaTest since it is used for running tests (or scraping) on the cloud Grid.

Step 4

Add the LambdaTest User Name and Access Key in the .env (or Makefile)that is located in the parent directory. Once done, save the Makefile.

Step 5

For realizing the Weather Information scenario, you need to register on OpenWeather. Once done, set the environment variable OPEN_WEATHER_API in .env. The API keys can be located from OpenWeather API Page

Dependency/Package Installation

Run the make install command on the terminal to install the desired packages (or dependencies) - Pytest, Selenium, Beautiful Soup, etc.

make install

For benchmarking (over a certain number of runs), we have used hyperfine, a command-line benchmarking tool. Installation of hyperfine on macOS is done by triggering brew install hyperfine on the terminal.

With this, all the dependencies and environment variables are set. It's time for some action !!

Execution

Follow the below mentioned steps to benchmark the performance of sync and async code (via the hyperfine tool):

Step 1

For the URL health checking scenario, the Chrome browser is invoked in the non-headless mode. It is recommended to install Chrome on your machine before you proceed to Step(4)

Step 2

Trigger the command make clean to clean the remove pycache folder(s) and .pyc files. It also cleans .DS_Store folder from the project.

Step 3

Trigger the make command on the terminal to realize the usecases in the sync & async mode.

Fetching Pokemon names using Pokemon APIs

Trigger the command make fetch-pokemon-names to fetch the Pokemon names using Pokemon APIs and running code in sync & async mode (using Asyncio in Python).

Result: Asyncio - 5.77+ seconds faster than sync execution

Fetching Weather information for certain cities in the US using Openweather APIs

In this scenario, the city names from Page-1 thru' Page-15 are first scraped using BeautifulSoup.

Now that we have the city names, let's fetch the weather of those cities by providing Latitude & Longitude to the OpenWeather API

Trigger the command make fetch-sync-weather-info to fetch the Pokemon names using Pokemon APIs and running code in sync mode.

Trigger the command make fetch-async-weather-info to fetch the Pokemon names using Pokemon APIs and running code in sync & async mode (using Asyncio in Python).

Result: Asyncio - 318.53+ seconds faster than sync execution

Checking URL health

In this scenario, the hyperlinks on LambdaTest Selenium Playground are first scraped using BeautifulSoup. Once the links are available, their health is checked using the response code via the requests library

Trigger the command make check-url-health to check the health (i.e. response code) of the links present on the LambdaTest Selenium Playground.

Result: Asyncio - 1.41+ seconds faster than sync execution

Web Scraping

In this scenario, the product details from Page-1 thru' Page-5 are scraped using the BeautifulSoup library.

The implementation using the sync programming is similar to the Web Scraping scenario that covered in Web Scraping with Python repo.

Trigger the command make perform-web-scraping to check scrap the product details on the test page mentioned earlier.

Result: Asyncio - 2.93+ seconds faster than sync execution

Have feedback or need assistance?

Feel free to fork the repo and contribute to make it better! Email to himanshu[dot]sheth[at]gmail[dot]com for any queries or ping me on the following social media sites:

LinkedIn: @hjsblogger
Twitter: @hjsblogger