About The Project

DataDelta is a very useful Python package for easily comparing two pandas dataframes for use in data analysis, data engineering, and tracking table changes across time.

DataDelta generates a report as both a Python dict and HTML file that summarizes the key changes between two dataframes through completing a series of tests (that can also be selected individually). The Python report is intended for use as part of a DevOps / DataOps pipeline for testing to ensure table changes are expected.

Report Bug · Request Feature

(back to top)

Getting Started

DataDelta is easy to install through pip or feel free to clone locally to make changes.

Dependencies

DataDelta has very few dependencies:

pandas: a fast, powerful, flexible and easy to use open source data analysis and manipulation tool - DataDelta is built on for comparing dataframes
numpy: The fundamental package for scientific computing with Python - used for transformations and calculations
jinja2: a fast, expressive, extensible templating engine - used to generate the HTML report
pytest (optional): a mature full-featured Python testing tool that helps you write better programs - used for testing

Installation

Install using Pip through PyPI:
```
pip install datadelta
```

OR

Clone the repo locally:

git clone https://github.com/gibbsbravo/DataDelta.git

(back to top)

Usage Examples

Quick starter code to get summary dataframe changes report:

import pandas as pd
import datadelta as delta

old_df = pd.read_csv('MainTestData_old_df.csv') # Add your old dataframe here
new_df = pd.read_csv('MainTestData_new_df.csv') # Add your new dataframe here
primary_key = 'A' # Set the primary key
column_subset = None # Specify the subset of columns of interest or leave None to compare all columns

# The consolidated_report dictionary will contain the summary changes
consolidated_report, record_changes_comparison_df = delta.create_consolidated_report(
    old_df, new_df, primary_key, column_subset)

# This will create a report named datadelta_html_report.html in the current working directory containing the summary changes
delta.export_html_report(consolidated_report, record_changes_comparison_df,
                      export_file_name='datadelta_html_report.html',
                      overwrite_existing_file=False)

Get dataframe summary:

  import pandas as pd
  import datadelta as delta

  new_df = pd.read_csv('MainTestData_new_df.csv') # Add your new dataframe here

  # Returns a report summarizing the key attributes and values of a dataframe
  summary_report = delta.get_df_summary(
    input_df=new_df, primary_key=primary_key, column_subset=column_subset, max_cols=15)

Get record count changes report:

  old_df = pd.read_csv('MainTestData_old_df.csv') # Add your old dataframe here
  new_df = pd.read_csv('MainTestData_new_df.csv') # Add your new dataframe here
  primary_key = 'A' # Set the primary key
  column_subset = None # Specify the subset of columns of interest or leave None to compare all columns

  # Returns a report summarizing any changes to the number of records (and composition) between two dataframes
  record_count_change_report = delta.check_record_count(
    old_df, new_df, primary_key)

Other functions include:

check_column_names: Returns a report summarizing any changes to column names between two dataframes
check_datatypes: Returns a report summarizing any columns with different datatypes
check_chg_in_values: Returns a report summarizing any records with changes in values
get_records_in_both_tables: Returns the records found in both dataframes
get_record_changes_comparison_df: Returns a dataframe comparing any records with changes in values by column
export_html_report: Exports an html report of the differences between two dataframes

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!