Team-DataWiz DatafestAfrica Datathon 2023 - Data Engineering Project

A wrap-up project for the 3-day DatafestAfrica Datathon 2023!
Explore the docs »

·

Table of Contents

1. About The Project
   • Built With
   • Architecture
2. Usage
3. Roadmap
4. Pipeline
5. Deliverables
   • Jobs
   • Environments
   • Runs

About The Project

Agricultural Monitoring Systems

Our business operates in the agricultural sector, where monitoring various environmental conditions and sensor data is crucial for optimizing crop yields and ensuring sustainable farming practices. This datathon is aligned with our commitment to leverage data engineering to streamline the processing and analysis of agricultural data.

(back to top)

Architecture

Built With

Language, frameworks, libraries, Services and Tools used to bootstrap this project.

• DBT
• Snowflake
• GitHub
• SQL
• Jinja Template

(back to top)

Usage

This dataset simulates real-time agricultural sensor data, providing a rich source of information to design and construct a robust data engineering solution. The primary objective of this project is to extract, transform, and load (ETL) the raw sensor data into structured fact and dimension tables within our snowflake data warehouse.

(back to top)

Roadmap

• Create a Snowflakes Account
• Setup DBT cloud or core using the snowflake as its warehouse
• Data Modelling (logical and business)
• Develop
• Test and Documentation
• Create Jobs
• Deploy to Production

Pipeline

Deliverables

Jobs

Environments

Runs