Skip to content
/ Stock-market-realtime-data-pipeline Public

Read the data from a source file using python and then produced that data to a kafka broker using a kafka producer , then consumed the message using a kafka consumer , uploaded the data to a aws s3 bucket then built crawler on top that and then queried that data using aws athena.

0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

Zain970/Stock-market-realtime-data-pipeline

BranchesTags

Repository files navigation

Stock Market Kafka Real Time Data Engineering Project

Introduction

In this project, I will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka.

I have used different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

Architecture

Technology Used

  • Programming Language - Python
  • Amazon Web Service (AWS)
  1. S3 (Simple Storage Service)
  2. Athena
  3. Glue Crawler
  4. Glue Catalog
  5. EC2
  • Apache Kafka

About

Read the data from a source file using python and then produced that data to a kafka broker using a kafka producer , then consumed the message using a kafka consumer , uploaded the data to a aws s3 bucket then built crawler on top that and then queried that data using aws athena.

Topics

python aws kafka ec2 athena s3-bucket glue-catalog

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages