Learn how to auto-ingest streaming data into Snowflake using Snowpipe.

AWS S3, Snowflake (Snowpipe, Snowpark) and PowerBi.

CLI script for interacting with Snowpipe REST APIs

SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

Generating Streaming Data for Snowpipe Streaming

SQL Scripts related to my learning on the Snowflake data cloud provider

SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

The goal is to build an example of a simple data collection pipeline which collects data from multiple customers and uploads the data to Snowflake could look like. The OpenMeteo Api acts as the "customer system" in this case.

❄️ Stream real-time data into Snowflake with Amazon Kinesis Firehose

Automatically move data from AWS S3 to Snowflake datawarehouse continuously with snowpipe

Implemented Snowflake project on AWS for efficient data storage and transformation. Utilized JSON-to-CSV conversion, Snowpipe for real-time ingestion, and reader accounts for secure data access. Employed streams, tasks, and materialized views for data synchronization and optimization. Implemented masking policies for enhanced data security.

This project showcases a scalable ETL pipeline that automates the extraction, transformation, and storage of Redfin housing market data using AWS, Apache Airflow, and Snowflake, with Power BI for data visualization. The pipeline is configured to run monthly, ensuring your data remains up-to-date.

Realtime data pipeline using Kafka + Spark + AWS S3 (Terraform) + Snowflake

Retail data analysis pipeline utilizing AWS S3, Snowflake, Python, SQL, and Tableau. It demonstrates data transformation and setup in Jupyter Notebook, integrates real-time retail insights via an automated Tableau dashboard with Snowflake, and employs a CRON job in Jupyter Lab connected to Amazon SQS for consistent data updates.

ETL with AWS Lambda and Snowflake with automation using Snowpipe.