Skip to content
@beam-pyio

Apache Beam Python I/O Connectors

README.md

Apache Beam Python I/O Connectors

This project aims to build Apache Beam® I/O Connectors using the Python SDK. Visit the project website for more details.

We keep looking into adding new connectors. If you have a new idea, please add a comment to this discussion.

AWS Services

I/O connectors for selective AWS services. They are developed by interfacing with the boto3 package.

Connector name Source Supported Sink Supported Batch Supported Streaming Supported
firehose_pyio ❌ ✔️ ✔️ ✔️
sqs_pyio 📅 ✔️ ✔️ ✔️
sns_pyio ❌ 📅 📅 📅
dynamodb_pyio 📅 ✔️ ✔️ ✔️
eventbridge_pyio ❌ 📅 📅 📅

SQL Sources

It connects to over 20 databases, data warehouses and query engines by integrating with the Daft package. See the SQL integration page for details.

Connector name Source Supported Sink Supported Batch Supported Streaming Supported
sql_pyio 📅 📅 📅 ❌

Open Table Format

These packages connect to popular open table formats including Apache Iceberg, Apache Hudi and Delta Lake. They are implemented by integrating with the Daft package. See the integrations page for details.

Connector name Source Supported Sink Supported Batch Supported Streaming Supported
iceberg_pyio 📅 📅 📅 ❌
hudi_pyio 📅 📅 📅 ❌
deltalake_pyio 📅 📅 📅 ❌

Code of Conduct

All Apache Beam Python I/O Connectors projects are governed by the project Code of Conduct. Please review that and keep it in mind in all interactions.

Popular repositories Loading

  1. firehose_pyio firehose_pyio Public

    Apache Beam Python I/O connector for Amazon Data Firehose

    Python 2

  2. sns_pyio sns_pyio Public

    Apache Beam Python I/O connector for Amazon SNS

    Python 1

  3. sqs_pyio sqs_pyio Public

    Apache Beam Python I/O connector for Amazon SQS

    Python 1

  4. dynamodb_pyio dynamodb_pyio Public

    Apache Beam Python I/O connector for Amazon DynamoDB

    Python 1

  5. .github .github Public

  6. pyio-cookiecutter pyio-cookiecutter Public

    Cookiecutter template for creating a package for the Apache Beam Python I/O Connectors project

    Python

Repositories

Showing 8 of 8 repositories
  • Daft Public Forked from Eventual-Inc/Daft

    Distributed data engine for Python/SQL designed for the cloud, powered by Rust

    beam-pyio/Daft’s past year of commit activity
    Rust 0 Apache-2.0 164 0 0 Updated Nov 8, 2024
  • beam-pyio.github.io Public

    Website for Apache Beam Python I/O Connectors

    beam-pyio/beam-pyio.github.io’s past year of commit activity
    SCSS 0 MIT 0 0 0 Updated Oct 20, 2024
  • dynamodb_pyio Public

    Apache Beam Python I/O connector for Amazon DynamoDB

    beam-pyio/dynamodb_pyio’s past year of commit activity
    Python 1 0 0 0 Updated Sep 21, 2024
  • .github Public
    beam-pyio/.github’s past year of commit activity
    0 0 0 0 Updated Sep 21, 2024
  • sqs_pyio Public

    Apache Beam Python I/O connector for Amazon SQS

    beam-pyio/sqs_pyio’s past year of commit activity
    Python 1 0 0 0 Updated Sep 20, 2024
  • firehose_pyio Public

    Apache Beam Python I/O connector for Amazon Data Firehose

    beam-pyio/firehose_pyio’s past year of commit activity
    Python 2 0 0 0 Updated Sep 20, 2024
  • pyio-cookiecutter Public

    Cookiecutter template for creating a package for the Apache Beam Python I/O Connectors project

    beam-pyio/pyio-cookiecutter’s past year of commit activity
    Python 0 0 0 0 Updated Sep 17, 2024
  • sns_pyio Public

    Apache Beam Python I/O connector for Amazon SNS

    beam-pyio/sns_pyio’s past year of commit activity
    Python 1 0 3 0 Updated Jul 24, 2024