Project Setup

Getting Started

Creating a Flink Stream Processing Project

Please check the following Project setup guidelines to set up a stream processing project with Apache Flink using Connectors.

Once after the set up, please follow the below instructions to add the Flink Pravega connectors to the project.

Add the Connector Dependencies

To add the Pravega connector dependencies to your project, add the following entry to your project file: (For example, pom.xml for Maven)

<dependency>
  <groupId>io.pravega</groupId>
  <artifactId>pravega-connectors-flink-<FLINK_MAJOR_VERSION>_2.12</artifactId>
  <version><CONNECTOR_VERSION></version>
</dependency>

Use appropriate version as necessary. The connector releases are aligned with the same version of the Pravega releases and the supported Flink version is in the middle of the artifact name. For example, if the your application running environment is Flink 1.12.3 and the Pravega server is 0.10.0. You should add the following entry in the project file:

<dependency>
  <groupId>io.pravega</groupId>
  <artifactId>pravega-connectors-flink-1.12_2.12</artifactId>
  <version>0.10.0</version>
</dependency>

The snapshot versions are published to Github Registry. The following credential is required to access the github registry artifacts repository.

maven {
    url "https://maven.pkg.github.com/pravega/flink-connectors"
    credentials {
        username = "pravega-public"
        password = "\u0067\u0068\u0070\u005F\u0048\u0034\u0046\u0079\u0047\u005A\u0031\u006B\u0056\u0030\u0051\u0070\u006B\u0079\u0058\u006D\u0035\u0063\u0034\u0055\u0033\u006E\u0032\u0065\u0078\u0039\u0032\u0046\u006E\u0071\u0033\u0053\u0046\u0076\u005A\u0049"
    }
}

The release artifacts are available in Maven Central repository.

Alternatively, we could build and publish the connector artifacts to local maven repository by executing the following command and make use of that version as your application dependency.

./gradlew clean publishToMavenLocal

Running / Deploying the Application

From Flink's perspective, the connector to Pravega is part of the streaming application (not part of Flink's core runtime), so the connector code must be part of the application's code artifact (JAR file). Typically, a Flink application is bundled as a fat-jar (also known as an uber-jar) , such that all its dependencies are embedded.

• The project set up should have been a success, if you have used the above linked templates/guides.

• If you set up a application's project and dependencies manually, you need to make sure that it builds a jar with dependencies, to include both the application and the connector classes.