Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update: SQL application to restart strategy #48

Merged
merged 7 commits into from
Mar 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 31 additions & 32 deletions docs/products/flink/howto/create-jar-application.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,20 @@ title: Create a JAR application
limited: true
---

Aiven for Apache Flink® enables you to upload and deploy
[custom code as a JAR file](/docs/products/flink/concepts/custom-jars), enhancing your
Flink applications with advanced data
processing capabilities.
Aiven for Apache Flink® enables you to upload and deploy [custom code as a JAR file](/docs/products/flink/concepts/custom-jars), enhancing your Apache Flink applications with advanced data processing capabilities.

## Prerequisite

- Custom JARs for Aiven for Apache Flink is a **limited availability** feature.
To try this feature, request it by contacting
our [Sales team](mailto:sales@aiven.io).
- Once activated on your Aiven account, enable it:
- For a new Aiven for Apache Flink service,
toggle the feature during service creation.
- For an existing service, in
[Aiven Console](https://console.aiven.io/):
1. Select your project and choose your Aiven for Apache Flink® service.
1. Click **Service settings** on the left sidebar.
1. Scroll to the **Advanced configuration** section, and click
**Configure**.
1. In the **Advanced configuration** screen, click **Add
configuration options**, and using the search box, find and set
`custom_code` configuration to **Enabled**.
To try this feature, request access by contacting the [sales team](mailto:sales@aiven.io).
- Create a new Aiven for Apache Flink service and click **Upload and deploy custom JARs**
to enable custom JARs during service creation.

:::note
Enabling Custom JARs for existing services is currently not possible. If you did not
enable this feature during service creation, you must create a new service with
Custom JARs enabled.
:::

## Create and deploy application

Expand All @@ -39,6 +31,12 @@ processing capabilities.
1. Click **Create application** to proceed.
1. Click **Upload first version** to upload the first version of the
application.

:::note
There is a limit on the size of the custom JAR file you can upload to an Apache Flink
JAR application. For more information, contact the sales or support team.
:::

1. In the **Upload new version** dialog:
1. Click **Choose file** to select your custom JAR file.
1. Select the **Terms of Service** checkbox to indicate your
Expand All @@ -48,20 +46,21 @@ processing capabilities.
page.
1. To deploy the application, click **Create deployment**. In the
**Create new deployment** dialog:
1. Select the application version to deploy.
1. Select a
[savepoint](/docs/products/flink/concepts/savepoints) if you wish to deploy from
a specific state. No
savepoints are available for the first application deployment.
1. Toggle **Restart on failure** to automatically restart Flink
jobs upon failure.
1. In the **Program args** field, provide command-line arguments
consisting of variables and configurations relevant to your
application'slogic upon submission. Each argument is limited
to 64 characters, with a total limit of 32 separate items.
1. Specify the number of [parallel
instances](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/)
you require for the task.
1. Select the application version to deploy.
1. Select a
[savepoint](/docs/products/flink/concepts/savepoints) if you wish to deploy from
a specific state. No
savepoints are available for the first application deployment.
1. Toggle **Restart on failure** to automatically restart Flink
jobs upon failure. See [Restart strategy in SQL and JAR applications](/docs/products/flink/howto/restart-strategy-jar-applications)
for details.
1. In the **Program args** field, provide command-line arguments
consisting of variables and configurations relevant to your
application'slogic upon submission. Each argument is limited
to 64 characters, with a total limit of 32 separate items.
1. Specify the number of [parallel
instances](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/)
you require for the task.
1. Click **Deploy without a savepoint** to begin the deployment
process.
1. While deploying, the application status shows **Initializing**. Once
Expand Down
121 changes: 59 additions & 62 deletions docs/products/flink/howto/create-sql-application.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,7 @@
title: Create an SQL application
---

In Aiven for Apache Flink®, you can create an SQL application that uses
Apache Flink SQL to streamline the process of building data processing
pipelines. The SQL application simplifies the process of defining source
and sink tables, implementing data processing logic, and managing
application deployment.
Build data processing pipelines in Aiven for Apache Flink® by creating SQL applications using Apache Flink SQL. Set up source and sink tables, define processing logic, and manage your deployments.

## Prerequisite

Expand All @@ -19,82 +15,83 @@ Flink applications.
Create an SQL application in Aiven for Apache Flink® using the [Aiven
Console](https://console.aiven.io/):

1. In the [Aiven Console](https://console.aiven.io/), select the Aiven
for Apache Flink service where you want to create and deploy a Flink
application.
1. In the [Aiven Console](https://console.aiven.io/), select the Aiven
for Apache Flink service where you want to create and deploy a Flink
application.

2. From the left sidebar, click **Applications** and then click
**Create application**.
1. From the left sidebar, click **Applications** and then click
**Create application**.

3. In the **Create application** dialog, enter the name of your
application and select **SQL** as the application type.
1. In the **Create application** dialog, enter the name of your
application and select **SQL** as the application type.

4. Click **Create application**.
1. Click **Create application**.

5. Click **Create first version** to create the first version of the
application.
1. Click **Create first version** to create the first version of the
application.

6. Click **Add your first source table** to add a source table.
1. Click **Add your first source table** to add a source table.

:::note
As this is your first application, no other applications are
available to import source tables.
:::
:::note
As this is your first application, no other applications are
available to import source tables.
:::

7. On the **Add new source table** screen:
1. On the **Add new source table** screen:

- Use the **Integrated service** drop-down to select the service.
- In the **Table SQL** section, enter the SQL statement to create
the source table.
- Optionally, click **Run** to test how data is being retrieved
from the data source. This may vary in time based on the data
volume and connection speed.
- Click **Add table**.
- Use the **Integrated service** drop-down to select the service.
- In the **Table SQL** section, enter the SQL statement to create
the source table.
- Optionally, click **Run** to test how data is being retrieved
from the data source. This may vary in time based on the data
volume and connection speed.
- Click **Add table**.

8. Click **Next** to proceed to adding a sink table and click **Add
your first sink table**.
1. Click **Next** to proceed to adding a sink table and click **Add
your first sink table**.

:::note
As this is your first application, no other applications are
available to import sink tables.
:::
:::note
As this is your first application, no other applications are
available to import sink tables.
:::

9. On the **Add new sink table** screen:
1. On the **Add new sink table** screen:

- Use the **Integrated service** drop-down to select the service.
- In the **Table SQL** section, enter the SQL statement to create
the sink table.
- Click **Add table**.
- Use the **Integrated service** drop-down to select the service.
- In the **Table SQL** section, enter the SQL statement to create
the sink table.
- Click **Add table**.

10. Click **Next** to enter the **SQL statement** that transforms the
data from the source stream. Optionally, click **Run** to see how
the data is extracted from the source.
1. Click **Next** to enter the **SQL statement** that transforms the
data from the source stream. Optionally, click **Run** to see how
the data is extracted from the source.

11. Click **Save and deploy later** to save the application. You can
view and access the application you created on the application
overview page.
1. Click **Save and deploy later** to save the application. You can
harshini-rangaswamy marked this conversation as resolved.
Show resolved Hide resolved
view and access the application you created on the application
overview page.

![Application landing page with a view of the source table, SQL statement, and sink table](/images/products/flink/application_landingpage_view.png)
![Application landing page with a view of the source table, SQL statement, and sink table](/images/products/flink/application_landingpage_view.png)

12. To deploy the application, click **Create deployment**. In the
**Create new deployment** dialog:
1. To deploy the application, click **Create deployment**. In the
**Create new deployment** dialog:

- Select the application version to deploy. The default version
for the first deployment is **Version: 1**.
- Select a
[savepoint](/docs/products/flink/concepts/savepoints) if you wish to deploy from a specific state. No
savepoints are available for the first application deployment.
- Toggle **Restart on failure** to automatically restart Flink
jobs upon failure.
- Specify the number of [parallel
instances](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/)
you require for the task.
- Select the application version to deploy. The default version
for the first deployment is **Version: 1**.
- Select a
[savepoint](/docs/products/flink/concepts/savepoints) if you wish to deploy from a specific state. No savepoints are available for the first
application deployment.
- Toggle **Restart on failure** to automatically restart Flink
jobs upon failure. See [Restart strategy in SQL and JAR applications](/docs/products/flink/howto/restart-strategy-jar-applications)
for details.
- Specify the number of [parallel
instances](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/)
you require for the task.

13. Click **Deploy without a savepoint** to begin the deployment
process.
1. Click **Deploy without a savepoint** to begin the deployment
process.

14. While deploying, the application status shows **Initializing**. Once
deployed, the status changes to **Running**.
1. While deploying, the application status shows **Initializing**. Once
deployed, the status changes to **Running**.

## Create SQL applications using Aiven CLI

Expand Down
111 changes: 63 additions & 48 deletions docs/products/flink/howto/restart-strategy-jar-applications.md
Original file line number Diff line number Diff line change
@@ -1,48 +1,55 @@
---
title: Restart strategy in JAR applications
title: Restart strategy in SQL and JAR applications
---
Learn how Aiven for Apache Flink® applications uses restart strategies to recover from job failures, ensuring high availability and fault tolerance for your distributed applications.

A restart strategy is a set of rules that Apache Flink® adheres to when
dealing with Flink job failures. These strategies enable the automatic
restart of a failed Flink job under specific conditions and parameters,
which is crucial for high availability and fault tolerance in
distributed and scalable systems.
## About restart strategy

## Default restart strategy for JAR applications
A restart strategy is a set of rules that Apache Flink® adheres to when dealing with
Flink job failures. These strategies enable the automatic restart of a failed Flink job
under specific conditions and parameters, which is crucial for high availability and fault
tolerance in distributed and scalable systems. Aiven for Apache Flink® supports restart
strategies for both JAR and SQL applications.

Aiven for Apache Flink® includes a default restart strategy for JAR
applications. This strategy uses the **exponential-delay** technique,
incrementally increasing the delay time between restarts up to a
specified maximum. Once this maximum delay is reached, it remains
constant for any subsequent restarts. The default strategy is fully
integrated into the Aiven for Apache Flink cluster configuration and
automatically applies to all JAR applications.
## Default restart strategy for JAR and SQL applications

### View the default strategy
Aiven for Apache Flink® uses the **exponential-delay** strategy as the default
restart mechanism for both **JAR and SQL applications**. This strategy incrementally
increases the delay between restarts, reaching a configurable maximum. After reaching
the maximum, the delay remains constant. The strategy resets the exponential delay after
a period of successful restarts, preventing it from staying at the maximum indefinitely.
This default strategy is integrated into the Aiven for Apache Flink cluster configuration
and automatically applies to JAR and SQL applications.

## View the default strategy

You can view the default restart strategy configurations for your Aiven
for Apache Flink cluster in the Apache Flink Dashboard. Follow these
steps to view the current settings:

1. Access the [Aiven Console](https://console.aiven.io/) and select the
Aiven for Apache Flink service.
2. From the **Connection information** section on the overview page,
copy the **Service URI** and paste it into your web browser's
address bar.
3. When prompted, log in using the **User** and **Password**
credentials specified in the **Connection information** section.
4. Once in the **Apache Flink Dashboard**, click the **Job Manager**
from the menu.
5. Switch to the **Configuration** tab.
6. Review the configurations and parameters related to the restart
strategy.
1. Access the [Aiven Console](https://console.aiven.io/) and select the
Aiven for Apache Flink service.
1. From the **Connection information** section on the overview page,
copy the **Service URI** and paste it into your web browser's
address bar.
1. When prompted, log in using the **User** and **Password**
credentials specified in the **Connection information** section.
1. Once in the **Apache Flink Dashboard**, click the **Job Manager**
from the menu.
1. Switch to the **Configuration** tab.
1. Review the configurations and parameters related to the restart
strategy.

## Disable default restart strategy

While Aiven for Apache Flink® typically recommends using the default
restart strategy for JAR applications, there are circumstances,
particularly during testing or debugging, where disabling automatic
restarts might be necessary. You cannot disable the default restart
While Aiven for Apache Flink® typically recommends following the default restart strategy
for high availability and fault tolerance, there might be scenarios,
especially during testing or debugging, where disabling automatic restarts
might be beneficial.

### JAR applications

For JAR applications, you cannot disable the default restart
strategy in Aiven for Apache Flink® through configuration files.
Instead, directly modify the code of your Jar application to achieve
this.
Expand All @@ -52,28 +59,36 @@ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironm
env.setRestartStrategy(RestartStrategies.noRestart());
```

This code sets the restart strategy to \'None\', preventing any restart
This code sets the restart strategy to `None`, preventing any restart
attempts in case of failures.

### Key considerations when disabling default restarts
### SQL applications

For SQL applications, you have a simplified approach to
restart strategies. You can enable or disable restarts on failure during
application deployment, providing a straightforward way
to manage applications without complex configurations.
For more information, see [Create an SQL application](/docs/products/flink/howto/create-sql-application).

## Key considerations when disabling default restarts

Before choosing to disable the default restart strategy, consider the
Before disabling the default restart strategy for your applications, consider the
following:

- **Persistent failures**: Disabling restarts means that if a Flink
Job fails, Flink will not attempt to recover it, leading to
permanent job failure.
- **Testing and debugging**: Disabling is beneficial when identifying
issues in the application code, as it prevents the masking of errors
through automatic restarts.
- **External factors**: Jobs can fail due to external factors, such as
infrastructure changes or maintenance activities. If you disable
restarts, your Flink jobs will become vulnerable to failures.
- **Operational risks**: In production environments, it is generally
advisable to use the default restart strategy to ensure high
availability and fault tolerance.
- **Persistent failures**: Disabling restarts means that if a Flink
Job fails, it will not attempt to recover it, leading to
permanent job failure.
- **Testing and debugging**: Disabling is beneficial when identifying
issues in the application code, as it prevents the masking of errors
through automatic restarts.
- **External factors**: Jobs can fail due to external factors, such as
infrastructure changes or maintenance activities. If you disable
restarts, your Flink jobs will become vulnerable to failures.
- **Operational risks**: In production environments, it is generally
advisable to use the default restart strategy to ensure high
availability and fault tolerance.

## Related pages

- [Restart strategies in Apache
Flink®](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/state/task_failure_recovery/#restart-strategies)
- [Restart strategies in Apache
Flink®](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/state/task_failure_recovery/#restart-strategies)