Streamis 0.3.1 Release

Streamis 0.3.1 includes all of Project project 0.3.1.

Streamis 0.3.1 provides enhancements in project management and job configuration, adds a heartbeat mechanism to log callback, and optimizes high availability and operation logs. Streamis 0.3.1 is compatible with multiple versions of Linkis (>= 1.4.0) and DSS (>= 1.1.2)

Streamis 0.3.1 mainly adds the following features:

• Support job-level resource downloading.
• Supports viewing and downloading of meta.json of existing execution history tasks.
• Add audit log recording to Streamis.
• Support batch enable and disable jobs.
• Adjust the default snapshot root address based on the high availability policy.
• Add cluster high availability query interface.
• Add heartbeat for callback log.
• add hook before flink apps stop.
• Add token in /upload interface.
• Add support for args with spaces.
• Error code optimization to identify failed submission exceptions.
• Adding project-level configuration templates.

Abbreviations:

• DSS: DataSphereStudio

New Feature

• #103 [streamis-projectmanager] Adding project-level configuration templates

• #107 [streamis-jobmanager] add hook before flink apps stop

• #108 [streamis-log] Add heartbeat for callback log

• #109 [streamis-higiavailablemanager] Add cluster high availability query interface

• #110 [streamis-jobmanager] Adjust the default snapshot root address based on the high availability policy

• #112 [streamis-projectmanager] Add audit log recording to Streamis

• #113 [streamis-jobmanager] Supports viewing and downloading of meta.json of existing execution history tasks

Enhancement

• #101 [streamis-jobmanager] next page button should be remained in full screen mode

• #100 [streamis-joblauncher] streamis flink task should load local 'env.java.opts' config in flink-config.yaml

• #104 [streamis-joblauncher] Error code optimization to identify failed submission exceptions

• #105 [streamis-joblauncher] Add support for args with spaces

• #106 [streamis-projectmanager] Add token in /upload interface

• #111 [streamis-jobmanager] Support batch enable and disable jobs

• #114 [streamis-jobmanager] Support job-level resource downloading

Bug Fixes

• #98 [streamis-projectmanager] upload big project resource files may cause oom

Credits

The release of Streamis 0.3.1 is inseparable from the contributors of the Streamis community, thanks to all community contributors.

Streamis 0.3.0 Release

Streamis 0.3.0 includes all of Project Streamis-0.3.0.

Streamis 0.3.0 has some important architecture updates and optimizations, including Separate Flink application support and Flink management engine connectors, and adds checks for snapshots and applications with the same name on yarn before starting jobs. Streamis 0.3.0 is compatible with multiple versions of Linkis (>= 1.1.2), but some related features require 1.4.0 or later.
Streamis 0.3.0 mainly adds the following features:

• Added separate management mode for Flink tasks. (requires Linkis 1.4.0)
• Add Flink management ec support. (requires Linkis 1.4.0)
• Add startup check for job task startup, check the snapshot address and the application with the same name on yarn. (requires Linkis 1.4.0)
• Support for DSS proxy users. (requires DSS 1.1.2)
• Collect and aggregate Yarn flow application log callbacks, and write them locally on the streamis side. The collection terminal supports filtering callback logs according to different levels

Abbreviations:

• DSS: DataSphereStudio

New Feature

• Streamis-74 [Streamis-jobmanager] Flink tasks add separate management mode
• Streamis-75 [Streamis-jobmanager] Add flink management ec support
• Streamis-77 [Streamis-jobmanager] Streamis job task startup adds startup check
• Streamis-73 [Streamis-jobmanager] When starting a job, it will automatically check the application with the same name on yarn
• Streamis-76 [Streamis-jobmanager] Streamis application parameter configuration import
• Streamis-95 [Streamis-jobmanager] collects and aggregates streaming application logs, and writes them locally on the streamis side
• Streamis-84 [Streamis-appconn] Streamis data interface permission adjustment
• Streamis-81 [Streamis-appconn] Cooperate with DSS to switch proxy users
• Streamis-68 [Streamis-jobmanager] jobs support label filtering

Enhancement

• Streamis-88 [Streamis-Web] Rendering and interface adjustment of task configuration interface
• Streamis-89 [Streamis-jobmanager] Refactor the automatic restart of streaming applications and optimize the idempotent problem
• Streamis-87 [Streamis-jobmanager] Streamis task abort process adjustment, optimized job stop function, added to scheduling
• Streamis-80 [Streamis-jobmanager] Added keyword filtering strategy on the collection side
• Streamis-78 [Streamis-jobmanager] Application initialization supports application online import and version management
• Streamis-71 [Streamis-jobmanager] Check the configuration of persion on task failure before the task starts
• Streamis-79 [Streamis-jobmanager] supports multiple local drivers to store streaming application logs
• Streamis-72 [Streamis-projectmanager] Upload project resource files to add md5

Bugs Fix

• Streamis-96 [Streamis-Web] Fix the problem that the next page cannot be displayed on the historical task log page, and display the flink ec paged log on the log interface
• Streamis-97 [Streamis] Code scanning bugs and smells fixed

Credits

The release of Streamis 0.3.0 is inseparable from the contributors of the Streamis community, thanks to all community contributors.

Compiled Version

Streamis0.3.0 installation package: click me to download

Streamis0.3.0 appconn plugin package: click me to download

DSS1.1.2&Linkis1.4.0 one-click installation package: click me to download

Streamis 0.2.0 Release

Streamis-0.2.0 includes all of Project Streamis-0.2.0.
This release mainly add the architecture optimization and tuning, provide some important features, including Streaming job state(checkpoint and savepoint) management which enable user to do a snapshot for Flink application and restore state from it, New Streaming job configuration module which supports to add or delete configuration item flexibly, Streamis AppConn, Streaming job management in batch(such as start and stop in batch) and so on. Streamis 0.2.0 support multiple Linkis versions (>= 1.1.1)，but some features require version greater than 1.1.2.
The following key features have been added:

• Provided AppConn for Streamis to synchronize project basic information and privilege relations from DataSphereStudio.
• Provided strategy to obtain or create checkpoint/savepoint file information of Flink application through Linkis server. (require Linkis 1.1.2)
• Provided strategy to restart the streaming job automatically and resolve the idempotent problem.
• Provided the tenancy scheduler module to manage the lifecycle of streaming jobs in batch.
• Provided the method to set the custom Flink configuration variables. (require Linkis 1.1.2)
• Provided the interface to view the log of yarn container through Linkis server. (require Linkis 1.1.2)

Abbreviations:

• SJL: Stream Job Launcher Module
• SJDA: Stream Job Deploy API Module
• SJHV: Stream Job History And Version Module
• SBO: Stream Bulk Operation Module
• SA: Stream AppConn Module

Features/Enhancement of version has been rolled up one pull request . [Streamis-28]

New Features

• SJL: Add the JobStateManager module to provide JobStateFetcher to obtain job state info (checkpoint/savepoint).[Streamis-23]
• SJL: Automatically restart failed tasks(related job) and automatically restore its state.[Streamis-22]
• SBO: Job batch operation collection. [Streamis-19]
• SJDA: Enable to fetch yarn application logs (should dependency on linkis version higher than 1.1.2). [Streamis-27]
• SA: Streamis AppConn to sync the project and privilege information (should dependency on dss version higher than 1.1.0). [Streamis-24]

Enhancement

• SJHV: Refactor the job configuration module and provide a new interface for job configuration acquisition and addition. [Streamis-21]
• SJDA: Provide two methods to stop job: "stop directly" and "snapshot and stop". [Streamis-20]

Bugs Fix

• SJDA: Fix the problem that exception in streaming job, but cannot view the error log. [Streamis-17]

Credits

   The release of Streamis 0.2.0 is inseparable from the contributors of the Streamis community. Thanks to all the community contributors!

Compiled Version

1. Tencent Cloud:
   wedatasphere-streamis-0.2.0-dist.tar.gz.
   streamis-0.2.0-dist.zip

Streamis 0.1.0 release

   Streamis-0.1.0 includes all of Project Streamis-0.1.0.

   This is the initial release of Streamis, which is an jointed development project, established by WeBank, CtYun, Samoyed Financial Cloud and XianWeng Technology.

   As a streaming application development and management system, Streamis allows users to complete the development, debugging, release and production management of streaming applications at low cost.

   The following key features are added in this release:

• Powerful streaming application development and debugging capabilities.
• Powerful streaming application production center capabilities.
• Supports multi-version management of Streamis jobs.
• Supports full life cycle management of Streamis jobs, such as start, stop, restart, etc.
• Supports monitoring alarm of Streamis jobs.
• Supports checkpoint and savepoint management capabilities of Streamis jobs.
• The application development layer of Streamis is organically integrated with the data application development process of DSS, providing simplier user experience.
• The application execution layer of Streamis integrates Linkis to submit streaming applications to Flink, the underlying computation engine integrated by Linkis.

   Welcome to download and use Streamis!

Credits

   The release of Streamis 0.1.0 is inseparable from the contributors of the Streamis community. Thanks to all the community contributors!

Compiled Version

1. Tencent Cloud:

Streamis-0.1.0 Compiled package (.tar.gz)