Skip to content

Latest commit

 

History

History
203 lines (155 loc) · 7.41 KB

StreamisDeployment.md

File metadata and controls

203 lines (155 loc) · 7.41 KB
Raw

Streamis installation and deployment documentation

1. Component introduction

Streamis0.3.0 provides the Streamis-JobManager component, the role of the component is

  1. Publish streaming applications
  2. Set streaming application parameters, such as the number of Flink slots, checkpoint related parameters, etc.
  3. Manage streaming applications (e.g. start and stop)
  4. Streaming application monitoring

2. Code compilation

Streamis does not require manual compilation. You can download the installation package directly for deployment. Please click to download the installation package

If you have already obtained the installation package, you can skip this step

  • The background compilation method is as follows
cd ${STREAMIS_CODE_HOME}
mvn -N install
mvn clean install

After successful compilation, the installation package will be generated in the 'assembly/target' directory of the project

  • The front-end compilation method is as follows

Pre dependency: nodejs, python 2.0

cd ${STREAMIS_CODE_HOME}/web
npm i
npm run build

After the compilation is successful, the installation package will be generated in the ${STREAMIS_CODE_HOME}/web directory

3. Installation preparation

3.1 Basic environment installation

        The following software must be installed:

3.2 Linkis and DSS environments

  • The execution of Streamis depends on Linkis, and it needs to be version 1.4.0 and above, so you need to install Linkis above 1.4.0 and ensure that the Flink engine can be used normally.Some functions need to be supported by linkis-1.4.0.
  • Datasphere studio (> =1.1.2), the development and debugging of streaming jobs depend on DSS scriptis, and the streaming production center needs to be embedded in the DSS engineering framework system, so it depends on * * dss-1.1.2 * * and above.

Before the formal installation of streamis, please install linkis-1.4.0 and dss-1.1.2 or above, and ensure that the linkis Flink engine and DSS can be used normally. For the installation of DSS and linkis, please refer to the dss & linkis one click installation and deployment document.

How to verify that DSS and linkis are basically available? You can create a flinksql script on DSS scriptis and execute it. If flinksql can execute correctly and return the result set, it means that the DSS and linkis environments are available.

4. Installation and startup

Background installation

1.installation package preparation

Upload the installation package to the installation directory of the Linux server (currently only supports linux environment deployment), such as /appcom/install/streams, and then extract it:

cd /appcom/Install/streamis
tar -xvf wedatasphere-streamis-${streamis-version}-dist.tar.gz

2.Modify the database configuration

vi conf/db.sh
#Configure basic database information

3.Modify the basic configuration file

vi conf/config.sh
### deploy user
deployUser=hadoop

### ssh port
SSH_PORT=22

##The Max Heap size for the JVM
SERVER_HEAP_SIZE="512M"

##The Port of Streamis
STREAMIS_PORT=9400

### The install home path of STREAMIS，Must provided
STREAMIS_INSTALL_HOME=/appcom/Install/streamis

###  Linkis EUREKA  information.  # Microservices Service Registration Discovery Center
EUREKA_INSTALL_IP=127.0.0.1
EUREKA_PORT=20303

### Linkis Gateway  information
GATEWAY_INSTALL_IP=127.0.0.1
GATEWAY_PORT=9001

################### The install Configuration of all Micro-Services #####################
#
#    NOTICE:
#       1. If you just wanna try, the following micro-service configuration can be set without any settings.
#            These services will be installed by default on this machine.
#       2. In order to get the most complete enterprise-level features, we strongly recommend that you install
#          the following microservice parameters
#

STREAMIS_SERVER_INSTALL_IP=127.0.0.1
STREAMIS_SERVER_INSTALL_PORT=9400

STREAMIS_VERSION=0.3.0
STREAMIS_FILE_NAME="STREAMIS-$STREAMIS_VERSION"

4.Installation

sh bin/install.sh
  • The install.sh script will ask you if you need to initialize the database and import metadata.

     Because the user is worried that the user repeatedly executes the install.sh script to clear the user data in the database, when the install.sh is executed, the user will be asked if they need to initialize the database and import metadata.

     Yes must be selected for the first installation.

5.start up

sh bin/start.sh
  • Start verification Verification method, because Streamis and Linkis use a set of Eureka, you need to check whether the Eureka page of Linkis already contains Streamis services, as shown in the figure, components

Front-end deployment

1.Install nginx

sudo yum install -y nginx

2.Deploy the front-end package

mkdir ${STREAMIS_FRONT_PATH}
cd ${STREAMIS_FRONT_PATH}
#Place the front-end package
unzip streamis-{streamis-version}.zip

3.Modify the nginx configuration file

cd /etc/nginx/conf.d
vi streamis.conf
# Copy the following template and modify it according to the actual situation
server {
    listen 9088;# access port
    server_name localhost;
    location / {
        root ${STREAMIS_FRONT_PAH}; # Please modify it to the appropriate static file directory of Streamis
    index index.html index.html;
    }
    location /api {
    proxy_pass http://${Linkis_GATEWAY_IP}:${LINKIS_GATEWY_PORT}; #Back-end Linkis address, please modify it to the ip and port of the Linkis gateway
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header x_real_ipP $remote_addr;
    proxy_set_header remote_addr $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_connect_timeout 4s;
    proxy_read_timeout 600s;
    proxy_send_timeout 12s;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection upgrade;
    }

    #error_page 404 /404.html;
    # redirect server error pages to the static page /50x.html
    #
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
    root /usr/share/nginx/html;
    }
}

4.Load nginx configuration

sudo nginx -s reload

5. Access to DSS

If you want to use the streamis0.3.0 front end normally, you also need to install the DSS StreamisAppConn plug-in. Please refer to: StreamisAppConn plug-in installation document

6. Linkis Flink engine compilation and installation

If you want to run streamis0.3.0 normally, you also need to install the linkis Flink engine. Please refer to: linkis Flink engine installation document

7. Streamis component upgrade document / script

If you want to upgrade from a lower version of streamis to streamis0.3.0, please refer to: streamis upgrade document