Skip to content

PiFlow V1.4 Release
Compare
Choose a tag to compare
@sosoll7 sosoll7 released this 24 Nov 07:33
v1.4
6fc9d37

Requirements

  • JDK 1.8
  • Scala 2.11.8
  • Spark-2.1.0、Spark-2.2.0、Spark-2.3.0（other spark version of piflow.jar should be built with code）
  • Hadoop-2.6.0 （other hadoop version of piflow.jar should be with code）
  • Hive-1.2.1（if you need to use hive，setup and modify the config.properties）

config.properties

  spark.master=yarn
  spark.deploy.mode=cluster
  
  #hdfs default file system
  fs.defaultFS=hdfs://10.0.85.83:9000
  
  #yarn resourcemanager hostname
  yarn.resourcemanager.hostname=10.0.85.83
  
  #if you want to use hive, set hive metastore uris
  #hive.metastore.uris=thrift://10.0.85.83:9083
  
  #show data in log, set 0 if you do not show the logs
  data.show=10

  #monitor the throughput of flow
  monitor.throughput=true

  #server port
  server.port=8001

  #h2db port
  h2.port=50001

Command

  ./start.sh
  ./stop.sh
  ./restart.sh
  ./status.sh
Assets 3
Loading