docker-hive

Docker build for hive 3.1.0 based on fredrikhgrelland/hadoop and tested with vagrant-hashistack

This image is built to work with object storage, only. Hive metastore requires connection to a postgres database.

The image can take role as both metastore and hiveserver and is based on the apache hive release bundle. NOTE: This image is ment to be used as a metastore and hiveserver DDL broker only. It has not been tested for carrying any load. This might work but it is currently missing tez binaries and it will probably be dead-slow. You may use this image as a metastore for presto, and as a HQL-endpoint for NIFI.

Note on versions

Compatiability of hadoop and hive is ensured by using a known working combination. See Hortonworks ( or cloudera ) compatibility in releasenotes. https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html

Published images

• dockerhub
• github

Prerequisites

Vagrant is uses for local run. You might need to install virtualbox and vagrant.

make prereq

Build locally for development

make build

This image can be built and operated behind a corporate proxy where the base os needs to trust a custom CA. See this

While building locally using the Makefile, you may set the environment variable CUSTOM_CA to a file or directory in order to import them. CUSTOM_CA=/usr/local/share/ca-certificates make

Run locally

Image could be run locally using vagrant box fredrikhgrelland/hashistack.

Prerequisite is to have vagrant Prerequisite is to have vagrant and virtualbox

// usual run
make up

// test
make test

Box runs HashiCorp products: Consul, Nomad, Vault, etc...

Hive with required dependencies(database, s3) will be deployed on Nomad as a docker container.

Stack:

• MinIO
• Postgres
• Hive - metastore mode
• Hive - server mode

NB: Nomad jobs are configured to run with consul-connect integration (service mesh).

Examples cli

To access hive-server or meta-store in nomad, go to http://localhost:4646, chose hive job and task:

hive metastore (connection)

beeline -u jdbc:hive2://

hive server (connection)

beeline -u "jdbc:hive2://localhost:10000/default;auth=noSasl" -n hive -p hive

beeline cli

SHOW DATABASES;
SHOW TABLES IN <database-name>;
SELECT * FROM <database-name>.<table-name>;

Credits:

Influenced by BDE