Shark Release 0.2

Shark 0.2 is the first Shark release since the original 0.1 prototype release. The new version brings new features, performance improvements, and stability to Shark. Major changes are documented below:

Hive Compatibility

• We have upgraded Shark to work with Hive 0.9, which introduces numerous features over the original Hive 0.7.
• Hive UDFs and UDAFs are fully supported now.
• Shark 0.2 also supports distributing resource files (e.g. jars) to the slaves using Hive's ADD FILE command.

Simpler Deployment

• We have significantly simplified the deployment process.
• For example, Running Shark Locally contains a guide to get Shark 0.2 up and running locally in ~ 5 mins.
• In addition to running on Mesos, Shark now supports Spark's standalone deploy mode that lets you quickly launch a cluster without installing an external cluster manager. The standalone mode only needs Java installed on each machine, and Spark deployed to it.

Hive Thrift Server

• Ram Sriharsha from Yahoo contributed a patch for the Shark Thrift server, which is compatible with Hive's Thrift server.
• The Thrift server starts a long-running server and support multiple clients connecting to it. These clients can access the same warehouse, using the same set of cached tables.
• To start the server on the default 10000 port, do bin/shark --service sharkserver

Query Execution and Performance Improvements

• Map side aggregation is now turned on by default, and if not enough reduction is observed, Shark will turn the map side aggregation off automatically. The user no longer needs to explitictly set hive.map.aggr.
• We have rewritten Shark's join and group by code. For queries that have a large number of distinct keys, join and group by performance can increase by 2X.

Spark Compatibility

• Shark 0.2 requires Spark 0.6 as it takes advantage of the new features and performance improvements from the new Spark release.

Miscellaneous

• If you feel _cached is a hacky way to indicate whether a table should be cached in memory, Shark 0.2 supports specifying the boolean flag using table properties when the table is created. For example

  CREATE TABLE myTable TBLPROPERTIES ("shark.cache" = "true") AS SELECT * FROM myInput;

Credits

Shark 0.2 was the work of a large set of new contributors from Berkeley and outside.

• Ram Sriharsha from Yahoo contributed a patch for the Shark Thrift server.
• Harvey Feng contributed the Hive 0.9 upgrade and improved map join implementation.
• Antonio Lupher contributed the map side aggregation tuning implementation.
• Denny Britz contributed support for ADD FILE and UDF/UDAF dynamic class loading.
• Patrick Wendell contributed the revamped documentation and extensive testing.
• Paul Ruan helped with testing.