Persistence of IBS cluster / HH files with ADAM format

[davidonlaptop]

Description

This feature allow persistence of the model created in issue #7 - #1 via the --cluster option.

The Spark models should integrates with the ADAM format. New record types may need to be added to the ADAM format.

The ADAM formats can be added as a maven dependency. The structure of the ADAM format is defined in an Avro file. The Avro file can be compiled into Java classes.

For updates to the ADAM format, choose the most simple solution for you as long as it is easy to diff the changes to the Avro file.

Analysis

Add a comment to this issue with:

• the missing fields and record types in the current ADAM format

Design

Add a comment to this issue describing how this will be implemented in Spark, and explain how the persistence model will work.

Implementation

The implementation should use:

• Scala
• Spark RDD
• Avro
• Parquet