Historical Data Store

In order to maintain a copy of Connector data in the simplest, most future proof format, it should be stored as simple JSON on disk. This poses query-ability challenges, so at very least, a byte offset index needs to be stored along with (but separate from) it, enabling retrieval of specific ranges of data. Optionally, higher-level indicies should be available for more convenient queries (e.g. by time stamp, instead of record ID).

Thus, immutable, CR delimited, JSON objects on disk, with a simple byte-offset index and SQLite tables for serious indexing. Modifications are appended as "deletes" and "updates".

Files:

• example.json - CR delimited JSON objects (UTF-8 encoded, of course)
• example.index - Byte offsets of the end of each object
• example.db - SQLite database for indexing additional fields (such as timeStamp)

Basic Public Methods:

• addRecord - Appends a record to the .json file, adds the byte offset to the .index, and adds any indexed fields to the SQLite db.
• getAfterRecordID - Returns all objects after the given recordID in the index
• getAfterFieldsValueEquals - Returns all objects with the value of a field greater than the passed value. Only works for monotonically increasing fields.