Protocol buffer format

Possible data format using protocol buffers: Author: D.Coderre (Update 14.04.2014)

We can easily define multiple types of protocol buffer. Here, for example, data type:

type kodiaq_data {
	required int64 eventNumber
	required int64 timeStamp

	type module {

		required int32 moduleID

		type channel {

			required int32 channelID	
			required bytes data         // we can compress this
		}

		repeated channel       // can have multiple channels in a module
	}
	repeated module   //can have multiple modules in a ‘kodiaq_data’
}

So one data object consists of any number of modules and channels. The channels contain the data. These are all the required fields. Other optional fields can be added without breaking compatibility.

In the file the data then looks like this:

{objectType}
{objectSize}
{kodiaq_data object}
{objectType}
{objectSize}
{kodiaq_data object}
.
. 
.
etc.

objectType is an identifier for what type of protocol buffer the next data object is. Right now they are all “kodiaq_data” type, but we can add others with muon veto data or meta data or whatever. The objectSize is simply the size of the next data object in bytes.

The complete data file might then look something like this:

:: Header ::     // contains options file, trigger settings, etc.
:: muon veto event :: // doesn’t have to, just whenever one comes in time
:: acquisition monitor data ::  //does this come in between each tpc event? 
:: tpc data event ::
:: acquisition monitor data ::
:: tpc data event ::
:: acquisition monitor data ::
:: tpc data event ::
.
.
.
:: acquisition monitor data ::
:: tpc data event ::
:: acquisition monitor data ::
:: muon veto event ::
:: acquisition monitor data ::
:: tpc data event ::
:: acquisition monitor data ::
:: tpc data event ::
.
.
.
:: acquisition monitor data ::
:: tpc data event ::
:: acquisition monitor data ::
:: muon veto event ::
:: acquisition monitor data ::
:: tpc data event ::
.
.
etc. 

The acquisition monitor data contains the sum waveform. Depending on what the file size looks like we might want to think of a clever way to store this rather than having so many objects, but it is not so much data so maybe it’s OK. Each type shown is a different protocol buffer with different required variables.

Edit: 14.04.14

It might not be the best idea to use the module/channel numbering for storage. Rather the PMT channel should be the only identifier. To maintain generality, the module could be included as an optional parameter (for certain use cases where we don't have a master channel map). For example:

type kodiaq_data {
	required int64 eventNumber
	required int64 timeStamp
	type channel {
                optional int32 moduleID     // in case of running with module/channel numbering
		required int32 channelID    // could be PMT number of 0-7 if moduleID is also set
		required bytes data         // we can compress this
	}
	repeated channel       // can have multiple channels in a module
}