FileCollector

FileCollector

Being a Collector means that it can receive data from a single source and then store it in some form.

This particular collector:

• Writes data received to a file

Purpose

Store data from a source in a file.

Main features

• Flushing of data from memory is based on time since last data received or amount of messages received
• By default system newline is used, but can be altered
• Multiple header lines allowed that use the set newline
• Allow rollover based on a fixed schedule and zipping of the old file
• Allow renaming based on max file size reached and optional zipping
• Execute commands on idle,max size and rollover with {path} as macro for the resulting file path

Example

  <collectors>
    <!-- Maximum options used --> 
    <file id="sensor" src="raw:test"> <!-- the id and the source of data -->
      <flush batchsize="10" age="15s"/> <!-- Write if 10 messages are received or the data is 15s old -->
      <rollover count="15" unit="min" zip="yes">HHmm</rollover> <!-- New file every 15minutes, {rollover} is replaced with calculated HHmm -->
      <maxsize zip="true">100mb</maxsize>  <!-- When an append increases the size over 100mb the file will be renamed to file.1.extension and zipped -->
      <path>csv/data_{rollover}.csv</path> <!-- file to write to with rollover -->
      <header>columnnames</header> <!-- header line -->
      <header>units</header> <!-- another header line -->
      <cmd trigger="rollover">email:admin,Rollover done,Get old file?</cmd> <!-- After the rollover was executed do this -->
      <cmd trigger="idle">email:admin,Not getting data,sensor issue?</cmd>  <!-- If a timeout was reached and the databuffer is empty, do this -->
      <cmd trigger="maxsize">email:admin,File got big!, Check {path}</cmd>  <!-- If max size limit reached, do this -->
    </file>
  </collectors>

Commands

Create/Alter the FileCollector

fc:addnew,id,src,path Create a blank filecollector with given id, source and path
fc:alter,id,param:value Alter some elements, options: eol, path, sizelimit, src

Add optional parts

fc:addrollover,id,count,unit,format,zip? Add rollover to the given fc (unit options:min,hour,day,week,month,year)
fc:addcmd,id,trigger:cmd Add a triggered command, triggers: maxsize,idle,rollover
fc:addheader,id,headerline Adds the header to the given fc
fc:addsizelimit,id,size,zip? Adds a limit of the given size with optional zipping

Example

How to use the commands to achieve the earlier given example:
fc:addnew,sensor,raw:test,csv/data_{rollover}.csv

  <collectors>
    <file id="sensor" src="raw:test">
      <path>csv/data_{rollover}.csv</path>
      <flush age="1m" batchsize="-1"/>
    </file>
  </collectors>

fc:addrollover,sensor,15,min,HHmm,yes
fc:addsizelimit,sensor,100mb,yes

  <collectors>
    <file id="sensor" src="raw:test">
      <path>csv/data_{rollover}.csv</path>
      <flush age="1m" batchsize="-1"/>
      <rollover count="15" unit="min" zip="yes">HHmm</rollover>
      <maxsize zip="true">100mb</maxsize>
    </file>
  </collectors>

fc:addheader,sensor,columnnames
fc:addcmd,sensor,rollover:email:admin,Rollover done,Get old file?

  <collectors>
    <file id="sensor" src="raw:test">
      <path>csv/data_{rollover}.csv</path>
      <flush age="1m" batchsize="-1"/>
      <rollover count="15" unit="min" zip="yes">HHmm</rollover>
      <maxsize zip="true">100mb</maxsize>
      <header>columnnames</header>
      <cmd trigger="rollover">email:admin,Rollover done,Get old file?</cmd>
    </file>
  </collectors>