The formatter does quite a bit of work in the middle of the system. It's primary function is to take the incoming packets and any required static information, and parse it into a form that can be streamed to any receivers and stored in the database.
Parsers must be written for each data source and type being sent to GATD. Parsers are python classes that are used on each packet as they come through the system.
The following is a template for what a parser must be:
class yourParser (parser): def __init__(): # you can create any state or whatnot # parse() get's called for each incoming packet. # # data : The raw string from the packet sent to the receiver. # meta : Extra data related to the packet being parsed. # type: dict # ['addr'] = ipv6 address of the sender # ['port'] = port of the sender socket # ['time'] = unix timestamp of when the receiver received the packet # extra : Static data associated with this parser/data source. This data is loaded # beforehand with this parser. An example of where this would be useful # is with location data. In theory the location of a sensor is known # ahead of time and the parser can then add it to the record so the # location is set in the database. # settings : This dictionary contains the settings info from the config file. # # return : This function must return a dict. The key, value pairs will be stored # in the database as key value pairs. def parse (self, data, meta, extra, settings):
Config files let the formatter register a parser. The basic format is:
[main] name: <Name for the parser> parser: <name of the python class of this parser> access: [public|private]
access is set to private, then only the owner of the data can retreive the data from the system.