User Tools

Site Tools


proj:gatd:formatter:start

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

proj:gatd:formatter:start [2012/11/26 18:47] (current)
Line 1: Line 1:
 +====== Get All The Data Formatter ======
 +
 +The formatter does quite a bit of work in the middle of the system. It's primary function is to take the incoming packets and any required static information,​ and parse it into a form that can be streamed to any receivers and stored in the database.
 +
 +
 +===== Parsers =====
 +
 +Parsers must be written for each data source and type being sent to GATD. Parsers are python classes that are used on each packet as they come through the system.
 +
 +The following is a template for what a parser must be:
 +<code python>
 +class yourParser (parser):
 +  def __init__():
 +    # you can create any state or whatnot
 +  ​
 +  ​
 +  ​
 +  # parse() get's called for each incoming packet.
 +  #
 +  # data  :    The raw string from the packet sent to the receiver.
 +  # meta  :    Extra data related to the packet being parsed.
 +  #            type: dict
 +  #             ​['​addr'​] = ipv6 address of the sender
 +  #             ​['​port'​] = port of the sender socket
 +  #             ​['​time'​] = unix timestamp of when the receiver received the packet
 +  # extra :    Static data associated with this parser/data source. This data is loaded
 +  #            beforehand with this parser. An example of where this would be useful
 +  #            is with location data. In theory the location of a sensor is known
 +  #            ahead of time and the parser can then add it to the record so the
 +  #            location is set in the database.
 +  # settings : This dictionary contains the settings info from the config file.
 +  #
 +  # return : This function must return a dict. The key, value pairs will be stored
 +  #          in the database as key value pairs.
 +  def parse (self, data, meta, extra, settings):
 +    ​
 +</​code>​
 +
 +===== Config Files =====
 +
 +Config files let the formatter register a parser. The basic format is:
 +
 +<​code>​
 +[main]
 +name: <Name for the parser>
 +parser: <name of the python class of this parser>
 +access: [public|private]
 +</​code>​
 +
 +If ''​access''​ is set to private, then only the owner of the data can retreive the data from the system.
 +
 +===== Formatter Exceptions =====
 +
 +  * BadPacket : The packet from the receiver is not valid. Most likely the source (aka UDP or TCP, etc.) is unknown.
 +  * ParserError : The parser for a packet type had some error.
 +  * ParserNotFound : There is no registered parser for this packet.
 +
  
proj/gatd/formatter/start.txt · Last modified: 2012/11/26 18:47 (external edit)