More on Data Types

While configuring a new message, you need to define the Message Parser Type. Supported Parser Types are Delimited, JSON, Regular Expression, and Custom.

Delimited

You can use any of the below delimiters for the Delimited parser type:

  • ‘;’ semi colon
  • ‘,’ Comma
  • ‘:’ Colon
  • ‘\t’ Tab
  • ‘|’ Pipe

 

Regular Expression

Regular Expression parser type allows you to construct a regex pattern for the message.

If the message contains multiple lines, select Multiline Required to define the message format. You can choose to either negate the line breaks or provide a regular expression to identify a complete message in the multiline message log.

Next step is to validate the Regular Expression. Configure a message field in the Field Configuration tab and click on the Validation button.

This will bring the RegEx validation screen as shown below:

To create a RegEx pattern for a field, click on the RegEx textbox. This will bring a new window of pre-defined patterns, and you can construct a new regular expression or select from the pre-defined patterns.

While constructing the field, you can define a delimiter out of Prefix or Postfix. A matched RegEx string appears automatically on entering the values in the fields ‘Sample Log Message’, ‘Delimiters’ and ‘RegEx’. If correct, RegEx string turns Green in color. Click on the Validate button to verify the regex pattern.

Custom Parser

Custom Message Parser allows you to define a custom parser type, for example, you can write a method to parse data from XML to JSON. To create a custom parser type, define a class that implements the interface “com.streamanalytix.model.parser.ICustomParser” and you can write your custom code inside the methods declared in it. Provide the fully qualified name of the custom class in the configuration settings of the Custom Channel. An example of a class name can be “com.streamanalytix.parser.TestParser”.

Multi-level JSON

For multi-level JSON messages, Field Name for the child will be prefixed by parent’s field name followed by JSON_MULTI_LEVEL_SYMBOL $$. See example below:

Sample JSON Field Name Data Type
{ address: {
city: New York,
country: USA}
}
Address Text
address$$city Text
address$$country Text

AVRO

The Avro-Schema defines the message-fields configured on the Field Configuration screen.

Following Avro Data Types are supported in StreamAnalytix:

Primitive Data Types

Avro Data Type Supported in StreamAnalytix Mapping to StreamAnalytix data-type
null Yes Text
int Yes Number
long Yes Number
float Yes Number
double Yes Number
bytes Yes Text
string Yes Text

Complex Data Types

Avro Data Type Supported in StreamAnalytix Remarks
Record Yes
Enum Yes – as Text Allows schema to be parsed and validate data as per Schema
Arrays No
Maps No
Unions Yes Support union with null; will choose the non-null data-type
Fixed Yes – as Text We are typecasting everything as UTF-8
Schedule a Demo