Streaming Configuration and Filtering - StreamAnalytix

Streaming Configuration and Filtering

Streaming Configuration

StreamAnalytix provides you multiple streaming feature for processing the data in parallel. You can configure custom streams between Channel-Processors, Channel-Emitters, Processor-Processors, or Processor-Emitters.

Right-click on the connection lines joinng the components to specify the Stream Name.

Below are the settings to configure a Stream:

  • Default: This is the default stream used by Apache Storm internally named defaultStream.
  • Custom: You can add custom Stream Name. The nomenclature supports alpha-numeric stream names which must start with an alphabet.
  • Stream Grouping:
    • Shuffle: If you have selected ‘Output Fields’ to be emitted by the channels/processor/emitter, then you can apply grouping over the emitted fields. Shuffle grouping sends the tuples to a random task.
    • Fields: A fields grouping let you group a stream by a subset of its fields. This causes equal values for that subset of fields to go to the same task every time.
    • Local/Shuffle: If the target processor has one or more tasks in the same worker process, tuples will be shuffled to just those in-process tasks. Otherwise, this acts like a normal shuffle grouping.

Message Filtering

StreamAnalytix provides you message filtering feature. You can configure a filter between Processor-Processor, Processor-Emitter or Emitter-Emitter but the feature is not supported between Channel-Processor and Channel-Emitter.

You can see the Filter tab in the configuration pop up open when right clicked on the connection between Processor-Processor, Processor-Emitter or Emitter-Emitter:

Below settings are used to filter out message fields:

  • Message Type: This allows you to select any of the configured messages.
  • Filter Rule: You can define the filter criteria using the message fields and required function from the drop down menu.
  • Negate: This feature provides a quick support to get the reverse filtering i.e. if negate is true the filtering criteria configured will be negated. Whereas if the negate value is false then the filtering criteria will be executed as it is.
  • Expression: The expression is created by providing conditions for the message fields. On the basis of sub-sections provided below Message Type, the data gets filtered. The user can add multiple conditions for any Message Type by clicking on the plus (+) icon.

Once the filter is configured, you will see a green “T” symbol on the particular connection symbolizing configured filter. If the Channel is having multi message type and you have selected more than one messages, then you can see “Add New Rule” link on the Filter tab. This enables you to configure multiple filtering criteria for multiple messages used.

Also, If you create a Sub-System with processors or emitters connected in parallel with the same Stream Name, and you configure a filter on any of the connection in parallel having the same Stream Name, then the same filter will get replicated on the other parallel connections. Although the replication of filtering criteria will not happen when the same Stream Name is used in serial connections.

Schedule a Demo