Custom Processor

Custom processor allows you to implement your custom logic in a pipeline. You can write your own custom code to ingest data from any data source and build it as a custom channel. You can use in your pipelines or even share it with other workspace users.

It is a processor available for you to create a custom business logic. Implements the ‘com.streamanalytix.framework.api.processor.JSONProcessor interface and provides custom business logic in implemented methods.

Pre-requisites – Create Custom Code Jar

To use a custom processor, first you need to create a jar file containing your custom code and then upload the jar file in a pipeline or as a registered component.

To write a custom logic for your custom channel, download the Sample Project.

Import the downloaded Sample project as a maven project in Eclipse. Ensure that Apache Maven is installed on your machine and that the PATH for the same is set.

Implement your custom code. Build the project, to create a jar file containing your code, using the following command: mvn clean install –DskipTests.

If the maven build is successful, upload the jar on the pipeline canvas.

Custom Processor for Spark

Custom processor for Spark supports writing code in Java and Scala programming languages. It provides two types of operations: Custom and Inline as Scala.

The Custom operation allows you to write the code in Java language whereas Inline as Scala operation allows you to write the code in Scala programming language.

The Inline as Scala is an enhancement over the Custom operation as it enables you to write the entire scala code in a processor itself and if the build is successful, code can be used in the pipeline.

Custom Code for your sample Project

You have to implement ‘com.streamanalytix.framework.api.processor.JSONProcessor interface and provide the custom business logic in implemented methods while using a custom processor in pipeline.

There are three methods for you to implement.

  1. Init: Allows you to enter any initialization calls.
  2. Process: Contains actual business logic and this method is called for each tuple.
  3. Cleanup: All resource cleanup occurs in this method.

Configure Custom Processor

To add a Custom Processor into your Spark pipeline, drag the custom processor added in the pipeline or available as a registered component on the canvas, and right click on it to configure as explained below.

The operation can be one of the two types: Custom or Inline as Scala.

  • Custom: Select the option as Custom if you wish to write custom business logic.
  • Inline as Scala:  Select the option as Inline as Scala if you wish to write the code in Scala language.

Option 1: selected Operation as Custom

Option 2: selected Operation as Inline as Scala

Configure Custom Processor for Storm

You have to implement ‘com.streamanalytix.framework.api.processor.JSONProcessor interface and provide the custom business logic in implemented methods while using a custom processor in pipeline.

To add a Custom Processor into your Strom pipeline, drag the custom processor added in the pipeline or available as a registered component on the canvas, and right click on it to configure as explained below.

 

Schedule a Demo