Hyper-Scale Data Processing and Storage - StreamAnalytix Case Study

Case Study

Hyper-Scale Data Processing and Storage Using StreamAnalytix

Enterprises generally need to compromise with running and maintaining multiple batch processes on the accumulated data due to throughput and management constraints. The overall business process turnaround can be improved if the data is made available after processing in real-time. A real-time system of such a large scale requires easy provisioning and monitoring.

Challenges

We developed a Proof of Concept (PoC) for financial service companies who face challenges in processing/storing large amounts of streaming data in real-time. The use case was applied to stock ticker data being streamed from over 80 different global exchanges and various sources with the following goals:

  • Receive simulated tick data from 80+ stock exchanges across the globe
  • High data ingestion rate of the incoming ticks
  • Real-time analytics on high volumes of data
  • Easy installation, configuration, and monitoring of required infrastructure
  • Real-time dashboard showing key statistics of all components and ingestion rates

Solution

The solution of the Stock Ticker PoC was divided in six segments:

  • Data generator: A multi-threaded process that can generate mocked-up stock tick data from multiple exchanges around the globe, with flexibility for increasing/decreasing data velocity
  • Real-time data processing and storage: A parallel processing engine, which can process high volumes of stock tick data and publish it to the UI in real-time with high data processing rate
  • Data store: A data store that stores and makes available huge volume of incoming stock tick data in real-time
  • Reporting on historical data: A reporting tool that can generate reports on the historical data, which can be viewed on the UI for analysis and decision-making
  • User Interface (UI): A UI that can handle huge real-time data and is capable of showing it as graphs in real-time
  • Provisioning and monitoring tool for cluster management

Results

  • Real-time data processing on hyper scale and storage
  • Seamless provisioning, management, and monitoring of clusters
  • Visualization of real-time graphs and reports on historical data

You may also be interested in…

 

Case Study

Real-Time Driver Profiling & Risk Assessment for Usage-based Insurance with StreamAnalytix

To keep up with the new digital consumer and remain competitive, the auto insurance industry is increasingly investing in connected…

Case Study

Real-time insider threat detection using machine learning for a Fortune 500 bank

Learn how a large US-based bank used predictive analytics and machine learning to identify and prevent insider information security threats…

Case Study

A leading Contact Centre builds a Real-Time Call Center Monitoring Solution with StreamAnalytix

A leading cloud-based communications technology company that offers hosted contact center services needed a way to improve performance metrics, eliminate…

White Paper

Harnessing IoT data in an always-connected world

As the Internet of Things (IoT) generates incessant data, organizations need smarter and more efficient ways to manage and process…

webinar

Streaming Analytics for IoT with Apache Spark

Modern IoT operations can drive digital transformation by analyzing the unprecedented amounts of data generated from devices and sensors in…

blog

Real-time analysis of weather impact on New York City taxi trips in minutes using StreamAnalytix

In this post, we will see how easy it is read data from a streaming source, apply data transformations, enrich…

Start your free trial

of StreamAnalytix

TRY NOW
Schedule a Demo