Read Data from File

Log Agent reads files data from remote data sources and ingests the data into Kafka or RabbitMQ channels.

Log Agent is installed on a remote machine and has three java processes running internally, LogAgentLogThread, and Main. LogAgent process reads system logs, whereas LogThread and Main processes read all other types of logs.

Steps to install Log Agent on a machine

  • Extract StreamAnalytix bundle, and configure agent.yaml file located at /StreamAnalytix/conf/yaml.
  • Extract logstash tar located at /StreamAnalytix/lib folder, and configure property logstash.bin.script.location in agent.yaml file.
  • Run the log agent using following commands:

$ sh logAgent.sh install

$ sh logAgent.sh start

Note: Install command is required only for the first time when Log Agent is to be installed, after that you need to run only start command.

Agent Configuration

Kafka

On the Agent Configuration page,  select message broker  as Kafka and provide the path from where to read log files to configure a log agent.

Field Description
Message Name of the message to be used in the pipeline
Message Broker Kafka
Connection Name Select a connection for the message broker
Topic Name Message broker topic name
Replication Factor Number of replications
Partitions Number of partitions
Directory or File Path Path from where to read logs. It is the path of directory on the server, for ex:

On Windows: c:\apache\apache.error.log,

On UNIX: /var/log/foo.log

Read Log File Whether log files should be read from the beginning to the end or from end  to the beginning. Beginning and End options specifies reading point of a new file. This option is applicable only for the first read operation. For successive read operations data will be fetched from the end of the last read
Encrypt Message If Yes, messages are encrypted before sending it to the broker
Apply Filter Filter messages for processing based on a given criteria

 

RabbitMQ

On the Agent Configuration page,  select message broker  as RabbitMQ and provide the path from where to read log files to configure a log agent.

Field Description
Message Name of the message to be used in the pipeline.
Message Broker Select  Message Broker as RabbitMQ.
Connection Name Select a connection for the message broker.
Exchange Name Is the Exchange Name where data is pushed.
Exchange Type Type of exchange where data is pushed.
Exchange Durable If RabbitMQ is selected as broker, and if this property is set to True then the exchange will not be deleted if you restart RabbitMQ .
Routing Key Routing key helps in routing specific data  to specific queue  based on binding using this key.
Queue Name Name of the queue where data is stored.
Queue Durable If this property is set to True then the exchange will not be deleted if you restart RabbitMQ
Add Config Queue Additional RabbitMQ properties can be configured using Add Config Queue.
Directory or File Path Path from where to read logs. It is the path of directory on the server, for example,

On Windows: c:\apache\apache.error.log,

On UNIX: /var/log/foo.log

Read Log File Whether log files should be read from the beginning to the end or from end to the beginning. Beginning and End options specifies reading point of a new file. This option is applicable only for the first read operation. For successive read operations data will be fetched from the end of the last read
Encrypt Message If Yes, messages are encrypted before sending it to the broker
Apply Filter Filter messages for processing based on a given criteria
Schedule a Demo