This article expands on my “Hello World” for Heka blog post. Check that one out first if you are new to Heka.
In this guide we introduce using Heka over the network by utilizing two Hekad processes on localhost. For discussion purposes we name one of the Hekad processes “sender” and the other “receiver”.
- The “sender” will watch a log file and emit messages to localhost TCP port 9612.
- The “receiver” will listen on localhost TCP port 9612 and emit message payloads to file.
# watch /tmp/input.log and output to TCP port 9612 on localhost [hello_heka_input_log] type = "LogstreamerInput" log_directory = "/tmp" file_match = 'input\.log' [tcp_out:9612] type = "TcpOutput" message_matcher = "TRUE" address = "127.0.0.1:9612" #encoder = "hello_heka_output_encoder" # #[hello_heka_output_encoder] #type = "PayloadEncoder" #append_newlines = false
# listen to TCP port 9612 and emit to /tmp/output.log [tcp_in:9612] type = "TcpInput" parser_type = "message.proto" decoder = "ProtobufDecoder" address = ":9612" [hello_heka_output_log] type = "FileOutput" message_matcher = "TRUE" path = "/tmp/output.log" perm = "664" encoder = "hello_heka_output_encoder" [hello_heka_output_encoder] type = "PayloadEncoder" append_newlines = false
Now that we have config files, let us start our hekad processes, Open three terminals.
- in terminal 1:
sudo hekad -config=hello-heka-file-in-tcp-out.toml
- in terminal 2:
sudo hekad -config=/tmp/hello-heka-tcp-in-file-out.toml
- in terminal 3:
echo 'Heka, World2!' >> /tmp/input.log
- in terminal 3:
By default TCP sender encodes the message with Protobuf (ProtobufEncoder) and the TCP reciever decodes the message with Protobuf (ProtobufDecoder).
In my testing I decided to make the TCP sender use the PayloadEncoder and then instead of using a second hekad process, I used
nc -l 9612 to listen on the port. When data was added to
/tmp/input.log it showed up in the netcat terminal because hekad was watching the file and emiting just payload portion of the message to TCP 9612 which netcat was listening on. I left this configuration in the examples above, simply uncomment to reproduce.
Read this for more fun with netcat.