Kafka Spark Streaming Integration
Spark Streaming has been getting some attention lately as a real-time data
processing tool, often mentioned alongside
Apache Storm. If you ask me, no real-time data
processing tool is complete without Kafka integration (smile), hence I added an example Spark Streaming application to
kafka-storm-starter that demonstrates how to read from Kafka and write
to Kafka, using
Avro as the data format and
Twitter Bijection for handling the data serialization.
In this post I will explain this Spark Streaming example in further detail and also shed some light on the current state
of Kafka integration in Spark Streaming. All this with the disclaimer that this happens to be my first experiment with
Spark Streaming.
Read more:
https://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/
More references:
https://databricks.com/blog/2017/04/04/real-time-end-to-end-integration-with-apache-kafka-in-apache-sparks-structured-streaming.html
