La donnée intelligente

Search This Blog

big data: une petit infographie qui date un peu mais toujours intéressante

on December 22, 2015

Get link
Facebook
X
Pinterest
Email
Other Apps

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

Spark Structured Streaming with Kafka: Understanding the startingOffset=“earliest” Issue

Spark Structured Streaming with Kafka: Understanding the startingOffset=“earliest” Issue Spark Structured Streaming is a powerful tool for processing large volumes of data in real-time. However, many data scientists have encountered an issue where the startingOffset="earliest" setting is not honored when using Kafka as a source. This blog post will delve into this issue, providing a comprehensive understanding and potential solutions. Understanding the Issue When using Spark Structured Streaming with Kafka, the startingOffset option allows you to specify where Spark should start reading data. The two options are "latest", which reads only new data, and "earliest", which reads from the beginning of the topic. However, some users have reported that Spark does not always honor the "earliest" setting. Instead of starting from the beginning, Spark starts reading from the latest offset, leading to data loss. Read more : https://saturncloud.io...

Can we really avoid vendor lock using Cloud & Platform for Data Engineering ?

https://open.substack.com/pub/performancede/p/avoiding-cloud-and-platform-lock?r=o2x0y&utm_campaign=post&utm_medium=web

Dive Deeper into Data Engineering on Databricks

Powered by Blogger

big data: une petit infographie qui date un peu mais toujours intéressante

Comments

Post a Comment