La donnée intelligente

Search This Blog

How to Easily Test Spark DataFrame Transformations

on November 30, 2021

Get link
Facebook
X
Pinterest
Email
Other Apps

Read here the full Medium post: Spark Scala Test

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

Dive Deeper into Data Engineering on Databricks

Spark Structured Streaming with Kafka: Understanding the startingOffset=“earliest” Issue

Spark Structured Streaming with Kafka: Understanding the startingOffset=“earliest” Issue Spark Structured Streaming is a powerful tool for processing large volumes of data in real-time. However, many data scientists have encountered an issue where the startingOffset="earliest" setting is not honored when using Kafka as a source. This blog post will delve into this issue, providing a comprehensive understanding and potential solutions. Understanding the Issue When using Spark Structured Streaming with Kafka, the startingOffset option allows you to specify where Spark should start reading data. The two options are "latest", which reads only new data, and "earliest", which reads from the beginning of the topic. However, some users have reported that Spark does not always honor the "earliest" setting. Instead of starting from the beginning, Spark starts reading from the latest offset, leading to data loss. Read more : https://saturncloud.io...

Six point checklist for Spark job optimization

Six point checklist for Spark job optimization 1. Use the correct file format If you are dealing with tabular data, you should choose the Apache Parquet format. The format is optimized for queries ( transformations ) on a subset of columns and on a large dataframe . If Spark is being used with Databricks , another particularly interesting format is the delta format which offers automatic optimization tools. 2. Maximize parallelism by splitting data using partitions To read and process data in parallel, we need to split data into multiple partitions. Spark organizes one thread per task and per CPU core. Each task is related to a single partition. (Configure a number of partitions at least as large as the number of available CPU cores) Goal is to split Spark job stages into number of tasks. How to create partitions? 3. Handle shuffle operations Shuffle partitions are special kinds of partitions which are created during stages of jobs involving a shuffle [wide tra...

Powered by Blogger

How to Easily Test Spark DataFrame Transformations

Comments

Post a Comment