How To Break DAG Lineage in Apache Spark — 3 Methods

In this post, I will introduce you to 3 methods how you can Brak DAG Lineage in Apache Spark. It’s very possible that 1 of them you weren’t even aware of! Check if you know all 3 methods which, depending on the conditions and requirements, can save you a lot of time! 

• What is DAG?
• Apache Spark Break DAG Lineage
• Why do we need to break DAG Lineage?
• Where to see the DAG graph?
• How do break DAG Lineage?
• #1: Checkpoint
• #2: LocalCheckpoint
• #3: ReCreate DataFrame / DataSet

Read more: https://medium.com/@bigdataetlcom/how-to-break-dag-lineage-in-apache-spark-3-methods-6d53f9cc1dab

Credits: Big Data Etls in Medium.com

Photo by <a href="https://unsplash.com/@conti_photos?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Fabrizio Conti</a> on <a href="https://unsplash.com/s/photos/mountain-climbing?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>