How To Break DAG Lineage in Apache Spark — 3 Methods
In this post, I will introduce you to 3 methods how you can Brak DAG Lineage in Apache Spark. It’s very possible that 1 of them you weren’t even aware of! Check if you know all 3 methods which, depending on the conditions and requirements, can save you a lot of time!
- What is DAG?
- Apache Spark Break DAG Lineage
- Why do we need to break DAG Lineage?
- Where to see the DAG graph?
- How do break DAG Lineage?
- #1: Checkpoint
- #2: LocalCheckpoint
- #3: ReCreate DataFrame / DataSet
Read more: https://medium.com/@bigdataetlcom/how-to-break-dag-lineage-in-apache-spark-3-methods-6d53f9cc1dab
Credits: Big Data Etls in Medium.com
Photo by <a href="https://unsplash.com/@conti_photos?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Fabrizio Conti</a> on <a href="https://unsplash.com/s/photos/mountain-climbing?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
Post a Comment