Simple method for choosing the number of partitions in Spark

on

 


Simple Method to choose Number of Partitions in Spark

At the end of this article, you will able to analyze your Spark Job and identify whether you have the right configurations settings for your spark environment and whether you utilize all your resources.

Whenever you work on a spark job, you should consider 2 things.

  • Avoid Spill
  • Maximize Parallelism by utilizing all the cores.

 Read more: https://medium.com/analytics-vidhya/simple-method-to-choose-number-of-partitions-in-spark-75315c636a94

 Credit: Tharun Kumar Sekar

Photo by Fran Couto 

Comments

Post a Comment