La donnée intelligente

Search This Blog

How to Easily Test Spark DataFrame Transformations

on November 30, 2021

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Read here the full Medium post: Spark Scala Test

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Comments

Six point checklist for Spark job optimization

Six point checklist for Spark job optimization 1. Use the correct file format If you are dealing with tabular data, you should choose the Apache Parquet format. The format is optimized for queries ( transformations ) on a subset of columns and on a large dataframe . If Spark is being used with Databricks , another particularly interesting format is the delta format which offers automatic optimization tools. 2. Maximize parallelism by splitting data using partitions To read and process data in parallel, we need to split data into multiple partitions. Spark organizes one thread per task and per CPU core. Each task is related to a single partition. (Configure a number of partitions at least as large as the number of available CPU cores) Goal is to split Spark job stages into number of tasks. How to create partitions? 3. Handle shuffle operations Shuffle partitions are special kinds of partitions which are created during stages of jobs involving a shuffle [wide transformations —

How we saved 90% of costs by moving from AWS Lambda to AWS Fargate

How we saved 90% of costs by moving from AWS Lambda to AWS Fargate AWS Fargate is a technology that you can use with Amazon ECS to run containers without managing servers or clusters of Amazon EC2 instances as a must. With Fargate, you no longer have to provision, configure, or scale clusters of virtual machines to run containers. This removes the need to choose server types, decide when to scale your clusters, or optimize cluster packing. AS IS TO BE Although replacing API Gateway with Application Load Balancer as a Lambda Trigger seems to be a quick solution, it does not solve our throttling problem. That’s why we’ve started to looking another solution in terms of cost, performance, limitations, and scalability. There are lots of solutions for implementation to cover all the things I mentioned earlier, but since this is not a comparison post about solutions, I will skip that part. After lots of research and long discussions, we’ve decided to move on with AW

What are LLMs, and how are they used in generative AI?

What are LLMs, and how are they used in generative AI? Large language models are the algorithmic basis for chatbots like OpenAI's ChatGPT and Google's Bard. The technology is tied back to billions — even trillions — of parameters that can make them both inaccurate and non-specific for vertical industry use. Here's what LLMs are and how they work. By Lucas Mearian Read more Photo de Alina Grubnyak sur Unsplash

Powered by Blogger

How to Easily Test Spark DataFrame Transformations

Comments

Post a Comment