Skip to main content

La donnée intelligente

Search This Blog

Quelques datasets

on December 09, 2016

Get link
Facebook
X
Pinterest
Email
Other Apps

Je reprends ici les liens fournis par Data Science Central vers des datasets intéressants

Big data sets available for free
Great Github list of public data sets
10 Great Healthcare Data Sets
Password and hijacked email dataset
20 Big Data Repositories You Should Check Out
Top 20 Open Data sources
Great IoT, Sensor and other Data Sets Repositories
The Free 'Big Data' Sources Everyone Should Know
100+ Interesting Data Sets for Data Science
Data sets and other machine learning resources from UC Irvine
Big data set - 3.5 billion web pages
The Free 'Big Data' Sources Everyone Should Know
Another large data set - 250 million data points
Two big datasets to challenge your data science expertise
13 Machine Learning Data Set Collections
Datasets for queueing modelling
Kaggle Releases Data Sets About Global Warming
Facebook Shares Large Data Sets to Help Improve its AI
DSC data sets

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

Post a Comment

Popular Posts

Dive Deeper into Data Engineering on Databricks

Spark Structured Streaming with Kafka: Understanding the startingOffset=“earliest” Issue

Spark Structured Streaming with Kafka: Understanding the startingOffset=“earliest” Issue Spark Structured Streaming is a powerful tool for processing large volumes of data in real-time. However, many data scientists have encountered an issue where the startingOffset="earliest" setting is not honored when using Kafka as a source. This blog post will delve into this issue, providing a comprehensive understanding and potential solutions. Understanding the Issue When using Spark Structured Streaming with Kafka, the startingOffset option allows you to specify where Spark should start reading data. The two options are "latest", which reads only new data, and "earliest", which reads from the beginning of the topic. However, some users have reported that Spark does not always honor the "earliest" setting. Instead of starting from the beginning, Spark starts reading from the latest offset, leading to data loss. Read more : https://saturncloud.io...

Six point checklist for Spark job optimization

Six point checklist for Spark job optimization 1. Use the correct file format If you are dealing with tabular data, you should choose the Apache Parquet format. The format is optimized for queries ( transformations ) on a subset of columns and on a large dataframe . If Spark is being used with Databricks , another particularly interesting format is the delta format which offers automatic optimization tools. 2. Maximize parallelism by splitting data using partitions To read and process data in parallel, we need to split data into multiple partitions. Spark organizes one thread per task and per CPU core. Each task is related to a single partition. (Configure a number of partitions at least as large as the number of available CPU cores) Goal is to split Spark job stages into number of tasks. How to create partitions? 3. Handle shuffle operations Shuffle partitions are special kinds of partitions which are created during stages of jobs involving a shuffle [wide tra...

Powered by Blogger