Apache Spark / Scala / Datastax Cassandra : set up development and deployment plateform
Last update: 24.10.2017
For developers working with Datastax Entreprises (Cassandra), Spark/Scala, after few tries in Scala REPL in one DSE Analytics (Spark) node, will come the question: ok now how I code my Spark application using IDE ?
Answer is not easy but I will give my findings and feedback.
Let's start from my environment:
- Windows 10 laptop with 16GB of RAM: my development workstation
- Datastax DSE 5.1.2: 3 nodes spark cluster with Spark 2.0.2.6 + Scala 2.11.11 installed.
Aim is to:
- develop using my windows 10 laptop
- deploy Spark app in the DSE spark cluster for production
Different options exist for IDE to used: Eclipse or Intellij Idea.
The best one looks to be Intellij Idea with Scala plugin.
How to setup Windows 10 for Spark dev workstation
Do not install locally in Windows 10 Spark, Scala, Hadoop, DSE.
Just download Intellij Idea, install it in Windows 10
Install JAVA SDK (not 9..I do not know why but does not work properly with SBT build tool in my laptop), set JAVA_HOME
Install SCALA plugin for Intellij Idea
You are now ready for coding.
For building and deploying, use SBT and deploy the JAR in Spark cluster nodes.