Cloud Dataproc

Question 1

What is Cloud Dataproc?

Answer 1

A fully managed cluster data processing service (Apache Spark and Apache Hadoop service)

Question 2

What are key points of Cloud Dataproc?

Answer 2

1. Compatible with Apache Hadoop, Spark and Hive
2. Runs in clusters
3. Allows existing projects to be moved without redevelopment
4. Fast cluster creation - can create workflow templates
5. Can scale clusters without stopping Job
6. Can switch to different versions
7. Can handle streaming and batch data

Question 3

When do you choose Cloud Dataproc over Cloud Dataflow?

Answer 3

If you have dependencies on Hadoop or Spark, or if you want more hands on management and control.

Question 4

How do you create a Cloud Dataproc cluster from the command line?

Answer 4

gcloud dataproc clusters create [CLUSTER NAME] –zone [ZONE]

Question 5

How do you submit a job to Cloud Dataproc via the shell?

Answer 5

gcloud dataproc jobs submit [TYPE] –cluster [CLUSTER NAME] –jar [JAR FILE]

Question 6

What cluster modes can you choose when setting up Cloud Dataproc?

Answer 6

1. Single - for development
2. Standard - one master node
3. High Availability - uses 3 master nodes

Question 7

What job types are available for Cloud Dataproc?

Answer 7

1. Spark
2. PySparck
3. SparkR
4. Hive
5. Spark SQL
6. Pig
7. Hadoop

Question 8

How do you import or export data to Cloud Dataproc?

Answer 8

You don’t. It’s a data analysis platform, not a database.
You can import and export to save/restore the cluster configuration data.
gcloud beta dataproc clusters export [CLUSTER NAME] – destination=[PATH TO EXPORT FILE]
gcloud beta dataproc clusters import [SOURCE FILE]