02 Spark Table & DF

Question 1

What does spark dataframe include

Answer 1

Includes spark df and spark df api’s

Question 2

Dataframe tables layers

Answer 2

Storage layer

Metadata layer

Logic layer

Question 3

Dataframe tables layers

Answer 3

Storage layer - stores data

Metadata layer - schema info

Logic layer - provides output

Question 4

What are Spark dataframe

Answer 4

Same like spark tables

Metadata is not stored in metadata storage instead in run time metadata catalog

This will NOT store permanent and hence metadata stored in runtime

Question 5

features of Spark tables

Answer 5

Stored data in a file named .dbf

Table schema stored in metadata store

Permanently stored