About this question
I am new to Spark and Spark SQL.
How does createOrReplaceTempView work in Spark?
If we register an RDD of objects as a table will spark keep all the data in memory?
I am new to Spark and Spark SQL.
How does createOrReplaceTempView work in Spark?
If we register an RDD of objects as a table will spark keep all the data in memory?
Log in to share your answer and help other learners.
Log in to answerBest Answer · By JanBask Hadoop Expert
Answered on May 10, 2021
Often we might want to store the spark Data frame as the table and query it, to convert Data frame into temporary view that is available for only that spark session, we use registerTempTable or createorreplacetempview (Spark > = 2.0) on our spark Dataframe.
createorreplacetempview is used when you desire to store the table for a specific spark session.
createorreplacetempview creates (or replaces if that view name already exists) a lazily evaluated "view" that you can then use like a hive table in Spark SQL. It does not persist to memory unless you cache the dataset that underpins the view.
scala> val s = Seq(1,2,3,4).toDF("num")
s: org.apache.spark.sql.DataFrame = [num: int]
scala> s.createOrReplaceTempView("nums")
scala> s.createOrReplaceTempView("nums")
scala> spark.table("nums")
res6: org.apache.spark.sql.DataFrame = [num: int]
scala> spark.table("nums").cache
res7: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [num: int]
scala> spark.table("nums").count
res8: Long = 4The data is cached fully only after the .count call.
However , the relevant quote (compared to persistent table): "Unlike the createOrReplaceTempView command, saveAsTable will materialize the contents of the DataFrame and create a pointer to the data in the Hive metastore."
Free tutorials and interview questions from industry experts — learn the skill, then get ready to prove it.
Most-read guides across JanBask
Common Spark interview questions, answered
Guides, tips and career advice on Spark from JanBask experts.
Hadoop Frequently Used Hive Commands in HQL with Examples
Top Hive Commands in HQL With Examples of the most commonly used Hadoop Hive commands for importing, exporting, and querying data…
Hadoop How to Compare Hive, Spark, Impala and Presto?
Explore the strengths and weaknesses of Presto vs Impala vs Hive vs Spark for big data processing. Learn which tool is best…
Hadoop What Is Hue? Hue Hadoop Tutorial Guide for Beginners
What is Hue? Hue Tutorial Guide for Beginner, We are covering Hue component, hadoop ecosystem, Hue features, Apache Hue Tutorial…