About this question
I am new to Spark and Spark SQL.
How does createOrReplaceTempView work in Spark? If we register an RDD of objects as a table will spark keep all the data in memory?
I am new to Spark and Spark SQL.
How does createOrReplaceTempView work in Spark? If we register an RDD of objects as a table will spark keep all the data in memory?
Log in to share your answer and help other learners.
Log in to answerBest Answer · By JanBask Hadoop Expert
Answered on Nov 17, 2022
createOrReplaceTempView creates (or replaces if that view name already exists) a lazily evaluated "view" that you can then use like a hive table in Spark SQL. It does not persist to memory unless you cache the dataset that underpins the view.
scala> val s = Seq(1,2,3).toDF("num")
s: org.apache.spark.sql.DataFrame = [num: int]
scala> s.createOrReplaceTempView("nums")
scala> spark.table("nums")
res22: org.apache.spark.sql.DataFrame = [num: int]
scala> spark.table("nums").cache
res23: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [num: int]
scala> spark.table("nums").count
res24: Long = 3
The data is cached fully only after the .count call.
Related SO: spark createOrReplaceTempView vs createGlobalTempView
Relevant quote (comparing to persistent table): "Unlike the createOrReplaceTempView command, saveAsTable will materialise the contents of the DataFrame and create a pointer to the data in the Hive metastore." from https://spark.apache.org/docs/latest/sql-programming-guide.html#saving-to-persistent-tables
Note : createOrReplaceTempView was formerly registerTempTableFree tutorials and interview questions from industry experts — learn the skill, then get ready to prove it.
Most-read guides across JanBask
Common Spark interview questions, answered
Guides, tips and career advice on Spark from JanBask experts.
Hadoop Frequently Used Hive Commands in HQL with Examples
Top Hive Commands in HQL With Examples of the most commonly used Hadoop Hive commands for importing, exporting, and querying data…
Hadoop How to Compare Hive, Spark, Impala and Presto?
Explore the strengths and weaknesses of Presto vs Impala vs Hive vs Spark for big data processing. Learn which tool is best…
Hadoop What Is Hue? Hue Hadoop Tutorial Guide for Beginners
What is Hue? Hue Tutorial Guide for Beginner, We are covering Hue component, hadoop ecosystem, Hue features, Apache Hue Tutorial…