Question

在尝试创建临时表的警告下获得此信息请帮助解决此警告

scala> df.registerTempTable（“ df”）警告：有一个弃用警告；使用-deprecation重新运行以获取详细信息

Answer 1

Spark 2.0中不推荐使用registerTempTable方法

createOrReplaceTempView是受支持的替换功能

Answer 2

火花代码DataSet.scala参见文档中的此消息

改为使用createOrReplaceTempView（viewName）

 /**
       * Registers this Dataset as a temporary table using the given name. The lifetime of this
       * temporary table is tied to the [[SparkSession]] that was used to create this Dataset.
       *
       * @group basic
       * @since 1.6.0
       */
      @deprecated("Use createOrReplaceTempView(viewName) instead.", "2.0.0")
      def registerTempTable(tableName: String): Unit = {
        createOrReplaceTempView(tableName)
      }

使用示例数据集的示例用法演示，使用createOrReplaceTempView：

   package com.examples

import com.droolsplay.util.SparkSessionSingleton
import org.apache.log4j.{Level, Logger}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

/**
  * Join Example and some basics demonstration using sample data.
  *
  * @author : Ram Ghadiyaram
  */
object JoinExamplesv2 extends Logging {
  // switch off  un necessary logs
  Logger.getLogger("org").setLevel(Level.OFF)
  Logger.getLogger("akka").setLevel(Level.OFF)
  //  val spark: SparkSession = SparkSession.builder.config("spark.master", "local").getOrCreate;
  val spark: SparkSession = SparkSessionSingleton.getInstance(Option(this.getClass.getName))

  /**
    * main
    *
    * @param args Array[String]
    */
  def main(args: Array[String]): Unit = {
    import spark.implicits._
    /**
      * create 2 dataframes here using case classes one is Person df1 and another one is profile df2
      */
    val df1 = spark.sqlContext.createDataFrame(
      spark.sparkContext.parallelize(
        Person("Sarath", 33, 2)
          :: Person("Vasudha Nanduri", 30, 2)
          :: Person("Ravikumar Ramasamy", 34, 5)
          :: Person("Ram Ghadiyaram", 42, 9)
          :: Person("Ravi chandra Kancharla", 43, 9)
          :: Nil))


    val df2 = spark.sqlContext.createDataFrame(
      Profile("Spark", 2, "SparkSQLMaster")
        :: Profile("Spark", 5, "SparkGuru")
        :: Profile("Spark", 9, "DevHunter")
        :: Nil
    )

    // you can do alias to refer column name with aliases to  increase readablity

    val df_asPerson = df1.as("dfperson")
    val df_asProfile = df2.as("dfprofile")
    /** *
      * Example displays how to join them in the dataframe level
      * next example demonstrates using sql with createOrReplaceTempView
      */
    val joined_df = df_asPerson.join(
      df_asProfile
      , col("dfperson.personid") === col("dfprofile.personid")
      , "inner")
    joined_df.select(
      col("dfperson.name")
      , col("dfperson.age")
      , col("dfprofile.name")
      , col("dfprofile.profileDescription"))
      .show

    /// example using sql statement after registering createOrReplaceTempView

    df_asPerson.createOrReplaceTempView("dfPerson");
    df_asProfile.createOrReplaceTempView("dfprofile")
    // this is example of plain sql
    val dfJoin = spark.sqlContext.sql(
      """SELECT dfperson.name, dfperson.age, dfprofile.profileDescription
                          FROM  dfperson JOIN  dfprofile
                          ON dfperson.personid == dfprofile.personid""")
    logInfo("Example using sql statement after registering createOrReplaceTempView ")
    dfJoin.show(false)

  }

  // models here

  case class Person(name: String, age: Int, personid: Int)

  case class Profile(name: String, personId: Int, profileDescription: String)

}

结果：

+--------------------+---+-----+------------------+
|                name|age| name|profileDescription|
+--------------------+---+-----+------------------+
|              Sarath| 33|Spark|    SparkSQLMaster|
|     Vasudha Nanduri| 30|Spark|    SparkSQLMaster|
|  Ravikumar Ramasamy| 34|Spark|         SparkGuru|
|      Ram Ghadiyaram| 42|Spark|         DevHunter|
|Ravi chandra Kanc...| 43|Spark|         DevHunter|
+--------------------+---+-----+------------------+

18/11/12 23:03:38 INFO JoinExamplesv2: Example using sql statement after registering createOrReplaceTempView 
+----------------------+---+------------------+
|name                  |age|profileDescription|
+----------------------+---+------------------+
|Sarath                |33 |SparkSQLMaster    |
|Vasudha Nanduri       |30 |SparkSQLMaster    |
|Ravikumar Ramasamy    |34 |SparkGuru         |
|Ram Ghadiyaram        |42 |DevHunter         |
|Ravi chandra Kancharla|43 |DevHunter         |

带有Scala的Spark SQL：registerTempTable的弃用警告

2 个答案: