DataFrame已保存到分区表,但无法查询

时间:2018-07-11 11:44:33

标签: apache-spark

我有以下简单代码:

  test("SparkSQLTest") {
    val spark = SparkSession.builder().master("local").config("spark.sql.cbo.enabled", true).appName("SparkSQLTest").enableHiveSupport().getOrCreate()
    import spark.implicits._


    spark.sql("create database if not exists test_db").count()
    spark.sql(
      """
         create table if not exists test_db.t_0149 (id int, name string) partitioned by(dt string) stored as parquet

      """.stripMargin(' ')).count()
    val dt = "20180701"
    val path = spark.sqlContext.getConf("spark.sql.warehouse.dir") + s"test_db.db/t_0149/dt=$dt"
    println(path)
    spark.createDataset(Seq((1, "a"), (2, "b"), (3, "c"))).toDF("id", "name").write.format("parquet").mode(SaveMode.Overwrite).save(path)
    spark.sql(s"select * from test_db.t_0149 where dt = '$dt' ").show()

  }

数据框已成功保存到表的目录中,但未查询任何内容。我不知道怎么了。

0 个答案:

没有答案