Question

尝试在DataFrame中删除一列，但我有一些带有点的列名，我将其转义。

在我逃脱之前，我的架构看起来像这样：

root
 |-- user_id: long (nullable = true)
 |-- hourOfWeek: string (nullable = true)
 |-- observed: string (nullable = true)
 |-- raw.hourOfDay: long (nullable = true)
 |-- raw.minOfDay: long (nullable = true)
 |-- raw.dayOfWeek: long (nullable = true)
 |-- raw.sensor2: long (nullable = true)

如果我尝试删除一列，我会得到：

df = df.drop("hourOfWeek")
org.apache.spark.sql.AnalysisException: cannot resolve 'raw.hourOfDay' given input columns raw.dayOfWeek, raw.sensor2, observed, raw.hourOfDay, hourOfWeek, raw.minOfDay, user_id;
        at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
        at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:60)
        at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:57)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:319)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:319)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53)

请注意，我甚至没有试图放下名字中带点的列。由于在没有转义列名的情况下似乎无能为力，我将模式转换为：

root
 |-- user_id: long (nullable = true)
 |-- hourOfWeek: string (nullable = true)
 |-- observed: string (nullable = true)
 |-- `raw.hourOfDay`: long (nullable = true)
 |-- `raw.minOfDay`: long (nullable = true)
 |-- `raw.dayOfWeek`: long (nullable = true)
 |-- `raw.sensor2`: long (nullable = true)

但这似乎没有帮助。我仍然得到同样的错误。

我尝试转义所有列名，然后使用转义名称删除，但这也不起作用。

root
 |-- `user_id`: long (nullable = true)
 |-- `hourOfWeek`: string (nullable = true)
 |-- `observed`: string (nullable = true)
 |-- `raw.hourOfDay`: long (nullable = true)
 |-- `raw.minOfDay`: long (nullable = true)
 |-- `raw.dayOfWeek`: long (nullable = true)
 |-- `raw.sensor2`: long (nullable = true)

df.drop("`hourOfWeek`")
org.apache.spark.sql.AnalysisException: cannot resolve 'user_id' given input columns `user_id`, `raw.dayOfWeek`, `observed`, `raw.minOfDay`, `raw.hourOfDay`, `raw.sensor2`, `hourOfWeek`;
        at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
        at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:60)

是否有另一种方法可以删除在此类数据上不会失败的列？

Answer 1

好吧，我似乎终于找到了解决方案：

df.drop(df.col("raw.hourOfWeek"))似乎有用

Answer 2

val data = df.drop("Customers");

适用于普通列

val new = df.drop(df.col("old.column"));

Spark 1.6：使用转义列名称

2 个答案: