更新

Question

我正在尝试将一列的数据框添加到更大的数据框中，但第一个数据框的问题是在创建它并尝试通过命令将其添加到主数据框之后：

  df.withColumn("name", dataframe)

我收到错误：

 **found   : org.apache.spark.sql.DataFrame
 (which expands to)  org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]
 required: org.apache.spark.sql.Column**

我知道数据集[Row]应该与Dataframe同义，但是我不知道如何解决这个错误。

对于上下文，我的代码的（真正）淡化版本如下：

// test function - will be used as part of the main script below
def Test(inputone: Double, inputtwo: Double): Double = { 
 var test = (2 * inputone) + inputtwo
 test 
}

对于主要脚本（即问题出在哪里）

//Importing the data via CSV
var df = sqlContext.read.format("csv").option("header",     "true").option("inferSchema", "true").load("/root/file.csv")

提供数据的上下文：

df: org.apache.spark.sql.DataFrame = [ID: int, blue: int ... 8 more fields]

+---+----+------+-----+------+------+----+---+-----+-----+
| ID|blue|purple|green|yellow|orange|pink|red|white|black|
+---+----+------+-----+------+------+----+---+-----+-----+
|  1| 500|    44|    0|     0|     3|   0|  5|   43|    2|
|  2| 560|    33|    1|     0|     4|   0| 22|   33|    4|
|  3| 744|    44|    1|    99|     3|1000| 78|   90|    0|
+---+----+------+-----+------+------+----+---+-----+-----+

root
 |-- ID: integer (nullable = true)
 |-- blue: integer (nullable = true)
 |-- purple: integer (nullable = true)
 |-- green: integer (nullable = true)
 |-- yellow: integer (nullable = true)
 |-- orange: integer (nullable = true)
 |-- pink: integer (nullable = true)
 |-- red: integer (nullable = true)
 |-- white: integer (nullable = true)
 |-- black: integer (nullable = true)

从那时起，脚本继续

// Creating a list for which columns to draw from the main dataframe
val a = List("green", "blue")

// Creating the mini dataframe to perform the function upon
val test_df = df.select(a.map(col): _*)

// The new dataframe will now go through the 'Test' function defined above
val df_function = test_df.rdd.map(col => Test(col(0).toString.toDouble, col(1).toString.toDouble))

// Converting the RDD output back to a dataframe (of one column)
val df_convert = df_function.toDF

作为参考，输出如下

+-----+
|value|
+-----+
|500.0|
|562.0|
|746.0|
+-----+

脚本的最后一行是将其添加到主数据框中，如下所示

 df = df.withColumn("new column", df_convert)

但如上所述，我收到以下错误：

found   : org.apache.spark.sql.DataFrame

   (which expands to)  org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]

required: org.apache.spark.sql.Column

////////// EDIT ////////////

@ user9819212解决方案适用于简单的方法但是当调用一个更复杂时，我得到以下错误

    test2_udf: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function5>,DoubleType,Some(List(DoubleType, IntegerType, StringType, DoubleType, DoubleType)))
    java.lang.ClassCastException: $anonfun$1 cannot be cast to scala.Function1

所以我尝试创建另一个简单版本的代码，对测试函数进行一些额外的更改，称为

// test function - will be used as part of the main script below
def Test (valueone: Double, valuetwo: Integer): Double = {
    val test = if(valuetwo > 2000) valueone + 4000 else valueone
    val fakeList = List(3000,4000,500000000)
    val index = fakeList.indexWhere(x => x>=valueone)
    val test2 = fakeList(index - 1) * valueone
    test2
}

val test_udf = udf(Test _)

df = df.withColumn(
   "new column", 
   test_udf(col("green").cast("double"), col("blue").cast("integer"))
)

起初似乎有效，但当我尝试使用命令

查看数据框时

df.show

我收到以下错误

    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 153.0 failed 1 times, most recent failure: Lost task 0.0 in stage 153.0 (TID 192, localhost, executor driver): 
org.apache.spark.SparkException: Failed to execute user defined function($anonfun$1: (double, int) => double)

Answer 1

您不能以这种方式添加其他DataFrame（或DataFrame）的列。只需使用UserDefinedFunction：

import org.apache.spark.sql.functions.udf._

val test_udf = udf(Test _)

df.withColumn(
   "new column", 
   test_udf(col("green").cast("double"), col("blue").cast("double"))
)

或具有如此简单的功能：

df.withColumn(
   "new column", 
   2 * col("green").cast("double") + col("blue").cast("double")
)

Answer 2

如果您转到api document，则会明确提及

AttributeError: 'Thief' object has no attribute 'human'

正如您所看到的那样，第二个参数应该是Person.__init__并且您已经传递了public DataFrame withColumn(java.lang.String colName, Column col) Returns a new DataFrame by adding a column or replacing the existing column that has the same name.。

这就是问题的原因。

您正在尝试添加从Column到DataFrame的列。这两个数据帧完全不同。对于这种情况，如果要分离数据帧，则必须查看join

或者spark functions与withColumn api一起用作列。

更新

查看您的第一个日志

df_convert

建议您将df函数定义为

test2_udf: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function5>,DoubleType,Some(List(DoubleType, IntegerType, StringType, DoubleType, DoubleType)))

和你的第二个日志

udf

建议您在def Test(valueone: Double, valuetwo: Integer, valuethree: String, valuefour: Double, valuefive: Double): Double = { ??? //calculation parts } val test2_udf = udf(Test _) //Test: Test[](val valueone: Double,val valuetwo: Integer,val valuethree: String,val valuefour: Double,val valuefive: Double) => Double //test2_udf: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function5>,DoubleType,Some(List(DoubleType, IntegerType, StringType, DoubleType, DoubleType)))调用中只传递一个参数

java.lang.ClassCastException: $anonfun$1 cannot be cast to scala.Function1

如果您专注于错误消息的test2_udf部分，它清楚地表明传递给udf函数的列数

如果传递三个参数，那么

后面的

df.withColumn("new column", test2_udf(col("green").cast("double"))).show(false)
//java.lang.ClassCastException: A$A30$A$A30$$anonfun$test2_udf$1 cannot be cast to scala.Function1

Scala - 如何将数据集[Row]转换为可添加到Dataframe的列

2 个答案:

更新