Question

我有一个看起来像这样的数据文件

+---------+---------+--------------------+--------+-------------------+---------+----------+--------------+
|InvoiceNo|StockCode|         Description|Quantity|        InvoiceDate|UnitPrice|CustomerID|       Country|
+---------+---------+--------------------+--------+-------------------+---------+----------+--------------+
|   536365|   85123A|WHITE HANGING HEA...|       6|2010-12-01 08:26:00|     2.55|   17850.0|United Kingdom|
|   536365|    71053| WHITE METAL LANTERN|       6|2010-12-01 08:26:00|     3.39|   17850.0|United Kingdom|
|   536365|   84406B|CREAM CUPID HEART...|       8|2010-12-01 08:26:00|     2.75|   17850.0|United Kingdom|
|   536365|   84029G|KNITTED UNION FLA...|       6|2010-12-01 08:26:00|     3.39|   17850.0|United Kingdom|
|   536365|   84029E|RED WOOLLY HOTTIE...|       6|2010-12-01 08:26:00|     3.39|   17850.0|United Kingdom|
|   536365|    22752|SET 7 BABUSHKA NE...|       2|2010-12-01 08:26:00|     7.65|   17850.0|United Kingdom|
|   536365|    21730|GLASS STAR FROSTE...|       6|2010-12-01 08:26:00|     4.25|   17850.0|United Kingdom|
|   536366|    22633|HAND WARMER UNION...|       6|2010-12-01 08:28:00|     1.85|   17850.0|United Kingdom|
|   536366|    22632|HAND WARMER RED P...|       6|2010-12-01 08:28:00|     1.85|   17850.0|United Kingdom|
|   536367|    84879|ASSORTED COLOUR B...|      32|2010-12-01 08:34:00|     1.69|   13047.0|United Kingdom|
+---------+---------+--------------------+--------+-------------------+---------+----------+--------------+

当我运行以下代码时

from pyspark.sql.functions import sum as sum_,count
relatil_data.groupBy('InvoiceNo').agg(sum_('UnitPrice'))

它可以正常工作并提供输出：

DataFrame[InvoiceNo: string, sum(UnitPrice): double]

但是当我运行以下代码

df=relatil_data.groupBy('InvoiceNo').agg(sum_('UnitPrice'))
df.show()

我收到以下错误

    C:\spark\spark-2.4.0-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o4839.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 198.0 failed 1 times, most recent failure: Lost task 0.0 in stage 198.0 (TID 214, localhost, executor driver): java.io.FileNotFoundException: C:\Users\pg186028\AppData\Local\Temp\blockmgr-e7aa0c35-ca53-4602-8411-bf816e010a46\17\temp_shuffle_f694f1cf-e72f-41b6-bf65-97ade34afc7c (The system cannot find the path specified)

当我尝试创建视图并在其上运行SQL时，发生了同样的事情。

Answer 1

尝试使用以下代码，而不导入库文件。

relatil_data.groupBy('InvoiceNo').agg("UnitPrice":"sum")

要从sum（UnitPrice）更改o / p列名称，请尝试以下代码

relatil_data.groupBy('InvoiceNo').agg("UnitPrice":"sum").withColumnRenamed("sum(UnitPrice)","Total_UnitPrice")

pyspark（版本2.4）与总和不起作用

1 个答案: