Question

我是pyspark的新手，正在尝试了解其下的不同操作和转换。

在jupyter笔记本上尝试count（）时，describe（）操作。我无法理解为什么找不到此文件异常？我遇到以下erorr：-


from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("ABC") \
    .getOrCreate()

df = spark.read.csv("Documents\hero.csv", inferSchema = True , header = True)

df.count()



Py4JJavaError                             Traceback (most recent call last)
<ipython-input-56-3c9a60fd698f> in <module>()
----> 1 df.count()

C:\spark-1\spark-2.3.3\python\pyspark\sql\dataframe.py in count(self)
    453         2
    454         """
--> 455         return int(self._jdf.count())
    456 
    457     @ignore_unicode_prefix

C:\spark-1\spark-2.3.3\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

C:\spark-1\spark-2.3.3\python\pyspark\sql\utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

C:\spark-1\spark-2.3.3\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o858.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 35.0 failed 1 times, most recent failure: Lost task 0.0 in stage 35.0 (TID 57, localhost, executor driver): java.io.FileNotFoundException: C:\Users\Tanya Srivastava\AppData\Local\Temp\blockmgr-619df226-2c9b-4ca5-b52f-8ec7a1ea8092\3c\temp_shuffle_f18cbb8d-3e52-4b0a-8662-b031e9c3fd6c (The system cannot find the path specified)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(Unknown Source)
    at java.io.FileOutputStream.<init>(Unknown Source)

Answer 1

PySpark中似乎有一个错误可以解决

尝试升级Spark版本。

OR

使用YARN客户端模式（-部署模式客户端）提交作业。

在pyspark数据帧中，很少有诸如count（），describe（）之类的操作会引发java.io.FileNotFoundException

1 个答案: