Question

我正在为我的PySpark应用程序编写一个自定义库，它需要使用某些CSV文件上的Pandas库进行一些预处理。由于输入文件本身存储在驱动程序中而不是HDFS中，因此在驱动程序节点上要进行“应该”的预处理（好吧，这就是我的想法）。但是，在使用addPyFile函数将库添加为包之后，导入所需的方法并执行该函数，则会引发ImportError。

包装结构是这样的

module
|- __init__.py
|- module_1.py
|- module_2.py
|- sub_module_1
   |- __init__.py
   |- sub_mod_1.py
|- ...

我在Python运行脚本中所做的是

spark = SparkSession.builder.enableHiveSupport().getOrCreate()
spark.sparkContext.addPyFile("module.zip")

from module import module_1

module_1.func(spark, configs) # Exception raised here

在module_1.py中，我有

import pandas as pd
from sub_module_1 import sub_mod_1

def func(spark, configs):

    input_local_file = configs.get("SOME_SECTION", "local_file")
    input_hdfs_file = configs.get("SOME_SECTION", "hdfs_file")
    output_hdfs_destination = configs.get("SOME_SECTION", "hdfs_dest")

    # Reads input file
    lf_pdf = pd.read_csv(input_local_file)
    # Convert pandas dataframe to dictionary object
    transformed_dict = to_dictionary(lf_pdf)
    # Log printed

    # Writes to hdfs, wraps a mapPartitions function
    another_method(transformed_dict, input_hdfs_file, output_hdfs_destination)

因此，这是否意味着即使我实际上并未在工作程序节点中使用Pandas，只要该包需要模块并通过addPyFile选项进行分发，它仍将需要Pandas库来也要安装在工人身上？事实是，module_2几乎做同样的事情，只不过将Pandas数据帧转换为Spark数据帧，但它不会引发相同的Exception。

完整的错误消息是：

WARN scheduler.TaskSetManager: Lost task 48.2 in stage 4.0 (TID 167, somewhere.org, executor 35): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2/python/pyspark/worker.py", line 166, in main
    func, profiler, deserializer, serializer = read_command(pickleSer, infile)
  File "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2/python/pyspark/worker.py", line 57, in read_command
    command = serializer.loads(command.value)
  File "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2/python/pyspark/serializers.py", line 454, in loads
    return pickle.loads(obj)
  File "./module.zip/module/module_1.py", line 15, in <module>
ImportError: No module named pandas

        at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
        at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
        at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
        at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

编辑：我也一直在记录应用程序中的步骤，并且在完成所有预处理之后才出现此错误，这就是为什么我不这样做的原因确定为什么会这样，因为不再使用熊猫了。

Answer 1

我发现了不一致的原因-只有使用mapPartitions方法的模块才出现此问题。我只是这样做

try:
    import pandas
except:
    pass

因为该库完全不在工作程序节点中使用。

即使在工作节点上不需要库，也会引发导入错误

1 个答案: