Question

我想下载一些xml文件（每个50MB-大约3000 = 150GB），进行处理，然后使用pyspark上传到BigQuery。出于开发目的，我使用了jupyter笔记本和少量文件10。我在dataproc上编写了非常复杂的代码设置集群。我的daproc群集具有6TB的HDFS，10个节点（每个4核）和120GB的RAM。

def context():
    import os
    os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.hadoop:hadoop-aws:2.7.3 pyspark-shell'
    import pyspark
    conf = pyspark.SparkConf()

    conf = (conf.setMaster('local[*]')
            .set('spark.executor.memory', '4G')
            .set('spark.driver.memory', '45G')
            .set('spark.driver.maxResultSize', '10G')
            .set("spark.python.profile", "true"))
    sc = pyspark.SparkContext(conf=conf)
    return sc

def job(sc):
    print("Job started")
    RDDread = sc.wholeTextFiles("s3a://custom-bucket/*/*.gz")
    models = RDDread.flatMap(process_xmls).groupByKey()
    tracking_all = (models.filter(lambda x: x[0] == TrackInformation)
                    .flatMap(lambda x: x[1])
                    .map(lambda model: (model.flight_ref, model))
                    .groupByKey())
    tracking_merged = tracking_all.map(lambda x: x[1]).map(merge_ti)
    flight_plans = (models.filter(lambda x: x[0] == FlightPlan).flatMap(lambda x: x[1]).map(lambda fp: (fp.flight_ref, fp)))
    fps_tracking = tracking_merged.union(flight_plans).groupByKey().filter(lambda x: len(x[1]) == 2)
    in_bq_batch = 1000
    n = fps_tracking.count()
    parts = ceil(n / in_bq_batch)
    many_n = fps_tracking.repartition(parts).mapPartitions(upload_fpm2)
    print("Job ended")
    return fps_tracking, tracking_merged, flight_plans, models, many_n

200条消息org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz]之后，我遇到2个错误：java.lang.OutOfMemoryError和MemoryError，主要是MemoryError。我以为RDDread之后只有2个分区，所以我修改了以下代码：sc.wholeTextFiles（“ s3a：// custom-bucket / / .gz”，minPartitions = 40）->并且破产得更快。我在一些随机位置添加了persistent（DISK）函数。

File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 684, in loads
    return s.decode("utf-8") if self.use_unicode else s
MemoryError

19/05/20 14:09:23 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz]
19/05/20 14:09:30 ERROR org.apache.spark.util.Utils: Uncaught exception in thread stdout writer for /opt/conda/default/bin/python
java.lang.OutOfMemoryError: Java heap space

我做错了什么以及如何调试代码？

Answer 1

您似乎在本地模式（local [*]）中运行spark。这意味着您正在使用具有45G RAM（spark.driver.memory）的单个jvm，并且所有工作线程都在该jvm中运行。 spark.executor.memory选项What does setMaster `local[*]` mean in spark?无效。

您应该在纱线调度器上设置火花机，或者在没有纱线的情况下使用独立模式https://spark.apache.org/docs/latest/spark-standalone.html。

如何在PySpark中调试MemoryError

1 个答案: