Question

我在Pyspark上运行一个小脚本，在那里我从hbase表中提取一些数据并创建一个Pyspark数据帧。我正在尝试将数据帧保存回本地hdfs，并且遇到退出50错误。我可以成功地为相对较小的数据帧执行相同的操作，但是无法对大文件执行此操作。我很乐意分享任何代码片段，并感谢任何帮助。此外，SparkUI的整个环境可以作为屏幕截图进行共享。

这是我的Spark（2.0.0）属性的配置（此处显示为字典）。部署在纱线客户端上。

configuration={'spark.executor.memory':'4g',
        'spark.executor.instances':'32',
        'spark.driver.memory':'12g',
        'spark.yarn.queue':'default' 
       }

获取数据框后，我试图将其保存为：

df.write.save('user//hdfs//test_df',format = 'com.databricks.spark.csv',mode = 'append')

以下错误块会一直重复，直到作业失败。我相信这可能是一个OOM错误，但我尝试过多达128个执行器，每个执行器有16GB内存，但无济于事。任何解决方法都将不胜感激。

Container exited with a non-zero exit code 50

17/09/25 15:19:35 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 64, fslhdppdata2611.imfs.micron.com): ExecutorLostFailure (executor 42 exited caused by one of the running tasks) Reason: Container marked as failed: container_e37_1502313369058_6420779_01_000043 on host: fslhdppdata2611.imfs.micron.com. Exit status: 50. Diagnostics: Exception from container-launch.
Container id: container_e37_1502313369058_6420779_01_000043
Exit code: 50
Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:109)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89)
    at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Shell output: main : command provided 1
main : run as user is hdfsprod
main : requested yarn user is hdfsprod
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /opt/hadoop/data/03/hadoop/yarn/local/nmPrivate/application_1502313369058_6420779/container_e37_1502313369058_6420779_01_000043/container_e37_1502313369058_6420779_01_000043.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths...

容器退出时使用非零退出代码50将Spark Dataframe保存到hdfs

0 个答案: