Question

我正在使用Spark-Submit，当我导入包裹时，我遇到了错误＆＃39; pandas＆＃39;。

从笔记本电脑中，我可以毫无问题地导入大熊猫。

但是当我通过spark-submit提交以下文件时：

from pymongo import MongoClient
import pandas as pd

mongo_url = "mongodb://..."

LOGDB = MongoClient(mongo_url).schedule.log_test
LOGDB.insert({'m': 'Worked Spark Submit Test'})

我收到以下错误：

stderr_1461683917691631694档案：

Spark Command: /usr/local/src/spark160master/ibm-java-x86_64-80//bin/java -cp /usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-launcher_2.10-1.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-network-shuffle_2.10-1.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/gson-2.2.4.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/guava-14.0.1.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/Java-WebSocket-1.3.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-ego_2.10-1.6.0.jar:/usr/local/src/spark160master/spark/profile/batch/:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/spark-assembly-1.6.0-hadoop2.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-api-jdo-3.2.6.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-core-3.2.10.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-rdbms-3.2.9.jar:/usr/local/src/data-connectors-1.4.1/* -Dspark.eventLog.enabled=true -Dspark.master=spark://yp-spark-dal09-env5-0005:7083 -Dspark.executor.memory=1024m -Dspark.executor.extraLibraryPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.driver.extraClassPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.driver.extraLibraryPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.app.name=test_submit.py -Dspark.executor.extraClassPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.files=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py -Dspark.eventLog.dir=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/events -Dspark.files.useFetchCache=false -Dspark.shuffle.service.port=7340 -Xms512m -Xmx512m org.apache.spark.deploy.ego.EGOClusterDriverWrapper {{WORKER_URL}} /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py org.apache.spark.deploy.PythonRunner --primary-py-file test_submit.py
========================================
log4j:ERROR Could not find value for key log4j.appender.FILE
log4j:ERROR Could not instantiate appender named "FILE".
16/04/26 10:18:43 INFO deploy.ego.EGOClusterDriverWrapper: Registered signal handlers for [TERM, HUP, INT]
16/04/26 10:18:44 WARN hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/26 10:18:44 INFO apache.spark.SecurityManager: Changing view acls to: s270-18029e40ba8077-05540e1c503a
16/04/26 10:18:44 INFO apache.spark.SecurityManager: Changing modify acls to: s270-18029e40ba8077-05540e1c503a
16/04/26 10:18:44 INFO apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s270-18029e40ba8077-05540e1c503a); users with modify permissions: Set(s270-18029e40ba8077-05540e1c503a)
16/04/26 10:18:45 INFO spark.util.Utils: Successfully started service 'EGOClusterDriverWrapper-driver-20160426101839-0010-41cca182-f2d6-408c-81c0-ec288530f82e' on port 55398.
16/04/26 10:18:45 INFO apache.spark.SecurityManager: Changing view acls to: s270-18029e40ba8077-05540e1c503a
16/04/26 10:18:45 INFO apache.spark.SecurityManager: Changing modify acls to: s270-18029e40ba8077-05540e1c503a
16/04/26 10:18:45 INFO apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s270-18029e40ba8077-05540e1c503a); users with modify permissions: Set(s270-18029e40ba8077-05540e1c503a)
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Fetching file from /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py to /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py
16/04/26 10:18:45 INFO spark.util.Utils: Copying /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py to /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Starting the user JAR in a separate Thread
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Waiting for spark context initialization ... 0
16/04/26 10:18:46 INFO deploy.ego.EGOClusterDriverWrapper: Final app status: 1, exitCode: 15, (reason: User class threw exception: User application exited with 1)
16/04/26 10:18:46 INFO deploy.ego.EGOClusterDriverWrapper: Sending driver program state to master
Exception in thread "Driver" org.apache.spark.SparkUserAppException: User application exited with 1
        at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:88)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:507)
        at org.apache.spark.deploy.ego.EGOClusterDriverWrapper$$anon$3.run(EGOClusterDriverWrapper.scala:434)

stdout_1461683917691631694

no extra config
load default config from : /usr/local/src/spark160master/spark/profile/batch/
/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsUTF8String
Traceback (most recent call last):
  File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py", line 19, in <module>
    import pandas as pd
  File "/usr/lib64/python2.7/site-packages/pandas-0.14.1-py2.7-linux-x86_64.egg/pandas/__init__.py", line 6, in <module>
    from . import hashtable, tslib, lib
  File "numpy.pxd", line 157, in init pandas.hashtable (pandas/hashtable.c:22150)
  File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsUTF8String

我做错了什么？

Answer 1

我无法完全重现您的错误，因为我没有您的整个代码spark-submit作业代码。

你是否正在初始化你的火花环境？

这是我的代码： -

from pymongo import MongoClient

import pandas as pd

from pyspark import SparkContext

if __name__ == "__main__":

    sc = SparkContext(appName="MongoDB CON TEST")
    print("Hello Spark Demo for Mongodb connection")
    print("Begining MongoCon")

    mongo_url = "mongodb://ds045252.mlab.com:45252"

    connection = MongoClient(mongo_url)

    db = connection["samplemongodb"]
    # MongoLab has user authentication
    db.authenticate("username", "password")
    db.Photos.insert({'m': 'Worked Spark Submit Test Real test'})
    print("Insert worked")
    sc.stop()

请在github中查看缩写的完整代码： - https://github.com/charles2588/sparkjobs/blob/master/mongodbtest.py

提交作业： -

./spark-submit.sh --vcap credentials/vcap.json --deploy-mode cluster --master https://169.54.219.20 mongodbtest.py

谢谢，

查尔斯。

Answer 2

笔记本电脑预装了Pandas和NumPy。您的stacktrace显示spark-submit正在尝试从.local加载NumPy，因此这是您自己安装的版本。您是否可以在火花提交环境中预先安装NumPy？

错误可能是由LD_LIBRARY_PATH的错误设置引起的。你是否从笔记本电脑安装了本地NumPy，现在正试图从spark-submit使用它？这些是不同的执行环境，它们可能不共享LD_LIBRARY_PATH的相同设置。

在Spark-Submit脚本

2 个答案: