Question

我正在将我的spark工作作为纱线客户端模式运行，为此，我需要番石榴罐来完成我的火花工作，该工作需要放在驱动程序和执行程序的类路径中。

所以，我在spark-submit命令下面运行：

spark-submit --class "com.bk.App" --master yarn --deploy-mode client --executor-cores 2  --driver-memory 1G --driver-cores 1 --driver-class-path /home/my_account/spark-jars/guava-19.0.jar  --conf spark.executor.extraClassPath=/home/my_account/spark-jars/guava-19.0.jar maprfs:///user/my_account/jobs/spark-jobs.jar parma1 parma2

这给了我以下例外：

Downloading maprfs:///user/my_account/jobs/spark-jobs.jar to /tmp/tmp732578642370806645/user/my_account/jobs/spark-jobs.jar.
2018-10-29 19:37:52,2025 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:566 Thread: 6832 Client initialization failed due to mismatch of libraries. Please make sure that the java library version matches the native build version 5.0.0.32987.GA and native patch version $Id: mapr-version: 5.0.0.32987.GA 40889:3056362e419b $
Exception in thread "main" java.io.IOException: Could not create FileClient
    at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:593)
    at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:654)
    at com.mapr.fs.MapRFileSystem.getMapRFileStatus(MapRFileSystem.java:1310)
    at com.mapr.fs.MapRFileSystem.getFileStatus(MapRFileSystem.java:942)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:345)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:297)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2066)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2035)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2011)
    at org.apache.spark.deploy.SparkSubmit$.downloadFile(SparkSubmit.scala:874)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$1.apply(SparkSubmit.scala:316)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$1.apply(SparkSubmit.scala:316)
    at scala.Option.map(Option.scala:146)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:316)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我什至尝试将番石榴罐放在hdfs位置，并在我的Spark提交中使用hdfs://甚至maprfs://添加了额外的类路径。也尝试提供local://。所有这些都以上述相同的例外结束。

注意：如果没有提供驱动程序和执行器额外的类路径jar，则Job可以正常工作。

有什么建议吗？我是否以错误的方式使用类路径参数？

为hdfs位置中的jar提供额外的类路径时，spark提交失败

0 个答案: