Question

当我试图在我的Spark conf中设置包依赖项“ spark-sftp”时，出现ClassNotFoundException。但是当我使用以下命令执行脚本时，它可以工作：

spark-submit-打包com.springml：spark-sftp_2.11：1.1.1 test.py

下面是我的代码。有人可以告诉我如何执行pyspark脚本通过不将软件包作为spark-submit的参数传递？

import sys
import datetime
import pyspark
from pyspark.sql import *
from pyspark.sql import SparkSession, SQLContext, Row, HiveContext
from pyspark import SparkContext

#Create new config
conf = (pyspark.conf.SparkConf()
.set("spark.driver.maxResultSize", "16g")
.set("spark.driver.memory", "20g")
.set("spark.executor.memory", "20g")
.set("spark.executor.cores", "5")
.set("spark.shuffle.service.enabled", "true")
.set("spark.dynamicAllocation.enabled", "true")
.set("spark.dynamicAllocation.initialExecutors", "24")
.set("spark.dynamicAllocation.minExecutors", "6")
.set("spark.submit.deployMode", "client")
.set("spark.jars.packages", "com.springml:spark-sftp_2.11:1.1.1")
.set("spark.python.worker.memory", "4g")
.set("spark.default.parallelism", "960")
.set("spark.executor.memoryOverhead", "4g")
.setMaster("yarn-client"))


# Create new context
spark = SparkSession.builder.appName("AppName").config(conf=conf).enableHiveSupport().getOrCreate()
spark.sparkContext.setLogLevel("WARN")


df = spark.read.format("com.springml.spark.sftp").option("host", "HOST").option("username", "HOSTNAME").option("password", "pass").option("fileType", "csv").option("inferSchema", "true").load("/test/sample.csv")

输出：：java.lang.ClassNotFoundException：无法找到数据源：com.springml.spark.sftp。请在http://spark.apache.org/third-party-projects.html中找到软件包在org.apache.spark.sql.execution.datasources.DataSource $ .lookupDataSource（DataSource.scala：635）在org.apache.spark.sql.DataFrameReader.load（DataFrameReader.scala：190）在org.apache.spark.sql.DataFrameReader.load（DataFrameReader.scala：174）在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）处在sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:62）在sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）在java.lang.reflect.Method.invoke（Method.java:498）在py4j.reflection.MethodInvoker.invoke（MethodInvoker.java:244）在py4j.reflection.ReflectionEngine.invoke（ReflectionEngine.java:357）在py4j.Gateway.invoke（Gateway.java:282）在py4j.commands.AbstractCommand.invokeMethod（AbstractCommand.java:132）在py4j.commands.CallCommand.execute（CallCommand.java:79）在py4j.GatewayConnection.run（GatewayConnection.java:238）在java.lang.Thread.run（Thread.java:748）引起原因：java.lang.ClassNotFoundException：com.springml.spark.sftp.DefaultSource

Answer 1

在提交Spark作业时，您可以指定要安装的软件包。为此，您可以将此Maven依赖项指定为：

> $SPARK_HOME/bin/spark-shell --packages com.springml:spark-sftp_2.11:1.1.3

如何在我的Pyspark代码中添加任何新的库，例如spark-sftp？

1 个答案: