Question

我想为spark-salesforce添加sbt软件包我收到以下异常

    Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: com.springml.spark.salesforce. Please find packages at http://spark.apache.org/third-party-projects.html
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:553)
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:89)
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:89)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:304)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
    at salesforce_check.salesforce_connect$.main(salesforce_connect.scala:35)
    at salesforce_check.salesforce_connect.main(salesforce_connect.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: 
com.springml.spark.salesforce.DefaultSource
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

在build.sbt中添加了以下内容。请验证sbt，并让我知道Salesforce依赖项如何包含在软件包中

   name := "SparkTestProject"

   version := "0.1"

   scalaVersion := "2.11.8"

   val sparkVersion = "2.1.0"
   scalacOptions ++=
   Seq("-encoding", "UTF8", "-unchecked", "-deprecation", "- 
language:postfixOps", "-language:implicitConversions", "- 
language:higherKinds", "-language:reflectiveCalls")



libraryDependencies ++= Seq("org.apache.spark" %% "spark-hive" % 
sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-yarn" % sparkVersion,
"org.spark-project.hive" % "hive-metastore" % "1.2.1.spark2",
"org.slf4j" % "slf4j-api" % "1.7.21",
"org.scalatest" %% "scalatest" % "3.0.0",
"com.holdenkarau" %% "spark-testing-base" % "2.0.0_0.4.4",
"com.github.scopt" %% "scopt" % "3.5.0",
"org.scalaz"  %% "scalaz-core" % "7.2.7",
"org.scalactic" %% "scalactic" % "3.0.1",
"com.springml" % "spark-salesforce_2.11" % "1.1.0"
)

尝试连接到Salesforce

val salesforce_user = spark.read.format("com.springml.spark.salesforce")
  .option("username", "migrationuser@cable.comcast.com.uat")
  .option("password", "migratioteon@123")
  .option("login", "https://test.salesforce.com/services/Soap/u/41.0")
  .option("soql", "SELECT AccountNumber,Active_Inactive__c,Billing_Account_Number__c,CSG_Customer_Id__c,Past_Due_Balance__c from Account")
  .load().toDF()

println("connected to salesforce")
salesforce_user.printSchema
salesforce_user.show()

如何为spark-salesforce设置sbt软件包

0 个答案: