Question

我使用这个在spark-shell中运行scala代码：

spark-shell --conf spark.neo4j.bolt.password=TestNeo4j --packages neo4j-contrib:neo4j-spark-connector:2.0.0-M2,graphframes:graphframes:0.2.0-spark2.0-s_2.11 -i neo4jsparkCluster.scala

这会在一个spark实例上运行得很好，现在我想对它进行聚类。

我有一个build.sbt文件，如下所示：

name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
        "org.apache.spark" %% "spark-core" % "2.2.0",
        "org.apache.spark" %% "spark-sql" % "2.2.0"
)

所以我不确定我在库中依赖于我想要加载的库我需要什么，我想它的语法？

我使用sbt package获得的错误是：

[info] Compiling 2 Scala sources to /Users/shane.thomas/SparkCourse/spark-sbt-builds/target/scala-2.11/classes...
[error] /Users/shane.thomas/SparkCourse/spark-sbt-builds/neo4jSparkCluster.scala:1: object neo4j is not a member of package org
[error] import org.neo4j.spark._
[error]            ^
[error] /Users/shane.thomas/SparkCourse/spark-sbt-builds/neo4jSparkCluster.scala:5: object streaming is not a member of package org.apache.spark
[error] import org.apache.spark.streaming._
[error]                         ^
[error] /Users/shane.thomas/SparkCourse/spark-sbt-builds/neo4jSparkCluster.scala:6: object streaming is not a member of package org.apache.spark
[error] import org.apache.spark.streaming.StreamingContext._
[error]                         ^
[error] /Users/shane.thomas/SparkCourse/spark-sbt-builds/neo4jSparkCluster.scala:539: not found: value Neo4j
[error]   val neo = Neo4j(sc)
[error]             ^
[error] four errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 5 s, completed Dec 7, 2017 2:45:00 PM

Answer 1

尝试添加以下内容

<script src="https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js"></script>

取自SBT部分的https://github.com/neo4j-contrib/neo4j-spark-connector

Answer 2

这些对我来说很好

scalaVersion := "2.12.13"

val spark_version:String = "3.1.0"

resolvers ++= Seq(
  "Spark Packages Repo" at "https://dl.bintray.com/spark-packages/maven"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % spark_version,
  "org.apache.spark" %% "spark-sql" % spark_version,
  "org.neo4j" % "neo4j-kernel" % "4.2.3",
  "neo4j-contrib" % "neo4j-spark-connector" % "2.4.5-M2",
  "graphframes" % "graphframes" % "0.8.1-spark3.0-s_2.12",
)

在build.sbt中为neo4j spark连接器使用哪些依赖项？

2 个答案: