Question

我正在尝试使用spark graphframe

创建图表

这是代码：

import org.graphframes._

// Node DataFrames
val v = sqlContext.createDataFrame(List(
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 30),
  ("d", "David", 29),
  ("e", "Esther", 32),
  ("f", "Fanny", 36),
  ("g", "Gabby", 60)
)).toDF("id", "name", "age")

// Edge DataFrame
val e = sqlContext.createDataFrame(List(
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
  ("f", "c", "follow"),
  ("e", "f", "follow"),
  ("e", "d", "friend"),
  ("d", "a", "friend"),
  ("a", "e", "friend")
)).toDF("src", "dst", "relationship")

// Create a GraphFrame
val g = GraphFrame(v, e)

但这是我得到的错误：

错误：加载类文件时检测到缺失或无效的依赖项＆＃39; GraphFrame.class＆＃39 ;.无法访问类型登录包 org.apache.spark，因为它（或其依赖项）缺失。校验缺少或冲突的依赖项的构建定义。（重办使用-Ylog-classpath查看有问题的类路径。）完整如果＆＃39; GraphFrame.class＆＃39;是针对一个编译的 org.apache.spark的不兼容版本。

我正在使用Apache Spark 2.1和Scala 2.11。有什么建议可以解决这个问题吗？

Answer 1

从maven central repo下载以下软件包

com.typesafe.scala-logging_scala-logging-api_2.11-2.1.2.jar    
graphframes_graphframes-0.5.0-spark2.1-s_2.11.jar  
org.slf4j_slf4j-api-1.7.7.jar
com.typesafe.scala-logging_scala-logging-slf4j_2.11-2.1.2.jar  
org.scala-lang_scala-reflect-2.11.0.jar

将以下内容添加到sparks-default.conf文件中（以逗号分隔的绝对路径列表，上面列出的jars所在的位置）

spark.jars            path_2_jar/org.slf4j_slf4j-api-1.7.7.jar, path_2_jar/org.scala-lang_scala-reflect-2.11.0.jar, path_2_jar/graphframes_graphframes-0.5.0-spark2.1-s_2.11.jar, path_2_jar/com.typesafe.scala-logging_scala-logging-slf4j_2.11-2.1.2.jar, path_2_jar/com.typesafe.scala-logging_scala-logging-api_2.11-2.1.2.jar

GraphFrame：加载类文件时检测到缺失或无效的依赖项

1 个答案: