Question

我正在运行以下代码，尝试在Apache Spark中创建GraphX中的Graph。

import org.apache.spark.SparkConf

import org.apache.spark.SparkContext

import org.apache.spark.graphx.GraphLoader 

import org.apache.spark.graphx.Graph

import org.apache.spark.rdd.RDD
import org.apache.spark.graphx.VertexId

//loads file from the array

val lines = sc.textFile("hdfs://moonshot-ha-nameservice/data/google-plus/2309.graph");

//maps lines and takes the first 21 characters of each line which is the node.

val result = lines.map( line => line.substring(0,20))

//creates a new variable with each node followed by a long .

val result2 = result.map(word => (word,1L).toLong)

//where i am getting an error

val vertexRDD: RDD[(Long,Long)] = sc.parallelize(result2)

我收到以下错误：

 error: type mismatch;

 found   : org.apache.spark.rdd.RDD[(Long, Long)]

 required: Seq[?]

Error occurred in an application involving default arguments.
         val vertexRDD: RDD[(Long, Long)] = sc.parallelize(result2)

Answer 1

首先，您的地图可以简化为以下代码：

val vertexRDD: RDD[(Long, Long)] = 
  lines.map(line => (line.substring(0, 17).toLong, 1L))

现在，您的错误：您无法使用RDD呼叫sc.parallelize。您的vertexRDD已由result2定义。然后，您可以使用result2和EdgesRDD创建图表：

val g = Graph(result2, edgesRDD)

或者，如果使用我的建议：

val g = Graph(vertexRDD, edgesRDD)

VertexRDD给出了类型不匹配错误

1 个答案: