Question

我试图反转有向图并将每个顶点的邻接列表以格式

写入文本文件

NodeId \t NeighbourId1,NeighbourId2,...,NeighbourIdn

到目前为止，我只尝试打印输出，如下所示：

(4,[J@13ad83aa)
(0,[J@338ff780)
(1,[J@6737f62b)
(3,[J@1250d788)
(2,[J@6d1fa6bb)

它应该是格式：

我目前使用的代码是

object Problem2{
def main(args: Array[String]){
val inputFile:String = args(0)
val outputFolder = args(1)
val conf = new SparkConf().setAppName("Problem2").setMaster("local")
val sc = new SparkContext(conf)

val graph = GraphLoader.edgeListFile(sc,inputFile)
val edges = graph.reverse.edges
val vertices = graph.vertices
val newGraph = Graph(vertices,edges)

val verticesWithSuccessors: VertexRDD[Array[VertexId]] = 
newGraph.ops.collectNeighborIds(EdgeDirection.Out)

val successorGraph = Graph(verticesWithSuccessors, edges)
val res = successorGraph.vertices.collect()

val adjList = successorGraph.vertices.foreach(println)

我不认为mkString()可以与图形对象一起使用。图表对象是否有类似的方法来获取字符串？

Answer 1

让我们再看一下这个例子：

val vertices: RDD[(VertexId, String)] =  
    sc.parallelize(Array((1L,""), (2L,""), (4L,""), (6L,"")))


val edges: RDD[Edge[String]] = 
    sc.parallelize(Array(
        Edge(1L, 2L, ""),
        Edge(1L, 4L, ""),
        Edge(1L, 6L, "")))
val inputGraph = Graph(vertices, edges)

val verticesWithSuccessors: VertexRDD[Array[VertexId]] = 
    inputGraph.ops.collectNeighborIds(EdgeDirection.Out)
val successorGraph = Graph(verticesWithSuccessors, edges)

一旦你有了这个：

val adjList = successorGraph.vertices

您可以轻松翻译成DataFrame：

val df = adjList.toDF(Seq("node", "adjacents"): _*)
df.show()
+----+---------+
|node|adjacents|
+----+---------+
|   1|[2, 4, 6]|
|   2|       []|
|   4|       []|
|   6|       []|
+----+---------+

现在可以轻松地使用列进行转换。这里有一个不那么漂亮的例子：

val result = df.rdd.collect().map(l=> l(0).asInstanceOf[Long] + "\t"  + l(1).asInstanceOf[Seq[Long]].mkString(" "))
result.foreach(println(_))

1   2 4 6
2   
4   
6

或者您也可以尝试使用UDF或根据需要处理列。

希望这有帮助！

Scala将图形的每个节点的邻接列表写入文本文件

1 个答案: