Question

我正在努力学习一些非常基本的火花代码。我想定义一个包含2列的矩阵x。这就是我的尝试：

scala> val s = breeze.linalg.linspace(-3,3,5)
s: breeze.linalg.DenseVector[Double] = DenseVector(-3.0, -1.5, 0.0, 1.5, 3.0) // in this case I want s to be both column 1 and column 2 of x

scala> val ss = s.toArray ++ s.toArray
ss: Array[Double] = Array(-3.0, -1.5, 0.0, 1.5, 3.0, -3.0, -1.5, 0.0, 1.5, 3.0)

scala> import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.mllib.linalg.distributed.RowMatrix

scala> val mat = new RowMatrix(ss, 5, 2)
<console>:17: error: type mismatch;
 found   : Array[Double]
 required: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector]
       val mat = new RowMatrix(ss, 5, 2)

我不明白如何才能将正确的转换传递给分布式矩阵^

编辑：也许我已经能够解决：

scala> val s = breeze.linalg.linspace(-3,3,5)
s: breeze.linalg.DenseVector[Double] = DenseVector(-3.0, -1.5, 0.0, 1.5, 3.0)

scala> val ss = s.to
toArray         toDenseMatrix   toDenseVector   toScalaVector   toString        
toVector        

scala> val ss = s.toArray ++ s.toArray
ss: Array[Double] = Array(-3.0, -1.5, 0.0, 1.5, 3.0, -3.0, -1.5, 0.0, 1.5, 3.0)

scala> val x = new breeze.linalg.Dense
DenseMatrix   DenseVector   

scala> val x = new breeze.linalg.DenseMatrix(5, 2, ss)
x: breeze.linalg.DenseMatrix[Double] = 
-3.0  -3.0  
-1.5  -1.5  
0.0   0.0   
1.5   1.5   
3.0   3.0   

scala> val xDist = sc.parallelize(x.toArray)
xDist: org.apache.spark.rdd.RDD[Double] = ParallelCollectionRDD[0] at parallelize at <console>:18

Answer 1

像这样的东西。这个类型检查，但由于某种原因不能在我的Scala工作表中运行。

import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.linalg.distributed._
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD

val conf = new SparkConf().setAppName("spark-scratch").setMaster("local")
val sc= new SparkContext(conf)

// the values for the column in each row
val col = List(-3.0, -1.5, 0.0, 1.5, 3.0) ;

// make two rows of the column values, transpose it,
// make Vectors of the result
val t = List(col,col).transpose.map(r=>Vectors.dense(r.toArray))

// make an RDD from the resultant sequence of Vectors, and 
// make a RowMatrix from that.
val rm = new RowMatrix(sc.makeRDD(t));

Spark中的简单矩阵乘法

1 个答案: