Question

我正在使用Spark Mllib对零售行业进行链接分析项目。我的架构是：

ID - Long Chain - Int Dept - int Category - Int Company - Long Brand - Long Date - Date ProductSize - Int ProductMeasure - Chararray PurchaseQuantity - Int PurchaseAmount - Double

我使用的代码是：

spark-shell
import org.apache.spark._
import org.apache.spark.rdd.RDD
import org.apache.spark.util.IntParam
import org.apache.spark.graphx._
import org.apache.spark.graphx.util.GraphGenerators

case class Transactions(ID:Long,Chain:String,Dept:String,Category:String,Company:String,Brand:String,Date:String,ProductSize:String,ProductMeasure:String,PurchaseQuantity:String,PurchaseAmount:String)

def parseTransactions(str:String): Transactions = {
     val line = str.split(",")
     Transactions(line(0).toLong,line(1),line(2),line(3),line(4),line(5),line(6),line(7),line(8),line(9),line(10))
     }

val textRDD = sc.textFile("/user/cloudera/transactions.csv")     

val transactionsRDD = textRDD.map(parseTransactions).cache()

val products = transactionsRDD.map(Transactions => (Transactions.ID,Transactions.Chain,Transactions.Dept,Transactions.Category,Transactions.Company,Transactions.Brand)).distinct

products.take(1)

val productMap = products.map { case ((ID), name) => (ID -> name) }.collect.toList.toMap

我收到了以下错误：

<console>:46: error: constructor cannot be instantiated to expected type;
 found   : (T1, T2)
 required: (Long, String, String, String, String, String)
         val productMap = products.map { case ((ID), name) => (ID -> name) }.collect.toList.toMap
                                              ^
<console>:46: error: not found: value ID
         val productMap = products.map { case ((ID), name) => (ID -> name) }.collect.toList.toMap
                                                               ^
<console>:46: error: value toList is not a member of Array[Nothing]
         val productMap = products.map { case ((ID), name) => (ID -> name) }.collect.toList.toMap
                                                                                     ^

任何人都知道我做错了什么？

非常感谢！

Answer 1

// this produces a collection of tuples of type (Long, String, String, String, String, String)
val products = transactionsRDD.map(Transactions => (Transactions.ID,Transactions.Chain,Transactions.Dept,Transactions.Category,Transactions.Company,Transactions.Brand)).distinct

// since products is a collection of sextuples, the pattern match should match the tuple length
val productMap = products.map { case (ID, chain, dept, cat, comp, brand) => (ID -> chain) }.collect.toList.toMap

（Spark - Scala）错误：构造函数无法实例化为期望的类型;

1 个答案: