我正在参加Scala Spark的 coursera课程，我正在尝试优化此片段：
val indexedMeansG = vectors.
map(v => findClosest(v, means) -> v).
groupByKey.mapValues(averageVectors)
vectors是RDD[(Int, Int)]，是为了查看依赖关系列表以及我使用过的RDD的血统：
println(s"""GroupBy:
| Deps: ${indexedMeansG.dependencies.size}
| Deps: ${indexedMeansG.dependencies}
| Lineage: ${indexedMeansG.toDebugString}""".stripMargin)
其中显示了这一点：
/* GroupBy:
* Deps: 1
* Deps: List(org.apache.spark.OneToOneDependency@44d1924)
* Lineage: (6) MapPartitionsRDD[18] at mapValues at StackOverflow.scala:207 []
* ShuffledRDD[17] at groupByKey at StackOverflow.scala:207 []
* +-(6) MapPartitionsRDD[16] at map at StackOverflow.scala:206 []
* MapPartitionsRDD[13] at map at StackOverflow.scala:139 []
* CachedPartitions: 6; MemorySize: 84.0 MB; ExternalBlockStoreSize: 0.0 B; DiskSize: 0.0 B
* MapPartitionsRDD[12] at values at StackOverflow.scala:116 []
* MapPartitionsRDD[11] at mapValues at StackOverflow.scala:115 []
* MapPartitionsRDD[10] at groupByKey at StackOverflow.scala:92 []
* MapPartitionsRDD[9] at join at StackOverflow.scala:91 []
* MapPartitionsRDD[8] at join at StackOverflow.scala:91 []
* CoGroupedRDD[7] at join at StackOverflow.scala:91 []
* +-(6) MapPartitionsRDD[4] at map at StackOverflow.scala:88 []
* | MapPartitionsRDD[3] at filter at StackOverflow.scala:88 []
* | MapPartitionsRDD[2] at map at StackOverflow.scala:69 []
* | src/main/resources/stackoverflow/stackoverflow.csv MapPartitionsRDD[1] at textFile at StackOverflow.scala:23 []
* | src/main/resources/stackoverflow/stackoverflow.csv HadoopRDD[0] at textFile at StackOverflow.scala:23 []
* +-(6) MapPartitionsRDD[6] at map at StackOverflow.scala:89 []
* MapPartitionsRDD[5] at filter at StackOverflow.scala:89 []
* MapPartitionsRDD[2] at map at StackOverflow.scala:69 []
* src/main/resources/stackoverflow/stackoverflow.csv MapPartitionsRDD[1] at textFile at StackOverflow.scala:23 []
* src/main/resources/stackoverflow/stackoverflow.csv HadoopRDD[0] at textFile at StackOverflow.scala:23 [] */
从这List(org.apache.spark.OneToOneDependency@44d1924)我推断出没有洗牌，我是对的吗？但是，打印ShuffledRDD[17]以下，这意味着实际上有洗牌。
我已尝试将groupByKey来电替换为reduceByKey，如下所示：
val indexedMeansR = vectors.
map(v => findClosest(v, means) -> v).
reduceByKey((a, b) => (a._1 + b._1) / 2 -> (a._2 + b._2) / 2)
它的依赖关系和血统是：
/* ReduceBy:
* Deps: 1
* Deps: List(org.apache.spark.ShuffleDependency@4d5e813f)
* Lineage: (6) ShuffledRDD[17] at reduceByKey at StackOverflow.scala:211 []
* +-(6) MapPartitionsRDD[16] at map at StackOverflow.scala:210 []
* MapPartitionsRDD[13] at map at StackOverflow.scala:139 []
* CachedPartitions: 6; MemorySize: 84.0 MB; ExternalBlockStoreSize: 0.0 B; DiskSize: 0.0 B
* MapPartitionsRDD[12] at values at StackOverflow.scala:116 []
* MapPartitionsRDD[11] at mapValues at StackOverflow.scala:115 []
* MapPartitionsRDD[10] at groupByKey at StackOverflow.scala:92 []
* MapPartitionsRDD[9] at join at StackOverflow.scala:91 []
* MapPartitionsRDD[8] at join at StackOverflow.scala:91 []
* CoGroupedRDD[7] at join at StackOverflow.scala:91 []
* +-(6) MapPartitionsRDD[4] at map at StackOverflow.scala:88 []
* | MapPartitionsRDD[3] at filter at StackOverflow.scala:88 []
* | MapPartitionsRDD[2] at map at StackOverflow.scala:69 []
* | src/main/resources/stackoverflow/stackoverflow.csv MapPartitionsRDD[1] at textFile at StackOverflow.scala:23 []
* | src/main/resources/stackoverflow/stackoverflow.csv HadoopRDD[0] at textFile at StackOverflow.scala:23 []
* +-(6) MapPartitionsRDD[6] at map at StackOverflow.scala:89 []
* MapPartitionsRDD[5] at filter at StackOverflow.scala:89 []
* MapPartitionsRDD[2] at map at StackOverflow.scala:69 []
* src/main/resources/stackoverflow/stackoverflow.csv MapPartitionsRDD[1] at textFile at StackOverflow.scala:23 []
* src/main/resources/stackoverflow/stackoverflow.csv HadoopRDD[0] at textFile at StackOverflow.scala:23 [] */
这一次，依赖关系是ShuffleDependency，我无法理解为什么。
由于RDD是一对键是Ints ，因此有一个排序，我也试图修改分区并使用RangePartitioner，但它没有＆ ＃39; t改善
答案 0 :(得分：3)
float x = 0, y = 0;
while (SDL_PollEvent(&event)){
switch (event.type) {
case SDL_QUIT: { board->setGameState(false); break;}
case SDL_FINGERDOWN: {
x = event.tfinger.x;
y = event.tfinger.y;
SDL_Log("\nDesplazamiento x: %f desplazamiento y: %f.\n", x, y);
window.get_AbsPixels(&x, &y);
if (x > keys->U.getX() && x < (keys->U.getX() + keys->U.getW()) &&
y > keys->U.getY() && y < keys->U.getY() + keys->U.getH()) {
SDL_Log("\nRetornado Up\n");
return Up;
}
if (x > keys->D.getX() && x < (keys->D.getX() + keys->D.getW()) &&
y > keys->D.getY() && y < keys->D.getY() + keys->D.getH()) {
SDL_Log("\nRetornado Down\n");
return Down;
}
if (x > keys->L.getX() && x < (keys->L.getX() + keys->L.getW()) &&
y > keys->L.getY() && y < keys->L.getY() + keys->L.getH()) {
SDL_Log("\nRetornado Left\n");
return Left;
}
if (x > keys->R.getX() && x < (keys->R.getX() + keys->R.getW()) &&
y > keys->R.getY() && y < keys->R.getY() + keys->R.getH()) {
SDL_Log("\nRetornado Right\n");
return Right;
}
break;
}
default:
//SDL_PumpEvents();
//SDL_FlushEvent(SDL_FINGERDOWN);
break;
}
}
return None;
操作仍然需要随机播放，因为仍然需要确保具有相同键的所有项都成为同一分区的一部分。
但是，与reduceByKey操作相比，这将是一个小得多的shuffle操作。 groupByKey将在混洗之前在每个分区内执行缩减操作，从而减少要洗牌的数据量。