是否有任何标志可避免源运算符和组合并之间的数据/元组混排

时间:2019-05-22 16:51:02

标签: apache-flink

我正在使用带有WordCount示例的GroupCombine和GroupReduce。但是与原始示例不同,我无法将 implementation 'com.github.ittianyu:BottomNavigationViewEx:1.2.4' implementation 'com.github.GrenderG:Toasty:1.4.2' implementation 'com.android.support:design:27.0.1' implementation 'com.android.support:appcompat-v7:27.0.1' implementation 'com.android.support.constraint:constraint-layout:1.0.2' implementation 'com.android.support:cardview-v7:27.0.1' implementation 'com.github.vivchar:ViewPagerIndicator:v1.0.1' implementation 'com.android.support:support-v4:27.0.1' implementation 'com.google.code.gson:gson:2.6.2' implementation 'com.squareup.retrofit2:retrofit:2.0.2' implementation 'com.squareup.retrofit2:converter-gson:2.0.2' implementation 'com.squareup.okhttp3:okhttp:3.3.1' implementation 'com.jakewharton.timber:timber:3.1.0' implementation 'com.squareup.okhttp3:logging-interceptor:3.2.0' implementation 'com.jakewharton.picasso:picasso2-okhttp3-downloader:1.0.2' implementation 'com.squareup.picasso:picasso:2.5.2' implementation 'com.android.support:multidex:1.0.1' implementation 'com.github.jd-alexander:LikeButton:0.2.3' implementation 'de.hdodenhof:circleimageview:2.1.0' implementation 'com.leo.simplearcloader:simplearcloader:1.0.+' implementation 'com.github.GrenderG:Toasty:1.2.5' implementation 'com.facebook.android:facebook-android-sdk:[4,5)' implementation 'com.facebook.android:facebook-share:[4,5)' implementation 'com.google.android.gms:play-services-auth:11.4.0' implementation 'com.google.firebase:firebase-core:11.4.0' implementation 'com.google.firebase:firebase-messaging:11.4.0' implementation 'com.google.firebase:firebase-ads:11.4.0' implementation 'com.anjlab.android.iab.v3:library:1.0.44' implementation 'com.getbase:floatingactionbutton:1.10.1' implementation 'com.github.shalskar:PeekAndPop:1.1.0' implementation 'com.google.android.exoplayer:exoplayer:r2.5.2' implementation 'com.facebook.android:audience-network-sdk:4.99.1' implementation 'com.google.android.ads.consent:consent-library:1.0.3' implementation 'devlight.io:navigationtabbar:1.2.5' implementation 'com.miguelcatalan:materialsearchview:1.4.0' implementation 'com.github.bhargavms:DotLoader:1.1.1' testImplementation 'junit:junit:4.12' androidTestImplementation 'com.android.support.test:runner:1.0.1' androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.1' } apply plugin: 'com.google.gms.google-services' 阶段与GroupCombine阶段结合在一起。这实际上意味着DataSource任务和DataSource任务之间将进行数据交换。这可能会降低工作性能。

我期望GroupCombine运算符应与GroupCombine任务结合使用。有没有标志可以帮助我做到这一点

根据DataSource的文档,可以看到

GroupCombine

这是我用来在本地组合然后减少所有本地组的代码段。

 * Generic interface used for combine functions ("combiners"). Combiners act as auxiliaries to a {@link GroupReduceFunction}
 * and "pre-reduce" the data. The combine functions typically do not see the entire group of elements, but
 * only a sub-group.

这是显示jobGraph的图 enter image description here GroupCombine阶段未与数据源阶段合并

仅供参考:我确实希望在GroupCombine和GroupReduce阶段之间可以进行洗牌,但不能在val counts = text.flatMap { _.toLowerCase.split("\\W+") filter { _.nonEmpty } } .map { (_, 1) } .combineGroup(new GroupCombineFunction[(String, Int), (String, Int)] { override def combine(values: lang.Iterable[(String, Int)], out: Collector[(String, Int)]): Unit = { values.iterator().asScala.toSeq.groupBy(p => p._1).map(t => (t._1, t._2.map(_._2).sum)).foreach(p => out.collect(p)) } }) .reduceGroup(new GroupReduceFunction[(String, Int), (String, Int)] { override def reduce(values: lang.Iterable[(String, Int)], out: Collector[(String, Int)]): Unit = { values.iterator().asScala.toSeq.groupBy(p => p._1).map(t => (t._1, t._2.map(_._2).sum)).foreach(p => out.collect(p)) } }) DataSource阶段之间进行洗牌。

0 个答案:

没有答案