如何在Flink

时间:2018-06-24 18:51:21

标签: apache-flink flink-streaming

需要使用parallelCamTasks(以args形式传递)通过cameraWithCube.getCam()1进行分区。

然后在每个分区内,需要通过cameraWithCube.getTs()再次进行分区,但需要确保每个 getTS()的第二个分区在同一物理节点上运行?

我该如何实现?

DataStream<CameraWithCube> cameraWithCubeDataStream = env
            .addSource(new Source(....))
            .keyBy((cameraWithCube) -> cameraWithCube.getCam() )
            .process(new ProcessFunction<CameraWithCube, CameraWithCube>() {
                public void processElement(CameraWithCube cameraWithCube, Context context, Collector<CameraWithCube> collector) throws Exception {
                    //do nothing
                }
            })
            .slotSharingGroup("camSharingGroup")//TODO: how to add camera# of the partition
            .setParallelism(parallelCamTasks)
            .keyBy((cameraWithCube) -> cameraWithCube.getTs())
            .process(new ProcessFunction<CameraWithCube, CameraWithCube>() {
                public void processElement(CameraWithCube cameraWithCube, Context context, Collector<CameraWithCube> collector) throws Exception {
                    //TODO: process code
                }
            })
            .setParallelism(noOfSlotsInEachPhysicalNode)//TODO: how many parallel tasks within physical node
            .slotSharingGroup("??");//TODO: in same physical node

TIA

0 个答案:

没有答案