更新
添加
env.getConfig().setAutoWatermarkInterval(1000L);
没有解决问题。
我想问题出在我代码的另一部分。所以首先要多一点背景。
该程序从单个kafka队列中使用混合消息类型的JSON流。程序最初转换为ObjectNode类型的流。然后使用.split()将此流拆分为大约10个单独的流。这些流映射到POJO流。
扩展代码示例
public class flinkkafka {
public static void main(String[] args) throws Exception {
//create object mapper to allow object to JSON transform
final ObjectMapper mapper = new ObjectMapper();
final String OUTPUT_QUEUE = "test";
//setup streaming environment
StreamExecutionEnvironment env =
StreamExecutionEnvironment
.getExecutionEnvironment();
//set streaming environment variables from command line
ParameterTool parameterTool = ParameterTool.fromArgs(args);
//set time characteristic to EventTime
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
//set watermark polling interval
env.getConfig().setAutoWatermarkInterval(1000L);
//Enable checkpoints to allow for graceful recovery
env.enableCheckpointing(1000);
//set parallelism
env.setParallelism(1);
//create an initial data stream of mixed messages
DataStream<ObjectNode> messageStream = env.addSource
(new FlinkKafkaConsumer09<>(
parameterTool.getRequired("topic"),
new JSONDeserializationSchema(),
parameterTool.getProperties()))
.assignTimestampsAndWatermarks(new
BoundedOutOfOrdernessTimestampExtractor<ObjectNode>
(Time.seconds(10)){
private static final long serialVersionUID = 1L;
@Override
public long extractTimestamp(ObjectNode value) {
DateFormat format = new SimpleDateFormat("yyyy-
MM-dd HH:mm:ss", Locale.ENGLISH);
long tmp = 0L;
try {
tmp =
format.parse(value.get("EventReceivedTime")
.asText()).getTime();
} catch (ParseException e) {
e.printStackTrace();
}
System.out.println("Assigning timestamp " +
tmp);
return tmp;
}
});
//split stream by message type
SplitStream<ObjectNode> split = messageStream.split(new
OutputSelector<ObjectNode>(){
private static final long serialVersionUID = 1L;
@Override
public Iterable<String> select(ObjectNode value){
List<String> output = new ArrayList<String>();
switch (value.get("name").asText()){
case "one":
switch (value.get("info").asText()){
case "two":
output.add("info");
System.out.println("Sending message to two
stream");
break;
case "three":
output.add("three");
System.out.println("Sending message to three stream");
break;
case "four":
output.add("four");
System.out.println("Sending message to four stream");
break;
case "five":
output.add("five");
System.out.println("Sending message to five stream");
break;
case "six":
output.add("six");
System.out.println("Sending message to six stream");
break;
default:
break;
}
break;
case "seven":
output.add("seven");
System.out.println("Sending message to seven stream");
break;
case "eight":
output.add("eight");
System.out.println("Sending message to eight stream");
break;
case "nine":
output.add("nine");
System.out.println("Sending message to nine stream");
break;
case "ten":
switch (value.get("info").asText()){
case "eleven":
output.add("eleven");
System.out.println("Sending message to eleven stream");
break;
case "twelve":
output.add("twelve");
System.out.println("Sending message to twelve stream");
break;
default:
break;
}
break;
default:
output.add("failed");
break;
}
return output;
}
});
//assign splits to new data streams
DataStream<ObjectNode> two = split.select("two");
//assigning more splits to streams
//convert ObjectNodes to POJO
DataStream<Two> twoStream = two.map(new MapFunction<ObjectNode, Two>(){
private static final long serialVersionUID = 1L;
@Override
public Twomap(ObjectNode value) throws Exception {
Two stream = new Two();
stream.Time = value.get("Time").asText();
stream.value = value.get("value").asLong();
return front;
}
});
DataStream<String> keyedTwo = twoStream
.keyBy("name")
.timeWindow(Time.minutes(5))
.apply(new twoSum())
.map(new MapFunction<Two, String>(){
private static final long serialVersionUID = 1L;
@Override
public String map(Two value) throws Exception {
return mapper.writeValueAsString(value);
}
});
keyedTwo.addSink(new FlinkKafkaProducer09<String>
(parameterTool.getRequired("bootstrap.servers"),
OUTPUT_QUEUE, new SimpleStringSchema()));
env.execute();
我正在尝试使用Flink聚合Kafka队列并将数据流推送回Kafka。聚合将使用5分钟的事件时间窗口，程序编译并运行，但收集的数据永远不会将窗口传递给聚合函数，因此永远不会向Kafka传递消息。但是，如果我注释掉eventTime特性，程序就会运行并产生结果。我不知道我哪里出错了。
EventTime代码
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
ParameterTool parameterTool = ParameterTool.fromArgs(args);
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.enableCheckpointing(1000);
DataStream<FrontEnd> frontEndStream = frontEnd.map(new
MapFunction<ObjectNode, FrontEnd>(){
private static final long serialVersionUID = 1L;
@Override
public FrontEnd map(ObjectNode value) throws Exception {
FrontEnd front = new FrontEnd();
front.eventTime = value.get("EventReceivedTime").asText();
return front;
}
}).assignTimestampsAndWatermarks(new
BoundedOutOfOrdernessTimestampExtractor<FrontEnd>(Time.seconds(10)){
private static final long serialVersionUID = 1L;
@Override
public long extractTimestamp(FrontEnd value) {
DateFormat format = new SimpleDateFormat("yyyy-MM-
ddHH:mm:ss",Locale.ENGLISH);
long tmp = 0L;
try {
tmp = format.parse(value.eventTime).getTime();
} catch (ParseException e) {
e.printStackTrace();
}
return tmp;
}
});
DataStream<String> keyedFrontEnd = frontEndStream
.keyBy("name")
.timeWindow(Time.minutes(5))
.apply(new FrontEndSum())
.map(new MapFunction<FrontEnd, String>(){
private static final long serialVersionUID = 1L;
@Override
public String map(FrontEnd value) throws Exception {
return mapper.writeValueAsString(value);
}
});
.map(new MapFunction<FrontEnd, String>(){
private static final long serialVersionUID = 1L;
@Override
public String map(FrontEnd value) throws Exception {
return mapper.writeValueAsString(value);
}
});
keyedFrontEnd.addSink(new FlinkKafkaProducer09<String>
(parameterTool.getRequired("bootstrap.servers"), OUTPUT_QUEUE, new
SimpleStringSchema()));
env.execute();
}
}
我尝试使用附加到传入流的时间戳提取器，并将一个附加到每个POJO流。同样，此代码与事件时间一起运行，并生成具有预期聚合的JSON字符串流的预期结果。但是，一旦启用了事件时间，窗口就不会产生结果
答案 0 :(得分：1)
AssignerWithPeriodicWatermarks实现ExecutionConfig接口，这意味着Flink会定期查询当前水印。
您必须通过env.getConfig.setAutoWatermarkInterval(1000L); // poll watermark every second
：
[ [001, 'A', '100'], [001, 'B', '94'], [002, 'A', '87'], [002, 'B', '85'] ]
答案 1 :(得分：0)
我的第一个倾向是始终假设TimeZone问题。
kafka有效负载中"EventReceivedTime"字段的时区是什么？
SimpleDateFormat将在LOCAL JVM时区解析：
DateFormat format = new SimpleDateFormat("yyyy-MM-ddHH:mm:ss",Locale.ENGLISH);
你可以添加
format.setTimeZone(TimeZone.getTimeZone("GMT"));
将您的字符串解析为GMT，例如，如果这是文本所代表的内容。您应该确保所有日期，水印等的时区/偏移量都匹配，并且全部按UTC /纪元时间进行比较（这是您从Long中提取Long后得到的结果）。