Question

我需要通过flink使用kafka，不幸的是，kafka消息正在Serde中使用protobuf，完全不知道如何处理它，这是来自Internet的代码，但是我无法使其工作。 / p>

...    
import com.google.protobuf.InvalidProtocolBufferException;
import com.twitter.chill.protobuf.ProtobufSerializer;

public class Protobuf2 {

    public static void main(String[] args) throws Exception {
        String inputTopic = "zz";
        String consumerGroup = "foobar";
        String address = "localhost:9092";
        StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();
        environment.getConfig().registerTypeWithKryoSerializer(ExchangeMessage.Order.class, ProtobufSerializer.class);

        FlinkKafkaConsumer<ExchangeMessage.Order> flinkKafkaConsumer = createStringConsumerForTopic(inputTopic, address,
                consumerGroup);
        DataStream<ExchangeMessage.Order> input = environment.addSource(flinkKafkaConsumer);

        input.map(order ->
            {
                System.out.println("foooooooo, " + order.getOid());
                return order;
            });

        environment.execute("kafka02");
    }

    public static FlinkKafkaConsumer<ExchangeMessage.Order> createStringConsumerForTopic(String topic,
            String kafkaAddress, String kafkaGroup) {
        Properties props = new Properties();
        props.setProperty("bootstrap.servers", kafkaAddress);
        props.setProperty("group.id", kafkaGroup);
        FlinkKafkaConsumer<ExchangeMessage.Order> consumer = new FlinkKafkaConsumer<>(topic,
                new OrderSerDeSchema(), props);
        return consumer;
    }


    @SuppressWarnings("serial")
    static class OrderSerDeSchema
            implements DeserializationSchema<ExchangeMessage.Order>, SerializationSchema<ExchangeMessage.Order> {

        @Override
        public ExchangeMessage.Order deserialize(byte[] message) throws IOException {
            ExchangeMessage.Order order = null;
            try {
                order = ExchangeMessage.Order.parseFrom(message);
            } catch (InvalidProtocolBufferException e) {
                e.printStackTrace();
            }
            return order;
        }

        @Override
        public boolean isEndOfStream(ExchangeMessage.Order nextElement) {
            return false;
        }

        @Override
        public TypeInformation<ExchangeMessage.Order> getProducedType() {
            return null;
        }

        @Override
        public byte[] serialize(ExchangeMessage.Order element) {
            return new byte[0];
        }
    }

}

这不起作用，它提供了NPE：

Caused by: java.lang.NullPointerException
    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:577)
    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
    at ......

有人知道我在做什么错吗？这是唯一一种放弃Protobuf的方法-使用Twitter ProtobufSerializer吗？还是还有另一条路要走？

Answer 1

添加依赖项

<dependency>
    <groupId>com.twitter</groupId>
    <artifactId>chill-protobuf</artifactId>
    <version>0.7.6</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-core</artifactId>
    <version>1.8.0</version>
</dependency>

并将其添加到您的代码中

env.getConfig().registerTypeWithKryoSerializer(MockMessageProto.MockMessage.class, ProtobufSerializer.class);

也许您会遇到版本冲突，请更改chill-protobuf版本以进行修复

Flink，如何使用ProtobufSerializer反序列化Protobuf？

1 个答案: