Question

我正在将Flink 1.8流与kafka 2.x（kafka_2.12-2.2.0，kafka_2.12-2.3.0）集成，并在启动flink作业几秒钟后遇到异常：

    org.apache.kafka.common.protocol.types.SchemaException: Error computing size for field 'topics': Error computing size for field 'name': Missing value for field 'name' which has no default value.

我知道kafka客户端的版本必须与kafka代理兼容。然后我再次检查了他们是否使用了相同的版本2.3.0。我的Flink作业项目中的一些相关依赖项：

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>2.3.0</version>
    </dependency>

    <properties>
    <scala.binary.version>2.12</scala.binary.version>
    <jackson.version>2.9.8</jackson.version>
    <flink.shaded.version>7.0</flink.shaded.version>
    <flink.version>1.8.0</flink.version>
</properties>

我使用FlinkKafkaConsumer作为Flink源，以如下方式使用来自kafka的数据：

Properties properties = new Properties();
    properties.put("group.id", "consumer");
    properties.put("bootstrap.servers", bootstrap);
    properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, JSONKeyValueDeserializationSchema.class.getName());
    properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JSONKeyValueDeserializationSchema.class.getName());
    DataStream<ObjectNode> inputStream = env.addSource(
            new FlinkKafkaConsumer<ObjectNode>(
                kafkaInputTopic, new JSONKeyValueDeserializationSchema(true), properties));

例外的详细信息：

org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 33e0a92790c3afa0c886fdb56b8b81b4)
        at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
        at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:483)
        at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
        at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1509)
        at com.stream.processor.main(processor.java:156)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
        at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
        at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
        at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
        at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
        at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126) Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
        at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
        at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
        ... 18 more


        Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error computing size for field 'topics': Error computing size for field 'name': Missing value  for field 'name' which has no default value.
        at org.apache.kafka.common.protocol.types.Schema.sizeOf(Schema.java:128)
        at org.apache.kafka.common.protocol.types.Struct.sizeOf(Struct.java:435)
        at org.apache.kafka.common.requests.AbstractRequestResponse.serialize(AbstractRequestResponse.java:28)
        at org.apache.kafka.common.requests.AbstractRequest.serialize(AbstractRequest.java:103)
        at org.apache.kafka.common.requests.AbstractRequest.toSend(AbstractRequest.java:96)
        at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:503)
        at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:474)
        at org.apache.kafka.clients.NetworkClient.send(NetworkClient.java:434)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:475)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:282)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
        at org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:347)
        at org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1842)
        at org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1810)
        at org.apache.flink.streaming.connectors.kafka.internal.KafkaPartitionDiscoverer.getAllPartitionsForTopics(KafkaPartitionDiscoverer.java:77)
        at org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.discoverPartitions(AbstractPartitionDiscoverer.java:131)
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:507)
        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
        at java.lang.Thread.run(Thread.java:745)

Answer 1

发现由于传递给job的topic参数为null而引发异常。

Flink作业引发SchemaException：计算字段“主题”的大小时出错

1 个答案: