Question

我正在试用Flink的新Python流API，并尝试使用./flink-1.6.1/bin/pyflink-stream.sh examples/read_from_kafka.py运行脚本。 python脚本非常简单，我只是想从一个现有的主题中使用，并将所有内容发送到stdout（或默认情况下output方法在其中发出输出数据的日志目录中的* .out文件）。

import glob
import os
import sys
from java.util import Properties
from org.apache.flink.streaming.api.functions.source import SourceFunction
from org.apache.flink.streaming.api.collector.selector import OutputSelector
from org.apache.flink.api.common.serialization import SimpleStringSchema

directories=['/home/user/flink/flink-1.6.1/lib']
for directory in directories:
    for jar in glob.glob(os.path.join(directory,'*.jar')):
                sys.path.append(jar)

from org.apache.flink.streaming.connectors.kafka import FlinkKafkaConsumer09

props = Properties()
config = {"bootstrap_servers": "localhost:9092",
          "group_id": "flink_test",
          "topics": ["TopicCategory-TopicName"]}
props.setProperty("bootstrap.servers", config['bootstrap_servers'])
props.setProperty("group_id", config['group_id'])
props.setProperty("zookeeper.connect", "localhost:2181")

def main(factory):
    consumer = FlinkKafkaConsumer09([config["topics"]], SimpleStringSchema(), props)

    env = factory.get_execution_environment()
    env.add_java_source(consumer) \
        .output()
    env.execute()

我从Maven仓库中抓取了一些jar文件，分别是flink-connector-kafka-0.9_2.11-1.6.1.jar，flink-connector-kafka-base_2.11-1.6.1.jar和kafka-clients-0.9.0.1.jar，并将它们复制到Flink的lib目录中。除非我误解了文档，否则Flink足以加载kafka连接器。确实，如果我删除所有这些jar，导入都会失败，但这似乎不足以实际调用该计划。添加for循环以将其动态添加到sys.path也不起作用。这是在控制台中打印的内容：

Starting execution of program
Failed to run plan: null
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/flink_streaming_plan_9cfed4d9-0288-429c-99ac-df02c86922ec/read_from_kafka.py", line 32, in main
    at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:267)
    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:486)
    at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
    at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1511)
    at org.apache.flink.streaming.python.api.environment.PythonStreamExecutionEnvironment.execute(PythonStreamExecutionEnvironment.java:245)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)

org.apache.flink.client.program.ProgramInvocationException: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: bbcc0cb2c4fe6e3012d228b06b270eba)

The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment.

这是我在日志中看到的：

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class:    org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09
ClassLoader info: URL ClassLoader:
    file: '/tmp/blobStore-9f6930fa-f1cf-4851-a0bf-2e620391596f/job_ca486746e7feb42d2d162026b74e9935/blob_p-9321896d165fec27a617d44ad50e3ef09c3211d9-405ccc9b490fa1e1348f0a76b1a48887' (valid JAR)
Class not resolvable through given classloader.
    at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:236)
    at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:104)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
    at java.lang.Thread.run(Thread.java:748)

是否有办法解决此问题并使连接器可用于Python？我怀疑这是Jython的Classloader问题，但我不知道如何进一步研究（同样鉴于我不具备Java知识）。非常感谢。

Answer 1

您在这里使用了错误的Kafka用户。在您的代码中，它是FlinkKafkaConsumer09，但是您使用的库是flink-connector-kafka-0.11_2.11-1.6.1.jar，它是FlinkKafkaConsumer011的库。尝试用此FlinkKafkaConsumer09替换FlinkKafkaConsumer011，或使用lib文件flink-connector-kafka-0.9_2.11-1.6.1.jar而不是当前文件。

Answer 2

我邀请jar文件可能具有内置的导入或依赖关系，因此这三个jar文件是不够的。关于如何找出java jar依赖关系，这就是java maven所做的事情。您可以查看官方网站“项目构建设置”以获取帮助。就我而言，我遵循官方的Java项目设置，使用“从org.apache.flink.streaming.connectors.kafka导入FlinkKafkaConsumer”并添加依赖项“                org.apache.flink
               flink-clients_2.11
               1.8.0“到pom.xml，然后我现在可以使用Python API将kafka记录输出到stdout。

Apache Flink：Python流API中的Kafka连接器，“无法加载用户类”

2 个答案: