我试图在Cloudera环境中将Kafka(Kafka_2.10版本0.8.2.1)与Storm(版本0.9.3)集成,并为生产者/消费者编写了一些代码。我能够与Kafka分别运行生产者代码,并且看到它正在使用我的消费者代码(在控制台上)。然后我使用KafkaSpout和HDFSBolt编写了一些代码来将数据写入HDFS。使用此代码,我可以创建拓扑(并在UI中查看),但KafkaSpout没有从生产者那里接收任何消息。
我的代码段如下所示:
public class LoadingData {
public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException {
String kafkaTopic = "test";
SpoutConfig spoutConfig = new SpoutConfig(new ZkHosts("localhost:2181"),
kafkaTopic, "/kafkastorm", "KafkaSpout");
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("KafkaSpout", new KafkaSpout(spoutConfig),4);
RecordFormat format = new DelimitedRecordFormat().withFieldDelimiter(",");
SyncPolicy syncPolicy = new CountSyncPolicy(10);
FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(5.0f, Units.MB);
FileNameFormat fileNameFormat = new DefaultFileNameFormat().withPath("/stormstuff");
builder.setBolt("stormbolt", new HdfsBolt()
.withFsUrl("hdfs://localhost:8020")
.withSyncPolicy(syncPolicy)
.withRecordFormat(format)
.withRotationPolicy(rotationPolicy)
.withFileNameFormat(fileNameFormat),1
).shuffleGrouping("KafkaSpout");
String topologyName = "EmployeeTopology";
Config config = new Config();
config.setNumWorkers(1);
StormSubmitter.submitTopology(topologyName, config, builder.createTopology());
}
}
关于我可能做错的任何想法/建议?我真的很感谢你的帮助!如果您需要更多详细信息,请与我们联系。