我在使用IDE向生产集群提交拓扑时遇到问题Must submit topologies using the 'storm' client script so that StormSubmitter knows which jar to upload
,而如果我使用storm jar
命令在命令行中执行,它就像天堂一样运行。我从githublink看过相同的例子。
提交拓扑我正在使用这些行
conf.put(Config.NIMBUS_HOST, NIMBUS_NODE);
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,ZOOKEEPER_ID);
conf.setNumWorkers(20);
conf.setMaxSpoutPending(5000);
StormSubmitter submitter = new StormSubmitter();
submitter.submitTopology("test", conf, builder.createTopology());
如果这是正确的方法,请建议我吗?
答案 0 :(得分:26)
很好地找到了解决方案。当我们运行“暴风罐”时,它会在提交的jar中触发storm.jar的属性标志。因此,如果我们想以编程方式提交jar,那么只需按照这种方式设置标志
System.setProperty("storm.jar", <path-to-jar>);
例如:
System.setProperty("storm.jar", "/Users/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
StormSubmitter.submitTopology("myTopology", config, builder.createTopology());
答案 1 :(得分:5)
要向远程Storm群集提交拓扑,您需要将该jar上传到nimbus机器,然后使用NimbusClient将该jar提交到Cluster。 你可以这样做:
Map storm_conf = Utils.readStormConfig();
storm_conf.put("nimbus.host", "<Nimbus Machine IP>");
Client client = NimbusClient.getConfiguredClient(storm_conf)
.getClient();
String inputJar = "C:\\workspace\\TestStormRunner\\target\\TestStormRunner-0.0.1-SNAPSHOT-jar-with-dependencies.jar";
NimbusClient nimbus = new NimbusClient(storm_conf, "<Nimbus Machine IP>",
<Nimbus Machine Port>);
// upload topology jar to Cluster using StormSubmitter
String uploadedJarLocation = StormSubmitter.submitJar(storm_conf,
inputJar);
String jsonConf = JSONValue.toJSONString(storm_conf);
nimbus.getClient().submitTopology("testtopology",
<uploadedJarLocation>, jsonConf, builder.createTopology());
答案 2 :(得分:4)
我没有运行java代码来提交自己,但我检查了storm命令 - 它是一个python文件,它运行java和http://nathanmarz.github.com/storm/doc/backtype/storm/StormSubmitter.html类
我认为你唯一应该担心的是 - 在执行时包含所有需要的库。
答案 3 :(得分:3)
我已经根据@ abhi和@Nishu Tayal的答案解决了这个问题,我想在这里发布我的代码:
public static void submitLocalTopologyWay1(String topologyName, Config topologyConf,
StormTopology topology, String localJar) {
try {
//get default storm config
Map defaultStormConf = Utils.readStormConfig();
defaultStormConf.putAll(topologyConf);
//set JAR
System.setProperty("storm.jar",localJar);
//submit topology
StormSubmitter.submitTopology(topologyName, defaultStormConf, topology);
} catch (Exception e) {
String errorMsg = "can't deploy topology " + topologyName + ", " + e.getMessage();
System.out.println(errorMsg);
e.printStackTrace();
}
}
public static void submitLocalTopologyWay2(String topologyName, Config topologyConf,
StormTopology topology, String localJar) {
try {
//get nimbus client
Map defaultStormConf = Utils.readStormConfig();
defaultStormConf.putAll(topologyConf);
Client client = NimbusClient.getConfiguredClient(defaultStormConf).getClient();
//upload JAR
String remoteJar = StormSubmitter.submitJar(defaultStormConf, localJar);
//submit topology
client.submitTopology(topologyName, remoteJar, JSONValue.toJSONString(topologyConf), topology);
} catch (Exception e) {
String errorMsg = "can't deploy topology " + topologyName + ", " + e.getMessage();
System.out.println(errorMsg);
e.printStackTrace();
}
}
然后这是一个测试,您必须首先将代码构建到JAR文件。
public void testSubmitTopologySubmitLocalTopologyWay1() {
Config config = new Config();
config.put(Config.NIMBUS_HOST,"9.119.84.179");
config.put(Config.NIMBUS_THRIFT_PORT, 6627);
config.put(Config.STORM_ZOOKEEPER_SERVERS, Arrays.asList("9.119.84.177","9.119.84.178","9.119.84.176"));
config.put(Config.STORM_ZOOKEEPER_PORT,2181);
config.put(Config.TOPOLOGY_WORKERS, 3);
RemoteSubmitter.submitLocalTopologyWay1("word-count-test-1", config,
WordCountTopology.buildTopology(), // your topology
"C:\\MyWorkspace\\project\\storm-sample-0.0.1-SNAPSHOT-jar-with-dependencies.jar");//the JAR file
}