Question

我有带有docker的服务器并创建3个Cassandra节点，2个工作者spark节点和一个主要spark节点。现在我想通过java应用程序从我的笔记本电脑连接到spark。我的java代码是：

public SparkTestPanel(String id, User user) {
    super(id);
    form = new Form("form");
    form.setOutputMarkupId(true);
    this.add(form);
    SparkConf conf = new SparkConf(true);
    conf.setAppName("Spark Test");
    conf.setMaster("spark://172.11.100.156:9050");
    conf.set("spark.cassandra.connection.host", "cassandra-0");
    conf.set("spark.cassandra.connection.port", "9042");
    conf.set("spark.cassandra.auth.username", "cassandra");
    conf.set("spark.cassandra.auth.password", "cassandra"); 
    JavaSparkContext sc = null;
    try {
        sc = new JavaSparkContext(conf);
        CassandraTableScanJavaRDD<com.datastax.spark.connector.japi.CassandraRow> cassandraTable = javaFunctions(sc).cassandraTable("test", "test_table");

        List<com.datastax.spark.connector.japi.CassandraRow> collect = cassandraTable.collect();
        for(com.datastax.spark.connector.japi.CassandraRow cassandraRow : collect){
            Logger.getLogger(SparkTestPanel.class).error(cassandraRow.toString());
        }
    } finally {
        sc.stop();
    }

}

我知道应用程序连接到spark master，因为我在网上看到了我的应用程序，但是在网上点亮：

CassandraTableScanJavaRDD<com.datastax.spark.connector.japi.CassandraRow> cassandraTable = javaFunctions(sc).cassandraTable("test", "test_table");

我得到以下错误：

2017-08-17 12:14:31,906 ERROR CassandraConnectorConf:72 - Unknown host 'cassandra-0'
java.net.UnknownHostException: cassandra-0: nodename nor servname provided, or not known
...

还有其他错误：

Caused by: java.lang.IllegalArgumentException: Cannot build a cluster without contact points
at com.datastax.driver.core.Cluster.checkNotEmpty(Cluster.java:119)
at com.datastax.driver.core.Cluster.<init>(Cluster.java:112)
at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178)
at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1335)
at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:131)
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:159)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154)
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32)
at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57)
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:79)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111)
at com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:122)
at com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:330)
at com.datastax.spark.connector.cql.Schema$.tableFromCassandra(Schema.scala:350)
at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.tableDef(CassandraTableRowReaderProvider.scala:50)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:62)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef(CassandraTableScanRDD.scala:62)
at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.verify(CassandraTableRowReaderProvider.scala:137)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify(CassandraTableScanRDD.scala:62)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:262)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)

然后我去服务器（＆＃34; 172.11.100.156＆＃34;）并得到spark-master容器ping cassandra-0并看到：

root@708d210056af:/# ping cassandra-0
PING cassandra-0 (21.1.0.21): 56 data bytes
64 bytes from 21.1.0.21: icmp_seq=0 ttl=64 time=0.554 ms
64 bytes from 21.1.0.21: icmp_seq=1 ttl=64 time=0.117 ms
64 bytes from 21.1.0.21: icmp_seq=2 ttl=64 time=0.116 ms
64 bytes from 21.1.0.21: icmp_seq=3 ttl=64 time=0.093 ms

我的应用程序中发生了什么可能导致此错误？任何人都可以帮忙吗？

通过Java中的spark连接到Cassandra的问题

0 个答案: