安装cassandra-spark-connector - 但是在创建SparkContext时出错

时间：2014-07-29 11:34:55

标签： apache-spark scala-2.10

安装cassandra-spark-connector - 但是创建SparkContext

时出错

请帮忙。我正在关注指南 - https://github.com/datastax/spark-cassandra-connector/blob/master/doc/0_quick_start.md

Env - Spark 1.0.1，Scala 2.10.4

但是当我开始创建SparkContext时出现以下错误消息。最后一行说所有主人都没有反应，放弃了。师父还在跑步

我的步骤是：     ./sbin/start-all - 成功启动所有工作

MASTER=spark://spark-master-hostname:7077 ./bin/spark-shell - to lunch spark on the master



scala> import org.apache.spark.SparkContext
import org.apache.spark.SparkContext

scala> import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext._

scala> import org.apache.spark.SparkConf
import org.apache.spark.SparkConf

scala> val conf = new SparkConf(true).set("spark.cassandra.connection.host","cassandra-host-ip")
conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@9f073

*scala> val sc = new SparkContext("spark://spark-master-ipaddress:7077", "test", conf)*
**14/07/29 12:18:23 WARN AbstractLifeCycle: FAILED
SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
   at   org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:293)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:192)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:191)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:205)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:99)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:223)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:97)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
    at $line15.$read$$iwC$$iwC$$iwC.<init>(<console>:26)
    at $line15.$read$$iwC$$iwC.<init>(<console>:28)
    at $line15.$read$$iwC.<init>(<console>:30)
    at $line15.$read.<init>(<console>:32)
    at $line15.$read$.<init>(<console>:36)
    at $line15.$read$.<clinit>(<console>)
    at $line15.$eval$.<init>(<console>:7)
    at $line15.$eval$.<clinit>(<console>)
    at $line15.$eval.$print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
    at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
    at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
    at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
    at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
    at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
    at   org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
    at     scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
    at org.apache.spark.repl.Main$.main(Main.scala:31)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    14/07/29 12:18:23 WARN AbstractLifeCycle: FAILED     org.eclipse.jetty.server.Server@dd53c8a:     java.net.BindException: Address already in use
    java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at  org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at  org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265        )
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:293)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:192)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:191)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:205)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:99)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:223)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:97)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
    at $line15.$read$$iwC$$iwC$$iwC.<init>(<console>:26)
    at $line15.$read$$iwC$$iwC.<init>(<console>:28)
    at $line15.$read$$iwC.<init>(<console>:30)
    at $line15.$read.<init>(<console>:32)
    at $line15.$read$.<init>(<console>:36)
    at $line15.$read$.<clinit>(<console>)
    at $line15.$eval$.<init>(<console>:7)
    at $line15.$eval$.<clinit>(<console>)
    at $line15.$eval.$print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
    at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
    at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
    at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
    at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
    at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
    at     org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
    at     scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
    at org.apache.spark.repl.Main$.main(Main.scala:31)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@4353d65f
    scala> 14/07/29 12:19:24 ERRstrong textOR SparkDeploySchedulerBackend: Application     has been killed. Reason: All master**s are unresponsive! Giving up.
    14/07/29 12:19:24 ERROR TaskSchedulerImpl: Exiting due to error from cluster     scheduler: All masters are unresponsive! Giving up.****

2 个答案:

答案 0 :(得分：0)

步骤1：要将连接器加载到Spark Shell，请使用以下命令启动shell：

../bin/spark-shell –jars ~/apps/spark-1.2/jars/spark-cassandra-connector-assembly-1.1.1-SNAPSHOT.jar

步骤2：将Spark Context连接到Cassandra集群。停止默认上下文。

sc.stop

步骤3：导入必要的jar文件。

import com.datastax.spark.connector._, org.apache.spark.SparkContext, org.apache.spark.SparkContext._, org.apache.spark.SparkConf

步骤4：使用Cassandra连接详细信息创建一个新的SparkConf：

val conf = new SparkConf(true).set("spark.cassandra.connection.host", "localhost")

步骤5：创建新的Spark上下文：

  val sc = new SparkContext(conf)

您现在有一个新的SparkContext连接到您的Cassandra集群。

答案 1 :(得分：0)

是否尝试过使用spark-packages？

Spark Cassandra Connector on spark-packages.org

归结为

$SPARK_HOME/bin/spark-shell --packages datastax:spark-cassandra-connector:2.0.0-M2-s_2.10

您需要为您的spark版本使用正确的版本。这应该自动设置所需的一切。

相关问题
最新问题