当手动设置驱动程序的端口和地址时，Spark“ERROR EndpointWriter：AssociationError”

时间：2016-01-04 10:42:33

标签： scala configuration apache-spark spark-streaming

我正在尝试在我的独立Spark群集上提交流式传输应用程序， 这是我的代码：

import akka.actor.{Props, ActorSystem, Actor} 
import akka.http.scaladsl.Http 
import akka.http.scaladsl.model.HttpRequest 
import akka.http.scaladsl.model.Uri 
import akka.stream.ActorMaterializer 
import org.apache.spark.streaming.{Seconds, StreamingContext} 
import org.apache.spark.streaming.receiver.ActorHelper 
import org.apache.spark.{SparkEnv, SparkContext, SparkConf} 

object Boot extends App { 
  val driverPort = 7777 
  val conf = new SparkConf(). 
    setAppName("TestSpark"). 
    set("spark.driver.port", driverPort.toString). 
    set("spark.driver.host", "localhost"). 
    setMaster("spark://<master-uri>:7077"). 
    set("spark.akka.logLifecycleEvents", "true") 

  val sc = new SparkContext(conf) 
  val ssc = new StreamingContext(sc, Seconds(1)) 
  val meetup = ssc.actorStream[String](Props[CustomReceiver], "attore") 
  ChunkReceiver.start(driverPort, "localhost", "attore") 
  meetup.print 
  ssc.start 
} 

object ChunkReceiver { 
  def start(driverPort:Int, driverHost:String, actorName:String) { 
    Thread.sleep(5000) 
    implicit val system = ActorSystem("client") 
    import system.dispatcher 

    implicit val materializer = ActorMaterializer() 
    val url = s"akka.tcp://sparkDriver@$driverHost:$driverPort/user/Supervisor0/$actorName" 
    val actor = SparkEnv.get.actorSystem.actorSelection(url) 
    val source = Uri("http://stream.meetup.com/2/rsvps") 
    Http(). 
      singleRequest(HttpRequest(uri = source)). 
      flatMap { response => 
        response.entity.dataBytes.runForeach { chunk => 
          actor ! chunk.utf8String 
        } 
      } 
  } 
} 

class CustomReceiver extends Actor with ActorHelper { 
  log.info("Starting receiver ...") 
  override def receive = { 
    case s:String => store(s) 
  } 
}

这些是Spark生成的消息：

15/12/21 06:43:24 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 
15/12/21 06:43:26 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@localhost:7777] <- [akka.tcp://driverPropsFetcher@<ip-cluster>:51397]: Error [Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:51397] [ 
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:51397 
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. 
] 
akka.event.Logging$Error$NoCause$ 
15/12/21 06:43:26 ERROR EndpointWriter: AssociationError [akka.tcp://sparkDriver@localhost:7777] <- [akka.tcp://driverPropsFetcher@<ip-cluster>:51397]: Error [Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:51397] [ 
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:51397 
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. 
] 
15/12/21 06:43:26 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@localhost:7777] <- [akka.tcp://driverPropsFetcher@<ip-cluster>:51463]: Error [Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:51463] [ 
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:51463 
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. 
] 
akka.event.Logging$Error$NoCause$ 
15/12/21 06:43:26 ERROR EndpointWriter: AssociationError [akka.tcp://sparkDriver@localhost:7777] <- [akka.tcp://driverPropsFetcher@<ip-cluster>:51463]: Error [Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:51463] [ 
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:51463 
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. 
] 
15/12/21 06:43:26 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@localhost:7777] <- [akka.tcp://driverPropsFetcher@<ip-cluster>:59128]: Error [Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:59128] [ 
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:59128 
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. 
] 
akka.event.Logging$Error$NoCause$ 
15/12/21 06:43:26 ERROR EndpointWriter: AssociationError [akka.tcp://sparkDriver@localhost:7777] <- [akka.tcp://driverPropsFetcher@<ip-cluster>:59128]: Error [Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:59128] [ 
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@<ip-cluster>:59128 
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. 
]

这是主人的日志文件：

15/12/22 03:23:05 ERROR FileAppender: Error writing stream to file /disco1/spark-1.5.1/work/app-20151222032252-0010/0/stderr 
java.io.IOException: Stream closed 
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162) 
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:272) 
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334) 
        at java.io.FilterInputStream.read(FilterInputStream.java:107) 
        at org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70) 
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39) 
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) 
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) 
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699) 
        at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38) 
15/12/22 03:23:05 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@<server-ip>:38095] has failed, address is now gated for [5000] ms. Reason: [Disassociated]

这是工作人员的日志文件：

15/12/22 03:23:05 ERROR FileAppender: Error writing stream to file /disco1/spark-1.5.1/work/app-20151222032252-0010/2/stderr 
java.io.IOException: Stream closed 
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162) 
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:272) 
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334) 
        at java.io.FilterInputStream.read(FilterInputStream.java:107) 
        at org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70) 
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39) 
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) 
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) 
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699) 
        at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38) 
15/12/22 03:23:05 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@<server-ip>:41370] has failed, address is now gated for [5000] ms. Reason: [Disassociated]

应用程序不会因错误而终止，但CustomReceiver不会收到任何数据。

1 个答案:

答案 0 :(得分：0)

请尝试以这种方式改变演员

// adding a timeout
val timeout = 100 seconds
val actor = Await.result(SparkEnv.get.actorSystem.actorSelection(url).resolveOne(timeout), timeout))

希望这有助于来源http://www.typesafe.com/activator/template/spark-streaming-scala-akka

相关问题
最新问题