Question

我在一个流媒体应用程序中调用了一个简单的函数，它可以发布如下数据：

def jsonPost(json: String) = {
  val result = Http("http://" + Address + "/api/v1/write").postData(json)
    .header("Content-Type", "application/json")
    .header("Charset", "UTF-8").asString
  result
}

如果我发送10条记录，则发送数据没有问题。当我想发送1000个时，我的Spark Streaming代码中出现以下错误：

java.net.SocketException: Connection reset
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1890)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1885)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1884)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1457)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at scalaj.http.HttpRequest.exec(Http.scala:351)
    at scalaj.http.HttpRequest.execute(Http.scala:322)
    at scalaj.http.HttpRequest.asString(Http.scala:537)
    at com.capitalone.streamingExample.streaming$.com$capitalone$streamingExample$streaming$$elasticPost$1(streaming.scala:83)
    at com.capitalone.streamingExample.streaming$$anonfun$createStream$1$$anonfun$apply$1.apply(streaming.scala:62)
    at com.capitalone.streamingExample.streaming$$anonfun$createStream$1$$anonfun$apply$1.apply(streaming.scala:59)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at org.apache.spark.util.NextIterator.foreach(NextIterator.scala:21)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:209)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:789)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
    at scalaj.http.HttpRequest.exec(Http.scala:349)
    ... 17 more

是否有任何解决方案，以便我可以扩展以发送我的API服务器1000的帖子而不会产生该错误？是否有一个我应该关注的更具可扩展性的解决方案。

scalaj HTTP在Spark Streaming中发送1000个帖子 - 错误

0 个答案: