spark-submit命令无法连接到bluemix分析引擎服务的spark节点

时间：2018-05-03 11:27:10

标签： apache-spark ibm-cloud spark-submit analytics-engine

我正在尝试将spark应用程序（Java）部署到Spark引擎，作为Bluemix上分析引擎服务的一部分。我按照这里提到的步骤进行了

https://console.bluemix.net/docs/services/AnalyticsEngine/WCE-CLI.html#analytics-engine-command-line-interface

完成所有步骤后，在spark-submit时，我收到以下错误

C:\Users\IBM_ADMIN\eclipse-workspace\mySparkApp\target>bx ae spark-submit --className mySparkApp.Main mySparkApp-0.0.1-S
NAPSHOT.jar
Current user is 'clsadmin'
Password>
Contacting endpoint 'https://159.122.220.119:8443'...
FAILED
Server call failed. Message: '<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /gateway/default/livy/v1/batches. Reason:
<pre>    Server Error</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>
'

我可以访问ambari服务器并可以看到启动和运行的spark服务。我也可以从浏览器

访问livy API端点

https://chs-uvi-769-mn001.bi.services.eu-gb.bluemix.net:8443/gateway/default/livy/v1/batches 

{"from":0,"total":5,"sessions":[{"id":0,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-0-t8fc4ebv in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]},{"id":1,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-1-1olxdmt5 in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]},{"id":2,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-2-xbjzpkbp in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]},{"id":3,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-3-sbilpm4a in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]},{"id":4,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-4-rvlzpl8o in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]}]}

我无法得到问题所在。我已启用bluemix跟踪，并且在调试模式下输出相同的命令是

C:\Users\IBM_ADMIN\eclipse-workspace\mySparkApp\target>SET BLUEMIX_TRACE=true

C:\Users\IBM_ADMIN\eclipse-workspace\mySparkApp\target>bx ae spark-submit --className mySparkApp.Main mySparkApp-0.0.1-S
NAPSHOT.jar
Current user is 'clsadmin'
Password>
DEBUG: INPUTS PROVIDED

 file mySparkApp-0.0.1-SNAPSHOT.jar
 proxyUser NOTSET
 className mySparkApp.Main
 inputArgs []
 jars []
 pyFiles []
 files []
 driverMemory NOTSET
 driverCores -1
 executorMemory NOTSET
 executorCores -1
 numExecutors -1
 archives []
 queue NOTSET
 name NOTSET
 conf map[]
 asynchronous false
 upload false

DEBUG: PROCESSED INPUTS

 file mySparkApp-0.0.1-SNAPSHOT.jar
 proxyUser NOTSET
 className mySparkApp.Main
 inputArgs []
 jars []
 pyFiles []
 files []
 driverMemory NOTSET
 driverCores -1
 executorMemory NOTSET
 executorCores -1
 numExecutors -1
 archives []
 queue NOTSET
 name NOTSET
 conf map[]
 asynchronous false
 upload false

Contacting endpoint 'https://159.122.220.119:8443'...

←[1;39mREQUEST:←[0m [2018-05-03T16:52:33+05:30]
POST /gateway/default/livy/v1/batches HTTP/1.1
Host: 159.122.220.119:8443
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
X-Requested-By: livy

{"file":"mySparkApp-0.0.1-SNAPSHOT.jar","className":"mySparkApp.Main"}

←[1;39mRESPONSE:←[0m [2018-05-03T16:52:34+05:30] ←[1;39mElapsed:←[0m 1382ms
HTTP/1.1 500 Server Error
Connection: close
Content-Length: 321
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html; charset=ISO-8859-1
Date: Thu, 03 May 2018 11:22:52 GMT
Date: Thu, 03 May 2018 11:22:52 GMT
Server: Jetty(9.2.16.v20160414)
Set-Cookie: JSESSIONID=vyw2g1lyyjmo2d2spzs5pnd1;Path=/gateway/default;Secure;HttpOnly
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Wed, 02-May-2018 11:22:52 GMT
Strict-Transport-Security: max-age=31536000

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /gateway/default/livy/v1/batches. Reason:
<pre>    Server Error</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

FAILED
Server call failed. Message: '<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /gateway/default/livy/v1/batches. Reason:
<pre>    Server Error</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>
'

FAILED
Server call failed. Message: '←[1;31m<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /gateway/default/livy/v1/batches. Reason:
<pre>    Server Error</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>
←[0m'

任何人都可以看到问题是什么？任何帮助或指针？

2 个答案:

答案 0 :(得分：1)

通常，当程序出现问题时，我注意到了这个错误。如果您可以访问集群，即ssh clsadmin @，则可以尝试使用spark-submit运行java程序并确认程序有效且没有问题。

答案 1 :(得分：0)

1）您能否查看火花节点（/ var / log / livy2）上的Livy日志，查看其中记录的任何错误消息？ 2）您还可以尝试在群集上重新启动Livy服务吗？

在我们的测试服务器上运行示例代码：

bx ae spark-submit --className org.apache.spark.examples.SparkPi local:/usr/hdp/current/spark2-client/jars/spark-examples.jar User (clsadmin)> Password> Contacting endpoint 'https://169.60.167.93:8443'... Job ID '8' Waiting for job to return application ID. Will check every 10 seconds, and stop checking after 2 minutes. Press Control C to stop waiting. Finished contacting endpoint 'https://169.60.167.93:8443' OK Job ID '8' Application ID 'application_1521738019714_0083' Done

相关问题
最新问题