Question

我正在使用apache spark，我希望将运行spark-submit命令时抛出的java.net.BindException: Address already in use.异常静音。 log4j属性设置在下面发布的单独文件中。

java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.spark-project.jetty.server.Server.doStart(Server.java:293)
    at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:252)
    at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
    at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
    at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1988)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
    at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1979)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:262)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
    at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
    at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:481)
    at com.mypackage.myclass.MyUtil$class.build_context(MyUtil.scala:500)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    16/03/14 11:13:01 WARN Utils: Service 'SparkUI' could not bind on port 4050. Attempting port 4051.

这是我的log4j.properties

log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.com.jiwire=INFO
log4j.logger.com.nd=INFO

知道我该怎么做？

Answer 1

导致错误是因为Spark尝试在已经在使用的端口上托管其web interface。

您可能同时运行两个Spark实例，使第一个实例获取端口。

抑制此类错误通常是一种不好的做法。

Read here 有关Spark配置的更多信息。

如何在log4j java.net.BindException中使一个类静音

1 个答案: