Question

我正在尝试在Azure云虚拟机中设置Spark独立群集.Spark 2.2设置已完成。如果我启动master（start-master.sh），我可以在web ui中看到主URL。但是，火花主URL具有该VM的主机名而不是IP地址。 VM IP有私有＆amp; public，我正在尝试使用publilc IP配置主URL，以便其他客户端可以访问。

我希望如果Spark主服务器包含IP地址而不是主机名，那么我的应用程序将访问主服务器。我试过按照在线表格中提到的几个步骤，但没有任何工作

选项：1 我在sbin文件夹下创建了spark-env.sh并添加了SPARK_MASTER_HOST = x.x.x.x，但遇到了同样的问题。

选项：2

启动master时我已将主机名作为参数传递，spark-master.sh -h x.x.x.x.但同样的错误

错误：

 your platform... using builtin-java classes where applicable
18/04/01 19:39:12 INFO SecurityManager: Changing view acls to: root
18/04/01 19:39:12 INFO SecurityManager: Changing modify acls to: root
18/04/01 19:39:12 INFO SecurityManager: Changing view acls groups to:
18/04/01 19:39:12 INFO SecurityManager: Changing modify acls groups to:
18/04/01 19:39:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7078. Attempting port 7079.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7079. Attempting port 7080.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7080. Attempting port 7081.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7081. Attempting port 7082.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7082. Attempting port 7083.
 Attempting port 7093.
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'sparkMaster' failed after 16 retries (starting from 7077)! Consider explicitly setting the appropriate port for the service 'sparkMaster' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:496)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:481)
@@@

如何更改spark master url？

Spark独立集群主URL更改

0 个答案: