Question

过去3天我尝试使用3个组件设置Docker机器： Spark Master，Spark Worker和Driver（Java）应用程序

从docker启动驱动程序OUTSIDE时，一切正常。但是，启动所有三个组件会导致端口防火墙 - 主机 - 噩梦

为了保持它（起初）简单，我使用docker-compose - 这是我的docker-compose.yml：

driver:
  hostname: driver
  image: driverimage
  command: -Dexec.args="0 192.168.99.100" -Dspark.driver.port=7001 -Dspark.driver.host=driver -Dspark.executor.port=7006 -Dspark.broadcast.port=15001 -Dspark.fileserver.port=15002 -Dspark.blockManager.port=15003 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory
  ports:
    - 10200:10200 # Module REST Port
    - 4040:4040 # Web UI (Spark)
    - 7001:7001 # Driver Port (Spark)
    - 15001:15001 # Broadcast (Spark)
    - 15002:15002 # File Server (Spark)
    - 15003:15003 # Blockmanager (Spark)
    - 7337:7337 # Shuffle? (Spark)
  extra_hosts:
    - sparkmaster:192.168.99.100
    - sparkworker:192.168.99.100
  environment:
    SPARK_LOCAL_IP: 192.168.99.100
    #SPARK_MASTER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    #SPARK_WORKER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    SPARK_JAVA_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=15001 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"

sparkmaster:
  extra_hosts:
    - driver:192.168.99.100
  image: gettyimages/spark
  command: /usr/spark/bin/spark-class org.apache.spark.deploy.master.Master -h sparkmaster
  hostname: sparkmaster
  environment:
    SPARK_CONF_DIR: /conf
    MASTER: spark://sparkmaster:7077
    SPARK_LOCAL_IP: 192.168.99.100
    SPARK_JAVA_OPTS:  "-Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    SPARK_WORKER_OPTS: "-Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    SPARK_MASTER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    #SPARK_WORKER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    #SPARK_JAVA_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
  expose:
    - 7001
    - 7002
    - 7003
    - 7004
    - 7005
    - 7006
    - 7077
    - 6066
  ports:
    - 6066:6066
    - 7077:7077 # Master (Main Port)
    - 8080:8080 # Web UI
    #- 7006:7006 # Executor

sparkworker:
  extra_hosts:
    - driver:192.168.99.100
  image: gettyimages/spark
  command: /usr/spark/bin/spark-class org.apache.spark.deploy.worker.Worker -h sparkworker spark://sparkmaster:7077
#  volumes:
#    - ./spark/logs:/log/spark
  hostname: sparkworker
  environment:
    SPARK_CONF_DIR: /conf
    SPARK_WORKER_CORES: 4
    SPARK_WORKER_MEMORY: 4g
    SPARK_WORKER_PORT: 8881
    SPARK_WORKER_WEBUI_PORT: 8081
    SPARK_LOCAL_IP: 192.168.99.100
    #SPARK_MASTER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    SPARK_JAVA_OPTS:  "-Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    SPARK_MASTER_OPTS: "-Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    SPARK_WORKER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=15003 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
    #SPARK_JAVA_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
  links:
    - sparkmaster
  expose:
    - 7001
    - 7002
    - 7003
    - 7004
    - 7005
    - 7006
    - 7012
    - 7013
    - 7014
    - 7015
    - 7016
    - 8881
  ports:
    - 8081:8081 # WebUI
    #- 15003:15003 # Blockmanager+
    - 7005:7005 # Executor
    - 7006:7006 # Executor
    #- 7006:7006 # Executor

我甚至不知道实际上使用了哪个端口等等。我所知道的是我目前的问题如下。司机可以与师傅沟通，师父可以与工作人员沟通，我认为司机可以与工人沟通！司机无法与执行人沟通。我也发现了问题。当我打开应用程序UI并打开exectuors选项卡时，它会显示“Executor 0 - Address 172.17.0.1:7005”。

所以问题是，驱动程序使用Docker网关地址寻址执行程序，这不起作用。我尝试了几件事（SPARK_LOCAL_IP，使用显式主机名等），但驱动程序总是尝试与Docker Gateway通信...任何想法如何实现驱动程序可以与执行程序/工作者通信？

Answer 1

这是由于Spark提供的配置选项不足。 Spark绑定侦听SPARK_LOCAL_HOSTNAME，并将此确切的主机名传播到群集。不幸的是，如果驱动程序位于NAT后面，例如Docker容器，则此设置不起作用。

您可以通过以下设置解决此问题（我已成功使用此hack）：

转发所有必要的端口1对1（与您一样）
为驱动程序使用自定义主机名：set eg SPARK_LOCAL_HOSTNAME: mydriver
对于主节点和工作节点，将192.168.99.100 mydriver添加到/etc/hosts，以便他们可以访问Spark驱动程序。
对于泊坞窗容器，将mydriver映射到0.0.0.0。这将使Spark驱动程序绑定到0.0.0.0，因此主服务器和工作人员可以访问它：

要在docker-compose.yml中执行此操作，只需添加以下行：

 extra_hosts:
  - "mydriver:0.0.0.0"

如何在Docker容器中设置执行程序IP？

1 个答案: