连接到托管在docker

Question

我在具有3个工作程序节点的Ubuntu服务器上安装了一个Spark集群（使用docker compose）。

您可以查看所有群集信息。也可以从我的电脑访问（web ui），如下所示：

现在，我想从Java驱动程序中添加作业，因此将是这样的：

SparkSession spark = SparkSession.builder()
            .master("spark://nodemaster:7077")
            .appName("MongoSparkConnectorIntro")
            .config("spark.network.timeout",2000000)
            .config("spark.driver.port","32772")
            .config("spark.driver.host","172.31.64.69")
            .config("spark.driver.bindAddress","172.18.0.2")
            .config("spark.mongodb.input.uri", "mongodb://")
            .config("spark.mongodb.output.uri", "mongodb://")
            .getOrCreate();

---

org.apache.spark.SparkContext: Running Spark version 2.4.1
org.apache.spark.SparkContext: Submitted application: MongoSparkConnectorIntro
org.apache.spark.SecurityManager: Changing view acls to: $USER
org.apache.spark.SecurityManager: Changing modify acls to: $USER
org.apache.spark.SecurityManager: Changing view acls groups to:
org.apache.spark.SecurityManager: Changing modify acls groups to:
org.apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set($USER); groups with view permissions: Set(); users  with modify permissions: Set($USER); groups with modify permissions: Set()
org.apache.spark.util.Utils: Service 'sparkDriver' could not bind on a random free port. You may check whether configuring an appropriate binding address.

我在Windows 10上使用Intellij IDE。 我也将我的IP添加到了/ etc / hosts

有帮助吗？