Question

我使用using details from you other question作为火花主容器，同时我还有一个火花图像，它将启动一个从属节点。这是我对应的docker文件。

FROM debian:jessie

RUN apt-get update \
 && apt-get install -y locales \
 && dpkg-reconfigure -f noninteractive locales \
 && locale-gen C.UTF-8 \
 && /usr/sbin/update-locale LANG=C.UTF-8 \
 && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
 && locale-gen \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*

# Users with other locales should set this in their derivative image
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

RUN apt-get update \
 && apt-get install -y curl unzip \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*


# JAVA
ARG JAVA_MAJOR_VERSION=8
ARG JAVA_UPDATE_VERSION=92
ARG JAVA_BUILD_NUMBER=14
ENV JAVA_HOME /usr/jdk1.${JAVA_MAJOR_VERSION}.0_${JAVA_UPDATE_VERSION}

ENV PATH $PATH:$JAVA_HOME/bin
RUN curl -sL --retry 3 --insecure \
  --header "Cookie: oraclelicense=accept-securebackup-cookie;" \
  "http://download.oracle.com/otn-pub/java/jdk/${JAVA_MAJOR_VERSION}u${JAVA_UPDATE_VERSION}-b${JAVA_BUILD_NUMBER}/server-jre-${JAVA_MAJOR_VERSION}u${JAVA_UPDATE_VERSION}-linux-x64.tar.gz" \
  | gunzip \
  | tar x -C /usr/ \
  && ln -s $JAVA_HOME /usr/java \
  && rm -rf $JAVA_HOME/man

# HADOOP
ENV HADOOP_VERSION 2.7.2
ENV HADOOP_HOME /usr/hadoop-$HADOOP_VERSION
ENV HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
ENV PATH $PATH:$HADOOP_HOME/bin
RUN curl -sL --retry 3 \
  "http://archive.apache.org/dist/hadoop/common/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz" \
  | gunzip \
  | tar -x -C /usr/ \
 && rm -rf $HADOOP_HOME/share/doc \
 && chown -R root:root $HADOOP_HOME

# SPARK
ENV SPARK_VERSION 2.0.1
ENV SPARK_PACKAGE spark-${SPARK_VERSION}-bin-without-hadoop
ENV SPARK_HOME /usr/spark-${SPARK_VERSION}
ENV SPARK_DIST_CLASSPATH="$HADOOP_HOME/etc/hadoop/*:$HADOOP_HOME/share/hadoop/common/lib/*:$HADOOP_HOME/share/hadoop/common/*:$HADOOP_HOME/share/hadoop/hdfs/*:$HADOOP_HOME/share/hadoop/hdfs/lib/*:$HADOOP_HOME/share/hadoop/hdfs/*:$HADOOP_HOME/share/hadoop/yarn/lib/*:$HADOOP_HOME/share/hadoop/yarn/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*:$HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/tools/lib/*"
ENV PATH $PATH:${SPARK_HOME}/bin
RUN curl -sL --retry 3 \
  "http://d3kbcqa49mib13.cloudfront.net/${SPARK_PACKAGE}.tgz" \
  | gunzip \
  | tar x -C /usr/ \
 && mv /usr/$SPARK_PACKAGE $SPARK_HOME \
 && chown -R root:root $SPARK_HOME

WORKDIR $SPARK_HOME 
CMD ["bin/spark-class","org.apache.spark.deploy.worker.Worker", //TODO: Figure out what this should be]

我想知道如果我通过docker compose进行设置，我怎么能让奴隶能够访问主人主机和端口。

Answer 1

假设您有docker-compose之类的内容：

version: '2'
services:
  spark-master:
   image: spark-master
   ports:
   - "7077:7077"
   - "8080:8080"
  spark-slave1:
   image: spark-slave
   ports:
   - "8081:8081"
   depends_on:
   - spark-master

在你的火花奴隶的Dockerfile中你需要定义大师像： ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://IP:PORT

但使用IP地址并不是一个好主意。因此，您可以使用docker-compose中的主机名（spark-master是主机名，并将其添加到/etc/hosts）：

CMD ["bin/spark-class","org.apache.spark.deploy.worker.Worker", "spark://spark-master:7077"]

现在你可以去：
DOCKER_IP:8080 - ＆gt;并看到“工人”中的1名工人 DOCKER_IP:8081 - ＆gt;并查看工人的详细信息

如果您想拥有更多员工，可以向docker-compose添加其他服务。在docker-compose之后将创建2个工作人员 - 端口8081上的1.工作人员和8082上的第二个工作人员的详细信息。

version: '2'
services:
  spark-master:
   image: spark-master
   ports:
   - "7077:7077"
   - "8080:8080"
  spark-slave1:
   image: spark-slave
   ports:
   - "8081:8081"
   depends_on:
   - spark-master
  spark-slave2:
   image: spark-slave
   ports:
   - "8082:8081"
   depends_on:
   - spark-master

通过docker compose将spark master连接到spark slave

1 个答案: