Question

我正在尝试在1个节点的AWS Fargate集群中设置构建作业。当我尝试运行Spark来构建我的数据时，我收到的错误似乎是Java无法找到“localHost”。

我通过运行添加spark-env.sh文件的脚本来设置配置，更新/etc/hosts文件并更新spark-defaults.conf文件。

在$SPARK_HOME/conf/spark-env.sh文件中，我添加：

SPARK_LOCAL_IP
SPARK_MASTER_HOST

在$SPARK_HOME/conf/spark-defaults.conf

中

spark.jars.packages <comma separated jars>
spark.master <ip or URL>
spark.driver.bindAddress <IP or URL>
spark.driver.host <IP or URL>

在/etc/hosts文件中，我追加：

<IP I get from http://169.254.170.2/v2/metadata> master

通过使用IP或URL传递spark-submit参数来调用-master <IP or URL>脚本似乎没有帮助。

我尝试使用local[*]，spark://<ip from metadata>:<port from metadata>，<ip>和<ip>:<port>变体，但无济于事。与使用127.0.0.1之类的内容和从元数据返回的IP相比，使用localhost和master似乎没有什么区别。

在AWS方面，Fargate集群在附加了NatGateway的私有子网中运行，因此据我所知，它确实有出口和入口网络路由。我已尝试使用公共网络并ENABLED ECS的设置自动将公共IP附加到容器。 Spark文档中的所有标准端口也在容器上打开。

它似乎运行良好，直到它试图收集自己的IP。

我得到的错误在堆栈中有这个：

spark.jars.packages com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.2
spark.master spark://10.0.41.190:7077
Spark Command: /docker-java-home/bin/java -cp /usr/spark/conf/:/usr/spark/jars/* -Xmx1gg org.apache.spark.deploy.SparkSubmit --master spark://10.0.41.190:7077 --verbose --jars lib/RedshiftJDBC42-1.2.12.1017.jar --packages org.apache.hadoop:hadoop-aws:2.7.3,com.amazonaws:aws-java-sdk:1.7.4,com.upplication:s3fs:2.2.1 ./build_phase.py
========================================
Using properties file: /usr/spark/conf/spark-defaults.conf
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.util.Utils$.redact(Utils.scala:2653)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$defaultSparkProperties$1.apply(SparkSubmitArguments.scala:93)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$defaultSparkProperties$1.apply(SparkSubmitArguments.scala:86)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.deploy.SparkSubmitArguments.defaultSparkProperties$lzycompute(SparkSubmitArguments.scala:86)
at org.apache.spark.deploy.SparkSubmitArguments.defaultSparkProperties(SparkSubmitArguments.scala:82)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:126)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.UnknownHostException: d4771b650361: d4771b650361: Name or service not known
at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:891)
at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:884)
at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:884)
at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:941)
at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:941)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.localHostName(Utils.scala:941)
at org.apache.spark.internal.config.package$.<init>(package.scala:204)
at org.apache.spark.internal.config.package$.<clinit>(package.scala)
... 10 more

尝试在本地运行时容器没有问题所以我认为它与Fargate的性质有关。

非常感谢任何帮助或指示！

修改

自从这篇文章以来，我尝试了几件不同的事情。我使用的是运行Spark 2.3，Hadoop 2.7和Python 3的图像，我尝试添加操作系统包以及我已经提到的配置的不同变体。

这一切都闻起来像我在做spark-defaults.conf和朋友们的错误，但我对这些东西很新，它可能只是木星和火星的错误对齐...

当前堆栈跟踪：

    Getting Spark Context...
    2018-06-08 22:39:40 INFO  SparkContext:54 - Running Spark version 2.3.0
    2018-06-08 22:39:40 INFO  SparkContext:54 - Submitted application: SmashPlanner
    2018-06-08 22:39:41 INFO  SecurityManager:54 - Changing view acls to: root
    2018-06-08 22:39:41 INFO  SecurityManager:54 - Changing modify acls to: root
    2018-06-08 22:39:41 INFO  SecurityManager:54 - Changing view acls groups to:
    2018-06-08 22:39:41 INFO  SecurityManager:54 - Changing modify acls groups to:
    2018-06-08 22:39:41 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
    2018-06-08 22:39:41 ERROR SparkContext:91 - Error initializing SparkContext.
    java.nio.channels.UnresolvedAddressException
        at sun.nio.ch.Net.checkAddress(Net.java:101)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:218)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1283)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:989)
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:364)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
    2018-06-08 22:39:41 INFO  SparkContext:54 - Successfully stopped SparkContext
    Traceback (most recent call last):
      File "/usr/local/smash_planner/build_phase.py", line 13, in <module>
        main()
      File "/usr/local/smash_planner/build_phase.py", line 9, in main
        build_all_data(pred_date)
      File "/usr/local/smash_planner/DataPiping/build_data.py", line 25, in build_all_data
        save_keyword(pred_date)
      File "/usr/local/smash_planner/DataPiping/build_data.py", line 52, in save_keyword
        df = get_dataframe(query)
      File "/usr/local/smash_planner/SparkUtil/data_piping.py", line 15, in get_dataframe
        sc = SparkCtx.get_sparkCtx()
      File "/usr/local/smash_planner/SparkUtil/context.py", line 20, in get_sparkCtx
        sc = SparkContext(conf=conf).getOrCreate()
      File "/usr/spark-2.3.0/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__
      File "/usr/spark-2.3.0/python/lib/pyspark.zip/pyspark/context.py", line 180, in _do_init
      File "/usr/spark-2.3.0/python/lib/pyspark.zip/pyspark/context.py", line 270, in _initialize_context
      File "/usr/local/lib/python3.4/dist-packages/py4j-0.10.6-py3.4.egg/py4j/java_gateway.py", line 1428, in __call__
        answer, self._gateway_client, None, self._fqn)
      File "/usr/local/lib/python3.4/dist-packages/py4j-0.10.6-py3.4.egg/py4j/protocol.py", line 320, in get_return_value
        format(target_id, ".", name), value)
    py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
    : java.nio.channels.UnresolvedAddressException
        at sun.nio.ch.Net.checkAddress(Net.java:101)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:218)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1283)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:989)
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:364)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)

    2018-06-08 22:39:41 INFO  ShutdownHookManager:54 - Shutdown hook called
    2018-06-08 22:39:41 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-80488ba8-2367-4fa6-8bb7-194b5ebf08cc
    Traceback (most recent call last):
      File "bin/smash_planner.py", line 76, in <module>
        raise RuntimeError("Spark hated your config and/or invocation...")
    RuntimeError: Spark hated your config and/or invocation...

SparkConf运行时配置：

def get_dataframe(query):
    ...
    sc = SparkCtx.get_sparkCtx()
    sql_context = SQLContext(sc)

    df = sql_context.read \
        .format("jdbc") \
        .option("driver", "com.amazon.redshift.jdbc42.Driver") \
        .option("url", os.getenv('JDBC_URL')) \
        .option("user", os.getenv('REDSHIFT_USER')) \
        .option("password", os.getenv('REDSHIFT_PASSWORD')) \
        .option("dbtable", "( " + query + " ) tmp ") \
        .load()

    return df

修改2

仅使用spark-env配置并使用gettyimages/docker-spark图片中的默认值运行会在浏览器中显示此错误。

java.util.NoSuchElementException
at java.util.Collections$EmptyIterator.next(Collections.java:4189)
at org.apache.spark.util.kvstore.InMemoryStore$InMemoryIterator.next(InMemoryStore.java:281)
at org.apache.spark.status.AppStatusStore.applicationInfo(AppStatusStore.scala:38)
at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:273)
at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:82)
at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:82)
at org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)
at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.spark_project.jetty.server.Server.handle(Server.java:534)
at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)

Answer 1

解决方案是避免用户错误...

这是一个完全面相的情况，但我希望我对Spark系统的误解可以帮助像我这样的可怜的傻瓜，他们花了太多时间停留在相同类型的问题上。

最后一次迭代（gettyimages/docker-spark Docker镜像）的答案是我试图在没有启动主服务器或工作服务器的情况下运行spark-submit命令。在gettyimages/docker-spark存储库中，您可以找到一个docker-compose文件，该文件向您显示在完成任何Spark工作之前，它已创建master和worker节点。映像创建主服务器或工作服务器的方式分别是使用spark-class脚本并传入org.apache.spark.deploy.<master|worker>.<Master|Worker>类。

因此，综上所述，我可以使用以前使用的配置，但是我必须先创建master和worker(s)，然后像我一样执行spark-submit命令已经在做。

这是一个快速而肮脏的实现，尽管我保证有更好的方法，由真正知道自己在做什么的人完成：

前3个步骤在集群启动脚本中进行。我是在由APIGateway触发的AWS Lambda中执行此操作

创建集群和队列或某种消息代理系统，例如zookeeper / kafka。（我正在为此使用API-Gateway-> lambda）
选择一个主节点（lambda中的逻辑）
创建一条包含一些基本信息的消息，例如主服务器的IP或域，并将其放入第1步的队列中（发生在lambda中）

以下所有内容均在Spark节点上的启动脚本中发生

在启动脚本中创建一个步骤，让节点检查队列中是否有来自步骤3的消息
使用您在步骤4中获取的消息中的信息，将SPARK_MASTER_HOST和SPARK_LOCAL_IP添加到$SPARK_HOME/conf/spark-env.sh文件中
使用您在步骤4中提取的消息中的信息，将spark.driver.bindAddress添加到$SPARK_HOME/conf/spark-defaults.conf文件中
在启动脚本中使用一些逻辑来确定“此”节点是主节点还是工作节点
启动主人或工人。在gettyimages/docker-spark图像中，您可以使用$SPARK_HOME/bin/spark-class org.apache.spark.deploy.master.Master -h <the master's IP or domain>来启动主服务器，而可以使用$SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker -h spark://<master's domain or IP>:7077来启动工作器
现在您可以运行spark-submit命令，它将把工作部署到集群中。

编辑：（一些代码供参考）这是lambda的补充

def handler(event, context):
    config = BuildConfig(event)
    res = create_job(config)
    return build_response(res)

以及修改后

def handler(event, context):
    config = BuildConfig(event)
    coordination_queue = config.cluster + '-coordination'

    sqs = boto3.client('sqs')
    message_for_master_node = {'type': 'master', 'count': config.count}
    queue_urls = sqs.list_queues(QueueNamePrefix=coordination_queue)['QueueUrls']

    if not queue_urls:
        queue_url = sqs.create_queue(QueueName=coordination_queue)['QueueUrl']
    else:
        queue_url = queue_urls[0]

     sqs.send_message(QueueUrl=queue_url,
                 MessageBody=message_for_master_node)

    res = create_job(config)
    return build_response(res)

然后我在启动时在Spark集群中运行节点的脚本中添加了一些内容：

# addition to the "main" in the Spark node's startup script
sqs = boto3.client('sqs')
boot_info_message = sqs.receive_message(
    QueueUrl=os.getenv('COORDINATIN_QUEUE_URL'),
    MaxNumberOfMessages=1)['Messages'][0]
boot_info = boot_info_message['Body']
message_for_worker = {'type': 'worker', 'master': self_url}

if boot_info['type'] == 'master':
    for i in range(int(boot_info['count'])):
        sqs.send_message(QueueUrl=os.getenv('COORDINATIN_QUEUE_URL'),
                         MessageBody=message_for_worker)
sqs.delete_message(QueueUrl=os.getenv('COORDINATIN_QUEUE_URL'),
                   ReceiptHandle=boot_info_message['ReceiptHandle'])

...
# starts a master or worker node
startup_command = "org.apache.spark.deploy.{}.{}".format(
    boot_info['type'], boot_info['type'].title())
subprocess.call(startup_command)

Answer 2

转到AWS控制台并在安全组配置下，允许实例的所有入站流量。

https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_SecurityGroups.html

在Fargate上的Spark无法找到本地IP

2 个答案: