试图在纱线集群上运行火花提交作业,但我不断收到以下警告。我该如何解决这个问题?

时间:2017-09-21 17:13:38

标签: apache-spark hdfs yarn spark-submit

WARN YarnClusterScheduler: Initial job has not accepted any resources; check 
your cluster UI to ensure that workers are registered and have sufficient 
resources.

我查看了类似的问题,并尝试了其他所有提到的内容。当我查看hdfs上的yarn-nodemanager登录时,我看到可能导致错误的以下警告。如何修复这些警告?

2017-09-13 14:29:52,640 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The 
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class 
org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. 
Because these are not the same tools trying to send ServiceData and read 
Service Meta Data may have issues unless the refer to the name in the config.

yarn-site.xml日志:

<configuration>
    <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
    </property>

<property>
  <name>yarn.application.classpath</name>
  <value>/usr/local/hadoop/etc/hadoop, /usr/local/hadoop/share/hadoop/common
/*, /usr/local/hadoop/share/hadoop/common/lib/*, 
/usr/local/hadoop/share/hadoop/
hdfs/*, /usr/local/hadoop/share/hadoop/hdfs/lib/*, 
/usr/local/hadoop/share/hadoo
p/mapreduce/*, /usr/local/hadoop/share/hadoop/mapreduce/lib/*, 
/usr/local/hadoop
/share/hadoop/yarn/*, /usr/local/hadoop/share/hadoop/yarn/lib/*</value>
</property>
<property>
   <name>nodemanager.resource.cpu-vcores</name>
   <value>2</value>
</property>
<property>
<description>
  Number of seconds after an application finishes before the nodemanager's
  DeletionService will delete the application's localized file directory
  and log directory.

core-site.xml日志:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://sandbox:9000</value>
</property>
<property>
  <name>dfs.client.use.legacy.blockreader</name>
  <value>true</value>
</property>

</configuration>

hdfs-site.xml日志:

<configuration>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
</configuration>

如果我试图找到错误方向的初始警告解决方案,请告诉我,因为应用程序一直在运行,但没有数据发送到hdfs。谢谢!

0 个答案:

没有答案