Question

我正在尝试在VirtualBox中的Hadoop“群集”中设置2个节点。在主节点中，我有设置

主奴隶配置：

hadoop@Master:/usr/local/hadoop/etc/hadoop$ cat slaves 
Master
Slave

Slave的主持人文件

hadoop@Slave:/usr/local/hadoop/sbin$ cat /etc/hosts
127.0.0.1   localhost
192.168.56.102  Master
127.0.0.1 Slave

Slave的core-site.xml

hadoop@Slave:/usr/local/hadoop/etc/hadoop$ cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://Slave:9000</value>
</property>
</configuration>

在掌握中我跑了：

hadoop@Master:/usr/local/hadoop/sbin$ ./start-dfs.sh 
18/03/18 12:46:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-Master.out
Slave: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-Slave.out
Master: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-Master.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-Master.out
18/03/18 12:46:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable


hadoop@Master:/usr/local/hadoop/sbin$ ./start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-Master.out
Slave: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-Slave.out
Master: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-Master.out


hadoop@Master:/usr/local/hadoop/sbin$ ./mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /usr/local/hadoop/logs/mapred-hadoop-historyserver-Master.out

这里似乎没有问题？然后我去http://localhost:8088/cluster/nodes

但是没有节点，为什么会这样？我应该在这里发布任何其他配置吗？

更新：检查我的日志......

似乎有些问题可能会启动我的节点？

hadoop@Master:/usr/local/hadoop/sbin$ tail -n 100 /usr/local/hadoop/logs/hadoop-hadoop-namenode-Master.out
ulimit -a for user hadoop
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 11659
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 11659
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

hadoop@Master:/usr/local/hadoop/sbin$ tail -n 100 /usr/local/hadoop/logs/hadoop-hadoop-datanode-Master.out
ulimit -a for user hadoop
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 11659
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 11659
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

更新2：资源管理器日志

2018-03-18 16:13:29,464 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2018-03-18 16:13:29,464 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: root: numChildQueue= 1, capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, numApps=0, numContainers=0
2018-03-18 16:13:29,464 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized root queue root: numChildQueue= 1, capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, numApps=0, numContainers=0
2018-03-18 16:13:29,464 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue mappings, override: false
2018-03-18 16:13:29,465 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized CapacityScheduler with calculator=class org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator, minimumAllocation=<<memory:1024, vCores:1>>, maximumAllocation=<<memory:8192, vCores:4>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
2018-03-18 16:13:29,497 INFO org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher: YARN system metrics publishing service is not enabled
2018-03-18 16:13:29,498 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to active state
2018-03-18 16:13:29,653 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating AMRMToken
2018-03-18 16:13:29,655 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager: Rolling master-key for container-tokens
2018-03-18 16:13:29,655 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Rolling master-key for nm-tokens
2018-03-18 16:13:29,656 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2018-03-18 16:13:29,656 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager: storing master key with keyID 1
2018-03-18 16:13:29,657 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing RMDTMasterKey.
2018-03-18 16:13:29,660 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2018-03-18 16:13:29,660 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2018-03-18 16:13:29,660 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager: storing master key with keyID 2
2018-03-18 16:13:29,660 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing RMDTMasterKey.
2018-03-18 16:13:29,662 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.nodelabels.event.NodeLabelsStoreEventType for class org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler
2018-03-18 16:13:29,720 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 5000
2018-03-18 16:13:29,732 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8031
2018-03-18 16:13:30,072 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.server.api.ResourceTrackerPB to the server
2018-03-18 16:13:30,086 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-03-18 16:13:30,086 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8031: starting
2018-03-18 16:13:30,271 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 5000
2018-03-18 16:13:30,278 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8030
2018-03-18 16:13:30,285 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB to the server
2018-03-18 16:13:30,285 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-03-18 16:13:30,286 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8030: starting
2018-03-18 16:13:30,386 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 5000
2018-03-18 16:13:30,386 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8032
2018-03-18 16:13:30,390 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ApplicationClientProtocolPB to the server
2018-03-18 16:13:30,394 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-03-18 16:13:30,394 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8032: starting
2018-03-18 16:13:30,405 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to active state
2018-03-18 16:13:30,686 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2018-03-18 16:13:30,705 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2018-03-18 16:13:30,711 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.resourcemanager is not defined
2018-03-18 16:13:30,733 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2018-03-18 16:13:30,742 INFO org.apache.hadoop.http.HttpServer2: Added filter RMAuthenticationFilter (class=org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter) to context cluster
2018-03-18 16:13:30,743 INFO org.apache.hadoop.http.HttpServer2: Added filter RMAuthenticationFilter (class=org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter) to context logs
2018-03-18 16:13:30,743 INFO org.apache.hadoop.http.HttpServer2: Added filter RMAuthenticationFilter (class=org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter) to context static
2018-03-18 16:13:30,743 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context cluster
2018-03-18 16:13:30,744 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2018-03-18 16:13:30,744 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2018-03-18 16:13:30,746 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /cluster/*
2018-03-18 16:13:30,746 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2018-03-18 16:13:31,369 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2018-03-18 16:13:31,371 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8088
2018-03-18 16:13:31,371 INFO org.mortbay.log: jetty-6.1.26
2018-03-18 16:13:31,426 INFO org.mortbay.log: Extract jar:file:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.4.jar!/webapps/cluster to /tmp/Jetty_0_0_0_0_8088_cluster____u0rgz3/webapp
2018-03-18 16:13:31,788 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2018-03-18 16:13:31,794 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2018-03-18 16:13:31,794 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2018-03-18 16:13:33,623 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8088
2018-03-18 16:13:33,623 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app cluster started at 8088
2018-03-18 16:13:33,752 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2018-03-18 16:13:33,752 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8033
2018-03-18 16:13:33,755 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocolPB to the server
2018-03-18 16:13:33,755 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-03-18 16:13:33,759 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8033: starting
2018-03-18 16:13:34,915 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: NodeManager from  Master doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager.

Slave节点有问题吗？

hadoop@Slave:/usr/local/hadoop/logs$ tail -n 50 hadoop-hadoop-datanode-Slave.log
2018-03-18 16:18:54,090 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: Slave/127.0.0.1:9000
2018-03-18 16:19:00,093 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-03-18 16:19:01,094 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-03-18 16:19:02,095 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-03-18 16:19:03,096 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-03-18 16:19:04,097 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-03-18 16:19:05,101 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-03-18 16:19:06,102 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-03-18 16:19:07,104 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-03-18 16:19:08,106 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Slave/127.0.0.1:9000. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

纱-site.xml中

hadoop@Master:/usr/local/hadoop/etc/hadoop$ cat yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>
 <property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
 </property>
</configuration>

Hadoop设置中没有节点

0 个答案: