Question

我是Hadoop的新手，所以非常感谢您提供有关此问题的反馈。

Hadoop设置似乎很好。我可以启动它，但是当我在http://localhost:50070或http://localhost:9870上检查Web UI时，它表明无法访问该网站。同样，要使用Web UI http://localhost:8088检查Yarn，我遇到了同样的问题。

命令jps显示以下详细信息： 50714 SecondaryNameNode 88442 51756每秒 50589 DataNode

Namenode，ResourceManager，NodeManager丢失。

我尝试更改端口配置，没有帮助。参考：http://localhost:50070 does not work HADOOP hadoop web UI at http://localhost:50070/ doesnt work

$ ./start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [Maggies-MacBook-Pro.local]
2019-09-01 17:33:33,523 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

sbin ./start-yarn.sh
Starting resourcemanager
Starting nodemanagers

重新设置名称节点和start-all.sh后：

sbin ./start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as zxiao in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [Maggies-MacBook-Pro.local]
2019-09-02 09:19:31,657 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers

sbin jps
98359 SecondaryNameNode
99014 Jps
98232 DataNode
88442

仍然无法启动namenode。网络用户界面仍然无法显示。

更新，这是名称节点的日志文件：

2019-09-02 10:57:12,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2019-09-02 10:57:12,850 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []
2019-09-02 10:57:12,965 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2019-09-02 10:57:13,089 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2019-09-02 10:57:13,090 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2019-09-02 10:57:13,112 INFO org.apache.hadoop.hdfs.server.namenode.NameNodeUtils: fs.defaultFS is hdfs://localhost:8020
2019-09-02 10:57:13,112 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients should use localhost:8020 to access this namenode/service.
2019-09-02 10:57:13,134 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-09-02 10:57:13,209 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
2019-09-02 10:57:13,226 INFO org.apache.hadoop.hdfs.DFSUtil: Starting Web-server for hdfs at: http://0.0.0.0:9870
2019-09-02 10:57:13,235 INFO org.eclipse.jetty.util.log: Logging initialized @839ms
2019-09-02 10:57:13,294 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2019-09-02 10:57:13,302 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.namenode is not defined
2019-09-02 10:57:13,306 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2019-09-02 10:57:13,307 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
2019-09-02 10:57:13,307 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2019-09-02 10:57:13,307 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2019-09-02 10:57:13,320 INFO org.apache.hadoop.http.HttpServer2: Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter)
2019-09-02 10:57:13,320 INFO org.apache.hadoop.http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
2019-09-02 10:57:13,333 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 9870
2019-09-02 10:57:13,333 INFO org.eclipse.jetty.server.Server: jetty-9.3.24.v20180605, build timestamp: 2018-06-05T10:11:56-07:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
2019-09-02 10:57:13,350 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2f2bf0e2{/logs,file:///usr/local/Cellar/hadoop/3.1.2/libexec/logs/,AVAILABLE}
2019-09-02 10:57:13,351 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@21ec5d87{/static,file:///usr/local/Cellar/hadoop/3.1.2/libexec/share/hadoop/hdfs/webapps/static/,AVAILABLE}
2019-09-02 10:57:13,404 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@4fdf8f12{/,file:///usr/local/Cellar/hadoop/3.1.2/libexec/share/hadoop/hdfs/webapps/hdfs/,AVAILABLE}{/hdfs}
2019-09-02 10:57:13,409 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@5710768a{HTTP/1.1,[http/1.1]}{0.0.0.0:9870}
2019-09-02 10:57:13,409 INFO org.eclipse.jetty.server.Server: Started @1013ms
2019-09-02 10:57:13,532 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories!
2019-09-02 10:57:13,532 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of data loss due to lack of redundant storage directories!
2019-09-02 10:57:13,559 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Edit logging is async:true
2019-09-02 10:57:13,567 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: KeyProvider: null
2019-09-02 10:57:13,568 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsLock is fair: true
2019-09-02 10:57:13,569 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2019-09-02 10:57:13,592 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             = zxiao (auth:SIMPLE)
2019-09-02 10:57:13,592 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          = supergroup
2019-09-02 10:57:13,592 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2019-09-02 10:57:13,593 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2019-09-02 10:57:13,622 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2019-09-02 10:57:13,630 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2019-09-02 10:57:13,630 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2019-09-02 10:57:13,634 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2019-09-02 10:57:13,634 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: The block deletion will start around 2019 Sep 02 10:57:13
2019-09-02 10:57:13,635 INFO org.apache.hadoop.util.GSet: Computing capacity for map BlocksMap
2019-09-02 10:57:13,635 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit
2019-09-02 10:57:13,636 INFO org.apache.hadoop.util.GSet: 2.0% max memory 4 GB = 81.9 MB
2019-09-02 10:57:13,636 INFO org.apache.hadoop.util.GSet: capacity      = 2^23 = 8388608 entries
2019-09-02 10:57:13,657 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable = false
2019-09-02 10:57:13,662 INFO org.apache.hadoop.conf.Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication         = 1
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication             = 512
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication             = 1
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams      = 2
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer        = false
2019-09-02 10:57:13,662 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2019-09-02 10:57:13,678 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: GLOBAL serial map: bits=24 maxEntries=16777215
2019-09-02 10:57:13,688 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2019-09-02 10:57:13,688 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit
2019-09-02 10:57:13,689 INFO org.apache.hadoop.util.GSet: 1.0% max memory 4 GB = 41.0 MB
2019-09-02 10:57:13,689 INFO org.apache.hadoop.util.GSet: capacity      = 2^22 = 4194304 entries
2019-09-02 10:57:13,697 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: ACLs enabled? false
2019-09-02 10:57:13,697 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: POSIX ACL inheritance enabled? true
2019-09-02 10:57:13,697 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: XAttrs enabled? true
2019-09-02 10:57:13,697 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occurring more than 10 times
2019-09-02 10:57:13,702 INFO org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2019-09-02 10:57:13,703 INFO org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager: SkipList is disabled
2019-09-02 10:57:13,706 INFO org.apache.hadoop.util.GSet: Computing capacity for map cachedBlocks
2019-09-02 10:57:13,706 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit
2019-09-02 10:57:13,706 INFO org.apache.hadoop.util.GSet: 0.25% max memory 4 GB = 10.2 MB
2019-09-02 10:57:13,706 INFO org.apache.hadoop.util.GSet: capacity      = 2^20 = 1048576 entries
2019-09-02 10:57:13,712 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2019-09-02 10:57:13,712 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2019-09-02 10:57:13,712 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2019-09-02 10:57:13,714 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2019-09-02 10:57:13,714 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2019-09-02 10:57:13,715 INFO org.apache.hadoop.util.GSet: Computing capacity for map NameNodeRetryCache
2019-09-02 10:57:13,715 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit
2019-09-02 10:57:13,715 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory 4 GB = 1.2 MB
2019-09-02 10:57:13,715 INFO org.apache.hadoop.util.GSet: capacity      = 2^17 = 131072 entries
2019-09-02 10:57:13,727 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/in_use.lock acquired by nodename 25057@Maggies-MacBook-Pro.local
2019-09-02 10:57:13,743 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current
2019-09-02 10:57:13,748 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Planning to load image: FSImageFile(file=/usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current/fsimage_0000000000000000000, cpktTxId=0000000000000000000)
2019-09-02 10:57:13,792 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode: Loading 1 INodes.
2019-09-02 10:57:13,809 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf: Loaded FSImage in 0 seconds.
2019-09-02 10:57:13,810 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current/fsimage_0000000000000000000
2019-09-02 10:57:13,812 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@5c748168 expecting start txid #1
2019-09-02 10:57:13,813 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current/edits_0000000000000000001-0000000000000000002 maxTxnsToRead = 9223372036854775807
2019-09-02 10:57:13,815 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream '/usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current/edits_0000000000000000001-0000000000000000002' to transaction ID 1
2019-09-02 10:57:13,826 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current/edits_0000000000000000001-0000000000000000002 of size 42 edits # 2 loaded in 0 seconds
2019-09-02 10:57:13,826 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Need to save fs image? false (staleImage=false, haEnabled=false, isRollingUpgrade=false)
2019-09-02 10:57:13,826 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 3
2019-09-02 10:57:13,910 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2019-09-02 10:57:13,911 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 193 msecs
2019-09-02 10:57:14,012 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: RPC server is binding to localhost:8020
2019-09-02 10:57:14,017 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 1000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2019-09-02 10:57:14,023 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8020
2019-09-02 10:57:14,154 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState, ReplicatedBlocksState and ECBlockGroupsState MBeans.
2019-09-02 10:57:14,170 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for active state
2019-09-02 10:57:14,170 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 3, 3
2019-09-02 10:57:14,175 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 2 Number of syncs: 3 SyncTimes(ms): 21 
2019-09-02 10:57:14,177 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current/edits_inprogress_0000000000000000003 -> /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current/edits_0000000000000000003-0000000000000000004
2019-09-02 10:57:14,178 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: FSEditLogAsync was interrupted, exiting
2019-09-02 10:57:14,178 INFO org.apache.hadoop.ipc.Server: Stopping server on 8020
2019-09-02 10:57:14,198 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for active state
2019-09-02 10:57:14,198 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for standby state
2019-09-02 10:57:14,201 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@4fdf8f12{/,null,UNAVAILABLE}{/hdfs}
2019-09-02 10:57:14,204 INFO org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@5710768a{HTTP/1.1,[http/1.1]}{0.0.0.0:9870}
2019-09-02 10:57:14,204 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@21ec5d87{/static,file:///usr/local/Cellar/hadoop/3.1.2/libexec/share/hadoop/hdfs/webapps/static/,UNAVAILABLE}
2019-09-02 10:57:14,204 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@2f2bf0e2{/logs,file:///usr/local/Cellar/hadoop/3.1.2/libexec/logs/,UNAVAILABLE}
2019-09-02 10:57:14,205 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2019-09-02 10:57:14,205 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2019-09-02 10:57:14,205 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2019-09-02 10:57:14,209 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Could not parse line: Su Mo Tu We Th Fr Sa  
    at org.apache.hadoop.fs.DF.parseOutput(DF.java:195)
    at org.apache.hadoop.fs.DF.getFilesystem(DF.java:76)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$CheckedVolume.<init>(NameNodeResourceChecker.java:69)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.addDirToCheck(NameNodeResourceChecker.java:165)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.<init>(NameNodeResourceChecker.java:134)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1166)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:788)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:714)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
2019-09-02 10:57:14,210 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.io.IOException: Could not parse line: Su Mo Tu We Th Fr Sa  
2019-09-02 10:57:14,212 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Maggies-MacBook-Pro.local/10.0.0.73
************************************************************/

Answer 1

尝试格式化namenode，然后再次使用start-all.sh启动所有这些节点将解决我猜到的问题

Answer 2

执行 ssh localhost ，可能会导致连接失败。

如果连接失败，则将ssh密钥添加到您的已知主机。

然后启动 ./ start-dfs.sh 。

Answer 3

就我而言，我在fortune中运行了一个.bashrc程序并打印了一些消息。似乎该输出会影响Hadoop脚本。我当前的版本是3.3.0。
Could not parse line: ***似乎在解析包含该程序的某些内容的输出并引发错误。所以我必须删除它，然后它消失了。

HADOOP 3.1.2 Namenode未启动

3 个答案: