我想在伪分布式模式下设置一个hadoop-cluster。我设法执行所有设置步骤,包括在我的机器上启动Namenode,Datanode,Jobtracker和Tasktracker。
然后我尝试运行一些示例程序并遇到java.net.ConnectException: Connection refused
错误。我回到了以独立模式运行某些操作的最初步骤,并遇到了同样的问题。
我执行了所有安装步骤的三重检查,并且不知道如何修复它。 (我是Hadoop和初学Ubuntu用户的新手,因此,如果提供任何指南或提示,我恳请您“考虑到它”)。
这是我一直收到的错误输出:
hduser@marta-komputer:/usr/local/hadoop$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'
15/02/22 18:23:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/02/22 18:23:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
java.net.ConnectException: Call From marta-komputer/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy9.delete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:521)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy10.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1929)
at org.apache.hadoop.hdfs.DistributedFileSystem$12.doCall(DistributedFileSystem.java:638)
at org.apache.hadoop.hdfs.DistributedFileSystem$12.doCall(DistributedFileSystem.java:634)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:634)
at org.apache.hadoop.examples.Grep.run(Grep.java:95)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.Grep.main(Grep.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 32 more
etc / hadoop / hadoop-env.sh 档案:
# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol. Jsvc is not required if SASL is configured for authentication of
# data transfer protocol using non-privileged ports.
#export JSVC_HOME=${JSVC_HOME}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done
# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE=
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""
# Extra Java runtime options. Empty by default.
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"
# On secure datanodes, user to run the datanode as after dropping privileges.
# This **MUST** be uncommented to enable secure HDFS if using privileged ports
# to provide authentication of data transfer protocol. This **MUST NOT** be
# defined if SASL is configured for authentication of data transfer protocol
# using non-privileged ports.
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}
# Where log files are stored. $HADOOP_HOME/logs by default.
#export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER
# Where log files are stored in the secure data environment.
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
# HDFS Mover specific parameters
###
# Specify the JVM options to be used when starting the HDFS Mover.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_MOVER_OPTS=""
###
# Advanced Users Only!
###
# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by
# the user that will run the hadoop daemons. Otherwise there is the
# potential for a symlink attack.
export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
# A string representing this instance of hadoop. $USER by default.
export HADOOP_IDENT_STRING=$USER
.bashrc 文件与Hadoop相关的片段:
# -- HADOOP ENVIRONMENT VARIABLES START -- #
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
# -- HADOOP ENVIRONMENT VARIABLES END -- #
/usr/local/hadoop/etc/hadoop/core-site.xml 档案:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop_tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
/usr/local/hadoop/etc/hadoop/hdfs-site.xml 档案:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop_tmp/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop_tmp/hdfs/datanode</value>
</property>
</configuration>
/usr/local/hadoop/etc/hadoop/yarn-site.xml 档案:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
/usr/local/hadoop/etc/hadoop/mapred-site.xml 档案:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<configuration>
运行hduser@marta-komputer:/usr/local/hadoop$ bin/hdfs namenode -format
会产生如下输出(我将部分内容改为(...)
):
hduser@marta-komputer:/usr/local/hadoop$ bin/hdfs namenode -format
15/02/22 18:50:47 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = marta-komputer/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.6.0
STARTUP_MSG: classpath = /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/htrace-core-3.0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli (...)2.6.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on 2014-11-13T21:10Z
STARTUP_MSG: java = 1.8.0_31
************************************************************/
15/02/22 18:50:47 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
15/02/22 18:50:47 INFO namenode.NameNode: createNameNode [-format]
15/02/22 18:50:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-0b65621a-eab3-47a4-bfd0-62b5596a940c
15/02/22 18:50:48 INFO namenode.FSNamesystem: No KeyProvider found.
15/02/22 18:50:48 INFO namenode.FSNamesystem: fsLock is fair:true
15/02/22 18:50:48 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
15/02/22 18:50:48 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
15/02/22 18:50:48 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
15/02/22 18:50:48 INFO blockmanagement.BlockManager: The block deletion will start around 2015 Feb 22 18:50:48
15/02/22 18:50:48 INFO util.GSet: Computing capacity for map BlocksMap
15/02/22 18:50:48 INFO util.GSet: VM type = 64-bit
15/02/22 18:50:48 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
15/02/22 18:50:48 INFO util.GSet: capacity = 2^21 = 2097152 entries
15/02/22 18:50:48 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
15/02/22 18:50:48 INFO blockmanagement.BlockManager: defaultReplication = 1
15/02/22 18:50:48 INFO blockmanagement.BlockManager: maxReplication = 512
15/02/22 18:50:48 INFO blockmanagement.BlockManager: minReplication = 1
15/02/22 18:50:48 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
15/02/22 18:50:48 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
15/02/22 18:50:48 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
15/02/22 18:50:48 INFO blockmanagement.BlockManager: encryptDataTransfer = false
15/02/22 18:50:48 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
15/02/22 18:50:48 INFO namenode.FSNamesystem: fsOwner = hduser (auth:SIMPLE)
15/02/22 18:50:48 INFO namenode.FSNamesystem: supergroup = supergroup
15/02/22 18:50:48 INFO namenode.FSNamesystem: isPermissionEnabled = true
15/02/22 18:50:48 INFO namenode.FSNamesystem: HA Enabled: false
15/02/22 18:50:48 INFO namenode.FSNamesystem: Append Enabled: true
15/02/22 18:50:48 INFO util.GSet: Computing capacity for map INodeMap
15/02/22 18:50:48 INFO util.GSet: VM type = 64-bit
15/02/22 18:50:48 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
15/02/22 18:50:48 INFO util.GSet: capacity = 2^20 = 1048576 entries
15/02/22 18:50:48 INFO namenode.NameNode: Caching file names occuring more than 10 times
15/02/22 18:50:48 INFO util.GSet: Computing capacity for map cachedBlocks
15/02/22 18:50:48 INFO util.GSet: VM type = 64-bit
15/02/22 18:50:48 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
15/02/22 18:50:48 INFO util.GSet: capacity = 2^18 = 262144 entries
15/02/22 18:50:48 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
15/02/22 18:50:48 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
15/02/22 18:50:48 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
15/02/22 18:50:48 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
15/02/22 18:50:48 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
15/02/22 18:50:48 INFO util.GSet: Computing capacity for map NameNodeRetryCache
15/02/22 18:50:48 INFO util.GSet: VM type = 64-bit
15/02/22 18:50:48 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
15/02/22 18:50:48 INFO util.GSet: capacity = 2^15 = 32768 entries
15/02/22 18:50:48 INFO namenode.NNConf: ACLs enabled? false
15/02/22 18:50:48 INFO namenode.NNConf: XAttrs enabled? true
15/02/22 18:50:48 INFO namenode.NNConf: Maximum size of an xattr: 16384
Re-format filesystem in Storage Directory /usr/local/hadoop_tmp/hdfs/namenode ? (Y or N) Y
15/02/22 18:50:50 INFO namenode.FSImage: Allocated new BlockPoolId: BP-948369552-127.0.1.1-1424627450316
15/02/22 18:50:50 INFO common.Storage: Storage directory /usr/local/hadoop_tmp/hdfs/namenode has been successfully formatted.
15/02/22 18:50:50 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/02/22 18:50:50 INFO util.ExitUtil: Exiting with status 0
15/02/22 18:50:50 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at marta-komputer/127.0.1.1
************************************************************/
启动dfs
和yarn
会产生以下输出:
hduser@marta-komputer:/usr/local/hadoop$ start-dfs.sh
15/02/22 18:53:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-marta-komputer.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-marta-komputer.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-marta-komputer.out
15/02/22 18:53:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hduser@marta-komputer:/usr/local/hadoop$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-marta-komputer.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-marta-komputer.out
在此之后不久致电jps
:
hduser@marta-komputer:/usr/local/hadoop$ jps
11696 ResourceManager
11842 NodeManager
11171 NameNode
11523 SecondaryNameNode
12167 Jps
netstat 输出:
hduser@marta-komputer:/usr/local/hadoop$ sudo netstat -lpten | grep java
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 1001 690283 11696/java
tcp 0 0 0.0.0.0:42745 0.0.0.0:* LISTEN 1001 684574 11842/java
tcp 0 0 0.0.0.0:13562 0.0.0.0:* LISTEN 1001 680955 11842/java
tcp 0 0 0.0.0.0:8030 0.0.0.0:* LISTEN 1001 684531 11696/java
tcp 0 0 0.0.0.0:8031 0.0.0.0:* LISTEN 1001 684524 11696/java
tcp 0 0 0.0.0.0:8032 0.0.0.0:* LISTEN 1001 680879 11696/java
tcp 0 0 0.0.0.0:8033 0.0.0.0:* LISTEN 1001 687392 11696/java
tcp 0 0 0.0.0.0:8040 0.0.0.0:* LISTEN 1001 680951 11842/java
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 1001 687242 11171/java
tcp 0 0 0.0.0.0:8042 0.0.0.0:* LISTEN 1001 680956 11842/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 1001 690252 11523/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 1001 687239 11171/java
/ etc / hosts 文件:
127.0.0.1 localhost
127.0.1.1 marta-komputer
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
更新1。
我更新了 core-site.xml ,现在我有了:
<property>
<name>fs.default.name</name>
<value>hdfs://marta-komputer:9000</value>
</property>
但我一直收到错误 - 现在开头:
15/03/01 00:59:34 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
java.net.ConnectException: Call From marta-komputer.home/192.168.1.8 to marta-komputer:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
我还注意到telnet localhost 9000
无效:
hduser@marta-komputer:~$ telnet localhost 9000
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
答案 0 :(得分:42)
对我来说,这些步骤有效
stop-all.sh
hadoop namenode -format
start-all.sh
答案 1 :(得分:14)
您好编辑您的conf / core-site.xml并将localhost更改为0.0.0.0。使用下面的conf。这应该有用。
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://0.0.0.0:9000</value>
</property>
答案 2 :(得分:10)
最终,我设法 make 我的服务收听端口9000 ,方法是在/etc/ssh/sshd_config
文件中添加以下行:
Port 9000
我遵循了这个serverguide/openssh-server(它还包含一些关于制作原始文件副本,重新启动sshd服务器应用程序等的重要说明。)
在此之后,我可以看到:
telnet 输出:
martakarass@marta-komputer:~$ telnet localhost 9000
Trying 127.0.0.1...
Connected to localhost.
nmap 输出:
martakarass@marta-komputer:~$ nmap localhost
Starting Nmap 6.40 ( http://nmap.org ) at 2015-05-01 18:28 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00023s latency).
Not shown: 994 closed ports
PORT STATE SERVICE
22/tcp open ssh
139/tcp open netbios-ssn
445/tcp open microsoft-ds
631/tcp open ipp
902/tcp open iss-realsecure
9000/tcp open cslistener
Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds
netstat 输出:
martakarass@marta-komputer:~$ sudo netstat -nlp | grep :9000
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 16397/sshd
tcp6 0 0 :::9000 :::* LISTEN 16397/sshd
答案 3 :(得分:3)
从netstat
输出中,您可以看到进程正在侦听地址127.0.0.1
tcp 0 0 127.0.0.1:9000 0.0.0.0:* ...
从异常消息中您可以看到它尝试连接到地址127.0.1.1
java.net.ConnectException: Call From marta-komputer/127.0.1.1 to localhost:9000 failed ...
进一步提及
的例外情况For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
在此页面上找到
检查您的主机名是否已在/ etc / hosts中映射到127.0.0.1或127.0.1.1(Ubuntu因此而臭名昭着)
所以结论就是删除/etc/hosts
127.0.1.1 marta-komputer
答案 4 :(得分:3)
我和OP有类似的问题。作为终端输出建议,我去了 http://wiki.apache.org/hadoop/ConnectionRefused
我尝试按照此处的建议更改我的/ etc / hosts文件,即删除127.0.1.1,因为OP建议它会创建另一个错误。
所以最后,我保持原样。以下是我的/ etc / hosts
127.0.0.1 localhost.localdomain localhost
127.0.1.1 linux
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
最后,我发现我的名字没有正确启动,即
在终端中键入sudo netstat -lpten | grep java
时,端口9000上将不会运行(侦听)任何JVM进程。
所以我分别为namenode和datanode创建了两个目录(如果你还没有这样做的话)。您不必放在我放置的位置,请根据您的hadoop目录替换它。 即。
mkdir -p /home/hadoopuser/hadoop-2.6.2/hdfs/namenode
mkdir -p /home/hadoopuser/hadoop-2.6.2/hdfs/datanode
我重新配置了 hdfs-site.xml 。
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoopuser/hadoop-2.6.2/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoopuser/hadoop-2.6.2/hdfs/datanode</value>
</property>
</configuration>
在终端中,使用脚本stop-dfs.sh
和stop-yarn.sh
停止hdfs和yarn。它们位于hadoop目录/ sbin中。就我而言,它是/home/hadoopuser/hadoop-2.6.2/sbin /.
然后使用脚本start-dfs.sh
和start-yarn.sh
启动hdfs和yarn
启动后,在终端中键入jps
以查看JVM进程是否正常运行。它应该显示以下内容。
15678 NodeManager
14982 NameNode
15347 SecondaryNameNode
23814 Jps
15119 DataNode
15548 ResourceManager
然后再次尝试使用netstat查看您的namenode是否正在侦听端口9000
sudo netstat -lpten | grep java
如果您成功设置了namenode,则应在终端输出中看到以下内容。
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 1001 175157 14982/java
然后尝试输入命令hdfs dfs -mkdir /user/hadoopuser
如果此命令成功执行,现在您可以通过hdfs dfs -ls /user
答案 5 :(得分:2)
对我而言,我无法将我的动物园管理员聚集在一起。
hdfs haadmin -getServiceState 1
active
hdfs haadmin -getServiceState 2
active
我的hadoop-hdfs-zkfc- [hostname] .log显示:
2017-04-14 11:46:55,351 WARN org.apache.hadoop.ha.HealthMonitor: 试图监视NameNode健康状况的传输级异常 HOST / 192.168.1.55:9000:java.net.ConnectException:连接被拒绝 呼叫从HOST / 192.168.1.55到主机:9000连接失败 exception:java.net.ConnectException:连接被拒绝;更多 详情请参阅:http://wiki.apache.org/hadoop/ConnectionRefused
溶液:
hdfs-site.xml
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>0.0.0.0</value>
</property>
前
netstat -plunt
tcp 0 0 192.168.1.55:9000 0.0.0.0:* LISTEN 13133/java
nmap localhost -p 9000
Starting Nmap 6.40 ( http://nmap.org ) at 2017-04-14 12:15 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000047s latency).
Other addresses for localhost (not scanned): 127.0.0.1
PORT STATE SERVICE
9000/tcp closed cslistener
后
netstat -plunt
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 14372/java
nmap localhost -p 9000
Starting Nmap 6.40 ( http://nmap.org ) at 2017-04-14 12:28 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000039s latency).
Other addresses for localhost (not scanned): 127.0.0.1
PORT STATE SERVICE
9000/tcp open cslistener
答案 6 :(得分:2)
确保HDFS在线。从$HADOOP_HOME/sbin/start-dfs.sh
开始
完成后,telnet localhost 9001
的测试应该有效。
答案 7 :(得分:1)
检查防火墙设置 并设置
<property>
<name>fs.default.name</name>
<value>hdfs://MachineName:9000</value>
</property>
将localhost替换为计算机名称
答案 8 :(得分:1)
hduser @ marta-komputer:/ usr / local / hadoop $ jps
11696 ResourceManager
11842 NodeManager
11171 NameNode
11523 SecondaryNameNode
12167 Jps
您的DataNode在哪里? Connection refused
问题也可能是因为没有有效DataNode
。检查datanode日志中的问题。
<强>更新:强>
对于此错误:
15/03/01 00:59:34 INFO client.RMProxy:在/0.0.0.0:8032连接到ResourceManager java.net.ConnectException:调用从marta-komputer.home / 192.168.1.8到marta-komputer:连接异常时9000失败:java.net.ConnectException:连接被拒绝;有关详细信息,请参阅:http://wiki.apache.org/hadoop/ConnectionRefused
在 yarn-site.xml :
中添加这些行<property>
<name>yarn.resourcemanager.address</name>
<value>192.168.1.8:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>192.168.1.8:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>192.168.1.8:8031</value>
</property>
重新启动hadoop进程。
答案 9 :(得分:0)
在我的经历中
15/02/22 18:23:04 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
您可能拥有64位版本操作系统,并且hadoop安装32位。参考this
java.net.ConnectException: Call From marta-komputer/127.0.1.1 to
localhost:9000 failed on connection exception: java.net.ConnectException:
connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
此问题是指您的ssh公钥授权。请提供有关ssh设置的详细信息。
请参阅this链接以查看完整步骤。
还提供信息
cat $HOME/.ssh/authorized_keys
返回任何结果。
答案 10 :(得分:0)
你的问题非常有趣。由于系统的复杂性和涉及的许多移动部件,Hadoop设置可能会令人沮丧。我认为你面临的问题绝对是防火墙问题。 我的hadoop集群有类似的设置。使用命令添加防火墙规则:
sudo iptables -A INPUT -p tcp --dport 9000 -j REJECT
我能够看到确切的问题:
15/03/02 23:46:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
java.net.ConnectException: Call From mybox/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
您可以使用以下命令验证防火墙设置:
/usr/local/hadoop/etc$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
REJECT tcp -- anywhere anywhere tcp dpt:9000 reject-with icmp-port-unreachable
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
一旦识别出可疑规则,就可以使用以下命令删除它:
sudo iptables -D INPUT -p tcp --dport 9000 -j REJECT
现在,连接应该通过。
答案 11 :(得分:0)
我通过将此属性添加到hdfs-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
答案 12 :(得分:0)
在/ etc / hosts中:
your-ip-address your-host-name
示例:192.168.1.8 master
在/ etc / hosts中:
删除127.0.1.1行(这将导致环回)
在您的核心站点中,将localhost更改为您的-ip或您的主机名
现在,重新启动群集。
答案 13 :(得分:0)
按以下方式停止它:stop-all.sh
格式化namenode-:hadoop namenode -format
再次启动-:start-all.sh
答案 14 :(得分:0)
我在Hortonworks中也面临着相同的问题
当我重新启动Ambari代理和服务器时,问题已解决。
systemctl stop ambari-agent
systemctl stop ambari-server
来源:Full Article With Resolution
systemctl start ambari-agent
systemctl start ambari-server
答案 15 :(得分:0)
我遇到了同样的问题,发现OpenSSH服务未运行,并且导致了问题。启动SSH服务后,它可以正常工作。
要检查SSH服务是否正在运行:
ssh localhost
要启动服务,如果已经安装了OpenSSH:
sudo /etc/init.d/ssh start
答案 16 :(得分:0)
进入$SPARK_HOME/conf
,然后打开文件spark-env.sh
并添加:
SPARK_MASTER_HOST= your-IP
SPARK_LOCAL_IP=127.0.0.1