ArchLinux上的Hadoop（伪分布式操作）

时间：2017-12-28 20:51:26

标签： java linux hadoop archlinux

我想在ArchLinux中使用hadoop，所以我在这里尝试（https://wiki.archlinux.org/index.php/Hadoop），但我能做到。

我试过......

  1. 从AUR安装hadoop（hadoop 3.0.0-1）

  2. 权限更改使用

    $sudo chmod o+r /etc/hadoop && sudo chmod o+rw /tmp/hadoop/log ...

  3. 在/etc/hadoop/hadoop-env.sh

    中添加此代码

    export JAVA_HOME=/usr/lib/jvm/java-8-openjdk

  4. 更改/etc/conf.d/hadoop

    中的名称

    HADOOP_USERNAME="<my user name>"

  5. （我可以在没有密码的情况下ssh到localhost，所以我没有为ssh做任何事情）

  6. 尝试hadoop namenode -format 我收到了一些WARM消息

    WARNING: HADOOP_SLAVES has been replaced by HADOOP_WORKERS. Using value of HADOOP_SLAVES.
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.

  7. 因此，我将<{1}}改为

    /etc/profile.d/hadoop.sh

  8. 我再次export HADOOP_CONF_DIR=/etc/hadoop export HADOOP_LOG_DIR=/tmp/hadoop/log export HADOOP_WORKERS=/etc/hadoop/workers export HADOOP_PID_DIR=/tmp/hadoop/run （我在shell中找不到hdfs）

  9. 关掉我的hadoop 我收到了这些消息

    hadoop namenode -format

  10. 启动服务使用[<my user name>@<pc name> etc]$ hadoop namenode -format WARNING: Use of this script to execute namenode is deprecated. WARNING: Attempting to execute replacement "hdfs namenode" instead. 2017-12-29 05:08:01,679 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = <my user name>/10.0.2.15 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 3.0.0 STARTUP_MSG: classpath = ... STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r c25427ceca461ee979d30edd7a4b0f50718e6533; compiled by 'andrew' on 2017-12-08T19:16Z STARTUP_MSG: java = 1.8.0_144 ************************************************************/ 2017-12-29 05:08:01,685 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2017-12-29 05:08:01,688 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-d2e0965e-76ff-456b-8552-9da5706b8a6b 2017-12-29 05:08:02,355 INFO namenode.FSEditLog: Edit logging is async:true 2017-12-29 05:08:02,389 INFO namenode.FSNamesystem: KeyProvider: null 2017-12-29 05:08:02,391 INFO namenode.FSNamesystem: fsLock is fair: true 2017-12-29 05:08:02,392 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false 2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: fsOwner = <my user name> (auth:SIMPLE) 2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: supergroup = supergroup 2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: isPermissionEnabled = true 2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: HA Enabled: false 2017-12-29 05:08:02,452 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 2017-12-29 05:08:02,471 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000 2017-12-29 05:08:02,471 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 2017-12-29 05:08:02,486 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 2017-12-29 05:08:02,486 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Dec 29 05:08:02 2017-12-29 05:08:02,488 INFO util.GSet: Computing capacity for map BlocksMap 2017-12-29 05:08:02,488 INFO util.GSet: VM type = 64-bit 2017-12-29 05:08:02,490 INFO util.GSet: 2.0% max memory 878.5 MB = 17.6 MB 2017-12-29 05:08:02,494 INFO util.GSet: capacity = 2^21 = 2097152 entries 2017-12-29 05:08:02,546 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false 2017-12-29 05:08:02,555 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: defaultReplication = 3 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxReplication = 512 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: minReplication = 1 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: encryptDataTransfer = false 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 2017-12-29 05:08:02,625 INFO util.GSet: Computing capacity for map INodeMap 2017-12-29 05:08:02,626 INFO util.GSet: VM type = 64-bit 2017-12-29 05:08:02,626 INFO util.GSet: 1.0% max memory 878.5 MB = 8.8 MB 2017-12-29 05:08:02,626 INFO util.GSet: capacity = 2^20 = 1048576 entries 2017-12-29 05:08:02,627 INFO namenode.FSDirectory: ACLs enabled? false 2017-12-29 05:08:02,627 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true 2017-12-29 05:08:02,627 INFO namenode.FSDirectory: XAttrs enabled? true 2017-12-29 05:08:02,627 INFO namenode.NameNode: Caching file names occurring more than 10 times 2017-12-29 05:08:02,631 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true 2017-12-29 05:08:02,643 INFO util.GSet: Computing capacity for map cachedBlocks 2017-12-29 05:08:02,643 INFO util.GSet: VM type = 64-bit 2017-12-29 05:08:02,643 INFO util.GSet: 0.25% max memory 878.5 MB = 2.2 MB 2017-12-29 05:08:02,643 INFO util.GSet: capacity = 2^18 = 262144 entries 2017-12-29 05:08:02,649 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 2017-12-29 05:08:02,654 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 2017-12-29 05:08:02,654 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 2017-12-29 05:08:02,670 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 2017-12-29 05:08:02,670 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 2017-12-29 05:08:02,685 INFO util.GSet: Computing capacity for map NameNodeRetryCache 2017-12-29 05:08:02,685 INFO util.GSet: VM type = 64-bit 2017-12-29 05:08:02,685 INFO util.GSet: 0.029999999329447746% max memory 878.5 MB = 269.9 KB 2017-12-29 05:08:02,686 INFO util.GSet: capacity = 2^15 = 32768 entries Re-format filesystem in Storage Directory /tmp/hadoop-elect/dfs/name ? (Y or N) Y 2017-12-29 05:08:04,846 INFO namenode.FSImage: Allocated new BlockPoolId: BP-142461510-10.0.2.15-1514491684838 2017-12-29 05:08:04,846 INFO common.Storage: Will remove files: [/tmp/hadoop-<my user name>/dfs/name/current/fsimage_0000000000000000000, /tmp/hadoop-<my user name>/dfs/name/current/fsimage_0000000000000000000.md5, /tmp/hadoop-<my user name>/dfs/name/current/seen_txid, /tmp/hadoop-<my user name>/dfs/name/current/VERSION] 2017-12-29 05:08:04,863 INFO common.Storage: Storage directory /tmp/hadoop-<my user name>/dfs/name has been successfully formatted. 2017-12-29 05:08:04,870 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-<my user name>/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 2017-12-29 05:08:04,981 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-<my user name>/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 387 bytes saved in 0 seconds. 2017-12-29 05:08:04,986 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 2017-12-29 05:08:04,989 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at <my user name>/10.0.2.15 ************************************************************/

  11. 检查服务使用sudo systemctl start hadoop-datanode(jobtracker etc...) 

    sudo systemctl status hadoop-...

  12. 我找不到这个解决方案

    请告诉我解决方案。

0 个答案:

没有答案
相关问题
最新问题