Hadoop MapReduce如何将字符串参数传递给Mapper

时间:2015-12-08 22:11:50

标签: python hadoop

我一直在SSH上使用Hadoop MapReduce命令,如下所示:

hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.0.1.jar -file /nfs_home/appers/user1/mapper.py -file /nfs_home/appers/user1/reducer.py -mapper '/usr/lib/python_2.7.3/bin/python mapper.py' -reducer '/usr/lib/python_2.7.3/bin/python reducer.py' -input /ccexp/data/test_xml/0901282-510179094535002-oozie-oozi-W/extractOut//.xml -output /user/ccexptest/output/user1/MRoutput

但是,我正在扩展Mapper.py脚本的功能,并希望在启动MapReduce作业时将字符串参数传递给Mapper。如何编辑上面的MR命令,以便将字符串作为参数包含在Mapper中?

1 个答案:

答案 0 :(得分:1)

根据您的情况,您可以将任意命令行输入传递到-mapper-reducer

-mapper '/usr/lib/python_2.7.3/bin/python mapper.py myargs'