我如何在Amazon EMR上使用Pydoop?
我试过谷歌搜索这个话题无济于事:它是否可能?
答案 0 :(得分:8)
我终于有了这个工作。一切都发生在主节点上... ssh到用户hadoop
的那个节点你需要一些包裹:
sudo easy_install argparse importlib
sudo apt-get update
sudo apt-get install libboost-python-dev
构建东西:
wget http://apache.mirrors.pair.com/hadoop/common/hadoop-0.20.205.0/hadoop-0.20.205.0.tar.gz
wget http://sourceforge.net/projects/pydoop/files/Pydoop-0.6/pydoop-0.6.0.tar.gz
tar xvf hadoop-0.20.205.0.tar.gz
tar xvf pydoop-0.6.0.tar.gz
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export JVM_ARCH=64 # I assume that 32 works for 32-bit systems
export HADOOP_HOME=/home/hadoop
export HADOOP_CPP_SRC=/home/hadoop/hadoop-0.20.205.0/src/c++/
export HADOOP_VERSION=0.20.205
export HDFS_LINK=/home/hadoop/hadoop-0.20.205.0/src/c++/libhdfs/
cd ~/hadoop-0.20.205.0/src/c++/libhdfs
sh ./configure
make
make install
cd ../install
tar cvfz ~/libhdfs.tar.gz lib
sudo tar xvf ~/libhdfs.tar.gz -C /usr
cd ~/pydoop-0.6.0
python setup.py bdist
cp dist/pydoop-0.6.0.linux-x86_64.tar.gz ~/
sudo tar xvf ~/pydoop-0.6.0.linux-x86_64.tar.gz -C /
保存两个tarball,将来可以跳过构建部分,只需执行以下操作即可安装(需要弄清楚如何在多节点集群上安装boostrap选项)
sudo tar xvf ~/libhdfs.tar.gz -C /usr
sudo tar xvf ~/pydoop-0.6.0.linux-x86_64.tar.gz -C /
然后我可以使用Full-fledged Hadoop API运行示例程序
(修复构造函数中的错误后调用super(WordCountMapper, self)
)。
#!/usr/bin/python
import pydoop.pipes as pp
class WordCountMapper(pp.Mapper):
def __init__(self, context):
super(WordCountMapper, self).__init__(context)
context.setStatus("initializing")
self.input_words = context.getCounter("WORDCOUNT", "INPUT_WORDS")
def map(self, context):
words = context.getInputValue().split()
for w in words:
context.emit(w, "1")
context.incrementCounter(self.input_words, len(words))
class WordCountReducer(pp.Reducer):
def reduce(self, context):
s = 0
while context.nextValue():
s += int(context.getInputValue())
context.emit(context.getInputKey(), str(s))
pp.runTask(pp.Factory(WordCountMapper, WordCountReducer))
我将该程序上传到存储桶并调用它运行。然后我使用了以下conf.xml:
<?xml version="1.0"?>
<configuration>
<property>
<name>hadoop.pipes.executable</name>
<value>s3://<my bucket>/run</value>
</property>
<property>
<name>mapred.job.name</name>
<value>myjobname</value>
</property>
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>true</value>
</property>
<property>
<name>hadoop.pipes.java.recordwriter</name>
<value>true</value>
</property>
</configuration>
最后,我使用了以下命令行:
hadoop pipes -conf conf.xml -input s3://elasticmapreduce/samples/wordcount/input -output s3://tmp.nou/asdf
答案 1 :(得分:2)
给出的答案只是部分正确,但解决方案更简单,就像这样做:
将此代码复制到您在计算机上创建的bash文件中:
#!/bin/bash
pip install pydoop
编写完此文件后,将其上传到s3存储桶。
然后你可以在emr中添加bootstrap动作:
选择&#34;自定义操作&#34;
给你的s3桶路径。
而且,你在你的emr集群中安装了一个pydoop。