如何在使用python MRJob库运行mapreduce程序时在终端上显示中间值(即打印变量或列表)?
答案 0 :(得分:6)
您可以使用sys.stderr.write()将结果输出为标准错误。这是一个例子:
from mrjob.job import MRJob
import sys
class MRWordCounter(MRJob):
def mapper(self, key, line):
sys.stderr.write("MAPPER INPUT: ({0},{1})\n".format(key,line))
for word in line.split():
yield word, 1
def reducer(self, word, occurrences):
occurencesList= list(occurrences)
sys.stderr.write("REDUCER INPUT: ({0},{1})\n".format(word,occurencesList))
yield word, sum(occurencesList)
if __name__ == '__main__':
MRWordCounter.run()