Question

我在AWS的弹性地图缩减群集中运行了大量工作。从大到大，我指的是我正在处理的超过800,000个文件，每个文件有25,000多条记录。在我的测试运行中，我一直在使用100 m1.medium spot实例进行处理。

作业似乎正在正常运行，但是我注意到输出（部分00000，部分00001等）具有在多个输出中列出的相同键的记录。这些应该在EMR中降低吗？

任何见解都将受到赞赏。

Answer 1

我遇到了同样的问题 - 我正在使用EMR使用流API创建“倒排索引”：

-input s3n：// mybucket / html2 -output s3n：// mybucket / results -mapper s3n：//mybucket/mapper.py -reducer s3n：//mybucket/reduce.py

// mybucket / html2有几个html文件和

mapper.py：

def main(args):
    for line in sys.stdin:
        line = line.strip()
        words = line.split()            
        for word in words:
            #do some preprocessing
            if word.startswith("http://"):
                #output the URL with a count of 1
                print "%s,%s" % (word, 1)
            else:           
                #cleanup HTML tags

                url = get_url() #irrelevant
                print "%s,%s" % (word, url)

if __name__ == "__main__":
    main(sys.argv)

和reduce.py是：

def main(args):
    current_word = None
    current_count = 0
    current_url_list = []
    key = None

    for line in sys.stdin:
        line = line.strip()
        (key, val) = line.split(',', 1)

        # If key is a URL - act as word count reducer
        if key.startswith("http:"):
            # convert count (currently a string) to int
            try:
                count = int(val)
            except:
                # count was not a number, so silently
                # ignore/discard this line
                continue

            # this IF-switch only works because Hadoop sorts map output
            # by key (here: word) before it is passed to the reducer
            if current_word == key:
                current_count += count
            else:
                if current_word:
                     #Check if previous word was a regular word
                    if current_word.startswith('http:'):
                        print '%s,%s' % (current_word, current_count)
                    else:
                        # previous word was a regular word
                        print '%s,%s' % (current_word, ','.join(current_url_list))
                current_count = count
                current_word = key
        else:
            #If key is a word - as act a URL-list-appending reducer
            if current_word == key:
                if val not in current_url_list:
                    current_url_list.append(val)
            else: #Got to a new key
                if current_word:
                    #Check if previous word was a URL
                    if(current_word.startswith("http:")):
                        print '%s,%s' % (current_word, current_count)
                    else:
                        # previous word was a regular word
                        print '%s,%s' % (current_word, ','.join(current_url_list))
                current_url_list = []
                current_url_list.append(val)
                current_word = key

我正在使用AWS控制台向导启动此流程（“创建新的作业流程”），除了设置输入，输出，映射和减少脚本之外，我将所有内容保留为默认值（日志路径除外）。

在输出中我得到的文件很少，在其中我看到相同的键（每次都有不同的值）。

也许这可以帮助更多地了解这个问题并帮助解决它

AWS EMR机器没有组合减少输出

1 个答案: