Question

我正在尝试使用python的MRJob包编写MapReduce作业。作业处理~336中存储的36,000个文件。每个文件大约2MB。当我在本地运行作业（将S3存储桶下载到我的计算机）时，运行大约需要1小时。但是，当我尝试在EMR上运行它时，需要更长时间（我在8小时停止它并且在映射器中完成了10％）。我已经为我的mapper_init和mapper附加了代码。有谁知道会导致像这样的问题？有谁知道如何修理它？我还应该注意，当我将输入限制为100个文件的样本时，它可以正常工作。

def mapper_init(self):
    """
    Set class variables that will be useful to our mapper:
        filename: the path and filename to the current recipe file
        previous_line: The line previously parsed. We need this because the
          ingredient name is in the line after the tag
    """

    #self.filename = os.environ["map_input_file"]  # Not currently used
    self.previous_line = "None yet"
    # Determining if an item is in a list is O(n) while determining if an
    #  item is in a set is O(1)
    self.stopwords = set(stopwords.words('english'))
    self.stopwords = set(self.stopwords_list)


def mapper(self, _, line):
    """
    Takes a line from an html file and yields ingredient words from it

    Given a line of input from an html file, we check to see if it
    contains the identifier that it is an ingredient. Due to the
    formatting of our html files from allrecipes.com, the ingredient name
    is actually found on the following line. Therefore, we save the
    current line so that it can be referenced in the next pass of the
    function to determine if we are on an ingredient line.

    :param line: a line of text from the html file as a str
    :yield: a tuple containing each word in the ingredient as well as a
        counter for each word. The counter is not currently being used,
        but is left in for future development. e.g. "chicken breast" would
        yield "chicken" and "breast"
    """

    # TODO is there a better way to get the tag?
    if re.search(r'span class="ingredient-name" id="lblIngName"',
                 self.previous_line):
        self.previous_line = line
        line = self.process_text(line)
        line_list = set(line.split())
        for word in line_list:
            if word not in self.stopwords:
                yield (word, 1)
    else:
        self.previous_line = line
    yield ('', 0)

Answer 1

问题是你有更多的小文件。使用s3distcp添加引导步骤以将文件复制到EMR。使用s3distcp时尝试将小文件聚合到~128MB文件中。

Hadoop对大量小文件不好。

由于您手动将文件下载到计算机并运行，因此运行速度更快。

使用S3distCP将文件复制到EMR后，使用HDFS中的文件。

MapReduce Job（用python编写）在EMR上运行缓慢

1 个答案: