我的MR程序有一些奇怪的问题，并且不知道它为什么会这样工作。 也许可以给我一个提示有什么不对吗？
我的Mapper功能如何：
Integer Click_ID = 0;
public void map(LongWritable key, Text value, Context context)
throws IOException , InterruptedException
{
String line = value.toString();
String []lineArr = line.split("\t");
String nm_uv_id = lineArr[0];
String session_id = lineArr[1];
String time_stamp = lineArr[2];
String click_counter = lineArr[3];
String is_robot = lineArr[4];
Click_ID++;
String full_line = Click_ID + "\t"+ nm_uv_id +"\t"+ session_id+"\t"+time_stamp+"\t"+click_counter+"\t"+ is_robot;
context.write(new Text(session_id), new Text(full_line));
}
到目前为止一切正常 - 当我设置Reducers的数量= 0时，我的mapper会产生预期的输出。
以下是我的Reducer的样子。我想要做的是在每个Iterable我的键中迭代两次。这样做，我试图在一个单独的ArrayList中缓存我的Iterable的每个值：
public void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
List<Text> cache = new ArrayList<Text>();
// first iterable
for (Text value : values) {
cache.add(value); }
//second iterable
for (Text entity : cache) {
context.write(key, entity); }
}
}
我用于MR的输入看起来像这样：
nm_uv_id_1 session_id_2 1234567891 1 is_robot_no
nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
nm_uv_id_1 session_id_2 1234567893 3 is_robot_no
nm_uv_id_1 session_id_2 1234567894 3 is_robot_no
nm_uv_id_1 session_id_1 1234567895 1 is_robot_no
nm_uv_id_1 session_id_1 1234567896 2 is_robot_no
nm_uv_id_1 session_id_1 1234567897 3 is_robot_no
nm_uv_id_1 session_id_1 1234567898 4 is_robot_no
nm_uv_id_1 session_id_1 1234567899 5 is_robot_no
nm_uv_id_1 session_id_1 1234567888 6 is_robot_no
nm_uv_id_1 session_id_1 1234567890 7 is_robot_no
nm_uv_id_1 session_id_1 1234567890 8 is_robot_no
nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
nm_uv_id_1 session_id_1 1234567890 10 is_robot_no
nm_uv_id_1 session_id_3 1234567890 1 is_robot_no
nm_uv_id_2 session_id_4 1234587890 1 is_robot_no
nm_uv_id_2 session_id_4 1234587890 2 is_robot_no
nm_uv_id_2 session_id_4 1234587890 3 is_robot_no
nm_uv_id_2 session_id_4 1234587890 4 is_robot_no
nm_uv_id_2 session_id_4 1234587890 5 is_robot_no
nm_uv_id_2 session_id_4 1234587890 6 is_robot_no
nm_uv_id_2 session_id_4 1234587890 7 is_robot_no
nm_uv_id_2 session_id_4 1234587890 8 is_robot_no
nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
nm_uv_id_2 session_id_5 1234587890 1 is_robot_no
nm_uv_id_2 session_id_5 1234587890 2 is_robot_no
nm_uv_id_2 session_id_5 1234587890 3 is_robot_yes
nm_uv_id_2 session_id_5 1234587890 4 is_robot_yes
nm_uv_id_2 session_id_5 1234587890 5 is_robot_no
nm_uv_id_2 session_id_5 123457890 6 is_robot_no
但是我的输出文件如下所示：
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
session_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
session_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
session_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
session_id_3 15 nm_uv_id_1 session_id_3 1234567890 1 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
我不知道为什么reducer总是为一个特定的密钥写相同的键值对。我尝试了几个东西，似乎第一个for-loop，我在那里进行缓存工作正常。当我写context.write（key，value）时，我得到了我期望的输出。 然而第二个，当我想在第二个for循环中使用缓存时，程序会为我写一些奇怪的东西。
有人可以帮忙吗？
答案 0 :(得分：3)
重用相同的Text缓冲区作为优化。因此，您需要手动克隆以对其进行缓存。
我只想改变你的缓存循环：
for (Text value : values) { cache.add(new Text(value)); }