Question

我正在学习一些MapReduce，但我遇到了一些问题，在这种情况下：我有两个文件：＆＃34;用户＆＃34;包含一些用户列表，其中包含一些数据（性别，年龄，国家/地区等），文件如下所示：

user_000003  m  22  United States   Oct 30, 2005

＆＃34;歌曲＆＃34;包含所有用户收听的歌曲数据（用户ID，收听日期和时间，艺术家ID，艺术家姓名，歌曲ID，歌曲名称）：

user_000999 2008-12-11T22:52:33Z    b7ffd2af-418f-4be2-bdd1-22f8b48613da    Nine Inch Nails 1d1bb32a-5bc6-4b6f-88cc-c043f6c52509    7 Ghosts I

目标是在某些国家/地区找到最受欢迎的歌曲。用k和输入中提供的国家列表。

我决定为映射器使用MultipleInputs类，因此一个映射器将输出一组键值对，如下所示：。另一个映射器将输出。据我所知，我应该能够读取与某个键配对的所有值（所以我应该在reducer中找到与userID相关联的值列表中的国家和一定数量的歌曲）并输出一组另一个MapReduce作业将成对的文件变为红色。

我非常确定映射器能够完成工作，因为我能够使用reducer编写输出。

更新：使用以下代码将文件传递给映射器：

Job job = Job.getInstance(conf);

        MultipleInputs.addInputPath(job, new Path(songsFile), TextInputFormat.class, SongMapper.class);
        MultipleInputs.addInputPath(job, new Path(usersFile), TextInputFormat.class, UserMapper.class);

        FileOutputFormat.setOutputPath(job, new Path(outFile));
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Text.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        job.setJarByClass(Songs.class);

        job.setCombinerClass(Combiner.class);
        job.setReducerClass(UserSongReducer.class);

Mappers代码：

public static class UserMapper extends Mapper<LongWritable, Text, Text, Text>{

        //empty cleanUp()

        protected void map(LongWritable key, Text value, Context context)
                throws IOException, InterruptedException {

            String record = value.toString();               
            String[] userData = record.split("\t");
            if(userData.length>3 && !userData[3].equals(""))
            {
                context.write(new Text(userData[0]), new Text(userData[3]));
            }        


        }

        //empty run() and setup()

    }

    public static class SongMapper extends Mapper<LongWritable, Text, Text, Text>{

        //empty cleanUp()

        protected void map(LongWritable key, Text value, Context context)
                throws IOException, InterruptedException {

            String record = value.toString();               
            String[] songData = record.split("\t");
            if(songData.length>3 && !songData[3].equals(""))
            {
                context.write(new Text(songData[0]), new Text(songData[5]+" |||  "+songData[3]));
            }        


        }

        //empty run() and setup()

    }

组合码：

public static class Combiner extends Reducer<Text, Text, Text, Text> 
    {

        private boolean isCountryAllowed(String toCheck, String[] countries)
        {

            for(int i=0; i<countries.length;i++)
            {
                if(toCheck.equals(countries[i]))
                    return true;
            }
            return false;
        }

        public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException 
        {



            ArrayList<String> list = new ArrayList<String>();



            String country = "foo";
            for(Text value : values) 
            {
                if(!value.toString().contains(" ||| "))
                {
                    country = value.toString();
                }else
                {
                    list.add(value.toString());
                }

            }

            if(isCountryAllowed(country, context.getConfiguration().getStrings("countries")))
            {

                for (String listVal : list) 
                {
                    context.write(new Text(country), 
                            new Text(listVal));
                }
            }

         }
    }

当我尝试使用reducer输出对时出现问题：

public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException 
        {


             for (Text value : values) 
            {
                context.write(key,value);
            }
            }

         }

我用＆＃34; ||| ＆＃34;建立艺术家+标题字符串问题是国家仍然是＃foo＆＃34;。我认为我应该看到至少有一行输出正确的国家作为关键，但输出总是＆＃34; foo＆＃34; （2,5kb歌曲文件）：

foo Deep Dish |||  Fuck Me Im Famous (Pacha Ibiza)-09-28-2007
foo Vnv Nation |||  Kingdom
foo Les Fleur De Lys |||  Circles
foo Home Video |||  Penguin
foo Of Montreal |||  Will You Come And Fetch Me
foo Godspeed You! Black Emperor |||  Bbf3
foo Alarum |||  Sustained Connection
foo Sneaker Pimps |||  Walking Zero
foo Cecilio And Kapono |||  I Love You
foo Garbage |||  Vow
foo The Brian Setzer Orchestra |||  Gettin' In The Mood
foo Nitin Sawhney |||  Sunset (J-Walk Remix)
foo Nine Inch Nails |||  Heresy
foo Collective Soul |||  Crowded Head
foo Vicarious Bliss |||  Limousine
foo Noisettes |||  Malice In Wonderland
foo Black Rebel Motorcycle Club |||  Lien On Your Dreams
foo Mae |||  Brink Of Disaster
foo Michael Andrews |||  Rosie Darko
foo A Perfect Circle |||  Blue

我做错了什么？

如果我使用自定义组合器，我应该能够避免第二个工作，组合器是否像减速器一样？

Answer 1

从我的代码中我看到，这个国家应该永远是＆＃34; foo＆＃34;，你到底想要实现什么？

 // IN YOUR MAPPER THE VALUE IS WRITTEN USING |||

context.write(new Text(songData[0]), new Text(songData[5]+" |||  "+songData[3]));

// THEN VALUE WILL ALWAYS CONTAIN ||| AND WILL NEVER CHANGE THE COUNTRY THAT WAS SET TO TRUE
 String country = "foo";
     if(!value.toString().contains(" ||| ")) //never ---

然后您输出国家/地区变量：

 context.write(new Text(country),

hadoop-mapreduce reducer-combiner输入

1 个答案: