Question

我正在使用Kaggle数据集开发一个简单的map reduce程序 https://www.kaggle.com/datasnaek/youtube-new

数据集包含40950个具有16个变量的视频记录，例如video_id，trending_date，标题，channel_title，category_id，publish_time，标签，观看次数，喜欢，不喜欢，comment_count，描述等。

我的MapReduce程序的目的是查找描述中包含“ iPhoneX”且具有至少10,000个点赞的所有视频。最终输出应仅包含（标题，视频数）

驱动程序类打包解决方案；

public class Driver extends Configured implements Tool{
    @Override
    public int run(String[] args) throws Exception{
        if(args.length != 2){
        System.out.printf("Usage: Driver <input dir> <output dir> \n");
        return -1;
        }
        Job job = new Job(getConf());
        job.setJarByClass(Driver.class);
        job.setJobName("iPhoneX");

        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.setMapperClass(Mapper.class);
        job.setReducerClass(Reducer.class);

        //Specify Combiner as the combiner class

        job.setCombinerClass(Reducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        if(job.getCombinerClass() == null){
            throw new Exception("Combiner not set");
        }

        boolean success = job.waitForCompletion(true);
        return success ? 0 : 1; 

    }
    /* The main method calls the ToolRunner.run method,
     * which calls the options parser that interprets Hadoop terminal
     * options and puts them into a config object
     * */
    public static void main(String[] args) throws Exception{

        int exitCode = ToolRunner.run(new Configuration(), new Driver(),args);
        System.exit(exitCode);
    }
}

减速器类

package solution;

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class Reducer extends Reducer<Text, IntWritable, Text, IntWritable>{

    @Override
    public void reduce(Text key, Iterable<IntWritable> values, Context context)
    throws IOException, InterruptedException{

        int video_count = 0;
        for(IntWritable value : values){
            video_count += value.get();
        }
        context.write(key, new IntWritable(video_count));
    }
}

映射器类

public class Mapper extends Mapper<LongWritable, Text, Text, IntWritable> {

    private Text description = new Text();
    private IntWritable likes = new IntWritable();
    @Override
    public void map(LongWritable key, Text value, Context context)

    throws IOException, InterruptedException{

        String line = value.toString();
        String str[] = line.split("\t");

        if(str.length > 3){
            description.set(str[8]);
        }

// Testing how many times the iPhoneX word is located in the data set       
//      StringTokenizer itr = new StringTokenizer(line);
//      
//      while(itr.hasMoreTokens()){
//          String token = itr.nextToken();
//          if(token.contains("iPhoneX")){
//              word.set("iPhoneX Count");
//              context.write(word, new IntWritable(1));
//          }
//      }
    }
}

Answer 1

您的代码看起来不错，但是您将需要取消注释映射器输出任何数据的部分，但是，映射器键应该只是“ iPhone”，并且您可能希望标记描述，而不是整个描述线

您还希望提取喜欢的人数，并仅过滤出符合问题集所列条件的那些人

顺便说一下，您至少需要9个元素才能获得该位置，而不仅仅是3个，因此请在此处更改条件

if(str.length >= 9){
    description.set(str[8]);

    likes = Integer.parseInt(str[...]);
    if (likes >= 10000) {
        // TODO: find when description string contains iPhoneX 
        context.write("IPhoneX", count);
    } 
} else {
    return; // skip line 
}

或者，您可以只为“ iPhoneX”的每个令牌写出（令牌，1），而不是在映射器中进行预聚合，然后让组合器和约简器为您求和

MapReduce字数统计功能可在数据集中找到特定的字词

1 个答案: