Question

我有一个项目，我需要使用包含许多列的逗号分隔文件，并提取公司名称，客户互动的结果以及发生的次数。
然后我需要计算不良互动与良好互动的百分比我正在使用Hadoop和Java 我有一个有效的Map和Reduce，它给我公司名称和有多少好的和坏的互动。

我的问题是，我找不到让Hadoop划分好坏的方法来给我一个百分比。
大多数公司没有任何不良的互动。

这是我的Mapp

public class TermProjectMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

    private final static IntWritable one = new IntWritable( 1); 
    private Text word = new Text();

      @Override
      public void map(LongWritable key, Text value, Context context)
          throws IOException, InterruptedException {

            String[] columb = value.toString().split(",");
            String companyName = columb[5];
            String companyResponseToConsumer = columb[12];
            String lookfor = "closed without relief";

                if (companyResponseToConsumer.toLowerCase().contains(lookfor)) {companyResponseToConsumer="Bad";}
                else {companyResponseToConsumer="Good";}
                //System.out.println(companyResponseToConsumer);
                if (companyName != "" && companyResponseToConsumer != "")
                {
                    word.set (companyName + " " + companyResponseToConsumer);
                    context.write( word, one); 
                }
      }
      }

这是我的减少

public class TermProjectReducer extends Reducer < Text, IntWritable, Text, IntWritable > 
{ 
    private IntWritable result = new IntWritable(); 
      @Override
        public void reduce( Text key, Iterable < IntWritable > values, Context context) throws IOException, InterruptedException 
        { 
            int sum = 0; 
            for (IntWritable val : values) 
            { 
                sum += val.get(); 
            } 
            if (sum > 0) 
            {
                result.set( sum); 
                context.write( key, result);
            }
        } 
    }

这是我现在所得到的一个例子。

AMERICAN EAGLE MORTGAGE COMPANY,Good,   4
AMERICAN EQUITY MORTGAGE,Good,  26 
AMERICAN EXPRESS COMPANY,Bad,   250 
AMERICAN EXPRESS COMPANY,Good,  9094 
AMERICAN FEDERAL MORTGAGE CORPORATION,Bad,  1 
AMERICAN FEDERAL MORTGAGE CORPORATION,Good, 3 
AMERICAN FINANCE HOUSE LARIBA,Good, 3 
AMERICAN FINANCIAL MORTGAGE COMPANY,Good,   3

Answer 1

为了聚合公司，您需要输出那些作为键，以便它们在reducer中组合。换句话说，您希望在同一个键上同时使用好的和坏的值，而不是像现在这样分开。

我最初认为您可以[1, 0]或[0, 1]，但只输出1或-1而不是("GOOD", 1)和("BAD", 1)更容易处理。（并且Hadoop的数据传输效率更高）

所以，例如，

private final static IntWritable ONE = new IntWritable(1); 
private final static IntWritable NEG_ONE = new IntWritable(-1); 

...

    IntWritable status;
    if (companyResponseToConsumer.toLowerCase().contains(lookfor)) {status=NEG_ONE;}
    else {status=ONE;}

    if (!companyName.isEmpty())
    {
        word.set (companyName);
        context.write(companyName, status); 
    }

现在，在reducer中，计算值并计算百分比。

public class TermProjectReducer extends Reducer < Text, IntWritable, Text, IntWritable > 
{ 
  private IntWritable result = new IntWritable(); 

  @Override
    public void reduce( Text key, Iterable < IntWritable > values, Context context) throws IOException, InterruptedException 
    { 
        int total = 0; 
        int good_sum = 0;
        for (IntWritable val : values) 
        { 
            good_sum += (val.get() == 1 ? 1 : 0); 
            total += 1
        } 
        if (total > 0) // Prevent division by zero
        {
            double percent = 1.0*good_sum/total;
            // Round it to how every many decimal places, if you want
            result.set(String.valueOf(percent)); // convert the floating number to a string
        } else {
            result.set("0.00"); 
        }
        context.write(key, result); 
    } 
}

我只计算好的值，因为在您的下游处理中，您可以自己(1 - good) = bad。

另外，我建议使用DoubleWritable作为Reducer值而不是Text

获得Hadoop的百分比

1 个答案: