如何减少主要BigInteger mapreduce代码的运行时间?

时间:2016-06-03 11:17:53

标签: performance mapreduce primes biginteger

我正在尝试生成大小为1764位(531位)的主要BigIntegers。当我在本地计算机上执行此操作时,需要很长时间。所以我尝试使用mapreduce生成BigIntegers并在单节点cloudera(CDH 4)上运行。但这需要花费大量时间来制作。我可以通过应用mapreduce并在多节点集群上实现它来减少时间吗?而我的第二个问题是这个程序可以改进以提高效率吗?

我的输入文件由90个包含“1764”的条目组成,这是随机BigInteger生成的位数。这是我的mapreduce代码

public final class Primes {

public final static void main(final String[] args) throws Exception {

   final Configuration conf = new Configuration();

   final Job job = new Job(conf, "Primes");
   job.setJarByClass(Primes.class);

    job.setOutputKeyClass(NullWritable.class);
    job.setOutputValueClass(Text.class);

    job.setMapperClass(PrimesMap.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.waitForCompletion(true);
}

public static final class PrimesMap extends Mapper<LongWritable, Text, NullWritable, Text> {

    final NullWritable nw = NullWritable.get();
    private Text str=new Text();

    public final void map(final LongWritable key, final Text value, final Context context)
            throws IOException, InterruptedException {

        final int number = Integer.parseInt(value.toString());
        BigInteger num=new BigInteger("1");
        num=num.probablePrime(number,new SecureRandom());            
        str.set(num.toString());
        context.write(nw, str);

    }
  }    
 }

0 个答案:

没有答案