我正在尝试生成大小为1764位(531位)的主要BigIntegers。当我在本地计算机上执行此操作时,需要很长时间。所以我尝试使用mapreduce生成BigIntegers并在单节点cloudera(CDH 4)上运行。但这需要花费大量时间来制作。我可以通过应用mapreduce并在多节点集群上实现它来减少时间吗?而我的第二个问题是这个程序可以改进以提高效率吗?
我的输入文件由90个包含“1764”的条目组成,这是随机BigInteger生成的位数。这是我的mapreduce代码
public final class Primes {
public final static void main(final String[] args) throws Exception {
final Configuration conf = new Configuration();
final Job job = new Job(conf, "Primes");
job.setJarByClass(Primes.class);
job.setOutputKeyClass(NullWritable.class);
job.setOutputValueClass(Text.class);
job.setMapperClass(PrimesMap.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
public static final class PrimesMap extends Mapper<LongWritable, Text, NullWritable, Text> {
final NullWritable nw = NullWritable.get();
private Text str=new Text();
public final void map(final LongWritable key, final Text value, final Context context)
throws IOException, InterruptedException {
final int number = Integer.parseInt(value.toString());
BigInteger num=new BigInteger("1");
num=num.probablePrime(number,new SecureRandom());
str.set(num.toString());
context.write(nw, str);
}
}
}