Question

我正在使用多个Jobs，我需要使用全局数组值。我在函数设置（Mapper）中使用de数组，我需要在函数清理（Reducer）中更改它。我在创建作业之前读了一个带有这个值的序列文件，然后我使用了conf.setInt（）。在Cleanup（Reducer）中，我用新数组编写了一个sequencefile。我正面临这个问题：

13/11/19 10:58:23 INFO mapred.JobClient: Task Id : attempt_201311190929_0005_m_000015_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 134.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201311190929_0005_m_000015_0: centers.size() - 10
attempt_201311190929_0005_m_000015_0: #
attempt_201311190929_0005_m_000015_0: # A fatal error has been detected by the Java Runtime Environment:
attempt_201311190929_0005_m_000015_0: #
attempt_201311190929_0005_m_000015_0: #  SIGSEGV (0xb) at pc=0x00007fed37941b78, pid=6494, tid=140656104789760
attempt_201311190929_0005_m_000015_0: #
attempt_201311190929_0005_m_000015_0: # JRE version: 7.0_25-b30
attempt_201311190929_0005_m_000015_0: # Java VM: OpenJDK 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)
attempt_201311190929_0005_m_000015_0: # Problematic frame:
attempt_201311190929_0005_m_000015_0: # V  [libjvm.so+0x6b2b78]  methodOopDesc::get_c2i_entry()+0x8
attempt_201311190929_0005_m_000015_0: #
attempt_201311190929_0005_m_000015_0: # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
attempt_201311190929_0005_m_000015_0: #
attempt_201311190929_0005_m_000015_0: # An error report file with more information is saved as:
attempt_201311190929_0005_m_000015_0: # /tmp/hadoop-vega/mapred/local/taskTracker/akuma/jobcache/job_201311190929_0005/attempt_201311190929_0005_m_000015_0/work/hs_err_pid6494.log
attempt_201311190929_0005_m_000015_0: #
attempt_201311190929_0005_m_000015_0: # If you would like to submit a bug report, please include
attempt_201311190929_0005_m_000015_0: # instructions on how to reproduce the bug and visit:
attempt_201311190929_0005_m_000015_0: #   https://bugs.launchpad.net/ubuntu/+source/openjdk-7/

我很绝望，不知道该怎么办= /

    public class Teste7 extends Configured implements Tool {
        public static int[] vetor;   
        private static int choosenIteration = 10;

        @Override    
        public int run(String[] strings) throws Exception {

        while(interations < choosenIteration) {

                    interations ++;
                    fs = FileSystem.get(conf);            
                    conf = new Configuration();            
                    conf.setInt("n", n);    

                    //This File was written on Cleanup (Reduce), I need to read it to pass to map 
                    try (SequenceFile.Reader reader = new SequenceFile.Reader(fs, dir, conf)) {            
                        IntWritable key = (IntWritable) reader.getKeyClass().newInstance();
                        IntWritable value = (IntWritable) reader.getValueClass().newInstance();                               
                        int a = 0;
                        while (reader.next(key, value)){        
                            System.out.println("vetor " + vetor[a]);
                            vetor[a] = value.get();                    
                            conf.setInt("vet["+a+"]", vetor[a]);
                            a++;
                        }              
                        reader.close();
                    } catch (InstantiationException | IllegalAccessException ex) {
                        Logger.getLogger(Teste7.class.getName()).log(Level.SEVERE, null, ex);
                    }

                    job = new Job(conf);            
                    job.setJarByClass(Teste7.class);
                    job.setJobName("KMeans Locão!");

                    if(fs.exists(saida)){       
                        fs.delete(saida, true);             
                    }

                    job.setMapperClass(MapTeste7.class);
                    job.setReducerClass(RedTeste7.class);
                    job.setJarByClass(Teste7.class);

                    job.setInputFormatClass(SequenceFileInputFormat.class);       

                    FileInputFormat.addInputPath(job, entrada);
                    FileOutputFormat.setOutputPath(job, saida);        

                    job.setOutputKeyClass(IntWritable.class);
                    job.setOutputValueClass(VectorWritable.class);

                    job.waitForCompletion(true);         
                }
}

    public class RedTeste7 extends Reducer<IntWritable, VectorWritable, IntWritable, VectorWritable> {
     @Override
        protected void cleanup(Context context) throws IOException, InterruptedException {     

            Configuration conf = context.getConfiguration();izado
            fs = FileSystem.get(conf); 
            try (SequenceFile.Writer write = SequenceFile.createWriter(fs, conf, dir, IntWritable.class, IntWritable.class)){
                IntWritable oi;
                IntWritable qtd;
                for(int i = 0; i < _partitonNumber; i++){
                    oi = new IntWritable(i);
                    qtd = new IntWritable(vetor[i]);
                    write.append(oi,qtd);  
                }
                write.close();
            }
        }
}

Answer 1

这表示JNI代码中的错误或者（更有可能是给定segfault的框架）运行任务的节点上的内存不良。

倍数工作和全球价值

1 个答案: