Question

我尝试使用分布式缓存的新api运行hadoop程序。我坚持使用以下错误消息。

14/11/04 10:54:36 WARN fs.FileUtil: Command 'ln -s /tmp/hadoop-hduser/mapred/local/1415078671812/normal_small /home/yogi/Desktop/normal_small' failed 1 with: ln: failed to create symbolic link ‘/home/yogi/Desktop/normal_small’: Permission denied

14/11/04 10:54:36 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /tmp/hadoop-hduser/mapred/local/1415078671812/normal_small <- /home/yogi/Desktop/normal_small

java.io.FileNotFoundException: hdfs:/master:54310/usr/local/hadoop/input/normal_small (No such file or directory)

我的代码中从未提及/ home / yogi / Desktop / normal_small。无法理解它试图访问该文件的位置。

另外，我应该如何在驱动程序类中给出输入文件路径以解决文件未找到的异常？

以下是我的mapper和驱动程序类片段：

映射器：

BufferedReader in = null;
  FileReader fr = null;
  private List<String> list = new ArrayList<String>();


  @Override
  protected void setup(Context context)
          throws IOException, InterruptedException {
      Configuration conf = context.getConfiguration();
      URI[] cacheFiles = context.getCacheFiles();


      try {
          fr = new FileReader(cacheFiles[0].toString());
          in = new BufferedReader(fr);
          String str;
          while ((str = in.readLine()) != null) {
              list.add(str);
          }
      } catch (Exception e) {
          e.printStackTrace();
      } finally {
          in.close();
          fr.close();
      }

  }


public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {


    FileOutputStream fos = new FileOutputStream("output");
    ObjectOutputStream oos = new ObjectOutputStream(fos);   
    oos.writeObject(list); // write MenuArray to ObjectOutputStream


    BufferedReader br=new BufferedReader(new FileReader("output"));


       String line=br.readLine();
        .........
}

驱动程序：

Job job = Job.getInstance(getConf());
job.setJobName("wordcount");
job.setJarByClass(driver.class);        
job.addCacheFile(new Path("hdfs://master:54310/usr/local/hadoop/input/normal_small").toUri());

Answer 1

将文件添加到分布式缓存时，它将创建一个临时目录。因此，将该目录的所有权更改为当前用户。

使用分布式缓存的新api的问题

1 个答案: