我正在关注使用分布式缓存的tutorial here。我对代码稍作修改,使其与Hadoop2.2兼容。
我发现当调用loadStopWords
方法时,抛出IO异常:
我确认stop_words.txt已复制到HDFS
。
我遗漏了mapper和reducer代码,这里很简单。
这是我的代码:
public static final String LOCAL_STOPWORD_LIST =
"/Users/sridhar/Documents/hadoop/stop_words.txt";
public static final String HDFS_STOPWORD_LIST = "/data/stop_words.txt";
//copies local file to HDFS and adds to Job's cache file
static void cacheStopWordList(Configuration conf, Job job) throws IOException, URISyntaxException {
FileSystem fs = FileSystem.get(conf);
URI hdfsPath = new URI(HDFS_STOPWORD_LIST);
System.out.println("coping files to HDFS");
// upload the file to hdfs. Overwrite any existing copy.
fs.copyFromLocalFile(false, true, new Path(LOCAL_STOPWORD_LIST),
new Path(hdfsPath));
System.out.println("done copying HDFS");
job.addCacheFile(hdfsPath);
}
protected void setup(Context context) {
try {
String stopwordCacheName = new Path(HDFS_STOPWORD_LIST).toString();
URI[] cacheFiles = context.getCacheFiles();
System.out.println(Arrays.toString(cacheFiles));
if (null != cacheFiles && cacheFiles.length > 0) {
for (URI cacheURI : cacheFiles) {
System.out.println(cacheURI.toString());
System.out.println(stopwordCacheName);
System.out.println("-----------------");
if (cacheURI.toString().equals(stopwordCacheName)) {
System.out.println("****************************************");
loadStopWords(new Path(cacheURI)); // IT BREAKS HERE
System.out.println(stopWords);
break;
}
}
}
} catch (IOException ioe) {
System.err.println("IOException reading from distributed cache");
System.err.println(ioe.toString());
}
}
void loadStopWords(Path cachePath) throws IOException {
// note use of regular java.io methods here - this is a local file now
BufferedReader wordReader = new BufferedReader(
new FileReader(cachePath.toString()));
try {
String line;
this.stopWords = new HashSet<String>();
while ((line = wordReader.readLine()) != null) {
this.stopWords.add(line.toLowerCase());
}
} finally {
wordReader.close();
}
}
public static void main(String[] args) throws IllegalArgumentException, IOException, InterruptedException, ClassNotFoundException, URISyntaxException {
Job job = new Job();
job.setJarByClass(LineIndexer.class);
job.setJobName("LineIndexer");
Configuration conf = job.getConfiguration();
cacheStopWordList(conf,job);
}
答案 0 :(得分:0)
我认为您应该尝试Path[] localPaths = context.getLocalCacheFiles();
而不是context.getCacheFiles();
让我知道它是否有效
答案 1 :(得分:-1)
在您提供的链接中提到使用DistributedCache.addCacheFile()。这是段落 要使用分布式缓存来传播文件,请在设置作业时创建DistributedCache类的实例。使用DistributedCache.addCacheFile()方法添加应发送到系统上所有节点的文件名。
而不是写
job.addCacheFile(hdfsPath);
尝试写作
DistributedCache.addCacheFile(hdfsPath, job.getConfiguration());