Question

我正在用Hadoop 1.1.1（Ubuntu）编写Java应用程序，它比较字符串以找到最长的公共子串。我已经为小型数据集成功运行了map和reduce阶段。每当我增加输入的大小时，我的reduce输出永远不会出现在我的目标输出目录中。它根本没有抱怨，这使得这一切都更加怪异。我在Eclipse中运行所有东西，我有1个mapper和1个reducer。

我的reducer在字符串集合中找到最长的公共子字符串，然后将子字符串作为键发出，并将包含它的字符串的索引作为值。我有一个简短的例子。

输入数据

0: ALPHAA

1: ALPHAB

2: ALZHA

输出已发布

Key: ALPHA  Value: 0

Key: ALPHA  Value: 1

Key: AL  Value: 0

Key: AL  Value: 1

Key: AL  Value: 2

前两个输入字符串共享“ALPHA”作为公共子字符串，而所有三个共享“AL”。我最终索引子字符串并在进程完成时将它们写入数据库。

另外一个观察，我可以看到中间文件是在我的输出目录中创建的，只是减少的数据永远不会放入输出文件中。

我已粘贴下面的Hadoop输出日志，它声称它有一些来自reducer的输出记录，只是它们似乎消失了。任何建议表示赞赏。

Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Use GenericOptionsParser for parsing the arguments. Applications should implement Tool     for the same.
No job jar file set.  User classes may not be found. See JobConf(Class) or     JobConf#setJar(String).
Total input paths to process : 1
Running job: job_local_0001
setsid exited with exit code 0
 Using ResourceCalculatorPlugin :     org.apache.hadoop.util.LinuxResourceCalculatorPlugin@411fd5a3
Snappy native library not loaded
io.sort.mb = 100
data buffer = 79691776/99614720
record buffer = 262144/327680
 map 0% reduce 0%
Spilling map output: record full = true
bufstart = 0; bufend = 22852573; bufvoid = 99614720
kvstart = 0; kvend = 262144; length = 327680
Finished spill 0
Starting flush of map output
Finished spill 1
Merging 2 sorted segments
Down to the last merge-pass, with 2 segments left of total size: 28981648 bytes

Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting

Task attempt_local_0001_m_000000_0 done.
 Using ResourceCalculatorPlugin :     org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3aff2f16

Merging 1 sorted segments
Down to the last merge-pass, with 1 segments left of total size: 28981646 bytes

 map 100% reduce 0%
reduce > reduce
 map 100% reduce 66%
reduce > reduce
 map 100% reduce 67%
reduce > reduce
reduce > reduce
 map 100% reduce 68%
reduce > reduce
reduce > reduce
reduce > reduce
 map 100% reduce 69%
reduce > reduce
reduce > reduce
 map 100% reduce 70%
reduce > reduce
job_local_0001
Job complete: job_local_0001
Counters: 22
  File Output Format Counters 
    Bytes Written=14754916
  FileSystemCounters
    FILE_BYTES_READ=61475617
    HDFS_BYTES_READ=97361881
    FILE_BYTES_WRITTEN=116018418
    HDFS_BYTES_WRITTEN=116746326
  File Input Format Counters 
    Bytes Read=46366176
  Map-Reduce Framework
    Reduce input groups=27774
    Map output materialized bytes=28981650
    Combine output records=0
    Map input records=4629524
    Reduce shuffle bytes=0
    Physical memory (bytes) snapshot=0
    Reduce output records=832559
    Spilled Records=651304
    Map output bytes=28289481
    CPU time spent (ms)=0
    Total committed heap usage (bytes)=2578972672
    Virtual memory (bytes) snapshot=0
    Combine input records=0
    Map output records=325652
    SPLIT_RAW_BYTES=136
    Reduce input records=27774
reduce > reduce
reduce > reduce

Answer 1

我将reduce（）和map（）逻辑放在try-catch块中，catch块递增一个计数器，其组为“Exception”，其名称为异常消息。这给了我一个快速的方法（通过查看计数器列表）来查看抛出的异常（如果有的话）。

Hadoop减少了从未为大数据创建的输出文件

1 个答案: