这可能是一个非常特殊的情况但是在我脑子里敲了一会儿之后我想从Stackoverflow社区获得帮助。
我正在构建大型数据集的倒排索引（大型系统的一天数据）。反向索引的构建在Hadoop上作为map reduce作业执行。在scala的帮助下构建反向索引。倒排索引的结构如下：{key:"New", ProductID:[1,2,3,4,5,...]}这些被写入avro文件。
在此过程中，我遇到了Java堆大小问题。我认为原因是像我上面所示的“新”这样的术语包含大量的productId（s）。我有一个粗略的想法，可以在我的Scala代码中发生问题：
def toIndexedRecord(ids: List[Long], token: String): IndexRecord = {
val javaList = ids.map(l => l: java.lang.Long).asJava //need to convert from scala long to java long
new IndexRecord(token, javaList)
}
这就是我使用这种方法的方法（它在很多地方使用，但代码结构和登录使用相同）
val titles = textPipeDump.map(vf => (vf.itemId, normalizer.customNormalizer(vf.title + " " + vf.subTitle).trim))
.flatMap {
case (id, title) =>
val ss = title.split("\\s+")
ss.map(word => (word, List(id)))
}
.filter(f => f._2.nonEmpty)
.group
.sum
.map {
case (token, ids) =>
toIndexedRecord(ids, token)
}
textPipeDump正在烫伤MultipleTextLine字段对象
case class MultipleTextLineFiles(p : String*) extends FixedPathSource(p:_*) with TextLineScheme
我有一个案例类要拆分并从该文本行中抓取我想要的字段，那就是对象ss
这是我的堆栈跟踪：
Exception in thread "IPC Client (47) connection to /127.0.0.1:55977 from job_201306241658_232590" java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:226)
at org.apache.hadoop.ipc.Client$Connection.close(Client.java:903)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:800)
28079664 [main] ERROR cascading.flow.stream.TrapHandler - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [WritableSequenceFile(h...][com.twitter.scalding.GroupBuilder$$anonfun$1.apply(GroupBuilder.scala:189)] operator Every failed executing operation: MRMAggregator[decl:'value']
at cascading.flow.stream.AggregatorEveryStage.receive(AggregatorEveryStage.java:136)
at cascading.flow.stream.AggregatorEveryStage.receive(AggregatorEveryStage.java:39)
at cascading.flow.stream.OpenReducingDuct.receive(OpenReducingDuct.java:49)
at cascading.flow.stream.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.run(HadoopGroupGate.java:90)
at cascading.flow.hadoop.FlowReducer.reduce(FlowReducer.java:133)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.OutOfMemoryError: Java heap space
at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:168)
at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:45)
at scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
at scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:176)
at scala.collection.immutable.List.$colon$colon$colon(List.scala:127)
at scala.collection.immutable.List.$plus$plus(List.scala:193)
at com.twitter.algebird.ListMonoid.plus(Monoid.scala:86)
at com.twitter.algebird.ListMonoid.plus(Monoid.scala:84)
at com.twitter.scalding.KeyedList$$anonfun$sum$1.apply(TypedPipe.scala:264)
at com.twitter.scalding.MRMAggregator.aggregate(Operations.scala:279)
at cascading.flow.stream.AggregatorEveryStage.receive(AggregatorEveryStage.java:128)
... 12 more
当我为小数据集执行map reduce作业时，我没有收到错误。这意味着随着数据的增加，我为新旧等词语索引的items / product_id数量会变大，导致堆大小溢出。
所以，问题是如何避免java堆大小溢出并完成此任务。