Question

Casbah（或mongodb的java驱动程序）似乎在处理大量返回的对象时遇到问题。例如，以下代码段将产生IllegalArgumentException，并且不会返回单个结果（下面的完整堆栈跟踪）。但是，如果我将“限制（...）”减少到1994年，一切似乎都能正常工作。

for (link <- links; query = link $exists true) {
    val group = new HashMap[String, Set[(String, String)]] with MultiMap[String, (String, String)]
    log.find(query, fieldsToGet.result).limit(1996) foreach {

      x => {
        group.addBinding(x.get(link).toString, (x.get("_id").toString(), x.get("eventType").toString))
      }

    }
    allGroups += link -> group
  }

Apr 26, 2011 8:23:40 PM com.mongodb.DBTCPConnector$MyPort error
SEVERE: MyPort.error called
java.lang.IllegalArgumentException: response too long: 1278031173
    at com.mongodb.Response.<init>(Response.java:40)
    at com.mongodb.DBPort.go(DBPort.java:101)
    at com.mongodb.DBPort.go(DBPort.java:66)
    at com.mongodb.DBPort.call(DBPort.java:56)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266)
    at com.mongodb.DBCursor._check(DBCursor.java:309)
    at com.mongodb.DBCursor._hasNext(DBCursor.java:431)
    at com.mongodb.DBCursor.hasNext(DBCursor.java:456)
    at com.mongodb.casbah.MongoCursorBase$class.hasNext(MongoCursor.scala:72)
    at com.mongodb.casbah.MongoCursor.hasNext(MongoCursor.scala:517)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at com.mongodb.casbah.MongoCursor.foreach(MongoCursor.scala:517)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:23)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:20)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at Sequencer$.<init>(Sequencer.scala:20)
    at Sequencer$.<clinit>(Sequencer.scala)
    at Sequencer.main(Sequencer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Exception in thread "main" java.lang.ExceptionInInitializerError
    at Sequencer.main(Sequencer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Caused by: java.lang.IllegalArgumentException: response too long: 1278031173
    at com.mongodb.Response.<init>(Response.java:40)
    at com.mongodb.DBPort.go(DBPort.java:101)
    at com.mongodb.DBPort.go(DBPort.java:66)
    at com.mongodb.DBPort.call(DBPort.java:56)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266)
    at com.mongodb.DBCursor._check(DBCursor.java:309)
    at com.mongodb.DBCursor._hasNext(DBCursor.java:431)
    at com.mongodb.DBCursor.hasNext(DBCursor.java:456)
    at com.mongodb.casbah.MongoCursorBase$class.hasNext(MongoCursor.scala:72)
    at com.mongodb.casbah.MongoCursor.hasNext(MongoCursor.scala:517)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at com.mongodb.casbah.MongoCursor.foreach(MongoCursor.scala:517)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:23)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:20)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at Sequencer$.<init>(Sequencer.scala:20)
    at Sequencer$.<clinit>(Sequencer.scala)
    ... 6 more

似乎异常是由java驱动程序中的“Response.java”中的以下检查产生的。

ByteArrayInputStream bin = new ByteArrayInputStream( b );
_len = Bits.readInt( bin );
if ( _len > ( 32 * 1024 * 1024 ) )
 throw new IllegalArgumentException( "response too long: " + _len );

是否可能是由返回的特定对象引起的？或者这可能是关于casbah？

谢谢，德里克

Answer 1

看起来Java驱动程序正在检查当前响应块是否大于32兆字节然后抛出异常。

如果在游标上设置batchSize（FEWER_NUMBER_OF_DOCS），这将减少数据库中的锁定时间并返回少于32 MB的数据。

我会使用batchSize来查看适合您应用程序的最佳选择。

http://api.mongodb.org/scala/casbah/2.1.2/scaladoc/

Java驱动程序中可能会增加max。

关于你的回答的一个奇怪的部分是，它说它返回了大约1.19 GB的数据。

如果您的回复没有那么多数据，则可能表明收集已损坏。

Casbah存在大量返回对象的问题

1 个答案: