使用postgresql进行批处理时,hibernate内存不足错误

时间:2018-05-29 08:47:05

标签: postgresql performance hibernate jpa out-of-memory

您好我是Hibernate的新手,我正在尝试这样做:从数据库表中选择一些数据,然后将它们插入到另一个表中:

EntityManager em = LocalEntityManagerFactory.createEntityManager();
List<Object> results = em.createNativeQuery("select id from bar")
for (Object o : results) {
    Foo foo = new Foo();
    foo.setBarId(o.toString());
    em.persist(foo);
}

我知道上面的代码将从数据库获取所有id,将它们存储在内存中并将它们放入List.This可能会得到OutOfMemoryError,如果有太多的ID。所以我已经检查了Hibernate官方用户指南进行批处理,但是我从示例中得到的内容不起作用:

EntityManager em= LocalEntityManagerFactory.createEntityManager();
txn = em.getTransaction();
txn.begin();
int batchSize = 25;
Session session = em.unwrap( Session.class );
ScrollableResults scrollableResults = session
    .createNativeQuery("select id from bar")
    .setCacheMode( CacheMode.IGNORE )
    .scroll( ScrollMode.FORWARD_ONLY );

int count = 0;
while ( scrollableResults.next() ) {
    String id= (String) scrollableResults.get( 0 );
    Foo foo = new Foo();
    foo.setBarId(id);
    em.persist(foo);
    if ( ++count % batchSize == 0 ) {
        //flush a batch of updates and release memory:
        em.flush();
        em.clear();
    }
}

txn.commit();

我尝试过hibernate stream()方法,它也不起作用:

try (Stream<Object> objs = session
            .createNativeQuery("select id from bar")
            .stream()){
        objs.forEach(m -> {
            Foo foo = new Foo();
            foo.setBarId(m.toString());
            em.persist(foo);
        });
    }

之后我找到了这个question,并禁用了二级缓存:

<property name="hibernate.cache.use_query_cache" value="false"/> 

它仍然返回OutOfMemoryError。我做错了什么?这是ErrorTrace:

 org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [Jersey Web Application] in context with path [] threw exception [org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space] with root cause
 java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3181)
    at java.util.ArrayList.grow(ArrayList.java:261)
    at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
    at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
    at java.util.ArrayList.add(ArrayList.java:458)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
    at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118)
    at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:60)
    at org.hibernate.loader.Loader.getResultSet(Loader.java:2168)
    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1931)
    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1893)
    at org.hibernate.loader.Loader.scroll(Loader.java:2768)
    at org.hibernate.loader.custom.CustomLoader.scroll(CustomLoader.java:380)
    at org.hibernate.internal.SessionImpl.scrollCustomQuery(SessionImpl.java:2175)
    at org.hibernate.internal.AbstractSharedSessionContract.scroll(AbstractSharedSessionContract.java:1021)
    at org.hibernate.query.internal.NativeQueryImpl.doScroll(NativeQueryImpl.java:199)
    at org.hibernate.query.internal.AbstractProducedQuery.scroll(AbstractProducedQuery.java:1371)
    at org.hibernate.query.internal.AbstractProducedQuery.stream(AbstractProducedQuery.java:1395)

0 个答案:

没有答案