Question

我有一个任务，我需要使用hibernate将大量数据插入数据库。我目前正在测试插入500,000个实体，每个实体都有一个关系，因此总共有1,000,000个插入。基于此guide，我创建了以下实际可行的代码。插入并提交所有数据时没有错误。

import javax.annotation.Resource;
import javax.ejb.*;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import javax.transaction.*;
import javax.xml.stream.XMLStreamException;
import javax.xml.transform.TransformerException;
import java.io.File;
import java.io.IOException;
import java.text.ParseException;

@Stateless
@TransactionManagement(TransactionManagementType.BEAN)
public class StackoverflowExample {
    @PersistenceContext
    private EntityManager entityManager;

    @Resource
    private SessionContext sessionContext;

    @EJB
    private XmlProcessorFactory xmlProcessorFactory;

    @EJB
    private TaskManagerBean taskManagerBean;

    public void processFile(String[] args, Task task) throws HeuristicRollbackException, RollbackException, HeuristicMixedException, SystemException, IOException, TransformerException, ParseException, XMLStreamException, NotSupportedException {
        UserTransaction tx = null;

        XmlProcessor instance = xmlProcessorFactory.getInstance(new File("data.xml"));

        XmlElement nextElement = instance.getNextElement();
        int i = 0;
        int batchSize = 50;
        entityManager.setProperty("hibernate.jdbc.batch_size", batchSize);
        tx = sessionContext.getUserTransaction();
        tx.begin();
        while (nextElement != null) {
            Entry entry = new Entry(nextElement.getUserReference(), nextElement.getXml());

            entityManager.persist(entry);

            if (i % batchSize == 0) {
                entityManager.flush();
                entityManager.clear();
            }

            nextElement = instance.getNextElement();
            i++;
        }

        task.setStatus(status);
        task.setEndTime(now());
        // This gives the OutOfMemoryError
        entityManager.merge(task);

        tx.commit();
    }
}

这将在我调用taskManagerBean.update（）的行中失败，并出现以下错误：

2017-03-31 08:49:30,212 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 3) WFLYEJB0034:
EJB Invocation failed on component TaskManagerBean for method public void
TaskManagerBean.update(Task,TaskStatus):
javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException:
could not load an entity: [Task#3]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:159)
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:256)
    ...
    at TaskManagerBean$$$view18.update(Unknown Source)
    at StoreEntriesBean.processFile(StoreEntriesBean.java:117)
    ...
    at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not load an entity: [Task#3]
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
    ...
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:254)
    ... 104 more
Caused by: org.hibernate.exception.GenericJDBCException: could not load an entity: [Task#3]
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
    ...
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.merge(AbstractEntityManagerImpl.java:1161)
    ... 135 more
Caused by: java.sql.SQLException: Error
    at org.jboss.jca.adapters.jdbc.WrappedConnection.checkException(WrappedConnection.java:1972)
    ...
    at org.hibernate.loader.Loader.loadEntity(Loader.java:2204)
    ... 155 more
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:3664)
    ...
    at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:478)

Runtime.getRuntime().freeMemory()在错误发生之前返回71540896。

我使用-Xmx2048m启动Wildfly，快速浏览内存使用情况表明它只使用不到一半。

我尝试在每1,000次插入后提交插入的条目。这会触发TaskManagerBean，并在几次更新后以相同的方式失败。在一些情况下，我在完成庞大的批处理作业并且更新拥有实体后也看到了这个错误。

我尝试过只有5,000个条目的文件，整个过程运行正常。

这是MySQL驱动程序中的错误还是我在这里做错了什么？

Answer 1

尝试通过EJB运行批处理过程几乎无法工作，因为内存不足或事务超时。

这是“批量申请”的原因之一开发了Java平台“（又名JSR-352）规范。

WildFly 10为您提供了此实现。

您可以在Batch Applications in Java EE 7 - Undertanding JSR 352 Concepts: TOTD #192

了解更多相关信息

Answer 2

你关闭会话或实体经理吗？

我从未使用过EntityManager，但是当您使用Java处理数据库时，您应该始终＆＃34;明确地＆＃34;完成作业（交易）时关闭连接

Answer 3

对public class Reports_Visit_Statastics extends AppCompatActivity { WebView wb; String ReportsURL, title; @SuppressLint("SetJavaScriptEnabled") @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.visit_statastics_reports); wb = (WebView) findViewById(webView); wb.getSettings().setLoadsImagesAutomatically(true); wb.getSettings().setJavaScriptEnabled(true); wb.setScrollBarStyle(View.SCROLLBARS_OUTSIDE_OVERLAY); Bundle b = getIntent().getExtras(); ReportsURL = b.getString("URL"); title = b.getString("title"); initToolbar(); wb.setWebViewClient(new MyWebViewClient()); wb.loadUrl(ReportsURL); } private class MyWebViewClient extends WebViewClient { @Override public void onReceivedError(WebView view, int errorCode, String description, String failingUrl) { Log.d("WEB_VIEW_TEST", "error code:" + errorCode + " - " + description); } @Override public boolean shouldOverrideUrlLoading(WebView view, String url) { // handle different requests for different type of files // this example handles downloads requests for .apk and .mp3 files // everything else the webview can handle normally if (url.endsWith(".pdf")) { Uri source = Uri.parse(url); // Make a new request pointing to the .apk url DownloadManager.Request request = new DownloadManager.Request(source); // appears the same in Notification bar while downloading request.setDescription("Description for the DownloadManager Bar"); request.setTitle("Document"); request.allowScanningByMediaScanner(); request.setNotificationVisibility(DownloadManager.Request.VISIBILITY_VISIBLE_NOTIFY_COMPLETED); // save the file in the "Downloads" folder of SDCARD request.setDestinationInExternalPublicDir(Environment.DIRECTORY_DOWNLOADS, "Document.doc"); // get download service and enqueue file DownloadManager manager = (DownloadManager) getSystemService(Context.DOWNLOAD_SERVICE); manager.enqueue(request); } else if (url.endsWith(".doc")) { Uri source = Uri.parse(url); // Make a new request pointing to the .apk url DownloadManager.Request request = new DownloadManager.Request(source); // appears the same in Notification bar while downloading request.setDescription("Description for the DownloadManager Bar"); request.setTitle("Document"); request.allowScanningByMediaScanner(); request.setNotificationVisibility(DownloadManager.Request.VISIBILITY_VISIBLE_NOTIFY_COMPLETED); // save the file in the "Downloads" folder of SDCARD request.setDestinationInExternalPublicDir(Environment.DIRECTORY_DOWNLOADS, "Document.doc"); // get download service and enqueue file DownloadManager manager = (DownloadManager) getSystemService(Context.DOWNLOAD_SERVICE); manager.enqueue(request); } else if (url.endsWith(".xls")) { Uri source = Uri.parse(url); // Make a new request pointing to the .apk url DownloadManager.Request request = new DownloadManager.Request(source); // appears the same in Notification bar while downloading request.setDescription("Description for the DownloadManager Bar"); request.setTitle("Document"); request.allowScanningByMediaScanner(); request.setNotificationVisibility(DownloadManager.Request.VISIBILITY_VISIBLE_NOTIFY_COMPLETED); // save the file in the "Downloads" folder of SDCARD request.setDestinationInExternalPublicDir(Environment.DIRECTORY_DOWNLOADS, "Document.doc"); // get download service and enqueue file DownloadManager manager = (DownloadManager) getSystemService(Context.DOWNLOAD_SERVICE); manager.enqueue(request); } // if there is a link to anything else than .apk or .mp3 load the URL in the webview else view.loadUrl(url); return true; } } private void initToolbar() { final Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar); setSupportActionBar(toolbar); final ActionBar actionBar = getSupportActionBar(); try { if (actionBar != null) { actionBar.setTitle(title); actionBar.setHomeButtonEnabled(true); actionBar.setDisplayHomeAsUpEnabled(true); } } catch (Exception e) { Log.d("Doctor_master", e.toString()); } } @Override public void onBackPressed() { super.onBackPressed(); } @Override public boolean onOptionsItemSelected(MenuItem item) { int id = item.getItemId(); if (id == android.R.id.home) { onBackPressed(); return true; } return super.onOptionsItemSelected(item); } }的调用是导致错误的原因。我并不是100％熟悉Hibernate，但显然EntityManger.merge()会在这种情况下获取从merge到Task的整个关系，在这种情况下是一个包含500,000个条目的集合 - 即使这样关系是拉斯加载的。

我用Entry替换了合并，并在该实例上设置了状态，这解决了我的问题。

与此同时，我已经介绍了Java Batch Processing，我只能推荐。它避免了自己编写批处理作业。

批量插入导致失败

3 个答案: