我从经典的Spring 4.3.2 + Hibernate(HQL)迁移到Spring Boot 1.4.1 + Hibernate(JPA)+ HikariCP连接池,并且在批量插入时遇到问题。我需要保存1000到527 000个元素的大型列表。
当我只保存一个实体的批处理,没有任何依赖关系(例如:Category
类的5000个元素)时,一切正常。
但是,如果我尝试同时保存子实体和父实体(Website
并为其设置Category
),则会出现例外情况:org.hibernate.PersistentObjectException: detached entity passed to persist: com.example.entities.Category
但当列表大小小于 hibernate.jdbc.batch_size
时,一切正常!两个实体都正确保存,父母和孩子。这看起来很奇怪!
我已经google了足够的搜索,在StackOverflow搜索,但无法找到特定问题的解决方案。我已经尝试将连接池替换为Apache DBCP,甚至根本没有包含一个),更改了Hibernate性能设置,将PostgreSQL数据库替换为H2 - 但没有结果。
哪里可能是错误?
是否有可能在JPA中做我想要的事情?或者我应该单独保存实体(首先是父母,然后是孩子)?另一种方法是从我的程序中排除flush()
和clear()
,但它对性能不利!
希望得到帮助。这是详细信息。
这就是我试图保存实体的方式:
@Repository("experimentDao")
@Transactional
public class Dao {
@PersistenceContext
private EntityManager entityManager;
@Value("${spring.jpa.properties.hibernate.jdbc.batch_size}")
private int batchSize;
public <T extends Entity> void bulkSave(Collection<T> entities) {
int i = 0;
for (Entity entity : entities) {
persistOrMerge(entity);
i++;
if (i % batchSize == 0) {
// Flush a batch of inserts and release memory.
entityManager.flush();
entityManager.clear();
}
}
}
private <T extends Entity> void persistOrMerge(T entity) {
if (entity.getId() == null) {
entityManager.persist(entity);
} else {
entityManager.merge(entity);
}
}
}
(示例见于:http://frightanic.com/software-development/jpa-batch-inserts/)。 而且我正在跟踪堆栈跟踪:
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:803) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:771) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
at com.example.DemoApplication.main(DemoApplication.java:24) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_101]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_101]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) [idea_rt.jar:na]
Caused by: org.springframework.dao.InvalidDataAccessApiUsageException: detached entity passed to persist: com.example.entities.Category; nested exception is org.hibernate.PersistentObjectException: detached entity passed to persist: com.example.entities.Category
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:299) ~[spring-orm-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:244) ~[spring-orm-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:491) ~[spring-orm-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:59) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:213) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:147) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at com.example.Dao$$EnhancerBySpringCGLIB$$79470367.bulkSave(<generated>) ~[classes/:na]
at com.example.DemoApplication.insertWebsites(DemoApplication.java:44) [classes/:na]
at com.example.DemoApplication.run(DemoApplication.java:30) [classes/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
... 11 common frames omitted
Caused by: org.hibernate.PersistentObjectException: detached entity passed to persist: com.example.entities.Category
at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:124) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.internal.SessionImpl.firePersist(SessionImpl.java:765) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.internal.SessionImpl.persist(SessionImpl.java:758) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.jpa.event.internal.core.JpaPersistEventListener$1.cascade(JpaPersistEventListener.java:80) ~[hibernate-entitymanager-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.engine.internal.Cascade.cascadeToOne(Cascade.java:398) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.engine.internal.Cascade.cascadeAssociation(Cascade.java:323) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.engine.internal.Cascade.cascadeProperty(Cascade.java:162) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.engine.internal.Cascade.cascade(Cascade.java:111) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.event.internal.AbstractSaveEventListener.cascadeBeforeSave(AbstractSaveEventListener.java:425) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.event.internal.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:249) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.event.internal.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:178) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:121) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.jpa.event.internal.core.JpaPersistEventListener.saveWithGeneratedId(JpaPersistEventListener.java:67) ~[hibernate-entitymanager-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.event.internal.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:189) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:132) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:58) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.internal.SessionImpl.firePersist(SessionImpl.java:775) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.internal.SessionImpl.persist(SessionImpl.java:748) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.internal.SessionImpl.persist(SessionImpl.java:753) ~[hibernate-core-5.0.11.Final.jar:5.0.11.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.persist(AbstractEntityManagerImpl.java:1146) ~[hibernate-entitymanager-5.0.11.Final.jar:5.0.11.Final]
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:298) ~[spring-orm-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at com.sun.proxy.$Proxy62.persist(Unknown Source) ~[na:na]
at com.example.Dao.persistOrMerge(Dao.java:36) ~[classes/:na]
at com.example.Dao.bulkSave(Dao.java:24) ~[classes/:na]
at com.example.Dao$$FastClassBySpringCGLIB$$2ada4001.invoke(<generated>) ~[classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE]
... 21 common frames omitted
Website
和Category
个实体具有一对一关系,Category
是父级。而且我也有抽象的超类,它应该坚持下去:
public abstract class Entity {
public abstract Long getId();
}
@javax.persistence.Entity
@Table(name = "websites")
public class Website extends Entity implements Serializable {
@Id
@GeneratedValue(strategy = SEQUENCE)
@Column(name = "website_id")
private Long id;
@Column(name = "url", unique = true)
private String url;
@OneToOne(cascade = CascadeType.ALL)
@JoinColumn(name="category_id")
private Category category;
// getters and setters, toString
}
@javax.persistence.Entity
@Table(name = "categories")
public class Category extends Entity implements Serializable {
@Id
@GeneratedValue(strategy = SEQUENCE)
@Column(name = "category_id")
private Long id;
@Column(name = "category_name", unique = true)
private String categoryName;
// getters, setters, toString
}
现在我要保存属于同一Website
的5000 Category
个。这是我如何批量插入:
@SpringBootApplication
public class DemoApplication implements CommandLineRunner {
private Dao dao;
@Autowired
public void setDao(Dao dao) {
this.dao = dao;
}
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
@Override
public void run(String... strings) throws Exception {
insertWebsites();
}
private void insertWebsites() {
List<Website> entities = new ArrayList<>(5000);
Category category = new Category();
category.setCategoryName("my category");
for (int i = 0; i < 5000; i++) {
Website website = new Website();
website.setUrl("http://example.com" + i);
website.setCategory(category);
entities.add(website);
}
dao.bulkSave(entities);
}
}
设置和参数:
(application.properties)
spring.datasource.driver-class-name = org.postgresql.Driver
spring.datasource.url = jdbc:postgresql://localhost:5432/spring_web
spring.datasource.username = postgres
spring.datasource.password = postgresql
spring.datasource.type = com.zaxxer.hikari.HikariDataSource
# Hibernate
spring.jpa.hibernate.ddl-auto = update
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.PostgreSQL94Dialect
spring.jpa.properties.hibernate.max_fetch_depth = 3
spring.jpa.properties.hibernate.jdbc.fetch_size = 30
spring.jpa.properties.hibernate.jdbc.batch_size = 30
spring.jpa.properties.hibernate.order_inserts = true
spring.jpa.properties.hibernate.order_updates = true
spring.jpa.properties.hibernate.cache.use_second_level_cache = false
(pom.xml中)
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>demo</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.1.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<!--<scope>runtime</scope>-->
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<version>2.5.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
很抱歉,我试图提供所有细节。期待得到任何帮助!
答案 0 :(得分:0)
执行后,问题就会明白:
batch_size
网站已创建并保留;他们都指向上面创建的单一类别
batch_size
个网站和类别都已保留,因此它们位于持久性上下文中。但是一旦你调用entityManager.clear()
,它们就会被抛弃持久化上下文(释放内存)。 因此单个类别变得分离(即从持久化上下文中删除)并且您获得异常。您提到的站点中的示例不会通过关系持续存在,这就是它工作的原因。鉴于这种关系,我不相信bulkSave()
能以这种通用的方式工作。如果您想批量插入网站,请执行以下操作:
@Value("${spring.jpa.properties.hibernate.jdbc.batch_size}")
private int batchSize;
@PersistenceContext
private EntityManager em; // you will need the EntityManager
private void insertWebsites() {
Category category = new Category();
category.setCategoryName("my category");
// persist it separately to keep the id for later
em.persist(category);
Long categoryId = category.getId();
for (int i = 1; i <= 5000; i++) {
Website website = new Website();
website.setUrl("http://example.com" + i);
website.setCategory(category);
if (i % batchSize == 0) {
em.flush();
em.clear();
// get an attached reference to the category (you know
// that it exists) without querying the database again
category = em.getReference(Category.class, categoryId);
}
}
}