我正在尝试使用一个postgres数据库将我的应用程序与k8s上的一些计划任务进行群集。想象以下设置:2x App,每个App 1 DB,例如1个cronjob。更新一些缓存。
由于我不想使用Zookeeper或任何其他实现,因此我尝试将Spring Boot领导选举与基于lock的表(INT_LOCK)结合使用,如spring文档中所述。
不幸的是,应用程序每隔几毫秒就会查询一次数据库锁定表,以查看谁是领导者。到目前为止,还不错,但是未持有锁的应用正在尝试插入新条目,因此数据库抛出了违反主键约束的情况。由于我的postgres db的错误日志条目,这种情况一直持续到我的pvc已满...
此过程似乎在Internet上已被不良记录。.我很高兴听到您可能有任何建议;) 如果需要提供更多信息,请告诉我。
谢谢!
我的领导者配置类:
@Configuration
public class LeadershipConfiguration {
private static final int LOCK_TIME_TO_LIVE = 120000;
private static final int LOCK_HEARTBEAT_TIME = 100000;
private static final int LOCK_ACQUIRE_WAIT_TIME = 110000;
@Value("${HOSTNAME:noname}")
private String podName;
@Value("${REGION:noregion}")
private String region;
@Bean
public DefaultLockRepository lockRepository(final DataSource dataSource) {
String clientId;
final DefaultLockRepository defaultLockRepository = new DefaultLockRepository(dataSource, this.podName);
defaultLockRepository.setRegion(this.region);
defaultLockRepository.setTimeToLive(LOCK_TIME_TO_LIVE);
return defaultLockRepository;
}
@Bean
public JdbcLockRegistry lockRegistry(final LockRepository lockRepository) {
return new JdbcLockRegistry(lockRepository);
}
@Bean
public LockRegistryLeaderInitiator leaderInitiator(final LockRegistry lockRegistry) {
final LockRegistryLeaderInitiator lockRegistryLeaderInitiator = new LockRegistryLeaderInitiator(lockRegistry, new DefaultCandidate());
lockRegistryLeaderInitiator.setHeartBeatMillis(LOCK_HEARTBEAT_TIME);
lockRegistryLeaderInitiator.setBusyWaitMillis(LOCK_ACQUIRE_WAIT_TIME);
return lockRegistryLeaderInitiator;
}
}
我更新了LeadershipConfiguration,因为根据此document和此document,这应该增加查询之间的时间,但是看来这也不起作用
这也是Pod中的一些日志语句:
DB日志提取,这3个日志stmt经常出现:
2019-05-27 13:52:04 UTC
ERROR: duplicate key value violates unique constraint "pk_int_lock_lock_key"
2019-05-27 13:52:04 UTC
DETAIL: Key (lock_key, region)=(c444858e-0aae-3727-9a73-d2eae62321ad, DEFAULT) already exists.
2019-05-27 13:52:04 UTC
STATEMENT: INSERT INTO INT_LOCK (REGION, LOCK_KEY, CLIENT_ID, CREATED_DATE) VALUES ($1, $2, $3, $4)
etc... re-occuring statements
从不是领导者的Pod中提取数据库日志:
13:49:59.441 DEBUG o.s.jdbc.datasource.DataSourceUtils - Resetting isolation level of JDBC Connection [HikariProxyConnection@512836674 wrapping org.postgresql.jdbc.PgConnection@61ab89b0] to 2
13:49:59.444 DEBUG o.s.orm.jpa.JpaTransactionManager - Closing JPA EntityManager [SessionImpl(1800314718<open>)] after transaction
13:49:59.546 DEBUG o.s.orm.jpa.JpaTransactionManager - Creating new transaction with name [org.springframework.integration.jdbc.lock.DefaultLockRepository.acquire]: PROPAGATION_REQUIRED,ISOLATION_SERIALIZABLE,timeout_1
13:49:59.547 DEBUG o.s.orm.jpa.JpaTransactionManager - Opened new EntityManager [SessionImpl(1465858215<open>)] for JPA transaction
13:49:59.547 DEBUG o.h.e.t.internal.TransactionImpl - On TransactionImpl creation, JpaCompliance#isJpaTransactionComplianceEnabled == false
13:49:59.549 DEBUG o.s.jdbc.datasource.DataSourceUtils - Changing isolation level of JDBC Connection [HikariProxyConnection@1224014205 wrapping org.postgresql.jdbc.PgConnection@61ab89b0] to 8
13:49:59.553 DEBUG o.h.e.t.internal.TransactionImpl - begin
13:49:59.553 DEBUG o.s.orm.jpa.JpaTransactionManager - Exposing JPA transaction as JDBC [org.springframework.orm.jpa.vendor.HibernateJpaDialect$HibernateConnectionHandle@5cd30775]
13:49:59.553 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:49:59.553 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [DELETE FROM INT_LOCK WHERE REGION=? AND LOCK_KEY=? AND CREATED_DATE<?]
13:49:59.555 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:49:59.555 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [UPDATE INT_LOCK SET CREATED_DATE=? WHERE REGION=? AND LOCK_KEY=? AND CLIENT_ID=?]
13:49:59.556 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:49:59.557 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [INSERT INTO INT_LOCK (REGION, LOCK_KEY, CLIENT_ID, CREATED_DATE) VALUES (?, ?, ?, ?)]
13:49:59.559 DEBUG o.s.j.s.SQLErrorCodeSQLExceptionTranslator - Translating SQLException with SQL state '23505', error code '0', message [ERROR: duplicate key value violates unique constraint "pk_int_lock_lock_key"
Detail: Key (lock_key, region)=(c444858e-0aae-3727-9a73-d2eae62321ad, DEFAULT) already exists.]; SQL was [INSERT INTO INT_LOCK (REGION, LOCK_KEY, CLIENT_ID, CREATED_DATE) VALUES (?, ?, ?, ?)] for task [PreparedStatementCallback]
13:49:59.559 DEBUG o.s.orm.jpa.JpaTransactionManager - Initiating transaction commit
13:49:59.559 DEBUG o.s.orm.jpa.JpaTransactionManager - Committing JPA transaction on EntityManager [SessionImpl(1465858215<open>)]
13:49:59.561 DEBUG o.h.e.t.internal.TransactionImpl - committing
13:49:59.561 DEBUG o.s.jdbc.datasource.DataSourceUtils - Resetting isolation level of JDBC Connection [HikariProxyConnection@1224014205 wrapping org.postgresql.jdbc.PgConnection@61ab89b0] to 2
....
从作为领导者的Pod中提取数据库日志:
13:52:08.861 DEBUG o.s.i.s.l.LockRegistryLeaderInitiator - Acquiring the lock for LockContext{role=leader, id=ce77af6e-4bb0-459e-a83d-f33ef9192fd3, isLeader=true}
13:52:08.862 DEBUG o.s.orm.jpa.JpaTransactionManager - Creating new transaction with name [org.springframework.integration.jdbc.lock.DefaultLockRepository.acquire]: PROPAGATION_REQUIRED,ISOLATION_SERIALIZABLE,timeout_1
13:52:08.864 DEBUG o.s.orm.jpa.JpaTransactionManager - Opened new EntityManager [SessionImpl(1712215220<open>)] for JPA transaction
13:52:08.865 DEBUG o.h.e.t.internal.TransactionImpl - On TransactionImpl creation, JpaCompliance#isJpaTransactionComplianceEnabled == false
13:52:08.865 DEBUG o.s.jdbc.datasource.DataSourceUtils - Changing isolation level of JDBC Connection [HikariProxyConnection@1507945608 wrapping org.postgresql.jdbc.PgConnection@7c72ecc] to 8
13:52:08.869 DEBUG o.h.e.t.internal.TransactionImpl - begin
13:52:08.869 DEBUG o.s.orm.jpa.JpaTransactionManager - Exposing JPA transaction as JDBC [org.springframework.orm.jpa.vendor.HibernateJpaDialect$HibernateConnectionHandle@7f1d0eae]
13:52:08.870 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:52:08.870 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [DELETE FROM INT_LOCK WHERE REGION=? AND LOCK_KEY=? AND CREATED_DATE<?]
13:52:08.870 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:52:08.871 DEBUG o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [UPDATE INT_LOCK SET CREATED_DATE=? WHERE REGION=? AND LOCK_KEY=? AND CLIENT_ID=?]
13:52:08.871 DEBUG o.s.orm.jpa.JpaTransactionManager - Initiating transaction commit
13:52:08.871 DEBUG o.s.orm.jpa.JpaTransactionManager - Committing JPA transaction on EntityManager [SessionImpl(1712215220<open>)]
13:52:08.872 DEBUG o.h.e.t.internal.TransactionImpl - committing
13:52:08.875 DEBUG o.s.jdbc.datasource.DataSourceUtils - Resetting isolation level of JDBC Connection [HikariProxyConnection@1507945608 wrapping org.postgresql.jdbc.PgConnection@7c72ecc] to 2
13:52:08.875 DEBUG o.s.orm.jpa.JpaTransactionManager - Closing JPA EntityManager [SessionImpl(1712215220<open>)] after transaction
13:52:09.376 DEBUG o.s.i.s.l.LockRegistryLeaderInitiator - Acquiring the lock for LockContext{role=leader, id=ce77af6e-4bb0-459e-a83d-f33ef9192fd3, isLeader=true}
....
我的预期结果将是Spring Boot可以自然地处理此问题,并且可以使我的pvc正常运行并运行一段时间...此问题导致我的1Gi pvc在3天内就装满了...