基于锁的领导者选举使K8S集群中的PVC死亡

时间:2019-05-27 14:00:30

标签: postgresql spring-boot kubernetes cluster-computing load-balancing

我正在尝试使用一个postgres数据库将我的应用程序与k8s上的一些计划任务进行群集。想象以下设置:2x App,每个App 1 DB,例如1个cronjob。更新一些缓存。

由于我不想使用Zookeeper或任何其他实现,因此我尝试将Spring Boot领导选举与基于lock的表(INT_LOCK)结合使用,如spring文档中所述。

不幸的是,应用程序每隔几毫秒就会查询一次数据库锁定表,以查看谁是领导者。到目前为止,还不错,但是未持有锁的应用正在尝试插入新条目,因此数据库抛出了违反主键约束的情况。由于我的postgres db的错误日志条目,这种情况一直持续到我的pvc已满...

此过程似乎在Internet上已被不良记录。.我很高兴听到您可能有任何建议;) 如果需要提供更多信息,请告诉我。

谢谢!

我的领导者配置类:

@Configuration
public class LeadershipConfiguration {

    private static final int LOCK_TIME_TO_LIVE = 120000;

    private static final int LOCK_HEARTBEAT_TIME = 100000;

    private static final int LOCK_ACQUIRE_WAIT_TIME = 110000;

    @Value("${HOSTNAME:noname}")
    private String podName;

    @Value("${REGION:noregion}")
    private String region;

    @Bean
    public DefaultLockRepository lockRepository(final DataSource dataSource) {
        String clientId;

        final DefaultLockRepository defaultLockRepository = new DefaultLockRepository(dataSource, this.podName);

        defaultLockRepository.setRegion(this.region);
        defaultLockRepository.setTimeToLive(LOCK_TIME_TO_LIVE);
        return defaultLockRepository;
    }

    @Bean
    public JdbcLockRegistry lockRegistry(final LockRepository lockRepository) {
        return new JdbcLockRegistry(lockRepository);
    }

    @Bean
    public LockRegistryLeaderInitiator leaderInitiator(final LockRegistry lockRegistry) {
        final LockRegistryLeaderInitiator lockRegistryLeaderInitiator = new LockRegistryLeaderInitiator(lockRegistry, new DefaultCandidate());
        lockRegistryLeaderInitiator.setHeartBeatMillis(LOCK_HEARTBEAT_TIME);
        lockRegistryLeaderInitiator.setBusyWaitMillis(LOCK_ACQUIRE_WAIT_TIME);
        return lockRegistryLeaderInitiator;
    }

}

我更新了LeadershipConfiguration,因为根据此document和此document,这应该增加查询之间的时间,但是看来这也不起作用

这也是Pod中的一些日志语句:

DB日志提取,这3个日志stmt经常出现:

2019-05-27 13:52:04 UTC 
ERROR:  duplicate key value violates unique constraint "pk_int_lock_lock_key"
2019-05-27 13:52:04 UTC 
DETAIL:  Key (lock_key, region)=(c444858e-0aae-3727-9a73-d2eae62321ad, DEFAULT) already exists.
2019-05-27 13:52:04 UTC 
STATEMENT:  INSERT INTO INT_LOCK (REGION, LOCK_KEY, CLIENT_ID, CREATED_DATE) VALUES ($1, $2, $3, $4)
etc... re-occuring statements

从不是领导者的Pod中提取数据库日志:

13:49:59.441  DEBUG  o.s.jdbc.datasource.DataSourceUtils - Resetting isolation level of JDBC Connection [HikariProxyConnection@512836674 wrapping org.postgresql.jdbc.PgConnection@61ab89b0] to 2
13:49:59.444  DEBUG  o.s.orm.jpa.JpaTransactionManager - Closing JPA EntityManager [SessionImpl(1800314718<open>)] after transaction
13:49:59.546  DEBUG  o.s.orm.jpa.JpaTransactionManager - Creating new transaction with name [org.springframework.integration.jdbc.lock.DefaultLockRepository.acquire]: PROPAGATION_REQUIRED,ISOLATION_SERIALIZABLE,timeout_1
13:49:59.547  DEBUG  o.s.orm.jpa.JpaTransactionManager - Opened new EntityManager [SessionImpl(1465858215<open>)] for JPA transaction
13:49:59.547  DEBUG  o.h.e.t.internal.TransactionImpl - On TransactionImpl creation, JpaCompliance#isJpaTransactionComplianceEnabled == false
13:49:59.549  DEBUG  o.s.jdbc.datasource.DataSourceUtils - Changing isolation level of JDBC Connection [HikariProxyConnection@1224014205 wrapping org.postgresql.jdbc.PgConnection@61ab89b0] to 8
13:49:59.553  DEBUG  o.h.e.t.internal.TransactionImpl - begin
13:49:59.553  DEBUG  o.s.orm.jpa.JpaTransactionManager - Exposing JPA transaction as JDBC [org.springframework.orm.jpa.vendor.HibernateJpaDialect$HibernateConnectionHandle@5cd30775]
13:49:59.553  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:49:59.553  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [DELETE FROM INT_LOCK WHERE REGION=? AND LOCK_KEY=? AND CREATED_DATE<?]
13:49:59.555  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:49:59.555  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [UPDATE INT_LOCK SET CREATED_DATE=? WHERE REGION=? AND LOCK_KEY=? AND CLIENT_ID=?]
13:49:59.556  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:49:59.557  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [INSERT INTO INT_LOCK (REGION, LOCK_KEY, CLIENT_ID, CREATED_DATE) VALUES (?, ?, ?, ?)]
13:49:59.559  DEBUG  o.s.j.s.SQLErrorCodeSQLExceptionTranslator - Translating SQLException with SQL state '23505', error code '0', message [ERROR: duplicate key value violates unique constraint "pk_int_lock_lock_key"
  Detail: Key (lock_key, region)=(c444858e-0aae-3727-9a73-d2eae62321ad, DEFAULT) already exists.]; SQL was [INSERT INTO INT_LOCK (REGION, LOCK_KEY, CLIENT_ID, CREATED_DATE) VALUES (?, ?, ?, ?)] for task [PreparedStatementCallback]
13:49:59.559  DEBUG  o.s.orm.jpa.JpaTransactionManager - Initiating transaction commit
13:49:59.559  DEBUG  o.s.orm.jpa.JpaTransactionManager - Committing JPA transaction on EntityManager [SessionImpl(1465858215<open>)]
13:49:59.561  DEBUG  o.h.e.t.internal.TransactionImpl - committing
13:49:59.561  DEBUG  o.s.jdbc.datasource.DataSourceUtils - Resetting isolation level of JDBC Connection [HikariProxyConnection@1224014205 wrapping org.postgresql.jdbc.PgConnection@61ab89b0] to 2
....

从作为领导者的Pod中提取数据库日志:

13:52:08.861  DEBUG  o.s.i.s.l.LockRegistryLeaderInitiator - Acquiring the lock for LockContext{role=leader, id=ce77af6e-4bb0-459e-a83d-f33ef9192fd3, isLeader=true}
13:52:08.862  DEBUG  o.s.orm.jpa.JpaTransactionManager - Creating new transaction with name [org.springframework.integration.jdbc.lock.DefaultLockRepository.acquire]: PROPAGATION_REQUIRED,ISOLATION_SERIALIZABLE,timeout_1
13:52:08.864  DEBUG  o.s.orm.jpa.JpaTransactionManager - Opened new EntityManager [SessionImpl(1712215220<open>)] for JPA transaction
13:52:08.865  DEBUG  o.h.e.t.internal.TransactionImpl - On TransactionImpl creation, JpaCompliance#isJpaTransactionComplianceEnabled == false
13:52:08.865  DEBUG  o.s.jdbc.datasource.DataSourceUtils - Changing isolation level of JDBC Connection [HikariProxyConnection@1507945608 wrapping org.postgresql.jdbc.PgConnection@7c72ecc] to 8
13:52:08.869  DEBUG  o.h.e.t.internal.TransactionImpl - begin
13:52:08.869  DEBUG  o.s.orm.jpa.JpaTransactionManager - Exposing JPA transaction as JDBC [org.springframework.orm.jpa.vendor.HibernateJpaDialect$HibernateConnectionHandle@7f1d0eae]
13:52:08.870  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:52:08.870  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [DELETE FROM INT_LOCK WHERE REGION=? AND LOCK_KEY=? AND CREATED_DATE<?]
13:52:08.870  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL update
13:52:08.871  DEBUG  o.s.jdbc.core.JdbcTemplate - Executing prepared SQL statement [UPDATE INT_LOCK SET CREATED_DATE=? WHERE REGION=? AND LOCK_KEY=? AND CLIENT_ID=?]
13:52:08.871  DEBUG  o.s.orm.jpa.JpaTransactionManager - Initiating transaction commit
13:52:08.871  DEBUG  o.s.orm.jpa.JpaTransactionManager - Committing JPA transaction on EntityManager [SessionImpl(1712215220<open>)]
13:52:08.872  DEBUG  o.h.e.t.internal.TransactionImpl - committing
13:52:08.875  DEBUG  o.s.jdbc.datasource.DataSourceUtils - Resetting isolation level of JDBC Connection [HikariProxyConnection@1507945608 wrapping org.postgresql.jdbc.PgConnection@7c72ecc] to 2
13:52:08.875  DEBUG  o.s.orm.jpa.JpaTransactionManager - Closing JPA EntityManager [SessionImpl(1712215220<open>)] after transaction
13:52:09.376  DEBUG  o.s.i.s.l.LockRegistryLeaderInitiator - Acquiring the lock for LockContext{role=leader, id=ce77af6e-4bb0-459e-a83d-f33ef9192fd3, isLeader=true}
....

我的预期结果将是Spring Boot可以自然地处理此问题,并且可以使我的pvc正常运行并运行一段时间...此问题导致我的1Gi pvc在3天内就装满了...

0 个答案:

没有答案