处理可运行任务期间的大量内存消耗

Question

我上了这个课：

@Component
public class TestClass extends Verify {

    private final ThreadPoolTaskExecutor processExecutor;
    private final BlockingQueue<KPIData> kpiData;

    @Autowired
    public TestClass(
            final DbAddresser dbAddresser,
            final MyDao myDao,
            final ThreadPoolTaskExecutor processExecutor) {
        super(VerificationType.KPI, dbAddresser, myDao);
        this.processExecutor = processExecutor;
        kpiData = new ArrayBlockingQueue<>(20);
    }

    @Override
    public void verification() throws Exception {
        startedVerification();
        verifyIndicator.setTitle("tasks");
        final Thread consumerKPI = new Thread(() -> {
            LOG.debug("Consumer thread started....");
            try {
                KPIData kpiData;
                while (!(kpiData = this.kpiData.take()).isLastElement()) {
                    final InconsistsKPICallable callable = new InconsistsKPICallable(((KPIDataImpl) kpiData).getRes());
                    execDispatcher.put(processExecutor.submit(callable));
                    verifyIndicator.incMaxStep();
                }
            } catch (Exception e) {
                LOG.error("Consumer thread error: ", e);
            }
            LOG.debug("Consumer thread finished....");
        });
        consumerKPI.start();

        try {

            dbAddresser.getKpiAll(kpiData, 5000);

            final Pair<Long, Long> minMaxEntities = dbAddresser::getMinMaxEntities;
            if (minMaxEntities != null) {
                long currentEntityId = minMaxEntities.getLeft();

                while (currentEntityId <= minMaxEntities.getRight()) {
                    if (stopped) {
                        break;
                    }

                    long fromEntityId = currentEntityId;
                    long toEntityId = currentEntityId + 4999;

                    incMaxStep();

                    InconsistsKPIneedUpdateCallable callable = new InconsistsKPIneedUpdateCallable(fromEntityId, toEntityId);
                    execDispatcher.put(processExecutor.submit(callable));
                    currentEntityId = toEntityId + 1;
                }
            }
        } finally {
            consumerKPI.join();
            waitFinished();
            finishedVerification();
        }
    }

    public class InconsistsKPICallable implements Runnable {
        private final List<Pair<Long, String>> kpiList;

        InconsistsKPICallable(final List<Pair<Long, String>> kpiList) {
            this.kpiList = kpiList;
        }

        @Override
        public void run() {
            try {
                // do hard work with kpiList
            } catch (Exception ex) {
            } finally {
                LOG.debug("InconsistsSmbpKPICallable finished!");
            }
            incCurStep();
        }
    }

    public class InconsistsKPIneedUpdateCallable implements Runnable {
        private final Long fromEntityId;
        private final Long toEntityId;

        InconsistsKPIneedUpdateCallable(final long fromEntityId, final long toEntityId) {
            this.fromEntityId = fromEntityId;
            this.toEntityId = toEntityId;
        }

        @Override
        public void run() {
            try {
                // do hard work with fromEntityId and toEntityId
            } catch (Exception ex) {
            } finally {
                LOG.debug("InconsistsKPIneedUpdateCallable finished for fromEntityId {} toEntityId {}", fromEntityId, toEntityId);
            }
            incCurStep();
        }
    }
}

整个问题是在这里：

while (!(kpiData = this.kpiData.take()).isLastElement()) {
      final InconsistsKPICallable callable = new InconsistsKPICallable(((KPIDataImpl) kpiData).getRes());
      execDispatcher.put(processExecutor.submit(callable));
      verifyIndicator.incMaxStep();
}

我们很快创建了一个任务，从而阻塞了所有内存。所有这些都是由于Runnable任务的处理速度非常慢，这与我们创建它们的方式不同。我尝试为ThreadPoolTaskExecutor指定队列的大小，但在此我还要依靠一个事实，即我们很快对其进行了评分并得到了一个例外：org.springframework.core.task.TaskRejectedException。提示如何重组此代码以减少内存消耗，并且不使任务异常退出？我尝试过寻找确保队列已满的方法，添加的调用线程只是在等待而不是异常退出，但是找不到足够的解决方案。