Question

我正在编写一个AWS SWF工作流，其中的第一个活动将评估分区的数量，并将其传递给子工作流。决策程序中的子工作流程会循环遍历第一个活动返回的分区并启动最大值。允许并行活动。假设最大允许值为50，一次并行启动50个活动。但是，我们面临的问题是接下来的50个问题要等到所有50个问题都完成才能启动，即它将所有其他分区停滞直到所有50个问题都完成。执行以50批次执行。下面是代码示例：

@Override
public void execute(@Nonnull final Execution execution, @Nonnull final Step step)
        throws ExecutionException {
    Promise<Queue<Step>> stepQueuePromise = activitiesClient.partition(step, context);
    Promise<Void> exitCode = executeSteps(stepQueuePromise, execution, context);
    activitiesClient.verify(step, exitCode);
}

@Asynchronous
private Promise<Void> executeStepsInBatches(@Nonnull final Queue<Step> stepQueue,
                                   @Wait final List<Promise<Void>> previousSteps,
                                   @Nonnull final Execution execution,
                                   @Nonnull final Context context) {

    List<Promise<Void>> stepListPromises = new ArrayList<>();

    for (int i = 0; i < concurrentStepThreshold && !stepQueue.isEmpty(); i++) {
        Promise<Void> stepPromise = activitiesClient.execute(execution, stepQueue.poll(), context);
        stepListPromises.add(stepPromise);
    }

    if (!stepListPromises.isEmpty()) {
        return executeStepsInBatches(stepQueue, stepListPromises, execution, context);
    } else {
        return Promise.Void();
    }
}

我们要分批执行50个活动，但是一旦完成几个活动，就应提交新活动以匹配50个并行运行计数。有人可以建议我们如何实现这一目标吗？

编辑（新代码）

我尝试了以下代码：

@Override
public void execute(@Nonnull final Execution execution, @Nonnull final Step step)
        throws ExecutionException {

    Promise<Queue<Step>> stepQueuePromise = activitiesClient.partition(step, context);
    executeSteps(stepQueuePromise, execution, context);
}

@Asynchronous
private void executeSteps(@Nonnull final Promise<Queue<Step>> stepQueuePromise,
                          @Nonnull final Execution execution,
                          @Nonnull final Context context) {
    Integer numNotReady = 0;
    List<Promise<Void>> currentPromises = new ArrayList<>();
    Iterator<Step> inputItr = stepQueuePromise.get().iterator();
    while (inputItr.hasNext() && numNotReady < 20) {
        Promise<Void> promise = activitiesClient.execute(execution, inputItr.next(), context);
        currentPromises.add(promise);

        if (!promise.isReady()) {
            numNotReady++;
        }
    }
    log.info("Num of not ready" + numNotReady);
    waitForPromises(currentPromises);
}

@Asynchronous
void waitForPromises(@Wait final List<Promise<Void>> activityOutputs) {
}

第一个循环并行启动20个活动。但是，即使决策者已运行，也未提交新活动。我可以看到我添加的日志，以验证决策者已运行： 2018年8月6日17：16：34,962 [信息]（SWF Decider ExecutorTaskList_1.0 1）com.amazon.traffic.cafe.orchestrator.swf.exec.impl.SwfExecutorImpl：未准备好的数字20 2018年8月6日17：16：50,808 [INFO]（SWF Decider ExecutorTaskList_1.0 1）com.amazon.traffic.cafe.orchestrator.swf.exec.impl.SwfExecutorImpl：

的数字

Answer 1

我相信以下方法应该有效：

@Asynchronous
private Promise<Void> executeStepsInBatches(@Nonnull final Queue<Step> stepQueue,
                                   @Wait final List<Promise<Void>> previousSteps,
                                   @Nonnull final Execution execution,
                                   @Nonnull final Context context) {

    List<Promise<Void>> stepListPromises = new ArrayList<>();

    for (int i = 0; i < concurrentStepThreshold; i++) {
        Promise<Void> stepPromise = executeNext(stepQueue, execution, context, stepPromise);
        stepListPromises.add(stepPromise);
    }
    return Promises.listOfPromisesToPromise(stepListPromises);
}

@Asynchronous
private Promise<Void> executeNext(Queue<Step> stepQueue, Execution execution, Context context, Promise<?> previous) {
   if (stepQueue.isEmpty()) {
     return Promise.Void();
   }
   Promise<Void> stepPromise = activitiesClient.execute(execution, stepQueue.poll(), context);
   // Loop recursively
   return executeNext(stepQueue, execution, stepPromise);    
}

Answer 2

最后，以下代码可以正常工作并经过测试：

@Override
public void execute(@Nonnull final Execution execution, @Nonnull final Step step)
        throws ExecutionException {


    Context context = getContext(execution, step);

    Promise<Queue<Step>> stepQueue = activitiesClient.partition(step, context);

    /**
     * List to hold the promise of started activities.
     */
    Promise<?>[] batchPromises = new Promise<?>[50];

    /**
     * Initialize the list with ready promises.
     */
    Arrays.fill(batchPromises, Promise.Void());

    /**
     * OrPromise list to unblock as soon as one of the activity is completed.
     */
    OrPromise waitForAtleastOneInBatch = new OrPromise(batchPromises);

    Promise<Void> exitCode = startActivityInBatch(execution, context, stepQueue, waitForAtleastOneInBatch);

}

@Asynchronous
private Promise<Void> startActivityInBatch(final Execution execution, final Context context,
                                            final Promise<Queue<Step>> stepQueue,
                                            final OrPromise waitForAtleastOneInBatch) {
    /**
     * Executes only when one of the promise is ready.
     */
    Promise<?>[] existingBatchPromises = waitForAtleastOneInBatch.getValues();

    /**
     * In this loop, we iterate over the promise list and if the promise is ready we replace it with
     * new promise by starting new activity.
     */
    for (int existingBatchIterator = 0; existingBatchIterator < existingBatchPromises.length;
         existingBatchIterator++) {
        /**
         * If the existing promise is ready, call the next task replace the ready promise.
         */
        if (existingBatchPromises[existingBatchIterator].isReady()) {
            final Step step = stepQueue.get().poll();
            if (step == null) {
                /**
                 * This means that queue is empty and we have run all the activities.
                 */
                existingBatchPromises[existingBatchIterator] = Promise.Void();
            } else {
                existingBatchPromises[existingBatchIterator] = activitiesClient.execute(execution, step, context);
            }
        }
    }

    /**
     * call recursively till  we have messages in queue.
     */
    if (stepQueue.get().size() > 0) {
        return startActivityInBatch(execution, context, stepQueue, new OrPromise(existingBatchPromises));
    } else {
        /**
         * AndPromise is used to make the workflow wait till all Promises are ready.
         */
        return new AndPromise(existingBatchPromises);
    }
}

在AWS SWF中运行并行活动

2 个答案: