Question

我有无穷无尽的事件，我想：

进入微处理器，每个100毫秒，或10000个元素
根据分组功能对它们进行分组
并行处理每个分组项目列表

我的代码如下：

Flux<SuchEvent> suchFlux = Flux.fromStream(events);
Scheduler parallel = Schedulers.newParallel("asd", 64);
suchFlux
            .bufferTimeout(10000, Duration.ofMillis(100))
            .map(rawEvents -> {
                Map<UUID, List<SuchEvent>> groupedEvents = new HashMap<>();
                for (SuchEvent stuff : rawEvents) {
                    if (!groupedEvents.containsKey(stuff.getProfileId())) {
                        groupedEvents.put(stuff.getProfileId(), new ArrayList<>());
                    }
                    groupedEvents.get(stuff.getProfileId()).add(stuff);
                }
                return groupedEvents.values();
            })
            .subscribe(groupedEvents -> {
                for (List<SuchEvent> suchEvents : groupedEvents) {
                    Flux.fromIterable(suchEvents)
                        .subscribeOn(parallel)
                        .subscribe(suchEvent -> {
                            //do stuff (this is fairly slow, each call takes 50ms)
                        });
                }
            });

我希望内部订阅（suchEvent）能够对生产产生一些背压，然而，在运行一段时间后，一切似乎都停滞不前。我逻辑中的谬误在哪里？

Answer 1

当您使用lambda订阅时，它将触发无限制请求。要微调背压，您需要实现自己的Subscriber。

建议的解决方案是从BaseSubscriber扩展。

在反应堆文件中，他们提供了有用的信息：

最低限度的实现是实现hookOnSubscribe（订阅订阅）和hookOnNext（T值）。

所以你可以这样做：

        var user =  (from a in db1.ExternalUsers
                    join b in db1.ExternalUserEmails on a.ExternalUserID equals b.ExternalUserID
                    join c in db1.ExternalUserPhones on a.ExternalUserID equals c.ExternalUserID
                    where a.ExternalUserID == id
                    select new EditProfile {ExternalUserID = a.ExternalUserID, FirstName = a.FirstName, LastName = a.LastName, EmailAddress = b.EmailAddress, PhoneNumber = c.PhoneNumber })
                    .FirstOrDefault();

并使用它：

public static class SubscriberWithBackPressure<T> extends BaseSubscriber<T> {
    private final int maxRequest;
    private final Consumer<T> consumer;

    public SubscriberWithBackPressure(int maxRequest, Consumer<T> consumer) {
        this.maxRequest = maxRequest;
        this.consumer = consumer;
    }

    @Override
    protected void hookOnSubscribe(Subscription subscription) {
        subscription.request(maxRequest);
    }

    @Override
    protected void hookOnNext(T value) {
        if (consumer != null) {
            this.consumer.accept(value);
        }
        request(maxRequest);
    }
}

更新版本（不确定在每个转换级别处理背压）：

suchFlux
        .bufferTimeout(10000, Duration.ofMillis(100))
        .map(rawEvents -> {
            Map<UUID, List<SuchEvent>> groupedEvents = new HashMap<>();
            for (SuchEvent stuff : rawEvents) {
                if (!groupedEvents.containsKey(stuff.getProfileId())) {
                    groupedEvents.put(stuff.getProfileId(), new ArrayList<>());
                }
                groupedEvents.get(stuff.getProfileId()).add(stuff);
            }
            return groupedEvents.values();
        })
        .subscribe(groupedEvents -> {
            for (List<SuchEvent> suchEvents : groupedEvents) {
                Flux.fromIterable(suchEvents)
                    .subscribeOn(parallel)
                    .subscribe(new SubscriberWithBackPressure<>(100, suchEvent -> /*do stuff*/));
            }
        });

未施加助焊剂背压

1 个答案: