Question

我需要使用webclient和flux调用其余的分页api。我已经尝试过一种阻塞的方式（一个接一个），但是我想使其平行。让我们一次说10个并行调用。每个调用都会获取1000条记录。
我已经在调用第0个请求以从标题中获取总记录数。请求完成后，我需要调用POST api发送此响应（1000条记录）。

如果任何请求完成，则将发送第11个请求，依此类推。我已经看过asyncRestTemplate和可听期货的其他示例，但是asyncRestTemplate已被弃用，替代方法是spring-webflux

也
由于rest template将被弃用

我做了什么。

除以总计数/ 1000->给出总页数
循环执行到5（如果我更改为totalpages，那么它会给我500内部服务器错误）
调用返回Mono的服务>
订阅每个请求

ObjectMapper objmapper = new ObjectMapper();

HttpHeaders headers = partsService.getHeaders();
long totalCount = Long.parseLong(headers.get("total-count").get(0));
log.info(totalCount);
long totalPages = (long) Math.ceil((double) totalCount / 1000);
log.info(totalPages);
// List<Mono<List<Parts>>> parts = new ArrayList<>();
for (long i = 1; i <= 5; i++) {
    partsService.fetchAllParts(1000L, i).log().subscribe(partList -> {
        try {
            // post each request response to another API
            log.info(objmapper.writeValueAsString(partList));
        } catch (JsonProcessingException ex) {
            ex.printStackTrace();
        }

    });
    log.info("Page Number:" + i);
}

我想在没有任何outOfmemoryerror的情况下并行执行，并且不会给调用api带来太多负担。另外，我尝试一次获取所有页面，但出现500内部服务器错误。

我是Flux（项目反应堆）的新手

在解决方案下方实现

它不是并行运行，单个请求大约需要2分钟的时间，这意味着所有10（并发级别）都应同时完成。

try {
        fetchTotalCount().log()
                .flatMapMany(totalCount -> createPageRange(totalCount, 1000)).log()
                .flatMap(pageNumber -> fetch(1000, pageNumber), 10).log()
                .flatMap(response -> create(response))
                .subscribe();

        } catch (Exception e) {
            e.printStackTrace();
        }

日志

2019-07-29T09:00:14,477 INFO  [scheduling-1] r.u.Loggers$Slf4JLogger: request(10)
2019-07-29T09:00:14,478 INFO  [scheduling-1] r.u.Loggers$Slf4JLogger: request(10)
2019-07-29T09:00:14,479 INFO  [scheduling-1] r.u.Loggers$Slf4JLogger: request(unbounded)
2019-07-29T09:00:14,679 INFO  [scheduling-1] c.o.q.l.Logging: fetch() execution time: 546 ms
2019-07-29T09:00:17,028 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(74577)
2019-07-29T09:00:17,042 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(1)
2019-07-29T09:00:17,068 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,1) execution time: 24 ms
2019-07-29T09:00:17,078 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(2)
2019-07-29T09:00:17,080 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,2) execution time: 2 ms
2019-07-29T09:00:17,083 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(3)
2019-07-29T09:00:17,087 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,3) execution time: 2 ms
2019-07-29T09:00:17,096 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(4)
2019-07-29T09:00:17,098 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,4) execution time: 1 ms
2019-07-29T09:00:17,100 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(5)
2019-07-29T09:00:17,101 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,5) execution time: 1 ms
2019-07-29T09:00:17,103 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(6)
2019-07-29T09:00:17,106 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,6) execution time: 3 ms
2019-07-29T09:00:17,108 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(7)
2019-07-29T09:00:17,110 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,7) execution time: 2 ms
2019-07-29T09:00:17,113 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(8)
2019-07-29T09:00:17,115 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,8) execution time: 1 ms
2019-07-29T09:00:17,116 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(9)
2019-07-29T09:00:17,118 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,9) execution time: 1 ms
2019-07-29T09:00:17,119 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(10)
2019-07-29T09:00:17,121 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,10) execution time: 1 ms
2019-07-29T09:00:17,123 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onComplete()
2019-07-29T09:09:03,295 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:03,296 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(11)
2019-07-29T09:09:03,296 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,11) execution time: 0 ms
2019-07-29T09:09:03,730 INFO  [reactor-http-nio-1] c.o.q.s.Scheduler: 200 OK
2019-07-29T09:09:03,730 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:05,106 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(// data print)
2019-07-29T09:09:05,196 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:05,196 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(12)
2019-07-29T09:09:05,198 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,12) execution time: 1 ms
2019-07-29T09:09:05,466 INFO  [reactor-http-nio-1] c.o.q.s.Scheduler: 200 OK
2019-07-29T09:09:05,466 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:09,565 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(//  data print)
2019-07-29T09:09:09,730 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:09,730 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(13)
2019-07-29T09:09:09,731 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,13) execution time: 0 ms
2019-07-29T09:09:10,049 INFO  [reactor-http-nio-1] c.o.q.s.Scheduler: 200 OK

Answer 1

Flux.flatMap具有用于设置并发级别的参数，可用于协调并行化。

在下面的示例中，我使用了伪URL，示例中的一些片段以及一些其他简单的代码来演示如何实现此目的：

keyExtractor={(item, index) =>

Answer 2

使用.flatMap()的重载版本，您可以在其中指定并发级别。通常，.flatMap()会热切地订阅所有内部流-在订阅下一个流之前，它不会等待任何流完成（与.concatMap()不同）。但是您指定了一个并发级别，它只会急切地（一次）订阅那么多内部流。在您的情况下，仅当最初的10个内部流中的至少一个完成时，第11个内部流才会被订阅。这完全可以解决您的问题。

使用WebClient和Flux进行多重异步其余分页调用

2 个答案: