我创建了一个RXJS observable,可以在输入一组URL的情况下执行任务:
let scrapeObservable: Observable<IProduct> = Observable.from(urlList)
.mergeMap(url => Observable.forkJoin(
Observable.of(url),
getPageDom(url), // Returns Observable that completes with PageDom
getExtractionRule(parseURL(url)["Host"]) // returns observable with data needed to perform work on the page dom
).delay(timeDelay += intervalTime)) // scrapes one url every set interval - using 0 in this case
.mergeMap(result => getProductInfoFromDOMObservable(result[0], result[1], result[2][0], result[2][1], requiredFields)) // performs work on page dom using data above.
.merge(maxConcurrentRequests) // prevents more than an certain number of processes from taking place simultaneously.
.catch(handleError);
当我订阅这个observable时,似乎对于某些输入(相当大的URL列表&gt; 500)它从不调用完整或错误。它就像可观察的悬挂一样。我无法弄清楚为什么会发生这种情况,但这会导致我的代码出现问题。对于小输入,它可以正常工作,并且对于大输入它总是不会失败,只是在某些时候。
关于可能导致此行为的任何想法?