Question

我的任务：我有一个包含许多项目的文件，每个项目都与我需要下载的图像URL数组有关。我想下载所有链接，我正在使用this library进行图像下载，并且正在使用Promise。

问题： 当我开始从许多项目中下载许多图像时，会出现问题，该程序在第一个请求完成之前发送了4000多个请求，程序崩溃了。

我的解决方案：我的想法是一次只能处理大约2个项目，因此我一次只能下载20张图片。我已经尝试过使用Promise和异步功能进行各种变体的操作，但是我对这些功能还很陌生，因此尝试失败了。

我的代码流是这样的：

csvRun()

function csvRun(){
    for(let i = 1; i <= itemsAmount; i++){  // Loops over the items
        // I want to be able to run only x items at a time
        console.log('Item number ' + i)
        itemHandle() 
    }
}

function itemHandle(){ // This function seems useless here but since the item has more data I kept it here
    handleImages()
}


function handleImages(){  // Loops over the images of the item
    for(let g = 0; g < imagesAmount; g++){        
        // Here there is a promise that downloads images
        // For the example I'll use settimout
        setTimeout(() => {
            console.log('Image downloaded ' + g)
        }, 3000);

        /** If you want the error just use ImgDonwload instead of
            settimeout and set imagesAmount to 20 and itemsAmount 
            to 400
        */ 

    }

}

// Only here to recreate the error. Not necessarily relevant.
function ImgDownload(){
    var download = require('image-downloader')
    download // returns the promise so the handling could resume in order
    .image({
        url:
            "https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg",
        dest: "/folder/img.jpg"
    })
    .then(({ filename, image }) => {
        console.log("File saved to", filename);
    })
    .catch((err: Error) => {
        console.error(err);
    });
}

当前，代码完成了csvRun中的循环并打印出Item number 1，直到Item number {itemsAmount}，并在3秒钟后打印出了所有Image downloaded messages。我知道为什么会这样。我想更改代码，以便每次仅同时进行两次对itemHandle的调用。

Answer 1

一种选择是让循环遍历图像并依次处理。然后要并行运行多个处理，请启动多个循环：

  // Goes over the "data" array, calls and waits for each "task" and processes "runnerCount" tasks in parallel
  function inParallel(task, data, runnerCount) {
    let i = 0, results = [];

    async function runner() {
      while(i < data.length) {
         const pos = i++; // be aware: concurrent modification of i
         const entry = data[pos]; 
         results[pos] = await task(entry);
      }
   }

    const runners = Array.from({ length: runnerCount }, runner);

    return Promise.all(runners).then(() => results);
 }

用作：

  const delay = ms => new Promise(res => setTimeout(res, ms));

 inParallel(async time => {
   console.log(`Timer for ${time}ms starts`);
   await delay(time);
   console.log(`Timer for ${time}ms ends`);
 }, [5000, 6000, 1000]/*ms*/, 2/*in parallel*/);

Answer 2

假设您的数据如下所示

const items = [
  { id: 1,
    images: [
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
     ]
  },
  { id: 2,
    images: [
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
     ]
  },
  { id: 3,
    images: [
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
      'https://cdn.vox-cdn.com/thumbor/XKPu8Ylce2Cq6yi_pgyLyw80vb4=/0x0:1920x1080/1200x800/filters:focal(807x387:1113x693)/cdn.vox-cdn.com/uploads/chorus_image/image/63380914/PIA16695_large.0.jpg',
     ]
  }
];

我将运行一个简单的for..of循环并遍历图像并逐项下载

// this function will try to download images per items
const download = require('image-downloader')
const downloadImages = async (items = []) => {
  let promises = [];
  for (const item of items) {
    const images = item.images;
    // dest is item.id/imageIndex.jpg
    promsies = images.map((url, index) => download({url, dest: `/folder/${item.id}/${index}.jpg`}));
    await Promise.all(promises);
  }
}

downloadImages(items);

Answer 3

带有香草保证，您可能会做类似的事情：

let pending_fetches = 0
const MAX_CONCURRENT = 2

const fetch_interval = setInterval(() => {
    if (items.length === 0) return clearInterval(fetch_interval)

    if (pending_fetches < MAX_CONCURRENT) {
        ++pending_fetches
        doFetch(items.pop()).then(response => {
            // do stuff with the response
            --pending_fetches
        })
    }    
}, 100)

与async / await类似：

const MAX_CONCURRENT = 2

const fetchLoop = async () => {
    while (items.length > 0) {
        const response = await doFetch(items.pop())
        // do stuff with the response
    }
}
for (let i = 0; i < MAX_CONCURRENT; ++i) fetchLoop()

Answer 4

我想我仍然更喜欢Jonas的实现，因为它简洁明了，但是我会在环上添加另一个。一些功能：

结果和错误位于稳定的数组中（基于位置）。
这将在单个工作者功能完成后立即开始处理另一个项目，而不是分批处理并等待每个Promise.all解决。

function parallelMap(values, workFn, maxConcurrency = 2) {
  const length = values.length;
  const results = Array.from({ length });

  let pos = 0;
  let completed = 0;

  return new Promise(resolve => {
    function work() {
      if (completed === length) {
        return resolve(results);
      }

      if (pos >= length) {
        return;
      }

      const workIndex = pos;
      const item = values[workIndex];
      pos = pos + 1;

      return workFn(item, workIndex)
        .then(result => {
          results[workIndex] = result;
          completed = completed + 1;
          work();
        })
        .catch(result => {
          results[workIndex] = result;
          completed = completed + 1;
          work();
        });
    }

    for (let i = 0; i < maxConcurrency; i++) {
      work();
    }
  });
}

用法：

async function fakeRequest({ value, time = 100, shouldFail = false }) {
  return new Promise((resolve, reject) => {
    setTimeout(() => {
      if (shouldFail) {
        reject("Failure: " + value);
      } else {
        resolve("Success: " + value);
      }
    }, time);
  });
}

test("basic 'working' prototype", async () => {
  const values = [1, 2, 3, 4, 5, 6];
  const results = await parallelMap(values, value => {
    return fakeRequest({ value, time: 100, shouldFail: value % 2 === 0 });
  });

  expect(results).toEqual([
    "Success: 1",
    "Failure: 2",
    "Success: 3",
    "Failure: 4",
    "Success: 5",
    "Failure: 6"
  ]);
}, 350); // each task takes ~100ms to complete, 6 tasks, two at a time = ~300 ms

有关完整的测试套件，请参见codesandbox。

等待异步功能并承诺完成

4 个答案: