Question

43

What is the best way to limit concurrency when using ES6's Promise.all()?

rated 0 times [ 47] [ 4] / answers: 1 / hits: 137272 / 8 Years ago, wed, november 16, 2016, 12:00:00

I have some code that is iterating over a list that was queried out of a database and making an HTTP request for each element in that list. That list can sometimes be a reasonably large number (in the thousands), and I would like to make sure I am not hitting a web server with thousands of concurrent HTTP requests.

An abbreviated version of this code currently looks something like this...

function getCounts() {

  return users.map(user => {

    return new Promise(resolve => {

      remoteServer.getCount(user) // makes an HTTP request

      .then(() => {

        /* snip */

        resolve();

      });

    });

  });

}



Promise.all(getCounts()).then(() => { /* snip */});

This code is running on Node 4.3.2. To reiterate, can Promise.all be managed so that only a certain number of Promises are in progress at any given time?

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

kinsley

Add To Favorites

Follow

Total Points: 352

Total Questions: 84

Total Answers: 94

Location: Denmark

Member since Tue, Jul 19, 2022

2 Years ago

kinsley questions

1 Module not found: Can't resolve '@material-ui/lab/TabContext' in '/home/sanika/Desktop/Coding/React/my-forms/src/Components'

Sat, Aug 21, 21, 00:00, 3 Years ago

1 TypeError [ERR_HTTP_INVALID_HEADER_VALUE] for Algolia with Firebase

Mon, Jul 20, 20, 00:00, 4 Years ago

1 ReactJS: propTypes validation for array of objects

Mon, Nov 25, 19, 00:00, 5 Years ago

1 styled-components: extend styles and change element type

Sat, Apr 20, 19, 00:00, 5 Years ago

1 Changing the display of an element via Javascript?

Wed, Feb 20, 19, 00:00, 5 Years ago

View All

answered 8 Years ago kristopher · Accepted Answer

Note that Promise.all() doesn't trigger the promises to start their work, creating the promise itself does.

With that in mind, one solution would be to check whenever a promise is resolved whether a new promise should be started or whether you're already at the limit.

However, there is really no need to reinvent the wheel here. One library that you could use for this purpose is es6-promise-pool. From their examples:

var PromisePool = require('es6-promise-pool')

 

var promiseProducer = function () {

  // Your code goes here. 

  // If there is work left to be done, return the next work item as a promise. 

  // Otherwise, return null to indicate that all promises have been created. 

  // Scroll down for an example. 

}

 

// The number of promises to process simultaneously. 

var concurrency = 3

 

// Create a pool. 

var pool = new PromisePool(promiseProducer, concurrency)

 

// Start the pool. 

var poolPromise = pool.start()

 

// Wait for the pool to settle. 

poolPromise.then(function () {

  console.log('All promises fulfilled')

}, function (error) {

  console.log('Some promise rejected: ' + error.message)

})