Question

120

How to fetch data over multiple pages?

rated 0 times [ 123] [ 3] / answers: 1 / hits: 17291 / 8 Years ago, fri, november 18, 2016, 12:00:00

My project is based on React, redux, redux-saga, es6 and I try to fetch data from this API:

http://api.dhsprogram.com/rest/dhs/data/BD,2000,2004,2007?&returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json

As you can see, this specific API call shows data with a limit of 100 data per page spread on 40 pages.

According to this answer:
http://userforum.dhsprogram.com/index.php?t=msg&th=2086&goto=9591&S=Google
it says that you can extend the limit to a maximum of 3000data per page.

However, in some cases I would do an API call that exceeds that limit which means I would not receive all my data doing it like this:

export function fetchMetaData(countryCode: string, surveyYears: string) {

return (fetch('http://api.dhsprogram.com/rest/dhs/data/' + countryCode + ',' + surveyYears + '?returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json')

    .then(response => response.json())

    .then(json => json.Data.map(survey => survey)))

}

So my question is; what is the best way to get all data from this API given that I know the total pages of data. The answer in the forum link suggest to loop through the API. However, I can't find the right syntax usage to do this.

My idea would be doing one api call to get the total number of pages. Then store this in a state using redux+redux-saga. Then do a new request sending the total pages as parameter and fetch this total number of pages times. And by doing this I can't figure out the syntax to store the data for each iteration.

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

savanar

Add To Favorites

Follow

Total Points: 237

Total Questions: 105

Total Answers: 99

Location: Wales

Member since Mon, May 17, 2021

3 Years ago

answered 8 Years ago sidneyh · Accepted Answer

A possible solution -the idea is to get the number of pages first, then make the appropriate number of API calls, pushing the promise from each call into an array. We then wait for all the promises to resolve, and do something with the returned data.

async function fetchMetaData() {



    const response = await fetch('apiUrlToGetPageNumber');



    const responses = await Promise.all(

        Array.from(

            Array(resp.data.pagesRequired),

            (_, i) => fetch(`apiUrlToSpecificPage?page=${i}`)

        )

    );

    

    // do something with processedResponses here



}