Monday, May 20, 2024
 Popular · Latest · Hot · Upcoming
-2
rated 0 times [  3] [ 5]  / answers: 1 / hits: 6214  / 5 Years ago, thu, july 4, 2019, 12:00:00

I'm trying to scrape some web pages in the TOR network, using Puppeteer and the tor package (apt install tor).
Probably due to the nature of TOR connections sometimes I get a timeout.
In addition, I'm new to asynchronous programming in JavaScript.



Usually I have a try-catch-construct like these:



await Promise.all([
page.goto(url),
page.waitForNavigation({
waitUntil: 'domcontentloaded'
}),
]).catch((err) => { logMyErrors(err, true); });


or



let langMenu = await page.waitForXPath('//*[contains(@class, .customer_name)]/ancestor::li').catch((err) => { logMyErrors(err, true); });


But I think often one or more retries would help to finally get the desired resource. Is there any best practice to implement retries?


More From » node.js

 Answers
4

I would recommend this rather simple approach:



async function retry(promiseFactory, retryCount) {
try {
return await promiseFactory();
} catch (error) {
if (retryCount <= 0) {
throw error;
}
return await retry(promiseFactory, retryCount - 1);
}
}


This function calls the promiseFactory, and waits for the returned Promise to finish. In case an error happens the process is (recursively) repeated until retryCount reaches 0.



Code Sample



You can use the function like this:



await retry(
() => page.waitForXPath('//*[contains(@class, .customer_name)]/ancestor::li'),
5 // retry this 5 times
);


You can also pass any other function returning a Promise like Promise.all:



await retry(
() => Promise.all([
page.goto(url),
page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
]),
1 // retry only once
);


Don't combine await and catch



Another advice: You should not combine await with .then or .catch as this will result in unexpected problems. Either use await and surround your code with a try..catch block or use .then and .catch. Otherwise your code might be waiting for the results of a catch function to finish, etc.



Instead, you use try..catch like this:



try {
// ...
} catch (error) {
logMyErrors(error);
}

[#7032] Tuesday, July 2, 2019, 5 Years  [reply] [flag answer]
Only authorized users can answer the question. Please sign in first, or register a free account.
griffinr

Total Points: 242
Total Questions: 91
Total Answers: 105

Location: Indonesia
Member since Wed, Jul 7, 2021
3 Years ago
griffinr questions
Sat, Jul 18, 20, 00:00, 4 Years ago
Mon, Sep 16, 19, 00:00, 5 Years ago
Wed, Jun 5, 19, 00:00, 5 Years ago
;