Question

38

Puppeteer browser useragent list

rated 0 times [ 41] [ 3] / answers: 1 / hits: 17829 / 4 Years ago, sat, september 19, 2020, 12:00:00

I would like to load random list of user-agents from my default location path , for example: 'agents.json' instead of adding direct and only 1 user-agent.

agents.json

["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36", "Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/79.0.3945.73 Mobile/15E148 Safari/604.1"]

My code:

const puppeteer = require('../core/puppeteer');

const referers = require('../core/referers.json');

const referers = require('../core/agents.json');

const viewVideosInBatch = async ({ targetUrls, durationInSeconds, port }) => {

  let browser;

  try {

    browser = await puppeteer.getBrowserInstance(port);

    const randomReferer = referers[Math.floor(Math.random() * referers.length)];

    const randomAgents = agents[Math.floor(Math.random() * agents.length)];

    const page = await browser.newPage();

    browser = await page.setUserAgent({ agents: randomAgents });

    page.setDefaultTimeout(PAGE_DEFAULT_TIMEOUT * 1000);

    page.on('error', handlePageCrash(page));

    page.on('pageerror', handlePageCrash(page));

    page.setExtraHTTPHeaders({ referer: randomReferer });



    await page.setViewport({

      width: 640,

      height: 480,

      deviceScaleFactor: 1,

    });

    const ipAddr = await getCurrentIP(page);

    const targetUrlsForAction = _take(_shuffle(targetUrls), VIEW_ACTION_COUNT);

    await watchVideosInSequence(page, ipAddr, targetUrlsForAction, durationInSeconds);

    await page.close();

  } catch (error) {

    logger.warn('Entire view action in a batch failed.');

    logger.debug(error);

  } finally {

    await browser.close();

  }

};

ERROR I'm getting:

(node:1) UnhandledPromiseRejectionWarning: Error: Protocol error (Network.setUserAgentOverride): Target closed.

view_4  |     at /app/node_modules/puppeteer/lib/Connection.js:183:56

view_4  |     at new Promise (<anonymous>)

view_4  |     at CDPSession.send (/app/node_modules/puppeteer/lib/Connection.js:182:12)

view_4  |     at next (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/sourceurl/index.js:30:43)

view_4  |     at CDPSession.send (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/sourceurl/index.js:46:18)

view_4  |     at Plugin.onPageCreated (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:75:18)

view_4  |     at runMicrotasks (<anonymous>)

view_4  |     at processTicksAndRejections (internal/process/task_queues.js:97:5)

view_4  |     at async Plugin._onTargetCreated (/app/node_modules/puppeteer-extra-plugin/dist/index.cjs.js:491:17)

view_4  | (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 79)

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

daja

Add To Favorites

Follow

Total Points: 407

Total Questions: 103

Total Answers: 103

Location: Ghana

Member since Sun, Mar 27, 2022

2 Years ago

daja questions

1 How to delete a file in antd Upload component

Tue, Dec 21, 21, 00:00, 2 Years ago

1 map() with async vs promise.all()

Thu, Apr 23, 20, 00:00, 4 Years ago

1 React useEffect (with []) is called every time I change page (React Router)

Fri, Oct 25, 19, 00:00, 5 Years ago

1 Detect URL change in querystring

Fri, Sep 6, 19, 00:00, 5 Years ago

1 How to query relations with Knex.js?

Tue, Jul 23, 19, 00:00, 5 Years ago

View All

answered 4 Years ago katieh · Accepted Answer

You seem to redeclare browser when you set the UA, which you shouldn't. Also you don't need to set it as an object: as page.setUserAgent accept a string: page.setUserAgent(userAgent).

await page.setUserAgent(randomAgent); will be just fine.

const browser = await puppeteer.getBrowserInstance(port);

const randomReferer = referers[Math.floor(Math.random() * referers.length)];

const randomAgent = agents[Math.floor(Math.random() * agents.length)]; // made it singular instead of plural

const page = await browser.newPage();

await page.setUserAgent(randomAgent); // like this

page.setDefaultTimeout(PAGE_DEFAULT_TIMEOUT * 1000);

page.on('error', handlePageCrash(page));

page.on('pageerror', handlePageCrash(page));

page.setExtraHTTPHeaders({ referer: randomReferer });

Note: I think it is only a mistake in your post, but you also have two const named referers, this one should be the agents:

const referers = require('../core/agents.json');