Question

44

Web scraping using PhantomJS

rated 0 times [ 48] [ 4] / answers: 1 / hits: 16574 / 10 Years ago, sun, december 14, 2014, 12:00:00

Is there a way to execute all the JavaScripts in a webpage exactly like the browser without specifying which function to execute? In most of the examples that I saw these seem to specify which portion of JavaScript you want to execute from the scraped webpage. I need to scrape all of the contents and execute all of the JavaScripts just like a browser and get me the final executed code which we can see using google inspect?

I am sure there must be some way, but the example code from PhantomJS did not seem to have any example addressing this.

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

victorw

Add To Favorites

Follow

Total Points: 484

Total Questions: 120

Total Answers: 107

Location: Faroe Islands

Member since Thu, Apr 8, 2021

3 Years ago

answered 10 Years ago rocioblancac · Accepted Answer

You don't specify what gets executed from the page with PhantomJS. You open the page with PhantomJS and all JavaScript that is executed in Chrome or Firefox is also executed in PhantomJS. It is a full browser without a head.

There are some differences though. Clicking a download link will not trigger a download. The rendering engine which PhantomJS 1.x is based upon is nearly 4 years old, so some pages are simply rendered differently, because PhantomJS 1.x might not support that feature. (PhantomJS 2 is on the way and now in unofficial alpha status)

So you need to script every interaction that a user is doing on the page with JavaScript or CoffeeScript. You don't call page functions. You manipulate DOM elements to simulate a user interacting with the page in the browser. This needs to be done in such a crude way, because the PhantomJS API doesn't provide high-level user-like functions. If you want those, you have to look at CasperJS which is built on top of PhantomJS/SlimerJS.

There you actually have functions like click, wait, fetchText, etc.